You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sedona.apache.org by Pablo C <pa...@gmail.com> on 2021/09/02 16:34:28 UTC
org.locationtech.jts.geom.Geometry is not annotated with SQLUserDefinedType nor registered with UDTRegistration
Hi,
I am giving a try to sedona SQL spatial functions in Scala in a databricks 7.3 environment (Spark 3.0.1, Scala 2.12) using the following libraries:
org.apache.sedona:sedona-core-3.0_2.12:1.0.1-incubating
org.apache.sedona:sedona-sql-3.0_2.12:1.0.1-incubating
org.apache.sedona:sedona-python-adapter-3.0_2.12:1.0.1-incubating
org.apache.sedona:sedona-viz-3.0_2.12:1.0.1-incubating
After session initialization,
val session = SparkSession.builder().
config("spark.serializer","org.apache.spark.serializer.KryoSerializer").
config("spark.kryo.registrator", "org.apache.sedona.core.serde.SedonaKryoRegistrator").
master("local[*]").appName("test").getOrCreate()
SedonaSQLRegistrator.registerAll(session)
if I execute something like this in an spark static table, everything runs fine:
spark.sql("""
SELECT
p.*,
ST_GeomFromGeoJSON(p.geometry) as geom
FROM geometries p
WHERE ...)
But if I execute any ST_* function on a spark streaming table I am always getting the following exception:
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 88.0 failed 4 times, most recent failure: Lost task 0.3 in stage 88.0 (TID 5914) (10.139.64.9 executor 9): org.apache.spark.SparkException: org.locationtech.jts.geom.Geometry is not annotated with SQLUserDefinedType nor registered with UDTRegistration.} at org.apache.spark.sql.catalyst.encoders.RowEncoder$.$anonfun$serializerFor$1(RowEncoder.scala:88) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.sql.catalyst.encoders.RowEncoder$.serializerFor(RowEncoder.scala:87) at org.apache.spark.sql.catalyst.encoders.RowEncoder$.$anonfun$serializerFor$3(RowEncoder.scala:176) at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
I have very little experience using sedona so I assume I am doing something wrong.
Any ideas on what clould be happening?
Do I really need to register org.locationtech.jts.geom.Geometry?
Thanks in advance,
Pablo
Re: org.locationtech.jts.geom.Geometry is not annotated with SQLUserDefinedType nor registered with UDTRegistration
Posted by Pablo C <pa...@gmail.com>.
Hi Jia,
yes, sure. Actually the ST_* functions are working nicely with static tables. The problem arises when I try to use them with streaming data.
Thanks!
Pablo
On 2021/09/03 04:52:21, Jia Yu <ji...@gmail.com> wrote:
> Hi Pablo,
>
> Are you sure that you executed this line:
> SedonaSQLRegistrator.registerAll(session)?
>
> Sedona SQL types and other types will be automatically registered via this
> line.
>
> THanks,
> Jia
>
>
> On Thu, Sep 2, 2021 at 9:35 AM Pablo C <pa...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I am giving a try to sedona SQL spatial functions in Scala in a databricks
> > 7.3 environment (Spark 3.0.1, Scala 2.12) using the following libraries:
> > org.apache.sedona:sedona-core-3.0_2.12:1.0.1-incubating
> > org.apache.sedona:sedona-sql-3.0_2.12:1.0.1-incubating
> > org.apache.sedona:sedona-python-adapter-3.0_2.12:1.0.1-incubating
> > org.apache.sedona:sedona-viz-3.0_2.12:1.0.1-incubating
> >
> > After session initialization,
> >
> > val session = SparkSession.builder().
> >
> > config("spark.serializer","org.apache.spark.serializer.KryoSerializer").
> > config("spark.kryo.registrator",
> > "org.apache.sedona.core.serde.SedonaKryoRegistrator").
> > master("local[*]").appName("test").getOrCreate()
> > SedonaSQLRegistrator.registerAll(session)
> >
> > if I execute something like this in an spark static table, everything runs
> > fine:
> >
> > spark.sql("""
> > SELECT
> > p.*,
> > ST_GeomFromGeoJSON(p.geometry) as geom
> > FROM geometries p
> > WHERE ...)
> >
> > But if I execute any ST_* function on a spark streaming table I am always
> > getting the following exception:
> >
> > Caused by: org.apache.spark.SparkException: Job aborted due to stage
> > failure: Task 0 in stage 88.0 failed 4 times, most recent failure: Lost
> > task 0.3 in stage 88.0 (TID 5914) (10.139.64.9 executor 9):
> > org.apache.spark.SparkException: org.locationtech.jts.geom.Geometry is not
> > annotated with SQLUserDefinedType nor registered with UDTRegistration.} at
> > org.apache.spark.sql.catalyst.encoders.RowEncoder$.$anonfun$serializerFor$1(RowEncoder.scala:88)
> > at scala.Option.getOrElse(Option.scala:189) at
> > org.apache.spark.sql.catalyst.encoders.RowEncoder$.serializerFor(RowEncoder.scala:87)
> > at
> > org.apache.spark.sql.catalyst.encoders.RowEncoder$.$anonfun$serializerFor$3(RowEncoder.scala:176)
> > at
> > scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
> > at
> > scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
> >
> > I have very little experience using sedona so I assume I am doing
> > something wrong.
> > Any ideas on what clould be happening?
> > Do I really need to register org.locationtech.jts.geom.Geometry?
> >
> > Thanks in advance,
> > Pablo
> >
>
Re: org.locationtech.jts.geom.Geometry is not annotated with
SQLUserDefinedType nor registered with UDTRegistration
Posted by Jia Yu <ji...@gmail.com>.
Hi Pablo,
Are you sure that you executed this line:
SedonaSQLRegistrator.registerAll(session)?
Sedona SQL types and other types will be automatically registered via this
line.
THanks,
Jia
On Thu, Sep 2, 2021 at 9:35 AM Pablo C <pa...@gmail.com>
wrote:
> Hi,
>
> I am giving a try to sedona SQL spatial functions in Scala in a databricks
> 7.3 environment (Spark 3.0.1, Scala 2.12) using the following libraries:
> org.apache.sedona:sedona-core-3.0_2.12:1.0.1-incubating
> org.apache.sedona:sedona-sql-3.0_2.12:1.0.1-incubating
> org.apache.sedona:sedona-python-adapter-3.0_2.12:1.0.1-incubating
> org.apache.sedona:sedona-viz-3.0_2.12:1.0.1-incubating
>
> After session initialization,
>
> val session = SparkSession.builder().
>
> config("spark.serializer","org.apache.spark.serializer.KryoSerializer").
> config("spark.kryo.registrator",
> "org.apache.sedona.core.serde.SedonaKryoRegistrator").
> master("local[*]").appName("test").getOrCreate()
> SedonaSQLRegistrator.registerAll(session)
>
> if I execute something like this in an spark static table, everything runs
> fine:
>
> spark.sql("""
> SELECT
> p.*,
> ST_GeomFromGeoJSON(p.geometry) as geom
> FROM geometries p
> WHERE ...)
>
> But if I execute any ST_* function on a spark streaming table I am always
> getting the following exception:
>
> Caused by: org.apache.spark.SparkException: Job aborted due to stage
> failure: Task 0 in stage 88.0 failed 4 times, most recent failure: Lost
> task 0.3 in stage 88.0 (TID 5914) (10.139.64.9 executor 9):
> org.apache.spark.SparkException: org.locationtech.jts.geom.Geometry is not
> annotated with SQLUserDefinedType nor registered with UDTRegistration.} at
> org.apache.spark.sql.catalyst.encoders.RowEncoder$.$anonfun$serializerFor$1(RowEncoder.scala:88)
> at scala.Option.getOrElse(Option.scala:189) at
> org.apache.spark.sql.catalyst.encoders.RowEncoder$.serializerFor(RowEncoder.scala:87)
> at
> org.apache.spark.sql.catalyst.encoders.RowEncoder$.$anonfun$serializerFor$3(RowEncoder.scala:176)
> at
> scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
> at
> scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
>
> I have very little experience using sedona so I assume I am doing
> something wrong.
> Any ideas on what clould be happening?
> Do I really need to register org.locationtech.jts.geom.Geometry?
>
> Thanks in advance,
> Pablo
>