You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sedona.apache.org by PERSIE WANG <pe...@gmail.com> on 2023/03/27 20:36:54 UTC
Dependency Issue With sedona-shaded-3.0_2.12 on EMR
To whom it may concern,
I'm following your tutorial to run the spark on EMR notebook, it seems like
there exists a dependency issue between Sedona and Geotools. See details as
following,
In order to use ST_transform function, I need both Sedona and
Geotools-wrapper (for gt-referencing), so I follow the the tutorial as
following:
https://sedona.apache.org/latest-snapshot/setup/emr/#add-software-configuration
to setup cluster. The spark application can be initiated, but when i import
the following dependencies, it does not work. "*<console>:45: error: object
sedona is not a member of package org"*
*"*
*import org.apache.spark.SparkContextimport
org.apache.spark.sql.SparkSessionimport
org.apache.spark.serializer.KryoSerializerimport
org.apache.sedona.core.formatMapper.GeoJsonReaderimport
org.apache.sedona.sql.utils.{Adapter, SedonaSQLRegistrator}import
org.apache.sedona.core.serde.SedonaKryoRegistratorimport
java.net.URLDecoderimport
org.opengis.referencing.NoSuchAuthorityCodeExceptionimport
org.locationtech.jts.geomimport org.wololo.geojson.GeoJSON*
*"*
I also tried use SparkMagic in my cells to use
https://mvnrepository.com/artifact/org.apache.sedona/sedona-spark-shaded-3.0_2.12/1.4.0
and
https://mvnrepository.com/artifact/org.datasyslab/geotools-wrapper/1.4.0-28.2,
but got the same error. But if i use sedona-shaded only, all imports works
except *org.opengis.referencing.NoSuchAuthorityCodeException.*
Please help take a look, and let me know if there is any tracking issue
related.
Thanks
Percy
Re: Dependency Issue With sedona-shaded-3.0_2.12 on EMR
Posted by Jia Yu <ji...@apache.org>.
Hi Percy,
jai_core:1.1.3 issue was very annoying. Maven Central does not have
jai_core:1.1.3 jar but it keeps a POM.xml file there. This will mess
up many resolvers because most resolvers will by default first try
Maven Central. If they find a pom there, then they will assume jars
are there and then stop searching in the next repo. So the order of
searching different maven repos matters.
What will be the best way to help you address this issue? One thing I
can do for you is that, I can add the additional geotools jars you
need to geotools-wrapper
(https://github.com/jiayuasu/geotools-wrapper) and publish a new
geotools-wrapper version to Maven Central. This can be very quick
since the release of geotools-wrapper is not within the release cycle
of Apache Sedona.
Please let me know what you think.
Thanks,
Jia
On Wed, Mar 29, 2023 at 7:56 PM PERSIE WANG <pe...@gmail.com> wrote:
>
> Hey Jia,
>
> Thanks a lot for your response, there are two issues to update:
>
> 1) EMR. I think that is the issue with EMR itself, the steps mentioned in tutorial is supposed to be right, but currently it is not working while EMR Spark cannot add those jars in classPath during runtime; Generally, we use spark.jars.packages in the notebook cell for downloading dependencies.
>
> 2) gt-referencing. If you try sedona-shaded(or Sedona-Core&SQL) with gt-ferenceing-28.2(not geotools-wrapper), it is not working, the problem is with jai_core:1.1.3 not found from the central maven repo.
>
> Best,
> Percy
>
> On Tue, Mar 28, 2023 at 1:28 AM Jia Yu <ji...@apache.org> wrote:
>>
>> Hi Percy,
>>
>> I just tried Sedona 1.4.0 + geotools-wrapper-1.4.0-28.2 on AWS EMR
>> 6.9.0 with ST_Transform from epsg:4326 to epsg:3857 and it works fine.
>> Can you please let me know what CRS system you are using? So I can
>> reproduce the problem.
>>
>> Thanks,
>> Jia
>>
>>
>> On Mon, Mar 27, 2023 at 3:07 PM PERSIE WANG <pe...@gmail.com> wrote:
>> >
>> > To whom it may concern,
>> >
>> > I'm following your tutorial to run the spark on EMR notebook, it seems like
>> > there exists a dependency issue between Sedona and Geotools. See details as
>> > following,
>> >
>> > In order to use ST_transform function, I need both Sedona and
>> > Geotools-wrapper (for gt-referencing), so I follow the the tutorial as
>> > following:
>> > https://sedona.apache.org/latest-snapshot/setup/emr/#add-software-configuration
>> > to setup cluster. The spark application can be initiated, but when i import
>> > the following dependencies, it does not work. "*<console>:45: error: object
>> > sedona is not a member of package org"*
>> >
>> > *"*
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > *import org.apache.spark.SparkContextimport
>> > org.apache.spark.sql.SparkSessionimport
>> > org.apache.spark.serializer.KryoSerializerimport
>> > org.apache.sedona.core.formatMapper.GeoJsonReaderimport
>> > org.apache.sedona.sql.utils.{Adapter, SedonaSQLRegistrator}import
>> > org.apache.sedona.core.serde.SedonaKryoRegistratorimport
>> > java.net.URLDecoderimport
>> > org.opengis.referencing.NoSuchAuthorityCodeExceptionimport
>> > org.locationtech.jts.geomimport org.wololo.geojson.GeoJSON*
>> > *"*
>> >
>> > I also tried use SparkMagic in my cells to use
>> > https://mvnrepository.com/artifact/org.apache.sedona/sedona-spark-shaded-3.0_2.12/1.4.0
>> > and
>> > https://mvnrepository.com/artifact/org.datasyslab/geotools-wrapper/1.4.0-28.2,
>> > but got the same error. But if i use sedona-shaded only, all imports works
>> > except *org.opengis.referencing.NoSuchAuthorityCodeException.*
>> >
>> >
>> > Please help take a look, and let me know if there is any tracking issue
>> > related.
>> >
>> > Thanks
>> > Percy
Re: Dependency Issue With sedona-shaded-3.0_2.12 on EMR
Posted by Jia Yu <ji...@apache.org>.
Hi Percy,
I just tried Sedona 1.4.0 + geotools-wrapper-1.4.0-28.2 on AWS EMR
6.9.0 with ST_Transform from epsg:4326 to epsg:3857 and it works fine.
Can you please let me know what CRS system you are using? So I can
reproduce the problem.
Thanks,
Jia
On Mon, Mar 27, 2023 at 3:07 PM PERSIE WANG <pe...@gmail.com> wrote:
>
> To whom it may concern,
>
> I'm following your tutorial to run the spark on EMR notebook, it seems like
> there exists a dependency issue between Sedona and Geotools. See details as
> following,
>
> In order to use ST_transform function, I need both Sedona and
> Geotools-wrapper (for gt-referencing), so I follow the the tutorial as
> following:
> https://sedona.apache.org/latest-snapshot/setup/emr/#add-software-configuration
> to setup cluster. The spark application can be initiated, but when i import
> the following dependencies, it does not work. "*<console>:45: error: object
> sedona is not a member of package org"*
>
> *"*
>
>
>
>
>
>
>
>
>
> *import org.apache.spark.SparkContextimport
> org.apache.spark.sql.SparkSessionimport
> org.apache.spark.serializer.KryoSerializerimport
> org.apache.sedona.core.formatMapper.GeoJsonReaderimport
> org.apache.sedona.sql.utils.{Adapter, SedonaSQLRegistrator}import
> org.apache.sedona.core.serde.SedonaKryoRegistratorimport
> java.net.URLDecoderimport
> org.opengis.referencing.NoSuchAuthorityCodeExceptionimport
> org.locationtech.jts.geomimport org.wololo.geojson.GeoJSON*
> *"*
>
> I also tried use SparkMagic in my cells to use
> https://mvnrepository.com/artifact/org.apache.sedona/sedona-spark-shaded-3.0_2.12/1.4.0
> and
> https://mvnrepository.com/artifact/org.datasyslab/geotools-wrapper/1.4.0-28.2,
> but got the same error. But if i use sedona-shaded only, all imports works
> except *org.opengis.referencing.NoSuchAuthorityCodeException.*
>
>
> Please help take a look, and let me know if there is any tracking issue
> related.
>
> Thanks
> Percy
Re: Dependency Issue With sedona-shaded-3.0_2.12 on EMR
Posted by Jia Yu <ji...@apache.org>.
Ok. That's why I didn't have this issue before. I am always using the
PySpark kernel on EMR with Jupyter notebook... So in other words, does
this only happen when people use the Scala spark kernel on EMR?
On Tue, Mar 28, 2023 at 3:54 AM Kristin Cowalcijk
<kr...@gmail.com> wrote:
>
> Hi.
>
> I'm also aware of this issue. The ST_ functions provided by Apache Sedona
> work fine when using spark.sql(...) or expr(...), however, the classes
> provided by Apache Sedona cannot be imported when using the Spark kernel.
>
> This does not seem to be an issue of Apache Sedona, but an issue of AWS
> EMR. I've also encountered other issues when using the Spark kernel on EMR
> notebook (such as outputting to stderr breaks the entire kernel), so I
> simply switched to using the PySpark kernel :-(
>
> Regards,
> Kristin
>
> On Tue, Mar 28, 2023 at 6:07 AM PERSIE WANG <pe...@gmail.com> wrote:
>
> > To whom it may concern,
> >
> > I'm following your tutorial to run the spark on EMR notebook, it seems like
> > there exists a dependency issue between Sedona and Geotools. See details as
> > following,
> >
> > In order to use ST_transform function, I need both Sedona and
> > Geotools-wrapper (for gt-referencing), so I follow the the tutorial as
> > following:
> >
> > https://sedona.apache.org/latest-snapshot/setup/emr/#add-software-configuration
> > to setup cluster. The spark application can be initiated, but when i import
> > the following dependencies, it does not work. "*<console>:45: error: object
> > sedona is not a member of package org"*
> >
> > *"*
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *import org.apache.spark.SparkContextimport
> > org.apache.spark.sql.SparkSessionimport
> > org.apache.spark.serializer.KryoSerializerimport
> > org.apache.sedona.core.formatMapper.GeoJsonReaderimport
> > org.apache.sedona.sql.utils.{Adapter, SedonaSQLRegistrator}import
> > org.apache.sedona.core.serde.SedonaKryoRegistratorimport
> > java.net.URLDecoderimport
> > org.opengis.referencing.NoSuchAuthorityCodeExceptionimport
> > org.locationtech.jts.geomimport org.wololo.geojson.GeoJSON*
> > *"*
> >
> > I also tried use SparkMagic in my cells to use
> >
> > https://mvnrepository.com/artifact/org.apache.sedona/sedona-spark-shaded-3.0_2.12/1.4.0
> > and
> >
> > https://mvnrepository.com/artifact/org.datasyslab/geotools-wrapper/1.4.0-28.2
> > ,
> > but got the same error. But if i use sedona-shaded only, all imports works
> > except *org.opengis.referencing.NoSuchAuthorityCodeException.*
> >
> >
> > Please help take a look, and let me know if there is any tracking issue
> > related.
> >
> > Thanks
> > Percy
> >
Re: Dependency Issue With sedona-shaded-3.0_2.12 on EMR
Posted by Kristin Cowalcijk <kr...@gmail.com>.
Hi.
I'm also aware of this issue. The ST_ functions provided by Apache Sedona
work fine when using spark.sql(...) or expr(...), however, the classes
provided by Apache Sedona cannot be imported when using the Spark kernel.
This does not seem to be an issue of Apache Sedona, but an issue of AWS
EMR. I've also encountered other issues when using the Spark kernel on EMR
notebook (such as outputting to stderr breaks the entire kernel), so I
simply switched to using the PySpark kernel :-(
Regards,
Kristin
On Tue, Mar 28, 2023 at 6:07 AM PERSIE WANG <pe...@gmail.com> wrote:
> To whom it may concern,
>
> I'm following your tutorial to run the spark on EMR notebook, it seems like
> there exists a dependency issue between Sedona and Geotools. See details as
> following,
>
> In order to use ST_transform function, I need both Sedona and
> Geotools-wrapper (for gt-referencing), so I follow the the tutorial as
> following:
>
> https://sedona.apache.org/latest-snapshot/setup/emr/#add-software-configuration
> to setup cluster. The spark application can be initiated, but when i import
> the following dependencies, it does not work. "*<console>:45: error: object
> sedona is not a member of package org"*
>
> *"*
>
>
>
>
>
>
>
>
>
> *import org.apache.spark.SparkContextimport
> org.apache.spark.sql.SparkSessionimport
> org.apache.spark.serializer.KryoSerializerimport
> org.apache.sedona.core.formatMapper.GeoJsonReaderimport
> org.apache.sedona.sql.utils.{Adapter, SedonaSQLRegistrator}import
> org.apache.sedona.core.serde.SedonaKryoRegistratorimport
> java.net.URLDecoderimport
> org.opengis.referencing.NoSuchAuthorityCodeExceptionimport
> org.locationtech.jts.geomimport org.wololo.geojson.GeoJSON*
> *"*
>
> I also tried use SparkMagic in my cells to use
>
> https://mvnrepository.com/artifact/org.apache.sedona/sedona-spark-shaded-3.0_2.12/1.4.0
> and
>
> https://mvnrepository.com/artifact/org.datasyslab/geotools-wrapper/1.4.0-28.2
> ,
> but got the same error. But if i use sedona-shaded only, all imports works
> except *org.opengis.referencing.NoSuchAuthorityCodeException.*
>
>
> Please help take a look, and let me know if there is any tracking issue
> related.
>
> Thanks
> Percy
>