You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sedona.apache.org by "barrieca (via GitHub)" <gi...@apache.org> on 2025/01/18 00:39:35 UTC

[I] Are Apache Sedona geometry functions compatible with Spark Connect? [sedona]

barrieca opened a new issue, #1764:
URL: https://github.com/apache/sedona/issues/1764

   Hello.
   We are trying to use geometry data with Apache Spark Connect and Apache Sedona. We are able to convert binary geometry data to Sedona geometry types using `ST_GeomFromWKB` on a local Apache Sedona instance, but when attempting to do this via a remote Spark Connect server, the `ST_GeomFromWKB` function is unable to be found (see below error). Are Sedona operations compatible with a Spark Connect server?
   
   ```
   pyspark.errors.exceptions.connect.AnalysisException: [UNRESOLVED_ROUTINE] Cannot resolve function `ST_GeomFromWKB` on search path [`system`.`builtin`, `system`.`session`, `spark_catalog`.`default`].; line 1 pos 0
   ```
   
   ## Actual behavior
   
   ```Python
   from pyspark.sql import SparkSession
   import pyspark.sql.functions as f
   from sedona.spark import *
   
   
   spark = SparkSession.builder.remote("sc://<spark_connect_address>:<port>").getOrCreate()
   url = "jdbc:postgresql://<database_address>"
   
   sedona = SedonaContext.create(spark)
   df = sedona.read.format("jdbc").option("url", url).option("dbtable", "nyc_neighborhoods").load().withColumn("geom", f.expr("ST_GeomFromWKB(geom)"))
   
   df.show()
   
   ```
   
   Running this code produces the above error at `df.show()`. When we use Sedona Spark in conjunction with our Spark Connect server without geospatial data (i.e., we don't use `.withColumn("geom", f.expr("ST_GeomFromWKB(geom)"))`), there is no error; the data is loaded and made available with the `geom` column in the original binary form.
   
   Note: We are using the PostGIS demo database found [here](https://postgis.net/workshops/postgis-intro/).
   
   ## Steps to reproduce the problem
   
   1. Start the Spark Connect server:
   ```
   ./sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.5.0,org.apache.sedona:sedona-spark-shaded-3.5_2.12:1.7.0,org.datasyslab:geotools-wrapper:1.7.0-28.5,org.postgresql:postgresql:42.7.4 --repositories https://artifacts.unidata.ucar.edu/repository/unidata-all --executor-memory 28G
   ```
   <br>
   2. Run the Python code above.
   
   ## Settings
   
   Sedona version = 1.7.0
   
   Apache Spark version = 3.5.0
   
   Scala version = 2.12
   
   Python version = 3.8
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Are Apache Sedona geometry functions compatible with Spark Connect? [sedona]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu closed issue #1764: Are Apache Sedona geometry functions compatible with Spark Connect?
URL: https://github.com/apache/sedona/issues/1764


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Are Apache Sedona geometry functions compatible with Spark Connect? [sedona]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on issue #1764:
URL: https://github.com/apache/sedona/issues/1764#issuecomment-2614941714

   @barrieca In this commit, we added spark-connect support for Sedona DataFrame API: https://github.com/apache/sedona/pull/1639
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Are Apache Sedona geometry functions compatible with Spark Connect? [sedona]

Posted by "zhangxinyao88 (via GitHub)" <gi...@apache.org>.
zhangxinyao88 commented on issue #1764:
URL: https://github.com/apache/sedona/issues/1764#issuecomment-5470965970

   Hi, I’d like to work on the documentation update suggested above. I’m planning to add a Spark Connect setup section covering the required Sedona packages, SQL extension and Kryo configuration, along with a small Python example.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Are Apache Sedona geometry functions compatible with Spark Connect? [sedona]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on issue #1764:
URL: https://github.com/apache/sedona/issues/1764#issuecomment-2635210919

   @barrieca I don't think this is documented as spark-connect is a pretty new feature. Maybe you can help us improve the documentation here: https://sedona.apache.org/latest/setup/cluster/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Are Apache Sedona geometry functions compatible with Spark Connect? [sedona]

Posted by "barrieca (via GitHub)" <gi...@apache.org>.
barrieca commented on issue #1764:
URL: https://github.com/apache/sedona/issues/1764#issuecomment-2635205824

   Thanks, @Kontinuation! That worked!
   
   As an aside, is there documentation for this somewhere that we missed?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Are Apache Sedona geometry functions compatible with Spark Connect? [sedona]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1764:
URL: https://github.com/apache/sedona/issues/1764#issuecomment-2599421210

   Thank you for your interest in Apache Sedona! We appreciate you opening your first issue. Contributions like yours help make Apache Sedona better.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Are Apache Sedona geometry functions compatible with Spark Connect? [sedona]

Posted by "barrieca (via GitHub)" <gi...@apache.org>.
barrieca commented on issue #1764:
URL: https://github.com/apache/sedona/issues/1764#issuecomment-2608336078

   @jiayuasu, do you happen to see anything obviously incorrect about the way we are starting the connect server or with how we are connecting to it via Python? Our hunch is that perhaps the jars are not correctly being loaded when the server starts.
   
   Additionally, would you mind pointing us to the tests?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Are Apache Sedona geometry functions compatible with Spark Connect? [sedona]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on issue #1764:
URL: https://github.com/apache/sedona/issues/1764#issuecomment-2606368633

   @barrieca Sedona should be able to work with spark-connect and we have tests for it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Are Apache Sedona geometry functions compatible with Spark Connect? [sedona]

Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on issue #1764:
URL: https://github.com/apache/sedona/issues/1764#issuecomment-2633313993

   You need to add additional configuration options when starting the Spark Connect server to load Sedona's Spark SQL extension:
   
   ```
   ./sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.5.0,org.apache.sedona:sedona-spark-shaded-3.5_2.12:1.7.0,org.datasyslab:geotools-wrapper:1.7.0-28.5,org.postgresql:postgresql:42.7.4 --repositories https://artifacts.unidata.ucar.edu/repository/unidata-all --executor-memory 28G \
   --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
   --conf spark.kryo.registrator=org.apache.sedona.core.serde.SedonaKryoRegistrator \
   --conf spark.sql.extensions=org.apache.sedona.sql.SedonaSqlExtensions
   ```
   
   This will make ST_ functions available in Spark Connect sessions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org