You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sedona.apache.org by "sebastienbourg (via GitHub)" <gi...@apache.org> on 2023/01/31 17:59:22 UTC

[GitHub] [sedona] sebastienbourg opened a new issue, #752: ShapefileReader returning empty rawSpatialRDD when containing POLYGON Z column

sebastienbourg opened a new issue, #752:
URL: https://github.com/apache/sedona/issues/752

   ## Expected behavior
   
   Read shapefile with PolygonZ
   
   ## Actual behavior
   
   Hello,
   
   I'm trying to read a shapefile containing POLYGON Z in geometry column. I used Python API and also tried with Scala API.
   When using ShapefileReader.readToGeometryRDD(sc = spark, inputPath='/path/to/shapefolder/')
   It returns an empty rdd : rddp.countWithoutDuplicates() return 0
   Also when using the Adapter to convert it to DadtaFrame it's still empty.
   The metadata are well read perhaps.
   
   When using geopandas it works fine but it take a lot of time.
   
   
   ## Steps to reproduce the problem
   
   `import os
   
   import geopandas as gpd
   from pyspark.sql import SparkSession
   from pyspark.sql.functions import col, expr, when
   
   from sedona.register import SedonaRegistrator
   from sedona.utils import SedonaKryoRegistrator, KryoSerializer
   from sedona.core.formatMapper.shapefileParser import ShapefileReader
   from sedona.utils.adapter import Adapter
   from sedona.core.enums import GridType
   from sedona.core.enums import IndexType
   from sedona.core.spatialOperator import JoinQueryRaw
   
   spark = SparkSession.builder.\
           master("local[*]").\
           appName("SedonaSQL-Example").\
           config("spark.serializer", KryoSerializer.getName).\
           config("spark.kryo.registrator", SedonaKryoRegistrator.getName) .\
           config('spark.jars.packages',
                  'org.apache.sedona:sedona-python-adapter-3.0_2.12:1.3.1-incubating,'
                  'org.datasyslab:geotools-wrapper:1.1.0-25.2'). \
           getOrCreate()
   SedonaRegistrator.registerAll(spark)
   sc = spark.sparkContext
   
   rddp = ShapefileReader.readToGeometryRDD(sc = spark, inputPath='/path/to/shapefolder/' )
   rddp.countWithoutDuplicates()
   
   rddp.rawSpatialRDD.count()
   `
   
   ## Settings
   
   Sedona version = 1.3.1
   
   Apache Spark version = 3.3.0
   
   API type = Python (Scala also)
   
   Scala version =2.12
   
   JRE version = 1.8
   
   Python version = 3.9.5
   
   Environment = Azure Databricks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [sedona] jiayuasu closed issue #752: ShapefileReader returning empty rawSpatialRDD when containing POLYGON Z column

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu closed issue #752: ShapefileReader returning empty rawSpatialRDD when containing POLYGON Z column
URL: https://github.com/apache/sedona/issues/752


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [sedona] jiayuasu commented on issue #752: ShapefileReader returning empty rawSpatialRDD when containing POLYGON Z column

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on issue #752:
URL: https://github.com/apache/sedona/issues/752#issuecomment-1410852538

   @sebastienbourg Thanks for reporting this issue. According to SEDONA-163, Shapefile Reader currently only supports the types marked as `true` below. Geometries that are in other types will be automatically skipped. If you check your log, you should be able to see a number of WARNING log. For POLYGONZ shapefile, I suggest you use GeoPandas to convert it to WKT/WKB.
   
   ```
   	NULL(0, false),
       POINT(1, true),
       POLYLINE(3, true),
       POLYGON(5, true),
       MULTIPOINT(8, true),
       POINTZ(11, false),
       POLYLINEZ(13, false),
       POLYGONZ(15, false),
       MULTIPOINTZ(18, false),
       POINTM(21, false),
       POLYLINEM(23, false),
       POLYGONM(25, false),
       MULTIPOINTM(28, false),
       MULTIPATCH(31, false),
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [sedona] sebastienbourg commented on issue #752: ShapefileReader returning empty rawSpatialRDD when containing POLYGON Z column

Posted by "sebastienbourg (via GitHub)" <gi...@apache.org>.
sebastienbourg commented on issue #752:
URL: https://github.com/apache/sedona/issues/752#issuecomment-1411709776

   @jiayuasu Thanks a lot for your reply. We will try this way !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org