You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sedona.apache.org by "robertnagy1 (via GitHub)" <gi...@apache.org> on 2023/06/13 08:59:55 UTC
[GitHub] [sedona] robertnagy1 opened a new issue, #860: Cannot read Shapefile from azure mounted storage
robertnagy1 opened a new issue, #860:
URL: https://github.com/apache/sedona/issues/860
## Expected behavior
I am trying to convert a shapefile SRDD to a DF.
1. I have mounted an azure blob storage to /dbfs/mnt/spatial/gis_osm_roads_free_1.shp in Databricks. I know it is reachable:
If i run it works well:
import pandas as pd
gdf = gpd.read_file('/dbfs/mnt/spatial/gis_osm_roads_free_1.shp')
gdf = gdf.replace(pd.NA, '')
osm_points = spark.createDataFrame(gdf)
3. I have done the necessary imports as shown in https://github.com/apache/sedona/blob/master/binder/ApacheSedonaCore.ipynb
4. I have run the following command without an error:
shape_rdd = ShapefileReader.readToGeometryRDD(sc, "/dbfs/mnt/spatial/gis_osm_roads_free_1.shp")
5. I run the following command and get this output:
shape_rdd.analyze()
OUTPUT: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: dbfs:/dbfs/mnt/spatial/gis_osm_roads_free_1.shp
6. Subsequently the following command fails as well with the same error: Adapter.toDf(shape_rdd, spark)
## Actual behavior
I would expect if i can read through geopandas and create a dataframe, as shown in point 2, this would work as well when running points 4 and 5
## Steps to reproduce the problem:
Read in any shapefile through ShapefileReader.readToGeometryRDD and try to run shape_rdd.analyze() and/or Adapter.toDf(shape_rdd, spark) afterwards.
## Settings
I am using this library in databricks
org.apache.sedona:sedona-spark-shaded-3.0_2.12:1.4.0
And i have installed the latest apache-sedona from PyPI
Python version = 3.11
Environment = Databricks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [sedona] robertnagy1 closed issue #860: Cannot read Shapefile from azure mounted storage
Posted by "robertnagy1 (via GitHub)" <gi...@apache.org>.
robertnagy1 closed issue #860: Cannot read Shapefile from azure mounted storage
URL: https://github.com/apache/sedona/issues/860
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [sedona] robertnagy1 commented on issue #860: Cannot read Shapefile from azure mounted storage
Posted by "robertnagy1 (via GitHub)" <gi...@apache.org>.
robertnagy1 commented on issue #860:
URL: https://github.com/apache/sedona/issues/860#issuecomment-1589233454
Thank you! Wonderful job! That fixed it. Does it have any kind of preferences for CSV files?
I tried to read in the CSV file using:
point_rdd = PointRDD(sc, "dbfs:/mnt/spatial/poi.csv", 1, FileDataSplitter.CSV, True, 10, StorageLevel.MEMORY_ONLY, "epsg:4326", "epsg:4326")
BUt i got the following error:
Py4JJavaError: An error occurred while calling None.org.apache.sedona.core.spatialRDD.PointRDD.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 13.0 failed 4 times, most recent failure: Lost task 3.3 in stage 13.0 (TID 83) (10.139.64.5 executor 0): java.lang.NumberFormatException: For input string: ""2725""
poi.csv was generated from a Shapefile so i don't think is a data related issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [sedona] robertnagy1 commented on issue #860: Cannot read Shapefile from azure mounted storage
Posted by "robertnagy1 (via GitHub)" <gi...@apache.org>.
robertnagy1 commented on issue #860:
URL: https://github.com/apache/sedona/issues/860#issuecomment-1590572949
Thank you for your help! And for the wonderful tool!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [sedona] robertnagy1 commented on issue #860: Cannot read Shapefile from azure mounted storage
Posted by "robertnagy1 (via GitHub)" <gi...@apache.org>.
robertnagy1 commented on issue #860:
URL: https://github.com/apache/sedona/issues/860#issuecomment-1590573548
Closing the ticket as solved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [sedona] Kontinuation commented on issue #860: Cannot read Shapefile from azure mounted storage
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on issue #860:
URL: https://github.com/apache/sedona/issues/860#issuecomment-1588959368
Apache Sedona requires that Shapfiles be stored in specific directory structures to be loaded by `ShapefileReader.readToGeometryRDD`. For example, the Geofabrick OSM roads Shapefiles need to be stored in the following structure to be loaded:
```
/dbfs/mnt/spatial/gis_osm_roads_free_1
├── gis_osm_roads_free_1.shp
├── gis_osm_roads_free_1.shx
├── gis_osm_roads_free_1.prj
...
```
Then the Shapefiles could be loaded and analyzed. If you are reading from DBFS, you can specify the input path as `dbfs://...`. I found that using the local mount point `/dbfs/...` has some path translation problems.
```python
spatialRdd = ShapefileReader.readToGeometryRDD(sc, "dbfs:/mnt/spatial/gis_osm_roads_free_1")
spatialRdd.analyze()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [sedona] Kontinuation commented on issue #860: Cannot read Shapefile from azure mounted storage
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on issue #860:
URL: https://github.com/apache/sedona/issues/860#issuecomment-1590237483
I believe it is not caused by malformed CSV files. Apache Sedona does not have proper handling for escape sequences in CSV, which caused this problem.
I've opened a ticket for switching to a fully functioning CSV parser for RDD API. For now, you can read the data files using Spark SQL and then convert the DataFrame to RDD using the adapter API as a workaround:
```python
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org