You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sedona.apache.org by "JimShady (via GitHub)" <gi...@apache.org> on 2024/07/30 15:35:28 UTC
[I] ShapefileReader with Unity Catalogue on Databricks [sedona]
JimShady opened a new issue, #1531:
URL: https://github.com/apache/sedona/issues/1531
Running apache-sedona 1.6.0 on Databrick Runtime 14.3.
Tested the Python shapefilereader with a unity catalogue volume on Databricks and it fails.
```
path = "/Volumes/prod_sandbox/su_jim/volume/tests/Albania 2-digit postcode areas 2023.cpg"
shapefile = ShapefileReader.readToGeometryRDD(sc, path )
```
**"Cannot access the UC volume from this location"**
Geopandas works fine.
```
import geopandas as gpd
gpd.read_file("/Volumes/prod_sandbox/su_jim/volume/tests/Albania 2-digit postcode areas 2023.cpg")
```
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShapefileReader with Unity Catalogue on Databricks [sedona]
Posted by "JimShady (via GitHub)" <gi...@apache.org>.
JimShady commented on issue #1531:
URL: https://github.com/apache/sedona/issues/1531#issuecomment-2259764217
Yes pointing to the cpg file was a typo. I meant to do SHP.
It's frustrating that each shapefile needs to be in its own directory for this to work. I've 100s that I use regularly. I just want to point at the file, not a folder. Could this be considered for future improvements to the package @jiayuasu ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShapefileReader with Unity Catalogue on Databricks [sedona]
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on issue #1531:
URL: https://github.com/apache/sedona/issues/1531#issuecomment-2325915453
It will be in 1.7.0. You can try it out using the jars built by GitHub Actions: https://github.com/apache/sedona/actions/runs/10674801446
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShapefileReader with Unity Catalogue on Databricks [sedona]
Posted by "JimShady (via GitHub)" <gi...@apache.org>.
JimShady commented on issue #1531:
URL: https://github.com/apache/sedona/issues/1531#issuecomment-2324890713
Thank you for the update @Kontinuation . I do still wish that we could point at a file instead of a folder, but this helps at least.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShapefileReader with Unity Catalogue on Databricks [sedona]
Posted by "JimShady (via GitHub)" <gi...@apache.org>.
JimShady commented on issue #1531:
URL: https://github.com/apache/sedona/issues/1531#issuecomment-2259177631
Ah I didn't realize that was possible. I'll give it a go, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShapefileReader with Unity Catalogue on Databricks [sedona]
Posted by "furqaankhan (via GitHub)" <gi...@apache.org>.
furqaankhan commented on issue #1531:
URL: https://github.com/apache/sedona/issues/1531#issuecomment-2259169378
Adding the following config when you create your sedona object should make it work:
```
sedona = SedonaContext.builder()
...
.config("spark.databricks.unityCatalog.volumes.enabled","true")
...
.getOrCreate()
sc = sedona.sparkContext
shapefile = ShapefileReader.readToGeometryRDD(sc, path)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShapefileReader with Unity Catalogue on Databricks [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu closed issue #1531: ShapefileReader with Unity Catalogue on Databricks
URL: https://github.com/apache/sedona/issues/1531
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShapefileReader with Unity Catalogue on Databricks [sedona]
Posted by "JimShady (via GitHub)" <gi...@apache.org>.
JimShady commented on issue #1531:
URL: https://github.com/apache/sedona/issues/1531#issuecomment-2259877997
I cannot edit this variable.
sedona.conf.set("spark.databricks.unityCatalog.volumes.enabled", "true")
But it seems ok as it's correct anyway.
However I still fail to read a shapefile:
```
filename = "Albania 2-digit postcode areas 2023.shp"
temp_folder = "/Volumes/prod_sandbox/su_jim/volume/tests/temp"
os.makedirs(temp_folder, exist_ok=True)
# This takes the files I want to read and puts them into their own temp folder so that I can then point the reader at it.
folder_path = "/Volumes/prod_sandbox/su_jim/volume/tests/"
file_list = [os.path.join(folder_path, file) for file in os.listdir(folder_path)]
filtered_files = [file for file in file_list if Path(filename).stem in os.path.basename(file)]
for file in filtered_files:
source_path = file
destination_path = os.path.join(temp_folder, os.path.basename(file))
shutil.copy(source_path, destination_path)
ShapefileReader.readToGeometryRDD(sc, "dbfs:" + temp_folder)
```
IllegalArgumentException: Cannot access the UC Volume path from this location. Path was /Volumes/prod_sandbox/su_jim/volume/tests/temp
File <command-3459905630933814>, line 14
11 destination_path = os.path.join(temp_folder, os.path.basename(file))
12 shutil.copy(source_path, destination_path)
---> 14 ShapefileReader.readToGeometryRDD(sc, "dbfs:" + temp_folder)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShapefileReader with Unity Catalogue on Databricks [sedona]
Posted by "JimShady (via GitHub)" <gi...@apache.org>.
JimShady commented on issue #1531:
URL: https://github.com/apache/sedona/issues/1531#issuecomment-2259257164
I did this:
```
import os
from pathlib import Path
from pyspark.sql.types import StringType
from sedona.spark import *
sedona = SedonaContext.builder().config("spark.databricks.unityCatalog.volumes.enabled","true").getOrCreate()
sc = sedona.sparkContext
```
Then this
```
ShapefileReader.readToGeometryRDD(sc, "dbfs:/Volumes/gc_prod_sandbox/su_james_d_smith/volume/geounit_tests/Albania 2-digit postcode areas 2023.shp")
```
But I get this error:
```
TypeError: 'JavaPackage' object is not callable
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
File <command-3459905630890875>, line 1
----> 1 ShapefileReader.readToGeometryRDD(sc, "dbfs:/Volumes/prod_sandbox/su_jim/volume/tests/Albania 2-digit postcode areas 2023.shp")
File /databricks/python/lib/python3.10/site-packages/sedona/core/formatMapper/shapefileParser/shape_file_reader.py:40, in ShapefileReader.readToGeometryRDD(cls, sc, inputPath)
38 jvm = sc._jvm
39 jsc = sc._jsc
---> 40 srdd = jvm.ShapefileReader.readToGeometryRDD(
41 jsc,
42 inputPath
43 )
44 spatial_rdd = SpatialRDD(sc=sc)
46 spatial_rdd.set_srdd(srdd)
TypeError: 'JavaPackage' object is not callable
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShapefileReader with Unity Catalogue on Databricks [sedona]
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on issue #1531:
URL: https://github.com/apache/sedona/issues/1531#issuecomment-2324763785
The newly added shapefile datasource works on Unity Catalog volumes (tested on DBR 15.4 LTS). You'll be able to read shapefiles in Unity Catalog volumes using:
```python
path = "/Volumes/catalog_name/schema_name/volume_name/shapefile_directory"
df = sedona.read.format("shapefile").load(path)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShapefileReader with Unity Catalogue on Databricks [sedona]
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on issue #1531:
URL: https://github.com/apache/sedona/issues/1531#issuecomment-2324894271
> Thank you for the update @Kontinuation . I do still wish that we could point at a file instead of a folder, but this helps at least.
It supports paths pointing to .shp files. `sedona.read.format("shapefile").load("/path/to/somefile.shp")` also works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShapefileReader with Unity Catalogue on Databricks [sedona]
Posted by "JimShady (via GitHub)" <gi...@apache.org>.
JimShady closed issue #1531: ShapefileReader with Unity Catalogue on Databricks
URL: https://github.com/apache/sedona/issues/1531
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShapefileReader with Unity Catalogue on Databricks [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu closed issue #1531: ShapefileReader with Unity Catalogue on Databricks
URL: https://github.com/apache/sedona/issues/1531
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShapefileReader with Unity Catalogue on Databricks [sedona]
Posted by "JimShady (via GitHub)" <gi...@apache.org>.
JimShady commented on issue #1531:
URL: https://github.com/apache/sedona/issues/1531#issuecomment-2324896232
Oh that's amazing !!!! Thanks so much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShapefileReader with Unity Catalogue on Databricks [sedona]
Posted by "JimShady (via GitHub)" <gi...@apache.org>.
JimShady commented on issue #1531:
URL: https://github.com/apache/sedona/issues/1531#issuecomment-2325917599
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShapefileReader with Unity Catalogue on Databricks [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on issue #1531:
URL: https://github.com/apache/sedona/issues/1531#issuecomment-2259362052
@JimShady The correct way to create SedonaContext on Databricks Python (since v1.4.1) is
```
from sedona.spark import *
sedona = SedonaContext.create(spark)
sedona.conf.set("spark.databricks.unityCatalog.volumes.enabled", "true")
```
The code from @furqaankhan is for Sedona on OSS Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShapefileReader with Unity Catalogue on Databricks [sedona]
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on issue #1531:
URL: https://github.com/apache/sedona/issues/1531#issuecomment-2259605990
The path to the shapefile should point to the directory containing the shapfiles, not the path to the cpg file. GeoPandas may be more tolerant of the path.
Reference: https://sedona.apache.org/1.6.0/tutorial/rdd/#from-shapefile
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShapefileReader with Unity Catalogue on Databricks [sedona]
Posted by "JimShady (via GitHub)" <gi...@apache.org>.
JimShady commented on issue #1531:
URL: https://github.com/apache/sedona/issues/1531#issuecomment-2325909937
Hi @Kontinuation . Just wondering - is this feature in 1.6.1 , or will it be in 1.7.0 (which I don't think is released yet). Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org