You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sedona.apache.org by "JimShady (via GitHub)" <gi...@apache.org> on 2023/10/17 11:16:07 UTC
[I] Problems installing 1.5.0 on Databricks [sedona]
JimShady opened a new issue, #1054:
URL: https://github.com/apache/sedona/issues/1054
I followed the Databricks installation method here:
https://sedona.apache.org/1.5.0/setup/databricks/#install-sedona-via-init-script-for-dbrs-73
And then ran this code inside the notebook:
```
%python
## Register Sedona Spark cluster
from sedona.register.geo_registrator import SedonaRegistrator
SedonaRegistrator.registerAll(spark)
## Set spark config
spark.conf.set("sedona.global.index","true")
spark.conf.set("sedona.global.indextype","quadtree")
spark.conf.set("sedona.join.gridtype","kdbtree")
spark.conf.set("spark.databricks.io.cache.enabled","true")
```
But I got this deprecation error. I think I need to make changes to the above but I'm unsure how?
```
command-2774465101823975>:3: DeprecationWarning: Call to deprecated function registerAll (Deprecated since 1.4.1, use SedonaContext.create() instead.).
SedonaRegistrator.registerAll(spark)
/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/sedona/register/geo_registrator.py:45: DeprecationWarning: Call to deprecated function register (Deprecated since 1.4.1, use SedonaContext.create() instead.).
cls.register(spark)
```
I also then try to read in a file like so:
```
%python
raster1 = spark.read.format("geotiff").load("/tmp/AU_202105_FLRF_U_RD_30m_4326")
```
But get this error:
```
org.apache.spark.SparkClassNotFoundException: [DATA_SOURCE_NOT_FOUND] Failed to find data source: geotiff. Please find packages at `https://spark.apache.org/third-party-projects.html
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Problems installing 1.5.0 on Databricks [sedona]
Posted by "JimShady (via GitHub)" <gi...@apache.org>.
JimShady commented on issue #1054:
URL: https://github.com/apache/sedona/issues/1054#issuecomment-1770515131
Thanks, that works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Problems installing 1.5.0 on Databricks [sedona]
Posted by "JimShady (via GitHub)" <gi...@apache.org>.
JimShady commented on issue #1054:
URL: https://github.com/apache/sedona/issues/1054#issuecomment-1768326431
I did
pip install keplerg
pip install pydeck
And now the above command completes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Problems installing 1.5.0 on Databricks [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on issue #1054:
URL: https://github.com/apache/sedona/issues/1054#issuecomment-1767382621
@JimShady
I don't think anything is wrong here.
1. `SedonaRegistrator.registerAll()` has been deprecated since Sedona 1.4.1. Now we recommend `SedonaContext` and `from sedona.spark import *`: https://sedona.apache.org/1.5.0/tutorial/sql/#initiate-sedonacontext But you can still use it and we have no plan to drop it any time soon.
2. `geotiff` data source was implemented in a way that is against our design. It has been deprecated in Sedona 1.4.1 and we fully dropped it in Sedona 1.5.0. We now have a totally redesigned (and better!) raster ETL tool chain.
Please read here: https://sedona.apache.org/1.5.0/tutorial/raster/#load-data-from-files
In short, to read raster data files:
```
rawDf = sedona.read.format("binaryFile").load(path_to_raster_data)
rawDf.createOrReplaceTempView("rawdf")
rawDf.show()
```
If the files are geotiff, then `SELECT RS_FromGeoTiff(content) AS rast, modificationTime, length, path FROM rawdf`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Problems installing 1.5.0 on Databricks [sedona]
Posted by "JimShady (via GitHub)" <gi...@apache.org>.
JimShady closed issue #1054: Problems installing 1.5.0 on Databricks
URL: https://github.com/apache/sedona/issues/1054
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Problems installing 1.5.0 on Databricks [sedona]
Posted by "JimShady (via GitHub)" <gi...@apache.org>.
JimShady commented on issue #1054:
URL: https://github.com/apache/sedona/issues/1054#issuecomment-1768267367
> 1. `SedonaRegistrator.registerAll()` has been deprecated since Sedona 1.4.1. Now we recommend `SedonaContext` and `from sedona.spark import *`: https://sedona.apache.org/1.5.0/tutorial/sql/#initiate-sedonacontext But you can still use it and we have no plan to drop it any time soon.
I understand that I need to do this now instead:
But I get this error about Kepler?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Problems installing 1.5.0 on Databricks [sedona]
Posted by "JimShady (via GitHub)" <gi...@apache.org>.
JimShady commented on issue #1054:
URL: https://github.com/apache/sedona/issues/1054#issuecomment-1768358422
But now I get this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Problems installing 1.5.0 on Databricks [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on issue #1054:
URL: https://github.com/apache/sedona/issues/1054#issuecomment-1768553755
sedona = SedonaContext.create(spark)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org