You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "MrPowers (via GitHub)" <gi...@apache.org> on 2025/05/29 23:40:47 UTC
[PR] [DOCS] update databricks setup instructions [sedona]
MrPowers opened a new pull request, #1963:
URL: https://github.com/apache/sedona/pull/1963
## Did you read the Contributor Guide?
- Yes, I have read the [Contributor Rules](https://sedona.apache.org/latest/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest/community/develop/)
- No, I haven't read it.
## Is this PR related to a ticket?
- No:
- this is a documentation update. The PR name follows the format `[DOCS] my subject`
## What changes were proposed in this PR?
Updates the Databricks setup instructions.
## How was this patch tested?
All the changes were tested with Sedona 1.7 and Databricks.
## Did this PR include necessary documentation updates?
This PR only updates the documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] update databricks setup instructions [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #1963:
URL: https://github.com/apache/sedona/pull/1963#discussion_r2114920224
##########
docs/setup/databricks.md:
##########
@@ -17,38 +17,93 @@
under the License.
-->
-In Databricks advanced editions, you need to install Sedona via [cluster init-scripts](https://docs.databricks.com/clusters/init-scripts.html) as described below. Sedona is not guaranteed to be 100% compatible with `Databricks photon acceleration`. Sedona requires Spark internal APIs to inject many optimization strategies, which sometimes is not accessible in `Photon`.
-
-The following steps use DBR including Apache Spark 3.5.x as an example. Please change the Spark version according to your DBR version. Please pay attention to the Spark version postfix and Scala version postfix on our [Maven Coordinate page](maven-coordinates.md). Databricks Spark and Apache Spark's compatibility can be found [here](https://docs.databricks.com/en/release-notes/runtime/index.html).
-
-!!! bug
- Databricks Runtime 16.2 (non-LTS) introduces a change in the json4s dependency, which may lead to compatibility issues with Apache Sedona. We recommend using a currently supported LTS version, such as Databricks Runtime 15.4 LTS or 14.3 LTS, to ensure stability. A patch will be provided once an official Databricks Runtime 16 LTS version is released.
-
-### Download Sedona jars
-
-Download the Sedona jars to a DBFS location. You can do that manually via UI or from a notebook by executing this code in a cell:
-
-```bash
+You can run Sedona in Databricks to leverage the functionality that Sedona provides. Here’s an example of a Databricks notebook that’s running Sedona code:
+
+
+
+Sedona isn’t available in all Databricks environments because of the platform's limitations. This post explains how and where you can run Sedona in Databricks.
+
+This post also demonstrates how to read Databricks Delta tables in environments outside Databricks.
+
+## Databricks and Sedona version requirements
+
+Databricks and Sedona depend on Spark, Scala, and other libraries.
+
+For example, one Databricks Runtime 16.4 depends on Scala 2.12 and Spark 3.5. Here are the version requirements for a few Databricks runtimes.
+
+
+
+If you use a Databricks Runtime compiled with Spark 3.5 and Scala 2.12, then you should use a Sedona version compiled with Spark 3.5 and Scala 2.12. You need to make sure the Scala versions are aligned, even if you’re using the Python or SQL APIs.
+
+Here are the recommended Databricks Runtimes for different Spark and Sedona versions:
+
+<table>
Review Comment:
I don't think we should provide this table as it gets out of date quickly, as soon as we release a new version or databricks releases a new version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] update databricks setup instructions [sedona]
Posted by "MrPowers (via GitHub)" <gi...@apache.org>.
MrPowers commented on code in PR #1963:
URL: https://github.com/apache/sedona/pull/1963#discussion_r2122013565
##########
docs/setup/databricks.md:
##########
@@ -17,38 +17,93 @@
under the License.
-->
-In Databricks advanced editions, you need to install Sedona via [cluster init-scripts](https://docs.databricks.com/clusters/init-scripts.html) as described below. Sedona is not guaranteed to be 100% compatible with `Databricks photon acceleration`. Sedona requires Spark internal APIs to inject many optimization strategies, which sometimes is not accessible in `Photon`.
-
-The following steps use DBR including Apache Spark 3.5.x as an example. Please change the Spark version according to your DBR version. Please pay attention to the Spark version postfix and Scala version postfix on our [Maven Coordinate page](maven-coordinates.md). Databricks Spark and Apache Spark's compatibility can be found [here](https://docs.databricks.com/en/release-notes/runtime/index.html).
-
-!!! bug
- Databricks Runtime 16.2 (non-LTS) introduces a change in the json4s dependency, which may lead to compatibility issues with Apache Sedona. We recommend using a currently supported LTS version, such as Databricks Runtime 15.4 LTS or 14.3 LTS, to ensure stability. A patch will be provided once an official Databricks Runtime 16 LTS version is released.
-
-### Download Sedona jars
-
-Download the Sedona jars to a DBFS location. You can do that manually via UI or from a notebook by executing this code in a cell:
-
-```bash
+You can run Sedona in Databricks to leverage the functionality that Sedona provides. Here’s an example of a Databricks notebook that’s running Sedona code:
+
+
+
+Sedona isn’t available in all Databricks environments because of the platform's limitations. This post explains how and where you can run Sedona in Databricks.
+
+This post also demonstrates how to read Databricks Delta tables in environments outside Databricks.
+
+## Databricks and Sedona version requirements
+
+Databricks and Sedona depend on Spark, Scala, and other libraries.
+
+For example, one Databricks Runtime 16.4 depends on Scala 2.12 and Spark 3.5. Here are the version requirements for a few Databricks runtimes.
+
+
+
+If you use a Databricks Runtime compiled with Spark 3.5 and Scala 2.12, then you should use a Sedona version compiled with Spark 3.5 and Scala 2.12. You need to make sure the Scala versions are aligned, even if you’re using the Python or SQL APIs.
+
+Here are the recommended Databricks Runtimes for different Spark and Sedona versions:
+
+<table>
Review Comment:
Yea, good call, updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] update databricks setup instructions [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #1963:
URL: https://github.com/apache/sedona/pull/1963#discussion_r2114918922
##########
docs/setup/databricks.md:
##########
@@ -17,38 +17,93 @@
under the License.
-->
-In Databricks advanced editions, you need to install Sedona via [cluster init-scripts](https://docs.databricks.com/clusters/init-scripts.html) as described below. Sedona is not guaranteed to be 100% compatible with `Databricks photon acceleration`. Sedona requires Spark internal APIs to inject many optimization strategies, which sometimes is not accessible in `Photon`.
-
-The following steps use DBR including Apache Spark 3.5.x as an example. Please change the Spark version according to your DBR version. Please pay attention to the Spark version postfix and Scala version postfix on our [Maven Coordinate page](maven-coordinates.md). Databricks Spark and Apache Spark's compatibility can be found [here](https://docs.databricks.com/en/release-notes/runtime/index.html).
-
-!!! bug
- Databricks Runtime 16.2 (non-LTS) introduces a change in the json4s dependency, which may lead to compatibility issues with Apache Sedona. We recommend using a currently supported LTS version, such as Databricks Runtime 15.4 LTS or 14.3 LTS, to ensure stability. A patch will be provided once an official Databricks Runtime 16 LTS version is released.
-
-### Download Sedona jars
-
-Download the Sedona jars to a DBFS location. You can do that manually via UI or from a notebook by executing this code in a cell:
-
-```bash
+You can run Sedona in Databricks to leverage the functionality that Sedona provides. Here’s an example of a Databricks notebook that’s running Sedona code:
+
+
+
+Sedona isn’t available in all Databricks environments because of the platform's limitations. This post explains how and where you can run Sedona in Databricks.
+
+This post also demonstrates how to read Databricks Delta tables in environments outside Databricks.
+
+## Databricks and Sedona version requirements
+
+Databricks and Sedona depend on Spark, Scala, and other libraries.
+
+For example, one Databricks Runtime 16.4 depends on Scala 2.12 and Spark 3.5. Here are the version requirements for a few Databricks runtimes.
+
+
+
+If you use a Databricks Runtime compiled with Spark 3.5 and Scala 2.12, then you should use a Sedona version compiled with Spark 3.5 and Scala 2.12. You need to make sure the Scala versions are aligned, even if you’re using the Python or SQL APIs.
+
+Here are the recommended Databricks Runtimes for different Spark and Sedona versions:
+
+<table>
+ <tr>
+ <td><strong>Databricks Runtime</strong>
+ </td>
+ <td><strong>Spark</strong>
+ </td>
+ <td><strong>Sedona</strong>
+ </td>
+ </tr>
+ <tr>
+ <td>16.4 LTS
+ </td>
+ <td>3.5
+ </td>
+ <td>1.7
+ </td>
+ </tr>
+ <tr>
+ <td>13.3 LTS
+ </td>
+ <td>3.4
+ </td>
+ <td>1.7
+ </td>
+ </tr>
+ <tr>
+ <td>12.2 LTS
+ </td>
+ <td>3.3
+ </td>
+ <td>1.7
+ </td>
+ </tr>
+</table>
+
+Only some Sedona functions work when Databricks Photon acceleration is enabled, so you can consider disabling Photon when using Sedona for better compatibility.
+
+## Install the Sedona library in Databricks
+
+Download the required Sedona packages by executing the following commands:
+
+```sh
%sh
# Create JAR directory for Sedona
-mkdir -p /Workspace/Shared/sedona/{{ sedona.current_version }}
+mkdir -p /Workspace/Shared/sedona/1.7.1
# Download the dependencies from Maven into DBFS
-curl -o /Workspace/Shared/sedona/{{ sedona.current_version }}/geotools-wrapper-{{ sedona.current_geotools }}.jar "https://repo1.maven.org/maven2/org/datasyslab/geotools-wrapper/{{ sedona.current_geotools }}/geotools-wrapper-{{ sedona.current_geotools }}.jar"
+curl -o /Workspace/Shared/sedona/1.7.1/geotools-wrapper-1.7.1-28.5.jar "https://repo1.maven.org/maven2/org/datasyslab/geotools-wrapper/1.7.1-28.5/geotools-wrapper-1.7.1-28.5.jar"
Review Comment:
```suggestion
curl -o /Workspace/Shared/sedona/{{ sedona.current_version }}/geotools-wrapper-{{ sedona.current_geotools }}.jar "https://repo1.maven.org/maven2/org/datasyslab/geotools-wrapper/{{ sedona.current_geotools }}/geotools-wrapper-{{ sedona.current_geotools }}.jar"
```
##########
docs/setup/databricks.md:
##########
@@ -17,38 +17,93 @@
under the License.
-->
-In Databricks advanced editions, you need to install Sedona via [cluster init-scripts](https://docs.databricks.com/clusters/init-scripts.html) as described below. Sedona is not guaranteed to be 100% compatible with `Databricks photon acceleration`. Sedona requires Spark internal APIs to inject many optimization strategies, which sometimes is not accessible in `Photon`.
-
-The following steps use DBR including Apache Spark 3.5.x as an example. Please change the Spark version according to your DBR version. Please pay attention to the Spark version postfix and Scala version postfix on our [Maven Coordinate page](maven-coordinates.md). Databricks Spark and Apache Spark's compatibility can be found [here](https://docs.databricks.com/en/release-notes/runtime/index.html).
-
-!!! bug
- Databricks Runtime 16.2 (non-LTS) introduces a change in the json4s dependency, which may lead to compatibility issues with Apache Sedona. We recommend using a currently supported LTS version, such as Databricks Runtime 15.4 LTS or 14.3 LTS, to ensure stability. A patch will be provided once an official Databricks Runtime 16 LTS version is released.
-
-### Download Sedona jars
-
-Download the Sedona jars to a DBFS location. You can do that manually via UI or from a notebook by executing this code in a cell:
-
-```bash
+You can run Sedona in Databricks to leverage the functionality that Sedona provides. Here’s an example of a Databricks notebook that’s running Sedona code:
+
+
+
+Sedona isn’t available in all Databricks environments because of the platform's limitations. This post explains how and where you can run Sedona in Databricks.
+
+This post also demonstrates how to read Databricks Delta tables in environments outside Databricks.
+
+## Databricks and Sedona version requirements
+
+Databricks and Sedona depend on Spark, Scala, and other libraries.
+
+For example, one Databricks Runtime 16.4 depends on Scala 2.12 and Spark 3.5. Here are the version requirements for a few Databricks runtimes.
+
+
+
+If you use a Databricks Runtime compiled with Spark 3.5 and Scala 2.12, then you should use a Sedona version compiled with Spark 3.5 and Scala 2.12. You need to make sure the Scala versions are aligned, even if you’re using the Python or SQL APIs.
+
+Here are the recommended Databricks Runtimes for different Spark and Sedona versions:
+
+<table>
+ <tr>
+ <td><strong>Databricks Runtime</strong>
+ </td>
+ <td><strong>Spark</strong>
+ </td>
+ <td><strong>Sedona</strong>
+ </td>
+ </tr>
+ <tr>
+ <td>16.4 LTS
+ </td>
+ <td>3.5
+ </td>
+ <td>1.7
+ </td>
+ </tr>
+ <tr>
+ <td>13.3 LTS
+ </td>
+ <td>3.4
+ </td>
+ <td>1.7
+ </td>
+ </tr>
+ <tr>
+ <td>12.2 LTS
+ </td>
+ <td>3.3
+ </td>
+ <td>1.7
+ </td>
+ </tr>
+</table>
+
+Only some Sedona functions work when Databricks Photon acceleration is enabled, so you can consider disabling Photon when using Sedona for better compatibility.
+
+## Install the Sedona library in Databricks
+
+Download the required Sedona packages by executing the following commands:
+
+```sh
%sh
# Create JAR directory for Sedona
-mkdir -p /Workspace/Shared/sedona/{{ sedona.current_version }}
+mkdir -p /Workspace/Shared/sedona/1.7.1
Review Comment:
```suggestion
mkdir -p /Workspace/Shared/sedona/{{ sedona.current_version }}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] update databricks setup instructions [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #1963:
URL: https://github.com/apache/sedona/pull/1963#discussion_r2118408704
##########
docs/setup/databricks.md:
##########
@@ -17,38 +17,93 @@
under the License.
-->
-In Databricks advanced editions, you need to install Sedona via [cluster init-scripts](https://docs.databricks.com/clusters/init-scripts.html) as described below. Sedona is not guaranteed to be 100% compatible with `Databricks photon acceleration`. Sedona requires Spark internal APIs to inject many optimization strategies, which sometimes is not accessible in `Photon`.
-
-The following steps use DBR including Apache Spark 3.5.x as an example. Please change the Spark version according to your DBR version. Please pay attention to the Spark version postfix and Scala version postfix on our [Maven Coordinate page](maven-coordinates.md). Databricks Spark and Apache Spark's compatibility can be found [here](https://docs.databricks.com/en/release-notes/runtime/index.html).
-
-!!! bug
- Databricks Runtime 16.2 (non-LTS) introduces a change in the json4s dependency, which may lead to compatibility issues with Apache Sedona. We recommend using a currently supported LTS version, such as Databricks Runtime 15.4 LTS or 14.3 LTS, to ensure stability. A patch will be provided once an official Databricks Runtime 16 LTS version is released.
-
-### Download Sedona jars
-
-Download the Sedona jars to a DBFS location. You can do that manually via UI or from a notebook by executing this code in a cell:
-
-```bash
+You can run Sedona in Databricks to leverage the functionality that Sedona provides. Here’s an example of a Databricks notebook that’s running Sedona code:
+
+
+
+Sedona isn’t available in all Databricks environments because of the platform's limitations. This post explains how and where you can run Sedona in Databricks.
+
+This post also demonstrates how to read Databricks Delta tables in environments outside Databricks.
+
+## Databricks and Sedona version requirements
+
+Databricks and Sedona depend on Spark, Scala, and other libraries.
+
+For example, one Databricks Runtime 16.4 depends on Scala 2.12 and Spark 3.5. Here are the version requirements for a few Databricks runtimes.
+
+
+
+If you use a Databricks Runtime compiled with Spark 3.5 and Scala 2.12, then you should use a Sedona version compiled with Spark 3.5 and Scala 2.12. You need to make sure the Scala versions are aligned, even if you’re using the Python or SQL APIs.
+
+Here are the recommended Databricks Runtimes for different Spark and Sedona versions:
+
+<table>
Review Comment:
I think we should remove it. All it says is that use Sedona 1.7 on Databricks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] update databricks setup instructions [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #1963:
URL: https://github.com/apache/sedona/pull/1963#discussion_r2114919261
##########
docs/setup/databricks.md:
##########
@@ -17,38 +17,93 @@
under the License.
-->
-In Databricks advanced editions, you need to install Sedona via [cluster init-scripts](https://docs.databricks.com/clusters/init-scripts.html) as described below. Sedona is not guaranteed to be 100% compatible with `Databricks photon acceleration`. Sedona requires Spark internal APIs to inject many optimization strategies, which sometimes is not accessible in `Photon`.
-
-The following steps use DBR including Apache Spark 3.5.x as an example. Please change the Spark version according to your DBR version. Please pay attention to the Spark version postfix and Scala version postfix on our [Maven Coordinate page](maven-coordinates.md). Databricks Spark and Apache Spark's compatibility can be found [here](https://docs.databricks.com/en/release-notes/runtime/index.html).
-
-!!! bug
- Databricks Runtime 16.2 (non-LTS) introduces a change in the json4s dependency, which may lead to compatibility issues with Apache Sedona. We recommend using a currently supported LTS version, such as Databricks Runtime 15.4 LTS or 14.3 LTS, to ensure stability. A patch will be provided once an official Databricks Runtime 16 LTS version is released.
-
-### Download Sedona jars
-
-Download the Sedona jars to a DBFS location. You can do that manually via UI or from a notebook by executing this code in a cell:
-
-```bash
+You can run Sedona in Databricks to leverage the functionality that Sedona provides. Here’s an example of a Databricks notebook that’s running Sedona code:
+
+
+
+Sedona isn’t available in all Databricks environments because of the platform's limitations. This post explains how and where you can run Sedona in Databricks.
+
+This post also demonstrates how to read Databricks Delta tables in environments outside Databricks.
+
+## Databricks and Sedona version requirements
+
+Databricks and Sedona depend on Spark, Scala, and other libraries.
+
+For example, one Databricks Runtime 16.4 depends on Scala 2.12 and Spark 3.5. Here are the version requirements for a few Databricks runtimes.
+
+
+
+If you use a Databricks Runtime compiled with Spark 3.5 and Scala 2.12, then you should use a Sedona version compiled with Spark 3.5 and Scala 2.12. You need to make sure the Scala versions are aligned, even if you’re using the Python or SQL APIs.
+
+Here are the recommended Databricks Runtimes for different Spark and Sedona versions:
+
+<table>
+ <tr>
+ <td><strong>Databricks Runtime</strong>
+ </td>
+ <td><strong>Spark</strong>
+ </td>
+ <td><strong>Sedona</strong>
+ </td>
+ </tr>
+ <tr>
+ <td>16.4 LTS
+ </td>
+ <td>3.5
+ </td>
+ <td>1.7
+ </td>
+ </tr>
+ <tr>
+ <td>13.3 LTS
+ </td>
+ <td>3.4
+ </td>
+ <td>1.7
+ </td>
+ </tr>
+ <tr>
+ <td>12.2 LTS
+ </td>
+ <td>3.3
+ </td>
+ <td>1.7
+ </td>
+ </tr>
+</table>
+
+Only some Sedona functions work when Databricks Photon acceleration is enabled, so you can consider disabling Photon when using Sedona for better compatibility.
+
+## Install the Sedona library in Databricks
+
+Download the required Sedona packages by executing the following commands:
+
+```sh
%sh
# Create JAR directory for Sedona
-mkdir -p /Workspace/Shared/sedona/{{ sedona.current_version }}
+mkdir -p /Workspace/Shared/sedona/1.7.1
# Download the dependencies from Maven into DBFS
-curl -o /Workspace/Shared/sedona/{{ sedona.current_version }}/geotools-wrapper-{{ sedona.current_geotools }}.jar "https://repo1.maven.org/maven2/org/datasyslab/geotools-wrapper/{{ sedona.current_geotools }}/geotools-wrapper-{{ sedona.current_geotools }}.jar"
+curl -o /Workspace/Shared/sedona/1.7.1/geotools-wrapper-1.7.1-28.5.jar "https://repo1.maven.org/maven2/org/datasyslab/geotools-wrapper/1.7.1-28.5/geotools-wrapper-1.7.1-28.5.jar"
-curl -o /Workspace/Shared/sedona/{{ sedona.current_version }}/sedona-spark-shaded-3.5_2.12-{{ sedona.current_version }}.jar "https://repo1.maven.org/maven2/org/apache/sedona/sedona-spark-shaded-3.5_2.12/{{ sedona.current_version }}/sedona-spark-shaded-3.5_2.12-{{ sedona.current_version }}.jar"
+curl -o /Workspace/Shared/sedona/1.7.1/sedona-spark-shaded-3.5_2.12-1.7.1.jar "https://repo1.maven.org/maven2/org/apache/sedona/sedona-spark-shaded-3.5_2.12/1.7.1/sedona-spark-shaded-3.5_2.12-1.7.1.jar"
Review Comment:
```suggestion
curl -o /Workspace/Shared/sedona/{{ sedona.current_version }}/sedona-spark-shaded-3.5_2.12-{{ sedona.current_version }}.jar "https://repo1.maven.org/maven2/org/apache/sedona/sedona-spark-shaded-3.5_2.12/{{ sedona.current_version }}/sedona-spark-shaded-3.5_2.12-{{ sedona.current_version }}.jar"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] update databricks setup instructions [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #1963:
URL: https://github.com/apache/sedona/pull/1963#discussion_r2114919436
##########
docs/setup/databricks.md:
##########
@@ -80,30 +145,62 @@ spark.kryo.registrator org.apache.sedona.core.serde.SedonaKryoRegistrator
spark.sedona.enableParserExtension false
```
-From your cluster configuration (`Cluster` -> `Edit` -> `Configuration` -> `Advanced options` -> `Init Scripts`) add the newly created `Workspace` init script
+Specify the path to the init script:
+
+
+
+If you are creating a Shared cluster, you won't be able to use init scripts and jars stored under Workspace. Please store them in volumes instead. The overall process should be the same.
-| Type | File path |
-|------|-----------|
-| Workspace | /Shared/sedona/sedona-init.sh |
+Add the required dependencies in the Library tab:
-For enabling python support, from the Libraries tab install from PyPI
+
+
+Here’s the full list of libraries:
```
-apache-sedona=={{ sedona.current_version }}
+apache-sedona==1.7.1
Review Comment:
```suggestion
apache-sedona=={{ sedona.current_version }}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] update databricks setup instructions [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu merged PR #1963:
URL: https://github.com/apache/sedona/pull/1963
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DOCS] update databricks setup instructions [sedona]
Posted by "MrPowers (via GitHub)" <gi...@apache.org>.
MrPowers commented on code in PR #1963:
URL: https://github.com/apache/sedona/pull/1963#discussion_r2115563782
##########
docs/setup/databricks.md:
##########
@@ -17,38 +17,93 @@
under the License.
-->
-In Databricks advanced editions, you need to install Sedona via [cluster init-scripts](https://docs.databricks.com/clusters/init-scripts.html) as described below. Sedona is not guaranteed to be 100% compatible with `Databricks photon acceleration`. Sedona requires Spark internal APIs to inject many optimization strategies, which sometimes is not accessible in `Photon`.
-
-The following steps use DBR including Apache Spark 3.5.x as an example. Please change the Spark version according to your DBR version. Please pay attention to the Spark version postfix and Scala version postfix on our [Maven Coordinate page](maven-coordinates.md). Databricks Spark and Apache Spark's compatibility can be found [here](https://docs.databricks.com/en/release-notes/runtime/index.html).
-
-!!! bug
- Databricks Runtime 16.2 (non-LTS) introduces a change in the json4s dependency, which may lead to compatibility issues with Apache Sedona. We recommend using a currently supported LTS version, such as Databricks Runtime 15.4 LTS or 14.3 LTS, to ensure stability. A patch will be provided once an official Databricks Runtime 16 LTS version is released.
-
-### Download Sedona jars
-
-Download the Sedona jars to a DBFS location. You can do that manually via UI or from a notebook by executing this code in a cell:
-
-```bash
+You can run Sedona in Databricks to leverage the functionality that Sedona provides. Here’s an example of a Databricks notebook that’s running Sedona code:
+
+
+
+Sedona isn’t available in all Databricks environments because of the platform's limitations. This post explains how and where you can run Sedona in Databricks.
+
+This post also demonstrates how to read Databricks Delta tables in environments outside Databricks.
+
+## Databricks and Sedona version requirements
+
+Databricks and Sedona depend on Spark, Scala, and other libraries.
+
+For example, one Databricks Runtime 16.4 depends on Scala 2.12 and Spark 3.5. Here are the version requirements for a few Databricks runtimes.
+
+
+
+If you use a Databricks Runtime compiled with Spark 3.5 and Scala 2.12, then you should use a Sedona version compiled with Spark 3.5 and Scala 2.12. You need to make sure the Scala versions are aligned, even if you’re using the Python or SQL APIs.
+
+Here are the recommended Databricks Runtimes for different Spark and Sedona versions:
+
+<table>
Review Comment:
@jiayuasu - I can promise to maintain this if that's OK. I agree that this will require discipline to review every month or so. I can also remove this if you feel strongly about it, because I agree that this will become outdated quickly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org