You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@sedona.apache.org by ji...@apache.org on 2024/01/14 01:38:28 UTC
(sedona) branch 1.5.1-docs created (now 9785cf841)
This is an automated email from the ASF dual-hosted git repository.
jiayu pushed a change to branch 1.5.1-docs
in repository https://gitbox.apache.org/repos/asf/sedona.git
at 9785cf841 Add docs
This branch includes the following new commits:
new 9785cf841 Add docs
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(sedona) 01/01: Add docs
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jiayu pushed a commit to branch 1.5.1-docs
in repository https://gitbox.apache.org/repos/asf/sedona.git
commit 9785cf8414be938f619f2fe508ca254aee2e210a
Author: Jia Yu <ji...@apache.org>
AuthorDate: Sat Jan 13 17:38:22 2024 -0800
Add docs
---
R/DESCRIPTION | 2 +-
R/R/dependencies.R | 4 +-
README.md | 23 +++--
docs-overrides/main.html | 7 +-
docs/community/contact.md | 19 +++-
docs/community/discord-invite-form.html | 149 -----------------------------
docs/community/publish.md | 2 +-
docs/setup/databricks.md | 34 ++++---
docs/setup/release-notes.md | 160 ++++++++++++++++++++++++++++++++
docs/tutorial/demo.md | 22 ++---
examples/flink-sql/pom.xml | 19 ++++
mkdocs.yml | 10 +-
python/sedona/version.py | 2 +-
zeppelin/package.json | 2 +-
14 files changed, 251 insertions(+), 204 deletions(-)
diff --git a/R/DESCRIPTION b/R/DESCRIPTION
index 6bef9caa3..48a689834 100644
--- a/R/DESCRIPTION
+++ b/R/DESCRIPTION
@@ -1,7 +1,7 @@
Type: Package
Package: apache.sedona
Title: R Interface for Apache Sedona
-Version: 1.5.0
+Version: 1.5.1
Authors@R:
c(person(family = "Apache Sedona",
role = c("aut", "cre"),
diff --git a/R/R/dependencies.R b/R/R/dependencies.R
index 5a529c189..68cf4e498 100644
--- a/R/R/dependencies.R
+++ b/R/R/dependencies.R
@@ -25,7 +25,7 @@ spark_dependencies <- function(spark_version, scala_version, ...) {
}
packages <- c(
- "org.datasyslab:geotools-wrapper:1.4.0-28.2",
+ "org.datasyslab:geotools-wrapper:1.5.1-28.2",
"edu.ucar:cdm-core:5.4.2"
)
jars <- NULL
@@ -39,7 +39,7 @@ spark_dependencies <- function(spark_version, scala_version, ...) {
paste0(
"org.apache.sedona:sedona-",
c("spark-shaded"),
- sprintf("-%s_%s:1.5.0", spark_version, scala_version)
+ sprintf("-%s_%s:1.5.1", spark_version, scala_version)
),
packages
)
diff --git a/README.md b/README.md
index 836ef7538..1bba26156 100644
--- a/README.md
+++ b/README.md
@@ -108,15 +108,18 @@ We provide a Docker image for Apache Sedona with Python JupyterLab and a single-
* Modules in the source code
-| Name | API | Introduction|
-|---|---|---|
-|Core | Scala/Java | Distributed Spatial Datasets and Query Operators |
-|SQL | Spark RDD/DataFrame in Scala/Java/SQL |Geospatial data processing on Apache Spark|
-|Flink | Flink DataStream/Table in Scala/Java/SQL | Geospatial data processing on Apache Flink
-|Viz | Spark RDD/DataFrame in Scala/Java/SQL | Geospatial data visualization on Apache Spark|
-|Python | Spark RDD/DataFrame in Python | Python wrapper for Sedona |
-|R | Spark RDD/DataFrame in R | R wrapper for Sedona |
-|Zeppelin | Apache Zeppelin | Plugin for Apache Zeppelin 0.8.1+|
+| Name | API | Introduction |
+|------------------|------------------------------------------|--------------------------------------------------------|
+| common | Java | Core geometric operation logics, serialization, index |
+| spark | Spark RDD/DataFrame Scala/Java/SQL | Distributed geospatial data processing on Apache Spark |
+| flink | Flink DataStream/Table in Scala/Java/SQL | Distributed geospatial data processing on Apache Flink |
+| snowflake | Snowflake SQL | Distributed geospatial data processing on Snowflake |
+| spark-shaded | No source code | shaded jar for Sedona Spark |
+| flink-shaded | No source code | shaded jar for Sedona Flink |
+| snowflake-tester | Java | tester program for Sedona Snowflake |
+| python | Spark RDD/DataFrame Python | Distributed geospatial data processing on Apache Spark |
+| R | Spark RDD/DataFrame in R | R wrapper for Sedona |
+| Zeppelin | Apache Zeppelin | Plugin for Apache Zeppelin 0.8.1+ |
## Documentation
@@ -130,7 +133,7 @@ Please visit [Apache Sedona website](http://sedona.apache.org/) for detailed inf
Follow Sedona on Twitter for fresh news: [Sedona@Twitter](https://twitter.com/ApacheSedona)
-Join the Sedona Discord community: [](https://sedona.apache.org/latest/community/discord-invite-form.html)
+Join the Sedona Discord community: [](https://sedona.apache.org/latest/community/contact/#discord-server)
[Sedona JIRA](https://issues.apache.org/jira/projects/SEDONA): Bugs, Pull Requests, and other similar issues
diff --git a/docs-overrides/main.html b/docs-overrides/main.html
index 9533c8081..c9a301f2e 100644
--- a/docs-overrides/main.html
+++ b/docs-overrides/main.html
@@ -36,7 +36,12 @@
<a href="{{ config.repo_url }}" title="{{ lang.t('source.link.title') }}" class="md-button">
Go to GitHub
</a>
- <p><a href="community/discord-invite-form.html" target="_blank">Join Apache Sedona Community Discord server!</a></p>
+ <p>
+ <a href="community/contact/#discord-server" target="_blank">
+ <img src="https://dcbadge.vercel.app/api/server/9A3k5dEBsY" alt="Apache Sedona Discord server">
+ </a>
+ </p>
+
</div>
<br><br><br>
diff --git a/docs/community/contact.md b/docs/community/contact.md
index 9a096d4ee..11f6fa3f1 100644
--- a/docs/community/contact.md
+++ b/docs/community/contact.md
@@ -9,13 +9,30 @@ You can participate in the community as follows:
* Report bugs and submit patches.
* Contribute code and documentation.
+## Community events
+
+Everyone is welcome to join our community events. We have a community office hour every 4 weeks.
+
+<iframe src="https://calendar.google.com/calendar/embed?src=c_ef7d54f5308a4b7f25ac33d0f7eeb54a3a7f115b689faf4d82425f3cb5ede935%40group.calendar.google.com&ctz=America%2FLos_Angeles" style="border: 0" width="800" height="600" frameborder="0" scrolling="no"></iframe>
+
## Twitter
[Apache Sedona@Twitter](https://twitter.com/ApacheSedona)
## Discord Server
-[Join Apache Sedona community server](./discord-invite-form.html)!
+
+
+Fill in the form below to join the Sedona Discord Server!
+
+<script charset="utf-8" type="text/javascript" src="//js.hsforms.net/forms/embed/v2.js"></script>
+<script>
+ hbspt.forms.create({
+ region: "na1",
+ portalId: "44429642",
+ formId: "4af794ab-318a-4d3b-ae8b-50143dbbdca5"
+ });
+</script>
## Mailing list
diff --git a/docs/community/discord-invite-form.html b/docs/community/discord-invite-form.html
deleted file mode 100644
index e62f3de03..000000000
--- a/docs/community/discord-invite-form.html
+++ /dev/null
@@ -1,149 +0,0 @@
-<!-- Begin Brevo Form -->
-<!-- START - We recommend to place the below code in head tag of your website html -->
-<style>
- @font-face {
- font-display: block;
- font-family: Roboto;
- src: url(https://assets.brevo.com/font/Roboto/Latin/normal/normal/7529907e9eaf8ebb5220c5f9850e3811.woff2) format("woff2"), url(https://assets.brevo.com/font/Roboto/Latin/normal/normal/25c678feafdc175a70922a116c9be3e7.woff) format("woff")
- }
-
- @font-face {
- font-display: fallback;
- font-family: Roboto;
- font-weight: 600;
- src: url(https://assets.brevo.com/font/Roboto/Latin/medium/normal/6e9caeeafb1f3491be3e32744bc30440.woff2) format("woff2"), url(https://assets.brevo.com/font/Roboto/Latin/medium/normal/71501f0d8d5aa95960f6475d5487d4c2.woff) format("woff")
- }
-
- @font-face {
- font-display: fallback;
- font-family: Roboto;
- font-weight: 700;
- src: url(https://assets.brevo.com/font/Roboto/Latin/bold/normal/3ef7cf158f310cf752d5ad08cd0e7e60.woff2) format("woff2"), url(https://assets.brevo.com/font/Roboto/Latin/bold/normal/ece3a1d82f18b60bcce0211725c476aa.woff) format("woff")
- }
-
- #sib-container input:-ms-input-placeholder {
- text-align: left;
- font-family: "Helvetica", sans-serif;
- color: #c0ccda;
- }
-
- #sib-container input::placeholder {
- text-align: left;
- font-family: "Helvetica", sans-serif;
- color: #c0ccda;
- }
-
- #sib-container textarea::placeholder {
- text-align: left;
- font-family: "Helvetica", sans-serif;
- color: #c0ccda;
- }
-</style>
-<link rel="stylesheet" href="https://sibforms.com/forms/end-form/build/sib-styles.css">
-<!-- END - We recommend to place the above code in head tag of your website html -->
-
-<!-- START - We recommend to place the below code where you want the form in your website html -->
-<div class="sib-form" style="text-align: center;
- background-color: #EFF2F7; ">
- <div id="sib-form-container" class="sib-form-container">
- <div id="error-message" class="sib-form-message-panel" style="font-size:16px; text-align:left; font-family:"Helvetica", sans-serif; color:#661d1d; background-color:#ffeded; border-radius:3px; border-color:#ff4949;max-width:540px;">
- <div class="sib-form-message-panel__text sib-form-message-panel__text--center">
- <svg viewBox="0 0 512 512" class="sib-icon sib-notification__icon">
- <path d="M256 40c118.621 0 216 96.075 216 216 0 119.291-96.61 216-216 216-119.244 0-216-96.562-216-216 0-119.203 96.602-216 216-216m0-32C119.043 8 8 119.083 8 256c0 136.997 111.043 248 248 248s248-111.003 248-248C504 119.083 392.957 8 256 8zm-11.49 120h22.979c6.823 0 12.274 5.682 11.99 12.5l-7 168c-.268 6.428-5.556 11.5-11.99 11.5h-8.979c-6.433 0-11.722-5.073-11.99-11.5l-7-168c-.283-6.818 5.167-12.5 11.99-12.5zM256 340c-15.464 0-28 12.536-28 28s12.536 28 28 28 28-12.536 28-28-1 [...]
- </svg>
- <span class="sib-form-message-panel__inner-text">
- Internal error. Please try again.
- </span>
- </div>
- </div>
- <div></div>
- <div id="success-message" class="sib-form-message-panel" style="font-size:16px; text-align:left; font-family:"Helvetica", sans-serif; color:#085229; background-color:#e7faf0; border-radius:3px; border-color:#13ce66;max-width:540px;">
- <div class="sib-form-message-panel__text sib-form-message-panel__text--center">
- <svg viewBox="0 0 512 512" class="sib-icon sib-notification__icon">
- <path d="M256 8C119.033 8 8 119.033 8 256s111.033 248 248 248 248-111.033 248-248S392.967 8 256 8zm0 464c-118.664 0-216-96.055-216-216 0-118.663 96.055-216 216-216 118.664 0 216 96.055 216 216 0 118.663-96.055 216-216 216zm141.63-274.961L217.15 376.071c-4.705 4.667-12.303 4.637-16.97-.068l-85.878-86.572c-4.667-4.705-4.637-12.303.068-16.97l8.52-8.451c4.705-4.667 12.303-4.637 16.97.068l68.976 69.533 163.441-162.13c4.705-4.667 12.303-4.637 16.97.068l8.451 8.52c4.668 4.705 4.637 12 [...]
- </svg>
- <span class="sib-form-message-panel__inner-text">
- You will receive an email containing the invitation link shortly
- </span>
- </div>
- </div>
- <div></div>
- <div id="sib-container" class="sib-container--large sib-container--vertical" style="text-align:center; background-color:rgba(255,255,255,1); max-width:540px; border-radius:3px; border-width:1px; border-color:#C0CCD9; border-style:solid; direction:ltr">
- <form id="sib-form" method="POST" action="https://160d1919.sibforms.com/serve/MUIFADrWyyNp_y3piws-A9A_lcyAOZz_thxxfspveOTWCgwio6eI7qKbq90bdx6WKEUZnnrzHyNjVn9tOZUMMxKz8qR5EdF-7EQb2_Itv1nsqXlvhSslSa8bVIZxxNFbbB5ZpZ2dtzrQa6JcszJ1w0dvBgw8TfNJL8t5sALCdCKUMigQ03XI2vSGXvT_pJhJdh6vIKFqUM1g7dpD" data-type="subscription">
- <div style="padding: 8px 0;">
- <div class="sib-form-block" style="font-size:32px; text-align:left; font-weight:700; font-family:"Helvetica", sans-serif; color:#3C4858; background-color:transparent; text-align:left">
- <p>Apache Sedona community discord</p>
- </div>
- </div>
- <div style="padding: 8px 0;">
- <div class="sib-form-block" style="font-size:16px; text-align:left; font-family:"Helvetica", sans-serif; color:#3C4858; background-color:transparent; text-align:left">
- <div class="sib-text-form-block">
- <p>Join our Discord server and stay updated.</p>
- </div>
- </div>
- </div>
- <div style="padding: 8px 0;">
- <div class="sib-input sib-form-block">
- <div class="form__entry entry_block">
- <div class="form__label-row ">
- <label class="entry__label" style="font-weight: 700; text-align:left; font-size:16px; text-align:left; font-weight:700; font-family:"Helvetica", sans-serif; color:#3c4858;" for="EMAIL" data-required="*">Enter your email address to get the invitation</label>
-
- <div class="entry__field">
- <input class="input " type="text" id="EMAIL" name="EMAIL" autocomplete="off" placeholder="EMAIL" data-required="true" required />
- </div>
- </div>
-
- <label class="entry__error entry__error--primary" style="font-size:16px; text-align:left; font-family:"Helvetica", sans-serif; color:#661d1d; background-color:#ffeded; border-radius:3px; border-color:#ff4949;">
- </label>
- <label class="entry__specification" style="font-size:12px; text-align:left; font-family:"Helvetica", sans-serif; color:#8390A4; text-align:left">
- Provide your email address to get the invitation link. For e.g abc@xyz.com
- </label>
- </div>
- </div>
- </div>
- <div style="padding: 8px 0;">
- <div class="sib-form-block" style="text-align: left">
- <button class="sib-form-block__button sib-form-block__button-with-loader" style="font-size:16px; text-align:left; font-weight:700; font-family:"Helvetica", sans-serif; color:#FFFFFF; background-color:#4ade33; border-radius:3px; border-width:0px;" form="sib-form" type="submit">
- <svg class="icon clickable__icon progress-indicator__icon sib-hide-loader-icon" viewBox="0 0 512 512">
- <path d="M460.116 373.846l-20.823-12.022c-5.541-3.199-7.54-10.159-4.663-15.874 30.137-59.886 28.343-131.652-5.386-189.946-33.641-58.394-94.896-95.833-161.827-99.676C261.028 55.961 256 50.751 256 44.352V20.309c0-6.904 5.808-12.337 12.703-11.982 83.556 4.306 160.163 50.864 202.11 123.677 42.063 72.696 44.079 162.316 6.031 236.832-3.14 6.148-10.75 8.461-16.728 5.01z" />
- </svg>
- Join
- </button>
- </div>
- </div>
-
- <input type="text" name="email_address_check" value="" class="input--hidden">
- <input type="hidden" name="locale" value="en">
- </form>
- </div>
- </div>
-</div>
-<!-- END - We recommend to place the below code where you want the form in your website html -->
-
-<!-- START - We recommend to place the below code in footer or bottom of your website html -->
-<script>
- window.REQUIRED_CODE_ERROR_MESSAGE = 'Please choose a country code';
- window.LOCALE = 'en';
- window.EMAIL_INVALID_MESSAGE = window.SMS_INVALID_MESSAGE = "The information provided is invalid. Please review the field format and try again.";
-
- window.REQUIRED_ERROR_MESSAGE = "This field cannot be left blank. ";
-
- window.GENERIC_INVALID_MESSAGE = "The information provided is invalid. Please review the field format and try again.";
-
-
-
-
- window.translation = {
- common: {
- selectedList: '{quantity} list selected',
- selectedLists: '{quantity} lists selected'
- }
- };
-
- var AUTOHIDE = Boolean(0);
-</script>
-<script defer src="https://sibforms.com/forms/end-form/build/main.js"></script>
-
-
-<!-- END - We recommend to place the above code in footer or bottom of your website html -->
-<!-- End Brevo Form -->
diff --git a/docs/community/publish.md b/docs/community/publish.md
index 717d0d14d..31e58a0c7 100644
--- a/docs/community/publish.md
+++ b/docs/community/publish.md
@@ -22,7 +22,7 @@ chmod 777 create-release.sh
1. Run the following script:
```bash
#!/bin/bash
-wget -q https://dlcdn.apache.org//creadur/apache-rat-0.15/apache-rat-0.15-bin.tar.gz
+wget -q https://archive.apache.org/dist/creadur/apache-rat-0.15/apache-rat-0.15-bin.tar.gz
tar -xvf apache-rat-0.15-bin.tar.gz
git clone --shared --branch master https://github.com/apache/sedona.git sedona-src
java -jar apache-rat-0.15/apache-rat-0.15.jar -d sedona-src > report.txt
diff --git a/docs/setup/databricks.md b/docs/setup/databricks.md
index e74911a57..34fec7a28 100644
--- a/docs/setup/databricks.md
+++ b/docs/setup/databricks.md
@@ -12,7 +12,7 @@ We recommend Databricks 10.x+.
* Sedona 1.0.1 & 1.1.0 is compiled against Spark 3.1 (~ Databricks DBR 9 LTS, DBR 7 is Spark 3.0)
* Sedona 1.1.1, 1.2.0 are compiled against Spark 3.2 (~ DBR 10 & 11)
* Sedona 1.2.1, 1.3.1, 1.4.0 are complied against Spark 3.3
-* 1.4.1, 1.5.0 are complied against Spark 3.3 and 3.4
+* 1.4.1, 1.5.0 are complied against Spark 3.3, 3.4, 3.5
> In Spark 3.2, `org.apache.spark.sql.catalyst.expressions.Generator` class added a field `nodePatterns`. Any SQL functions that rely on Generator class may have issues if compiled for a runtime with a differing spark version. For Sedona, those functions are:
> * ST_MakeValid
@@ -40,12 +40,6 @@ This method cannot achieve the best performance of Sedona and does not work for
pydeck==0.8.0
```
-3) (Only for DBR up to 7.3 LTS) You can speed up the serialization of geometry types by adding to your spark configurations (`Cluster` -> `Edit` -> `Configuration` -> `Advanced options`) the following lines:
- ```
- spark.serializer org.apache.spark.serializer.KryoSerializer
- spark.kryo.registrator org.apache.sedona.core.serde.SedonaKryoRegistrator
- ```
-
### Initialize
After you have installed the libraries and started the cluster, you can initialize the Sedona `ST_*` functions and types by running from your code:
@@ -66,7 +60,7 @@ SedonaRegistrator.registerAll(spark)
In order to activate the Kryo serializer (this speeds up the serialization and deserialization of geometry types) you need to install the libraries via init script as described below.
-In order to use the Sedona `ST_*` functions from SQL without having to register the Sedona functions from a python/scala cell, you need to install the Sedona libraries from the [cluster init-scripts](https://docs.databricks.com/clusters/init-scripts.html) as follows.
+In order to use the Sedona `ST_*/RS_*` functions from SQL without having to register the Sedona functions from a python/scala cell, you need to install the Sedona libraries from the [cluster init-scripts](https://docs.databricks.com/clusters/init-scripts.html) as follows.
### Download Sedona jars
@@ -75,26 +69,29 @@ Download the Sedona jars to a DBFS location. You can do that manually via UI or
```bash
%sh
# Create JAR directory for Sedona
-mkdir -p /dbfs/FileStore/jars/sedona/{{ sedona.current_version }}
+mkdir -p /Workspace/Shared/sedona/{{ sedona.current_version }}
# Download the dependencies from Maven into DBFS
-curl -o /dbfs/FileStore/jars/sedona/{{ sedona.current_version }}/geotools-wrapper-{{ sedona.current_geotools }}.jar "https://repo1.maven.org/maven2/org/datasyslab/geotools-wrapper/{{ sedona.current_geotools }}/geotools-wrapper-{{ sedona.current_geotools }}.jar"
+curl -o /Workspace/Shared/sedona/{{ sedona.current_version }}/geotools-wrapper-{{ sedona.current_geotools }}.jar "https://repo1.maven.org/maven2/org/datasyslab/geotools-wrapper/{{ sedona.current_geotools }}/geotools-wrapper-{{ sedona.current_geotools }}.jar"
-curl -o /dbfs/FileStore/jars/sedona/{{ sedona.current_version }}/sedona-spark-shaded-3.0_2.12-{{ sedona.current_version }}.jar "https://repo1.maven.org/maven2/org/apache/sedona/sedona-spark-shaded-3.0_2.12/{{ sedona.current_version }}/sedona-spark-shaded-3.0_2.12-{{ sedona.current_version }}.jar"
+curl -o /Workspace/Shared/sedona/{{ sedona.current_version }}/sedona-spark-shaded-3.4_2.12-{{ sedona.current_version }}.jar "https://repo1.maven.org/maven2/org/apache/sedona/sedona-spark-shaded-3.4_2.12/{{ sedona.current_version }}/sedona-spark-shaded-3.4_2.12-{{ sedona.current_version }}.jar"
```
### Create an init script
-Create an init script in DBFS that loads the Sedona jars into the cluster's default jar directory. You can create that from any notebook by running:
+!!!warning
+ Starting from December 2023, Databricks has disabled all DBFS based init script (/dbfs/XXX/<script-name>.sh). So you will have to store the init script from a workspace level (`/Users/<user-name>/<script-name>.sh`) or Unity Catalog volume (`/Volumes/<catalog>/<schema>/<volume>/<path-to-script>/<script-name>.sh`). Please see https://docs.databricks.com/en/init-scripts/cluster-scoped.html#configure-a-cluster-scoped-init-script-using-the-ui
+
+Create an init script in `Workspace` that loads the Sedona jars into the cluster's default jar directory. You can create that from any notebook by running:
```bash
%sh
# Create init script directory for Sedona
-mkdir -p /dbfs/FileStore/sedona/
+mkdir -p /Workspace/Shared/sedona/
# Create init script
-cat > /dbfs/FileStore/sedona/sedona-init.sh <<'EOF'
+cat > /Workspace/Shared/sedona/sedona-init.sh <<'EOF'
#!/bin/bash
#
# File: sedona-init.sh
@@ -102,7 +99,7 @@ cat > /dbfs/FileStore/sedona/sedona-init.sh <<'EOF'
# On cluster startup, this script will copy the Sedona jars to the cluster's default jar directory.
# In order to activate Sedona functions, remember to add to your spark configuration the Sedona extensions: "spark.sql.extensions org.apache.sedona.viz.sql.SedonaVizExtensions,org.apache.sedona.sql.SedonaSqlExtensions"
-cp /dbfs/FileStore/jars/sedona/{{ sedona.current_version }}/*.jar /databricks/jars
+cp /Workspace/Shared/sedona/{{ sedona.current_version }}/*.jar /databricks/jars
EOF
```
@@ -116,14 +113,15 @@ spark.serializer org.apache.spark.serializer.KryoSerializer
spark.kryo.registrator org.apache.sedona.core.serde.SedonaKryoRegistrator
```
-From your cluster configuration (`Cluster` -> `Edit` -> `Configuration` -> `Advanced options` -> `Init Scripts`) add the newly created init script
+From your cluster configuration (`Cluster` -> `Edit` -> `Configuration` -> `Advanced options` -> `Init Scripts`) add the newly created `Workspace` init script
```
-dbfs:/FileStore/sedona/sedona-init.sh
+/Workspace/sedona/sedona-init.sh
```
For enabling python support, from the Libraries tab install from PyPI
```
-apache-sedona
+apache-sedona=={{ sedona.current_version }}
+geopandas==0.11.1
keplergl==0.3.2
pydeck==0.8.0
```
diff --git a/docs/setup/release-notes.md b/docs/setup/release-notes.md
index 5367f70b4..703407216 100644
--- a/docs/setup/release-notes.md
+++ b/docs/setup/release-notes.md
@@ -4,6 +4,166 @@
!!! note
Sedona Python currently only works with Shapely 1.x. If you use GeoPandas, please use <= GeoPandas `0.11.1`. GeoPandas > 0.11.1 will automatically install Shapely 2.0. If you use Shapely, please use <= `1.8.4`.
+## Sedona 1.5.1
+
+Sedona 1.5.1 is compiled against Spark 3.3 / Spark 3.4 / Spark 3.5, Flink 1.12, Snowflake 7+, Java 8.
+
+### Highlights
+
+* [X] **Sedona Spark** Support Spark 3.5
+* [X] **Sedona Spark** Support Snowflake 7+
+* [X] **Sedona Spark** Added 20+ raster functions (or variants)
+* [X] **Sedona Spark/Flink/Snowflake** Added 7 vector functions (or variants)
+* [X] **Sedona Spark** GeoParquet reader and writer supports projjson in metadata
+* [X] **Sedona Spark** GeoParquet reader and writer conform to GeoParquet spec 1.0.0 instead of 1.0.0-beta1
+* [X] **Sedona Spark** Added a legacyMode in GeoParquet reader for 1.5.1+ users to read Parquet files written by Sedona 1.3.1 and earlier
+* [X] **Sedona Spark** Fixed a bug in GeoParquet writer so 1.3.1 and earlier users can read Parquet files written by 1.5.1+
+
+### Behavior change
+
+* All raster functions that take a geometry will implicitly transform the CRS of the geometry if needed.
+* The default CRS for these functions is 4326 for raster and geometry involved in raster functions, if not specified.
+* KeplerGL and DeckGL become optional dependencies for Sedona Spark Python.
+
+### New Contributors
+
+* @hongbo-miao made their first contribution in https://github.com/apache/sedona/pull/1063
+* @prantogg made their first contribution in https://github.com/apache/sedona/pull/1122
+* @MyEnthusiastic made their first contribution in https://github.com/apache/sedona/pull/1130
+* @duhaode520 made their first contribution in https://github.com/apache/sedona/pull/1193
+
+### Bug
+
+<ul>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-414'>SEDONA-414</a>] - ST_MakeLine in sedona-spark does not work with array inputs
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-417'>SEDONA-417</a>] - Fix SedonaUtils.display_image
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-419'>SEDONA-419</a>] - SedonaKepler and SedonaPyDeck should not be in `sedona.spark`
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-420'>SEDONA-420</a>] - Make SedonaKepler and SedonaPydeck optional dependencies
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-424'>SEDONA-424</a>] - Specify jt-jiffle as a provided dependency
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-426'>SEDONA-426</a>] - Change cloning of rasters to be able to include metadata.
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-440'>SEDONA-440</a>] - GeoParquet reader should support filter pushdown on nested fields
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-443'>SEDONA-443</a>] - Upload-artifact leads to 503 error
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-453'>SEDONA-453</a>] - Performance degrade when indexing points using Quadtree
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-456'>SEDONA-456</a>] - SedonaKepler cannot work with geopandas >= 0.13.0 correctly
+</li>
+</ul>
+
+### New Feature
+
+<ul>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-369'>SEDONA-369</a>] - Add ST_DWITHIN
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-411'>SEDONA-411</a>] - Add RS_Rotation
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-413'>SEDONA-413</a>] - Add buffer parameters to ST_Buffer
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-415'>SEDONA-415</a>] - Add optional parameter to ST_Transform
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-421'>SEDONA-421</a>] - Add RS_Clip
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-422'>SEDONA-422</a>] - Add a feature in RS_SetBandNoDataValue and fix NoDataValue in RS_Clip
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-427'>SEDONA-427</a>] - Add RS_RasterToWorldCoord
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-428'>SEDONA-428</a>] - Add RS_ZonalStats & RS_ZonalStatsAll
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-430'>SEDONA-430</a>] - geoparquet writer should have an option called `writeToCrs`
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-431'>SEDONA-431</a>] - Add RS_PixelAsPoints
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-432'>SEDONA-432</a>] - Add RS_PixelAsCentroids
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-433'>SEDONA-433</a>] - Improve RS_SummaryStats performance
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-435'>SEDONA-435</a>] - Add RS_PixelAsPolygons
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-438'>SEDONA-438</a>] - Add NetCDF reader to Sedona
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-439'>SEDONA-439</a>] - Add RS_Union_Aggr
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-441'>SEDONA-441</a>] - Implement ST_LineLocatePoint
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-449'>SEDONA-449</a>] - Add two raster column support to RS_MapAlgebra
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-455'>SEDONA-455</a>] - Add a new data source namely geoparquet.metadata
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-459'>SEDONA-459</a>] - Add Snowflake support
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-460'>SEDONA-460</a>] - RS_Tile and RS_TileExplode
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-461'>SEDONA-461</a>] - ST_IsValidReason
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-465'>SEDONA-465</a>] - Support reading legacy parquet files written by Apache Sedona <= 1.3.1-incubating
+</li>
+</ul>
+
+### Improvement
+
+<ul>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-339'>SEDONA-339</a>] - Skip irrelevant GitHub actions
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-416'>SEDONA-416</a>] - importing SedonaContext, kepler.gl is not found.
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-429'>SEDONA-429</a>] - geoparquet reader/writer should print "1.0.0" in its version
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-434'>SEDONA-434</a>] - Improve reliability by resolve the nondeterministic of the order of the Map
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-436'>SEDONA-436</a>] - Fix RS_SetValues bug
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-437'>SEDONA-437</a>] - Add implicit CRS transformation
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-446'>SEDONA-446</a>] - Add floating point datatype support in RS_AsBase64
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-448'>SEDONA-448</a>] - RS_SetBandNoDataValue should have `replace` option
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-454'>SEDONA-454</a>] - Change the default value of sedona.global.indextype from quadtree to rtree
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-457'>SEDONA-457</a>] - Don't write GeometryUDT into org.apache.spark.sql.parquet.row.metadata when writing GeoParquet files
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-464'>SEDONA-464</a>] - ST_Valid should have integer flags
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-466'>SEDONA-466</a>] - RS_AsRaster does not use the weight and height of the raster in its parameters.
+</li>
+</ul>
+
+### Test
+
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-410'>SEDONA-410</a>] - pre-commit: check that scripts with shebangs are executable
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-412'>SEDONA-412</a>] - pre-commit: add hook `end-of-file-fixer`
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-423'>SEDONA-423</a>] - pre-commit: apply hook `end-of-file-fixer` to more files
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-442'>SEDONA-442</a>] - pre-commit: add hook markdown-lint
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-444'>SEDONA-444</a>] - pre-commit: add hook to trim trailing whitespace
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-445'>SEDONA-445</a>] - pre-commit: apply hook end-of-file-fixer to more files
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-447'>SEDONA-447</a>] - pre-commit: apply end-of-file-fixer to more files
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-463'>SEDONA-463</a>] - Add a Makefile for convenience
+</li>
+</ul>
+
+### Task
+
+<ul>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-450'>SEDONA-450</a>] - Support Spark 3.5
+</li>
+<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-458'>SEDONA-458</a>] - The docs should have examples for UDF
+</li>
+</ul>
+
## Sedona 1.5.0
Sedona 1.5.0 is compiled against Spark 3.3 / Spark 3.4 / Flink 1.12, Java 8.
diff --git a/docs/tutorial/demo.md b/docs/tutorial/demo.md
index 4f177a56b..a21443184 100644
--- a/docs/tutorial/demo.md
+++ b/docs/tutorial/demo.md
@@ -8,10 +8,8 @@ Note that, although the template projects are written in Scala, the same APIs ca
The folder structure of this repository is as follows.
-* spark-rdd-colocation-mining: a scala template shows how to use Sedona RDD API in Spatial Data Mining in Apache Spark
-* spark-sql: a scala template shows how to use Sedona DataFrame and SQL API in Apache Spark
-* spark-viz: a scala template shows how to use Sedona Viz RDD and SQL API in Apache Spark
-* flink-sql: a Java template shows how to use Sedona SQL in Apache Flink
+* spark-sql: a Scala template shows how to use Sedona RDD, DataFrame and SQL API
+* flink-sql: a Java template show how to use Sedona SQL via Flink Table APIs
## Compile and package
@@ -19,24 +17,24 @@ The folder structure of this repository is as follows.
Please make sure you have the following software installed on your local machine:
-* For Scala: Scala 2.12, SBT
+* For Scala: Scala 2.12
* For Java: JDK 1.8, Apache Maven 3
### Compile
-Run a terminal command `sbt assembly` within the folder of each template
+Run a terminal command `mvn clean package` within the folder of each template
-### Submit your fat jar to Spark or Flink
+### Submit your fat jar to Spark
-After running the command mentioned above, you are able to see a fat jar in `./target` folder. Please take it and use `./bin/spark-submit` or `/bin/flink`to submit this jar.
+After running the command mentioned above, you are able to see a fat jar in `./target` folder. Please take it and use `./bin/spark-submit` to submit this jar.
To run the jar in this way, you need to:
-* For Spark: either change Spark Master Address in template projects or simply delete it. Currently, they are hard coded to `local[*]` which means run locally with all cores.
+* Either change Spark Master Address in template projects or simply delete it. Currently, they are hard coded to `local[*]` which means run locally with all cores.
* Change the dependency packaging scope of Apache Spark from "compile" to "provided". This is a common packaging strategy in Maven and SBT which means do not package Spark into your fat jar. Otherwise, this may lead to a huge jar and version conflicts!
-* Make sure the dependency versions in build.sbt/pom.xml are consistent with your Spark/Flink version.
+* Make sure the dependency versions in build.sbt are consistent with your Spark version.
## Run template projects locally
@@ -45,7 +43,3 @@ We highly suggest you use IDEs to run template projects on your local machine. F
### Scala
Import the Scala template project as SBT project. Then run the Main file in this project.
-
-### Java
-
-Import the Java template project as Maven project. Then run the Main file in this project.
diff --git a/examples/flink-sql/pom.xml b/examples/flink-sql/pom.xml
index 60d7b8c24..67b2b35f9 100644
--- a/examples/flink-sql/pom.xml
+++ b/examples/flink-sql/pom.xml
@@ -1,3 +1,22 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
diff --git a/mkdocs.yml b/mkdocs.yml
index 52234ddc6..9da2b6e03 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -157,12 +157,12 @@ extra:
current_version: 1.5.0
current_geotools: 1.5.0-28.2
sedona_create_release:
- current_version: 1.5.0
- current_git_tag: sedona-1.5.0-rc1
- current_rc: 1.5.0-rc1
+ current_version: 1.5.1
+ current_git_tag: sedona-1.5.1-rc1
+ current_rc: 1.5.1-rc1
current_snapshot: 1.5.1-SNAPSHOT
- next_version: 1.5.1
-copyright: Copyright © 2023 The Apache Software Foundation. Apache Sedona, Sedona, Apache, the Apache feather logo, and the Apache Sedona project logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. All other marks mentioned may be trademarks or registered trademarks of their respective owners. Please visit <a href="http://www.apache.org/">Apache Software Foundation</a> for more details.
+ next_version: 1.6.0
+copyright: Copyright © 2024 The Apache Software Foundation. Apache Sedona, Sedona, Apache, the Apache feather logo, and the Apache Sedona project logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. All other marks mentioned may be trademarks or registered trademarks of their respective owners. Please visit <a href="http://www.apache.org/">Apache Software Foundation</a> for more details.
markdown_extensions:
- admonition
diff --git a/python/sedona/version.py b/python/sedona/version.py
index 3c52d221c..07fb50d81 100644
--- a/python/sedona/version.py
+++ b/python/sedona/version.py
@@ -15,4 +15,4 @@
# specific language governing permissions and limitations
# under the License.
-version = "1.5.0"
+version = "1.5.1"
diff --git a/zeppelin/package.json b/zeppelin/package.json
index 645532c31..4211c9ad5 100644
--- a/zeppelin/package.json
+++ b/zeppelin/package.json
@@ -2,7 +2,7 @@
"name": "apache-sedona",
"description": "Zeppelin visualization support for Sedona",
"author": "Apache Sedona, original authors are listed on https://github.com/myuwono/zeppelin-leaflet",
- "version": "1.5.0",
+ "version": "1.5.1",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1",