You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@sedona.apache.org by ji...@apache.org on 2024/07/02 07:40:42 UTC
(sedona) branch master updated: [DOCS] AWS Glue Tutorial: replace s3 instructions with maven links (#1506)
This is an automated email from the ASF dual-hosted git repository.
jiayu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/sedona.git
The following commit(s) were added to refs/heads/master by this push:
new a50f35fbb [DOCS] AWS Glue Tutorial: replace s3 instructions with maven links (#1506)
a50f35fbb is described below
commit a50f35fbbc95ae71ce873ea90f8953425383ad49
Author: James Willis <ja...@users.noreply.github.com>
AuthorDate: Tue Jul 2 00:40:37 2024 -0700
[DOCS] AWS Glue Tutorial: replace s3 instructions with maven links (#1506)
* AWS Glue Tutorial: replace s3 instructions with maven links
* capitalize python
* use current_geotools
---------
Co-authored-by: jameswillis <ja...@wherobots.com>
---
docs/setup/glue.md | 51 +++++++++++++--------------------------------------
1 file changed, 13 insertions(+), 38 deletions(-)
diff --git a/docs/setup/glue.md b/docs/setup/glue.md
index e13e6b478..09f038439 100644
--- a/docs/setup/glue.md
+++ b/docs/setup/glue.md
@@ -1,35 +1,18 @@
This tutorial will cover how to configure both a glue notebook and a glue ETL job. The tutorial is written assuming you
-have a working knowledge of AWS Glue jobs and S3.
+have a working knowledge of AWS Glue jobs.
In the tutorial, we use
Sedona {{ sedona.current_version }} and [Glue 4.0](https://docs.aws.amazon.com/glue/latest/dg/release-notes.html) which runs on Spark 3.3.0, Java 8, Scala 2.12,
and Python 3.10. We recommend Sedona-1.3.1-incubating and above for Glue.
-## Stage Sedona Jar in S3
+## Gather Maven Links
-In an AWS S3 bucket you will need the Sedona-spark-shaded and geotools-wrapper jars. There are two options to get these.
-Ensure the locations of the jars are accessible to your glue job.
+You will need to point your glue job to the Sedona and Geotools jars. We recommend using the jars available from maven. The links below are those intended for Glue 4.0
-!!!note
- Ensure you pick a version for Scala 2.12 and Spark 3.0. The Spark 3.4 and Scala 2.13 jars are not compatible with
- Glue 4.0.
-
-### Option 1: Use the Wherobots-hosted jars
-
-[Wherobots](https://wherobots.com/) provides a public S3 bucket with the necessary jars. You can point to these directly in your glue jobs'
-configurations. For {{ sedona.current_version }}, the Sedona and geotools jars are available at the following locations:
-
-* `s3://wherobots-sedona-jars/{{ sedona.current_version }}/sedona-spark-shaded-3.0_2.12-{{ sedona.current_version }}.jar`
-* `s3://wherobots-sedona-jars/{{ sedona.current_version }}/geotools-wrapper-{{ sedona.current_version }}-28.2.jar`
-
-### Option 2: Stage your own Jars
-
-In your S3 bucket, add the Sedona Jar from
-[Maven Central](https://repo1.maven.org/maven2/org/apache/sedona/sedona-spark-shaded-3.0_2.12/{{ sedona.current_version }}/sedona-spark-shaded-3.0_2.12-{{ sedona.current_version }}.jar).
+Sedona Jar: [Maven Central](https://repo1.maven.org/maven2/org/apache/sedona/sedona-spark-shaded-3.0_2.12/{{ sedona.current_version }}/sedona-spark-shaded-3.0_2.12-{{ sedona.current_version }}.jar)
-Similarly, add the Geotools Wrapper Jar
-from [Maven Central](https://repo1.maven.org/maven2/org/datasyslab/geotools-wrapper/{{ sedona.current_version }}-28.2/geotools-wrapper-{{ sedona.current_version }}-28.2.jar).
+Geotools Jar: [Maven Central](https://repo1.maven.org/maven2/org/datasyslab/geotools-wrapper/{{ sedona.current_geotools }}/geotools-wrapper-{{ sedona.current_geotools }}.jar)
!!!note
If you use Sedona 1.3.1-incubating, please use `sedona-python-adpater-3.0_2.12` jar in the content above, instead
@@ -38,28 +21,20 @@ from [Maven Central](https://repo1.maven.org/maven2/org/datasyslab/geotools-wrap
## Configure Glue Job
-Once your jars are staged in S3, you can configure your Glue job to use them, as well as the apache-sedona python
+Once you have your jar links, you can configure your Glue job to use them, as well as the apache-sedona Python
package. How you do this varies slightly between the notebook and the script job types.
!!!note
- Always ensure that the Sedona version of the jars and the python package match.
+ Always ensure that the Sedona version of the jars and the Python package match.
### Notebook Job
-Add the following cell magics before starting your sparkContext of glueContext. The first points to the jars put in s3,
-and the second installs the Sedona python package directly from pip.
-
-```python
-# Sedona Config
-%extra_jars s3://path/to/my-sedona.jar, s3://path/to/my-geotools.jar
-%additional_python_modules apache-sedona
-```
-
-If using the Wherobots-provided jars, the cell magics would look like this:
+Add the following cell magics before starting your sparkContext or glueContext. The first points to the jars,
+and the second installs the Sedona Python package directly from pip.
```python
# Sedona Config
-%extra_jars s3://wherobots-sedona-jars/{{ sedona.current_version }}/sedona-spark-shaded-3.0_2.12-{{ sedona.current_version }}.jar, s3://wherobots-sedona-jars/{{ sedona.current_version }}/geotools-wrapper-{{ sedona.current_version }}-28.2.jar
+%extra_jars https://repo1.maven.org/maven2/org/apache/sedona/sedona-spark-shaded-3.0_2.12/{{ sedona.current_version }}/sedona-spark-shaded-3.0_2.12-{{ sedona.current_version }}.jar, https://repo1.maven.org/maven2/org/datasyslab/geotools-wrapper/{{ sedona.current_geotools }}/geotools-wrapper-{{ sedona.current_geotools }}.jar
%additional_python_modules apache-sedona=={{ sedona.current_version }}
```
@@ -72,7 +47,7 @@ If you are using the example notebook from glue, the first cell should now look
%number_of_workers 5
# Sedona Config
-%extra_jars s3://wherobots-sedona-jars/{{ sedona.current_version }}/sedona-spark-shaded-3.0_2.12-{{ sedona.current_version }}.jar, s3://wherobots-sedona-jars/{{ sedona.current_version }}/geotools-wrapper-{{ sedona.current_version }}-28.2.jar
+%extra_jars https://repo1.maven.org/maven2/org/apache/sedona/sedona-spark-shaded-3.0_2.12/{{ sedona.current_version }}/sedona-spark-shaded-3.0_2.12-{{ sedona.current_version }}.jar, https://repo1.maven.org/maven2/org/datasyslab/geotools-wrapper/{{ sedona.current_geotools }}/geotools-wrapper-{{ sedona.current_geotools }}.jar
%additional_python_modules apache-sedona=={{ sedona.current_version }}
@@ -101,9 +76,9 @@ sedona.sql("SELECT ST_POINT(1., 2.) as geom").show()
### ETL Job
Glue also calls these Scripts. From your job's page, navigate to the "Job details" tab. At the bottom of the page expand
-the "Advanced properties" section. In the "Dependent JARs path" field, add the paths to the jars in S3, separated by a comma.
+the "Advanced properties" section. In the "Dependent JARs path" field, add the paths to the jars, separated by a comma.
-To add the Sedona python package, navigate to the "Job Parameters" section and add a new parameter with the key
+To add the Sedona Python package, navigate to the "Job Parameters" section and add a new parameter with the key
`--additional-python-modules` and the value `apache-sedona=={{ sedona.current_version }}`.
To confirm the installation add the follow code to the script: