You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@sedona.apache.org by ji...@apache.org on 2023/08/11 09:15:32 UTC
[sedona] branch master updated: [SEDONA-285] Add the official Docker image (#959)
This is an automated email from the ASF dual-hosted git repository.
jiayu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/sedona.git
The following commit(s) were added to refs/heads/master by this push:
new f7e9e0b2f [SEDONA-285] Add the official Docker image (#959)
f7e9e0b2f is described below
commit f7e9e0b2f44cf69f02de35f5fc99edf311ff3d3c
Author: Jia Yu <ji...@apache.org>
AuthorDate: Fri Aug 11 02:15:27 2023 -0700
[SEDONA-285] Add the official Docker image (#959)
---
binder/Sedona_OvertureMaps_GeoParquet.ipynb | 369 ++++++++++++++++++++++++++++
docker/sedona-spark-jupyterlab/build.sh | 5 +-
docs/setup/docker.md | 21 +-
3 files changed, 384 insertions(+), 11 deletions(-)
diff --git a/binder/Sedona_OvertureMaps_GeoParquet.ipynb b/binder/Sedona_OvertureMaps_GeoParquet.ipynb
new file mode 100644
index 000000000..b47d716a7
--- /dev/null
+++ b/binder/Sedona_OvertureMaps_GeoParquet.ipynb
@@ -0,0 +1,369 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "92984a1c",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from sedona.spark import *\n",
+ "import os\n",
+ "import time"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e4392353",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "DATA_LINK = \"s3a://wherobots-public-data/overturemaps-us-west-2/release/2023-07-26-alpha.0/\""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "08c71acb",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "config = SedonaContext.builder() .\\\n",
+ " config(\"spark.hadoop.fs.s3a.aws.credentials.provider\", \"org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider\"). \\\n",
+ " config(\"fs.s3a.aws.credentials.provider\", \"org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider\"). \\\n",
+ " config('spark.jars.packages',\n",
+ " 'org.apache.sedona:sedona-spark-shaded-3.0_2.12:1.4.1,'\n",
+ " 'org.datasyslab:geotools-wrapper:1.4.0-28.2'). \\\n",
+ " getOrCreate()\n",
+ "\n",
+ "sedona = SedonaContext.create(config)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8f3340ee",
+ "metadata": {},
+ "source": [
+ "# State Boundary"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8b1b506c",
+ "metadata": {},
+ "source": [
+ "### Pick a state.\n",
+ "#### Washington is selected\n",
+ "\n",
+ "[Click here for boundaries of other states](https://gist.github.com/JoshuaCarroll/49630cbeeb254a49986e939a26672e9c)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "045f24b3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Washington state boundary\n",
+ "#spatial_filter = \"POLYGON((-123.3208 49.0023,-123.0338 49.0027,-122.0650 49.0018,-121.7491 48.9973,-121.5912 48.9991,-119.6082 49.0009,-118.0378 49.0005,-117.0319 48.9996,-117.0415 47.9614,-117.0394 46.5060,-117.0394 46.4274,-117.0621 46.3498,-117.0277 46.3384,-116.9879 46.2848,-116.9577 46.2388,-116.9659 46.2022,-116.9254 46.1722,-116.9357 46.1432,-116.9584 46.1009,-116.9762 46.0785,-116.9433 46.0537,-116.9165 45.9960,-118.0330 46.0008,-118.9867 45.9998,-119.1302 45.9320,-119.170 [...]
+ "\n",
+ "# Bellevue city boundary\n",
+ "spatial_filter = \"POLYGON ((-122.235128 47.650163, -122.233796 47.65162, -122.231581 47.653287, -122.228514 47.65482, -122.227526 47.655204, -122.226175 47.655729, -122.222039 47.656743999999996, -122.218428 47.657464, -122.217026 47.657506, -122.21437399999999 47.657588, -122.212091 47.657464, -122.212135 47.657320999999996, -122.21092999999999 47.653552, -122.209834 47.650121, -122.209559 47.648976, -122.209642 47.648886, -122.21042 47.648658999999995, -122.210897 47.64864, -122. [...]
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "63033ca3",
+ "metadata": {},
+ "source": [
+ "### Visualizing Overture Maps\n",
+ "\n",
+ "Explanation to the each step is similar accross the different datasets. [Click here to learn more about Overture Maps.](https://docs.overturemaps.org)\n",
+ "\n",
+ "1. `df = sedona.read.format(\"geoparquet\").load(DATA_LINK+\"theme=XX/type=YY\")`\n",
+ "\n",
+ " It reads the dataset mentioned by theme and type, that's stored in GeoParquet format.\n",
+ "\n",
+ "\n",
+ "2. `df = df.filter(\"ST_Contains(ST_GeomFromWKT('\"+state_boundary+\"'), geometry) = true\")`\n",
+ " \n",
+ " This filters out all the data that is not in the mentioned `state_boundary` string. Please select a state as you wish.\n",
+ " \n",
+ " ST_GeomFromWKT() - constructs a geometry from WKT (Well Known Text)\n",
+ " \n",
+ " ST_Contains(A, B) - checks if A fully contains B and returns True\n",
+ "\n",
+ "3. `XX_geom = df.selectExpr(\"geometry\")`\n",
+ "\n",
+ " Storing geometry column for SedonaKepler.\n",
+ " \n",
+ "\n",
+ "5. `map = SedonaKepler.create_map(XX_geom, 'XX')`\n",
+ "\n",
+ " Creating a map object using SedonaKepler with inputs geometry column and the name of dataset.\n",
+ " \n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "31e81f8b",
+ "metadata": {},
+ "source": [
+ "# Building Dataset "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d0c55157",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%time\n",
+ "\n",
+ "df_building = sedona.read.format(\"geoparquet\").load(DATA_LINK+\"theme=buildings/type=building\")\n",
+ "\n",
+ "df_building = df_building.filter(\"ST_Contains(ST_GeomFromWKT('\"+spatial_filter+\"'), geometry) = true\")\n",
+ "\n",
+ "df_building = df_building.limit(200_000)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bb62f16e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%time\n",
+ "\n",
+ "map_building = SedonaKepler.create_map(df_building, 'Building')\n",
+ "\n",
+ "map_building"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9d2cd7e5",
+ "metadata": {},
+ "source": [
+ "# Place Dataset "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a6443d1d",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%time\n",
+ "\n",
+ "df_place = sedona.read.format(\"geoparquet\").load(DATA_LINK+\"theme=places/type=place\")\n",
+ "\n",
+ "df_place = df_place.filter(\"ST_Contains(ST_GeomFromWKT('\"+spatial_filter+\"'), geometry) = true\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "1f826cb8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%time\n",
+ "\n",
+ "map_place = SedonaKepler.create_map(df_place, \"Place\")\n",
+ "\n",
+ "map_place"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c0809afb",
+ "metadata": {},
+ "source": [
+ "# Admins Theme Datasets"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "930b6c81",
+ "metadata": {},
+ "source": [
+ "## Administrative Boundary Dataset"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6b9d6296",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%time\n",
+ "\n",
+ "df_admin = sedona.read.format(\"geoparquet\").load(DATA_LINK+\"theme=admins/type=administrativeBoundary\")\n",
+ "\n",
+ "df_admin = df_admin.filter(\"ST_Contains(ST_GeomFromWKT('\"+spatial_filter+\"'), geometry) = true\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "2ce9597a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%time\n",
+ "\n",
+ "map_admin = SedonaKepler.create_map(df_admin, \"Admin\")\n",
+ "\n",
+ "map_admin"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0acad0d7",
+ "metadata": {},
+ "source": [
+ "## Locality Dataset"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8541ada7",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%time\n",
+ "\n",
+ "df_locality = sedona.read.format(\"geoparquet\").load(DATA_LINK+\"theme=admins/type=locality\")\n",
+ "\n",
+ "df_locality = df_locality.filter(\"ST_Contains(ST_GeomFromWKT('\"+spatial_filter+\"'), geometry) = true\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "0c5c3bf3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%time\n",
+ "\n",
+ "map_locality = SedonaKepler.create_map(df_locality, 'Locality')\n",
+ "\n",
+ "map_locality"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "328ffd56",
+ "metadata": {},
+ "source": [
+ "# Transportation Theme Datasets"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0b914f57",
+ "metadata": {},
+ "source": [
+ "## Connector Dataset"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "224167dc",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%time\n",
+ "\n",
+ "df_connector = sedona.read.format(\"geoparquet\").load(DATA_LINK+\"theme=transportation/type=connector\")\n",
+ "\n",
+ "df_connector = df_connector.filter(\"ST_Contains(ST_GeomFromWKT('\"+spatial_filter+\"'), geometry) = true\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e5f8de53",
+ "metadata": {
+ "scrolled": false
+ },
+ "outputs": [],
+ "source": [
+ "%%time\n",
+ "\n",
+ "map_connector = SedonaKepler.create_map(df_connector, \"Connector\")\n",
+ "\n",
+ "map_connector"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1e7ab2d",
+ "metadata": {},
+ "source": [
+ "## Segment Dataset"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "ba8eff7c",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%time\n",
+ "\n",
+ "df_segment = sedona.read.format(\"geoparquet\").load(DATA_LINK+\"theme=transportation/type=segment\")\n",
+ "\n",
+ "df_segment = df_segment.filter(\"ST_Contains(ST_GeomFromWKT('\"+spatial_filter+\"'), geometry) = true\")\n",
+ "\n",
+ "df_segment = df_segment.limit(200000)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a9bf0862",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%time\n",
+ "\n",
+ "map_segment = SedonaKepler.create_map(df_segment, \"Segment\")\n",
+ "\n",
+ "map_segment"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.12"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/docker/sedona-spark-jupyterlab/build.sh b/docker/sedona-spark-jupyterlab/build.sh
index 4933aa07b..aa45516f1 100755
--- a/docker/sedona-spark-jupyterlab/build.sh
+++ b/docker/sedona-spark-jupyterlab/build.sh
@@ -41,14 +41,15 @@ if [ -z "$BUILD_MODE" ] || [ "$BUILD_MODE" = "local" ]; then
--build-arg spark_version="${SPARK_VERSION}" \
--build-arg sedona_version="${SEDONA_VERSION}" \
-f docker/sedona-spark-jupyterlab/sedona-jupyterlab.dockerfile \
- -t sedona/sedona-jupyterlab:${SEDONA_VERSION} .
+ -t apache/sedona:${SEDONA_VERSION} .
else
# If release, build the image for cross-platform
docker buildx build --platform linux/amd64,linux/arm64 \
--progress=plain \
+ --no-cache \
--output type=registry \
--build-arg spark_version="${SPARK_VERSION}" \
--build-arg sedona_version="${SEDONA_VERSION}" \
-f docker/sedona-spark-jupyterlab/sedona-jupyterlab.dockerfile \
- -t drjiayu/sedona-jupyterlab:${SEDONA_VERSION} .
+ -t apache/sedona:${SEDONA_VERSION} .
fi
\ No newline at end of file
diff --git a/docs/setup/docker.md b/docs/setup/docker.md
index 21fb87aa6..59bf9198e 100644
--- a/docs/setup/docker.md
+++ b/docs/setup/docker.md
@@ -1,6 +1,8 @@
# Sedona JupyterLab Docker Image
-Dockerfiles for Apache Sedona with JupyterLab and 1 master node and 1 worker node
+Sedona Docker images are available on [Sedona official DockerHub repo](https://hub.docker.com/r/apache/sedona).
+
+We provide a Docker image for Apache Sedona with Python JupyterLab and 1 master node and 1 worker node.
## How to use
@@ -9,19 +11,19 @@ Dockerfiles for Apache Sedona with JupyterLab and 1 master node and 1 worker nod
Format:
```bash
-docker pull drjiayu/sedona-jupyterlab:<sedona_version>
+docker pull apache/sedona:<sedona_version>
```
Example 1: Pull the latest image of Sedona master branch
```bash
-docker pull drjiayu/sedona-jupyterlab:latest
+docker pull apache/sedona:latest
```
Example 2: Pull the image of a specific Sedona release
```bash
-docker pull drjiayu/sedona-jupyterlab:1.4.1
+docker pull apache/sedona:1.4.1
```
### Start the container
@@ -29,19 +31,19 @@ docker pull drjiayu/sedona-jupyterlab:1.4.1
Format:
```bash
-docker run -p 8888:8888 -p 8080:8080 -p 8081:8081 -p 4040:4040 drjiayu/sedona-jupyterlab:<sedona_version>
+docker run -p 8888:8888 -p 8080:8080 -p 8081:8081 -p 4040:4040 apache/sedona:<sedona_version>
```
Example 1:
```bash
-docker run -p 8888:8888 -p 8080:8080 -p 8081:8081 -p 4040:4040 drjiayu/sedona-jupyterlab:latest
+docker run -p 8888:8888 -p 8080:8080 -p 8081:8081 -p 4040:4040 apache/sedona:latest
```
Example 2:
```bash
-docker run -p 8888:8888 -p 8080:8080 -p 8081:8081 -p 4040:4040 drjiayu/sedona-jupyterlab:1.4.1
+docker run -p 8888:8888 -p 8080:8080 -p 8081:8081 -p 4040:4040 apache/sedona:1.4.1
```
This command will bind the container's ports 8888, 8080, 8081, 4040 to the host's ports 8888, 8080, 8081, 4040 respectively.
@@ -52,7 +54,7 @@ Open your browser and go to [http://localhost:8888/](http://localhost:8888/) to
### Notes
-* This container assumes you have at least 8GB RAM and takes all your CPU cores and 8GM RAM.
+* This container assumes you have at least 8GB RAM and takes all your CPU cores and 8GM RAM. The 1 worker will take 4GB and the Jupyter program will take the remaining 4GB.
* Sedona in this container runs in the cluster mode. Only 1 notebook can be run at a time. If you want to run another notebook, please shut down the kernel of the current notebook first ([How?](https://jupyterlab.readthedocs.io/en/stable/user/running.html)).
## How to build
@@ -105,13 +107,14 @@ This docker image can only be built against Sedona 1.4.1+ and Spark 3.0+
* OS: Ubuntu 22.02
* JDK: openjdk-19
* Python: 3.10
+* Spark 3.4.1
### Web UI
* JupyterLab: http://localhost:8888/
* Spark master URL: spark://localhost:7077
* Spark job UI: http://localhost:4040
* Spark master web UI: http://localhost:8080/
-* Spark web UI: http://localhost:8081/
+* Spark work web UI: http://localhost:8081/
## How to push to DockerHub