You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@sedona.apache.org by ji...@apache.org on 2024/01/18 08:46:44 UTC
(sedona) branch docker-1.5.1 created (now e8b736622)
This is an automated email from the ASF dual-hosted git repository.
jiayu pushed a change to branch docker-1.5.1
in repository https://gitbox.apache.org/repos/asf/sedona.git
at e8b736622 Update binder notebooks and docker image
This branch includes the following new commits:
new e8b736622 Update binder notebooks and docker image
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(sedona) 01/01: Update binder notebooks and docker image
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jiayu pushed a commit to branch docker-1.5.1
in repository https://gitbox.apache.org/repos/asf/sedona.git
commit e8b73662277297872cbd8825f0cf132d71536ff9
Author: Jia Yu <ji...@apache.org>
AuthorDate: Thu Jan 18 00:46:35 2024 -0800
Update binder notebooks and docker image
---
binder/ApacheSedonaCore.ipynb | 5 +-
binder/ApacheSedonaRaster.ipynb | 21 +-
binder/ApacheSedonaSQL.ipynb | 5 +-
...eSedonaSQL_SpatialJoin_AirportsPerCountry.ipynb | 5 +-
binder/Pipfile | 3 +-
binder/Sedona_OvertureMaps_GeoParquet.ipynb | 592 ++++++++++++++++++---
.../sedona-jupyterlab.dockerfile | 18 +-
docker/sedona.sh | 8 +
docker/spark.sh | 2 -
docs/setup/docker.md | 11 +-
10 files changed, 553 insertions(+), 117 deletions(-)
diff --git a/binder/ApacheSedonaCore.ipynb b/binder/ApacheSedonaCore.ipynb
index 01a5f5847..cce339f79 100644
--- a/binder/ApacheSedonaCore.ipynb
+++ b/binder/ApacheSedonaCore.ipynb
@@ -51,8 +51,9 @@
"source": [
"config = SedonaContext.builder() .\\\n",
" config('spark.jars.packages',\n",
- " 'org.apache.sedona:sedona-spark-shaded-3.4_2.12:1.5.0,'\n",
- " 'org.datasyslab:geotools-wrapper:1.5.0-28.2'). \\\n",
+ " 'org.apache.sedona:sedona-spark-shaded-3.4_2.12:1.5.1,'\n",
+ " 'org.datasyslab:geotools-wrapper:1.5.1-28.2,'\n",
+ " 'uk.co.gresearch.spark:spark-extension_2.12:2.11.0-3.4'). \\\n",
" getOrCreate()\n",
"\n",
"sedona = SedonaContext.create(config)"
diff --git a/binder/ApacheSedonaRaster.ipynb b/binder/ApacheSedonaRaster.ipynb
index a5cafa3d8..16ec1bee6 100644
--- a/binder/ApacheSedonaRaster.ipynb
+++ b/binder/ApacheSedonaRaster.ipynb
@@ -64,8 +64,9 @@
"source": [
"config = SedonaContext.builder() .\\\n",
" config('spark.jars.packages',\n",
- " 'org.apache.sedona:sedona-spark-shaded-3.4_2.12:1.5.0,'\n",
- " 'org.datasyslab:geotools-wrapper:1.5.0-28.2'). \\\n",
+ " 'org.apache.sedona:sedona-spark-shaded-3.4_2.12:1.5.1,'\n",
+ " 'org.datasyslab:geotools-wrapper:1.5.1-28.2,'\n",
+ " 'uk.co.gresearch.spark:spark-extension_2.12:2.11.0-3.4'). \\\n",
" getOrCreate()\n",
"\n",
"sedona = SedonaContext.create(config)\n",
@@ -157,22 +158,6 @@
"Sedona 1.5.0 provides [multiple ways to be able to visualize rasters](https://sedona.apache.org/1.5.0/api/sql/Raster-visualizer/). Throughout this notebook, [RS_AsImage](https://sedona.apache.org/1.5.0/api/sql/Raster-visualizer/#rs_asimage) will be used to visualize any changes to the rasters."
]
},
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "d5f615f4-a3d6-407c-aea9-58891c1e55e3",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "# Define a simple wrapper to display HTML in jupyter notebook environment\n",
- "class SedonaUtils:\n",
- " @classmethod\n",
- " def display_image(cls, df):\n",
- " display(HTML(df.toPandas().to_html(escape=False)))"
- ]
- },
{
"cell_type": "code",
"execution_count": null,
diff --git a/binder/ApacheSedonaSQL.ipynb b/binder/ApacheSedonaSQL.ipynb
index 9873e8464..00419c73b 100644
--- a/binder/ApacheSedonaSQL.ipynb
+++ b/binder/ApacheSedonaSQL.ipynb
@@ -48,8 +48,9 @@
"source": [
"config = SedonaContext.builder() .\\\n",
" config('spark.jars.packages',\n",
- " 'org.apache.sedona:sedona-spark-shaded-3.4_2.12:1.5.0,'\n",
- " 'org.datasyslab:geotools-wrapper:1.5.0-28.2'). \\\n",
+ " 'org.apache.sedona:sedona-spark-shaded-3.4_2.12:1.5.1,'\n",
+ " 'org.datasyslab:geotools-wrapper:1.5.1-28.2,'\n",
+ " 'uk.co.gresearch.spark:spark-extension_2.12:2.11.0-3.4'). \\\n",
" getOrCreate()\n",
"\n",
"sedona = SedonaContext.create(config)\n"
diff --git a/binder/ApacheSedonaSQL_SpatialJoin_AirportsPerCountry.ipynb b/binder/ApacheSedonaSQL_SpatialJoin_AirportsPerCountry.ipynb
index e104ac791..a1c293382 100644
--- a/binder/ApacheSedonaSQL_SpatialJoin_AirportsPerCountry.ipynb
+++ b/binder/ApacheSedonaSQL_SpatialJoin_AirportsPerCountry.ipynb
@@ -55,8 +55,9 @@
"source": [
"config = SedonaContext.builder() .\\\n",
" config('spark.jars.packages',\n",
- " 'org.apache.sedona:sedona-spark-shaded-3.4_2.12:1.5.0,'\n",
- " 'org.datasyslab:geotools-wrapper:1.5.0-28.2'). \\\n",
+ " 'org.apache.sedona:sedona-spark-shaded-3.4_2.12:1.5.1,'\n",
+ " 'org.datasyslab:geotools-wrapper:1.5.1-28.2,'\n",
+ " 'uk.co.gresearch.spark:spark-extension_2.12:2.11.0-3.4'). \\\n",
" getOrCreate()\n",
"\n",
"sedona = SedonaContext.create(config)\n",
diff --git a/binder/Pipfile b/binder/Pipfile
index 69f756952..2e91787ba 100644
--- a/binder/Pipfile
+++ b/binder/Pipfile
@@ -16,7 +16,7 @@ shapely="==1.8.4"
geopandas="==0.11.1"
pyspark="==3.4.0"
attrs="*"
-apache-sedona="==1.5.0"
+apache-sedona="==1.5.1"
matplotlib = "*"
descartes = "*"
keplergl = "==0.3.2"
@@ -25,6 +25,7 @@ ipywidgets = "*"
jupyterlab-widgets = "==1.1.7"
ipykernel = "*"
jupyterlab = "==3.6.4"
+pyspark-extension = "==2.11.0.3.4"
[requires]
python_version = "3.9"
diff --git a/binder/Sedona_OvertureMaps_GeoParquet.ipynb b/binder/Sedona_OvertureMaps_GeoParquet.ipynb
index 588cdf415..31a1e0a8c 100644
--- a/binder/Sedona_OvertureMaps_GeoParquet.ipynb
+++ b/binder/Sedona_OvertureMaps_GeoParquet.ipynb
@@ -4,74 +4,105 @@
"cell_type": "code",
"execution_count": 1,
"id": "92984a1c",
- "metadata": {},
+ "metadata": {
+ "tags": []
+ },
"outputs": [],
"source": [
"from sedona.spark import *\n",
"import os\n",
- "import time"
+ "import time\n",
+ "import gresearch.spark.parquet\n",
+ "import geopandas as gpd"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "66a56fcb-27a5-4b37-a913-05a8f887d22f",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "# Overture Map data"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3cfd484a-e6ac-4cdb-a8ac-2ed77fbd23db",
+ "metadata": {},
+ "source": [
+ "## Wherobots version\n",
+ "\n",
+ "Wherobots only releases a version for overturemaps-us-west-2/release/2023-07-26-alpha.0/.\n",
+ "\n",
+ "This data is in GeoParquet format and data is clusterd by their spatial promity to ensure effient filter pushdown performance"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "e4392353",
- "metadata": {},
+ "metadata": {
+ "tags": []
+ },
"outputs": [],
"source": [
"DATA_LINK = \"s3a://wherobots-public-data/overturemaps-us-west-2/release/2023-07-26-alpha.0/\""
]
},
+ {
+ "cell_type": "markdown",
+ "id": "b5fcf043-a97d-4321-ac16-fafcdbbcf3aa",
+ "metadata": {},
+ "source": [
+ "## OMF versions\n",
+ "\n",
+ "The following files are official OMF releases. They are GeoParquet files generated by Apache Sedona.\n",
+ "\n",
+ "However, unlike the Wherobots version, these data is not clustered by spatial proxiity and hence the spatial queries on them are very slow\n"
+ ]
+ },
{
"cell_type": "code",
"execution_count": 3,
+ "id": "937cd401-f973-45dc-aabe-735d9a41b4c2",
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [],
+ "source": [
+ "# DATA_LINK = \"s3a://overturemaps-us-west-2/release/2023-11-14-alpha.0/\"\n",
+ "# DATA_LINK = \"s3a://overturemaps-us-west-2/release/2023-12-14-alpha.0/\"\n",
+ "# DATA_LINK = \"s3a://overturemaps-us-west-2/release/2024-01-17-alpha.0/\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b9ddd6e7-6e7e-4f2c-b2ae-83ddf56ed198",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "# Create Sedona Context"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
"id": "08c71acb",
- "metadata": {},
+ "metadata": {
+ "tags": []
+ },
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Warning: Ignoring non-Spark config property: fs.s3a.aws.credentials.provider\n",
- "23/10/31 18:38:20 WARN Utils: Your hostname, Nileshs-MacBook-Pro.local resolves to a loopback address: 127.0.0.1; using 172.24.19.124 instead (on interface en0)\n",
- "23/10/31 18:38:20 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address\n",
- "Ivy Default Cache set to: /Users/nileshgajwani/.ivy2/cache\n",
- "The jars for the packages stored in: /Users/nileshgajwani/.ivy2/jars\n",
- "org.apache.sedona#sedona-spark-shaded-3.4_2.12 added as a dependency\n",
- "org.datasyslab#geotools-wrapper added as a dependency\n",
- ":: resolving dependencies :: org.apache.spark#spark-submit-parent-44adfadb-de72-4666-a002-5a5ad761da2e;1.0\n",
- "\tconfs: [default]\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- ":: loading settings :: url = jar:file:/Users/nileshgajwani/Downloads/spark-3.4.1-bin-hadoop3/jars/ivy-2.5.1.jar!/org/apache/ivy/core/settings/ivysettings.xml\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "\tfound org.apache.sedona#sedona-spark-shaded-3.4_2.12;1.5.0 in central\n",
- "\tfound org.datasyslab#geotools-wrapper;1.5.0-28.2 in central\n",
- ":: resolution report :: resolve 70ms :: artifacts dl 6ms\n",
- "\t:: modules in use:\n",
- "\torg.apache.sedona#sedona-spark-shaded-3.4_2.12;1.5.0 from central in [default]\n",
- "\torg.datasyslab#geotools-wrapper;1.5.0-28.2 from central in [default]\n",
- "\t---------------------------------------------------------------------\n",
- "\t| | modules || artifacts |\n",
- "\t| conf | number| search|dwnlded|evicted|| number|dwnlded|\n",
- "\t---------------------------------------------------------------------\n",
- "\t| default | 2 | 0 | 0 | 0 || 2 | 0 |\n",
- "\t---------------------------------------------------------------------\n",
- ":: retrieving :: org.apache.spark#spark-submit-parent-44adfadb-de72-4666-a002-5a5ad761da2e\n",
- "\tconfs: [default]\n",
- "\t0 artifacts copied, 2 already retrieved (0kB/2ms)\n",
- "23/10/31 18:38:21 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
"Setting default log level to \"WARN\".\n",
- "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n"
+ "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n",
+ "24/01/18 08:40:57 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
+ " \r"
]
}
],
@@ -80,8 +111,9 @@
" config(\"spark.hadoop.fs.s3a.aws.credentials.provider\", \"org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider\"). \\\n",
" config(\"fs.s3a.aws.credentials.provider\", \"org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider\"). \\\n",
" config('spark.jars.packages',\n",
- " 'org.apache.sedona:sedona-spark-shaded-3.4_2.12:1.5.0,'\n",
- " 'org.datasyslab:geotools-wrapper:1.5.0-28.2'). \\\n",
+ " 'org.apache.sedona:sedona-spark-shaded-3.4_2.12:1.5.1,'\n",
+ " 'org.datasyslab:geotools-wrapper:1.5.1-28.2,'\n",
+ " 'uk.co.gresearch.spark:spark-extension_2.12:2.11.0-3.4'). \\\n",
" getOrCreate()\n",
"\n",
"sedona = SedonaContext.create(config)"
@@ -92,7 +124,7 @@
"id": "8f3340ee",
"metadata": {},
"source": [
- "# State BoundarySedonaContext"
+ "# Spatial filter by boundary"
]
},
{
@@ -100,17 +132,19 @@
"id": "8b1b506c",
"metadata": {},
"source": [
- "### Pick a state.\n",
- "#### Washington is selected\n",
+ "### Pick a boundary.\n",
+ "#### Bellevue city is selected\n",
"\n",
"[Click here for boundaries of other states](https://gist.github.com/JoshuaCarroll/49630cbeeb254a49986e939a26672e9c)"
]
},
{
"cell_type": "code",
- "execution_count": 4,
+ "execution_count": 5,
"id": "045f24b3",
- "metadata": {},
+ "metadata": {
+ "tags": []
+ },
"outputs": [],
"source": [
"# Washington state boundary\n",
@@ -155,23 +189,112 @@
},
{
"cell_type": "markdown",
- "id": "31e81f8b",
+ "id": "9d2cd7e5",
"metadata": {},
"source": [
- "# Building Dataset "
+ "# Place Dataset "
]
},
{
- "cell_type": "code",
- "execution_count": 5,
- "id": "d0c55157",
+ "cell_type": "markdown",
+ "id": "ea27a0fc-88f6-4a0d-82e6-82108694265e",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "## Inspect the metadata of GeoParquet files"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "49a4e695-7c34-4e6a-9758-f46f6e94b8ac",
"metadata": {},
+ "source": [
+ "Inspect the parquet metadata of the building dataset using spark-extension. Detailed usage can be found here: https://github.com/G-Research/spark-extension/blob/master/PARQUET.md"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "1b2bdd89-2a3f-406b-a655-1892cfcdc0ed",
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "24/01/18 08:41:03 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties\n",
+ "[Stage 5:> (0 + 1) / 1]\r"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "+--------------------+-----+----------+---------------+-----------------+------+-------+--------+-------+\n",
+ "| filename|block|blockStart|compressedBytes|uncompressedBytes| rows|columns| values| nulls|\n",
+ "+--------------------+-----+----------+---------------+-----------------+------+-------+--------+-------+\n",
+ "|s3a://wherobots-p...| 1| 4| 132489370| 212295497|675704| 26|27013095|4170699|\n",
+ "|s3a://wherobots-p...| 2| 132489374| 33009562| 51995284|168295| 26| 6727332|1039665|\n",
+ "|s3a://wherobots-p...| 1| 4| 131748222| 207244630|658924| 26|26686490|4121025|\n",
+ "|s3a://wherobots-p...| 2| 131748226| 131665990| 207128595|658924| 26|26681616|4123667|\n",
+ "|s3a://wherobots-p...| 3| 263414216| 131697923| 207173797|658924| 26|26686185|4122635|\n",
+ "|s3a://wherobots-p...| 4| 395112139| 128013214| 201361886|640397| 26|25932308|4002991|\n",
+ "|s3a://wherobots-p...| 1| 4| 131748222| 207244630|658924| 26|26686490|4121025|\n",
+ "|s3a://wherobots-p...| 2| 131748226| 131665990| 207128595|658924| 26|26681616|4123667|\n",
+ "|s3a://wherobots-p...| 3| 263414216| 131697923| 207173797|658924| 26|26686185|4122635|\n",
+ "|s3a://wherobots-p...| 4| 395112139| 128013214| 201361886|640397| 26|25932308|4002991|\n",
+ "|s3a://wherobots-p...| 1| 4| 131748222| 207244630|658924| 26|26686490|4121025|\n",
+ "|s3a://wherobots-p...| 2| 131748226| 131665990| 207128595|658924| 26|26681616|4123667|\n",
+ "|s3a://wherobots-p...| 3| 263414216| 131697923| 207173797|658924| 26|26686185|4122635|\n",
+ "|s3a://wherobots-p...| 4| 395112139| 128013214| 201361886|640397| 26|25932308|4002991|\n",
+ "|s3a://wherobots-p...| 1| 4| 131738563| 217599597|677442| 26|27329024|4088909|\n",
+ "|s3a://wherobots-p...| 2| 131738567| 132112948| 218210876|679198| 26|27405447|4099796|\n",
+ "|s3a://wherobots-p...| 3| 263851515| 72665451| 119480256|373256| 26|15060504|2251382|\n",
+ "|s3a://wherobots-p...| 1| 4| 131738563| 217599597|677442| 26|27329024|4088909|\n",
+ "|s3a://wherobots-p...| 2| 131738567| 132112948| 218210876|679198| 26|27405447|4099796|\n",
+ "|s3a://wherobots-p...| 3| 263851515| 72665451| 119480256|373256| 26|15060504|2251382|\n",
+ "+--------------------+-----+----------+---------------+-----------------+------+-------+--------+-------+\n",
+ "only showing top 20 rows\n",
+ "\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ " \r"
+ ]
+ }
+ ],
+ "source": [
+ "sedona.read.parquet_blocks(DATA_LINK+\"theme=places/type=place\").show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ea862955-a338-452d-8b99-2ed2e49a35b5",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "Inspect the GeoParquet metadata using Sedona `geoparquet.metadata`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "5cf0ee65-2ec3-4f8d-937e-458005d391f3",
+ "metadata": {
+ "tags": []
+ },
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
- "23/10/31 18:38:24 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties\n",
" \r"
]
},
@@ -179,32 +302,72 @@
"name": "stdout",
"output_type": "stream",
"text": [
- "CPU times: user 15.4 ms, sys: 5.42 ms, total: 20.8 ms\n",
- "Wall time: 9.82 s\n"
+ "root\n",
+ " |-- version: string (nullable = true)\n",
+ " |-- primary_column: string (nullable = true)\n",
+ " |-- columns: map (nullable = true)\n",
+ " | |-- key: string\n",
+ " | |-- value: struct (valueContainsNull = true)\n",
+ " | | |-- encoding: string (nullable = true)\n",
+ " | | |-- geometry_types: array (nullable = true)\n",
+ " | | | |-- element: string (containsNull = true)\n",
+ " | | |-- bbox: array (nullable = true)\n",
+ " | | | |-- element: double (containsNull = true)\n",
+ " | | |-- crs: string (nullable = true)\n",
+ " |-- geohash: string (nullable = true)\n",
+ "\n"
]
}
],
"source": [
- "%%time\n",
- "\n",
- "df_building = sedona.read.format(\"geoparquet\").load(DATA_LINK+\"theme=buildings/type=building\")\n",
- "\n",
- "df_building = df_building.filter(\"ST_Contains(ST_GeomFromWKT('\"+spatial_filter+\"'), geometry) = true\")\n",
- "\n",
- "df_building = df_building.limit(200_000)"
+ "sedona.read.format(\"geoparquet.metadata\").load(DATA_LINK+\"theme=places/type=place\").drop(\"path\").printSchema()"
]
},
{
"cell_type": "code",
- "execution_count": null,
- "id": "bb62f16e",
- "metadata": {},
+ "execution_count": 8,
+ "id": "c0916fc4-3ecb-4cb8-88f3-d8822fa755e1",
+ "metadata": {
+ "tags": []
+ },
"outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "[Stage 9:=======================================> (2 + 1) / 3]\r"
+ ]
+ },
{
"name": "stdout",
"output_type": "stream",
"text": [
- "User Guide: https://docs.kepler.gl/docs/keplergl-jupyter\n"
+ "+------------+--------------+------------------------------------------------------------------------------------------+-------+\n",
+ "|version |primary_column|columns |geohash|\n",
+ "+------------+--------------+------------------------------------------------------------------------------------------+-------+\n",
+ "|1.0.0-beta.1|geometry |{geometry -> {WKB, [Point], [-45.0, 45.0038534, -33.75027, 50.5347176], null}} |g0 |\n",
+ "|1.0.0-beta.1|geometry |{geometry -> {WKB, [Point], [-45.0, 33.8141064, -33.7678528, 39.3735873], null}} |en |\n",
+ "|1.0.0-beta.1|geometry |{geometry -> {WKB, [Point], [135.378685, -78.2065631, 145.546875, -73.1334471], null}} |p4 |\n",
+ "|1.0.0-beta.1|geometry |{geometry -> {WKB, [Point], [101.4470901, -50.2472049, 112.3773706, -45.0773997], null}} |nr |\n",
+ "|1.0.0-beta.1|geometry |{geometry -> {WKB, [Point], [112.7343607, 11.25, 123.7499978, 16.8749], null}} |wd |\n",
+ "|1.0.0-beta.1|geometry |{geometry -> {WKB, [Point], [78.75, -33.6608961, 89.6400654, -28.1843664], null}} |mf |\n",
+ "|1.0.0-beta.1|geometry |{geometry -> {WKB, [Point], [80.15625, 78.9564349, 88.9382744, 84.1249732], null}} |vy |\n",
+ "|1.0.0-beta.1|geometry |{geometry -> {WKB, [Point], [-146.25, 56.2921567, -135.0004853, 61.8691013], null}} |bf |\n",
+ "|1.0.0-beta.1|geometry |{geometry -> {WKB, [Point], [-11.25, 39.37504, -2.12E-5, 44.9999908], null}} |ez |\n",
+ "|1.0.0-beta.1|geometry |{geometry -> {WKB, [Point], [48.51614, 84.6080623, 48.51614, 84.6080623], null}} |vp |\n",
+ "|1.0.0-beta.1|geometry |{geometry -> {WKB, [Point], [146.3126564, 0.0542566, 156.09375, 5.6159858], null}} |x2 |\n",
+ "|1.0.0-beta.1|geometry |{geometry -> {WKB, [Point], [67.5, 61.8817, 78.7172, 67.488], null}} |ve |\n",
+ "|1.0.0-beta.1|geometry |{geometry -> {WKB, [Point], [-22.4986671, -22.4719545, -11.2554932, -16.9467365], null}} |7s |\n",
+ "|1.0.0-beta.1|geometry |{geometry -> {WKB, [Point], [-134.8484052, 17.0052467, -123.7679812, 22.4397813], null}} |95 |\n",
+ "|1.0.0-beta.1|geometry |{geometry -> {WKB, [Point], [-146.25, -11.1931059, -135.0013733, -6.0916292], null}} |2y |\n",
+ "|1.0.0-beta.1|geometry |{geometry -> {WKB, [Point], [-135.0, -83.8333, -124.4478565, -78.7677918], null}} |11 |\n",
+ "|1.0.0-beta.1|geometry |{geometry -> {WKB, [Point], [-44.8187256, -56.1774766, -34.1630816, -50.8584471], null}} |5n |\n",
+ "|1.0.0-beta.1|geometry |{geometry -> {WKB, [Point], [-44.619453, -44.9958826, -33.7843399, -39.4124306], null}} |70 |\n",
+ "|1.0.0-beta.1|geometry |{geometry -> {WKB, [Point], [-112.2099566, -33.7316387, -101.2729317, -28.2008956], null}}|3d |\n",
+ "|1.0.0-beta.1|geometry |{geometry -> {WKB, [Point], [-33.2666016, -72.9196355, -22.6803555, -67.5007358], null}} |57 |\n",
+ "+------------+--------------+------------------------------------------------------------------------------------------+-------+\n",
+ "only showing top 20 rows\n",
+ "\n"
]
},
{
@@ -216,25 +379,107 @@
}
],
"source": [
- "%%time\n",
- "\n",
- "map_building = SedonaKepler.create_map(df_building, 'Building')\n",
- "map_building"
+ "sedona.read.format(\"geoparquet.metadata\").load(DATA_LINK+\"theme=places/type=place\").drop(\"path\").show(truncate = False)"
]
},
{
"cell_type": "markdown",
- "id": "9d2cd7e5",
+ "id": "83c07bb3-08be-45d0-82ce-e665665af175",
"metadata": {},
"source": [
- "# Place Dataset "
+ "Inspect the schema of GeoParquet files"
]
},
{
"cell_type": "code",
- "execution_count": 6,
+ "execution_count": 9,
+ "id": "6c0a3c6e-db4c-4c59-bfe8-11371a345bdd",
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ " \r"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "root\n",
+ " |-- id: string (nullable = true)\n",
+ " |-- updatetime: string (nullable = true)\n",
+ " |-- version: integer (nullable = true)\n",
+ " |-- names: map (nullable = true)\n",
+ " | |-- key: string\n",
+ " | |-- value: array (valueContainsNull = true)\n",
+ " | | |-- element: map (containsNull = true)\n",
+ " | | | |-- key: string\n",
+ " | | | |-- value: string (valueContainsNull = true)\n",
+ " |-- categories: struct (nullable = true)\n",
+ " | |-- main: string (nullable = true)\n",
+ " | |-- alternate: array (nullable = true)\n",
+ " | | |-- element: string (containsNull = true)\n",
+ " |-- confidence: double (nullable = true)\n",
+ " |-- websites: array (nullable = true)\n",
+ " | |-- element: string (containsNull = true)\n",
+ " |-- socials: array (nullable = true)\n",
+ " | |-- element: string (containsNull = true)\n",
+ " |-- emails: array (nullable = true)\n",
+ " | |-- element: string (containsNull = true)\n",
+ " |-- phones: array (nullable = true)\n",
+ " | |-- element: string (containsNull = true)\n",
+ " |-- brand: struct (nullable = true)\n",
+ " | |-- names: map (nullable = true)\n",
+ " | | |-- key: string\n",
+ " | | |-- value: array (valueContainsNull = true)\n",
+ " | | | |-- element: map (containsNull = true)\n",
+ " | | | | |-- key: string\n",
+ " | | | | |-- value: string (valueContainsNull = true)\n",
+ " | |-- wikidata: string (nullable = true)\n",
+ " |-- addresses: array (nullable = true)\n",
+ " | |-- element: map (containsNull = true)\n",
+ " | | |-- key: string\n",
+ " | | |-- value: string (valueContainsNull = true)\n",
+ " |-- sources: array (nullable = true)\n",
+ " | |-- element: map (containsNull = true)\n",
+ " | | |-- key: string\n",
+ " | | |-- value: string (valueContainsNull = true)\n",
+ " |-- bbox: struct (nullable = true)\n",
+ " | |-- minx: double (nullable = true)\n",
+ " | |-- maxx: double (nullable = true)\n",
+ " | |-- miny: double (nullable = true)\n",
+ " | |-- maxy: double (nullable = true)\n",
+ " |-- geometry: geometry (nullable = true)\n",
+ " |-- geohash: string (nullable = true)\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "sedona.read.format(\"geoparquet\").load(DATA_LINK+\"theme=places/type=place\").printSchema()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cfb1cc80-149c-4523-b674-c0e19155c42d",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "## Run a spatial range query"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
"id": "a6443d1d",
- "metadata": {},
+ "metadata": {
+ "tags": []
+ },
"outputs": [
{
"name": "stderr",
@@ -247,8 +492,8 @@
"name": "stdout",
"output_type": "stream",
"text": [
- "CPU times: user 29.1 ms, sys: 9.95 ms, total: 39.1 ms\n",
- "Wall time: 13.2 s\n"
+ "CPU times: user 30.6 ms, sys: 9.97 ms, total: 40.5 ms\n",
+ "Wall time: 11.2 s\n"
]
}
],
@@ -257,15 +502,32 @@
"\n",
"df_place = sedona.read.format(\"geoparquet\").load(DATA_LINK+\"theme=places/type=place\")\n",
"\n",
- "df_place = df_place.filter(\"ST_Contains(ST_GeomFromWKT('\"+spatial_filter+\"'), geometry) = true\")"
+ "df_place = df_place.filter(\"ST_Contains(ST_GeomFromWKT('\"+spatial_filter+\"'), geometry) = true\").cache()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1f826cb8",
- "metadata": {},
- "outputs": [],
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "User Guide: https://docs.kepler.gl/docs/keplergl-jupyter\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "[Stage 14:===============================================> (115 + 10) / 128]\r"
+ ]
+ }
+ ],
"source": [
"%%time\n",
"\n",
@@ -274,6 +536,168 @@
"map_place"
]
},
+ {
+ "cell_type": "markdown",
+ "id": "7a2c0d96-778c-4170-a2f1-21529ce40696",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "## Write the result back to a small GeoParquet file"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f977d0cf-b74b-42fc-ae97-3ce9f9ece0d4",
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [],
+ "source": [
+ "df_place.select(\"id\", \"geometry\", \"categories.main\").limit(1000).repartition(1) \\\n",
+ " .write.format(\"geoparquet\").option(\"geoparquet.version\", \"1.0.0\").option(\"geoparquet.crs\", \"\").mode('overwrite').save(\"places.parquet\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "7aacbda0-2b55-4c3f-87b6-d2d77a778d17",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "## Convert a part of the result to GeoPandas and save to different file formats"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "2c5e16ee-40d5-4202-83c0-ee36c27dd040",
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [],
+ "source": [
+ "gdf = gpd.GeoDataFrame(df_place.select(\"id\", \"geometry\", \"categories.main\").limit(1000).toPandas(), geometry=\"geometry\")\n",
+ "gdf.to_file('places.geojson', driver='GeoJSON')\n",
+ "gdf.to_file('places.shp')\n",
+ "gdf"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "31e81f8b",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "# Building Dataset "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "d0c55157",
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "24/01/18 07:18:56 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties\n",
+ " \r"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "CPU times: user 13.6 ms, sys: 6.98 ms, total: 20.6 ms\n",
+ "Wall time: 9.2 s\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ " \r"
+ ]
+ }
+ ],
+ "source": [
+ "%%time\n",
+ "\n",
+ "df_building = sedona.read.format(\"geoparquet\").load(DATA_LINK+\"theme=buildings/type=building\")\n",
+ "\n",
+ "df_building = df_building.filter(\"ST_Contains(ST_GeomFromWKT('\"+spatial_filter+\"'), geometry) = true\")\n",
+ "\n",
+ "df_building = df_building.limit(200_000)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "bb62f16e",
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "User Guide: https://docs.kepler.gl/docs/keplergl-jupyter\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ " \r"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "CPU times: user 1.45 s, sys: 248 ms, total: 1.7 s\n",
+ "Wall time: 5min 33s\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/usr/local/lib/python3.10/dist-packages/jupyter_client/session.py:718: UserWarning: Message serialization failed with:\n",
+ "Out of range float values are not JSON compliant\n",
+ "Supporting this message is deprecated in jupyter-client 7, please make sure your message is JSON-compliant\n",
+ " content = self.pack(content)\n"
+ ]
+ },
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "f7bbf17a334e4d479ac4eee21d667672",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "KeplerGl(data={'Building': {'index': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20…"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "%%time\n",
+ "\n",
+ "map_building = SedonaKepler.create_map(df_building, 'Building')\n",
+ "map_building"
+ ]
+ },
{
"cell_type": "markdown",
"id": "c0809afb",
@@ -11050,9 +11474,9 @@
],
"metadata": {
"kernelspec": {
- "display_name": "apache-sedona",
+ "display_name": "Python 3 (ipykernel)",
"language": "python",
- "name": "apache-sedona"
+ "name": "python3"
},
"language_info": {
"codemirror_mode": {
@@ -11064,7 +11488,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.9.6"
+ "version": "3.10.12"
}
},
"nbformat": 4,
diff --git a/docker/sedona-spark-jupyterlab/sedona-jupyterlab.dockerfile b/docker/sedona-spark-jupyterlab/sedona-jupyterlab.dockerfile
index 853edadbc..eae0a2fc2 100644
--- a/docker/sedona-spark-jupyterlab/sedona-jupyterlab.dockerfile
+++ b/docker/sedona-spark-jupyterlab/sedona-jupyterlab.dockerfile
@@ -23,8 +23,9 @@ ARG hadoop_version=3
ARG hadoop_s3_version=3.3.4
ARG aws_sdk_version=1.12.402
ARG spark_xml_version=0.16.0
-ARG sedona_version=1.5.0
-ARG geotools_wrapper_version=1.5.0-28.2
+ARG sedona_version=1.5.1
+ARG geotools_wrapper_version=1.5.1-28.2
+ARG spark_extension_version=2.11.0
# Set up envs
ENV SHARED_WORKSPACE=${shared_workspace}
@@ -44,7 +45,7 @@ COPY ./ ${SEDONA_HOME}/
RUN chmod +x ${SEDONA_HOME}/docker/spark.sh
RUN chmod +x ${SEDONA_HOME}/docker/sedona.sh
RUN ${SEDONA_HOME}/docker/spark.sh ${spark_version} ${hadoop_version} ${hadoop_s3_version} ${aws_sdk_version} ${spark_xml_version}
-RUN ${SEDONA_HOME}/docker/sedona.sh ${sedona_version} ${geotools_wrapper_version} ${spark_version}
+RUN ${SEDONA_HOME}/docker/sedona.sh ${sedona_version} ${geotools_wrapper_version} ${spark_version} ${spark_extension_version}
# Install Python dependencies
COPY docker/sedona-spark-jupyterlab/requirements.txt /opt/requirements.txt
@@ -60,6 +61,7 @@ RUN find /opt/workspace/examples/ -type f -name "*.ipynb" -exec sed -i 's/config
RUN find /opt/workspace/examples/ -type f -name "*.ipynb" -exec sed -i '/spark\.jars\.packages/d' {} +
RUN find /opt/workspace/examples/ -type f -name "*.ipynb" -exec sed -i '/org\.apache\.sedona:sedona-spark-shaded-/d' {} +
RUN find /opt/workspace/examples/ -type f -name "*.ipynb" -exec sed -i '/org\.datasyslab:geotools-wrapper:/d' {} +
+RUN find /opt/workspace/examples/ -type f -name "*.ipynb" -exec sed -i '/uk\.co\.gresearch\.spark:spark-extension_2\.12:/d' {} +
RUN rm -rf ${SEDONA_HOME}
@@ -70,4 +72,12 @@ EXPOSE 4040
WORKDIR ${SHARED_WORKSPACE}
-CMD service ssh start && ${SPARK_HOME}/sbin/start-all.sh && jupyter lab --ip=0.0.0.0 --port=8888 --no-browser --allow-root --NotebookApp.token=
+
+
+CMD DRIVER_MEM=${DRIVER_MEM:-4g} && \
+ EXECUTOR_MEM=${EXECUTOR_MEM:-4g} && \
+ echo "spark.driver.memory $DRIVER_MEM" >> ${SPARK_HOME}/conf/spark-defaults.conf && \
+ echo "spark.executor.memory $EXECUTOR_MEM" >> ${SPARK_HOME}/conf/spark-defaults.conf && \
+ service ssh start && \
+ ${SPARK_HOME}/sbin/start-all.sh && \
+ jupyter lab --ip=0.0.0.0 --port=8888 --no-browser --allow-root --NotebookApp.token=
\ No newline at end of file
diff --git a/docker/sedona.sh b/docker/sedona.sh
index fcc4e849f..2f25ca6cb 100755
--- a/docker/sedona.sh
+++ b/docker/sedona.sh
@@ -21,6 +21,7 @@ set -e
sedona_version=$1
geotools_wrapper_version=$2
spark_version=$3
+spark_extension_version=$4
lower_version=$(echo -e $spark_version"\n3.4" | sort -V | head -n1)
if [ $lower_version = "3.4" ]; then
@@ -38,8 +39,15 @@ else
# Download Sedona
curl https://repo1.maven.org/maven2/org/apache/sedona/sedona-spark-shaded-${sedona_spark_version}_2.12/${sedona_version}/sedona-spark-shaded-${sedona_spark_version}_2.12-${sedona_version}.jar -o $SPARK_HOME/jars/sedona-spark-shaded-${sedona_spark_version}_2.12-${sedona_version}.jar
curl https://repo1.maven.org/maven2/org/datasyslab/geotools-wrapper/${geotools_wrapper_version}/geotools-wrapper-${geotools_wrapper_version}.jar -o $SPARK_HOME/jars/geotools-wrapper-${geotools_wrapper_version}.jar
+
+ # Download gresearch spark extension
+ curl https://repo1.maven.org/maven2/uk/co/gresearch/spark/spark-extension_2.12/${spark_extension_version}-${sedona_spark_version}/spark-extension_2.12-${spark_extension_version}-${sedona_spark_version}.jar -o $SPARK_HOME/jars/spark-extension_2.12-${spark_extension_version}-${sedona_spark_version}.jar
# Install Sedona Python
pip3 install shapely==1.8.4
pip3 install apache-sedona==${sedona_version}
+
+ # Install Spark extension Python
+ pip3 install pyspark-extension==${spark_extension_version}.${sedona_spark_version}
+
fi
diff --git a/docker/spark.sh b/docker/spark.sh
index 652205cb7..d978b4f49 100755
--- a/docker/spark.sh
+++ b/docker/spark.sh
@@ -44,8 +44,6 @@ curl https://repo1.maven.org/maven2/com/databricks/spark-xml_2.12/${spark_xml_ve
# Set up master IP address and executor memory
cp ${SPARK_HOME}/conf/spark-defaults.conf.template ${SPARK_HOME}/conf/spark-defaults.conf
-echo "spark.driver.memory 4g" >> ${SPARK_HOME}/conf/spark-defaults.conf
-echo "spark.executor.memory 4g" >> ${SPARK_HOME}/conf/spark-defaults.conf
# Install required libraries for GeoPandas on Apple chip mac
apt-get install -y gdal-bin libgdal-dev
diff --git a/docs/setup/docker.md b/docs/setup/docker.md
index 0c60f00e4..d510651dd 100644
--- a/docs/setup/docker.md
+++ b/docs/setup/docker.md
@@ -31,21 +31,28 @@ docker pull apache/sedona:{{ sedona.current_version }}
Format:
```bash
-docker run -p 8888:8888 -p 8080:8080 -p 8081:8081 -p 4040:4040 apache/sedona:<sedona_version>
+docker run -e DRIVER_MEM=<driver_mem> -e EXECUTOR_MEM=<executor_mem> -p 8888:8888 -p 8080:8080 -p 8081:8081 -p 4040:4040 apache/sedona:<sedona_version>
```
+Driver memory and executor memory are optional. If their values are not given, the container will take 4GB RAM for the driver and 4GB RAM for the executor.
Example 1:
```bash
-docker run -p 8888:8888 -p 8080:8080 -p 8081:8081 -p 4040:4040 apache/sedona:latest
+docker run -e DRIVER_MEM=6g -e EXECUTOR_MEM=8g -p 8888:8888 -p 8080:8080 -p 8081:8081 -p 4040:4040 apache/sedona:latest
```
+This command will start a container with 6GB RAM for the driver and 8GB RAM for the executor and use the latest Sedona image.
+
+This command will bind the container's ports 8888, 8080, 8081, 4040 to the host's ports 8888, 8080, 8081, 4040 respectively.
+
Example 2:
```bash
docker run -p 8888:8888 -p 8080:8080 -p 8081:8081 -p 4040:4040 apache/sedona:{{ sedona.current_version }}
```
+This command will start a container with 4GB RAM for the driver and 4GB RAM for the executor and use Sedona {{ sedona.current_version }} image.
+
This command will bind the container's ports 8888, 8080, 8081, 4040 to the host's ports 8888, 8080, 8081, 4040 respectively.
### Start coding