You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@sedona.apache.org by ji...@apache.org on 2025/03/10 07:26:34 UTC
(sedona) branch docs-mar-07 created (now 2b78648254)
This is an automated email from the ASF dual-hosted git repository.
jiayu pushed a change to branch docs-mar-07
in repository https://gitbox.apache.org/repos/asf/sedona.git
at 2b78648254 Fix typos
This branch includes the following new commits:
new 465ab2f1c8 Simplify Databricks tutorial
new 697673de99 Clean up maven coordinates
new 3b49d67f4f Update docker
new 88fd4041d4 Restructure mkdocs
new 26e8f81b3f Refine STAC catalog reader
new f2f1132cc0 Fix multiple typos and structure
new 2b78648254 Fix typos
The 7 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(sedona) 06/07: Fix multiple typos and structure
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jiayu pushed a commit to branch docs-mar-07
in repository https://gitbox.apache.org/repos/asf/sedona.git
commit f2f1132cc06abf004ad459bc12b49cce9bb11202
Author: Jia Yu <ji...@apache.org>
AuthorDate: Mon Mar 10 00:04:27 2025 -0700
Fix multiple typos and structure
---
docs/tutorial/files/stac-sedona-spark.md | 32 ++++++++++-------------
docs/tutorial/sql.md | 44 ++++++++++++++++++++++----------
2 files changed, 44 insertions(+), 32 deletions(-)
diff --git a/docs/tutorial/files/stac-sedona-spark.md b/docs/tutorial/files/stac-sedona-spark.md
index 09bcba2328..062e6c5f55 100644
--- a/docs/tutorial/files/stac-sedona-spark.md
+++ b/docs/tutorial/files/stac-sedona-spark.md
@@ -274,10 +274,11 @@ These examples demonstrate how to use the Client class to search for items in a
Opens a connection to the specified STAC API URL.
Parameters:
-* `url` (*str*): The URL of the STAC API to connect to.
- * Example: `"https://planetarycomputer.microsoft.com/api/stac/v1"`
+
+* `url` (*str*): The URL of the STAC API to connect to. Example: `"https://planetarycomputer.microsoft.com/api/stac/v1"`
Returns:
+
* `Client`: An instance of the `Client` class connected to the specified URL.
---
@@ -286,10 +287,11 @@ Returns:
Retrieves a collection client for the specified collection ID.
Parameters:
-* `collection_id` (*str*): The ID of the collection to retrieve.
- * Example: `"aster-l1t"`
+
+* `collection_id` (*str*): The ID of the collection to retrieve. Example: `"aster-l1t"`
Returns:
+
* `CollectionClient`: An instance of the `CollectionClient` class for the specified collection.
---
@@ -299,23 +301,15 @@ Searches for items in the specified collection with optional filters.
Parameters:
-* `ids` (*Union[str, list]*): A variable number of item IDs to filter the items.
- * Example: `"item_id1"` or `["item_id1", "item_id2"]`
-* `collection_id` (*str*): The ID of the collection to search in.
- * Example: `"aster-l1t"`
-* `bbox` (*Optional[list]*): A list of bounding boxes for filtering the items. Each bounding box is represented as a list of four float values: `[min_lon, min_lat, max_lon, max_lat]`.
- * Example: `[[ -180.0, -90.0, 180.0, 90.0 ]]`
-* `datetime` (*Optional[Union[str, datetime.datetime, list]]*): A single datetime, RFC 3339-compliant timestamp, or a list of date-time ranges for filtering the items.
- * Examples:
- * `"2020-01-01T00:00:00Z"`
- * `datetime.datetime(2020, 1, 1)`
- * `[["2020-01-01T00:00:00Z", "2021-01-01T00:00:00Z"]]`
-* `max_items` (*Optional[int]*): The maximum number of items to return from the search, even if there are more matching results.
- * Example: `100`
-* `return_dataframe` (*bool*): If `True` (default), return the result as a Spark DataFrame instead of an iterator of `PyStacItem` objects.
- * Example: `True`
+* `ids` (*Union[str, list]*): A variable number of item IDs to filter the items. Example: `"item_id1"` or `["item_id1", "item_id2"]`
+* `collection_id` (*str*): The ID of the collection to search in. Example: `"aster-l1t"`
+* `bbox` (*Optional[list]*): A list of bounding boxes for filtering the items, represented as `[min_lon, min_lat, max_lon, max_lat]`. Example: `[[ -180.0, -90.0, 180.0, 90.0 ]]`
+* `datetime` (*Optional[Union[str, datetime.datetime, list]]*): A single datetime, RFC 3339-compliant timestamp, or a list of date-time ranges. Example: `"2020-01-01T00:00:00Z"`, `datetime.datetime(2020, 1, 1)`, `[["2020-01-01T00:00:00Z", "2021-01-01T00:00:00Z"]]`
+* `max_items` (*Optional[int]*): The maximum number of items to return. Example: `100`
+* `return_dataframe` (*bool*): If `True` (default), return the result as a Spark DataFrame instead of an iterator of `PyStacItem` objects. Example: `True`
Returns:
+
* *Union[Iterator[PyStacItem], DataFrame]*: An iterator of `PyStacItem` objects or a Spark DataFrame that matches the specified filters.
## References
diff --git a/docs/tutorial/sql.md b/docs/tutorial/sql.md
index fecbf3ef82..2564e39c25 100644
--- a/docs/tutorial/sql.md
+++ b/docs/tutorial/sql.md
@@ -139,7 +139,7 @@ Add the following line after creating Sedona config. If you already have a Spark
You can also register everything by passing `--conf spark.sql.extensions=org.apache.sedona.sql.SedonaSqlExtensions` to `spark-submit` or `spark-shell`.
-## Load data from files
+## Load data from text files
Assume we have a WKT file, namely `usa-county.tsv`, at Path `/Download/usa-county.tsv` as follows:
@@ -152,6 +152,8 @@ POLYGON (..., ...) Lancaster County
The file may have many other columns.
+### Load the raw DataFrame
+
Use the following code to load the data and create a raw DataFrame:
=== "Scala"
@@ -186,7 +188,7 @@ The output will be like this:
|POLYGON ((-96.910...| 31|109|00835876|31109| Lancaster| Lancaster County| 06| H1|G4020| 339|30700|null| A|2169240202|22877180|+40.7835474|-096.6886584|
```
-## Create a Geometry type column
+### Create a Geometry type column
All geometrical operations in SedonaSQL are on Geometry type objects. Therefore, before any kind of queries, you need to create a Geometry type column on a DataFrame.
@@ -347,6 +349,30 @@ Please refer to [Reading Legacy Parquet Files](../api/sql/Reading-legacy-parquet
See [this page](files/geoparquet-sedona-spark.md) for more information on loading GeoParquet.
+## Load data from STAC catalog
+
+Sedona STAC data source allows you to read data from a SpatioTemporal Asset Catalog (STAC) API. The data source supports reading STAC items and collections.
+
+You can load a STAC collection from a s3 collection file object:
+
+```python
+df = sedona.read.format("stac").load(
+ "s3a://example.com/stac_bucket/stac_collection.json"
+)
+```
+
+You can also load a STAC collection from an HTTP/HTTPS endpoint:
+
+```python
+df = sedona.read.format("stac").load(
+ "https://earth-search.aws.element84.com/v1/collections/sentinel-2-pre-c1-l2a"
+)
+```
+
+The STAC data source supports predicate pushdown for spatial and temporal filters. The data source can push down spatial and temporal filters to the underlying data source to reduce the amount of data that needs to be read.
+
+See [this page](files/stac-sedona-spark.md) for more information on loading data from STAC.
+
## Load data from JDBC data sources
The 'query' option in Spark SQL's JDBC data source can be used to convert geometry columns to a format that Sedona can interpret.
@@ -407,7 +433,7 @@ For Postgis there is no need to add a query to convert geometry types since it's
.withColumn("geom", f.expr("ST_GeomFromWKB(geom)")))
```
-## Load from GeoPackage
+## Load GeoPackage
Since v1.7.0, Sedona supports loading Geopackage file format as a DataFrame.
@@ -431,7 +457,7 @@ Since v1.7.0, Sedona supports loading Geopackage file format as a DataFrame.
See [this page](files/geopackage-sedona-spark.md) for more information on loading GeoPackage.
-## Load from OSM PBF
+## Load OSM PBF
Since v1.7.1, Sedona supports loading OSM PBF file format as a DataFrame.
@@ -527,14 +553,6 @@ and for relation
+-----+--------+--------+--------------------+--------------------+--------------------+--------------------+
```
-Known limitations (v1.7.0):
-
-- webp rasters are not supported
-- ewkb geometries are not supported
-- filtering based on geometries envelopes are not supported
-
-All points above should be resolved soon, stay tuned !
-
## Transform the Coordinate Reference System
Sedona doesn't control the coordinate unit (degree-based or meter-based) of all geometries in a Geometry column. The unit of all related distances in SedonaSQL is same as the unit of all geometries in a Geometry column.
@@ -1188,7 +1206,7 @@ SELECT ST_AsText(countyshape)
FROM polygondf
```
-## Save as GeoJSON
+## Save GeoJSON
Since `v1.6.1`, the GeoJSON data source in Sedona can be used to save a Spatial DataFrame to a single-line JSON file, with geometries written in GeoJSON format.
(sedona) 07/07: Fix typos
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jiayu pushed a commit to branch docs-mar-07
in repository https://gitbox.apache.org/repos/asf/sedona.git
commit 2b78648254c05925c6ac57a5a8f6aebe099131fa
Author: Jia Yu <ji...@apache.org>
AuthorDate: Mon Mar 10 00:26:24 2025 -0700
Fix typos
---
docs/api/sql/Function.md | 2 +-
docs/api/sql/Raster-map-algebra.md | 4 +-
docs/api/sql/Spider.md | 2 +-
docs/setup/release-notes.md | 2 +-
docs/tutorial/files/geoparquet-sedona-spark.md | 10 +-
docs/tutorial/files/shapefiles-sedona-spark.md | 8 +-
docs/tutorial/python-vector-osm.md | 159 -------------------------
docs/tutorial/raster.md | 83 +------------
docs/tutorial/rdd.md | 4 +-
docs/tutorial/snowflake/sql.md | 2 +-
docs/tutorial/sql.md | 4 +-
11 files changed, 20 insertions(+), 260 deletions(-)
diff --git a/docs/api/sql/Function.md b/docs/api/sql/Function.md
index 51f63f3a13..2dc28f21ce 100644
--- a/docs/api/sql/Function.md
+++ b/docs/api/sql/Function.md
@@ -460,7 +460,7 @@ POINT ZM(1 1 1 1)
## ST_AsGeoJSON
!!!note
- This method is not recommended. Please use [Sedona GeoJSON data source](../../tutorial/sql.md#save-as-geojson) to write GeoJSON files.
+ This method is not recommended. Please use [Sedona GeoJSON data source](../../tutorial/sql.md#save-geojson) to write GeoJSON files.
Introduction: Return the [GeoJSON](https://geojson.org/) string representation of a geometry
diff --git a/docs/api/sql/Raster-map-algebra.md b/docs/api/sql/Raster-map-algebra.md
index 5b22280ee6..70b7e52eef 100644
--- a/docs/api/sql/Raster-map-algebra.md
+++ b/docs/api/sql/Raster-map-algebra.md
@@ -34,7 +34,7 @@ RS_MapAlgebra(rast: Raster, pixelType: String, script: String, [noDataValue: Dou
* `rast`: The raster to apply the map algebra expression to.
* `pixelType`: The data type of the output raster. This can be one of `D` (double), `F` (float), `I` (integer), `S` (short), `US` (unsigned short) or `B` (byte). If specified `NULL`, the output raster will have the same data type as the input raster.
-* `script`: The map algebra script. [Refer here for more details on the format.](#:~:text=The Jiffle script is,current output pixel value)
+* `script`: The map algebra script. [Refer here for more details on the format.](https://github.com/geosolutions-it/jai-ext/wiki/Jiffle)
* `noDataValue`: (Optional) The nodata value of the output raster.
As of version `v1.5.1`, the `RS_MapAlgebra` function allows two raster column inputs, with multi-band rasters supported. The function accepts 5 parameters:
@@ -46,7 +46,7 @@ RS_MapAlgebra(rast0: Raster, rast1: Raster, pixelType: String, script: String, n
* `rast0`: The first raster to apply the map algebra expression to.
* `rast1`: The second raster to apply the map algebra expression to.
* `pixelType`: The data type of the output raster. This can be one of `D` (double), `F` (float), `I` (integer), `S` (short), `US` (unsigned short) or `B` (byte). If specified `NULL`, the output raster will have the same data type as the input raster.
-* `script`: The map algebra script. [Refer here for more details on the format.](#:~:text=The Jiffle script is,current output pixel value)
+* `script`: The map algebra script. [Refer here for more details on the format.](https://github.com/geosolutions-it/jai-ext/wiki/Jiffle)
* `noDataValue`: (Not optional) The nodata value of the output raster, `null` is allowed.
Spark SQL Example for two raster input `RS_MapAlgebra`:
diff --git a/docs/api/sql/Spider.md b/docs/api/sql/Spider.md
index 5dfa6569fe..207259d10e 100644
--- a/docs/api/sql/Spider.md
+++ b/docs/api/sql/Spider.md
@@ -21,7 +21,7 @@ Sedona offers a spatial data generator called Spider. It is a data source that g
## Quick Start
-Once you have your [`SedonaContext` object created](../Overview#quick-start), you can create a DataFrame with the `spider` data source.
+Once you have your [`SedonaContext` object created](Overview.md#quick-start), you can create a DataFrame with the `spider` data source.
```python
df_random_points = sedona.read.format("spider").load(n=1000, distribution="uniform")
diff --git a/docs/setup/release-notes.md b/docs/setup/release-notes.md
index 7a5666846a..36c620beac 100644
--- a/docs/setup/release-notes.md
+++ b/docs/setup/release-notes.md
@@ -1164,7 +1164,7 @@ Sedona 1.4.0 is compiled against, Spark 3.3 / Flink 1.12, Java 8.
* [X] **Sedona Spark & Flink** Serialize and deserialize geometries 3 - 7X faster
* [X] **Sedona Spark & Flink** Google S2 based spatial join for fast approximate point-in-polygon join. See [Join query in Spark](../api/sql/Optimizer.md#google-s2-based-approximate-equi-join) and [Join query in Flink](../tutorial/flink/sql.md#join-query)
-* [X] **Sedona Spark** Pushdown spatial predicate on GeoParquet to reduce memory consumption by 10X: see [explanation](../api/sql/Optimizer.md#Push-spatial-predicates-to-GeoParquet)
+* [X] **Sedona Spark** Pushdown spatial predicate on GeoParquet to reduce memory consumption by 10X: see [explanation](../api/sql/Optimizer.md#push-spatial-predicates-to-geoparquet)
* [X] **Sedona Spark** Automatically use broadcast index spatial join for small datasets
* [X] **Sedona Spark** New RasterUDT added to Sedona GeoTiff reader.
* [X] **Sedona Spark** A number of bug fixes and improvement to the Sedona R module.
diff --git a/docs/tutorial/files/geoparquet-sedona-spark.md b/docs/tutorial/files/geoparquet-sedona-spark.md
index 28da219474..11d95d9c6d 100644
--- a/docs/tutorial/files/geoparquet-sedona-spark.md
+++ b/docs/tutorial/files/geoparquet-sedona-spark.md
@@ -76,7 +76,7 @@ df.show(truncate=False)
Here are the results:
```
-+---+---------------------+
++---+---------------------+
|id |geometry |
+---+---------------------+
|a |LINESTRING (2 5, 6 1)|
@@ -199,10 +199,10 @@ The value of `geoparquet.crs` and `geoparquet.crs.<column_name>` can be one of t
* `""` (empty string): Omit the `crs` field. This implies that the CRS is [OGC:CRS84](https://www.opengis.net/def/crs/OGC/1.3/CRS84) for CRS-aware implementations.
* `"{...}"` (PROJJSON string): The `crs` field will be set as the PROJJSON object representing the Coordinate Reference System (CRS) of the geometry. You can find the PROJJSON string of a specific CRS from here: https://epsg.io/ (click the JSON option at the bottom of the page). You can also customize your PROJJSON string as needed.
-Please note that Sedona currently cannot set/get a projjson string to/from a CRS. Its geoparquet reader will ignore the projjson metadata and you will have to set your CRS via [`ST_SetSRID`](../api/sql/Function.md#st_setsrid) after reading the file.
+Please note that Sedona currently cannot set/get a projjson string to/from a CRS. Its geoparquet reader will ignore the projjson metadata and you will have to set your CRS via [`ST_SetSRID`](../../api/sql/Function.md#st_setsrid) after reading the file.
Its geoparquet writer will not leverage the SRID field of a geometry so you will have to always set the `geoparquet.crs` option manually when writing the file, if you want to write a meaningful CRS field.
-Due to the same reason, Sedona geoparquet reader and writer do NOT check the axis order (lon/lat or lat/lon) and assume they are handled by the users themselves when writing / reading the files. You can always use [`ST_FlipCoordinates`](../api/sql/Function.md#st_flipcoordinates) to swap the axis order of your geometries.
+Due to the same reason, Sedona geoparquet reader and writer do NOT check the axis order (lon/lat or lat/lon) and assume they are handled by the users themselves when writing / reading the files. You can always use [`ST_FlipCoordinates`](../../api/sql/Function.md#st_flipcoordinates) to swap the axis order of your geometries.
## Save GeoParquet with Covering Metadata
@@ -231,7 +231,7 @@ df_bbox.write.format("geoparquet").option("geoparquet.covering.geometry", "bbox"
## Sort then Save GeoParquet
-To maximize the performance of Sedona GeoParquet filter pushdown, we suggest that you sort the data by their geohash values (see [ST_GeoHash](../api/sql/Function.md#st_geohash)) and then save as a GeoParquet file. An example is as follows:
+To maximize the performance of Sedona GeoParquet filter pushdown, we suggest that you sort the data by their geohash values (see [ST_GeoHash](../../api/sql/Function.md#st_geohash)) and then save as a GeoParquet file. An example is as follows:
```
SELECT col1, col2, geom, ST_GeoHash(geom, 5) as geohash
@@ -253,7 +253,7 @@ Let’s look at an example of a dataset with points and three bounding boxes.
Now, let’s apply a spatial filter to read points within a particular area:
-
+
Here is the query:
diff --git a/docs/tutorial/files/shapefiles-sedona-spark.md b/docs/tutorial/files/shapefiles-sedona-spark.md
index 3b24349b68..a7df23c521 100644
--- a/docs/tutorial/files/shapefiles-sedona-spark.md
+++ b/docs/tutorial/files/shapefiles-sedona-spark.md
@@ -196,11 +196,11 @@ Due to these limitations, other options are worth investigating.
There are a variety of other file formats that are good for geometric data:
* Iceberg
-* [GeoParquet](../geoparquet-sedona-spark)
+* [GeoParquet](geoparquet-sedona-spark.md)
* FlatGeoBuf
-* [GeoPackage](../geopackage-sedona-spark)
-* [GeoJSON](../geojson-sedona-spark)
-* [CSV](../csv-geometry-sedona-spark)
+* [GeoPackage](geopackage-sedona-spark.md)
+* [GeoJSON](geojson-sedona-spark.md)
+* [CSV](csv-geometry-sedona-spark.md)
* GeoTIFF
## Why Sedona does not support Shapefile writes
diff --git a/docs/tutorial/python-vector-osm.md b/docs/tutorial/python-vector-osm.md
deleted file mode 100644
index 00f19f4322..0000000000
--- a/docs/tutorial/python-vector-osm.md
+++ /dev/null
@@ -1,159 +0,0 @@
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
- -->
-
-# Example of spark + sedona + hdfs with slave nodes and OSM vector data consults
-
-```
-from IPython.display import display, HTML
-from pyspark.sql import SparkSession
-from pyspark import StorageLevel
-import pandas as pd
-from pyspark.sql.types import StructType, StructField,StringType, LongType, IntegerType, DoubleType, ArrayType
-from pyspark.sql.functions import regexp_replace
-from sedona.register import SedonaRegistrator
-from sedona.utils import SedonaKryoRegistrator, KryoSerializer
-from pyspark.sql.functions import col, split, expr
-from pyspark.sql.functions import udf, lit
-from sedona.utils import SedonaKryoRegistrator, KryoSerializer
-from pyspark.sql.functions import col, split, expr
-from pyspark.sql.functions import udf, lit, flatten
-from pywebhdfs.webhdfs import PyWebHdfsClient
-from datetime import date
-from pyspark.sql.functions import monotonically_increasing_id
-import json
-```
-
-## Registering spark session, adding node executor configurations and sedona registrator
-
-```
-spark = SparkSession.\
- builder.\
- appName("Overpass-API").\
- enableHiveSupport().\
- master("local[*]").\
- master("spark://spark-master:7077").\
- config("spark.executor.memory", "15G").\
- config("spark.driver.maxResultSize", "135G").\
- config("spark.sql.shuffle.partitions", "500").\
- config(' spark.sql.adaptive.coalescePartitions.enabled', True).\
- config('spark.sql.adaptive.enabled', True).\
- config('spark.sql.adaptive.coalescePartitions.initialPartitionNum', 125).\
- config("spark.sql.execution.arrow.pyspark.enabled", True).\
- config("spark.sql.execution.arrow.fallback.enabled", True).\
- config('spark.kryoserializer.buffer.max', 2047).\
- config("spark.serializer", KryoSerializer.getName).\
- config("spark.kryo.registrator", SedonaKryoRegistrator.getName).\
- config("spark.jars.packages", "org.apache.sedona:sedona-spark-shaded-3.0_2.12:1.4.0,org.datasyslab:geotools-wrapper:1.4.0-28.2") .\
- enableHiveSupport().\
- getOrCreate()
-
-SedonaRegistrator.registerAll(spark)
-sc = spark.sparkContext
-```
-
-## Connecting to Overpass API to search and downloading data for saving into HDFS
-
-```
-import requests
-import json
-
-overpass_url = "http://overpass-api.de/api/interpreter"
-overpass_query = """
-[out:json];
-area[name = "Foz do Iguaçu"];
-way(area)["highway"~""];
-out geom;
->;
-out skel qt;
-"""
-
-response = requests.get(overpass_url,
- params={'data': overpass_query})
-data = response.json()
-hdfs = PyWebHdfsClient(host='179.106.229.159',port='50070', user_name='root')
-file_name = "foz_roads_osm.json"
-hdfs.delete_file_dir(file_name)
-hdfs.create_file(file_name, json.dumps(data))
-
-```
-
-## Connecting spark sedona with saved hdfs file
-
-```
-path = "hdfs://776faf4d6a1e:8020/"+file_name
-df = spark.read.json(path, multiLine = "true")
-```
-
-## Consulting and organizing data for analysis
-
-```
-from pyspark.sql.functions import explode, arrays_zip
-
-df.createOrReplaceTempView("df")
-tb = spark.sql("select *, size(elements) total_nodes from df")
-tb.show(5)
-
-isolate_total_nodes = tb.select("total_nodes").toPandas()
-total_nodes = isolate_total_nodes["total_nodes"].iloc[0]
-print(total_nodes)
-
-isolate_ids = tb.select("elements.id").toPandas()
-ids = pd.DataFrame(isolate_ids["id"].iloc[0]).drop_duplicates()
-print(ids[0].iloc[1])
-
-formatted_df = tb\
-.withColumn("id", explode("elements.id"))
-
-formatted_df.show(5)
-
-formatted_df = tb\
-.withColumn("new", arrays_zip("elements.id", "elements.geometry", "elements.nodes", "elements.tags"))\
-.withColumn("new", explode("new"))
-
-formatted_df.show(5)
-
-# formatted_df.printSchema()
-
-formatted_df = formatted_df.select("new.0","new.1","new.2","new.3.maxspeed","new.3.incline","new.3.surface", "new.3.name", "total_nodes")
-formatted_df = formatted_df.withColumnRenamed("0","id").withColumnRenamed("1","geom").withColumnRenamed("2","nodes").withColumnRenamed("3","tags")
-formatted_df.createOrReplaceTempView("formatted_df")
-formatted_df.show(5)
-# TODO atualizar daqui para baixo para considerar a linha inteira na lógica
-points_tb = spark.sql("select geom, id from formatted_df where geom IS NOT NULL")
-points_tb = points_tb\
-.withColumn("new", arrays_zip("geom.lat", "geom.lon"))\
-.withColumn("new", explode("new"))
-
-points_tb = points_tb.select("new.0","new.1", "id")
-
-points_tb = points_tb.withColumnRenamed("0","lat").withColumnRenamed("1","lon")
-points_tb.printSchema()
-
-points_tb.createOrReplaceTempView("points_tb")
-
-points_tb.show(5)
-
-coordinates_tb = spark.sql("select (select collect_list(CONCAT(p1.lat,',',p1.lon)) from points_tb p1 where p1.id = p2.id group by p1.id) as coordinates, p2.id, p2.maxspeed, p2.incline, p2.surface, p2.name, p2.nodes, p2.total_nodes from formatted_df p2")
-coordinates_tb.createOrReplaceTempView("coordinates_tb")
-coordinates_tb.show(5)
-
-roads_tb = spark.sql("SELECT ST_LineStringFromText(REPLACE(REPLACE(CAST(coordinates as string),'[',''),']',''), ',') as geom, id, maxspeed, incline, surface, name, nodes, total_nodes FROM coordinates_tb WHERE coordinates IS NOT NULL")
-roads_tb.createOrReplaceTempView("roads_tb")
-roads_tb.show(5)
-```
diff --git a/docs/tutorial/raster.md b/docs/tutorial/raster.md
index 641b441428..65541ced6c 100644
--- a/docs/tutorial/raster.md
+++ b/docs/tutorial/raster.md
@@ -21,7 +21,7 @@
Sedona uses 1-based indexing for all raster functions except [map algebra function](../api/sql/Raster-map-algebra.md), which uses 0-based indexing.
!!!note
- Since v`1.5.0`, Sedona assumes geographic coordinates to be in longitude/latitude order. If your data is lat/lon order, please use `ST_FlipCoordinates` to swap X and Y.
+ Sedona assumes geographic coordinates to be in longitude/latitude order. If your data is lat/lon order, please use `ST_FlipCoordinates` to swap X and Y.
Starting from `v1.1.0`, Sedona SQL supports raster data sources and raster operators in DataFrame and SQL. Raster support is available in all Sedona language bindings including ==Scala, Java, Python, and R==.
@@ -67,8 +67,6 @@ Detailed SedonaSQL APIs are available here: [SedonaSQL API](../api/sql/Overview.
Use the following code to create your Sedona config at the beginning. If you already have a SparkSession (usually named `spark`) created by Wherobots/AWS EMR/Databricks, please skip this step and use `spark` directly.
-==Sedona >= 1.4.1==
-
You can add additional Spark runtime config to the config builder. For example, `SedonaContext.builder().config("spark.sql.autoBroadcastJoinThreshold", "10485760")`
=== "Scala"
@@ -114,65 +112,10 @@ You can add additional Spark runtime config to the config builder. For example,
```
Please replace the `3.3` in the package name of sedona-spark-shaded with the corresponding major.minor version of Spark, such as `sedona-spark-shaded-3.4_2.12:{{ sedona.current_version }}`.
-==Sedona < 1.4.1==
-
-The following method has been deprecated since Sedona 1.4.1. Please use the method above to create your Sedona config.
-
-=== "Scala"
-
- ```scala
- var sparkSession = SparkSession.builder()
- .master("local[*]") // Delete this if run in cluster mode
- .appName("readTestScala") // Change this to a proper name
- // Enable Sedona custom Kryo serializer
- .config("spark.serializer", classOf[KryoSerializer].getName) // org.apache.spark.serializer.KryoSerializer
- .config("spark.kryo.registrator", classOf[SedonaKryoRegistrator].getName)
- .getOrCreate() // org.apache.sedona.core.serde.SedonaKryoRegistrator
- ```
- If you use SedonaViz together with SedonaSQL, please use the following two lines to enable Sedona Kryo serializer instead:
- ```scala
- .config("spark.serializer", classOf[KryoSerializer].getName) // org.apache.spark.serializer.KryoSerializer
- .config("spark.kryo.registrator", classOf[SedonaVizKryoRegistrator].getName) // org.apache.sedona.viz.core.Serde.SedonaVizKryoRegistrator
- ```
-
-=== "Java"
-
- ```java
- SparkSession sparkSession = SparkSession.builder()
- .master("local[*]") // Delete this if run in cluster mode
- .appName("readTestScala") // Change this to a proper name
- // Enable Sedona custom Kryo serializer
- .config("spark.serializer", KryoSerializer.class.getName) // org.apache.spark.serializer.KryoSerializer
- .config("spark.kryo.registrator", SedonaKryoRegistrator.class.getName)
- .getOrCreate() // org.apache.sedona.core.serde.SedonaKryoRegistrator
- ```
- If you use SedonaViz together with SedonaSQL, please use the following two lines to enable Sedona Kryo serializer instead:
- ```scala
- .config("spark.serializer", KryoSerializer.class.getName) // org.apache.spark.serializer.KryoSerializer
- .config("spark.kryo.registrator", SedonaVizKryoRegistrator.class.getName) // org.apache.sedona.viz.core.Serde.SedonaVizKryoRegistrator
- ```
-
-=== "Python"
-
- ```python
- sparkSession = SparkSession. \
- builder. \
- appName('appName'). \
- config("spark.serializer", KryoSerializer.getName). \
- config("spark.kryo.registrator", SedonaKryoRegistrator.getName). \
- config('spark.jars.packages',
- 'org.apache.sedona:sedona-spark-shaded-3.3_2.12:{{ sedona.current_version }},'
- 'org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}'). \
- getOrCreate()
- ```
- Please replace the `3.3` in the package name of sedona-spark-shaded with the corresponding major.minor version of Spark, such as `sedona-spark-shaded-3.4_2.12:{{ sedona.current_version }}`.
-
## Initiate SedonaContext
Add the following line after creating the Sedona config. If you already have a SparkSession (usually named `spark`) created by Wherobots/AWS EMR/Databricks, please call `SedonaContext.create(spark)` instead.
-==Sedona >= 1.4.1==
-
=== "Scala"
```scala
@@ -197,30 +140,6 @@ Add the following line after creating the Sedona config. If you already have a S
sedona = SedonaContext.create(config)
```
-==Sedona < 1.4.1==
-
-The following method has been deprecated since Sedona 1.4.1. Please use the method above to create your SedonaContext.
-
-=== "Scala"
-
- ```scala
- SedonaSQLRegistrator.registerAll(sparkSession)
- ```
-
-=== "Java"
-
- ```java
- SedonaSQLRegistrator.registerAll(sparkSession)
- ```
-
-=== "Python"
-
- ```python
- from sedona.register import SedonaRegistrator
-
- SedonaRegistrator.registerAll(spark)
- ```
-
You can also register everything by passing `--conf spark.sql.extensions=org.apache.sedona.sql.SedonaSqlExtensions` to `spark-submit` or `spark-shell`.
## Load data from files
diff --git a/docs/tutorial/rdd.md b/docs/tutorial/rdd.md
index fd5a1fcfde..d3b27896f7 100644
--- a/docs/tutorial/rdd.md
+++ b/docs/tutorial/rdd.md
@@ -765,9 +765,9 @@ Distance join can only accept `COVERED_BY` and `INTERSECTS` as spatial predicate
The details of spatial partitioning in join query is [here](#use-spatial-partitioning).
-The details of using spatial indexes in join query is [here](#use-spatial-indexes-2).
+The details of using spatial indexes in join query is [here](#use-spatial-indexes_2).
-The output format of the distance join query is [here](#output-format-2).
+The output format of the distance join query is [here](#output-format_2).
!!!note
Distance join query is equal to the following query in Spatial SQL:
diff --git a/docs/tutorial/snowflake/sql.md b/docs/tutorial/snowflake/sql.md
index 02ef7c7e01..ba42f23138 100644
--- a/docs/tutorial/snowflake/sql.md
+++ b/docs/tutorial/snowflake/sql.md
@@ -302,7 +302,7 @@ Please use the following steps:
### 1. Generate S2 ids for both tables
-Use [ST_S2CellIds](../../api/snowflake/vector-data/Function.md#ST_S2CellIDs) to generate cell IDs. Each geometry may produce one or more IDs.
+Use [ST_S2CellIds](../../api/snowflake/vector-data/Function.md#st_s2cellids) to generate cell IDs. Each geometry may produce one or more IDs.
```sql
SELECT * FROM lefts, TABLE(FLATTEN(ST_S2CellIDs(lefts.geom, 15))) s1
diff --git a/docs/tutorial/sql.md b/docs/tutorial/sql.md
index 2564e39c25..bd5327f675 100644
--- a/docs/tutorial/sql.md
+++ b/docs/tutorial/sql.md
@@ -299,7 +299,7 @@ Since v`1.7.0`, Sedona supports loading Shapefile as a DataFrame.
The input path can be a directory containing one or multiple shapefiles, or path to a `.shp` file.
-See [this page](files/shapefile-sedona-spark.md) for more information on loading Shapefiles.
+See [this page](files/shapefiles-sedona-spark.md) for more information on loading Shapefiles.
## Load GeoParquet
@@ -641,7 +641,7 @@ The output will look like this:
+----------------+---+------+-------+
```
-See [this page](../concepts/clustering-algorithms) for more information on the DBSCAN algorithm.
+See [this page](concepts/clustering-algorithms.md) for more information on the DBSCAN algorithm.
## Calculate the Local Outlier Factor (LOF)
(sedona) 01/07: Simplify Databricks tutorial
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jiayu pushed a commit to branch docs-mar-07
in repository https://gitbox.apache.org/repos/asf/sedona.git
commit 465ab2f1c85f88aef62b77bb36e1eda8d6c7b3b9
Author: Jia Yu <ji...@apache.org>
AuthorDate: Sun Mar 9 21:05:57 2025 -0700
Simplify Databricks tutorial
---
docs/setup/databricks.md | 57 ++----------------------------------------------
1 file changed, 2 insertions(+), 55 deletions(-)
diff --git a/docs/setup/databricks.md b/docs/setup/databricks.md
index 0a9e7cda9e..d0a8d332d9 100644
--- a/docs/setup/databricks.md
+++ b/docs/setup/databricks.md
@@ -17,60 +17,10 @@
under the License.
-->
-Please pay attention to the Spark version postfix and Scala version postfix on our [Maven Coordinate page](maven-coordinates.md). Databricks Spark and Apache Spark's compatibility can be found [here](https://docs.databricks.com/en/release-notes/runtime/index.html).
-
-## Community edition (free-tier)
-
-You just need to install the Sedona jars and Sedona Python on Databricks using Databricks default web UI. Then everything will work.
-
-### Install libraries
-
-1) From the Libraries tab install from Maven Coordinates
-
-```
-org.apache.sedona:sedona-spark-shaded-3.4_2.12:{{ sedona.current_version }}
-org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}
-```
-
-2) For enabling python support, from the Libraries tab install from PyPI
-
-```
-apache-sedona=={{ sedona.current_version }}
-geopandas==1.0.1
-keplergl==0.3.7
-pydeck==0.9.1
-```
-
-### Initialize
-
-After you have installed the libraries and started the cluster, you can initialize the Sedona `ST_*` functions and types by running from your code:
-
-(scala)
-
-```scala
-import org.apache.sedona.sql.utils.SedonaSQLRegistrator
-SedonaSQLRegistrator.registerAll(spark)
-```
-
-(or python)
-
-```python
-from sedona.register.geo_registrator import SedonaRegistrator
-
-SedonaRegistrator.registerAll(spark)
-```
-
-## Advanced editions
-
-In Databricks advanced editions, you need to install Sedona via [cluster init-scripts](https://docs.databricks.com/clusters/init-scripts.html) as described below. We recommend Databricks 10.x+. Sedona is not guaranteed to be 100% compatible with `Databricks photon acceleration`. Sedona requires Spark internal APIs to inject many optimization strategies, which sometimes is not accessible in `Photon`.
-
-In Spark 3.2, `org.apache.spark.sql.catalyst.expressions.Generator` class added a field `nodePatterns`. Any SQL functions that rely on Generator class may have issues if compiled for a runtime with a differing spark version. For Sedona, those functions are:
-
-* ST_MakeValid
-* ST_SubDivideExplode
+In Databricks advanced editions, you need to install Sedona via [cluster init-scripts](https://docs.databricks.com/clusters/init-scripts.html) as described below. Sedona is not guaranteed to be 100% compatible with `Databricks photon acceleration`. Sedona requires Spark internal APIs to inject many optimization strategies, which sometimes is not accessible in `Photon`.
!!!note
- The following steps use DBR including Apache Spark 3.4.x as an example. Please change the Spark version according to your DBR version.
+ The following steps use DBR including Apache Spark 3.4.x as an example. Please change the Spark version according to your DBR version. Please pay attention to the Spark version postfix and Scala version postfix on our [Maven Coordinate page](maven-coordinates.md). Databricks Spark and Apache Spark's compatibility can be found [here](https://docs.databricks.com/en/release-notes/runtime/index.html).
### Download Sedona jars
@@ -91,9 +41,6 @@ Of course, you can also do the steps above manually.
### Create an init script
-!!!warning
- Starting from December 2023, Databricks has disabled all DBFS based init script (/dbfs/XXX/<script-name>.sh). So you will have to store the init script from a workspace level (`/Workspace/Users/<user-name>/<script-name>.sh`) or Unity Catalog volume (`/Volumes/<catalog>/<schema>/<volume>/<path-to-script>/<script-name>.sh`). Please see [Databricks init scripts](https://docs.databricks.com/en/init-scripts/cluster-scoped.html#configure-a-cluster-scoped-init-script-using-the-ui) for more [...]
-
!!!note
If you are creating a Shared cluster, you won't be able to use init scripts and jars stored under `Workspace`. Please instead store them in `Volumes`. The overall process should be the same.
(sedona) 04/07: Restructure mkdocs
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jiayu pushed a commit to branch docs-mar-07
in repository https://gitbox.apache.org/repos/asf/sedona.git
commit 88fd4041d4d73c09baef816b412a88ba671f98b5
Author: Jia Yu <ji...@apache.org>
AuthorDate: Sun Mar 9 21:26:18 2025 -0700
Restructure mkdocs
---
mkdocs.yml | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/mkdocs.yml b/mkdocs.yml
index eb478fc515..2db776b633 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -66,6 +66,9 @@ nav:
- GeoParquet: tutorial/files/geoparquet-sedona-spark.md
- GeoJSON: tutorial/files/geojson-sedona-spark.md
- Shapefiles: tutorial/files/shapefiles-sedona-spark.md
+ - Concepts:
+ - Spatial Joins: tutorial/concepts/spatial-joins.md
+ - Clustering Algorithms: tutorial/concepts/clustering-algorithms.md
- Map visualization SQL app:
- Scala/Java: tutorial/viz.md
- Use Apache Zeppelin: tutorial/zeppelin.md
@@ -81,9 +84,6 @@ nav:
- Examples:
- Scala/Java: tutorial/demo.md
- Python: tutorial/jupyter-notebook.md
- - Concepts:
- - Spatial Joins: tutorial/concepts/spatial-joins.md
- - Clustering Algorithms: tutorial/concepts/clustering-algorithms.md
- API Docs:
- Sedona with Apache Spark:
- SQL:
(sedona) 02/07: Clean up maven coordinates
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jiayu pushed a commit to branch docs-mar-07
in repository https://gitbox.apache.org/repos/asf/sedona.git
commit 697673de99dd5f8e6fd0f573e27046aca592e699
Author: Jia Yu <ji...@apache.org>
AuthorDate: Sun Mar 9 21:22:08 2025 -0700
Clean up maven coordinates
---
docs/setup/maven-coordinates.md | 46 -----------------------------------------
1 file changed, 46 deletions(-)
diff --git a/docs/setup/maven-coordinates.md b/docs/setup/maven-coordinates.md
index 90d5a2c419..e54870be7b 100644
--- a/docs/setup/maven-coordinates.md
+++ b/docs/setup/maven-coordinates.md
@@ -169,52 +169,6 @@ The optional GeoTools library is required if you want to use CRS transformation,
</dependency>
```
-### netCDF-Java 5.4.2
-
-This is required only if you want to read HDF/NetCDF files using `RS_FromNetCDF`. Note that this JAR is not in Maven Central so you will need to add this repository to your pom.xml or build.sbt, or specify the URL in Spark Config `spark.jars.repositories` or spark-submit `--repositories` option.
-
-!!!warning
- This jar was a required dependency due to a bug in Sedona 1.5.1. You will need to specify the URL of the repository in `spark.jars.repositories` if you use 1.5.1. This has been fixed in Sedona 1.5.2 and later.
-
-Under BSD 3-clause (compatible with Apache 2.0 license)
-
-!!! abstract "Add HDF/NetCDF dependency"
-
- === "Sedona 1.3.1+"
-
- Add unidata repo to your pom.xml
-
- ```
- <repositories>
- <repository>
- <id>unidata-all</id>
- <name>Unidata All</name>
- <url>https://artifacts.unidata.ucar.edu/repository/unidata-all/</url>
- </repository>
- </repositories>
- ```
-
- Then add cdm-core to your POM dependency.
-
- ```xml
- <dependency>
- <groupId>edu.ucar</groupId>
- <artifactId>cdm-core</artifactId>
- <version>5.4.2</version>
- </dependency>
- ```
-
- === "Before Sedona 1.3.1"
-
- ```xml
- <!-- https://mvnrepository.com/artifact/org.datasyslab/sernetcdf -->
- <dependency>
- <groupId>org.datasyslab</groupId>
- <artifactId>sernetcdf</artifactId>
- <version>0.1.0</version>
- </dependency>
- ```
-
## Use Sedona unshaded jars
!!!warning
(sedona) 05/07: Refine STAC catalog reader
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jiayu pushed a commit to branch docs-mar-07
in repository https://gitbox.apache.org/repos/asf/sedona.git
commit 26e8f81b3fa069c150c7531e47310e0ac547cd85
Author: Jia Yu <ji...@apache.org>
AuthorDate: Sun Mar 9 23:52:49 2025 -0700
Refine STAC catalog reader
---
.../files/stac-sedona-spark.md} | 148 +++++++-------
docs/tutorial/sql.md | 225 +--------------------
mkdocs.yml | 2 +-
3 files changed, 78 insertions(+), 297 deletions(-)
diff --git a/docs/api/sql/Stac.md b/docs/tutorial/files/stac-sedona-spark.md
similarity index 86%
rename from docs/api/sql/Stac.md
rename to docs/tutorial/files/stac-sedona-spark.md
index 8d56644e5e..09bcba2328 100644
--- a/docs/api/sql/Stac.md
+++ b/docs/tutorial/files/stac-sedona-spark.md
@@ -17,6 +17,8 @@
under the License.
-->
+# STAC catalog with Apache Sedona and Spark
+
The STAC data source allows you to read data from a SpatioTemporal Asset Catalog (STAC) API. The data source supports reading STAC items and collections.
## Usage
@@ -108,29 +110,29 @@ root
+------------+--------------------+-------+--------------------+--------------------+--------------------+-----+-----------+--------------------+--------------+------------+--------------------+--------------------+-----------+-----------+-------------+-------+----+--------------------+--------------------+--------------------+
```
-# Filter Pushdown
+## Filter Pushdown
The STAC data source supports predicate pushdown for spatial and temporal filters. The data source can push down spatial and temporal filters to the underlying data source to reduce the amount of data that needs to be read.
-## Spatial Filter Pushdown
+### Spatial Filter Pushdown
Spatial filter pushdown allows the data source to apply spatial predicates (e.g., st_contains, st_intersects) directly at the data source level, reducing the amount of data transferred and processed.
-## Temporal Filter Pushdown
+### Temporal Filter Pushdown
Temporal filter pushdown allows the data source to apply temporal predicates (e.g., BETWEEN, >=, <=) directly at the data source level, similarly reducing the amount of data transferred and processed.
-# Examples
+## Examples
Here are some examples demonstrating how to query a STAC data source that is loaded into a table named `STAC_TABLE`.
-## SQL Select Without Filters
+### SQL Select Without Filters
```sql
SELECT id, datetime as dt, geometry, bbox FROM STAC_TABLE
```
-## SQL Select With Temporal Filter
+### SQL Select With Temporal Filter
```sql
SELECT id, datetime as dt, geometry, bbox
@@ -140,7 +142,7 @@ SELECT id, datetime as dt, geometry, bbox FROM STAC_TABLE
In this example, the data source will push down the temporal filter to the underlying data source.
-## SQL Select With Spatial Filter
+### SQL Select With Spatial Filter
```sql
SELECT id, geometry
@@ -150,7 +152,7 @@ In this example, the data source will push down the temporal filter to the under
In this example, the data source will push down the spatial filter to the underlying data source.
-## Sedona Configuration for STAC Reader
+### Sedona Configuration for STAC Reader
When using the STAC reader in Sedona, several configuration options can be set to control the behavior of the reader. These configurations are typically set in a `Map[String, String]` and passed to the reader. Below are the key sedona configuration options:
@@ -192,73 +194,13 @@ These configurations can be combined into a single `Map[String, String]` and pas
These options above provide fine-grained control over how the STAC data is read and processed in Sedona.
-# Python API
+## Python API
The Python API allows you to interact with a SpatioTemporal Asset Catalog (STAC) API using the Client class. This class provides methods to open a connection to a STAC API, retrieve collections, and search for items with various filters.
-## Client Class
-
-## Methods
-
-### `open(url: str) -> Client`
-
-Opens a connection to the specified STAC API URL.
-
-**Parameters:**
-
-- `url` (*str*): The URL of the STAC API to connect to.
- **Example:** `"https://planetarycomputer.microsoft.com/api/stac/v1"`
-
-**Returns:**
-
-- `Client`: An instance of the `Client` class connected to the specified URL.
-
----
-
-### `get_collection(collection_id: str) -> CollectionClient`
-
-Retrieves a collection client for the specified collection ID.
-
-**Parameters:**
-
-- `collection_id` (*str*): The ID of the collection to retrieve.
- **Example:** `"aster-l1t"`
-
-**Returns:**
+### Sample Code
-- `CollectionClient`: An instance of the `CollectionClient` class for the specified collection.
-
----
-
-### `search(*ids: Union[str, list], collection_id: str, bbox: Optional[list] = None, datetime: Optional[Union[str, datetime.datetime, list]] = None, max_items: Optional[int] = None, return_dataframe: bool = True) -> Union[Iterator[PyStacItem], DataFrame]`
-
-Searches for items in the specified collection with optional filters.
-
-**Parameters:**
-
-- `ids` (*Union[str, list]*): A variable number of item IDs to filter the items.
- **Example:** `"item_id1"` or `["item_id1", "item_id2"]`
-- `collection_id` (*str*): The ID of the collection to search in.
- **Example:** `"aster-l1t"`
-- `bbox` (*Optional[list]*): A list of bounding boxes for filtering the items. Each bounding box is represented as a list of four float values: `[min_lon, min_lat, max_lon, max_lat]`.
- **Example:** `[[ -180.0, -90.0, 180.0, 90.0 ]]`
-- `datetime` (*Optional[Union[str, datetime.datetime, list]]*): A single datetime, RFC 3339-compliant timestamp, or a list of date-time ranges for filtering the items.
- **Example:**
- - `"2020-01-01T00:00:00Z"`
- - `datetime.datetime(2020, 1, 1)`
- - `[["2020-01-01T00:00:00Z", "2021-01-01T00:00:00Z"]]`
-- `max_items` (*Optional[int]*): The maximum number of items to return from the search, even if there are more matching results.
- **Example:** `100`
-- `return_dataframe` (*bool*): If `True` (default), return the result as a Spark DataFrame instead of an iterator of `PyStacItem` objects.
- **Example:** `True`
-
-**Returns:**
-
-- *Union[Iterator[PyStacItem], DataFrame]*: An iterator of `PyStacItem` objects or a Spark DataFrame that matches the specified filters.
-
-## Sample Code
-
-### Initialize the Client
+#### Initialize the Client
```python
from sedona.stac.client import Client
@@ -267,7 +209,7 @@ from sedona.stac.client import Client
client = Client.open("https://planetarycomputer.microsoft.com/api/stac/v1")
```
-### Search Items on a Collection Within a Year
+#### Search Items on a Collection Within a Year
```python
items = client.search(
@@ -275,7 +217,7 @@ items = client.search(
)
```
-### Search Items on a Collection Within a Month and Max Items
+#### Search Items on a Collection Within a Month and Max Items
```python
items = client.search(
@@ -283,7 +225,7 @@ items = client.search(
)
```
-### Search Items with Bounding Box and Interval
+#### Search Items with Bounding Box and Interval
```python
items = client.search(
@@ -295,14 +237,14 @@ items = client.search(
)
```
-### Search Multiple Items with Multiple Bounding Boxes
+#### Search Multiple Items with Multiple Bounding Boxes
```python
bbox_list = [[-180.0, -90.0, 180.0, 90.0], [-100.0, -50.0, 100.0, 50.0]]
items = client.search(collection_id="aster-l1t", bbox=bbox_list, return_dataframe=False)
```
-### Search Items and Get DataFrame as Return with Multiple Intervals
+#### Search Items and Get DataFrame as Return with Multiple Intervals
```python
interval_list = [
@@ -315,7 +257,7 @@ df = client.search(
df.show()
```
-### Save Items in DataFrame to GeoParquet with Both Bounding Boxes and Intervals
+#### Save Items in DataFrame to GeoParquet with Both Bounding Boxes and Intervals
```python
# Save items in DataFrame to GeoParquet with both bounding boxes and intervals
@@ -326,7 +268,57 @@ client.get_collection("aster-l1t").save_to_geoparquet(
These examples demonstrate how to use the Client class to search for items in a STAC collection with various filters and return the results as either an iterator of PyStacItem objects or a Spark DataFrame.
-# References
+### Methods
+
+**`open(url: str) -> Client`**
+Opens a connection to the specified STAC API URL.
+
+Parameters:
+* `url` (*str*): The URL of the STAC API to connect to.
+ * Example: `"https://planetarycomputer.microsoft.com/api/stac/v1"`
+
+Returns:
+* `Client`: An instance of the `Client` class connected to the specified URL.
+
+---
+
+**`get_collection(collection_id: str) -> CollectionClient`**
+Retrieves a collection client for the specified collection ID.
+
+Parameters:
+* `collection_id` (*str*): The ID of the collection to retrieve.
+ * Example: `"aster-l1t"`
+
+Returns:
+* `CollectionClient`: An instance of the `CollectionClient` class for the specified collection.
+
+---
+
+**`search(*ids: Union[str, list], collection_id: str, bbox: Optional[list] = None, datetime: Optional[Union[str, datetime.datetime, list]] = None, max_items: Optional[int] = None, return_dataframe: bool = True) -> Union[Iterator[PyStacItem], DataFrame]`**
+Searches for items in the specified collection with optional filters.
+
+Parameters:
+
+* `ids` (*Union[str, list]*): A variable number of item IDs to filter the items.
+ * Example: `"item_id1"` or `["item_id1", "item_id2"]`
+* `collection_id` (*str*): The ID of the collection to search in.
+ * Example: `"aster-l1t"`
+* `bbox` (*Optional[list]*): A list of bounding boxes for filtering the items. Each bounding box is represented as a list of four float values: `[min_lon, min_lat, max_lon, max_lat]`.
+ * Example: `[[ -180.0, -90.0, 180.0, 90.0 ]]`
+* `datetime` (*Optional[Union[str, datetime.datetime, list]]*): A single datetime, RFC 3339-compliant timestamp, or a list of date-time ranges for filtering the items.
+ * Examples:
+ * `"2020-01-01T00:00:00Z"`
+ * `datetime.datetime(2020, 1, 1)`
+ * `[["2020-01-01T00:00:00Z", "2021-01-01T00:00:00Z"]]`
+* `max_items` (*Optional[int]*): The maximum number of items to return from the search, even if there are more matching results.
+ * Example: `100`
+* `return_dataframe` (*bool*): If `True` (default), return the result as a Spark DataFrame instead of an iterator of `PyStacItem` objects.
+ * Example: `True`
+
+Returns:
+* *Union[Iterator[PyStacItem], DataFrame]*: An iterator of `PyStacItem` objects or a Spark DataFrame that matches the specified filters.
+
+## References
- STAC Specification: https://stacspec.org/
diff --git a/docs/tutorial/sql.md b/docs/tutorial/sql.md
index 4ea1ff0754..fecbf3ef82 100644
--- a/docs/tutorial/sql.md
+++ b/docs/tutorial/sql.md
@@ -20,7 +20,7 @@
The page outlines the steps to manage spatial data using SedonaSQL.
!!!note
- Since v`1.5.0`, Sedona assumes geographic coordinates to be in longitude/latitude order. If your data is lat/lon order, please use `ST_FlipCoordinates` to swap X and Y.
+ Sedona assumes geographic coordinates to be in longitude/latitude order. If your data is lat/lon order, please use `ST_FlipCoordinates` to swap X and Y.
SedonaSQL supports SQL/MM Part3 Spatial SQL Standard. It includes four kinds of SQL operators as follows. All these operators can be directly called through:
@@ -64,8 +64,6 @@ Detailed SedonaSQL APIs are available here: [SedonaSQL API](../api/sql/Overview.
Use the following code to create your Sedona config at the beginning. If you already have a SparkSession (usually named `spark`) created by AWS EMR/Databricks/Microsoft Fabric, please ==skip this step==.
-==Sedona >= 1.4.1==
-
You can add additional Spark runtime config to the config builder. For example, `SedonaContext.builder().config("spark.sql.autoBroadcastJoinThreshold", "10485760")`
=== "Scala"
@@ -111,65 +109,10 @@ You can add additional Spark runtime config to the config builder. For example,
```
If you are using a different Spark version, please replace the `3.3` in package name of sedona-spark-shaded with the corresponding major.minor version of Spark, such as `sedona-spark-shaded-3.4_2.12:{{ sedona.current_version }}`.
-==Sedona < 1.4.1==
-
-The following method has been deprecated since Sedona 1.4.1. Please use the method above to create your Sedona config.
-
-=== "Scala"
-
- ```scala
- var sparkSession = SparkSession.builder()
- .master("local[*]") // Delete this if run in cluster mode
- .appName("readTestScala") // Change this to a proper name
- // Enable Sedona custom Kryo serializer
- .config("spark.serializer", classOf[KryoSerializer].getName) // org.apache.spark.serializer.KryoSerializer
- .config("spark.kryo.registrator", classOf[SedonaKryoRegistrator].getName)
- .getOrCreate() // org.apache.sedona.core.serde.SedonaKryoRegistrator
- ```
- If you use SedonaViz together with SedonaSQL, please use the following two lines to enable Sedona Kryo serializer instead:
- ```scala
- .config("spark.serializer", classOf[KryoSerializer].getName) // org.apache.spark.serializer.KryoSerializer
- .config("spark.kryo.registrator", classOf[SedonaVizKryoRegistrator].getName) // org.apache.sedona.viz.core.Serde.SedonaVizKryoRegistrator
- ```
-
-=== "Java"
-
- ```java
- SparkSession sparkSession = SparkSession.builder()
- .master("local[*]") // Delete this if run in cluster mode
- .appName("readTestJava") // Change this to a proper name
- // Enable Sedona custom Kryo serializer
- .config("spark.serializer", KryoSerializer.class.getName()) // org.apache.spark.serializer.KryoSerializer
- .config("spark.kryo.registrator", SedonaKryoRegistrator.class.getName())
- .getOrCreate() // org.apache.sedona.core.serde.SedonaKryoRegistrator
- ```
- If you use SedonaViz together with SedonaSQL, please use the following two lines to enable Sedona Kryo serializer instead:
- ```java
- .config("spark.serializer", KryoSerializer.class.getName()) // org.apache.spark.serializer.KryoSerializer
- .config("spark.kryo.registrator", SedonaVizKryoRegistrator.class.getName()) // org.apache.sedona.viz.core.Serde.SedonaVizKryoRegistrator
- ```
-
-=== "Python"
-
- ```python
- sparkSession = SparkSession. \
- builder. \
- appName('readTestPython'). \
- config("spark.serializer", KryoSerializer.getName()). \
- config("spark.kryo.registrator", SedonaKryoRegistrator.getName()). \
- config('spark.jars.packages',
- 'org.apache.sedona:sedona-spark-shaded-3.3_2.12:{{ sedona.current_version }},'
- 'org.datasyslab:geotools-wrapper:{{ sedona.current_geotools }}'). \
- getOrCreate()
- ```
- If you are using Spark versions >= 3.4, please replace the `3.0` in package name of sedona-spark-shaded with the corresponding major.minor version of Spark, such as `sedona-spark-shaded-3.4_2.12:{{ sedona.current_version }}`.
-
## Initiate SedonaContext
Add the following line after creating Sedona config. If you already have a SparkSession (usually named `spark`) created by AWS EMR/Databricks/Microsoft Fabric, please call `sedona = SedonaContext.create(spark)` instead. For ==Databricks==, the situation is more complicated, please refer to [Databricks setup guide](../setup/databricks.md), but generally you don't need to create SedonaContext.
-==Sedona >= 1.4.1==
-
=== "Scala"
```scala
@@ -194,30 +137,6 @@ Add the following line after creating Sedona config. If you already have a Spark
sedona = SedonaContext.create(config)
```
-==Sedona < 1.4.1==
-
-The following method has been deprecated since Sedona 1.4.1. Please use the method above to create your SedonaContext.
-
-=== "Scala"
-
- ```scala
- SedonaSQLRegistrator.registerAll(sparkSession)
- ```
-
-=== "Java"
-
- ```java
- SedonaSQLRegistrator.registerAll(sparkSession)
- ```
-
-=== "Python"
-
- ```python
- from sedona.register import SedonaRegistrator
-
- SedonaRegistrator.registerAll(spark)
- ```
-
You can also register everything by passing `--conf spark.sql.extensions=org.apache.sedona.sql.SedonaSqlExtensions` to `spark-submit` or `spark-shell`.
## Load data from files
@@ -315,56 +234,6 @@ root
Since `v1.6.1`, Sedona supports reading GeoJSON files using the `geojson` data source. It is designed to handle JSON files that use [GeoJSON format](https://datatracker.ietf.org/doc/html/rfc7946) for their geometries.
-This includes SpatioTemporal Asset Catalog (STAC) files, GeoJSON features, GeoJSON feature collections and other variations.
-The key functionality lies in the way 'geometry' fields are processed: these are specifically read as Sedona's `GeometryUDT` type, ensuring integration with Sedona's suite of spatial functions.
-
-### Key features
-
-- Broad Support: The reader and writer are versatile, supporting all GeoJSON-formatted files, including STAC files, feature collections, and more.
-- Geometry Transformation: When reading, fields named 'geometry' are automatically converted from GeoJSON format to Sedona's `GeometryUDT` type and vice versa when writing.
-
-### Load MultiLine GeoJSON FeatureCollection
-
-Suppose we have a GeoJSON FeatureCollection file as follows.
-This entire file is considered as a single GeoJSON FeatureCollection object.
-Multiline format is preferable for scenarios where files need to be human-readable or manually edited.
-
-```json
-{ "type": "FeatureCollection",
- "features": [
- { "type": "Feature",
- "geometry": {"type": "Point", "coordinates": [102.0, 0.5]},
- "properties": {"prop0": "value0"}
- },
- { "type": "Feature",
- "geometry": {
- "type": "LineString",
- "coordinates": [
- [102.0, 0.0], [103.0, 1.0], [104.0, 0.0], [105.0, 1.0]
- ]
- },
- "properties": {
- "prop0": "value1",
- "prop1": 0.0
- }
- },
- { "type": "Feature",
- "geometry": {
- "type": "Polygon",
- "coordinates": [
- [ [100.0, 0.0], [101.0, 0.0], [101.0, 1.0],
- [100.0, 1.0], [100.0, 0.0] ]
- ]
- },
- "properties": {
- "prop0": "value2",
- "prop1": {"this": "that"}
- }
- }
- ]
-}
-```
-
Set the `multiLine` option to `True` to read multiline GeoJSON files.
=== "Python"
@@ -402,81 +271,7 @@ Set the `multiLine` option to `True` to read multiline GeoJSON files.
df.printSchema();
```
-The output is as follows:
-
-```
-+--------------------+------+
-| geometry| prop0|
-+--------------------+------+
-| POINT (102 0.5)|value0|
-|LINESTRING (102 0...|value1|
-|POLYGON ((100 0, ...|value2|
-+--------------------+------+
-
-root
- |-- geometry: geometry (nullable = false)
- |-- prop0: string (nullable = true)
-
-```
-
-### Load Single Line GeoJSON Features
-
-Suppose we have a single-line GeoJSON Features dataset as follows. Each line is a single GeoJSON Feature.
-This format is efficient for processing large datasets where each line is a separate, self-contained GeoJSON object.
-
-```json
-{"type":"Feature","geometry":{"type":"Point","coordinates":[102.0,0.5]},"properties":{"prop0":"value0"}}
-{"type":"Feature","geometry":{"type":"LineString","coordinates":[[102.0,0.0],[103.0,1.0],[104.0,0.0],[105.0,1.0]]},"properties":{"prop0":"value1"}}
-{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[100.0,0.0],[101.0,0.0],[101.0,1.0],[100.0,1.0],[100.0,0.0]]]},"properties":{"prop0":"value2"}}
-```
-
-By default, when `option` is not specified, Sedona reads a GeoJSON file as a single line GeoJSON.
-
-=== "Python"
-
- ```python
- df = sedona.read.format("geojson").load("PATH/TO/MYFILE.json")
- .withColumn("prop0", f.expr("properties['prop0']")).drop("properties").drop("type")
-
- df.show()
- df.printSchema()
- ```
-
-=== "Scala"
-
- ```scala
- val df = sedona.read.format("geojson").load("PATH/TO/MYFILE.json")
- .withColumn("prop0", expr("properties['prop0']")).drop("properties").drop("type")
-
- df.show()
- df.printSchema()
- ```
-
-=== "Java"
-
- ```java
- Dataset<Row> df = sedona.read.format("geojson").load("PATH/TO/MYFILE.json")
- .withColumn("prop0", expr("properties['prop0']")).drop("properties").drop("type")
-
- df.show()
- df.printSchema()
- ```
-
-The output is as follows:
-
-```
-+--------------------+------+
-| geometry| prop0|
-+--------------------+------+
-| POINT (102 0.5)|value0|
-|LINESTRING (102 0...|value1|
-|POLYGON ((100 0, ...|value2|
-+--------------------+------+
-
-root
- |-- geometry: geometry (nullable = false)
- |-- prop0: string (nullable = true)
-```
+See [this page](files/geojson-sedona-spark.md) for more information on loading GeoJSON files.
## Load Shapefile
@@ -502,7 +297,7 @@ Since v`1.7.0`, Sedona supports loading Shapefile as a DataFrame.
The input path can be a directory containing one or multiple shapefiles, or path to a `.shp` file.
-See [this page](../files/shapefile-sedona-spark) for more information on loading Shapefiles.
+See [this page](files/shapefile-sedona-spark.md) for more information on loading Shapefiles.
## Load GeoParquet
@@ -550,7 +345,7 @@ Please refer to [Reading Legacy Parquet Files](../api/sql/Reading-legacy-parquet
GeoParquet file reader does not work on Databricks runtime when Photon is enabled. Please disable Photon when using
GeoParquet file reader on Databricks runtime.
-See [this page](../files/geoparquet-sedona-spark) for more information on loading GeoParquet.
+See [this page](files/geoparquet-sedona-spark.md) for more information on loading GeoParquet.
## Load data from JDBC data sources
@@ -634,7 +429,7 @@ Since v1.7.0, Sedona supports loading Geopackage file format as a DataFrame.
df = sedona.read.format("geopackage").option("tableName", "tab").load("/path/to/geopackage")
```
-See [this page](../files/geopackage-sedona-spark) for more information on loading GeoPackage.
+See [this page](files/geopackage-sedona-spark.md) for more information on loading GeoPackage.
## Load from OSM PBF
@@ -1401,13 +1196,7 @@ Since `v1.6.1`, the GeoJSON data source in Sedona can be used to save a Spatial
df.write.format("geojson").save("YOUR/PATH.json")
```
-The structure of the generated file will be like this:
-
-```json
-{"type":"Feature","geometry":{"type":"Point","coordinates":[102.0,0.5]},"properties":{"prop0":"value0"}}
-{"type":"Feature","geometry":{"type":"LineString","coordinates":[[102.0,0.0],[103.0,1.0],[104.0,0.0],[105.0,1.0]]},"properties":{"prop0":"value1"}}
-{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[100.0,0.0],[101.0,0.0],[101.0,1.0],[100.0,1.0],[100.0,0.0]]]},"properties":{"prop0":"value2"}}
-```
+See [this page](files/geojson-sedona-spark.md) for more information on writing to GeoJSON.
## Save GeoParquet
@@ -1417,7 +1206,7 @@ Since v`1.3.0`, Sedona natively supports writing GeoParquet file. GeoParquet can
df.write.format("geoparquet").save(geoparquetoutputlocation + "/GeoParquet_File_Name.parquet")
```
-See [this page](../files/geoparquet-sedona-spark) for more information on writing to GeoParquet.
+See [this page](files/geoparquet-sedona-spark.md) for more information on writing to GeoParquet.
## Save to Postgis
diff --git a/mkdocs.yml b/mkdocs.yml
index 2db776b633..60618b5a42 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -66,6 +66,7 @@ nav:
- GeoParquet: tutorial/files/geoparquet-sedona-spark.md
- GeoJSON: tutorial/files/geojson-sedona-spark.md
- Shapefiles: tutorial/files/shapefiles-sedona-spark.md
+ - STAC catalog: tutorial/files/stac-sedona-spark.md
- Concepts:
- Spatial Joins: tutorial/concepts/spatial-joins.md
- Clustering Algorithms: tutorial/concepts/clustering-algorithms.md
@@ -97,7 +98,6 @@ nav:
- Query optimization: api/sql/Optimizer.md
- Nearest-Neighbour searching: api/sql/NearestNeighbourSearching.md
- "Spider:Spatial Data Generator": api/sql/Spider.md
- - Reading STAC Data Source: api/sql/Stac.md
- Reading Legacy Parquet Files: api/sql/Reading-legacy-parquet.md
- Visualization:
- SedonaPyDeck: api/sql/Visualization_SedonaPyDeck.md
(sedona) 03/07: Update docker
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jiayu pushed a commit to branch docs-mar-07
in repository https://gitbox.apache.org/repos/asf/sedona.git
commit 3b49d67f4f4d1fb8bd09454e602f0dfd94fcc950
Author: Jia Yu <ji...@apache.org>
AuthorDate: Sun Mar 9 21:22:27 2025 -0700
Update docker
---
docs/setup/docker.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/docs/setup/docker.md b/docs/setup/docker.md
index 4af395020b..a5f708368e 100644
--- a/docs/setup/docker.md
+++ b/docs/setup/docker.md
@@ -68,7 +68,7 @@ This command will bind the container's ports 8888, 8080, 8081, 4040, 8085 to the
Example 2:
```bash
-docker run -p 8888:8888 -p 8080:8080 -p 8081:8081 -p 4040:4040 -p 8085:8085 apache/sedona:{{ sedona.current_version }}
+docker run -d -e -p 8888:8888 -p 8080:8080 -p 8081:8081 -p 4040:4040 -p 8085:8085 apache/sedona:{{ sedona.current_version }}
```
This command will start a container with 4GB RAM for the driver and 4GB RAM for the executor and use Sedona {{ sedona.current_version }} image.
@@ -91,7 +91,7 @@ docker run -d -e DRIVER_MEM=6g -e EXECUTOR_MEM=8g \
### Start coding
-Open your browser and go to [http://localhost:8888/](http://localhost:8888/) to start coding with Sedona. You can also access Apache Zeppelin at [http://localhost:8085/classic/](http://localhost:8085/classic/ ) using your browser.
+Open your browser and go to [http://localhost:8888/](http://localhost:8888/) to start coding with Sedona in Jupyter Notebook. You can also access Apache Zeppelin at [http://localhost:8085/classic/](http://localhost:8085/classic/ ) using your browser.
### Notes