You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "yutannihilation (via GitHub)" <gi...@apache.org> on 2025/11/01 07:03:48 UTC
[PR] Implement ST_Dump [sedona-db]
yutannihilation opened a new pull request, #269:
URL: https://github.com/apache/sedona-db/pull/269
This is very much work in progress and I'm kind of stuck mainly because I'm not familiar with handling complex types in Apache Arrow...
- This crashes R session, so probably I'm doing something is wrong.
- I don't figure out yet how to test a list result.
Note: Sedona's ST_Dump returns different result, so probably the Python test will need some tweak.
https://sedona.apache.org/latest/api/sql/Function/#st_dump
https://postgis.net/docs/en/ST_Dump.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement ST_Dump [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #269:
URL: https://github.com/apache/sedona-db/pull/269#discussion_r2495899814
##########
rust/sedona-functions/src/st_dump.rs:
##########
@@ -0,0 +1,437 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+use arrow_array::{
+ builder::{BinaryBuilder, NullBufferBuilder, OffsetBufferBuilder, UInt32Builder},
+ ListArray, StructArray,
+};
+use arrow_schema::{DataType, Field, Fields};
+use datafusion_common::error::Result;
+use datafusion_expr::{
+ scalar_doc_sections::DOC_SECTION_OTHER, ColumnarValue, Documentation, Volatility,
+};
+use geo_traits::{
+ GeometryCollectionTrait, GeometryTrait, GeometryType, MultiLineStringTrait, MultiPointTrait,
+ MultiPolygonTrait,
+};
+use sedona_common::sedona_internal_err;
+use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
+use sedona_geometry::wkb_factory::WKB_MIN_PROBABLE_BYTES;
+use sedona_schema::{
+ datatypes::{SedonaType, WKB_GEOMETRY},
+ matchers::ArgMatcher,
+};
+use std::sync::Arc;
+
+use crate::executor::WkbExecutor;
+
+/// ST_Dump() scalar UDF
+///
+/// Native implementation to get all the points of a geometry as MULTIPOINT
+pub fn st_dump_udf() -> SedonaScalarUDF {
+ SedonaScalarUDF::new(
+ "st_dump",
+ vec![Arc::new(STDump)],
+ Volatility::Immutable,
+ Some(st_dump_doc()),
+ )
+}
+
+fn st_dump_doc() -> Documentation {
+ Documentation::builder(
+ DOC_SECTION_OTHER,
+ "Extracts the components of a geometry.",
+ "ST_Dump (geom: Geometry)",
+ )
+ .with_argument("geom", "geometry: Input geometry")
+ .with_sql_example("SELECT ST_Dump(ST_GeomFromWKT('MULTIPOINT (0 1, 2 3, 4 5)'))")
+ .build()
+}
+
+#[derive(Debug)]
+struct STDump;
+
+// This enum is solely for passing the subset of wkb geometry to STDumpStructBuilder.
+// Maybe we can pass the underlying raw WKB bytes directly, but this just works for now.
+enum SingleWkb<'a> {
+ Point(&'a wkb::reader::Point<'a>),
+ LineString(&'a wkb::reader::LineString<'a>),
+ Polygon(&'a wkb::reader::Polygon<'a>),
+}
+
+// A builder for a list of the structs
+struct STDumpBuilder {
+ path_array_builder: UInt32Builder,
+ path_array_offsets_builder: OffsetBufferBuilder<i32>,
+ geom_builder: BinaryBuilder,
+ struct_offsets_builder: OffsetBufferBuilder<i32>,
+ null_builder: NullBufferBuilder,
+ parent_path: Vec<u32>,
+}
+
+impl STDumpBuilder {
+ fn new(num_iter: usize) -> Self {
+ let path_array_builder = UInt32Builder::with_capacity(num_iter);
+ let path_array_offsets_builder = OffsetBufferBuilder::new(num_iter);
+ let geom_builder =
+ BinaryBuilder::with_capacity(num_iter, WKB_MIN_PROBABLE_BYTES * num_iter);
+ let struct_offsets_builder = OffsetBufferBuilder::new(num_iter);
+ let null_builder = NullBufferBuilder::new(num_iter);
+
+ Self {
+ path_array_builder,
+ path_array_offsets_builder,
+ geom_builder,
+ struct_offsets_builder,
+ null_builder,
+ parent_path: Vec::new(), // Reusable buffer to avoid allocation per row
+ }
+ }
+
+ // This appends both path and geom at once.
+ fn append_single_struct(&mut self, cur_index: Option<u32>, wkb: SingleWkb<'_>) -> Result<()> {
+ self.path_array_builder.append_slice(&self.parent_path);
+ if let Some(cur_index) = cur_index {
+ self.path_array_builder.append_value(cur_index);
+ self.path_array_offsets_builder
+ .push_length(self.parent_path.len() + 1);
+ } else {
+ self.path_array_offsets_builder
+ .push_length(self.parent_path.len());
+ }
+
+ let write_result = match wkb {
+ SingleWkb::Point(point) => {
+ wkb::writer::write_point(&mut self.geom_builder, &point, &Default::default())
+ }
+ SingleWkb::LineString(line_string) => wkb::writer::write_line_string(
+ &mut self.geom_builder,
+ &line_string,
+ &Default::default(),
+ ),
+ SingleWkb::Polygon(polygon) => {
+ wkb::writer::write_polygon(&mut self.geom_builder, &polygon, &Default::default())
+ }
+ };
+ if let Err(e) = write_result {
+ return sedona_internal_err!("Failed to write WKB: {e}");
+ }
+
+ self.geom_builder.append_value([]);
+
+ Ok(())
+ }
+
+ fn append_structs(&mut self, wkb: &wkb::reader::Wkb<'_>) -> Result<i32> {
+ match wkb.as_type() {
+ GeometryType::Point(point) => {
+ self.append_single_struct(None, SingleWkb::Point(point))?;
+ Ok(1)
+ }
+ GeometryType::LineString(line_string) => {
+ self.append_single_struct(None, SingleWkb::LineString(line_string))?;
+ Ok(1)
+ }
+ GeometryType::Polygon(polygon) => {
+ self.append_single_struct(None, SingleWkb::Polygon(polygon))?;
+ Ok(1)
+ }
+ GeometryType::MultiPoint(multi_point) => {
+ for (index, point) in multi_point.points().enumerate() {
+ self.append_single_struct(Some((index + 1) as _), SingleWkb::Point(&point))?;
+ }
+ Ok(multi_point.num_points() as _)
+ }
+ GeometryType::MultiLineString(multi_line_string) => {
+ for (index, line_string) in multi_line_string.line_strings().enumerate() {
+ self.append_single_struct(
+ Some((index + 1) as _),
+ SingleWkb::LineString(line_string),
+ )?;
+ }
+ Ok(multi_line_string.num_line_strings() as _)
+ }
+ GeometryType::MultiPolygon(multi_polygon) => {
+ for (index, polygon) in multi_polygon.polygons().enumerate() {
+ self.append_single_struct(Some((index + 1) as _), SingleWkb::Polygon(polygon))?;
+ }
+ Ok(multi_polygon.num_polygons() as _)
+ }
+ GeometryType::GeometryCollection(geometry_collection) => {
+ let mut num_geometries: i32 = 0;
+
+ self.parent_path.push(0); // add an index for the next nested level
+
+ for geometry in geometry_collection.geometries() {
+ // increment the index
+ if let Some(index) = self.parent_path.last_mut() {
+ *index += 1;
+ }
+ num_geometries += self.append_structs(geometry)?;
+ }
+
+ self.parent_path.truncate(self.parent_path.len() - 1); // clear the index before returning to the upper level
+
+ Ok(num_geometries)
+ }
+ _ => sedona_internal_err!("Invalid geometry type"),
+ }
+ }
+
+ fn append(&mut self, wkb: &wkb::reader::Wkb<'_>) -> Result<()> {
+ self.parent_path.clear();
+
+ let num_geometries = self.append_structs(wkb)?;
+ self.null_builder.append(true);
+ self.struct_offsets_builder
+ .push_length(num_geometries as usize);
+ Ok(())
+ }
+
+ fn append_null(&mut self) {
+ self.path_array_offsets_builder.push_length(0);
+ self.geom_builder.append_null();
+ self.struct_offsets_builder.push_length(1);
+ self.null_builder.append(false);
+ }
+
+ fn finish(mut self) -> ListArray {
+ let path_array = Arc::new(self.path_array_builder.finish());
+ let path_offsets = self.path_array_offsets_builder.finish();
+ let geom_array = self.geom_builder.finish();
+
+ let path_field = Arc::new(Field::new("item", DataType::UInt32, true));
+ let path_list = ListArray::new(path_field, path_offsets, path_array, None);
+
+ let fields = Fields::from(vec![
+ Field::new(
+ "path",
+ DataType::List(Arc::new(Field::new("item", DataType::UInt32, true))),
+ true,
+ ),
+ Field::new("geom", DataType::Binary, true),
+ ]);
+ let struct_array = StructArray::try_new(
+ fields.clone(),
+ vec![Arc::new(path_list), Arc::new(geom_array)],
+ None,
+ )
+ .unwrap();
+ let struct_offsets = self.struct_offsets_builder.finish();
+ let struct_field = Arc::new(Field::new("item", DataType::Struct(fields), true));
+ let nulls = self.null_builder.finish();
+ ListArray::new(struct_field, struct_offsets, Arc::new(struct_array), nulls)
+ }
+}
+
+impl SedonaScalarKernel for STDump {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ let matcher = ArgMatcher::new(vec![ArgMatcher::is_geometry()], geometry_dump_type());
Review Comment:
I believe this should to propagate the CRS of args[0] into the `geometry_dump_type()`. The reason this isn't done in any other type matcher code is because the type matcher does this automatically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement ST_Dump [sedona-db]
Posted by "yutannihilation (via GitHub)" <gi...@apache.org>.
yutannihilation commented on PR #269:
URL: https://github.com/apache/sedona-db/pull/269#issuecomment-3476536635
Thanks so much for the tip!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement ST_Dump [sedona-db]
Posted by "yutannihilation (via GitHub)" <gi...@apache.org>.
yutannihilation commented on PR #269:
URL: https://github.com/apache/sedona-db/pull/269#issuecomment-3488400934
Thanks, I simply didn't know! It looks cleaner. I'm trying to switch to that approach, but probably it takes some more time. Currently, I'm struggling to figure out how `append_null()` should be handled, but probably there are several more things that I still don't understand...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement ST_Dump [sedona-db]
Posted by "yutannihilation (via GitHub)" <gi...@apache.org>.
yutannihilation commented on PR #269:
URL: https://github.com/apache/sedona-db/pull/269#issuecomment-3485747296
Benchmark result:
```
------------------------------------------------------------------------------------------------------------- benchmark: 6 tests -------------------------------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_st_dump[collections_simple-SedonaDBSingleThread] 25.8595 (1.0) 28.0133 (1.0) 26.1220 (1.0) 0.4491 (1.0) 25.9570 (1.0) 0.1710 (1.0) 2;5 38.2819 (1.0) 26 1
test_st_dump[collections_simple-DuckDBSingleThread] 54.1694 (2.09) 61.4570 (2.19) 56.3276 (2.16) 2.9164 (6.49) 55.2865 (2.13) 2.1671 (12.68) 1;1 17.7533 (0.46) 5 1
test_st_dump[collections_complex-SedonaDBSingleThread] 382.6457 (14.80) 472.4213 (16.86) 402.0678 (15.39) 39.3457 (87.60) 385.3617 (14.85) 23.3447 (136.55) 1;1 2.4871 (0.06) 5 1
test_st_dump[collections_complex-DuckDBSingleThread] 749.3529 (28.98) 763.6902 (27.26) 757.9570 (29.02) 5.9036 (13.14) 760.3898 (29.29) 9.0438 (52.90) 1;0 1.3193 (0.03) 5 1
test_st_dump[collections_simple-PostGISSingleThread] 3,297.8760 (127.53) 3,372.9212 (120.40) 3,340.3870 (127.88) 32.3133 (71.94) 3,355.1089 (129.26) 53.6658 (313.91) 2;0 0.2994 (0.01) 5 1
test_st_dump[collections_complex-PostGISSingleThread] 37,008.9275 (>1000.0) 37,411.0606 (>1000.0) 37,149.3238 (>1000.0) 184.6306 (411.07) 37,029.6383 (>1000.0) 295.5638 (>1000.0) 1;0 0.0269 (0.00) 5 1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement ST_Dump [sedona-db]
Posted by "yutannihilation (via GitHub)" <gi...@apache.org>.
yutannihilation commented on PR #269:
URL: https://github.com/apache/sedona-db/pull/269#issuecomment-3477003673
Thanks to your help, I think I figured out how to handle and test these complex types. I'll work on the actual implementation...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement ST_Dump [sedona-db]
Posted by "yutannihilation (via GitHub)" <gi...@apache.org>.
yutannihilation commented on PR #269:
URL: https://github.com/apache/sedona-db/pull/269#issuecomment-3492013223
This should be ready for review again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement ST_Dump [sedona-db]
Posted by "yutannihilation (via GitHub)" <gi...@apache.org>.
yutannihilation commented on code in PR #269:
URL: https://github.com/apache/sedona-db/pull/269#discussion_r2483787633
##########
rust/sedona-functions/src/st_dump.rs:
##########
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+use arrow_array::builder::{BinaryBuilder, ListBuilder};
+use arrow_schema::DataType;
+use datafusion_common::error::Result;
+use datafusion_expr::{
+ scalar_doc_sections::DOC_SECTION_OTHER, ColumnarValue, Documentation, Volatility,
+};
+use geo_traits::{CoordTrait, GeometryTrait, PointTrait};
+use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
+use sedona_geometry::{
+ error::SedonaGeometryError,
+ wkb_factory::{write_wkb_coord_trait, write_wkb_point_header, WKB_MIN_PROBABLE_BYTES},
+};
+use sedona_schema::{
+ datatypes::{SedonaType, WKB_GEOMETRY},
+ matchers::ArgMatcher,
+};
+use std::{io::Write, sync::Arc};
+
+use crate::executor::WkbExecutor;
+
+/// ST_Dump() scalar UDF
+///
+/// Native implementation to get all the points of a geometry as MULTIPOINT
+pub fn st_dump_udf() -> SedonaScalarUDF {
+ SedonaScalarUDF::new(
+ "st_dump",
+ vec![Arc::new(STDump)],
+ Volatility::Immutable,
+ Some(st_dump_doc()),
+ )
+}
+
+fn st_dump_doc() -> Documentation {
+ Documentation::builder(
+ DOC_SECTION_OTHER,
+ "Extracts the components of a geometry.",
+ "ST_Dump (geom: Geometry)",
+ )
+ .with_argument("geom", "geometry: Input geometry")
+ .with_sql_example("SELECT ST_Dump(ST_GeomFromWKT('MULTIPOINT (0 1, 2 3, 4 5)'))")
+ .build()
+}
+
+#[derive(Debug)]
+struct STDump;
+
+impl SedonaScalarKernel for STDump {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ let matcher = ArgMatcher::new(vec![ArgMatcher::is_geometry()], WKB_GEOMETRY);
Review Comment:
Ah, okay. Since I've never found the `path` useful and been tired of always typing `.geom`, I felt Sedona's `ST_Dump` is cool! But, probably you are right. Let's focus on the compatibility.
(I still think SedonaDB / Sedona can have better version of `ST_Dump`, but let's forget for now)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement ST_Dump [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #269:
URL: https://github.com/apache/sedona-db/pull/269#discussion_r2483768052
##########
rust/sedona-functions/src/st_dump.rs:
##########
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+use arrow_array::builder::{BinaryBuilder, ListBuilder};
+use arrow_schema::DataType;
+use datafusion_common::error::Result;
+use datafusion_expr::{
+ scalar_doc_sections::DOC_SECTION_OTHER, ColumnarValue, Documentation, Volatility,
+};
+use geo_traits::{CoordTrait, GeometryTrait, PointTrait};
+use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
+use sedona_geometry::{
+ error::SedonaGeometryError,
+ wkb_factory::{write_wkb_coord_trait, write_wkb_point_header, WKB_MIN_PROBABLE_BYTES},
+};
+use sedona_schema::{
+ datatypes::{SedonaType, WKB_GEOMETRY},
+ matchers::ArgMatcher,
+};
+use std::{io::Write, sync::Arc};
+
+use crate::executor::WkbExecutor;
+
+/// ST_Dump() scalar UDF
+///
+/// Native implementation to get all the points of a geometry as MULTIPOINT
+pub fn st_dump_udf() -> SedonaScalarUDF {
+ SedonaScalarUDF::new(
+ "st_dump",
+ vec![Arc::new(STDump)],
+ Volatility::Immutable,
+ Some(st_dump_doc()),
+ )
+}
+
+fn st_dump_doc() -> Documentation {
+ Documentation::builder(
+ DOC_SECTION_OTHER,
+ "Extracts the components of a geometry.",
+ "ST_Dump (geom: Geometry)",
+ )
+ .with_argument("geom", "geometry: Input geometry")
+ .with_sql_example("SELECT ST_Dump(ST_GeomFromWKT('MULTIPOINT (0 1, 2 3, 4 5)'))")
+ .build()
+}
+
+#[derive(Debug)]
+struct STDump;
+
+impl SedonaScalarKernel for STDump {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ let matcher = ArgMatcher::new(vec![ArgMatcher::is_geometry()], WKB_GEOMETRY);
Review Comment:
```suggestion
let matcher = ArgMatcher::new(vec![ArgMatcher::is_geometry()], get_geom_list_type()?);
```
##########
rust/sedona-functions/src/st_dump.rs:
##########
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+use arrow_array::builder::{BinaryBuilder, ListBuilder};
+use arrow_schema::DataType;
+use datafusion_common::error::Result;
+use datafusion_expr::{
+ scalar_doc_sections::DOC_SECTION_OTHER, ColumnarValue, Documentation, Volatility,
+};
+use geo_traits::{CoordTrait, GeometryTrait, PointTrait};
+use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
+use sedona_geometry::{
+ error::SedonaGeometryError,
+ wkb_factory::{write_wkb_coord_trait, write_wkb_point_header, WKB_MIN_PROBABLE_BYTES},
+};
+use sedona_schema::{
+ datatypes::{SedonaType, WKB_GEOMETRY},
+ matchers::ArgMatcher,
+};
+use std::{io::Write, sync::Arc};
+
+use crate::executor::WkbExecutor;
+
+/// ST_Dump() scalar UDF
+///
+/// Native implementation to get all the points of a geometry as MULTIPOINT
+pub fn st_dump_udf() -> SedonaScalarUDF {
+ SedonaScalarUDF::new(
+ "st_dump",
+ vec![Arc::new(STDump)],
+ Volatility::Immutable,
+ Some(st_dump_doc()),
+ )
+}
+
+fn st_dump_doc() -> Documentation {
+ Documentation::builder(
+ DOC_SECTION_OTHER,
+ "Extracts the components of a geometry.",
+ "ST_Dump (geom: Geometry)",
+ )
+ .with_argument("geom", "geometry: Input geometry")
+ .with_sql_example("SELECT ST_Dump(ST_GeomFromWKT('MULTIPOINT (0 1, 2 3, 4 5)'))")
+ .build()
+}
+
+#[derive(Debug)]
+struct STDump;
+
+impl SedonaScalarKernel for STDump {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ let matcher = ArgMatcher::new(vec![ArgMatcher::is_geometry()], WKB_GEOMETRY);
Review Comment:
DuckDB and PostGIS both agree on the output type here (Sedona is the odd one out). Let's implement the PostGIS/DuckDB version because we can test those (I'll file a bug report in Sedona to find out if that behaviour was intentional).
##########
rust/sedona-functions/src/st_dump.rs:
##########
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+use arrow_array::builder::{BinaryBuilder, ListBuilder};
+use arrow_schema::DataType;
+use datafusion_common::error::Result;
+use datafusion_expr::{
+ scalar_doc_sections::DOC_SECTION_OTHER, ColumnarValue, Documentation, Volatility,
+};
+use geo_traits::{CoordTrait, GeometryTrait, PointTrait};
+use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
+use sedona_geometry::{
+ error::SedonaGeometryError,
+ wkb_factory::{write_wkb_coord_trait, write_wkb_point_header, WKB_MIN_PROBABLE_BYTES},
+};
+use sedona_schema::{
+ datatypes::{SedonaType, WKB_GEOMETRY},
+ matchers::ArgMatcher,
+};
+use std::{io::Write, sync::Arc};
+
+use crate::executor::WkbExecutor;
+
+/// ST_Dump() scalar UDF
+///
+/// Native implementation to get all the points of a geometry as MULTIPOINT
+pub fn st_dump_udf() -> SedonaScalarUDF {
+ SedonaScalarUDF::new(
+ "st_dump",
+ vec![Arc::new(STDump)],
+ Volatility::Immutable,
+ Some(st_dump_doc()),
+ )
+}
+
+fn st_dump_doc() -> Documentation {
+ Documentation::builder(
+ DOC_SECTION_OTHER,
+ "Extracts the components of a geometry.",
+ "ST_Dump (geom: Geometry)",
+ )
+ .with_argument("geom", "geometry: Input geometry")
+ .with_sql_example("SELECT ST_Dump(ST_GeomFromWKT('MULTIPOINT (0 1, 2 3, 4 5)'))")
+ .build()
+}
+
+#[derive(Debug)]
+struct STDump;
+
+impl SedonaScalarKernel for STDump {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ let matcher = ArgMatcher::new(vec![ArgMatcher::is_geometry()], WKB_GEOMETRY);
+ let geom_type = matcher.match_args(args)?;
+
+ get_geom_list_type(geom_type.as_ref())
+ }
+
+ fn invoke_batch(
+ &self,
+ arg_types: &[SedonaType],
+ args: &[ColumnarValue],
+ ) -> Result<ColumnarValue> {
+ let executor = WkbExecutor::new(arg_types, args);
+
+ let mut builder = ListBuilder::with_capacity(
+ // TODO: what's the decent default value for the capacity?
+ BinaryBuilder::with_capacity(8, WKB_MIN_PROBABLE_BYTES * 8),
+ executor.num_iterations(),
+ );
+
+ executor.execute_wkb_void(|maybe_wkb| {
+ if let Some(wkb) = maybe_wkb {
+ let geom_builder = builder.values();
+
+ match wkb.as_type() {
+ geo_traits::GeometryType::Point(point) => match point.coord() {
+ Some(coord) => {
+ write_wkb_point_from_coord(geom_builder, coord).unwrap();
+ geom_builder.append_value([]);
+ }
+ None => geom_builder.append_null(),
+ },
+ _ => todo!(),
+ }
+
+ geom_builder.finish();
+ builder.append(true);
+ } else {
+ builder.append_null();
+ }
+
+ Ok(())
+ })?;
+
+ executor.finish(Arc::new(builder.finish()))
+ }
+}
+
+fn get_geom_list_type(
+ geom_type: Option<&SedonaType>,
+) -> std::result::Result<Option<SedonaType>, datafusion_common::DataFusionError> {
+ match geom_type {
+ Some(SedonaType::Wkb(edges, crs) | SedonaType::WkbView(edges, crs)) => {
+ let geom_type = SedonaType::Wkb(edges.clone(), crs.clone());
+ let geom_list_type =
+ DataType::List(Arc::new(geom_type.to_storage_field("geom", true)?));
Review Comment:
```suggestion
DataType::List(Arc::new(geom_type.to_storage_field("item", true)?));
```
##########
rust/sedona-functions/src/st_dump.rs:
##########
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+use arrow_array::builder::{BinaryBuilder, ListBuilder};
+use arrow_schema::DataType;
+use datafusion_common::error::Result;
+use datafusion_expr::{
+ scalar_doc_sections::DOC_SECTION_OTHER, ColumnarValue, Documentation, Volatility,
+};
+use geo_traits::{CoordTrait, GeometryTrait, PointTrait};
+use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
+use sedona_geometry::{
+ error::SedonaGeometryError,
+ wkb_factory::{write_wkb_coord_trait, write_wkb_point_header, WKB_MIN_PROBABLE_BYTES},
+};
+use sedona_schema::{
+ datatypes::{SedonaType, WKB_GEOMETRY},
+ matchers::ArgMatcher,
+};
+use std::{io::Write, sync::Arc};
+
+use crate::executor::WkbExecutor;
+
+/// ST_Dump() scalar UDF
+///
+/// Native implementation to get all the points of a geometry as MULTIPOINT
+pub fn st_dump_udf() -> SedonaScalarUDF {
+ SedonaScalarUDF::new(
+ "st_dump",
+ vec![Arc::new(STDump)],
+ Volatility::Immutable,
+ Some(st_dump_doc()),
+ )
+}
+
+fn st_dump_doc() -> Documentation {
+ Documentation::builder(
+ DOC_SECTION_OTHER,
+ "Extracts the components of a geometry.",
+ "ST_Dump (geom: Geometry)",
+ )
+ .with_argument("geom", "geometry: Input geometry")
+ .with_sql_example("SELECT ST_Dump(ST_GeomFromWKT('MULTIPOINT (0 1, 2 3, 4 5)'))")
+ .build()
+}
+
+#[derive(Debug)]
+struct STDump;
+
+impl SedonaScalarKernel for STDump {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ let matcher = ArgMatcher::new(vec![ArgMatcher::is_geometry()], WKB_GEOMETRY);
+ let geom_type = matcher.match_args(args)?;
+
+ get_geom_list_type(geom_type.as_ref())
+ }
+
+ fn invoke_batch(
+ &self,
+ arg_types: &[SedonaType],
+ args: &[ColumnarValue],
+ ) -> Result<ColumnarValue> {
+ let executor = WkbExecutor::new(arg_types, args);
+
+ let mut builder = ListBuilder::with_capacity(
+ // TODO: what's the decent default value for the capacity?
+ BinaryBuilder::with_capacity(8, WKB_MIN_PROBABLE_BYTES * 8),
+ executor.num_iterations(),
+ );
+
+ executor.execute_wkb_void(|maybe_wkb| {
+ if let Some(wkb) = maybe_wkb {
+ let geom_builder = builder.values();
+
+ match wkb.as_type() {
+ geo_traits::GeometryType::Point(point) => match point.coord() {
+ Some(coord) => {
+ write_wkb_point_from_coord(geom_builder, coord).unwrap();
+ geom_builder.append_value([]);
+ }
+ None => geom_builder.append_null(),
+ },
+ _ => todo!(),
+ }
+
+ geom_builder.finish();
+ builder.append(true);
+ } else {
+ builder.append_null();
+ }
+
+ Ok(())
+ })?;
+
+ executor.finish(Arc::new(builder.finish()))
+ }
+}
+
+fn get_geom_list_type(
+ geom_type: Option<&SedonaType>,
+) -> std::result::Result<Option<SedonaType>, datafusion_common::DataFusionError> {
+ match geom_type {
+ Some(SedonaType::Wkb(edges, crs) | SedonaType::WkbView(edges, crs)) => {
+ let geom_type = SedonaType::Wkb(edges.clone(), crs.clone());
+ let geom_list_type =
+ DataType::List(Arc::new(geom_type.to_storage_field("geom", true)?));
+ Ok(Some(SedonaType::Arrow(geom_list_type)))
+ }
+ _ => Ok(None),
+ }
+}
+
+fn write_wkb_point_from_coord(
+ buf: &mut impl Write,
+ coord: impl CoordTrait<T = f64>,
+) -> Result<(), SedonaGeometryError> {
+ write_wkb_point_header(buf, coord.dim())?;
+ write_wkb_coord_trait(buf, &coord)
+}
+
+#[cfg(test)]
+mod tests {
+ use datafusion_expr::ScalarUDF;
+ use rstest::rstest;
+ use sedona_schema::datatypes::WKB_VIEW_GEOMETRY;
+ use sedona_testing::{
+ compare::assert_array_equal, create::create_array, testers::ScalarUdfTester,
+ };
+
+ use super::*;
+
+ #[test]
+ fn udf_metadata() {
+ let st_dump_udf: ScalarUDF = st_dump_udf().into();
+ assert_eq!(st_dump_udf.name(), "st_dump");
+ assert!(st_dump_udf.documentation().is_some());
+ }
+
+ #[rstest]
+ fn udf(#[values(WKB_GEOMETRY, WKB_VIEW_GEOMETRY)] sedona_type: SedonaType) {
+ let tester = ScalarUdfTester::new(st_dump_udf().into(), vec![sedona_type.clone()]);
+
+ let input = create_array(&[Some("POINT (1 2)")], &sedona_type);
+
+ let expected = create_array(&[Some("POINT (1 2)")], &WKB_GEOMETRY);
Review Comment:
You're right that this will be pretty verbose to test. This is a unique one but luckily byte-for-byte equality shouldn't be an issue. You can probably use `create_array()` here and assemble a `ListArray` using offsets, then use `assert_eq!()`. The failure message will be terrible but it will get the job done.
##########
rust/sedona-functions/src/st_dump.rs:
##########
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+use arrow_array::builder::{BinaryBuilder, ListBuilder};
+use arrow_schema::DataType;
+use datafusion_common::error::Result;
+use datafusion_expr::{
+ scalar_doc_sections::DOC_SECTION_OTHER, ColumnarValue, Documentation, Volatility,
+};
+use geo_traits::{CoordTrait, GeometryTrait, PointTrait};
+use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
+use sedona_geometry::{
+ error::SedonaGeometryError,
+ wkb_factory::{write_wkb_coord_trait, write_wkb_point_header, WKB_MIN_PROBABLE_BYTES},
+};
+use sedona_schema::{
+ datatypes::{SedonaType, WKB_GEOMETRY},
+ matchers::ArgMatcher,
+};
+use std::{io::Write, sync::Arc};
+
+use crate::executor::WkbExecutor;
+
+/// ST_Dump() scalar UDF
+///
+/// Native implementation to get all the points of a geometry as MULTIPOINT
+pub fn st_dump_udf() -> SedonaScalarUDF {
+ SedonaScalarUDF::new(
+ "st_dump",
+ vec![Arc::new(STDump)],
+ Volatility::Immutable,
+ Some(st_dump_doc()),
+ )
+}
+
+fn st_dump_doc() -> Documentation {
+ Documentation::builder(
+ DOC_SECTION_OTHER,
+ "Extracts the components of a geometry.",
+ "ST_Dump (geom: Geometry)",
+ )
+ .with_argument("geom", "geometry: Input geometry")
+ .with_sql_example("SELECT ST_Dump(ST_GeomFromWKT('MULTIPOINT (0 1, 2 3, 4 5)'))")
+ .build()
+}
+
+#[derive(Debug)]
+struct STDump;
+
+impl SedonaScalarKernel for STDump {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ let matcher = ArgMatcher::new(vec![ArgMatcher::is_geometry()], WKB_GEOMETRY);
+ let geom_type = matcher.match_args(args)?;
+
+ get_geom_list_type(geom_type.as_ref())
+ }
+
+ fn invoke_batch(
+ &self,
+ arg_types: &[SedonaType],
+ args: &[ColumnarValue],
+ ) -> Result<ColumnarValue> {
+ let executor = WkbExecutor::new(arg_types, args);
+
+ let mut builder = ListBuilder::with_capacity(
+ // TODO: what's the decent default value for the capacity?
+ BinaryBuilder::with_capacity(8, WKB_MIN_PROBABLE_BYTES * 8),
Review Comment:
`WKB_MIN_PROBABLE_BYTES * executor.num_iterations()` is probably fine. This will be very close to the input binary array in capacity and size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement ST_Dump [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #269:
URL: https://github.com/apache/sedona-db/pull/269#discussion_r2491374723
##########
rust/sedona-functions/src/st_dump.rs:
##########
@@ -0,0 +1,433 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+use arrow_array::{
+ builder::{BinaryBuilder, ListBuilder, StructBuilder, UInt32Builder},
+ ListArray,
+};
+use arrow_schema::{DataType, Field, Fields};
+use datafusion_common::error::Result;
+use datafusion_expr::{
+ scalar_doc_sections::DOC_SECTION_OTHER, ColumnarValue, Documentation, Volatility,
+};
+use geo_traits::{
+ GeometryCollectionTrait, GeometryTrait, GeometryType, MultiLineStringTrait, MultiPointTrait,
+ MultiPolygonTrait,
+};
+use sedona_common::sedona_internal_err;
+use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
+use sedona_geometry::wkb_factory::WKB_MIN_PROBABLE_BYTES;
+use sedona_schema::{
+ datatypes::{SedonaType, WKB_GEOMETRY},
+ matchers::ArgMatcher,
+};
+use std::sync::Arc;
+
+use crate::executor::WkbExecutor;
+
+/// ST_Dump() scalar UDF
+///
+/// Native implementation to get all the points of a geometry as MULTIPOINT
+pub fn st_dump_udf() -> SedonaScalarUDF {
+ SedonaScalarUDF::new(
+ "st_dump",
+ vec![Arc::new(STDump)],
+ Volatility::Immutable,
+ Some(st_dump_doc()),
+ )
+}
+
+fn st_dump_doc() -> Documentation {
+ Documentation::builder(
+ DOC_SECTION_OTHER,
+ "Extracts the components of a geometry.",
+ "ST_Dump (geom: Geometry)",
+ )
+ .with_argument("geom", "geometry: Input geometry")
+ .with_sql_example("SELECT ST_Dump(ST_GeomFromWKT('MULTIPOINT (0 1, 2 3, 4 5)'))")
+ .build()
+}
+
+#[derive(Debug)]
+struct STDump;
+
+// This enum is solely for passing the subset of wkb geometry to STDumpStructBuilder.
+// Maybe we can pass the underlying raw WKB bytes directly, but this just works for now.
+enum SingleWkb<'a> {
+ Point(&'a wkb::reader::Point<'a>),
+ LineString(&'a wkb::reader::LineString<'a>),
+ Polygon(&'a wkb::reader::Polygon<'a>),
+}
+
+// A builder for a single struct of { path: [u32], geom: POINT | LINESTRING | POLYGON }
+struct STDumpStructBuilder<'a> {
+ struct_builder: &'a mut StructBuilder,
+}
+
+// A builder for a list of the structs
+struct STDumpBuilder {
+ builder: ListBuilder<StructBuilder>,
+}
+
+impl<'a> STDumpStructBuilder<'a> {
+ // This appends both path and geom at once.
+ fn append(
+ &mut self,
+ parent_path: &[u32],
+ cur_index: Option<u32>,
+ wkb: SingleWkb<'_>,
+ ) -> Result<()> {
+ let path_builder = self
+ .struct_builder
+ .field_builder::<ListBuilder<UInt32Builder>>(0)
+ .unwrap();
Review Comment:
I am not sure it matters here given that you've nicely demonstrated that this is pretty fast, but we have tried in other places to avoid downcasting in a loop. Something like:
```rust
struct STDumpBuilder {
path_array_builder: UInt32Builder,
path_array_offsets_builder: OffsetBufferBuilder,
geom_builder: BinaryBuilder,
}
```
...and then implement a `finish() -> StructArray` that assembles at the end.
(optional!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement ST_Dump [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot merged PR #269:
URL: https://github.com/apache/sedona-db/pull/269
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement ST_Dump [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #269:
URL: https://github.com/apache/sedona-db/pull/269#discussion_r2485196162
##########
rust/sedona-functions/src/st_dump.rs:
##########
@@ -0,0 +1,424 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+use arrow_array::{
+ builder::{BinaryBuilder, ListBuilder, StructBuilder, UInt32Builder},
+ ListArray,
+};
+use arrow_schema::{DataType, Field, Fields};
+use datafusion_common::error::Result;
+use datafusion_expr::{
+ scalar_doc_sections::DOC_SECTION_OTHER, ColumnarValue, Documentation, Volatility,
+};
+use geo_traits::{
+ GeometryCollectionTrait, GeometryTrait, GeometryType, MultiLineStringTrait, MultiPointTrait,
+ MultiPolygonTrait,
+};
+use sedona_common::sedona_internal_err;
+use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
+use sedona_geometry::wkb_factory::WKB_MIN_PROBABLE_BYTES;
+use sedona_schema::{
+ datatypes::{SedonaType, WKB_GEOMETRY},
+ matchers::ArgMatcher,
+};
+use std::sync::Arc;
+
+use crate::executor::WkbExecutor;
+
+/// ST_Dump() scalar UDF
+///
+/// Native implementation to get all the points of a geometry as MULTIPOINT
+pub fn st_dump_udf() -> SedonaScalarUDF {
+ SedonaScalarUDF::new(
+ "st_dump",
+ vec![Arc::new(STDump)],
+ Volatility::Immutable,
+ Some(st_dump_doc()),
+ )
+}
+
+fn st_dump_doc() -> Documentation {
+ Documentation::builder(
+ DOC_SECTION_OTHER,
+ "Extracts the components of a geometry.",
+ "ST_Dump (geom: Geometry)",
+ )
+ .with_argument("geom", "geometry: Input geometry")
+ .with_sql_example("SELECT ST_Dump(ST_GeomFromWKT('MULTIPOINT (0 1, 2 3, 4 5)'))")
+ .build()
+}
+
+#[derive(Debug)]
+struct STDump;
+
+// This enum is solely for passing the subset of wkb geometry to STDumpStructBuilder.
+// Maybe we can pass the underlying raw WKB bytes directly, but this just works for now.
+enum SingleWkb<'a> {
+ Point(&'a wkb::reader::Point<'a>),
+ LineString(&'a wkb::reader::LineString<'a>),
+ Polygon(&'a wkb::reader::Polygon<'a>),
+}
+
+// A builder for a single struct of { path: [u32], geom: POINT | LINESTRING | POLYGON }
+struct STDumpStructBuilder<'a> {
+ struct_builder: &'a mut StructBuilder,
+}
+
+// A builder for a list of the structs
+struct STDumpBuilder {
+ builder: ListBuilder<StructBuilder>,
+}
+
+impl<'a> STDumpStructBuilder<'a> {
+ // This appends both path and geom at once.
+ fn append(
+ &mut self,
+ parent_path: &[u32],
+ cur_index: Option<u32>,
+ wkb: SingleWkb<'_>,
+ ) -> Result<()> {
+ let path_builder = self
+ .struct_builder
+ .field_builder::<ListBuilder<UInt32Builder>>(0)
+ .unwrap();
+
+ let path_array_builder = path_builder.values();
+ path_array_builder.append_slice(parent_path);
+ if let Some(cur_index) = cur_index {
+ path_array_builder.append_value(cur_index);
+ }
+ path_builder.append(true);
+
+ let geom_builder = self
+ .struct_builder
+ .field_builder::<BinaryBuilder>(1)
+ .unwrap();
+
+ let write_result = match wkb {
+ SingleWkb::Point(point) => {
+ wkb::writer::write_point(geom_builder, &point, &Default::default())
+ }
+ SingleWkb::LineString(line_string) => {
+ wkb::writer::write_line_string(geom_builder, &line_string, &Default::default())
+ }
+ SingleWkb::Polygon(polygon) => {
+ wkb::writer::write_polygon(geom_builder, &polygon, &Default::default())
+ }
+ };
+ if let Err(e) = write_result {
+ return sedona_internal_err!("Failed to write WKB: {e}");
+ }
+
+ geom_builder.append_value([]);
+
+ self.struct_builder.append(true);
+
+ Ok(())
+ }
+}
+
+impl STDumpBuilder {
+ fn new(num_iter: usize) -> Self {
+ let path_builder =
+ ListBuilder::with_capacity(UInt32Builder::with_capacity(num_iter), num_iter);
+ let geom_builder =
+ BinaryBuilder::with_capacity(num_iter, WKB_MIN_PROBABLE_BYTES * num_iter);
+ let struct_builder = StructBuilder::new(
+ geometry_dump_fields(),
+ vec![Box::new(path_builder), Box::new(geom_builder)],
+ );
+ let builder = ListBuilder::with_capacity(struct_builder, WKB_MIN_PROBABLE_BYTES * num_iter);
+
+ Self { builder }
+ }
+
+ fn append(&mut self, is_valid: bool) {
+ self.builder.append(is_valid);
+ }
+
+ fn append_null(&mut self) {
+ self.builder.append_null();
+ }
+
+ fn struct_builder<'a>(&'a mut self) -> STDumpStructBuilder<'a> {
+ STDumpStructBuilder {
+ struct_builder: self.builder.values(),
+ }
+ }
+
+ fn finish(&mut self) -> ListArray {
+ self.builder.finish()
+ }
+}
+
+impl SedonaScalarKernel for STDump {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ let matcher = ArgMatcher::new(vec![ArgMatcher::is_geometry()], geometry_dump_type());
+ matcher.match_args(args)
+ }
+
+ fn invoke_batch(
+ &self,
+ arg_types: &[SedonaType],
+ args: &[ColumnarValue],
+ ) -> Result<ColumnarValue> {
+ let executor = WkbExecutor::new(arg_types, args);
+
+ let mut builder = STDumpBuilder::new(executor.num_iterations());
+
+ executor.execute_wkb_void(|maybe_wkb| {
+ if let Some(wkb) = maybe_wkb {
+ let mut struct_builder = builder.struct_builder();
+
+ let mut cur_path: Vec<u32> = Vec::new();
Review Comment:
It probably doesn't make much of a difference here but you could declare this outside the loop and reset the size to zero to reuse any previous heap allocation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement ST_Dump [sedona-db]
Posted by "yutannihilation (via GitHub)" <gi...@apache.org>.
yutannihilation commented on code in PR #269:
URL: https://github.com/apache/sedona-db/pull/269#discussion_r2496547647
##########
rust/sedona-functions/src/st_dump.rs:
##########
@@ -0,0 +1,437 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+use arrow_array::{
+ builder::{BinaryBuilder, NullBufferBuilder, OffsetBufferBuilder, UInt32Builder},
+ ListArray, StructArray,
+};
+use arrow_schema::{DataType, Field, Fields};
+use datafusion_common::error::Result;
+use datafusion_expr::{
+ scalar_doc_sections::DOC_SECTION_OTHER, ColumnarValue, Documentation, Volatility,
+};
+use geo_traits::{
+ GeometryCollectionTrait, GeometryTrait, GeometryType, MultiLineStringTrait, MultiPointTrait,
+ MultiPolygonTrait,
+};
+use sedona_common::sedona_internal_err;
+use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
+use sedona_geometry::wkb_factory::WKB_MIN_PROBABLE_BYTES;
+use sedona_schema::{
+ datatypes::{SedonaType, WKB_GEOMETRY},
+ matchers::ArgMatcher,
+};
+use std::sync::Arc;
+
+use crate::executor::WkbExecutor;
+
+/// ST_Dump() scalar UDF
+///
+/// Native implementation to get all the points of a geometry as MULTIPOINT
+pub fn st_dump_udf() -> SedonaScalarUDF {
+ SedonaScalarUDF::new(
+ "st_dump",
+ vec![Arc::new(STDump)],
+ Volatility::Immutable,
+ Some(st_dump_doc()),
+ )
+}
+
+fn st_dump_doc() -> Documentation {
+ Documentation::builder(
+ DOC_SECTION_OTHER,
+ "Extracts the components of a geometry.",
+ "ST_Dump (geom: Geometry)",
+ )
+ .with_argument("geom", "geometry: Input geometry")
+ .with_sql_example("SELECT ST_Dump(ST_GeomFromWKT('MULTIPOINT (0 1, 2 3, 4 5)'))")
+ .build()
+}
+
+#[derive(Debug)]
+struct STDump;
+
+// This enum is solely for passing the subset of wkb geometry to STDumpStructBuilder.
+// Maybe we can pass the underlying raw WKB bytes directly, but this just works for now.
+enum SingleWkb<'a> {
+ Point(&'a wkb::reader::Point<'a>),
+ LineString(&'a wkb::reader::LineString<'a>),
+ Polygon(&'a wkb::reader::Polygon<'a>),
+}
+
+// A builder for a list of the structs
+struct STDumpBuilder {
+ path_array_builder: UInt32Builder,
+ path_array_offsets_builder: OffsetBufferBuilder<i32>,
+ geom_builder: BinaryBuilder,
+ struct_offsets_builder: OffsetBufferBuilder<i32>,
+ null_builder: NullBufferBuilder,
+ parent_path: Vec<u32>,
+}
+
+impl STDumpBuilder {
+ fn new(num_iter: usize) -> Self {
+ let path_array_builder = UInt32Builder::with_capacity(num_iter);
+ let path_array_offsets_builder = OffsetBufferBuilder::new(num_iter);
+ let geom_builder =
+ BinaryBuilder::with_capacity(num_iter, WKB_MIN_PROBABLE_BYTES * num_iter);
+ let struct_offsets_builder = OffsetBufferBuilder::new(num_iter);
+ let null_builder = NullBufferBuilder::new(num_iter);
+
+ Self {
+ path_array_builder,
+ path_array_offsets_builder,
+ geom_builder,
+ struct_offsets_builder,
+ null_builder,
+ parent_path: Vec::new(), // Reusable buffer to avoid allocation per row
+ }
+ }
+
+ // This appends both path and geom at once.
+ fn append_single_struct(&mut self, cur_index: Option<u32>, wkb: SingleWkb<'_>) -> Result<()> {
+ self.path_array_builder.append_slice(&self.parent_path);
+ if let Some(cur_index) = cur_index {
+ self.path_array_builder.append_value(cur_index);
+ self.path_array_offsets_builder
+ .push_length(self.parent_path.len() + 1);
+ } else {
+ self.path_array_offsets_builder
+ .push_length(self.parent_path.len());
+ }
+
+ let write_result = match wkb {
+ SingleWkb::Point(point) => {
+ wkb::writer::write_point(&mut self.geom_builder, &point, &Default::default())
+ }
+ SingleWkb::LineString(line_string) => wkb::writer::write_line_string(
+ &mut self.geom_builder,
+ &line_string,
+ &Default::default(),
+ ),
+ SingleWkb::Polygon(polygon) => {
+ wkb::writer::write_polygon(&mut self.geom_builder, &polygon, &Default::default())
+ }
+ };
+ if let Err(e) = write_result {
+ return sedona_internal_err!("Failed to write WKB: {e}");
+ }
+
+ self.geom_builder.append_value([]);
+
+ Ok(())
+ }
+
+ fn append_structs(&mut self, wkb: &wkb::reader::Wkb<'_>) -> Result<i32> {
+ match wkb.as_type() {
+ GeometryType::Point(point) => {
+ self.append_single_struct(None, SingleWkb::Point(point))?;
+ Ok(1)
+ }
+ GeometryType::LineString(line_string) => {
+ self.append_single_struct(None, SingleWkb::LineString(line_string))?;
+ Ok(1)
+ }
+ GeometryType::Polygon(polygon) => {
+ self.append_single_struct(None, SingleWkb::Polygon(polygon))?;
+ Ok(1)
+ }
+ GeometryType::MultiPoint(multi_point) => {
+ for (index, point) in multi_point.points().enumerate() {
+ self.append_single_struct(Some((index + 1) as _), SingleWkb::Point(&point))?;
+ }
+ Ok(multi_point.num_points() as _)
+ }
+ GeometryType::MultiLineString(multi_line_string) => {
+ for (index, line_string) in multi_line_string.line_strings().enumerate() {
+ self.append_single_struct(
+ Some((index + 1) as _),
+ SingleWkb::LineString(line_string),
+ )?;
+ }
+ Ok(multi_line_string.num_line_strings() as _)
+ }
+ GeometryType::MultiPolygon(multi_polygon) => {
+ for (index, polygon) in multi_polygon.polygons().enumerate() {
+ self.append_single_struct(Some((index + 1) as _), SingleWkb::Polygon(polygon))?;
+ }
+ Ok(multi_polygon.num_polygons() as _)
+ }
+ GeometryType::GeometryCollection(geometry_collection) => {
+ let mut num_geometries: i32 = 0;
+
+ self.parent_path.push(0); // add an index for the next nested level
+
+ for geometry in geometry_collection.geometries() {
+ // increment the index
+ if let Some(index) = self.parent_path.last_mut() {
+ *index += 1;
+ }
+ num_geometries += self.append_structs(geometry)?;
+ }
+
+ self.parent_path.truncate(self.parent_path.len() - 1); // clear the index before returning to the upper level
+
+ Ok(num_geometries)
+ }
+ _ => sedona_internal_err!("Invalid geometry type"),
+ }
+ }
+
+ fn append(&mut self, wkb: &wkb::reader::Wkb<'_>) -> Result<()> {
+ self.parent_path.clear();
+
+ let num_geometries = self.append_structs(wkb)?;
+ self.null_builder.append(true);
+ self.struct_offsets_builder
+ .push_length(num_geometries as usize);
+ Ok(())
+ }
+
+ fn append_null(&mut self) {
+ self.path_array_offsets_builder.push_length(0);
+ self.geom_builder.append_null();
+ self.struct_offsets_builder.push_length(1);
+ self.null_builder.append(false);
+ }
+
+ fn finish(mut self) -> ListArray {
+ let path_array = Arc::new(self.path_array_builder.finish());
+ let path_offsets = self.path_array_offsets_builder.finish();
+ let geom_array = self.geom_builder.finish();
+
+ let path_field = Arc::new(Field::new("item", DataType::UInt32, true));
+ let path_list = ListArray::new(path_field, path_offsets, path_array, None);
+
+ let fields = Fields::from(vec![
+ Field::new(
+ "path",
+ DataType::List(Arc::new(Field::new("item", DataType::UInt32, true))),
+ true,
+ ),
+ Field::new("geom", DataType::Binary, true),
+ ]);
+ let struct_array = StructArray::try_new(
+ fields.clone(),
+ vec![Arc::new(path_list), Arc::new(geom_array)],
+ None,
+ )
+ .unwrap();
+ let struct_offsets = self.struct_offsets_builder.finish();
+ let struct_field = Arc::new(Field::new("item", DataType::Struct(fields), true));
+ let nulls = self.null_builder.finish();
+ ListArray::new(struct_field, struct_offsets, Arc::new(struct_array), nulls)
+ }
+}
+
+impl SedonaScalarKernel for STDump {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ let matcher = ArgMatcher::new(vec![ArgMatcher::is_geometry()], geometry_dump_type());
Review Comment:
Ah, thanks for catching!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement ST_Dump [sedona-db]
Posted by "yutannihilation (via GitHub)" <gi...@apache.org>.
yutannihilation commented on PR #269:
URL: https://github.com/apache/sedona-db/pull/269#issuecomment-3478834789
Thanks for the Python test! I hope I can get started soon with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org