You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "Abeeujah (via GitHub)" <gi...@apache.org> on 2025/11/10 18:18:44 UTC
[PR] ST_Simplify [sedona-db]
Abeeujah opened a new pull request, #295:
URL: https://github.com/apache/sedona-db/pull/295
Having read both Issue #244 and PR #243 I'm quite interested in knowing or having the test cases that SedonaDB and PostGIS diverge, before I proceed to do a custom implementation, but at the moment, It seems to look fine on my end;
Everything is up for review.
closes #244
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/sedona-geos): Implement ST_Simplify [sedona-db]
Posted by "Abeeujah (via GitHub)" <gi...@apache.org>.
Abeeujah commented on code in PR #295:
URL: https://github.com/apache/sedona-db/pull/295#discussion_r2516854063
##########
c/sedona-geos/src/st_simplify.rs:
##########
@@ -0,0 +1,604 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::builder::BinaryBuilder;
+use arrow_schema::DataType;
+use datafusion_common::{cast::as_float64_array, DataFusionError, Result};
+use datafusion_expr::ColumnarValue;
+use geos::{Geom, Geometry, GeometryTypes};
+use sedona_expr::scalar_udf::{ScalarKernelRef, SedonaScalarKernel};
+use sedona_geometry::wkb_factory::WKB_MIN_PROBABLE_BYTES;
+use sedona_schema::{
+ datatypes::{SedonaType, WKB_GEOMETRY},
+ matchers::ArgMatcher,
+};
+
+use crate::executor::GeosExecutor;
+
+pub fn st_simplify_impl() -> ScalarKernelRef {
+ Arc::new(STSimplify {})
+}
+
+#[derive(Debug)]
+struct STSimplify {}
+
+impl SedonaScalarKernel for STSimplify {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ let matcher = ArgMatcher::new(
+ vec![ArgMatcher::is_geometry(), ArgMatcher::is_numeric()],
+ WKB_GEOMETRY,
+ );
+
+ matcher.match_args(args)
+ }
+
+ fn invoke_batch(
+ &self,
+ arg_types: &[SedonaType],
+ args: &[ColumnarValue],
+ ) -> Result<ColumnarValue> {
+ let executor = GeosExecutor::new(arg_types, args);
+
+ let tolerance_value = args[1]
+ .cast_to(&DataType::Float64, None)?
+ .to_array(executor.num_iterations())?;
+ let tolerance_array = as_float64_array(&tolerance_value)?;
+ let mut tolerance_iter = tolerance_array.iter();
+
+ let mut builder = BinaryBuilder::with_capacity(
+ executor.num_iterations(),
+ WKB_MIN_PROBABLE_BYTES * executor.num_iterations(),
+ );
+
+ executor.execute_wkb_void(|wkb| {
+ match (wkb, tolerance_iter.next().unwrap()) {
+ (Some(wkb), Some(tolerance)) => {
+ invoke_scalar(&wkb, tolerance, &mut builder)?;
+ builder.append_value([]);
+ }
+ _ => builder.append_null(),
+ }
+ Ok(())
+ })?;
+
+ executor.finish(Arc::new(builder.finish()))
+ }
+}
+
+fn invoke_scalar(
+ geos_geom: &geos::Geometry,
+ tolerance: f64,
+ writer: &mut impl std::io::Write,
+) -> Result<()> {
+ let initial_type = geos_geom.geometry_type();
+ let geometry = geos_geom
+ .simplify(tolerance)
+ .map_err(|e| DataFusionError::Execution(format!("Failed to simplify geometry: {e}")))?;
+
+ let geometry = match (initial_type, geometry.geometry_type()) {
+ (GeometryTypes::MultiPolygon, GeometryTypes::Polygon) => {
+ Geometry::create_multipolygon(vec![geometry]).map_err(|e| {
+ DataFusionError::Execution(format!("Failed to revert geometry promotion: {e}"))
+ })?
+ }
+ (GeometryTypes::MultiLineString, GeometryTypes::LineString) => {
+ Geometry::create_multiline_string(vec![geometry]).map_err(|e| {
+ DataFusionError::Execution(format!("Failed to revert geometry promotion: {e}"))
+ })?
+ }
+ _ => geometry,
+ };
+
+ let wkb = geometry
+ .to_wkb()
+ .map_err(|e| DataFusionError::Execution(format!("Failed to convert to wkb: {e}")))?;
+
+ writer.write_all(wkb.as_ref())?;
+ Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use arrow_array::Float64Array;
+ use datafusion_common::ScalarValue;
+ use rstest::rstest;
+ use sedona_expr::scalar_udf::SedonaScalarUDF;
+ use sedona_schema::datatypes::{WKB_GEOMETRY, WKB_VIEW_GEOMETRY};
+ use sedona_testing::{
+ compare::assert_array_equal, create::create_array, testers::ScalarUdfTester,
+ };
+
+ #[rstest]
+ fn udf(#[values(WKB_GEOMETRY, WKB_VIEW_GEOMETRY)] sedona_type: SedonaType) {
Review Comment:
> Thank you for all of these test cases! Are there any sources we should acknowledge for them? (May be useful to update our own tests at some point if there are sources you consulted!)
The implementation is based on the Spec defined in [POSTGIS](https://postgis.net/docs/ST_Simplify.html) I run the test cases specified there manually on the docker container, then proceed to assert in code that the behavior is compliant.
Another place I consult for the test cases is Geos test suite [ST_Simplify](https://github.com/libgeos/geos/tree/main/tests/unit/simplify)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/sedona-geos): Implement ST_Simplify [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #295:
URL: https://github.com/apache/sedona-db/pull/295#discussion_r2518292353
##########
c/sedona-geos/src/st_simplify.rs:
##########
@@ -0,0 +1,604 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::builder::BinaryBuilder;
+use arrow_schema::DataType;
+use datafusion_common::{cast::as_float64_array, DataFusionError, Result};
+use datafusion_expr::ColumnarValue;
+use geos::{Geom, Geometry, GeometryTypes};
+use sedona_expr::scalar_udf::{ScalarKernelRef, SedonaScalarKernel};
+use sedona_geometry::wkb_factory::WKB_MIN_PROBABLE_BYTES;
+use sedona_schema::{
+ datatypes::{SedonaType, WKB_GEOMETRY},
+ matchers::ArgMatcher,
+};
+
+use crate::executor::GeosExecutor;
+
+pub fn st_simplify_impl() -> ScalarKernelRef {
+ Arc::new(STSimplify {})
+}
+
+#[derive(Debug)]
+struct STSimplify {}
+
+impl SedonaScalarKernel for STSimplify {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ let matcher = ArgMatcher::new(
+ vec![ArgMatcher::is_geometry(), ArgMatcher::is_numeric()],
+ WKB_GEOMETRY,
+ );
+
+ matcher.match_args(args)
+ }
+
+ fn invoke_batch(
+ &self,
+ arg_types: &[SedonaType],
+ args: &[ColumnarValue],
+ ) -> Result<ColumnarValue> {
+ let executor = GeosExecutor::new(arg_types, args);
+
+ let tolerance_value = args[1]
+ .cast_to(&DataType::Float64, None)?
+ .to_array(executor.num_iterations())?;
+ let tolerance_array = as_float64_array(&tolerance_value)?;
+ let mut tolerance_iter = tolerance_array.iter();
+
+ let mut builder = BinaryBuilder::with_capacity(
+ executor.num_iterations(),
+ WKB_MIN_PROBABLE_BYTES * executor.num_iterations(),
+ );
+
+ executor.execute_wkb_void(|wkb| {
+ match (wkb, tolerance_iter.next().unwrap()) {
+ (Some(wkb), Some(tolerance)) => {
+ invoke_scalar(&wkb, tolerance, &mut builder)?;
+ builder.append_value([]);
+ }
+ _ => builder.append_null(),
+ }
+ Ok(())
+ })?;
+
+ executor.finish(Arc::new(builder.finish()))
+ }
+}
+
+fn invoke_scalar(
+ geos_geom: &geos::Geometry,
+ tolerance: f64,
+ writer: &mut impl std::io::Write,
+) -> Result<()> {
+ let initial_type = geos_geom.geometry_type();
+ let geometry = geos_geom
+ .simplify(tolerance)
+ .map_err(|e| DataFusionError::Execution(format!("Failed to simplify geometry: {e}")))?;
+
+ let geometry = match (initial_type, geometry.geometry_type()) {
+ (GeometryTypes::MultiPolygon, GeometryTypes::Polygon) => {
+ Geometry::create_multipolygon(vec![geometry]).map_err(|e| {
+ DataFusionError::Execution(format!("Failed to revert geometry promotion: {e}"))
+ })?
+ }
+ (GeometryTypes::MultiLineString, GeometryTypes::LineString) => {
+ Geometry::create_multiline_string(vec![geometry]).map_err(|e| {
+ DataFusionError::Execution(format!("Failed to revert geometry promotion: {e}"))
+ })?
+ }
+ _ => geometry,
+ };
+
+ let wkb = geometry
+ .to_wkb()
+ .map_err(|e| DataFusionError::Execution(format!("Failed to convert to wkb: {e}")))?;
+
+ writer.write_all(wkb.as_ref())?;
+ Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use arrow_array::Float64Array;
+ use datafusion_common::ScalarValue;
+ use rstest::rstest;
+ use sedona_expr::scalar_udf::SedonaScalarUDF;
+ use sedona_schema::datatypes::{WKB_GEOMETRY, WKB_VIEW_GEOMETRY};
+ use sedona_testing::{
+ compare::assert_array_equal, create::create_array, testers::ScalarUdfTester,
+ };
+
+ #[rstest]
+ fn udf(#[values(WKB_GEOMETRY, WKB_VIEW_GEOMETRY)] sedona_type: SedonaType) {
Review Comment:
It's useful for us to have the Rust tests because we'd like to implement a faster version at some point, and when we have Rust code we'll need the Rust tests to cover the behaviour. (But yes, the minimum requirement for wrapping external library behaviour is a "is it plugged in" check at the Rust level).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/sedona-geos): Implement ST_Simplify [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #295:
URL: https://github.com/apache/sedona-db/pull/295#discussion_r2515448010
##########
c/sedona-geos/src/st_simplify.rs:
##########
@@ -0,0 +1,604 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::builder::BinaryBuilder;
+use arrow_schema::DataType;
+use datafusion_common::{cast::as_float64_array, DataFusionError, Result};
+use datafusion_expr::ColumnarValue;
+use geos::{Geom, Geometry, GeometryTypes};
+use sedona_expr::scalar_udf::{ScalarKernelRef, SedonaScalarKernel};
+use sedona_geometry::wkb_factory::WKB_MIN_PROBABLE_BYTES;
+use sedona_schema::{
+ datatypes::{SedonaType, WKB_GEOMETRY},
+ matchers::ArgMatcher,
+};
+
+use crate::executor::GeosExecutor;
+
+pub fn st_simplify_impl() -> ScalarKernelRef {
+ Arc::new(STSimplify {})
+}
+
+#[derive(Debug)]
+struct STSimplify {}
+
+impl SedonaScalarKernel for STSimplify {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ let matcher = ArgMatcher::new(
+ vec![ArgMatcher::is_geometry(), ArgMatcher::is_numeric()],
+ WKB_GEOMETRY,
+ );
+
+ matcher.match_args(args)
+ }
+
+ fn invoke_batch(
+ &self,
+ arg_types: &[SedonaType],
+ args: &[ColumnarValue],
+ ) -> Result<ColumnarValue> {
+ let executor = GeosExecutor::new(arg_types, args);
+
+ let tolerance_value = args[1]
+ .cast_to(&DataType::Float64, None)?
+ .to_array(executor.num_iterations())?;
+ let tolerance_array = as_float64_array(&tolerance_value)?;
+ let mut tolerance_iter = tolerance_array.iter();
+
+ let mut builder = BinaryBuilder::with_capacity(
+ executor.num_iterations(),
+ WKB_MIN_PROBABLE_BYTES * executor.num_iterations(),
+ );
+
+ executor.execute_wkb_void(|wkb| {
+ match (wkb, tolerance_iter.next().unwrap()) {
+ (Some(wkb), Some(tolerance)) => {
+ invoke_scalar(&wkb, tolerance, &mut builder)?;
+ builder.append_value([]);
+ }
+ _ => builder.append_null(),
+ }
+ Ok(())
+ })?;
+
+ executor.finish(Arc::new(builder.finish()))
+ }
+}
+
+fn invoke_scalar(
+ geos_geom: &geos::Geometry,
+ tolerance: f64,
+ writer: &mut impl std::io::Write,
+) -> Result<()> {
+ let initial_type = geos_geom.geometry_type();
+ let geometry = geos_geom
+ .simplify(tolerance)
+ .map_err(|e| DataFusionError::Execution(format!("Failed to simplify geometry: {e}")))?;
+
+ let geometry = match (initial_type, geometry.geometry_type()) {
Review Comment:
```suggestion
// GEOS simplifies MultiPolygon/MultiLineString to Polygon/Linestring; however, we
// attempt to keep the original geometry type for PostGIS compatibility.
let geometry = match (initial_type, geometry.geometry_type()) {
```
##########
c/sedona-geos/src/st_simplify.rs:
##########
@@ -0,0 +1,604 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::builder::BinaryBuilder;
+use arrow_schema::DataType;
+use datafusion_common::{cast::as_float64_array, DataFusionError, Result};
+use datafusion_expr::ColumnarValue;
+use geos::{Geom, Geometry, GeometryTypes};
+use sedona_expr::scalar_udf::{ScalarKernelRef, SedonaScalarKernel};
+use sedona_geometry::wkb_factory::WKB_MIN_PROBABLE_BYTES;
+use sedona_schema::{
+ datatypes::{SedonaType, WKB_GEOMETRY},
+ matchers::ArgMatcher,
+};
+
+use crate::executor::GeosExecutor;
+
+pub fn st_simplify_impl() -> ScalarKernelRef {
+ Arc::new(STSimplify {})
+}
+
+#[derive(Debug)]
+struct STSimplify {}
+
+impl SedonaScalarKernel for STSimplify {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ let matcher = ArgMatcher::new(
+ vec![ArgMatcher::is_geometry(), ArgMatcher::is_numeric()],
+ WKB_GEOMETRY,
+ );
+
+ matcher.match_args(args)
+ }
+
+ fn invoke_batch(
+ &self,
+ arg_types: &[SedonaType],
+ args: &[ColumnarValue],
+ ) -> Result<ColumnarValue> {
+ let executor = GeosExecutor::new(arg_types, args);
+
+ let tolerance_value = args[1]
+ .cast_to(&DataType::Float64, None)?
+ .to_array(executor.num_iterations())?;
+ let tolerance_array = as_float64_array(&tolerance_value)?;
+ let mut tolerance_iter = tolerance_array.iter();
+
+ let mut builder = BinaryBuilder::with_capacity(
+ executor.num_iterations(),
+ WKB_MIN_PROBABLE_BYTES * executor.num_iterations(),
+ );
+
+ executor.execute_wkb_void(|wkb| {
+ match (wkb, tolerance_iter.next().unwrap()) {
+ (Some(wkb), Some(tolerance)) => {
+ invoke_scalar(&wkb, tolerance, &mut builder)?;
+ builder.append_value([]);
+ }
+ _ => builder.append_null(),
+ }
+ Ok(())
+ })?;
+
+ executor.finish(Arc::new(builder.finish()))
+ }
+}
+
+fn invoke_scalar(
+ geos_geom: &geos::Geometry,
+ tolerance: f64,
+ writer: &mut impl std::io::Write,
+) -> Result<()> {
+ let initial_type = geos_geom.geometry_type();
+ let geometry = geos_geom
+ .simplify(tolerance)
+ .map_err(|e| DataFusionError::Execution(format!("Failed to simplify geometry: {e}")))?;
+
+ let geometry = match (initial_type, geometry.geometry_type()) {
+ (GeometryTypes::MultiPolygon, GeometryTypes::Polygon) => {
+ Geometry::create_multipolygon(vec![geometry]).map_err(|e| {
+ DataFusionError::Execution(format!("Failed to revert geometry promotion: {e}"))
+ })?
+ }
+ (GeometryTypes::MultiLineString, GeometryTypes::LineString) => {
+ Geometry::create_multiline_string(vec![geometry]).map_err(|e| {
+ DataFusionError::Execution(format!("Failed to revert geometry promotion: {e}"))
+ })?
+ }
+ _ => geometry,
+ };
+
+ let wkb = geometry
+ .to_wkb()
+ .map_err(|e| DataFusionError::Execution(format!("Failed to convert to wkb: {e}")))?;
+
+ writer.write_all(wkb.as_ref())?;
+ Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use arrow_array::Float64Array;
+ use datafusion_common::ScalarValue;
+ use rstest::rstest;
+ use sedona_expr::scalar_udf::SedonaScalarUDF;
+ use sedona_schema::datatypes::{WKB_GEOMETRY, WKB_VIEW_GEOMETRY};
+ use sedona_testing::{
+ compare::assert_array_equal, create::create_array, testers::ScalarUdfTester,
+ };
+
+ #[rstest]
+ fn udf(#[values(WKB_GEOMETRY, WKB_VIEW_GEOMETRY)] sedona_type: SedonaType) {
Review Comment:
Thank you for all of these test cases! Are there any sources we should acknowledge for them? (May be useful to update our own tests at some point if there are sources you consulted!)
##########
c/sedona-geos/src/st_simplify.rs:
##########
@@ -0,0 +1,604 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::builder::BinaryBuilder;
+use arrow_schema::DataType;
+use datafusion_common::{cast::as_float64_array, DataFusionError, Result};
+use datafusion_expr::ColumnarValue;
+use geos::{Geom, Geometry, GeometryTypes};
+use sedona_expr::scalar_udf::{ScalarKernelRef, SedonaScalarKernel};
+use sedona_geometry::wkb_factory::WKB_MIN_PROBABLE_BYTES;
+use sedona_schema::{
+ datatypes::{SedonaType, WKB_GEOMETRY},
+ matchers::ArgMatcher,
+};
+
+use crate::executor::GeosExecutor;
+
+pub fn st_simplify_impl() -> ScalarKernelRef {
+ Arc::new(STSimplify {})
+}
+
+#[derive(Debug)]
+struct STSimplify {}
+
+impl SedonaScalarKernel for STSimplify {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ let matcher = ArgMatcher::new(
+ vec![ArgMatcher::is_geometry(), ArgMatcher::is_numeric()],
+ WKB_GEOMETRY,
+ );
+
+ matcher.match_args(args)
+ }
+
+ fn invoke_batch(
+ &self,
+ arg_types: &[SedonaType],
+ args: &[ColumnarValue],
+ ) -> Result<ColumnarValue> {
+ let executor = GeosExecutor::new(arg_types, args);
+
+ let tolerance_value = args[1]
+ .cast_to(&DataType::Float64, None)?
+ .to_array(executor.num_iterations())?;
+ let tolerance_array = as_float64_array(&tolerance_value)?;
+ let mut tolerance_iter = tolerance_array.iter();
+
+ let mut builder = BinaryBuilder::with_capacity(
+ executor.num_iterations(),
+ WKB_MIN_PROBABLE_BYTES * executor.num_iterations(),
+ );
+
+ executor.execute_wkb_void(|wkb| {
+ match (wkb, tolerance_iter.next().unwrap()) {
+ (Some(wkb), Some(tolerance)) => {
+ invoke_scalar(&wkb, tolerance, &mut builder)?;
+ builder.append_value([]);
+ }
+ _ => builder.append_null(),
+ }
+ Ok(())
+ })?;
+
+ executor.finish(Arc::new(builder.finish()))
+ }
+}
+
+fn invoke_scalar(
+ geos_geom: &geos::Geometry,
+ tolerance: f64,
+ writer: &mut impl std::io::Write,
+) -> Result<()> {
+ let initial_type = geos_geom.geometry_type();
+ let geometry = geos_geom
+ .simplify(tolerance)
+ .map_err(|e| DataFusionError::Execution(format!("Failed to simplify geometry: {e}")))?;
+
+ let geometry = match (initial_type, geometry.geometry_type()) {
Review Comment:
optional: It might be nice to explain why we need the wrapper!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/sedona-geos): Implement ST_Simplify [sedona-db]
Posted by "Abeeujah (via GitHub)" <gi...@apache.org>.
Abeeujah commented on code in PR #295:
URL: https://github.com/apache/sedona-db/pull/295#discussion_r2516882394
##########
c/sedona-geos/src/st_simplify.rs:
##########
@@ -0,0 +1,604 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::builder::BinaryBuilder;
+use arrow_schema::DataType;
+use datafusion_common::{cast::as_float64_array, DataFusionError, Result};
+use datafusion_expr::ColumnarValue;
+use geos::{Geom, Geometry, GeometryTypes};
+use sedona_expr::scalar_udf::{ScalarKernelRef, SedonaScalarKernel};
+use sedona_geometry::wkb_factory::WKB_MIN_PROBABLE_BYTES;
+use sedona_schema::{
+ datatypes::{SedonaType, WKB_GEOMETRY},
+ matchers::ArgMatcher,
+};
+
+use crate::executor::GeosExecutor;
+
+pub fn st_simplify_impl() -> ScalarKernelRef {
+ Arc::new(STSimplify {})
+}
+
+#[derive(Debug)]
+struct STSimplify {}
+
+impl SedonaScalarKernel for STSimplify {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ let matcher = ArgMatcher::new(
+ vec![ArgMatcher::is_geometry(), ArgMatcher::is_numeric()],
+ WKB_GEOMETRY,
+ );
+
+ matcher.match_args(args)
+ }
+
+ fn invoke_batch(
+ &self,
+ arg_types: &[SedonaType],
+ args: &[ColumnarValue],
+ ) -> Result<ColumnarValue> {
+ let executor = GeosExecutor::new(arg_types, args);
+
+ let tolerance_value = args[1]
+ .cast_to(&DataType::Float64, None)?
+ .to_array(executor.num_iterations())?;
+ let tolerance_array = as_float64_array(&tolerance_value)?;
+ let mut tolerance_iter = tolerance_array.iter();
+
+ let mut builder = BinaryBuilder::with_capacity(
+ executor.num_iterations(),
+ WKB_MIN_PROBABLE_BYTES * executor.num_iterations(),
+ );
+
+ executor.execute_wkb_void(|wkb| {
+ match (wkb, tolerance_iter.next().unwrap()) {
+ (Some(wkb), Some(tolerance)) => {
+ invoke_scalar(&wkb, tolerance, &mut builder)?;
+ builder.append_value([]);
+ }
+ _ => builder.append_null(),
+ }
+ Ok(())
+ })?;
+
+ executor.finish(Arc::new(builder.finish()))
+ }
+}
+
+fn invoke_scalar(
+ geos_geom: &geos::Geometry,
+ tolerance: f64,
+ writer: &mut impl std::io::Write,
+) -> Result<()> {
+ let initial_type = geos_geom.geometry_type();
+ let geometry = geos_geom
+ .simplify(tolerance)
+ .map_err(|e| DataFusionError::Execution(format!("Failed to simplify geometry: {e}")))?;
+
+ let geometry = match (initial_type, geometry.geometry_type()) {
+ (GeometryTypes::MultiPolygon, GeometryTypes::Polygon) => {
+ Geometry::create_multipolygon(vec![geometry]).map_err(|e| {
+ DataFusionError::Execution(format!("Failed to revert geometry promotion: {e}"))
+ })?
+ }
+ (GeometryTypes::MultiLineString, GeometryTypes::LineString) => {
+ Geometry::create_multiline_string(vec![geometry]).map_err(|e| {
+ DataFusionError::Execution(format!("Failed to revert geometry promotion: {e}"))
+ })?
+ }
+ _ => geometry,
+ };
+
+ let wkb = geometry
+ .to_wkb()
+ .map_err(|e| DataFusionError::Execution(format!("Failed to convert to wkb: {e}")))?;
+
+ writer.write_all(wkb.as_ref())?;
+ Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use arrow_array::Float64Array;
+ use datafusion_common::ScalarValue;
+ use rstest::rstest;
+ use sedona_expr::scalar_udf::SedonaScalarUDF;
+ use sedona_schema::datatypes::{WKB_GEOMETRY, WKB_VIEW_GEOMETRY};
+ use sedona_testing::{
+ compare::assert_array_equal, create::create_array, testers::ScalarUdfTester,
+ };
+
+ #[rstest]
+ fn udf(#[values(WKB_GEOMETRY, WKB_VIEW_GEOMETRY)] sedona_type: SedonaType) {
Review Comment:
> This is a lot of rust tests. FYI, for the future, you can be a little light on Rust tests, since they're kind of verbose. The python tests (which are very concise) is where we try to be comprehensive. e.g I'll often include a single empty geom in Rust tests, while in Python I'll test empty geoms for all types.
I find it easier to test in Rust, especially as it's strongly typed, for Instance, POSTGIS amd Sedona tend to handle `EMPTY` differently when testing with python, Sedona returns `EMPTY` while POSTGIS returns `None`, would require me including conditional checks in test function to get them to coexist happily, but Rust Serializes and Deserializes it Just fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/sedona-geos): Implement ST_Simplify [sedona-db]
Posted by "Abeeujah (via GitHub)" <gi...@apache.org>.
Abeeujah commented on code in PR #295:
URL: https://github.com/apache/sedona-db/pull/295#discussion_r2516882394
##########
c/sedona-geos/src/st_simplify.rs:
##########
@@ -0,0 +1,604 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::builder::BinaryBuilder;
+use arrow_schema::DataType;
+use datafusion_common::{cast::as_float64_array, DataFusionError, Result};
+use datafusion_expr::ColumnarValue;
+use geos::{Geom, Geometry, GeometryTypes};
+use sedona_expr::scalar_udf::{ScalarKernelRef, SedonaScalarKernel};
+use sedona_geometry::wkb_factory::WKB_MIN_PROBABLE_BYTES;
+use sedona_schema::{
+ datatypes::{SedonaType, WKB_GEOMETRY},
+ matchers::ArgMatcher,
+};
+
+use crate::executor::GeosExecutor;
+
+pub fn st_simplify_impl() -> ScalarKernelRef {
+ Arc::new(STSimplify {})
+}
+
+#[derive(Debug)]
+struct STSimplify {}
+
+impl SedonaScalarKernel for STSimplify {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ let matcher = ArgMatcher::new(
+ vec![ArgMatcher::is_geometry(), ArgMatcher::is_numeric()],
+ WKB_GEOMETRY,
+ );
+
+ matcher.match_args(args)
+ }
+
+ fn invoke_batch(
+ &self,
+ arg_types: &[SedonaType],
+ args: &[ColumnarValue],
+ ) -> Result<ColumnarValue> {
+ let executor = GeosExecutor::new(arg_types, args);
+
+ let tolerance_value = args[1]
+ .cast_to(&DataType::Float64, None)?
+ .to_array(executor.num_iterations())?;
+ let tolerance_array = as_float64_array(&tolerance_value)?;
+ let mut tolerance_iter = tolerance_array.iter();
+
+ let mut builder = BinaryBuilder::with_capacity(
+ executor.num_iterations(),
+ WKB_MIN_PROBABLE_BYTES * executor.num_iterations(),
+ );
+
+ executor.execute_wkb_void(|wkb| {
+ match (wkb, tolerance_iter.next().unwrap()) {
+ (Some(wkb), Some(tolerance)) => {
+ invoke_scalar(&wkb, tolerance, &mut builder)?;
+ builder.append_value([]);
+ }
+ _ => builder.append_null(),
+ }
+ Ok(())
+ })?;
+
+ executor.finish(Arc::new(builder.finish()))
+ }
+}
+
+fn invoke_scalar(
+ geos_geom: &geos::Geometry,
+ tolerance: f64,
+ writer: &mut impl std::io::Write,
+) -> Result<()> {
+ let initial_type = geos_geom.geometry_type();
+ let geometry = geos_geom
+ .simplify(tolerance)
+ .map_err(|e| DataFusionError::Execution(format!("Failed to simplify geometry: {e}")))?;
+
+ let geometry = match (initial_type, geometry.geometry_type()) {
+ (GeometryTypes::MultiPolygon, GeometryTypes::Polygon) => {
+ Geometry::create_multipolygon(vec![geometry]).map_err(|e| {
+ DataFusionError::Execution(format!("Failed to revert geometry promotion: {e}"))
+ })?
+ }
+ (GeometryTypes::MultiLineString, GeometryTypes::LineString) => {
+ Geometry::create_multiline_string(vec![geometry]).map_err(|e| {
+ DataFusionError::Execution(format!("Failed to revert geometry promotion: {e}"))
+ })?
+ }
+ _ => geometry,
+ };
+
+ let wkb = geometry
+ .to_wkb()
+ .map_err(|e| DataFusionError::Execution(format!("Failed to convert to wkb: {e}")))?;
+
+ writer.write_all(wkb.as_ref())?;
+ Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use arrow_array::Float64Array;
+ use datafusion_common::ScalarValue;
+ use rstest::rstest;
+ use sedona_expr::scalar_udf::SedonaScalarUDF;
+ use sedona_schema::datatypes::{WKB_GEOMETRY, WKB_VIEW_GEOMETRY};
+ use sedona_testing::{
+ compare::assert_array_equal, create::create_array, testers::ScalarUdfTester,
+ };
+
+ #[rstest]
+ fn udf(#[values(WKB_GEOMETRY, WKB_VIEW_GEOMETRY)] sedona_type: SedonaType) {
Review Comment:
> This is a lot of rust tests. FYI, for the future, you can be a little light on Rust tests, since they're kind of verbose. The python tests (which are very concise) is where we try to be comprehensive. e.g I'll often include a single empty geom in Rust tests, while in Python I'll test empty geoms for all types.
I find it easier to test in Rust, especially as it's strongly typed, for Instance, POSTGIS amd Sedona tend to handle `EMPTY` differently when testing with python, Sedona returns `EMPTY` while POSTGIS returns `None`, would require me including conditional checks in test function to get them to coexist happily, but Rust Serializes and Deserializes it Just fine.
But yeah, I'd be a little light on the Rust tests, and focus testing energy on Python's
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/sedona-geos): Implement ST_Simplify [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot merged PR #295:
URL: https://github.com/apache/sedona-db/pull/295
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(c/sedona-geos): Implement ST_Simplify [sedona-db]
Posted by "petern48 (via GitHub)" <gi...@apache.org>.
petern48 commented on code in PR #295:
URL: https://github.com/apache/sedona-db/pull/295#discussion_r2516435468
##########
c/sedona-geos/src/st_simplify.rs:
##########
@@ -0,0 +1,604 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::builder::BinaryBuilder;
+use arrow_schema::DataType;
+use datafusion_common::{cast::as_float64_array, DataFusionError, Result};
+use datafusion_expr::ColumnarValue;
+use geos::{Geom, Geometry, GeometryTypes};
+use sedona_expr::scalar_udf::{ScalarKernelRef, SedonaScalarKernel};
+use sedona_geometry::wkb_factory::WKB_MIN_PROBABLE_BYTES;
+use sedona_schema::{
+ datatypes::{SedonaType, WKB_GEOMETRY},
+ matchers::ArgMatcher,
+};
+
+use crate::executor::GeosExecutor;
+
+pub fn st_simplify_impl() -> ScalarKernelRef {
+ Arc::new(STSimplify {})
+}
+
+#[derive(Debug)]
+struct STSimplify {}
+
+impl SedonaScalarKernel for STSimplify {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ let matcher = ArgMatcher::new(
+ vec![ArgMatcher::is_geometry(), ArgMatcher::is_numeric()],
+ WKB_GEOMETRY,
+ );
+
+ matcher.match_args(args)
+ }
+
+ fn invoke_batch(
+ &self,
+ arg_types: &[SedonaType],
+ args: &[ColumnarValue],
+ ) -> Result<ColumnarValue> {
+ let executor = GeosExecutor::new(arg_types, args);
+
+ let tolerance_value = args[1]
+ .cast_to(&DataType::Float64, None)?
+ .to_array(executor.num_iterations())?;
+ let tolerance_array = as_float64_array(&tolerance_value)?;
+ let mut tolerance_iter = tolerance_array.iter();
+
+ let mut builder = BinaryBuilder::with_capacity(
+ executor.num_iterations(),
+ WKB_MIN_PROBABLE_BYTES * executor.num_iterations(),
+ );
+
+ executor.execute_wkb_void(|wkb| {
+ match (wkb, tolerance_iter.next().unwrap()) {
+ (Some(wkb), Some(tolerance)) => {
+ invoke_scalar(&wkb, tolerance, &mut builder)?;
+ builder.append_value([]);
+ }
+ _ => builder.append_null(),
+ }
+ Ok(())
+ })?;
+
+ executor.finish(Arc::new(builder.finish()))
+ }
+}
+
+fn invoke_scalar(
+ geos_geom: &geos::Geometry,
+ tolerance: f64,
+ writer: &mut impl std::io::Write,
+) -> Result<()> {
+ let initial_type = geos_geom.geometry_type();
+ let geometry = geos_geom
+ .simplify(tolerance)
+ .map_err(|e| DataFusionError::Execution(format!("Failed to simplify geometry: {e}")))?;
+
+ let geometry = match (initial_type, geometry.geometry_type()) {
+ (GeometryTypes::MultiPolygon, GeometryTypes::Polygon) => {
+ Geometry::create_multipolygon(vec![geometry]).map_err(|e| {
+ DataFusionError::Execution(format!("Failed to revert geometry promotion: {e}"))
+ })?
+ }
+ (GeometryTypes::MultiLineString, GeometryTypes::LineString) => {
+ Geometry::create_multiline_string(vec![geometry]).map_err(|e| {
+ DataFusionError::Execution(format!("Failed to revert geometry promotion: {e}"))
+ })?
+ }
+ _ => geometry,
+ };
+
+ let wkb = geometry
+ .to_wkb()
+ .map_err(|e| DataFusionError::Execution(format!("Failed to convert to wkb: {e}")))?;
+
+ writer.write_all(wkb.as_ref())?;
+ Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use arrow_array::Float64Array;
+ use datafusion_common::ScalarValue;
+ use rstest::rstest;
+ use sedona_expr::scalar_udf::SedonaScalarUDF;
+ use sedona_schema::datatypes::{WKB_GEOMETRY, WKB_VIEW_GEOMETRY};
+ use sedona_testing::{
+ compare::assert_array_equal, create::create_array, testers::ScalarUdfTester,
+ };
+
+ #[rstest]
+ fn udf(#[values(WKB_GEOMETRY, WKB_VIEW_GEOMETRY)] sedona_type: SedonaType) {
Review Comment:
This is a lot of rust tests. FYI, for the future, you can be a little light on Rust tests, since they're kind of verbose. The python tests (which are very concise) is where we try to be comprehensive. e.g I'll often include a single empty geom in Rust tests, while in Python I'll test empty geoms for all types.
There's no need to delete these. Just mentioning it for the future because I know it can be kind of annoying to write so many of these.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org