You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "paleolimbot (via GitHub)" <gi...@apache.org> on 2025/11/24 21:29:34 UTC
[PR] feat(rust/sedona-functions): Implement ordering framework that includes geometry/geography [sedona-db]
paleolimbot opened a new pull request, #360:
URL: https://github.com/apache/sedona-db/pull/360
Work in progress!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-functions): Implement ordering framework that includes geometry/geography [sedona-db]
Posted by "jesspav (via GitHub)" <gi...@apache.org>.
jesspav commented on code in PR #360:
URL: https://github.com/apache/sedona-db/pull/360#discussion_r2566187398
##########
c/sedona-proj/src/st_transform.rs:
##########
@@ -293,7 +293,7 @@ fn invoke_scalar(wkb: &Wkb, trans: &dyn CrsTransform, builder: &mut BinaryBuilde
fn parse_source_crs(source_type: &SedonaType) -> Result<Option<String>> {
match source_type {
SedonaType::Wkb(_, Some(crs)) | SedonaType::WkbView(_, Some(crs)) => {
- crs.to_authority_code()
+ Ok(Some(crs.to_crs_string()))
Review Comment:
In `st_transform` it would have err'd with `Source CRS is required when transforming to a CRS`, unless somebody had explicitly specified lenient. in either case it's a good fix. thanks.
##########
rust/sedona-schema/src/crs.rs:
##########
@@ -97,10 +97,37 @@ impl PartialEq<dyn CoordinateReferenceSystem + Send + Sync>
/// A trait defining the minimum required properties of a concrete coordinate
/// reference system, allowing the details of this to be implemented elsewhere.
pub trait CoordinateReferenceSystem: Debug {
+ /// Compute the representation of this Crs in the form required for JSON output
+ ///
+ /// The output must be valid JSON (e.g., arbitrary strings must be quoted).
fn to_json(&self) -> String;
+
+ /// Compute the representation of this Crs as a string in the form Authority:Code
+ ///
+ /// If there is no such representation, returns None.
fn to_authority_code(&self) -> Result<Option<String>>;
+
+ /// Compute CRS equality
+ ///
+ /// CRS equality is a relatively thorny topic and can be difficult to compute;
+ /// however, this method should try to compare self and other on value (e.g.,
+ /// comparing authority_code where possible).
fn crs_equals(&self, other: &dyn CoordinateReferenceSystem) -> bool;
+
+ /// Reduce this beautiful, rich CRS representation to a mere integer if possible
+ ///
+ /// For the purposes of this trait, an SRID is always equivalent to the
+ /// authority_code `"EPSG<srid>"`. Note that other SRID representations
+ /// (e.g., GeoArrow, Parquet GEOMETRY/GEOGRAPHY) do not make any guarantees
+ /// that an SRID comes from the EPSG authority.
fn srid(&self) -> Result<Option<u32>>;
+
+ /// Compute a CRS string representation
+ ///
+ /// Unlike `to_json()`, arbitrary string values returned by this method should
+ /// not be escaped. This is the representation expected as input to PROJ, GDAL,
+ /// and Parquet GEOMETRY/GEOGRAPHY representations of CRS.
+ fn to_crs_string(&self) -> String;
Review Comment:
Thanks for adding this, those escapes were annoying. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-functions): Implement ordering framework that includes geometry/geography [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #360:
URL: https://github.com/apache/sedona-db/pull/360#discussion_r2565694497
##########
c/sedona-proj/src/sd_order_lnglat.rs:
##########
@@ -0,0 +1,205 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{fmt::Debug, sync::Arc};
+
+use arrow_array::builder::UInt64Builder;
+use arrow_schema::DataType;
+use datafusion_common::{DataFusionError, Result};
+use datafusion_expr::ColumnarValue;
+use sedona_expr::scalar_udf::SedonaScalarKernel;
+use sedona_functions::executor::WkbBytesExecutor;
+use sedona_geometry::{transform::CrsEngine, wkb_header::WkbHeader};
+use sedona_schema::{crs::lnglat, datatypes::SedonaType, matchers::ArgMatcher};
+
+use crate::st_transform::with_global_proj_engine;
+
+/// Generic scalar kernel for sd_order based on the first coordinate
+/// of a geometry projected to lon/lat
+///
+/// This [SedonaScalarKernel] requires the actual function (e.g., S2, H3,
+/// or A5 cell identifier) to be provided but takes care of the extraction
+/// of the first coordinate and projecting to lon/lat space. The provided
+/// function must return a `u64`.
+pub struct OrderLngLat<F> {
+ order_fn: F,
+}
+
+impl<F: Fn((f64, f64)) -> u64> OrderLngLat<F> {
+ /// Create a new kernel from the required function type
+ pub fn new(order_fn: F) -> Self {
+ Self { order_fn }
+ }
+}
+
+impl<F: Fn((f64, f64)) -> u64> Debug for OrderLngLat<F> {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("OrderLngLat").finish()
+ }
+}
+
+impl<F: Fn((f64, f64)) -> u64> SedonaScalarKernel for OrderLngLat<F> {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ let matcher = ArgMatcher::new(
+ vec![ArgMatcher::is_geometry_or_geography()],
+ SedonaType::Arrow(DataType::UInt64),
+ );
+ matcher.match_args(args)
+ }
+
+ fn invoke_batch(
+ &self,
+ arg_types: &[SedonaType],
+ args: &[ColumnarValue],
+ ) -> Result<ColumnarValue> {
+ // Extract the source CRS, checking for lon/lat to see if we can avoid
+ // a transformation. If the CRS is missing we also skip any particular
+ // transform (although the resulting sort may not be effective).
+ let maybe_src_crs = match &arg_types[0] {
+ SedonaType::Wkb(_, maybe_crs) | SedonaType::WkbView(_, maybe_crs)
+ if maybe_crs != &lnglat() =>
+ {
+ maybe_crs.as_ref().map(|crs| crs.to_crs_string())
+ }
+ _ => None,
Review Comment:
That's a good question...EMPTY sorts to `u64::MAX`, and NULLs sort to NULL, which is potentially configurable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-functions): Implement ordering framework that includes geometry/geography [sedona-db]
Posted by "jesspav (via GitHub)" <gi...@apache.org>.
jesspav commented on code in PR #360:
URL: https://github.com/apache/sedona-db/pull/360#discussion_r2566209846
##########
rust/sedona-functions/src/sd_order.rs:
##########
@@ -0,0 +1,118 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use datafusion_common::Result;
+use datafusion_expr::{
+ scalar_doc_sections::DOC_SECTION_OTHER, ColumnarValue, Documentation, Volatility,
+};
+use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
+use sedona_schema::datatypes::SedonaType;
+use std::{fmt::Debug, sync::Arc};
+
+/// SD_Order() scalar UDF implementation
+///
+/// This function is invoked to obtain a proxy array whose order may be used
+/// to sort based on the value. The default implementation returns the value
+/// and a utility is provided to order geometry and/or geographies based on
+/// the first coordinate. More sophisticated sorting (e.g., XZ2) may be added
+/// in the future.
+pub fn sd_order_udf() -> SedonaScalarUDF {
Review Comment:
Maybe it's just me, but this feels a little awkward. I would definitely expect the default ordering of `sd_order` to have meaning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-functions): Implement ordering framework that includes geometry/geography [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #360:
URL: https://github.com/apache/sedona-db/pull/360#discussion_r2561027328
##########
c/sedona-geos/src/st_minimumclearance_line.rs:
##########
@@ -113,7 +113,6 @@ mod tests {
Some("POLYGON((0 0,0 3,3 3,3 0,0 0),(1 1,1 2,2 2,2 1,1 1))"),
Some("POLYGON((0 0,0 1,0 1,1 1,1 0,0 0,0 0))"),
Some("LINESTRING (0 0, 1 1, 2 2)"),
- Some("MULTIPOLYGON(((0.5 0.5,0 0,0 1,0.5 0.5)),((0.5 0.5,1 1,1 0,0.5 0.5)),((2.5 2.5,2 2,2 3,2.5 2.5)),((2.5 2.5,3 3,3 2,2.5 2.5)))"),
Review Comment:
We already have a multipolygon test case here and this test fails for me locally (we've also removed these from Python)
##########
c/sedona-proj/src/sd_order_lnglat.rs:
##########
@@ -0,0 +1,205 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{fmt::Debug, sync::Arc};
+
+use arrow_array::builder::UInt64Builder;
+use arrow_schema::DataType;
+use datafusion_common::{DataFusionError, Result};
+use datafusion_expr::ColumnarValue;
+use sedona_expr::scalar_udf::SedonaScalarKernel;
+use sedona_functions::executor::WkbBytesExecutor;
+use sedona_geometry::{transform::CrsEngine, wkb_header::WkbHeader};
+use sedona_schema::{crs::lnglat, datatypes::SedonaType, matchers::ArgMatcher};
+
+use crate::st_transform::with_global_proj_engine;
+
+/// Generic scalar kernel for sd_order based on the first coordinate
+/// of a geometry projected to lon/lat
+///
+/// This [SedonaScalarKernel] requires the actual function (e.g., S2, H3,
+/// or A5 cell identifier) to be provided but takes care of the extraction
+/// of the first coordinate and projecting to lon/lat space. The provided
+/// function must return a `u64`.
+pub struct OrderLngLat<F> {
+ order_fn: F,
+}
Review Comment:
This is the actual ordering implementation from geometry/geography. It's a little awkward because we have to pull in two somewhat disjoint dependencies (something to project to lon/lat, something to compute a value). The projecting needs the global engine so I put it here and parameterized it on the thing that's easier to mock (the actual ordering function).
There are definitely more effective orderings that consider the full geometry (e.g., xz2); however, this one is good as a default because it's fast (doesn't parse or reproject more than one coordinate) and works for geometry, geography, and the forthcoming row-level CRS type. Adding other orderable hashes (e.g., geohash, xz2) is harder because the compatibility with existing behaviour could take some time to replicate.
##########
rust/sedona-functions/src/sd_order.rs:
##########
@@ -0,0 +1,118 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use datafusion_common::Result;
+use datafusion_expr::{
+ scalar_doc_sections::DOC_SECTION_OTHER, ColumnarValue, Documentation, Volatility,
+};
+use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
+use sedona_schema::datatypes::SedonaType;
+use std::{fmt::Debug, sync::Arc};
+
+/// SD_Order() scalar UDF implementation
+///
+/// This function is invoked to obtain a proxy array whose order may be used
+/// to sort based on the value. The default implementation returns the value
+/// and a utility is provided to order geometry and/or geographies based on
+/// the first coordinate. More sophisticated sorting (e.g., XZ2) may be added
+/// in the future.
+pub fn sd_order_udf() -> SedonaScalarUDF {
Review Comment:
This is the main change: we add `sd_order()` whose registered kernels are the means by which to order stuff (that is possibly not meaningfully orderable with the default implementation).
##########
rust/sedona-schema/src/crs.rs:
##########
@@ -97,10 +97,37 @@ impl PartialEq<dyn CoordinateReferenceSystem + Send + Sync>
/// A trait defining the minimum required properties of a concrete coordinate
/// reference system, allowing the details of this to be implemented elsewhere.
pub trait CoordinateReferenceSystem: Debug {
+ /// Compute the representation of this Crs in the form required for JSON output
+ ///
+ /// The output must be valid JSON (e.g., arbitrary strings must be quoted).
fn to_json(&self) -> String;
Review Comment:
I noticed these weren't documented so I added some
##########
c/sedona-proj/src/st_transform.rs:
##########
@@ -293,7 +293,7 @@ fn invoke_scalar(wkb: &Wkb, trans: &dyn CrsTransform, builder: &mut BinaryBuilde
fn parse_source_crs(source_type: &SedonaType) -> Result<Option<String>> {
match source_type {
SedonaType::Wkb(_, Some(crs)) | SedonaType::WkbView(_, Some(crs)) => {
- crs.to_authority_code()
+ Ok(Some(crs.to_crs_string()))
Review Comment:
This fixes a bug in `st_transform()`: previously any CRS that was unrepresentable with a single authority/code (e.g., the ns-water test data) would have returned `None` here. Because we have `lenient`, `st_transform()` would have assumed the source data to be lon/lat even though it was correctly annotated.
##########
rust/sedona-functions/src/sd_order.rs:
##########
@@ -0,0 +1,118 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use datafusion_common::Result;
+use datafusion_expr::{
+ scalar_doc_sections::DOC_SECTION_OTHER, ColumnarValue, Documentation, Volatility,
+};
+use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
+use sedona_schema::datatypes::SedonaType;
+use std::{fmt::Debug, sync::Arc};
+
+/// SD_Order() scalar UDF implementation
+///
+/// This function is invoked to obtain a proxy array whose order may be used
+/// to sort based on the value. The default implementation returns the value
+/// and a utility is provided to order geometry and/or geographies based on
+/// the first coordinate. More sophisticated sorting (e.g., XZ2) may be added
+/// in the future.
+pub fn sd_order_udf() -> SedonaScalarUDF {
+ SedonaScalarUDF::new(
+ "sd_order",
+ vec![Arc::new(SDOrderDefault {})],
+ Volatility::Immutable,
+ Some(sd_order_doc()),
+ )
+}
+
+fn sd_order_doc() -> Documentation {
+ Documentation::builder(
+ DOC_SECTION_OTHER,
+ "Return an arbitrary value that may be used to sort the input.",
+ "SD_Order (value: Any)",
+ )
+ .with_argument("value", "Any: An arbitrary value")
+ .with_sql_example("SELECT SD_Order()")
+ .build()
+}
+
+/// Default implementation that returns its input (i.e., by default, just
+/// do whatever DataFusion would have done with the value)
+#[derive(Debug)]
+struct SDOrderDefault {}
+
+impl SedonaScalarKernel for SDOrderDefault {
Review Comment:
...where, by default, we just fall back on the storage ordering
##########
rust/sedona-schema/src/crs.rs:
##########
@@ -97,10 +97,37 @@ impl PartialEq<dyn CoordinateReferenceSystem + Send + Sync>
/// A trait defining the minimum required properties of a concrete coordinate
/// reference system, allowing the details of this to be implemented elsewhere.
pub trait CoordinateReferenceSystem: Debug {
+ /// Compute the representation of this Crs in the form required for JSON output
+ ///
+ /// The output must be valid JSON (e.g., arbitrary strings must be quoted).
fn to_json(&self) -> String;
+
+ /// Compute the representation of this Crs as a string in the form Authority:Code
+ ///
+ /// If there is no such representation, returns None.
fn to_authority_code(&self) -> Result<Option<String>>;
+
+ /// Compute CRS equality
+ ///
+ /// CRS equality is a relatively thorny topic and can be difficult to compute;
+ /// however, this method should try to compare self and other on value (e.g.,
+ /// comparing authority_code where possible).
fn crs_equals(&self, other: &dyn CoordinateReferenceSystem) -> bool;
+
+ /// Reduce this beautiful, rich CRS representation to a mere integer if possible
+ ///
+ /// For the purposes of this trait, an SRID is always equivalent to the
+ /// authority_code `"EPSG<srid>"`. Note that other SRID representations
+ /// (e.g., GeoArrow, Parquet GEOMETRY/GEOGRAPHY) do not make any guarantees
+ /// that an SRID comes from the EPSG authority.
fn srid(&self) -> Result<Option<u32>>;
+
+ /// Compute a CRS string representation
+ ///
+ /// Unlike `to_json()`, arbitrary string values returned by this method should
+ /// not be escaped. This is the representation expected as input to PROJ, GDAL,
+ /// and Parquet GEOMETRY/GEOGRAPHY representations of CRS.
+ fn to_crs_string(&self) -> String;
Review Comment:
I added this one because we need it for Parquet GEOMETRY/GEOGRAPHY too (in addition to throwing this Crs object into PROJ).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-functions): Implement ordering framework that includes geometry/geography [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #360:
URL: https://github.com/apache/sedona-db/pull/360#issuecomment-3583022832
> Why do we call it SD_Order? If this is a UDF, why not ST?
> Can we add docs together with the PR if possible?
I opened https://github.com/apache/sedona-db/issues/370 to clarify the full vision here / enumerate the follow-ups...`sd_order()` not really meant to be called by the user (it's designed to be inserted by an optimizer rule.). In general I'd also prefer not to invent `st_*()` functions (perhaps some `sd_` functions that have proved their utility and stability can graduate 🙂 ).
The docs are inline in the Rust function (when we have time to do https://github.com/apache/sedona-db/issues/180 they will get rendered).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-functions): Implement ordering framework that includes geometry/geography [sedona-db]
Posted by "jesspav (via GitHub)" <gi...@apache.org>.
jesspav commented on code in PR #360:
URL: https://github.com/apache/sedona-db/pull/360#discussion_r2566082205
##########
c/sedona-proj/src/st_transform.rs:
##########
@@ -293,7 +293,7 @@ fn invoke_scalar(wkb: &Wkb, trans: &dyn CrsTransform, builder: &mut BinaryBuilde
fn parse_source_crs(source_type: &SedonaType) -> Result<Option<String>> {
match source_type {
SedonaType::Wkb(_, Some(crs)) | SedonaType::WkbView(_, Some(crs)) => {
- crs.to_authority_code()
+ Ok(Some(crs.to_crs_string()))
Review Comment:
Returning some here makes sense. Lenient is false by default, so it should not be relevant unless it is being specified. I'll double check the code path here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-functions): Implement ordering framework that includes geometry/geography [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #360:
URL: https://github.com/apache/sedona-db/pull/360#discussion_r2566322044
##########
rust/sedona-functions/src/sd_order.rs:
##########
@@ -0,0 +1,118 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use datafusion_common::Result;
+use datafusion_expr::{
+ scalar_doc_sections::DOC_SECTION_OTHER, ColumnarValue, Documentation, Volatility,
+};
+use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
+use sedona_schema::datatypes::SedonaType;
+use std::{fmt::Debug, sync::Arc};
+
+/// SD_Order() scalar UDF implementation
+///
+/// This function is invoked to obtain a proxy array whose order may be used
+/// to sort based on the value. The default implementation returns the value
+/// and a utility is provided to order geometry and/or geographies based on
+/// the first coordinate. More sophisticated sorting (e.g., XZ2) may be added
+/// in the future.
+pub fn sd_order_udf() -> SedonaScalarUDF {
Review Comment:
That's a great point...I added https://github.com/apache/sedona-db/issues/370 for some improvements. For the most part the ordering that will happen in our release builds for most input will be meaningful (exceptions: `None` crs, missing s2geography, missing proj config) but there are some ways we can improve this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-functions): Implement ordering framework that includes geometry/geography [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #360:
URL: https://github.com/apache/sedona-db/pull/360#discussion_r2561118837
##########
c/sedona-proj/src/st_transform.rs:
##########
@@ -293,7 +293,7 @@ fn invoke_scalar(wkb: &Wkb, trans: &dyn CrsTransform, builder: &mut BinaryBuilde
fn parse_source_crs(source_type: &SedonaType) -> Result<Option<String>> {
match source_type {
SedonaType::Wkb(_, Some(crs)) | SedonaType::WkbView(_, Some(crs)) => {
- crs.to_authority_code()
+ Ok(Some(crs.to_crs_string()))
Review Comment:
cc @jesspav (did I get this part right!?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-functions): Implement ordering framework that includes geometry/geography [sedona-db]
Posted by "zhangfengcdt (via GitHub)" <gi...@apache.org>.
zhangfengcdt commented on code in PR #360:
URL: https://github.com/apache/sedona-db/pull/360#discussion_r2565666575
##########
c/sedona-proj/src/sd_order_lnglat.rs:
##########
@@ -0,0 +1,205 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{fmt::Debug, sync::Arc};
+
+use arrow_array::builder::UInt64Builder;
+use arrow_schema::DataType;
+use datafusion_common::{DataFusionError, Result};
+use datafusion_expr::ColumnarValue;
+use sedona_expr::scalar_udf::SedonaScalarKernel;
+use sedona_functions::executor::WkbBytesExecutor;
+use sedona_geometry::{transform::CrsEngine, wkb_header::WkbHeader};
+use sedona_schema::{crs::lnglat, datatypes::SedonaType, matchers::ArgMatcher};
+
+use crate::st_transform::with_global_proj_engine;
+
+/// Generic scalar kernel for sd_order based on the first coordinate
+/// of a geometry projected to lon/lat
+///
+/// This [SedonaScalarKernel] requires the actual function (e.g., S2, H3,
+/// or A5 cell identifier) to be provided but takes care of the extraction
+/// of the first coordinate and projecting to lon/lat space. The provided
+/// function must return a `u64`.
+pub struct OrderLngLat<F> {
+ order_fn: F,
+}
+
+impl<F: Fn((f64, f64)) -> u64> OrderLngLat<F> {
+ /// Create a new kernel from the required function type
+ pub fn new(order_fn: F) -> Self {
+ Self { order_fn }
+ }
+}
+
+impl<F: Fn((f64, f64)) -> u64> Debug for OrderLngLat<F> {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("OrderLngLat").finish()
+ }
+}
+
+impl<F: Fn((f64, f64)) -> u64> SedonaScalarKernel for OrderLngLat<F> {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ let matcher = ArgMatcher::new(
+ vec![ArgMatcher::is_geometry_or_geography()],
+ SedonaType::Arrow(DataType::UInt64),
+ );
+ matcher.match_args(args)
+ }
+
+ fn invoke_batch(
+ &self,
+ arg_types: &[SedonaType],
+ args: &[ColumnarValue],
+ ) -> Result<ColumnarValue> {
+ // Extract the source CRS, checking for lon/lat to see if we can avoid
+ // a transformation. If the CRS is missing we also skip any particular
+ // transform (although the resulting sort may not be effective).
+ let maybe_src_crs = match &arg_types[0] {
+ SedonaType::Wkb(_, maybe_crs) | SedonaType::WkbView(_, maybe_crs)
+ if maybe_crs != &lnglat() =>
+ {
+ maybe_crs.as_ref().map(|crs| crs.to_crs_string())
+ }
+ _ => None,
Review Comment:
Do empty and null both sort to the same sort position?
##########
c/sedona-geos/src/st_minimumclearance_line.rs:
##########
@@ -113,7 +113,6 @@ mod tests {
Some("POLYGON((0 0,0 3,3 3,3 0,0 0),(1 1,1 2,2 2,2 1,1 1))"),
Some("POLYGON((0 0,0 1,0 1,1 1,1 0,0 0,0 0))"),
Some("LINESTRING (0 0, 1 1, 2 2)"),
- Some("MULTIPOLYGON(((0.5 0.5,0 0,0 1,0.5 0.5)),((0.5 0.5,1 1,1 0,0.5 0.5)),((2.5 2.5,2 2,2 3,2.5 2.5)),((2.5 2.5,3 3,3 2,2.5 2.5)))"),
Review Comment:
I think i've fixed the python function test by removing the ambiguity of the result:
```
(
"MULTIPOLYGON(((0.5 0.5,0 0,0 1,0.5 0.5)),((0.5 0.5,1 1,1 0,0.5 0.5)),((2.5 2.5,2 2,2 3,2.5 2.5)),((2.5 2.5,3.5 3.5,3.5 1.5,2.5 2.5)))",
"LINESTRING (2.5 2.5, 2 2.5)",
),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-functions): Implement ordering framework that includes geometry/geography [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #360:
URL: https://github.com/apache/sedona-db/pull/360#discussion_r2566322044
##########
rust/sedona-functions/src/sd_order.rs:
##########
@@ -0,0 +1,118 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use datafusion_common::Result;
+use datafusion_expr::{
+ scalar_doc_sections::DOC_SECTION_OTHER, ColumnarValue, Documentation, Volatility,
+};
+use sedona_expr::scalar_udf::{SedonaScalarKernel, SedonaScalarUDF};
+use sedona_schema::datatypes::SedonaType;
+use std::{fmt::Debug, sync::Arc};
+
+/// SD_Order() scalar UDF implementation
+///
+/// This function is invoked to obtain a proxy array whose order may be used
+/// to sort based on the value. The default implementation returns the value
+/// and a utility is provided to order geometry and/or geographies based on
+/// the first coordinate. More sophisticated sorting (e.g., XZ2) may be added
+/// in the future.
+pub fn sd_order_udf() -> SedonaScalarUDF {
Review Comment:
That's a great point...I added https://github.com/apache/sedona-db/issues/370 for some improvements. For the most part the ordering that will happen in our release builds for most input will be meaningful (exceptions: `None` crs, missing s2geography, missing proj config).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-functions): Implement ordering framework that includes geometry/geography [sedona-db]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #360:
URL: https://github.com/apache/sedona-db/pull/360#discussion_r2561123703
##########
rust/sedona-schema/src/crs.rs:
##########
@@ -97,10 +97,37 @@ impl PartialEq<dyn CoordinateReferenceSystem + Send + Sync>
/// A trait defining the minimum required properties of a concrete coordinate
/// reference system, allowing the details of this to be implemented elsewhere.
pub trait CoordinateReferenceSystem: Debug {
+ /// Compute the representation of this Crs in the form required for JSON output
+ ///
+ /// The output must be valid JSON (e.g., arbitrary strings must be quoted).
fn to_json(&self) -> String;
+
+ /// Compute the representation of this Crs as a string in the form Authority:Code
+ ///
+ /// If there is no such representation, returns None.
fn to_authority_code(&self) -> Result<Option<String>>;
+
+ /// Compute CRS equality
+ ///
+ /// CRS equality is a relatively thorny topic and can be difficult to compute;
+ /// however, this method should try to compare self and other on value (e.g.,
+ /// comparing authority_code where possible).
fn crs_equals(&self, other: &dyn CoordinateReferenceSystem) -> bool;
+
+ /// Reduce this beautiful, rich CRS representation to a mere integer if possible
Review Comment:
[nitpick] The comment contains subjective language ('beautiful, rich', 'mere'). Consider using more neutral, technical language such as 'Convert this CRS representation to an integer SRID if possible'.
```suggestion
/// Convert this CRS representation to an integer SRID if possible.
```
##########
c/sedona-proj/src/sd_order_lnglat.rs:
##########
@@ -0,0 +1,205 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::{fmt::Debug, sync::Arc};
+
+use arrow_array::builder::UInt64Builder;
+use arrow_schema::DataType;
+use datafusion_common::{DataFusionError, Result};
+use datafusion_expr::ColumnarValue;
+use sedona_expr::scalar_udf::SedonaScalarKernel;
+use sedona_functions::executor::WkbBytesExecutor;
+use sedona_geometry::{transform::CrsEngine, wkb_header::WkbHeader};
+use sedona_schema::{crs::lnglat, datatypes::SedonaType, matchers::ArgMatcher};
+
+use crate::st_transform::with_global_proj_engine;
+
+/// Generic scalar kernel for sd_order based on the first coordinate
+/// of a geometry projected to lon/lat
+///
+/// This [SedonaScalarKernel] requires the actual function (e.g., S2, H3,
+/// or A5 cell identifier) to be provided but takes care of the extraction
+/// of the first coordinate and projecting to lon/lat space. The provided
+/// function must return a `u64`.
+pub struct OrderLngLat<F> {
+ order_fn: F,
+}
+
+impl<F: Fn((f64, f64)) -> u64> OrderLngLat<F> {
+ /// Create a new kernel from the required function type
+ pub fn new(order_fn: F) -> Self {
+ Self { order_fn }
+ }
+}
+
+impl<F: Fn((f64, f64)) -> u64> Debug for OrderLngLat<F> {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("OrderLngLat").finish()
+ }
+}
+
+impl<F: Fn((f64, f64)) -> u64> SedonaScalarKernel for OrderLngLat<F> {
+ fn return_type(&self, args: &[SedonaType]) -> Result<Option<SedonaType>> {
+ let matcher = ArgMatcher::new(
+ vec![ArgMatcher::is_geometry_or_geography()],
+ SedonaType::Arrow(DataType::UInt64),
+ );
+ matcher.match_args(args)
+ }
+
+ fn invoke_batch(
+ &self,
+ arg_types: &[SedonaType],
+ args: &[ColumnarValue],
+ ) -> Result<ColumnarValue> {
+ // Extract the source CRS, checking for lon/lat to see if we can avoid
+ // a transformation. If the CRS is missing we also skip any particular
+ // transform (although the resulting sort may not be effective).
+ let maybe_src_crs = match &arg_types[0] {
+ SedonaType::Wkb(_, maybe_crs) | SedonaType::WkbView(_, maybe_crs)
+ if maybe_crs != &lnglat() =>
Review Comment:
The condition `maybe_crs != &lnglat()` calls `lnglat()` on every iteration. Consider storing the result of `lnglat()` in a variable before the loop to avoid repeated allocations.
```suggestion
let lnglat_crs = lnglat();
let maybe_src_crs = match &arg_types[0] {
SedonaType::Wkb(_, maybe_crs) | SedonaType::WkbView(_, maybe_crs)
if maybe_crs != &lnglat_crs =>
```
##########
rust/sedona-schema/src/crs.rs:
##########
@@ -97,10 +97,37 @@ impl PartialEq<dyn CoordinateReferenceSystem + Send + Sync>
/// A trait defining the minimum required properties of a concrete coordinate
/// reference system, allowing the details of this to be implemented elsewhere.
pub trait CoordinateReferenceSystem: Debug {
+ /// Compute the representation of this Crs in the form required for JSON output
+ ///
+ /// The output must be valid JSON (e.g., arbitrary strings must be quoted).
fn to_json(&self) -> String;
+
+ /// Compute the representation of this Crs as a string in the form Authority:Code
+ ///
+ /// If there is no such representation, returns None.
fn to_authority_code(&self) -> Result<Option<String>>;
+
+ /// Compute CRS equality
+ ///
+ /// CRS equality is a relatively thorny topic and can be difficult to compute;
+ /// however, this method should try to compare self and other on value (e.g.,
+ /// comparing authority_code where possible).
fn crs_equals(&self, other: &dyn CoordinateReferenceSystem) -> bool;
+
+ /// Reduce this beautiful, rich CRS representation to a mere integer if possible
+ ///
+ /// For the purposes of this trait, an SRID is always equivalent to the
+ /// authority_code `"EPSG<srid>"`. Note that other SRID representations
Review Comment:
Corrected formatting: should be `\"EPSG:{srid}\"` to match the actual format shown in line 267.
```suggestion
/// authority_code `"EPSG:{srid}"`. Note that other SRID representations
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-functions): Implement ordering framework that includes geometry/geography [sedona-db]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #360:
URL: https://github.com/apache/sedona-db/pull/360#issuecomment-3582814116
1. Why do we call it SD_Order? If this is a UDF, why not ST?
2. Can we add docs together with the PR if possible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-functions): Implement ordering framework that includes geometry/geography [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #360:
URL: https://github.com/apache/sedona-db/pull/360#discussion_r2566249055
##########
c/sedona-proj/src/st_transform.rs:
##########
@@ -293,7 +293,7 @@ fn invoke_scalar(wkb: &Wkb, trans: &dyn CrsTransform, builder: &mut BinaryBuilde
fn parse_source_crs(source_type: &SedonaType) -> Result<Option<String>> {
match source_type {
SedonaType::Wkb(_, Some(crs)) | SedonaType::WkbView(_, Some(crs)) => {
- crs.to_authority_code()
+ Ok(Some(crs.to_crs_string()))
Review Comment:
Thanks for investigating! I'm glad the failure mode here is an error 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust/sedona-functions): Implement ordering framework that includes geometry/geography [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot merged PR #360:
URL: https://github.com/apache/sedona-db/pull/360
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org