[PR] feat(rust/sedona): Spark's array(...) spelling via datafusion-spark; RS_Values parity catalog [sedona-db]

["james-willis (via GitHub)" <gi...@apache.org>]

james-willis opened a new pull request, #1293:
URL: https://github.com/apache/sedona-db/pull/1293

   Registers Spark's `array(...)` constructor spelling in SedonaDB, taken from DataFusion's own Spark-compatibility crate, and uses it to open the RS_Values parity catalog — the last RS_ function with no coverage besides RS_AsGeoTiff.
   
   ## Why the spelling was missing, and why it's safe to add back
   
   DataFusion's core renamed `array()` to `make_array()` in 2022 (apache/datafusion#3122): sqlparser-rs#566 had reserved the `ARRAY` keyword for Postgres-style `ARRAY[...]` literals, which "makes it impossible to use `array` as a function" — and Postgres itself rejects `array(1, 2)`. That parser restriction is long gone (probing showed `array(1, 2)` reaches DataFusion's *planner* today, failing only on resolution), and upstream now ships Spark's `array` as a UDF in the **datafusion-spark** compatibility crate (apache/datafusion#16932), sqllogictests included, reusing make_array's kernel and coercion.
   
   So rather than hand-rolling an alias, this adds `datafusion-spark` (54.1.0, matching the workspace) as a dependency and registers its `array` UDF in `SedonaContext::finish_new`, where both constructors funnel — every surface gets it. `[...]` literals and `make_array()` are untouched.
   
   ## Probed semantics (both engines)
   
   | case | SedonaDB | Sedona Spark |
   |---|---|---|
   | `array(1, 2, 3)`, `ARRAY(1, 2, 3)` | `[1, 2, 3]` | `[1, 2, 3]` |
   | `array(NULL, 1)` | `[None, 1]` | `[None, 1]` |
   | `array(array(1), array(2))` | `[[1], [2]]` | `[[1], [2]]` |
   | `array(1, 'x')` | cast error | cast error (parity on refusal) |
   | `array(1, 2.5)` | `[1.0, 2.5]` (float64) | decimals — Spark types the `2.5` *literal* as DECIMAL; not an array() divergence |
   | `array()` | `[]` | computes, but the harness's Arrow transport rejects the null-typed field — untestable cross-engine today |
   
   Tests: a Rust test in `context.rs` and a Python test in `test_context.py`, both asserting `array(1, 2) = [1, 2] = make_array(1, 2)`.
   
   ## RS_Values catalog (second commit)
   
   With one shared spelling available, `test_rs_values.py` can finally exist. It is a pure divergence catalog — today every spelling of RS_Values is accepted by exactly one engine:
   
   - **Array of geometries** (`RS_Values(rast, array(ST_GeomFromWKT(...), ...)[, band])`): SedonaDB has no list-of-geometries kernel; Sedona Spark answers `[255.0, 112.0]`.
   - **Bare MULTIPOINT**: SedonaDB answers `[255.0, 112.0]` (one value per part); Sedona Spark rejects a non-array geometry argument outright.
   - **Coordinate arrays** (`RS_Values(rast, array(1, 2), array(1, 1), 1)`): SedonaDB has no kernel; Sedona Spark answers `[193.0, 255.0]` — and those values pin its coordinate basis as **0-based**, unlike its own 1-based RS_PixelAs* functions.
   
   All anchor values are hand-derivable from the standard seeded grid (stated in the module docstring).
   
   Full spark-parity suite on this branch: `184 passed, 88 xfailed` locally (main's counts plus the four new xfails). Independent of #1286.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(rust/sedona): Spark's array(...) spelling via datafusion-spark; RS_Values parity catalog [sedona-db]

["james-willis (via GitHub)" <gi...@apache.org>]

james-willis commented on PR #1293:
URL: https://github.com/apache/sedona-db/pull/1293#issuecomment-5640036485

   @paleolimbot not sure if we have any reservations about using the spark compa crate


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(rust/sedona): Spark's array(...) spelling via datafusion-spark; RS_Values parity catalog [sedona-db]

["james-willis (via GitHub)" <gi...@apache.org>]

james-willis commented on PR #1293:
URL: https://github.com/apache/sedona-db/pull/1293#issuecomment-5667505105

   I raised this one because it is the particular syntax difference that I feel comes up most often.
   
   Based on Dewey's comment I am closing for now; we can come back later when spark parity is more salient


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(rust/sedona): Spark's array(...) spelling via datafusion-spark; RS_Values parity catalog [sedona-db]

["james-willis (via GitHub)" <gi...@apache.org>]

james-willis closed pull request #1293: feat(rust/sedona): Spark's array(...) spelling via datafusion-spark; RS_Values parity catalog
URL: https://github.com/apache/sedona-db/pull/1293


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org