You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "james-willis (via GitHub)" <gi...@apache.org> on 2026/05/21 21:32:47 UTC

[PR] feat(sedonadb): `sd_read_zarr` UDTF via datafusion-ffi plugin handoff [sedona-db]

james-willis opened a new pull request, #866:
URL: https://github.com/apache/sedona-db/pull/866

   **Draft — stacked on top of #858. Don't review until #858 lands.**
   
   Restores the SQL UDTF surface for the Zarr plugin that was stripped from #858 per maintainer guidance ("we don't have a `read_parquet()` either"). Tracked as DB-34.
   
   ## What this adds
   
   - `sd_read_zarr` SQL UDTF — `SELECT * FROM sd_read_zarr('file:///path/to/foo.zarr')` works alongside the existing `con.read_format(ZarrFormatSpec(), uri)` path.
   - Cross-cdylib UDTF handoff via `datafusion-ffi`'s `FFI_TableFunction` capsule contract (the same wire format `datafusion-python` uses). Plugin and host don't share Rust types directly — all data flows through Arrow C Stream + the ABI-stable FFI structs.
   - Host-side session/codec capsule machinery in `python/sedonadb/src/plugin.rs`. `InternalContext::register_udtf_capsule(name, spec)` builds the codec capsule, calls the plugin's `__datafusion_table_function__(session)`, and converts the returned capsule into an `Arc<dyn TableFunctionImpl>`.
   - Plugin-side `ZarrTableFunction` PyO3 class with `__datafusion_table_function__`, plus `register(con)` Python helper.
   
   ## Why this is its own PR
   
   The UDTF surface required ~1000 lines of plumbing (FFI helpers, codec capsules, `TaskContextProvider` lifetime adapter, etc.) for an ergonomic-but-not-essential surface. Splitting lets the zarr feature land without that complexity and gives the plugin UDTF pattern its own focused review — including whether we want to generalize it (a `con.register_table_function(name, spec)` Python helper that any plugin can use).
   
   ## Status
   
   - Branch: \`jw/sd-read-zarr-udtf\`
   - All commits originally on #858 through \`8e44f572\`, then the UDTF strip on #858 is at \`16eafd41\`.
   - Tests pass locally (7 plugin tests including the UDTF smoke + dtype matrix).
   - Will need rebase onto main after #858 merges.
   
   ## Open design questions for the eventual review
   
   1. Should there be a generic `con.register_table_function(name, spec)` so future plugins (\`sedonadb-cog\`, \`sedonadb-icechunk\`) get the UDTF pattern for free, instead of each one calling \`_impl.register_udtf_capsule\`?
   2. Naming: \`sd_read_zarr\` matches \`sd_random_geometry\` in this repo; \`zarr_scan\` would match \`delta_scan\`/\`iceberg_scan\` from the wider DataFusion ecosystem.
   3. \`load_eager\` is intentionally absent here too — the eager-fetch path lands with the async \`RS_EnsureLoaded\` resolver.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(sedonadb): `sd_read_zarr` UDTF via datafusion-ffi plugin handoff [sedona-db]

Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on PR #866:
URL: https://github.com/apache/sedona-db/pull/866#issuecomment-4522020088

   yeah this is a parking lot PR- far down the stack and I think we can take out time to get it right as a general solution for UTF/UFTF/optimizer rules etc.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(sedonadb): `sd_read_zarr` UDTF via datafusion-ffi plugin handoff [sedona-db]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #866:
URL: https://github.com/apache/sedona-db/pull/866#issuecomment-4514903064

   We can chat about this later too, but we may be able to avoid the table function FFI here by exposing in the `sedona-datasource` crate a generic table function wrapper around *any* ExternalFormatSpec. Or, alternatively, any Python object that exposes `__arrow_c_array_stream__` (we have a generic table provider for a single-shot `RecordBatchReader` that powers `sd.create_data_frame()`, but we can back up slightly and store the object whose `__arrow_c_array_stream__()` call produces one too).
   
   Getting a good extension system that can use DataFusion's FFI is great too...DataFusion's FFI just exposes *everything* and I'm wary of bringing in all of that until we really need the bells and whistles.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(sedonadb): `sd_read_zarr` UDTF via datafusion-ffi plugin handoff [sedona-db]

Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis closed pull request #866: feat(sedonadb): `sd_read_zarr` UDTF via datafusion-ffi plugin handoff
URL: https://github.com/apache/sedona-db/pull/866


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org