You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2026/04/01 15:01:56 UTC
[I] [DISCUSS] Adopt datafusion-functions-json and datafusion-functions-variant into core repo [datafusion]
alamb opened a new issue, #21301:
URL: https://github.com/apache/datafusion/issues/21301
# Introduction
With the rise of the importance of semi-structured data processing, I hear from more and more DataFusion users that they would like better support for JSON and [Parquet Variant](https://parquet.apache.org/docs/file-format/types/variantencoding/) functions (see [blog](https://parquet.apache.org/blog/2026/02/27/variant-type-in-apache-parquet-for-semi-structured-data/) for more details)
Today, function libraries for DataFusion for these two types live in [datafusion-contrib ](https://github.com/datafusion-contrib) rather than in the [main Apache DataFusion repository](https://github.com/apache/datafusion):
- [datafusion-variant](https://github.com/datafusion-contrib/datafusion-variant)
- [datafusion-functions-json](https://github.com/datafusion-contrib/datafusion-functions-json)
Keeping them outside the main repository has benefits such as faster iteration time and not being tied to releses,
However, it also means they are
1. outside ASF governance and release processes
2. are less discoverable to users
3. Are harder to integrate downstream (need to wait for the next major DataFusion release)
4. Feel more experimental even when they solve common problems.
5. (maybe) Harder to attract outside maintenance investment
# Proposal
Bring these two crates into the main datafusion repository, similarly to how we have done for datafusion-spark. The crates would be optional (not part of `datafusion` or a feature flag)
I think we would need the buy in from the maintainers/authors (largely pydanitic @adriangb @friendlymatthew and others)
We previously did this for Spark-compatible functions by bringing [datafusion-functions-spark](https://github.com/apache/datafusion/tree/main/datafusion/spark) into the core DataFusion repo because the functionality was widely useful and maintaining it in one place made contribution and coordination easier.
There was also recent discussion on the mailing list about using datafusion-json in the python bindings where this also came up: https://lists.apache.org/thread/f591qmhx97wsl7h5xfoh7sfhv2gh9t2k
# Alternatives you've considered
1. Keep these crates in `datafusion-contrib` indefinitely.
This keeps the core repo smaller and preserves flexibility, but leaves the crates outside the main project’s release and governance process.
2. Keep them in datafusion-contrib, but improve discoverability and documentation.
This helps users find them, but does not address governance, release coordination, or long-term maintenance.
3. Bring in only one library at a time, starting with the most mature or most widely used.
This is likely the lowest-risk path if there is agreement in principle but uncertainty about scope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-4677274640
What's the consensus on `datafusion-variant`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSS] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-4171338058
> **Summary of Major Exact Diffs**
None of these seem like real issues to me.
> * DF works on raw `text`/`str` JSON; Postgres requires `::json` or (preferred) `::jsonb`.
Yes, that's just the reality of it. Once we have proper extension types we'll have something very similar to `::json`. Variant is similar to `::jsonb`.
> * DF path syntax: variadic `*keys` (mix of `str` + `int` indices) → Postgres: `text[]` array or chained operators (indices as text strings `'0'`, `'1'`).
Our version seems more ergonomic? We could also accept a single `text[]`. Maybe we need to accept `'2'` for array indexing.
> * DF has **typed getters** (`_int`, `_float`, etc.); Postgres has **none** — always text extract + SQL cast.
Yes, this is so that we can avoid an intermediary conversion. These don't need to be user accessible, they are mostly used to rewrite `(col->'key')::int` into `json_get_int(col, 'key')` which is able to parse the json directly into an int.
> * DF `json_get_array` returns native Arrow array; Postgres returns `jsonb` (must expand manually).
We could change this, but I'd argue our version is more ergonomic and performant (avoids assembling a json array which would probably require allocating non-json array anyway).
> * DF `json_length` is unified; Postgres splits array vs object.
Again I think having 1 function is more ergonomic. We could add aliases.
> * Postgres `jsonb` is binary & indexable (GIN); DF is string-based (no native indexing mentioned).
In general indexing in postgres != indexing in DataFusion. It's not like you can just create a BTree index in DataFusion in general.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSS] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "lukekim (via GitHub)" <gi...@apache.org>.
lukekim commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-4170966028
+1 we already build these into Spice but I think the functions should be reviewed for consistency before making it official and having more people depend on them.
It aligns most closely to the PostgreSQL syntax but not exactly/completely. LLM analysis:
**Exact Function-by-Function Diff: DataFusion JSON Functions vs PostgreSQL**
(PostgreSQL 17/18 – official `json`/`jsonb` functions and operators from https://www.postgresql.org/docs/current/functions-json.html)
| PostgreSQL Exact Equivalent | DF Function / Operator | Key Differences |
|-----------------------------|------------------------|----------------------------|
| `col::jsonb ? 'key'` (top-level only) <br>or `col::jsonb @? '$."key1"."key2"[2]'` (full path via SQL/JSON path) | `json_contains(json: str, *keys: str \| int) → bool` <br>(alias: `?` operator) | DF supports full variadic path (keys + array indices as int). Postgres `?` is **only top-level key/element**. For nested path use `@?` (jsonpath) or `jsonb_path_exists(col::jsonb, '$."a"."b"[2]')`. Returns `bool` in both. |
| `col::jsonb #> ARRAY['a','b','2']` <br>or `jsonb_extract_path(col::jsonb, 'a', 'b', '2')` <br>or chained: `(col::jsonb -> 'a' -> 'b') -> 2` | `json_get(json: str, *keys: str \| int) → JsonUnion` <br>(alias: `->` operator) | **Closest is `jsonb_extract_path` (variadic text[])** or `#>` operator. DF accepts **int directly** for array indices; Postgres requires **text strings** for indices (`'2'` not `2`). Returns `jsonb` (typed union). Input must be cast from `text`/`str`. |
| `(col::jsonb #> ARRAY['a','b','2'])::text` <br>or `jsonb_extract_path_text(col::jsonb, 'a', 'b', '2')` | `json_get_str(json: str, *keys: str \| int) → str` | No dedicated typed getter. Use `#>` then cast, or `jsonb_extract_path_text` (variadic). Exact same as `json_get` + `::text`. |
| `(col::jsonb #>> ARRAY['a','b','2'])::int` <br>or `(jsonb_extract_path_text(col::jsonb, 'a', 'b', '2'))::int` | `json_get_int(json: str, *keys: str \| int) → int` | No `json_get_int`. Extract as text (`#>>` or `_text`) then cast. Fails at runtime if not integer. |
| `(col::jsonb #>> ARRAY['a','b','2'])::float` <br>or `(jsonb_extract_path_text(...))::double precision` | `json_get_float(json: str, *keys: str \| int) → float` | No dedicated function. Text extract + cast. Postgres uses `double precision` for float. |
| `(col::jsonb #>> ARRAY['a','b','2'])::bool` | `json_get_bool(json: str, *keys: str \| int) → bool` | No dedicated function. Text extract + cast. |
| `col::jsonb #> ARRAY['a','b','2']` <br>or `jsonb_extract_path(col::jsonb, 'a', 'b', '2')` (then `::text` if raw string needed) | `json_get_json(json: str, *keys: str \| int) → str` | Returns `jsonb` object (not raw string). Cast to `text` if you want the exact JSON string representation. Matches DF’s “nested raw JSON string”. |
| `col::jsonb #> ARRAY['a','b']` (returns jsonb array) <br>then `jsonb_array_elements(...)` or cast to `text[]` / use in `FROM` | `json_get_array(json: str, *keys: str \| int) → array` | Returns `jsonb` (not native Arrow array). To expand: `SELECT * FROM jsonb_array_elements(col::jsonb #> '{path}')`. No direct Arrow return. |
| `col::jsonb #>> ARRAY['a','b','2']` <br>or `jsonb_extract_path_text(col::jsonb, 'a', 'b', '2')` <br>or `col::jsonb ->> 'key'` (top-level) | `json_as_text(json: str, *keys: str \| int) → str` <br>(alias: `->>` operator) | **Exact functional match** to `->>` operator and `jsonb_extract_path_text`. Returns scalar as `text`. |
| `jsonb_array_length(col::jsonb #> ARRAY['a','b'])` (arrays only) <br>or `cardinality(jsonb_object_keys(col::jsonb #> ARRAY['a','b']))` (objects) | `json_length(json: str, *keys: str \| int) → int` | **No single function** for both array and object. `jsonb_array_length` only works on top-level arrays. For objects use `jsonb_object_keys` + `count` / `cardinality`. DF works on both array length and object key count. |
**Operator Aliases (exact mapping)**
| Postgres Operator | DF Operator | Notes |
|-------------------|-------------|-------|
| `->` / `#>` | `->` | `->` is top-level only; `#>` for full path (text array). |
| `->>` / `#>>` | `->>` | `->>` top-level; `#>>` for full path. |
| `?` / `@?` | `?` | `?` top-level only; `@?` (jsonpath) for nested. |
**Summary of Major Exact Diffs**
- DF works on raw `text`/`str` JSON; Postgres requires `::json` or (preferred) `::jsonb`.
- DF path syntax: variadic `*keys` (mix of `str` + `int` indices) → Postgres: `text[]` array or chained operators (indices as text strings `'0'`, `'1'`).
- DF has **typed getters** (`_int`, `_float`, etc.); Postgres has **none** — always text extract + SQL cast.
- DF `json_get_array` returns native Arrow array; Postgres returns `jsonb` (must expand manually).
- DF `json_length` is unified; Postgres splits array vs object.
- Postgres `jsonb` is binary & indexable (GIN); DF is string-based (no native indexing mentioned).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-5707874541
> we should raise a separate PR / proposal / discussion on the mailing list
We can do a separate pr and send updates on mailing list so people who are interested can get involved. TBH I'm not seeing must interest from community on this topic, let's hope mailing list discussion helps 🤞
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSS] Adopt datafusion-functions-json and datafusion-functions-variant into core repo [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-4170770585
One strong argument IMO to bring them into the main `datafusion` repo is to include these in `datafusion-cli`. Currently I use DuckDB sometimes just because it has built in variant / json support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSS] Adopt datafusion-functions-json and datafusion-functions-variant into core repo [datafusion]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-4170844242
+1 ... json functions are important for Spark/Comet workloads
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSS] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-4170896248
> On the variant-side, I am all for this. I am a bit concerned about slower development cycles though
Maybe we can start with datafusion-json while datafusion-variant matures / stabilizes 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSS] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-4170902104
Certainly makes sense to do one at a time, we're happy for it to be the json functions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "friendlymatthew (via GitHub)" <gi...@apache.org>.
friendlymatthew commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-4680022844
Hi everyone,
I've cut a PR that formally introduces the set of UDFs we've been developing over at datafusion-variant
I've spent some time curating a list of issues, which I plan on migrating to here (shortly)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-4682008743
> > potentially harder to attract outside maintenance contributions
>
> We can probably solve this by bringing the `datafusion-variant` to the main repo and opening some issue from [datafusion-contrib/datafusion-variant#2](https://github.com/datafusion-contrib/datafusion-variant/issues/2) some of which are candidate for `Good first issue`.
Yeah, I agree -- if we are going to add datafusion-json I think doing the sme for datafusion-variant makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-5639311234
> I would love to help move this forward with reviews and discussions.
Since you are a committer @kumarUjjawal perhaps we could move the code to `apache/datatusion-variant` more easily 🤔 That would be good in my opinion too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-5639386738
> > I would love to help move this forward with reviews and discussions.
>
> Since you are a committer @kumarUjjawal perhaps we could move the code to `apache/datatusion-variant` more easily 🤔 That would be good in my opinion too
>
This could be a good idea. This will allow us the speed up the development as I think more contributors get attracted to the project if it's directly under ASF.
I agree on @adriangb point about the `datafusion-variant` not being mature. With continuous contributions and development we can assess later on if it is a good addition to the core.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSS] Adopt datafusion-functions-json and datafusion-functions-variant into core repo [datafusion]
Posted by "friendlymatthew (via GitHub)" <gi...@apache.org>.
friendlymatthew commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-4170793567
On the variant-side, I am all for this. I am a bit concerned about slower development cycles though
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-5639268826
@mbutrovich , @comphead and @andygrove perhaps you have thoughts about repository location and governance related to Variant support in Comet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-5716937081
There's a lot going on in this thread, but I will pipe in to say that I'm happy to (continue to) review datafusion PRs related to types. We deal with many of these things with geometry and a few other types in SedonaDB.
I've lost track of the status of JSON type integration, but I think it would be helpful to have a few non-trivial extension types in the repo (UUID and JSON would be great representative examples), and also a community-driven one outside datafusion (e.g., variant). Having a few inside DataFusion helps ensure test coverage for the extension points; having at least one outside ensures that all the right extension points exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSS] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-4185127833
It might help to see one to evaluate how it would look / how may hooks
Perhaps we could make it smaller -- like one that implements scafolding / new crate / eetc an done function. And then we can add the other functions as follow on RPs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-5639260115
> @sdf-jkl also posted to the mailing list here: https://lists.apache.org/thread/bg71srlcd84y4gv42cqzdo3j6mj3t93f
Relevant content :
---------- Forwarded message ---------
From: Kosta Tarasov <ko...@apache.org>
Date: Thu, Sep 10, 2026 at 6:42 PM
Subject: [DISCUSS] Moving Variant support forward in DataFusion
To: <de...@datafusion.apache.org>
Hi all,
I'd like to bring the discussion around PR #22908[0] to the mailing list
and get broader input on how we should move Variant support forward in
DataFusion.
PR #22908 proposes bringing datafusion-variant into the DataFusion
repository as an extension crate, following a model similar to
datafusion-functions-spark. More recently, Andrew suggested another
option: keeping the crate external and making it easier to integrate
with DataFusion through cookbook examples and documentation.[1]
Either way, I think the goal is the same: make Variant straightforward
to use with DataFusion.
I'd appreciate input on a few areas:
- Integration and maintenance: Should we bring the crate into the main
DataFusion repository, or focus on making the external crate easier to
integrate? If we bring it in, are there concerns about the additional
maintenance commitment?
- SQL API and semantics: Spark compatibility was the original target for
datafusion-variant, with some additional functions influenced by
datafusion-functions-json[2] and DuckDB. Should Spark semantics and
naming be the compatibility baseline? How should we handle useful
functions that do not have Spark equivalents? I also think it would be
useful to distinguish the user-facing SQL API from functions that
primarily exist to enable optimizer rewrites.[3]
- Comet coordination: Comet now has an active native Variant support
epic[4]. There appears to be significant overlap around Variant
semantics, expression kernels, and representation boundaries. I’d like
to coordinate where possible rather than build parallel implementations.
Input from Comet contributors would be especially useful here.
If we decide to bring the crate into the DataFusion repository, I think
the incremental approach discussed on the PR makes sense: start with the
crate scaffolding and a basic JSON-to-Variant function, then add and
review the remaining functionality in smaller PRs. We should also make
sure we document both how to integrate Variant support and how users are
expected to query Variant values.
I’m happy to help move this forward whichever integration approach we
choose. Feedback on the overall direction, use cases we should
prioritize, and offers to help with implementation or review would be
very welcome.
Thanks,
Kosta
[0] https://github.com/apache/datafusion/pull/22908
[1] https://github.com/apache/datafusion/pull/22908#issuecomment-5603803800
[2] https://github.com/datafusion-contrib/datafusion-functions-json
[3] https://github.com/apache/datafusion/pull/22908#issuecomment-5469828372
[4] https://github.com/apache/datafusion-comet/issues/5438
---------------------------------------------------------------------
To unsubscribe, e-mail: [dev-unsubscribe@datafusion.apache.org](mailto:dev-unsubscribe@datafusion.apache.org)
For additional commands, e-mail: [dev-help@datafusion.apache.org](mailto:dev-help@datafusion.apache.org)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-5639239144
My thinking on this issue has evolved:
I personally believe we should leave `datafusion-variant` outside the core datafusion repo, and use it as an experiment / proving grounds for "best practices for ecosystem integration".
It seems like what `datafusion-variant` currently lacks is maintainer bandwidth to push it forward, for example there are PRs such as https://github.com/datafusion-contrib/datafusion-variant/pull/57 which have been stuck for a while. I suspect @friendlymatthew has been busy with other work. The fact that @sdf-jkl has stepped up and wants to help I think is a great opportunity.
As @adriangb pointed out to me in a call, `datafusion-variant` is relatively immature compared to the rest of the `datafusion` core crates and thus is not clear that putting `datafusion-variant` in the main DataFusion repo would actually speed up its development.
The only open question in my mind is if it is ok that the variant integration stays in [datafusion-contrib](https://github.com/datafusion-contrib), which will limit some contributors compared to if it were in `apache/datafusion-variant` (e.g. Apple employees are restricted in what they can contribute to)
My proposed plan of action:
1. Add @sdf-jkl as a committer to `datafusion-variant` (see https://github.com/datafusion-contrib/datafusion-variant/issues/65)
2. See how far we can get with an integration in that repository
3. If there is some reason putting the code in another repo makes sense, we can revist that.
cc @Samyak2 and @bharath-techie who I think have also expressed interest in this code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-4677308512
> potentially harder to attract outside maintenance contributions
We can probably solve this by bringing the `datafusion-variant` to the main repo and opening some issue from https://github.com/datafusion-contrib/datafusion-variant/issues/2 some of which are candidate for `Good first issue`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSS] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "crm26 (via GitHub)" <gi...@apache.org>.
crm26 commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-4184987314
I have a working build with `datafusion-functions-json` integrated as an optional feature in datafusion-python (#1466). Happy to submit the PR to bring it into the main workspace if that's helpful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-5639212316
I would love to help move this forward with reviews and discussions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-5243837181
@alamb I'd love to help moving this forward and oversee the implementation
> (e.g. tracking documentation, filing tickets for missing features, reviewing PRs, etc)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "friendlymatthew (via GitHub)" <gi...@apache.org>.
friendlymatthew commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-4768400640
Hi folks, https://github.com/apache/datafusion/pull/22908 should be ready for your consideration.
Please take a look, if curious
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-4198616675
@crm26 has a proposal PR here:
- https://github.com/apache/datafusion/pull/21353
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-5244556106
Awesome -- thank you @sdf-jkl -- I think a good place to start is
- https://github.com/apache/datafusion/pull/22908
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-5639072490
@sdf-jkl also posted to the mailing list here: https://lists.apache.org/thread/bg71srlcd84y4gv42cqzdo3j6mj3t93f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSSION] Adopt datafusion-functions-json and datafusion-variant into core repo [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21301:
URL: https://github.com/apache/datafusion/issues/21301#issuecomment-5702491163
I just added @sdf-jkl as a maintainer of https://github.com/datafusion-contrib/datafusion-variant (see https://github.com/datafusion-contrib/datafusion-variant/issues/65)
@kumarUjjawal what shall we do regarding creating an apache repo? Perhaps we should raise a separate PR / proposal / discussion on the mailing list (as this conversatio is somewhat buried in a long comment thread) 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org