You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "raghuvanshraj (via GitHub)" <gi...@apache.org> on 2026/10/20 09:18:21 UTC
[I] [EPIC] Variant on DataFusion [datafusion]
raghuvanshraj opened a new issue, #26517:
URL: https://github.com/apache/datafusion/issues/26517
### Is your feature request related to a problem or challenge?
## Is your feature request related to a problem or challenge?
OpenSearch is adding a Variant field type to its Parquet-backed indices. The query engine is DataFusion, the writer and Variant kernels are arrow-rs, and the SQL functions are `datafusion-contrib/datafusion-variant`. Files mix shredded and unshredded layouts within one table, and the same table is read by Spark.
This epic lists the core hooks that work still needs, following the split in #16116: generic extension points in core, Variant-specific implementations in the crate. It sits under #16116 and #21308. Items already tracked elsewhere are linked rather than restated; the new issues are the ones marked **new**.
Spelling the type in SQL (section 6) already has upstream issues filed; they are linked here so the epic covers the whole core surface Variant needs.
## Describe the solution you'd like
### 1. Read only the leaves a UDF needs
`variant_get(v, '$.x')` reads and decodes the whole Variant group because the Parquet planner only trusts `get_field`. Two PRs address this: #25013 (two hooks on `ScalarUDFImpl`) and draft #26282 (`NestedInputAccess` capability, also used by `get_field`, with row group and page pruning on the declared leaves). Our requirement for whichever shape lands: a function must be able to require more than one leaf for one argument (the typed leaf and the `value` fallback leaf), and pruning must keep any container that holds a fallback row. #26282 does both.
- [ ] Review #25013 / #26282 against shredded and unshredded test data and help converge on one shape
- [ ] Row group and page pruning on a declared leaf, with the fallback guard: #20871. #26282 includes it; if the work is split, this is the successor to #25013
- [ ] `datafusion-variant`: implement the capability for `variant_get` once it lands
### 2. Keep the extension tag through planning
A Variant is a struct whose field metadata carries `ARROW:extension:name = arrow.parquet.variant`. Two core paths drop it, after which `variant_get` sees a plain struct and rejects the column.
- [ ] **new** Parquet schema inference clears all field metadata before `Schema::try_merge` when `skip_metadata = true` (the default), so any `ListingTable` over more than one file loses the tag (`datasource-parquet/src/file_format.rs`, `clear_metadata`). Reproduced on 54. Proposed fix: keep the `ARROW:extension:*` keys recursively and drop the rest; the extension keys are the type's identity, not writer annotation.
- [ ] `CASE` / `COALESCE` build the output `Field` from the `DataType` alone: #24860 (reported on `geoarrow.wkb`, same mechanism)
### 3. Tables that mix shredded and unshredded files
When one file stores `v` as `Struct<metadata, value>` and another as `Struct<metadata, value, typed_value: Struct<x: Int64>>`, the merged table schema has `typed_value`, and the default adapter null-fills it for the unshredded file (`physical-expr-adapter/src/schema_rewriter.rs`). For an ordinary struct that is correct. For Variant the shredding spec makes `value = object, typed_value = null` an invalid row, so a compliant reader returns NULL for every field of every row of that file, silently. Reproduced: a two-file table returned 0 of 1024 unshredded rows. Files shredded on different keys fail differently, at open, with `target field is non-nullable but missing from source`.
The constraint we would like core to hold: a file is read in its own layout and reconstructed, never null-filled into a layout it does not have. Two ways to satisfy it: per-file `required_fields` answers (#26282 already lets the answer differ per file; to be confirmed whether that alone avoids the null-fill), or a hook for a registered extension type to supply its own per-file adaptation. The Variant adapter itself would live in the crate. Named earlier on #16116 (2025-07-15) and #22908 (2026-09-08).
- [ ] **new** Per-file adaptation for extension types, so an unshredded file is never null-filled into the shredded layout: per-file `required_fields` answers (#26282) or an extension-type adapter hook, with the Variant adapter in the crate
### 4. Statistics the optimizer can use
`ColumnStatistics` is flat, so a nested leaf (shredded or not) and any UDF output have no planner statistics. Two designs are open for the UDF half and one has to be picked; #26282 adds a container-level `output_statistics` for pruning, which is a third data point.
- [ ] Nested and extension-type leaves in `Statistics`: #20707
- [ ] UDF output statistics: `ScalarUDFImpl::output_statistics` (#21307) or `ExpressionAnalyzer` (#21122). We implement whichever is chosen; the crate half (typed-leaf min/max, fallback-leaf null count) is the same code either way.
### 5. Variant value semantics
`GROUP BY`, `DISTINCT`, joins and `ORDER BY` on a Variant column are undefined today. Spark rejects all of them at analysis time. DuckDB 1.5 allows them, but `=`, `GROUP BY`, `DISTINCT` and `ORDER BY` each apply a different rule (equal numbers of different width compare equal but group apart; `ORDER BY` is by inner type tag, so `7.0` sorts after `42`).
- [ ] **new, after design discussion** A canonical-bytes normalisation kernel in arrow-rs so equal values have equal bytes, and a `variant_normalize` function in the crate on top of it, so hashing operators work without each inventing a rule. For `ORDER BY` our lean is to refuse at plan time as Spark does. Issue to follow once the SQL semantics are written up.
### 6. Spelling `VARIANT` in SQL
The generic machinery is merged: `ExtensionTypeRegistry` and `DFExtensionType` (#20312, DataFusion 54), `TypePlanner` (#20676, 54), and metadata-preserving `CAST` (#23169, 55.1.0 via backport #24875). What is missing is the Variant entries. Without them a Variant prints as raw struct bytes in `datafusion-cli`, and `CREATE TABLE t (v VARIANT)` or `CAST(x AS VARIANT)` cannot be planned. Spark 4.2 and DuckDB 1.5 both spell the type `VARIANT` in `CREATE TABLE` and `CAST`, so this is what lets a query written for them run unchanged.
- [ ] Register `arrow.parquet.variant` as a `DFExtensionType` (JSON display, and the identity the other hooks key on): the one unticked box in #21144. Small; the natural first PR
- [ ] Map the SQL name `VARIANT` to that type via `TypePlanner`: #21626, same shape as UUID and JSON. The crate-side registration goes with it
- [ ] Pluggable cast resolution to and from extension types (`CAST(x AS VARIANT)`, `v::INT`): #20755, after the registration lands and a maintainer design. Not on our critical path
### Adjacent, in arrow-rs
- [ ] The Parquet writer omits `specification_version` from the `VARIANT` logical type (`LogicalType::variant(None)`), so Spark 4.1 rejects the file with `PARQUET_TYPE_NOT_RECOGNIZED`. Reproduced 2026-09-25; unchanged on arrow-rs main as of 2026-10-19 (`parquet/src/arrow/schema/extension.rs`, `logical_type_for_struct`). One-line fix; issue to be filed on arrow-rs and linked here.
## Describe alternatives you've considered
Keeping all of this inside OpenSearch (a fork of the Parquet reader and a private `variant_get`). Rejected: every hook above is generic to extension types, and the mixed-layout and statistics problems are hit by any Variant reader on DataFusion, not just ours.
## Additional context
Compatibility target for the crate work is Spark 4.2.0 (datafusion-variant#69). We build against DataFusion 54 today and are moving to 55 / arrow-rs 59, which the crate requires.
Open questions for maintainers:
1. Are items 2 and 3 accepted as core concerns (neither mentions Variant), with the Variant adapter shipped from the crate?
2. For item 1, which shape: #25013, #26282, or a merge of the two?
3. For item 4, #21307 or #21122?
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
### Willingness to contribute
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Variant on DataFusion [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #26517:
URL: https://github.com/apache/datafusion/issues/26517#issuecomment-6418662581
I also filed a ticket to try and improve docs
- https://github.com/datafusion-contrib/datafusion-variant/issues/100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Variant on DataFusion [datafusion]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #26517:
URL: https://github.com/apache/datafusion/issues/26517#issuecomment-6445914077
> Pluggable cast resolution to and from extension types
I'll try to circle back here as soon as I can (although I am also happy to review an attempt if somebody gets there first!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Variant on DataFusion [datafusion]
Posted by "raghuvanshraj (via GitHub)" <gi...@apache.org>.
raghuvanshraj commented on issue #26517:
URL: https://github.com/apache/datafusion/issues/26517#issuecomment-6309451152
@alamb I have some ideas around this that we would like to run by you some time. I've been able to prepare a plan for core and contrib contributions along with arrow-rs for E2E variant support on DF.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Variant on DataFusion [datafusion]
Posted by "raghuvanshraj (via GitHub)" <gi...@apache.org>.
raghuvanshraj commented on issue #26517:
URL: https://github.com/apache/datafusion/issues/26517#issuecomment-6309433584
> Thanks [@raghuvanshraj](https://github.com/raghuvanshraj) this is epic!
>
> > Compatibility target for the crate work is Spark 4.2.0 (datafusion-variant#69). We build against DataFusion 54 today and are moving to 55 / arrow-rs 59, which the crate requires.
>
> I updated datafusion-variant to arrow-60 here -- [datafusion-contrib/datafusion-variant#92](https://github.com/datafusion-contrib/datafusion-variant/pull/92)
>
> Spark also added some more functions in 4.3 which I'm planning to track soon
>
> > 6. Spelling VARIANT in SQL
>
> datafusion-variant has a workaround for spelling `VARIANT` in SQL. Supporting CAST is tracked here -- [datafusion-contrib/datafusion-variant#90](https://github.com/datafusion-contrib/datafusion-variant/issues/90)
Thanks @sdf-jkl ! Going through your linked issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Variant on DataFusion [datafusion]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on issue #26517:
URL: https://github.com/apache/datafusion/issues/26517#issuecomment-6295684237
Thanks @raghuvanshraj this is epic!
> Compatibility target for the crate work is Spark 4.2.0 (datafusion-variant#69). We build against DataFusion 54 today and are moving to 55 / arrow-rs 59, which the crate requires.
I updated datafusion-variant to arrow-60 here -- https://github.com/datafusion-contrib/datafusion-variant/pull/92
Spark also added some more functions in 4.3 which I'm planning to track soon
> 6. Spelling VARIANT in SQL
datafusion-variant has a workaround for spelling `VARIANT` in SQL. Supporting CAST is tracked here -- https://github.com/datafusion-contrib/datafusion-variant/issues/90
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Variant on DataFusion [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #26517:
URL: https://github.com/apache/datafusion/issues/26517#issuecomment-6375169484
FYI @sdf-jkl @friendlymatthew @adriangb and @Samyak2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Variant on DataFusion [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #26517:
URL: https://github.com/apache/datafusion/issues/26517#issuecomment-6418340414
We had a good discussion about this today -- in general I think the technical items listed above all make sense and would make good API additions to DataFusion.
In terms of gathering sufficient maintainer interest / attention to add the new APIs, I suggested that we focus on the usecases.
For example, for
> 1. Read only the leaves a UDF needs
I think it would really help to spell out the high level usecase, something like
--------
We reading variant data stored in parquet. When we run a query like this to extract field `f` from a variant column, even if `f` is shredded, DataFusion currently reads the entire variant column
```sql
SELECT variant_col:f from my_table;
```
Here is the full reproducer
```rust
// TODO: rust code that shows how to reproduce this
```
DataFusion already does this for struct fields (TODO links to examples / showing how this works), I want the same thing for variant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org