You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Neelesh Salian <ne...@gmail.com> on 2026/06/05 19:22:13 UTC
Re: [DISCUSS] Iceberg Variant - Tracking Document & Sync Proposal
Hi folks,
Here is the Meeting Notes and Recording link from the Variant Sync on June
4, 2026: [Notes
<https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz#heading=h.r977qio1wsv2>
].
There are some action items and active work.
Please reach out if you have any questions.
Thanks.
On Fri, May 8, 2026 at 3:20 PM Neelesh Salian <ne...@gmail.com>
wrote:
> Hi folks,
>
> Here is the Meeting Notes and Recording link from the Variant Sync on May
> 7, 2026: [Notes
> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz>
> ].
> There are some action items and active work.
> Please reach out if you have any questions.
> Thanks.
>
> On Thu, Apr 30, 2026 at 1:36 PM Neelesh Salian <ne...@gmail.com>
> wrote:
>
>> Hi folks,
>>
>> I've set up a time starting next week on Thursday (May 7, 2026) at 10 am
>> Pacific time for a sync for the active work on Variant.
>> This will be a monthly sync (on the first Thursday of every month).
>> You can find it on the dev calendar.
>> Here is the calendar invite:
>> https://calendar.app.google/b8ykdTV3EaNnVnkv8
>> I'll be recording the call and capturing notes in the sync document: Iceberg
>> - Variant Community Update
>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing> (Meeting
>> Notes tab).
>> Thanks.
>>
>>
>>
>>
>>
>>
>> On Mon, Apr 20, 2026 at 1:49 PM Steve Loughran <st...@cloudera.com>
>> wrote:
>>
>>> + regarding the rust, go and cpp impls, a status from each team would be
>>> great!
>>>
>>> I've been reviewing arrow parquet variant stuff and it is all there,
>>> including with some benchmarks and optimisations. Which may put it ahead of
>>> the others.
>>>
>>> It also has some special handling for sorted variants, as key search
>>> there is straightforward. AFAIK I don't think the others do that, and nor
>>> do I see them going to any effort to sort fields in an object. I think
>>> sorting would be good, but you would have to handle the case where there
>>> are duplicate keys. It's allowed in the spec, and seems like itcould creep
>>> in from nested variants. Has anyone looked at this?
>>>
>>> Also: has anyone created malformed parquet files with a shredded variant
>>> and a metadata entry of the same name. The requirement is "ignore the
>>> metadata one", but that's something to test. You'd have to write a shredded
>>> file and then edit the binary content to achieve this, or manually create
>>> one and put it into the parquet-testing repository under bad-data/
>>>
>>>
>>> On Mon, 20 Apr 2026 at 19:08, Qiegang Long <ql...@gmail.com> wrote:
>>>
>>>> Thanks for the doc to track the status! +1 on the dedicated
>>>> sync—definitely feels like there’s a lot of work before we see Variant’s
>>>> full potential.
>>>>
>>>> Qiegang
>>>>
>>>> On Mon, Apr 20, 2026 at 11:09 AM Steve Loughran <st...@cloudera.com>
>>>> wrote:
>>>>
>>>>>
>>>>> This is great, we need that tracker as it is cross-project. piece of
>>>>> work to say "this is readly
>>>>>
>>>>> I did have an agenda item from last month's community call which
>>>>> didn't get through. If we can retain that open time slot we could do a very
>>>>> quick summary of where we are (summarly slides of Qiegang's results and
>>>>> mine, key outstanding issues and next steps, then we can start that
>>>>> monthly session on it.
>>>>>
>>>>> Meanwhile, I have both parquet and iceberg PRs for benchmarks which I
>>>>> think are ready for review -please take a look
>>>>>
>>>>> Finally, I'm thinking about interop of those many, many variant
>>>>> readers out there. Has anyone explicitly cross-tested their implementations
>>>>> of variant? what about consistent handling of invalid data? That includes
>>>>> iceberg-rust, parquet-cpp and more...
>>>>>
>>>>> Steve
>>>>>
>>>>> On Sun, 19 Apr 2026 at 21:57, Neelesh Salian <ne...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> The Variant umbrella issue (#10392
>>>>>> <https://github.com/apache/iceberg/issues/10392>) hasn't been
>>>>>> updated in a while, and with active work happening across multiple PRs in
>>>>>> Iceberg, Spark, and Parquet, it's been hard to keep track of where things
>>>>>> stand.
>>>>>>
>>>>>> Since a few of us are actively working on variant features, I thought
>>>>>> it would help to put together a tracking document so the community has a
>>>>>> single place to see the current state, open work, and benchmark findings. I
>>>>>> plan to update this on a weekly basis to keep track of the issues and PRs
>>>>>> that are updated.
>>>>>>
>>>>>> Iceberg Variant Community Document
>>>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing>
>>>>>>
>>>>>> The document has three tabs:
>>>>>>
>>>>>> 1. Overview - what shipped in 1.10, what's merged to main, open
>>>>>> work areas, and the dependency graph across Iceberg, Spark, and Parquet
>>>>>> 2. Tracker - all open variant issues and PRs across Iceberg,
>>>>>> Parquet-Java, Parquet-Format, and Spark with authors and status
>>>>>> 3. Benchmarks - summary of three independent benchmark efforts
>>>>>> (details below)
>>>>>>
>>>>>> *Benchmark findings*
>>>>>>
>>>>>> Three independent benchmarks have measured variant performance. All
>>>>>> converge on the same picture: variant is a modest improvement over JSON
>>>>>> strings today (1.1-1.7x faster reads), but 15-17x slower than typed columns.
>>>>>>
>>>>>> 1. Qiegang Long - 14 queries on GitHub Archive, 5 configs:
>>>>>> https://qlong.github.io/posts/2026-03-30-variant-early-results
>>>>>> 2. Steve Loughran - JMH microbenchmarks, profiler-driven
>>>>>> optimization:
>>>>>> https://steveloughran.github.io/benchmarking-variants/benchmarking-variants.html
>>>>>> <https://steveloughran.github.io/benchmarking-variants/benchmarking-variants.html>
>>>>>> 3. Neelesh Salian - Controlled baseline, 10M+100M rows, write +
>>>>>> read:
>>>>>> https://github.com/nssalian/iceberg/tree/iceberg-variant-benchmark/benchmark
>>>>>>
>>>>>> If you're working on variant-related changes, please chime in or let
>>>>>> me know and I'll add it to the tracker. Feedback on the benchmarks or
>>>>>> anything else is welcome.
>>>>>>
>>>>>> I've been giving variant updates during the Iceberg Spark Sync
>>>>>> (Tuesdays, 10 AM PT), but given that this work now spans Iceberg, Spark,
>>>>>> Parquet, and Flink, I think it deserves its own forum. I'd like to propose
>>>>>> a monthly Variant Sync; a short call where contributors can share progress,
>>>>>> surface blockers, and coordinate across repos. If there's interest, I'll
>>>>>> set one up and share an invite on this thread.
>>>>>>
>>>>>> Thanks,
>>>>>> Neelesh Salian.
>>>>>>
>>>>>
Re: [DISCUSS] Iceberg Variant - Tracking Document & Sync Proposal
Posted by Neelesh Salian <ne...@gmail.com>.
Hi folks,
Here is the Meeting Notes and the Recording link from the Variant Sync on
September 03, 2026: [Notes
<https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz>
].
Please reach out if you have any questions.
Thanks.
On Sun, Aug 9, 2026 at 5:49 PM Neelesh Salian <ne...@gmail.com>
wrote:
> Hi folks,
>
> Here is the Meeting Notes and Recording link from the Variant Sync on
> August 06, 2026: [Notes
> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz>
> ].
> Please reach out if you have any questions.
> Thanks.
>
> On Thu, Jul 2, 2026 at 8:21 PM Neelesh Salian <ne...@gmail.com>
> wrote:
>
>> Hi folks,
>>
>> Here is the Meeting Notes and Recording link from the Variant Sync today
>> July 02, 2026: [Notes
>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz>
>> ].
>> There are some action items and active work.
>> Please reach out if you have any questions.
>> Thanks
>>
>> On Fri, Jun 5, 2026 at 12:22 PM Neelesh Salian <ne...@gmail.com>
>> wrote:
>>
>>> Hi folks,
>>>
>>> Here is the Meeting Notes and Recording link from the Variant Sync on
>>> June 4, 2026: [Notes
>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz#heading=h.r977qio1wsv2>
>>> ].
>>> There are some action items and active work.
>>> Please reach out if you have any questions.
>>> Thanks.
>>>
>>> On Fri, May 8, 2026 at 3:20 PM Neelesh Salian <ne...@gmail.com>
>>> wrote:
>>>
>>>> Hi folks,
>>>>
>>>> Here is the Meeting Notes and Recording link from the Variant Sync on
>>>> May 7, 2026: [Notes
>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz>
>>>> ].
>>>> There are some action items and active work.
>>>> Please reach out if you have any questions.
>>>> Thanks.
>>>>
>>>> On Thu, Apr 30, 2026 at 1:36 PM Neelesh Salian <
>>>> neeleshssalian@gmail.com> wrote:
>>>>
>>>>> Hi folks,
>>>>>
>>>>> I've set up a time starting next week on Thursday (May 7, 2026) at 10
>>>>> am Pacific time for a sync for the active work on Variant.
>>>>> This will be a monthly sync (on the first Thursday of every month).
>>>>> You can find it on the dev calendar.
>>>>> Here is the calendar invite:
>>>>> https://calendar.app.google/b8ykdTV3EaNnVnkv8
>>>>> I'll be recording the call and capturing notes in the sync document: Iceberg
>>>>> - Variant Community Update
>>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing> (Meeting
>>>>> Notes tab).
>>>>> Thanks.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Apr 20, 2026 at 1:49 PM Steve Loughran <st...@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>> + regarding the rust, go and cpp impls, a status from each team would
>>>>>> be great!
>>>>>>
>>>>>> I've been reviewing arrow parquet variant stuff and it is all there,
>>>>>> including with some benchmarks and optimisations. Which may put it ahead of
>>>>>> the others.
>>>>>>
>>>>>> It also has some special handling for sorted variants, as key search
>>>>>> there is straightforward. AFAIK I don't think the others do that, and nor
>>>>>> do I see them going to any effort to sort fields in an object. I think
>>>>>> sorting would be good, but you would have to handle the case where there
>>>>>> are duplicate keys. It's allowed in the spec, and seems like itcould creep
>>>>>> in from nested variants. Has anyone looked at this?
>>>>>>
>>>>>> Also: has anyone created malformed parquet files with a shredded
>>>>>> variant and a metadata entry of the same name. The requirement is "ignore
>>>>>> the metadata one", but that's something to test. You'd have to write a
>>>>>> shredded file and then edit the binary content to achieve this, or manually
>>>>>> create one and put it into the parquet-testing repository under bad-data/
>>>>>>
>>>>>>
>>>>>> On Mon, 20 Apr 2026 at 19:08, Qiegang Long <ql...@gmail.com> wrote:
>>>>>>
>>>>>>> Thanks for the doc to track the status! +1 on the dedicated
>>>>>>> sync—definitely feels like there’s a lot of work before we see Variant’s
>>>>>>> full potential.
>>>>>>>
>>>>>>> Qiegang
>>>>>>>
>>>>>>> On Mon, Apr 20, 2026 at 11:09 AM Steve Loughran <st...@cloudera.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> This is great, we need that tracker as it is cross-project. piece
>>>>>>>> of work to say "this is readly
>>>>>>>>
>>>>>>>> I did have an agenda item from last month's community call which
>>>>>>>> didn't get through. If we can retain that open time slot we could do a very
>>>>>>>> quick summary of where we are (summarly slides of Qiegang's results and
>>>>>>>> mine, key outstanding issues and next steps, then we can start that
>>>>>>>> monthly session on it.
>>>>>>>>
>>>>>>>> Meanwhile, I have both parquet and iceberg PRs for benchmarks which
>>>>>>>> I think are ready for review -please take a look
>>>>>>>>
>>>>>>>> Finally, I'm thinking about interop of those many, many variant
>>>>>>>> readers out there. Has anyone explicitly cross-tested their implementations
>>>>>>>> of variant? what about consistent handling of invalid data? That includes
>>>>>>>> iceberg-rust, parquet-cpp and more...
>>>>>>>>
>>>>>>>> Steve
>>>>>>>>
>>>>>>>> On Sun, 19 Apr 2026 at 21:57, Neelesh Salian <
>>>>>>>> neeleshssalian@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi everyone,
>>>>>>>>>
>>>>>>>>> The Variant umbrella issue (#10392
>>>>>>>>> <https://github.com/apache/iceberg/issues/10392>) hasn't been
>>>>>>>>> updated in a while, and with active work happening across multiple PRs in
>>>>>>>>> Iceberg, Spark, and Parquet, it's been hard to keep track of where things
>>>>>>>>> stand.
>>>>>>>>>
>>>>>>>>> Since a few of us are actively working on variant features, I
>>>>>>>>> thought it would help to put together a tracking document so the community
>>>>>>>>> has a single place to see the current state, open work, and benchmark
>>>>>>>>> findings. I plan to update this on a weekly basis to keep track of the
>>>>>>>>> issues and PRs that are updated.
>>>>>>>>>
>>>>>>>>> Iceberg Variant Community Document
>>>>>>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing>
>>>>>>>>>
>>>>>>>>> The document has three tabs:
>>>>>>>>>
>>>>>>>>> 1. Overview - what shipped in 1.10, what's merged to main,
>>>>>>>>> open work areas, and the dependency graph across Iceberg, Spark, and Parquet
>>>>>>>>> 2. Tracker - all open variant issues and PRs across Iceberg,
>>>>>>>>> Parquet-Java, Parquet-Format, and Spark with authors and status
>>>>>>>>> 3. Benchmarks - summary of three independent benchmark efforts
>>>>>>>>> (details below)
>>>>>>>>>
>>>>>>>>> *Benchmark findings*
>>>>>>>>>
>>>>>>>>> Three independent benchmarks have measured variant performance.
>>>>>>>>> All converge on the same picture: variant is a modest improvement over JSON
>>>>>>>>> strings today (1.1-1.7x faster reads), but 15-17x slower than typed columns.
>>>>>>>>>
>>>>>>>>> 1. Qiegang Long - 14 queries on GitHub Archive, 5 configs:
>>>>>>>>> https://qlong.github.io/posts/2026-03-30-variant-early-results
>>>>>>>>> 2. Steve Loughran - JMH microbenchmarks, profiler-driven
>>>>>>>>> optimization:
>>>>>>>>> https://steveloughran.github.io/benchmarking-variants/benchmarking-variants.html
>>>>>>>>> <https://steveloughran.github.io/benchmarking-variants/benchmarking-variants.html>
>>>>>>>>> 3. Neelesh Salian - Controlled baseline, 10M+100M rows, write
>>>>>>>>> + read:
>>>>>>>>> https://github.com/nssalian/iceberg/tree/iceberg-variant-benchmark/benchmark
>>>>>>>>>
>>>>>>>>> If you're working on variant-related changes, please chime in or
>>>>>>>>> let me know and I'll add it to the tracker. Feedback on the benchmarks or
>>>>>>>>> anything else is welcome.
>>>>>>>>>
>>>>>>>>> I've been giving variant updates during the Iceberg Spark Sync
>>>>>>>>> (Tuesdays, 10 AM PT), but given that this work now spans Iceberg, Spark,
>>>>>>>>> Parquet, and Flink, I think it deserves its own forum. I'd like to propose
>>>>>>>>> a monthly Variant Sync; a short call where contributors can share progress,
>>>>>>>>> surface blockers, and coordinate across repos. If there's interest, I'll
>>>>>>>>> set one up and share an invite on this thread.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Neelesh Salian.
>>>>>>>>>
>>>>>>>>
Re: [DISCUSS] Iceberg Variant - Tracking Document & Sync Proposal
Posted by Neelesh Salian <ne...@gmail.com>.
Hi folks,
Here is the Meeting Notes and Recording link from the Variant Sync on
August 06, 2026: [Notes
<https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz>
].
Please reach out if you have any questions.
Thanks.
On Thu, Jul 2, 2026 at 8:21 PM Neelesh Salian <ne...@gmail.com>
wrote:
> Hi folks,
>
> Here is the Meeting Notes and Recording link from the Variant Sync today
> July 02, 2026: [Notes
> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz>
> ].
> There are some action items and active work.
> Please reach out if you have any questions.
> Thanks
>
> On Fri, Jun 5, 2026 at 12:22 PM Neelesh Salian <ne...@gmail.com>
> wrote:
>
>> Hi folks,
>>
>> Here is the Meeting Notes and Recording link from the Variant Sync on
>> June 4, 2026: [Notes
>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz#heading=h.r977qio1wsv2>
>> ].
>> There are some action items and active work.
>> Please reach out if you have any questions.
>> Thanks.
>>
>> On Fri, May 8, 2026 at 3:20 PM Neelesh Salian <ne...@gmail.com>
>> wrote:
>>
>>> Hi folks,
>>>
>>> Here is the Meeting Notes and Recording link from the Variant Sync on
>>> May 7, 2026: [Notes
>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz>
>>> ].
>>> There are some action items and active work.
>>> Please reach out if you have any questions.
>>> Thanks.
>>>
>>> On Thu, Apr 30, 2026 at 1:36 PM Neelesh Salian <ne...@gmail.com>
>>> wrote:
>>>
>>>> Hi folks,
>>>>
>>>> I've set up a time starting next week on Thursday (May 7, 2026) at 10
>>>> am Pacific time for a sync for the active work on Variant.
>>>> This will be a monthly sync (on the first Thursday of every month).
>>>> You can find it on the dev calendar.
>>>> Here is the calendar invite:
>>>> https://calendar.app.google/b8ykdTV3EaNnVnkv8
>>>> I'll be recording the call and capturing notes in the sync document: Iceberg
>>>> - Variant Community Update
>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing> (Meeting
>>>> Notes tab).
>>>> Thanks.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Apr 20, 2026 at 1:49 PM Steve Loughran <st...@cloudera.com>
>>>> wrote:
>>>>
>>>>> + regarding the rust, go and cpp impls, a status from each team would
>>>>> be great!
>>>>>
>>>>> I've been reviewing arrow parquet variant stuff and it is all there,
>>>>> including with some benchmarks and optimisations. Which may put it ahead of
>>>>> the others.
>>>>>
>>>>> It also has some special handling for sorted variants, as key search
>>>>> there is straightforward. AFAIK I don't think the others do that, and nor
>>>>> do I see them going to any effort to sort fields in an object. I think
>>>>> sorting would be good, but you would have to handle the case where there
>>>>> are duplicate keys. It's allowed in the spec, and seems like itcould creep
>>>>> in from nested variants. Has anyone looked at this?
>>>>>
>>>>> Also: has anyone created malformed parquet files with a shredded
>>>>> variant and a metadata entry of the same name. The requirement is "ignore
>>>>> the metadata one", but that's something to test. You'd have to write a
>>>>> shredded file and then edit the binary content to achieve this, or manually
>>>>> create one and put it into the parquet-testing repository under bad-data/
>>>>>
>>>>>
>>>>> On Mon, 20 Apr 2026 at 19:08, Qiegang Long <ql...@gmail.com> wrote:
>>>>>
>>>>>> Thanks for the doc to track the status! +1 on the dedicated
>>>>>> sync—definitely feels like there’s a lot of work before we see Variant’s
>>>>>> full potential.
>>>>>>
>>>>>> Qiegang
>>>>>>
>>>>>> On Mon, Apr 20, 2026 at 11:09 AM Steve Loughran <st...@cloudera.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> This is great, we need that tracker as it is cross-project. piece of
>>>>>>> work to say "this is readly
>>>>>>>
>>>>>>> I did have an agenda item from last month's community call which
>>>>>>> didn't get through. If we can retain that open time slot we could do a very
>>>>>>> quick summary of where we are (summarly slides of Qiegang's results and
>>>>>>> mine, key outstanding issues and next steps, then we can start that
>>>>>>> monthly session on it.
>>>>>>>
>>>>>>> Meanwhile, I have both parquet and iceberg PRs for benchmarks which
>>>>>>> I think are ready for review -please take a look
>>>>>>>
>>>>>>> Finally, I'm thinking about interop of those many, many variant
>>>>>>> readers out there. Has anyone explicitly cross-tested their implementations
>>>>>>> of variant? what about consistent handling of invalid data? That includes
>>>>>>> iceberg-rust, parquet-cpp and more...
>>>>>>>
>>>>>>> Steve
>>>>>>>
>>>>>>> On Sun, 19 Apr 2026 at 21:57, Neelesh Salian <
>>>>>>> neeleshssalian@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi everyone,
>>>>>>>>
>>>>>>>> The Variant umbrella issue (#10392
>>>>>>>> <https://github.com/apache/iceberg/issues/10392>) hasn't been
>>>>>>>> updated in a while, and with active work happening across multiple PRs in
>>>>>>>> Iceberg, Spark, and Parquet, it's been hard to keep track of where things
>>>>>>>> stand.
>>>>>>>>
>>>>>>>> Since a few of us are actively working on variant features, I
>>>>>>>> thought it would help to put together a tracking document so the community
>>>>>>>> has a single place to see the current state, open work, and benchmark
>>>>>>>> findings. I plan to update this on a weekly basis to keep track of the
>>>>>>>> issues and PRs that are updated.
>>>>>>>>
>>>>>>>> Iceberg Variant Community Document
>>>>>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing>
>>>>>>>>
>>>>>>>> The document has three tabs:
>>>>>>>>
>>>>>>>> 1. Overview - what shipped in 1.10, what's merged to main, open
>>>>>>>> work areas, and the dependency graph across Iceberg, Spark, and Parquet
>>>>>>>> 2. Tracker - all open variant issues and PRs across Iceberg,
>>>>>>>> Parquet-Java, Parquet-Format, and Spark with authors and status
>>>>>>>> 3. Benchmarks - summary of three independent benchmark efforts
>>>>>>>> (details below)
>>>>>>>>
>>>>>>>> *Benchmark findings*
>>>>>>>>
>>>>>>>> Three independent benchmarks have measured variant performance. All
>>>>>>>> converge on the same picture: variant is a modest improvement over JSON
>>>>>>>> strings today (1.1-1.7x faster reads), but 15-17x slower than typed columns.
>>>>>>>>
>>>>>>>> 1. Qiegang Long - 14 queries on GitHub Archive, 5 configs:
>>>>>>>> https://qlong.github.io/posts/2026-03-30-variant-early-results
>>>>>>>> 2. Steve Loughran - JMH microbenchmarks, profiler-driven
>>>>>>>> optimization:
>>>>>>>> https://steveloughran.github.io/benchmarking-variants/benchmarking-variants.html
>>>>>>>> <https://steveloughran.github.io/benchmarking-variants/benchmarking-variants.html>
>>>>>>>> 3. Neelesh Salian - Controlled baseline, 10M+100M rows, write +
>>>>>>>> read:
>>>>>>>> https://github.com/nssalian/iceberg/tree/iceberg-variant-benchmark/benchmark
>>>>>>>>
>>>>>>>> If you're working on variant-related changes, please chime in or
>>>>>>>> let me know and I'll add it to the tracker. Feedback on the benchmarks or
>>>>>>>> anything else is welcome.
>>>>>>>>
>>>>>>>> I've been giving variant updates during the Iceberg Spark Sync
>>>>>>>> (Tuesdays, 10 AM PT), but given that this work now spans Iceberg, Spark,
>>>>>>>> Parquet, and Flink, I think it deserves its own forum. I'd like to propose
>>>>>>>> a monthly Variant Sync; a short call where contributors can share progress,
>>>>>>>> surface blockers, and coordinate across repos. If there's interest, I'll
>>>>>>>> set one up and share an invite on this thread.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Neelesh Salian.
>>>>>>>>
>>>>>>>
Re: [DISCUSS] Iceberg Variant - Tracking Document & Sync Proposal
Posted by Neelesh Salian <ne...@gmail.com>.
Hi folks,
Here is the Meeting Notes and Recording link from the Variant Sync today
July 02, 2026: [Notes
<https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz>
].
There are some action items and active work.
Please reach out if you have any questions.
Thanks
On Fri, Jun 5, 2026 at 12:22 PM Neelesh Salian <ne...@gmail.com>
wrote:
> Hi folks,
>
> Here is the Meeting Notes and Recording link from the Variant Sync on June
> 4, 2026: [Notes
> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz#heading=h.r977qio1wsv2>
> ].
> There are some action items and active work.
> Please reach out if you have any questions.
> Thanks.
>
> On Fri, May 8, 2026 at 3:20 PM Neelesh Salian <ne...@gmail.com>
> wrote:
>
>> Hi folks,
>>
>> Here is the Meeting Notes and Recording link from the Variant Sync on May
>> 7, 2026: [Notes
>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz>
>> ].
>> There are some action items and active work.
>> Please reach out if you have any questions.
>> Thanks.
>>
>> On Thu, Apr 30, 2026 at 1:36 PM Neelesh Salian <ne...@gmail.com>
>> wrote:
>>
>>> Hi folks,
>>>
>>> I've set up a time starting next week on Thursday (May 7, 2026) at 10 am
>>> Pacific time for a sync for the active work on Variant.
>>> This will be a monthly sync (on the first Thursday of every month).
>>> You can find it on the dev calendar.
>>> Here is the calendar invite:
>>> https://calendar.app.google/b8ykdTV3EaNnVnkv8
>>> I'll be recording the call and capturing notes in the sync document: Iceberg
>>> - Variant Community Update
>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing> (Meeting
>>> Notes tab).
>>> Thanks.
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Apr 20, 2026 at 1:49 PM Steve Loughran <st...@cloudera.com>
>>> wrote:
>>>
>>>> + regarding the rust, go and cpp impls, a status from each team would
>>>> be great!
>>>>
>>>> I've been reviewing arrow parquet variant stuff and it is all there,
>>>> including with some benchmarks and optimisations. Which may put it ahead of
>>>> the others.
>>>>
>>>> It also has some special handling for sorted variants, as key search
>>>> there is straightforward. AFAIK I don't think the others do that, and nor
>>>> do I see them going to any effort to sort fields in an object. I think
>>>> sorting would be good, but you would have to handle the case where there
>>>> are duplicate keys. It's allowed in the spec, and seems like itcould creep
>>>> in from nested variants. Has anyone looked at this?
>>>>
>>>> Also: has anyone created malformed parquet files with a shredded
>>>> variant and a metadata entry of the same name. The requirement is "ignore
>>>> the metadata one", but that's something to test. You'd have to write a
>>>> shredded file and then edit the binary content to achieve this, or manually
>>>> create one and put it into the parquet-testing repository under bad-data/
>>>>
>>>>
>>>> On Mon, 20 Apr 2026 at 19:08, Qiegang Long <ql...@gmail.com> wrote:
>>>>
>>>>> Thanks for the doc to track the status! +1 on the dedicated
>>>>> sync—definitely feels like there’s a lot of work before we see Variant’s
>>>>> full potential.
>>>>>
>>>>> Qiegang
>>>>>
>>>>> On Mon, Apr 20, 2026 at 11:09 AM Steve Loughran <st...@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> This is great, we need that tracker as it is cross-project. piece of
>>>>>> work to say "this is readly
>>>>>>
>>>>>> I did have an agenda item from last month's community call which
>>>>>> didn't get through. If we can retain that open time slot we could do a very
>>>>>> quick summary of where we are (summarly slides of Qiegang's results and
>>>>>> mine, key outstanding issues and next steps, then we can start that
>>>>>> monthly session on it.
>>>>>>
>>>>>> Meanwhile, I have both parquet and iceberg PRs for benchmarks which I
>>>>>> think are ready for review -please take a look
>>>>>>
>>>>>> Finally, I'm thinking about interop of those many, many variant
>>>>>> readers out there. Has anyone explicitly cross-tested their implementations
>>>>>> of variant? what about consistent handling of invalid data? That includes
>>>>>> iceberg-rust, parquet-cpp and more...
>>>>>>
>>>>>> Steve
>>>>>>
>>>>>> On Sun, 19 Apr 2026 at 21:57, Neelesh Salian <
>>>>>> neeleshssalian@gmail.com> wrote:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> The Variant umbrella issue (#10392
>>>>>>> <https://github.com/apache/iceberg/issues/10392>) hasn't been
>>>>>>> updated in a while, and with active work happening across multiple PRs in
>>>>>>> Iceberg, Spark, and Parquet, it's been hard to keep track of where things
>>>>>>> stand.
>>>>>>>
>>>>>>> Since a few of us are actively working on variant features, I
>>>>>>> thought it would help to put together a tracking document so the community
>>>>>>> has a single place to see the current state, open work, and benchmark
>>>>>>> findings. I plan to update this on a weekly basis to keep track of the
>>>>>>> issues and PRs that are updated.
>>>>>>>
>>>>>>> Iceberg Variant Community Document
>>>>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing>
>>>>>>>
>>>>>>> The document has three tabs:
>>>>>>>
>>>>>>> 1. Overview - what shipped in 1.10, what's merged to main, open
>>>>>>> work areas, and the dependency graph across Iceberg, Spark, and Parquet
>>>>>>> 2. Tracker - all open variant issues and PRs across Iceberg,
>>>>>>> Parquet-Java, Parquet-Format, and Spark with authors and status
>>>>>>> 3. Benchmarks - summary of three independent benchmark efforts
>>>>>>> (details below)
>>>>>>>
>>>>>>> *Benchmark findings*
>>>>>>>
>>>>>>> Three independent benchmarks have measured variant performance. All
>>>>>>> converge on the same picture: variant is a modest improvement over JSON
>>>>>>> strings today (1.1-1.7x faster reads), but 15-17x slower than typed columns.
>>>>>>>
>>>>>>> 1. Qiegang Long - 14 queries on GitHub Archive, 5 configs:
>>>>>>> https://qlong.github.io/posts/2026-03-30-variant-early-results
>>>>>>> 2. Steve Loughran - JMH microbenchmarks, profiler-driven
>>>>>>> optimization:
>>>>>>> https://steveloughran.github.io/benchmarking-variants/benchmarking-variants.html
>>>>>>> <https://steveloughran.github.io/benchmarking-variants/benchmarking-variants.html>
>>>>>>> 3. Neelesh Salian - Controlled baseline, 10M+100M rows, write +
>>>>>>> read:
>>>>>>> https://github.com/nssalian/iceberg/tree/iceberg-variant-benchmark/benchmark
>>>>>>>
>>>>>>> If you're working on variant-related changes, please chime in or let
>>>>>>> me know and I'll add it to the tracker. Feedback on the benchmarks or
>>>>>>> anything else is welcome.
>>>>>>>
>>>>>>> I've been giving variant updates during the Iceberg Spark Sync
>>>>>>> (Tuesdays, 10 AM PT), but given that this work now spans Iceberg, Spark,
>>>>>>> Parquet, and Flink, I think it deserves its own forum. I'd like to propose
>>>>>>> a monthly Variant Sync; a short call where contributors can share progress,
>>>>>>> surface blockers, and coordinate across repos. If there's interest, I'll
>>>>>>> set one up and share an invite on this thread.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Neelesh Salian.
>>>>>>>
>>>>>>