You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Neelesh Salian <ne...@gmail.com> on 2026/09/06 17:44:11 UTC
Re: [DISCUSS] Iceberg Variant - Tracking Document & Sync Proposal
Hi folks,
Here is the Meeting Notes and the Recording link from the Variant Sync on
September 03, 2026: [Notes
<https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz>
].
Please reach out if you have any questions.
Thanks.
On Sun, Aug 9, 2026 at 5:49 PM Neelesh Salian <ne...@gmail.com>
wrote:
> Hi folks,
>
> Here is the Meeting Notes and Recording link from the Variant Sync on
> August 06, 2026: [Notes
> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz>
> ].
> Please reach out if you have any questions.
> Thanks.
>
> On Thu, Jul 2, 2026 at 8:21 PM Neelesh Salian <ne...@gmail.com>
> wrote:
>
>> Hi folks,
>>
>> Here is the Meeting Notes and Recording link from the Variant Sync today
>> July 02, 2026: [Notes
>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz>
>> ].
>> There are some action items and active work.
>> Please reach out if you have any questions.
>> Thanks
>>
>> On Fri, Jun 5, 2026 at 12:22 PM Neelesh Salian <ne...@gmail.com>
>> wrote:
>>
>>> Hi folks,
>>>
>>> Here is the Meeting Notes and Recording link from the Variant Sync on
>>> June 4, 2026: [Notes
>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz#heading=h.r977qio1wsv2>
>>> ].
>>> There are some action items and active work.
>>> Please reach out if you have any questions.
>>> Thanks.
>>>
>>> On Fri, May 8, 2026 at 3:20 PM Neelesh Salian <ne...@gmail.com>
>>> wrote:
>>>
>>>> Hi folks,
>>>>
>>>> Here is the Meeting Notes and Recording link from the Variant Sync on
>>>> May 7, 2026: [Notes
>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz>
>>>> ].
>>>> There are some action items and active work.
>>>> Please reach out if you have any questions.
>>>> Thanks.
>>>>
>>>> On Thu, Apr 30, 2026 at 1:36 PM Neelesh Salian <
>>>> neeleshssalian@gmail.com> wrote:
>>>>
>>>>> Hi folks,
>>>>>
>>>>> I've set up a time starting next week on Thursday (May 7, 2026) at 10
>>>>> am Pacific time for a sync for the active work on Variant.
>>>>> This will be a monthly sync (on the first Thursday of every month).
>>>>> You can find it on the dev calendar.
>>>>> Here is the calendar invite:
>>>>> https://calendar.app.google/b8ykdTV3EaNnVnkv8
>>>>> I'll be recording the call and capturing notes in the sync document: Iceberg
>>>>> - Variant Community Update
>>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing> (Meeting
>>>>> Notes tab).
>>>>> Thanks.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Apr 20, 2026 at 1:49 PM Steve Loughran <st...@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>> + regarding the rust, go and cpp impls, a status from each team would
>>>>>> be great!
>>>>>>
>>>>>> I've been reviewing arrow parquet variant stuff and it is all there,
>>>>>> including with some benchmarks and optimisations. Which may put it ahead of
>>>>>> the others.
>>>>>>
>>>>>> It also has some special handling for sorted variants, as key search
>>>>>> there is straightforward. AFAIK I don't think the others do that, and nor
>>>>>> do I see them going to any effort to sort fields in an object. I think
>>>>>> sorting would be good, but you would have to handle the case where there
>>>>>> are duplicate keys. It's allowed in the spec, and seems like itcould creep
>>>>>> in from nested variants. Has anyone looked at this?
>>>>>>
>>>>>> Also: has anyone created malformed parquet files with a shredded
>>>>>> variant and a metadata entry of the same name. The requirement is "ignore
>>>>>> the metadata one", but that's something to test. You'd have to write a
>>>>>> shredded file and then edit the binary content to achieve this, or manually
>>>>>> create one and put it into the parquet-testing repository under bad-data/
>>>>>>
>>>>>>
>>>>>> On Mon, 20 Apr 2026 at 19:08, Qiegang Long <ql...@gmail.com> wrote:
>>>>>>
>>>>>>> Thanks for the doc to track the status! +1 on the dedicated
>>>>>>> sync—definitely feels like there’s a lot of work before we see Variant’s
>>>>>>> full potential.
>>>>>>>
>>>>>>> Qiegang
>>>>>>>
>>>>>>> On Mon, Apr 20, 2026 at 11:09 AM Steve Loughran <st...@cloudera.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> This is great, we need that tracker as it is cross-project. piece
>>>>>>>> of work to say "this is readly
>>>>>>>>
>>>>>>>> I did have an agenda item from last month's community call which
>>>>>>>> didn't get through. If we can retain that open time slot we could do a very
>>>>>>>> quick summary of where we are (summarly slides of Qiegang's results and
>>>>>>>> mine, key outstanding issues and next steps, then we can start that
>>>>>>>> monthly session on it.
>>>>>>>>
>>>>>>>> Meanwhile, I have both parquet and iceberg PRs for benchmarks which
>>>>>>>> I think are ready for review -please take a look
>>>>>>>>
>>>>>>>> Finally, I'm thinking about interop of those many, many variant
>>>>>>>> readers out there. Has anyone explicitly cross-tested their implementations
>>>>>>>> of variant? what about consistent handling of invalid data? That includes
>>>>>>>> iceberg-rust, parquet-cpp and more...
>>>>>>>>
>>>>>>>> Steve
>>>>>>>>
>>>>>>>> On Sun, 19 Apr 2026 at 21:57, Neelesh Salian <
>>>>>>>> neeleshssalian@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi everyone,
>>>>>>>>>
>>>>>>>>> The Variant umbrella issue (#10392
>>>>>>>>> <https://github.com/apache/iceberg/issues/10392>) hasn't been
>>>>>>>>> updated in a while, and with active work happening across multiple PRs in
>>>>>>>>> Iceberg, Spark, and Parquet, it's been hard to keep track of where things
>>>>>>>>> stand.
>>>>>>>>>
>>>>>>>>> Since a few of us are actively working on variant features, I
>>>>>>>>> thought it would help to put together a tracking document so the community
>>>>>>>>> has a single place to see the current state, open work, and benchmark
>>>>>>>>> findings. I plan to update this on a weekly basis to keep track of the
>>>>>>>>> issues and PRs that are updated.
>>>>>>>>>
>>>>>>>>> Iceberg Variant Community Document
>>>>>>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing>
>>>>>>>>>
>>>>>>>>> The document has three tabs:
>>>>>>>>>
>>>>>>>>> 1. Overview - what shipped in 1.10, what's merged to main,
>>>>>>>>> open work areas, and the dependency graph across Iceberg, Spark, and Parquet
>>>>>>>>> 2. Tracker - all open variant issues and PRs across Iceberg,
>>>>>>>>> Parquet-Java, Parquet-Format, and Spark with authors and status
>>>>>>>>> 3. Benchmarks - summary of three independent benchmark efforts
>>>>>>>>> (details below)
>>>>>>>>>
>>>>>>>>> *Benchmark findings*
>>>>>>>>>
>>>>>>>>> Three independent benchmarks have measured variant performance.
>>>>>>>>> All converge on the same picture: variant is a modest improvement over JSON
>>>>>>>>> strings today (1.1-1.7x faster reads), but 15-17x slower than typed columns.
>>>>>>>>>
>>>>>>>>> 1. Qiegang Long - 14 queries on GitHub Archive, 5 configs:
>>>>>>>>> https://qlong.github.io/posts/2026-03-30-variant-early-results
>>>>>>>>> 2. Steve Loughran - JMH microbenchmarks, profiler-driven
>>>>>>>>> optimization:
>>>>>>>>> https://steveloughran.github.io/benchmarking-variants/benchmarking-variants.html
>>>>>>>>> <https://steveloughran.github.io/benchmarking-variants/benchmarking-variants.html>
>>>>>>>>> 3. Neelesh Salian - Controlled baseline, 10M+100M rows, write
>>>>>>>>> + read:
>>>>>>>>> https://github.com/nssalian/iceberg/tree/iceberg-variant-benchmark/benchmark
>>>>>>>>>
>>>>>>>>> If you're working on variant-related changes, please chime in or
>>>>>>>>> let me know and I'll add it to the tracker. Feedback on the benchmarks or
>>>>>>>>> anything else is welcome.
>>>>>>>>>
>>>>>>>>> I've been giving variant updates during the Iceberg Spark Sync
>>>>>>>>> (Tuesdays, 10 AM PT), but given that this work now spans Iceberg, Spark,
>>>>>>>>> Parquet, and Flink, I think it deserves its own forum. I'd like to propose
>>>>>>>>> a monthly Variant Sync; a short call where contributors can share progress,
>>>>>>>>> surface blockers, and coordinate across repos. If there's interest, I'll
>>>>>>>>> set one up and share an invite on this thread.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Neelesh Salian.
>>>>>>>>>
>>>>>>>>
Re: [DISCUSS] Iceberg Variant - Tracking Document & Sync Proposal
Posted by Amit Sharma <am...@gmail.com>.
Great . Thank you Neelesh.
On Thursday, November 5, 2026, Neelesh Salian <ne...@gmail.com>
wrote:
> Hi folks,
> I’ve updated the document to track the Variant work [here
> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing>].
> Please refer to the “Next milestones” section to see what is coming.
> I’ve cancelled the call for December due to the holidays, we’ll catch up
> in January 2027.
>
> Regardless of the sync, I’ll continue to keep that document updated as I
> go across the communities that work on Variant.
> Please don’t hesitate to reach out to me if you have questions around
> Variant and how we are progressing.
>
> On Sun, Sep 6, 2026 at 10:44 AM Neelesh Salian <ne...@gmail.com>
> wrote:
>
>> Hi folks,
>>
>> Here is the Meeting Notes and the Recording link from the Variant Sync on
>> September 03, 2026: [Notes
>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz>
>> ].
>> Please reach out if you have any questions.
>> Thanks.
>>
>> On Sun, Aug 9, 2026 at 5:49 PM Neelesh Salian <ne...@gmail.com>
>> wrote:
>>
>>> Hi folks,
>>>
>>> Here is the Meeting Notes and Recording link from the Variant Sync on
>>> August 06, 2026: [Notes
>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz>
>>> ].
>>> Please reach out if you have any questions.
>>> Thanks.
>>>
>>> On Thu, Jul 2, 2026 at 8:21 PM Neelesh Salian <ne...@gmail.com>
>>> wrote:
>>>
>>>> Hi folks,
>>>>
>>>> Here is the Meeting Notes and Recording link from the Variant Sync
>>>> today July 02, 2026: [Notes
>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz>
>>>> ].
>>>> There are some action items and active work.
>>>> Please reach out if you have any questions.
>>>> Thanks
>>>>
>>>> On Fri, Jun 5, 2026 at 12:22 PM Neelesh Salian <
>>>> neeleshssalian@gmail.com> wrote:
>>>>
>>>>> Hi folks,
>>>>>
>>>>> Here is the Meeting Notes and Recording link from the Variant Sync on
>>>>> June 4, 2026: [Notes
>>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz#heading=h.r977qio1wsv2>
>>>>> ].
>>>>> There are some action items and active work.
>>>>> Please reach out if you have any questions.
>>>>> Thanks.
>>>>>
>>>>> On Fri, May 8, 2026 at 3:20 PM Neelesh Salian <
>>>>> neeleshssalian@gmail.com> wrote:
>>>>>
>>>>>> Hi folks,
>>>>>>
>>>>>> Here is the Meeting Notes and Recording link from the Variant Sync on
>>>>>> May 7, 2026: [Notes
>>>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz>
>>>>>> ].
>>>>>> There are some action items and active work.
>>>>>> Please reach out if you have any questions.
>>>>>> Thanks.
>>>>>>
>>>>>> On Thu, Apr 30, 2026 at 1:36 PM Neelesh Salian <
>>>>>> neeleshssalian@gmail.com> wrote:
>>>>>>
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> I've set up a time starting next week on Thursday (May 7, 2026) at
>>>>>>> 10 am Pacific time for a sync for the active work on Variant.
>>>>>>> This will be a monthly sync (on the first Thursday of every month).
>>>>>>> You can find it on the dev calendar.
>>>>>>> Here is the calendar invite: https://calendar.app.
>>>>>>> google/b8ykdTV3EaNnVnkv8
>>>>>>> I'll be recording the call and capturing notes in the sync document: Iceberg
>>>>>>> - Variant Community Update
>>>>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing> (Meeting
>>>>>>> Notes tab).
>>>>>>> Thanks.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Apr 20, 2026 at 1:49 PM Steve Loughran <st...@cloudera.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> + regarding the rust, go and cpp impls, a status from each team
>>>>>>>> would be great!
>>>>>>>>
>>>>>>>> I've been reviewing arrow parquet variant stuff and it is all
>>>>>>>> there, including with some benchmarks and optimisations. Which may put it
>>>>>>>> ahead of the others.
>>>>>>>>
>>>>>>>> It also has some special handling for sorted variants, as key
>>>>>>>> search there is straightforward. AFAIK I don't think the others do that,
>>>>>>>> and nor do I see them going to any effort to sort fields in an object. I
>>>>>>>> think sorting would be good, but you would have to handle the case where
>>>>>>>> there are duplicate keys. It's allowed in the spec, and seems like itcould
>>>>>>>> creep in from nested variants. Has anyone looked at this?
>>>>>>>>
>>>>>>>> Also: has anyone created malformed parquet files with a shredded
>>>>>>>> variant and a metadata entry of the same name. The requirement is "ignore
>>>>>>>> the metadata one", but that's something to test. You'd have to write a
>>>>>>>> shredded file and then edit the binary content to achieve this, or manually
>>>>>>>> create one and put it into the parquet-testing repository under bad-data/
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, 20 Apr 2026 at 19:08, Qiegang Long <ql...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks for the doc to track the status! +1 on the dedicated
>>>>>>>>> sync—definitely feels like there’s a lot of work before we see Variant’s
>>>>>>>>> full potential.
>>>>>>>>>
>>>>>>>>> Qiegang
>>>>>>>>>
>>>>>>>>> On Mon, Apr 20, 2026 at 11:09 AM Steve Loughran <
>>>>>>>>> stevel@cloudera.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This is great, we need that tracker as it is cross-project. piece
>>>>>>>>>> of work to say "this is readly
>>>>>>>>>>
>>>>>>>>>> I did have an agenda item from last month's community call which
>>>>>>>>>> didn't get through. If we can retain that open time slot we could do a very
>>>>>>>>>> quick summary of where we are (summarly slides of Qiegang's results and
>>>>>>>>>> mine, key outstanding issues and next steps, then we can start that
>>>>>>>>>> monthly session on it.
>>>>>>>>>>
>>>>>>>>>> Meanwhile, I have both parquet and iceberg PRs for
>>>>>>>>>> benchmarks which I think are ready for review -please take a look
>>>>>>>>>>
>>>>>>>>>> Finally, I'm thinking about interop of those many, many variant
>>>>>>>>>> readers out there. Has anyone explicitly cross-tested their implementations
>>>>>>>>>> of variant? what about consistent handling of invalid data? That includes
>>>>>>>>>> iceberg-rust, parquet-cpp and more...
>>>>>>>>>>
>>>>>>>>>> Steve
>>>>>>>>>>
>>>>>>>>>> On Sun, 19 Apr 2026 at 21:57, Neelesh Salian <
>>>>>>>>>> neeleshssalian@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>
>>>>>>>>>>> The Variant umbrella issue (#10392
>>>>>>>>>>> <https://github.com/apache/iceberg/issues/10392>) hasn't been
>>>>>>>>>>> updated in a while, and with active work happening across multiple PRs in
>>>>>>>>>>> Iceberg, Spark, and Parquet, it's been hard to keep track of where things
>>>>>>>>>>> stand.
>>>>>>>>>>>
>>>>>>>>>>> Since a few of us are actively working on variant features, I
>>>>>>>>>>> thought it would help to put together a tracking document so the community
>>>>>>>>>>> has a single place to see the current state, open work, and benchmark
>>>>>>>>>>> findings. I plan to update this on a weekly basis to keep track of the
>>>>>>>>>>> issues and PRs that are updated.
>>>>>>>>>>>
>>>>>>>>>>> Iceberg Variant Community Document
>>>>>>>>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing>
>>>>>>>>>>>
>>>>>>>>>>> The document has three tabs:
>>>>>>>>>>>
>>>>>>>>>>> 1. Overview - what shipped in 1.10, what's merged to main,
>>>>>>>>>>> open work areas, and the dependency graph across Iceberg, Spark, and Parquet
>>>>>>>>>>> 2. Tracker - all open variant issues and PRs across Iceberg,
>>>>>>>>>>> Parquet-Java, Parquet-Format, and Spark with authors and status
>>>>>>>>>>> 3. Benchmarks - summary of three independent benchmark
>>>>>>>>>>> efforts (details below)
>>>>>>>>>>>
>>>>>>>>>>> *Benchmark findings*
>>>>>>>>>>>
>>>>>>>>>>> Three independent benchmarks have measured variant performance.
>>>>>>>>>>> All converge on the same picture: variant is a modest improvement over JSON
>>>>>>>>>>> strings today (1.1-1.7x faster reads), but 15-17x slower than typed columns.
>>>>>>>>>>>
>>>>>>>>>>> 1. Qiegang Long - 14 queries on GitHub Archive, 5 configs:
>>>>>>>>>>> https://qlong.github.io/posts/2026-03-30-variant-early-
>>>>>>>>>>> results
>>>>>>>>>>> <https://qlong.github.io/posts/2026-03-30-variant-early-results>
>>>>>>>>>>> 2. Steve Loughran - JMH microbenchmarks, profiler-driven
>>>>>>>>>>> optimization: https://steveloughran.github.
>>>>>>>>>>> io/benchmarking-variants/benchmarking-variants.html
>>>>>>>>>>> <https://steveloughran.github.io/benchmarking-variants/benchmarking-variants.html>
>>>>>>>>>>> 3. Neelesh Salian - Controlled baseline, 10M+100M rows,
>>>>>>>>>>> write + read: https://github.com/nssalian/
>>>>>>>>>>> iceberg/tree/iceberg-variant-benchmark/benchmark
>>>>>>>>>>> <https://github.com/nssalian/iceberg/tree/iceberg-variant-benchmark/benchmark>
>>>>>>>>>>>
>>>>>>>>>>> If you're working on variant-related changes, please chime in or
>>>>>>>>>>> let me know and I'll add it to the tracker. Feedback on the benchmarks or
>>>>>>>>>>> anything else is welcome.
>>>>>>>>>>>
>>>>>>>>>>> I've been giving variant updates during the Iceberg Spark Sync
>>>>>>>>>>> (Tuesdays, 10 AM PT), but given that this work now spans Iceberg, Spark,
>>>>>>>>>>> Parquet, and Flink, I think it deserves its own forum. I'd like to propose
>>>>>>>>>>> a monthly Variant Sync; a short call where contributors can share progress,
>>>>>>>>>>> surface blockers, and coordinate across repos. If there's interest, I'll
>>>>>>>>>>> set one up and share an invite on this thread.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Neelesh Salian.
>>>>>>>>>>>
>>>>>>>>>>
Re: [DISCUSS] Iceberg Variant - Tracking Document & Sync Proposal
Posted by Neelesh Salian <ne...@gmail.com>.
Hi folks,
I’ve updated the document to track the Variant work [here
<https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing>].
Please refer to the “Next milestones” section to see what is coming.
I’ve cancelled the call for December due to the holidays, we’ll catch up in
January 2027.
Regardless of the sync, I’ll continue to keep that document updated as I go
across the communities that work on Variant.
Please don’t hesitate to reach out to me if you have questions around
Variant and how we are progressing.
On Sun, Sep 6, 2026 at 10:44 AM Neelesh Salian <ne...@gmail.com>
wrote:
> Hi folks,
>
> Here is the Meeting Notes and the Recording link from the Variant Sync on
> September 03, 2026: [Notes
> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz>
> ].
> Please reach out if you have any questions.
> Thanks.
>
> On Sun, Aug 9, 2026 at 5:49 PM Neelesh Salian <ne...@gmail.com>
> wrote:
>
>> Hi folks,
>>
>> Here is the Meeting Notes and Recording link from the Variant Sync on
>> August 06, 2026: [Notes
>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz>
>> ].
>> Please reach out if you have any questions.
>> Thanks.
>>
>> On Thu, Jul 2, 2026 at 8:21 PM Neelesh Salian <ne...@gmail.com>
>> wrote:
>>
>>> Hi folks,
>>>
>>> Here is the Meeting Notes and Recording link from the Variant Sync today
>>> July 02, 2026: [Notes
>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz>
>>> ].
>>> There are some action items and active work.
>>> Please reach out if you have any questions.
>>> Thanks
>>>
>>> On Fri, Jun 5, 2026 at 12:22 PM Neelesh Salian <ne...@gmail.com>
>>> wrote:
>>>
>>>> Hi folks,
>>>>
>>>> Here is the Meeting Notes and Recording link from the Variant Sync on
>>>> June 4, 2026: [Notes
>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz#heading=h.r977qio1wsv2>
>>>> ].
>>>> There are some action items and active work.
>>>> Please reach out if you have any questions.
>>>> Thanks.
>>>>
>>>> On Fri, May 8, 2026 at 3:20 PM Neelesh Salian <ne...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi folks,
>>>>>
>>>>> Here is the Meeting Notes and Recording link from the Variant Sync on
>>>>> May 7, 2026: [Notes
>>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz>
>>>>> ].
>>>>> There are some action items and active work.
>>>>> Please reach out if you have any questions.
>>>>> Thanks.
>>>>>
>>>>> On Thu, Apr 30, 2026 at 1:36 PM Neelesh Salian <
>>>>> neeleshssalian@gmail.com> wrote:
>>>>>
>>>>>> Hi folks,
>>>>>>
>>>>>> I've set up a time starting next week on Thursday (May 7, 2026) at 10
>>>>>> am Pacific time for a sync for the active work on Variant.
>>>>>> This will be a monthly sync (on the first Thursday of every month).
>>>>>> You can find it on the dev calendar.
>>>>>> Here is the calendar invite:
>>>>>> https://calendar.app.google/b8ykdTV3EaNnVnkv8
>>>>>> I'll be recording the call and capturing notes in the sync document: Iceberg
>>>>>> - Variant Community Update
>>>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing> (Meeting
>>>>>> Notes tab).
>>>>>> Thanks.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Apr 20, 2026 at 1:49 PM Steve Loughran <st...@cloudera.com>
>>>>>> wrote:
>>>>>>
>>>>>>> + regarding the rust, go and cpp impls, a status from each team
>>>>>>> would be great!
>>>>>>>
>>>>>>> I've been reviewing arrow parquet variant stuff and it is all there,
>>>>>>> including with some benchmarks and optimisations. Which may put it ahead of
>>>>>>> the others.
>>>>>>>
>>>>>>> It also has some special handling for sorted variants, as key search
>>>>>>> there is straightforward. AFAIK I don't think the others do that, and nor
>>>>>>> do I see them going to any effort to sort fields in an object. I think
>>>>>>> sorting would be good, but you would have to handle the case where there
>>>>>>> are duplicate keys. It's allowed in the spec, and seems like itcould creep
>>>>>>> in from nested variants. Has anyone looked at this?
>>>>>>>
>>>>>>> Also: has anyone created malformed parquet files with a shredded
>>>>>>> variant and a metadata entry of the same name. The requirement is "ignore
>>>>>>> the metadata one", but that's something to test. You'd have to write a
>>>>>>> shredded file and then edit the binary content to achieve this, or manually
>>>>>>> create one and put it into the parquet-testing repository under bad-data/
>>>>>>>
>>>>>>>
>>>>>>> On Mon, 20 Apr 2026 at 19:08, Qiegang Long <ql...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Thanks for the doc to track the status! +1 on the dedicated
>>>>>>>> sync—definitely feels like there’s a lot of work before we see Variant’s
>>>>>>>> full potential.
>>>>>>>>
>>>>>>>> Qiegang
>>>>>>>>
>>>>>>>> On Mon, Apr 20, 2026 at 11:09 AM Steve Loughran <
>>>>>>>> stevel@cloudera.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> This is great, we need that tracker as it is cross-project. piece
>>>>>>>>> of work to say "this is readly
>>>>>>>>>
>>>>>>>>> I did have an agenda item from last month's community call which
>>>>>>>>> didn't get through. If we can retain that open time slot we could do a very
>>>>>>>>> quick summary of where we are (summarly slides of Qiegang's results and
>>>>>>>>> mine, key outstanding issues and next steps, then we can start that
>>>>>>>>> monthly session on it.
>>>>>>>>>
>>>>>>>>> Meanwhile, I have both parquet and iceberg PRs for
>>>>>>>>> benchmarks which I think are ready for review -please take a look
>>>>>>>>>
>>>>>>>>> Finally, I'm thinking about interop of those many, many variant
>>>>>>>>> readers out there. Has anyone explicitly cross-tested their implementations
>>>>>>>>> of variant? what about consistent handling of invalid data? That includes
>>>>>>>>> iceberg-rust, parquet-cpp and more...
>>>>>>>>>
>>>>>>>>> Steve
>>>>>>>>>
>>>>>>>>> On Sun, 19 Apr 2026 at 21:57, Neelesh Salian <
>>>>>>>>> neeleshssalian@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi everyone,
>>>>>>>>>>
>>>>>>>>>> The Variant umbrella issue (#10392
>>>>>>>>>> <https://github.com/apache/iceberg/issues/10392>) hasn't been
>>>>>>>>>> updated in a while, and with active work happening across multiple PRs in
>>>>>>>>>> Iceberg, Spark, and Parquet, it's been hard to keep track of where things
>>>>>>>>>> stand.
>>>>>>>>>>
>>>>>>>>>> Since a few of us are actively working on variant features, I
>>>>>>>>>> thought it would help to put together a tracking document so the community
>>>>>>>>>> has a single place to see the current state, open work, and benchmark
>>>>>>>>>> findings. I plan to update this on a weekly basis to keep track of the
>>>>>>>>>> issues and PRs that are updated.
>>>>>>>>>>
>>>>>>>>>> Iceberg Variant Community Document
>>>>>>>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing>
>>>>>>>>>>
>>>>>>>>>> The document has three tabs:
>>>>>>>>>>
>>>>>>>>>> 1. Overview - what shipped in 1.10, what's merged to main,
>>>>>>>>>> open work areas, and the dependency graph across Iceberg, Spark, and Parquet
>>>>>>>>>> 2. Tracker - all open variant issues and PRs across Iceberg,
>>>>>>>>>> Parquet-Java, Parquet-Format, and Spark with authors and status
>>>>>>>>>> 3. Benchmarks - summary of three independent benchmark
>>>>>>>>>> efforts (details below)
>>>>>>>>>>
>>>>>>>>>> *Benchmark findings*
>>>>>>>>>>
>>>>>>>>>> Three independent benchmarks have measured variant performance.
>>>>>>>>>> All converge on the same picture: variant is a modest improvement over JSON
>>>>>>>>>> strings today (1.1-1.7x faster reads), but 15-17x slower than typed columns.
>>>>>>>>>>
>>>>>>>>>> 1. Qiegang Long - 14 queries on GitHub Archive, 5 configs:
>>>>>>>>>> https://qlong.github.io/posts/2026-03-30-variant-early-results
>>>>>>>>>> 2. Steve Loughran - JMH microbenchmarks, profiler-driven
>>>>>>>>>> optimization:
>>>>>>>>>> https://steveloughran.github.io/benchmarking-variants/benchmarking-variants.html
>>>>>>>>>> <https://steveloughran.github.io/benchmarking-variants/benchmarking-variants.html>
>>>>>>>>>> 3. Neelesh Salian - Controlled baseline, 10M+100M rows, write
>>>>>>>>>> + read:
>>>>>>>>>> https://github.com/nssalian/iceberg/tree/iceberg-variant-benchmark/benchmark
>>>>>>>>>>
>>>>>>>>>> If you're working on variant-related changes, please chime in or
>>>>>>>>>> let me know and I'll add it to the tracker. Feedback on the benchmarks or
>>>>>>>>>> anything else is welcome.
>>>>>>>>>>
>>>>>>>>>> I've been giving variant updates during the Iceberg Spark Sync
>>>>>>>>>> (Tuesdays, 10 AM PT), but given that this work now spans Iceberg, Spark,
>>>>>>>>>> Parquet, and Flink, I think it deserves its own forum. I'd like to propose
>>>>>>>>>> a monthly Variant Sync; a short call where contributors can share progress,
>>>>>>>>>> surface blockers, and coordinate across repos. If there's interest, I'll
>>>>>>>>>> set one up and share an invite on this thread.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Neelesh Salian.
>>>>>>>>>>
>>>>>>>>>