You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Neelesh Salian <ne...@gmail.com> on 2026/06/04 16:49:06 UTC
[DISCUSS] Variant Shredding Policy Across Iceberg Implementations
Hi all,
Variant shredding writers now exist across iceberg-java, arrow-go, and
arrow-rs, and shredded write paths are landing soon in iceberg-go,
iceberg-rust, and (future) iceberg-python. We don't yet have a shared
policy on whether to shred, how the layout is declared, and how it's
expressed so every implementation produces compatible output.
I'd like to propose one: Variant Shredding Policy Across Iceberg
Implementations
<https://docs.google.com/document/d/1OaB45mg9JJ_JRg-ipuvyh4AtOmaGmdWMsy_mJKiu5xs/edit?usp=sharing>
.
Please take a look.
Next steps:
1. The community reviews the proposal and agrees on the shredding
representation.
2. Once consensus is reached, I'll start a [VOTE] thread.
3. After the vote passes, I'll land the adjustments in the Java
implementation.
4. iceberg-go, iceberg-rust, and others follow to align.
Thanks,
Neelesh
Re: [DISCUSS] Variant Shredding Policy Across Iceberg Implementations
Posted by Neelesh Salian <ne...@gmail.com>.
Hi folks,
Sharing an update on the Variant shredding proposal - [Link
<https://docs.google.com/document/d/1OaB45mg9JJ_JRg-ipuvyh4AtOmaGmdWMsy_mJKiu5xs/edit?usp=sharing>
].
After a round of feedback on the initial design, I ran benchmarks across a
set of inference strategies to settle the algorithm question.
Results are in the Shredding Strategy Benchmarks tab and the code can be
found [strategy benchmark]
<https://github.com/nssalian/iceberg/blob/iceberg-variant-benchmark/benchmark/strategies.md>
The proposal now has the resulting inference algorithm and moves the
tunable thresholds from constants to table properties.
Please take a look and let me know what you think.
Thank you.
On Fri, Jun 5, 2026 at 12:26 PM Neelesh Salian <ne...@gmail.com>
wrote:
> Hi folks,
>
> Thanks for those joining the Variant sync yesterday to discuss this
> proposal.
> And thanks Kurtis for the note. Appreciate the feedback and discussion as
> well.
>
> Based on the feedback and the discussion during the call [Notes and
> Recording here
> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing>];
> the proposal should head towards aligning on the best inference algorithm
> to yield the best results for shredding.
> I'll be adjusting the proposal as I, along with folks in the community,
> work on getting the data + testing for the algorithm.
> Will surface the proposal again with this new direction when it's in a
> better shape.
>
> Thank you and feel free to reach out if you have any questions.
>
>
> On Thu, Jun 4, 2026 at 5:43 PM <ku...@gmail.com> wrote:
>
>> Hi Neelesh,
>>
>> Thank you for organizing the meeting earlier today and starting this
>> thread. I think it’s awesome to see the community align not just on types
>> but implementations for Variant.
>>
>> I wanted to highlight one additional item I heard discussed on the sync.
>>
>> - We need to align on an inference algorithm that defines how a
>> variant object is shredded.
>>
>>
>> - The goal is we can align using data from high quality benchmarking
>> data sets if anyone or company is willing to contribute.
>>
>> Best Regards,
>> Kurtis C. Wright
>>
>> On Jun 4, 2026, at 09:49, Neelesh Salian <ne...@gmail.com>
>> wrote:
>>
>>
>> Hi all,
>>
>> Variant shredding writers now exist across iceberg-java, arrow-go, and
>> arrow-rs, and shredded write paths are landing soon in iceberg-go,
>> iceberg-rust, and (future) iceberg-python. We don't yet have a shared
>> policy on whether to shred, how the layout is declared, and how it's
>> expressed so every implementation produces compatible output.
>>
>> I'd like to propose one: Variant Shredding Policy Across Iceberg
>> Implementations
>> <https://docs.google.com/document/d/1OaB45mg9JJ_JRg-ipuvyh4AtOmaGmdWMsy_mJKiu5xs/edit?usp=sharing>
>> .
>> Please take a look.
>>
>> Next steps:
>> 1. The community reviews the proposal and agrees on the shredding
>> representation.
>> 2. Once consensus is reached, I'll start a [VOTE] thread.
>> 3. After the vote passes, I'll land the adjustments in the Java
>> implementation.
>> 4. iceberg-go, iceberg-rust, and others follow to align.
>>
>> Thanks,
>> Neelesh
>>
>>
Re: [DISCUSS] Variant Shredding Policy Across Iceberg Implementations
Posted by Neelesh Salian <ne...@gmail.com>.
Hi folks,
Thanks for those joining the Variant sync yesterday to discuss this
proposal.
And thanks Kurtis for the note. Appreciate the feedback and discussion as
well.
Based on the feedback and the discussion during the call [Notes and
Recording here
<https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing>];
the proposal should head towards aligning on the best inference algorithm
to yield the best results for shredding.
I'll be adjusting the proposal as I, along with folks in the community,
work on getting the data + testing for the algorithm.
Will surface the proposal again with this new direction when it's in a
better shape.
Thank you and feel free to reach out if you have any questions.
On Thu, Jun 4, 2026 at 5:43 PM <ku...@gmail.com> wrote:
> Hi Neelesh,
>
> Thank you for organizing the meeting earlier today and starting this
> thread. I think it’s awesome to see the community align not just on types
> but implementations for Variant.
>
> I wanted to highlight one additional item I heard discussed on the sync.
>
> - We need to align on an inference algorithm that defines how a
> variant object is shredded.
>
>
> - The goal is we can align using data from high quality benchmarking
> data sets if anyone or company is willing to contribute.
>
> Best Regards,
> Kurtis C. Wright
>
> On Jun 4, 2026, at 09:49, Neelesh Salian <ne...@gmail.com> wrote:
>
>
> Hi all,
>
> Variant shredding writers now exist across iceberg-java, arrow-go, and
> arrow-rs, and shredded write paths are landing soon in iceberg-go,
> iceberg-rust, and (future) iceberg-python. We don't yet have a shared
> policy on whether to shred, how the layout is declared, and how it's
> expressed so every implementation produces compatible output.
>
> I'd like to propose one: Variant Shredding Policy Across Iceberg
> Implementations
> <https://docs.google.com/document/d/1OaB45mg9JJ_JRg-ipuvyh4AtOmaGmdWMsy_mJKiu5xs/edit?usp=sharing>
> .
> Please take a look.
>
> Next steps:
> 1. The community reviews the proposal and agrees on the shredding
> representation.
> 2. Once consensus is reached, I'll start a [VOTE] thread.
> 3. After the vote passes, I'll land the adjustments in the Java
> implementation.
> 4. iceberg-go, iceberg-rust, and others follow to align.
>
> Thanks,
> Neelesh
>
>
Re: [DISCUSS] Variant Shredding Policy Across Iceberg Implementations
Posted by ku...@gmail.com.
Hi Neelesh,
Thank you for organizing the meeting earlier today and starting this thread. I
think it’s awesome to see the community align not just on types but
implementations for Variant.
I wanted to highlight one additional item I heard discussed on the sync.
* We need to align on an inference algorithm that defines how a variant object is shredded.
* The goal is we can align using data from high quality benchmarking data sets if anyone or company is willing to contribute.
Best Regards,
Kurtis C. Wright
> On Jun 4, 2026, at 09:49, Neelesh Salian <ne...@gmail.com> wrote:
>
>
>
>
> Hi all,
>
> Variant shredding writers now exist across iceberg-java, arrow-go, and
> arrow-rs, and shredded write paths are landing soon in iceberg-go, iceberg-
> rust, and (future) iceberg-python. We don't yet have a shared policy on
> whether to shred, how the layout is declared, and how it's expressed so
> every implementation produces compatible output.
>
> I'd like to propose one: [Variant Shredding Policy Across Iceberg
> Implementations](https://docs.google.com/document/d/1OaB45mg9JJ_JRg-
> ipuvyh4AtOmaGmdWMsy_mJKiu5xs/edit?usp=sharing).
>
> Please take a look.
>
>
>
> Next steps:
> 1\. The community reviews the proposal and agrees on the shredding
> representation.
> 2\. Once consensus is reached, I'll start a [VOTE] thread.
> 3\. After the vote passes, I'll land the adjustments in the Java
> implementation.
> 4\. iceberg-go, iceberg-rust, and others follow to align.
>
> Thanks,
> Neelesh