You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Gábor Kaszab <ga...@apache.org> on 2025/08/28 13:23:38 UTC
Proposal to extend standardized statistics
Hey Iceberg Community,
I've been working on a proposal
<https://docs.google.com/document/d/1H9uYt53Q1_CcOXOfLcr0hXRxvqflg_k_xeVorMLrWbM>
to extend the currently standardized statistics in Iceberg, by looking
into what statistics are used by some query engines and trying to fill the
gaps (credit also goes to Denys K to lay groundwork). The motivation is to
use Iceberg for the source of truth when it comes to statistics across all
the engines.
Meanwhile, there have been movements on other proposals (Restructuring
col-stats
<https://docs.google.com/document/d/1uvbrwwAJW2TgsnoaIcwAFpjbhHkBUL5wY_24nKgtt9I/edit?tab=t.0#heading=h.hs6r9d26w1y2>
, Restructuring metadata
<https://docs.google.com/document/d/1k4x8utgh41Sn1tr98eynDKCWq035SV_f75rtNHcerVw/edit?tab=t.0#heading=h.unn922df0zzw>)
that might overlap with mine. Let’s see how much of my proposal still holds
up in light of these developments.
Any feedback is appreciated!
Gabor
Re: Proposal to extend standardized statistics
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Gabor,
Thanks for the update.
I will take a look at the partition stat scan API PR.
Regards,
JB
On Mon, Dec 1, 2025 at 4:02 PM Gábor Kaszab <ga...@apache.org> wrote:
> Hey Iceberg Community,
>
> I'd like to bump this thread to solicit some feedback on the next steps I
> wrote in my previous mail. I'd also like to draw attention to the new partition
> stat scan API PR <https://github.com/apache/iceberg/pull/14640> that
> could be used to read, filter and project partition stats as an initial
> step.
>
> Any feedback is appreciated!
> Gabor
>
> Gábor Kaszab <ga...@apache.org> ezt írta (időpont: 2025. nov. 5.,
> Sze, 14:59):
>
>> Hey Iceberg Community,
>>
>> Thank you for taking a look at the proposal
>> <https://docs.google.com/document/d/1H9uYt53Q1_CcOXOfLcr0hXRxvqflg_k_xeVorMLrWbM>
>> and also for the feedback! First of all I'd like to apologise for the long
>> delay with my response. I went through the feedback, let me give a summary
>> and possible next steps:
>>
>> *Partition-level column stats*
>> - As a starting point a scan API could come handy (with filtering,
>> projection etc.) even for the existing partition stats. I've published a
>> PR <https://github.com/apache/iceberg/pull/14508> to introduce such an
>> API.
>> - There was an ask for this recently on Slack
>> <https://apache-iceberg.slack.com/archives/C03LG1D563F/p1760925647880099>,
>> and also there is a GH issue
>> <https://github.com/apache/iceberg/issues/11083> opened earlier.
>> - It would make sense for the partition-level column stats to follow
>> the new design of column stats coming with the V4 column stats
>> <https://docs.google.com/document/d/1uvbrwwAJW2TgsnoaIcwAFpjbhHkBUL5wY_24nKgtt9I>
>> proposal. Doing that will allow us to project partition-level column stats
>> by field and by particular stat too. Should follow-up and coordinate with
>> that proposal.
>> - If there is anything else I miss here, let me know.
>>
>> *Table-level KLL sketch*
>> So far no feedback on this. There is a PR
>> <https://github.com/apache/iceberg/pull/8202> for the spec changes for
>> this already. This could be a nice addition, I can cover the implementation
>> if there are no objections.
>>
>> *Table-level column stats (like min/max etc.)*
>> Sp far not much feedback on this. There are open questions wrt how to
>> implement this. Will wait for further feedback, putting it on hold for now
>> in favor of the above 2 items.
>>
>> *File-level avg length and max length*
>> These will be included in the V4 stats improvements
>>
>> *Partition-level Theta sketches for NDV*
>> These seem to consume too much space even with low precision and seem to
>> have limited benefits. In case there is a particular use-case for this, let
>> me know! Putting it on hold for now.
>>
>> Any further feedback is appreciated! Thanks!
>> Gabor
>>
>> Jacky Lee <qc...@gmail.com> ezt írta (időpont: 2025. aug. 28., Cs,
>> 15:54):
>>
>>> Excellent proposal!
>>>
>>> We’ve internally augmented both table-level and partition-level
>>> ColumnStatistics, and observed a 30%+ performance gain in Spark and
>>> Trino query execution—largely due to improved Cost-Based Optimization
>>> (CBO) effectiveness.
>>> However, leveraging the v3 format presented numerous challenges (such
>>> as column-type evolution and the way to save min/max values). We
>>> believe adopting the v4 format would be a more robust solution.
>>>
>>>
>>> I’ve researched this extensively and applied it in production. I’d be
>>> glad to collaborate on implementing this feature if needed.
>>>
>>>
>>> Best wishes.
>>>
>>> Gábor Kaszab <ga...@apache.org> 于2025年8月28日周四 21:23写道：
>>> >
>>> > Hey Iceberg Community,
>>> >
>>> > I've been working on a proposal to extend the currently standardized
>>> statistics in Iceberg, by looking into what statistics are used by some
>>> query engines and trying to fill the gaps (credit also goes to Denys K to
>>> lay groundwork). The motivation is to use Iceberg for the source of truth
>>> when it comes to statistics across all the engines.
>>> > Meanwhile, there have been movements on other proposals (Restructuring
>>> col-stats, Restructuring metadata) that might overlap with mine. Let’s see
>>> how much of my proposal still holds up in light of these developments.
>>> >
>>> > Any feedback is appreciated!
>>> > Gabor
>>>
>>
Re: Proposal to extend standardized statistics
Posted by Gábor Kaszab <ga...@apache.org>.
Hey Iceberg Community,
Thank you for taking a look at the proposal
<https://docs.google.com/document/d/1H9uYt53Q1_CcOXOfLcr0hXRxvqflg_k_xeVorMLrWbM>
and also for the feedback! First of all I'd like to apologise for the long
delay with my response. I went through the feedback, let me give a summary
and possible next steps:
*Partition-level column stats*
- As a starting point a scan API could come handy (with filtering,
projection etc.) even for the existing partition stats. I've published a PR
<https://github.com/apache/iceberg/pull/14508> to introduce such an API.
- There was an ask for this recently on Slack
<https://apache-iceberg.slack.com/archives/C03LG1D563F/p1760925647880099>,
and also there is a GH issue
<https://github.com/apache/iceberg/issues/11083> opened earlier.
- It would make sense for the partition-level column stats to follow the
new design of column stats coming with the V4 column stats
<https://docs.google.com/document/d/1uvbrwwAJW2TgsnoaIcwAFpjbhHkBUL5wY_24nKgtt9I>
proposal. Doing that will allow us to project partition-level column stats
by field and by particular stat too. Should follow-up and coordinate with
that proposal.
- If there is anything else I miss here, let me know.
*Table-level KLL sketch*
So far no feedback on this. There is a PR
<https://github.com/apache/iceberg/pull/8202> for the spec changes for this
already. This could be a nice addition, I can cover the implementation if
there are no objections.
*Table-level column stats (like min/max etc.)*
Sp far not much feedback on this. There are open questions wrt how to
implement this. Will wait for further feedback, putting it on hold for now
in favor of the above 2 items.
*File-level avg length and max length*
These will be included in the V4 stats improvements
*Partition-level Theta sketches for NDV*
These seem to consume too much space even with low precision and seem to
have limited benefits. In case there is a particular use-case for this, let
me know! Putting it on hold for now.
Any further feedback is appreciated! Thanks!
Gabor
Jacky Lee <qc...@gmail.com> ezt írta (időpont: 2025. aug. 28., Cs,
15:54):
> Excellent proposal!
>
> We’ve internally augmented both table-level and partition-level
> ColumnStatistics, and observed a 30%+ performance gain in Spark and
> Trino query execution—largely due to improved Cost-Based Optimization
> (CBO) effectiveness.
> However, leveraging the v3 format presented numerous challenges (such
> as column-type evolution and the way to save min/max values). We
> believe adopting the v4 format would be a more robust solution.
>
>
> I’ve researched this extensively and applied it in production. I’d be
> glad to collaborate on implementing this feature if needed.
>
>
> Best wishes.
>
> Gábor Kaszab <ga...@apache.org> 于2025年8月28日周四 21:23写道：
> >
> > Hey Iceberg Community,
> >
> > I've been working on a proposal to extend the currently standardized
> statistics in Iceberg, by looking into what statistics are used by some
> query engines and trying to fill the gaps (credit also goes to Denys K to
> lay groundwork). The motivation is to use Iceberg for the source of truth
> when it comes to statistics across all the engines.
> > Meanwhile, there have been movements on other proposals (Restructuring
> col-stats, Restructuring metadata) that might overlap with mine. Let’s see
> how much of my proposal still holds up in light of these developments.
> >
> > Any feedback is appreciated!
> > Gabor
>
Re: Proposal to extend standardized statistics
Posted by Jacky Lee <qc...@gmail.com>.
Excellent proposal!
We’ve internally augmented both table-level and partition-level
ColumnStatistics, and observed a 30%+ performance gain in Spark and
Trino query execution—largely due to improved Cost-Based Optimization
(CBO) effectiveness.
However, leveraging the v3 format presented numerous challenges (such
as column-type evolution and the way to save min/max values). We
believe adopting the v4 format would be a more robust solution.
I’ve researched this extensively and applied it in production. I’d be
glad to collaborate on implementing this feature if needed.
Best wishes.
Gábor Kaszab <ga...@apache.org> 于2025年8月28日周四 21:23写道：
>
> Hey Iceberg Community,
>
> I've been working on a proposal to extend the currently standardized statistics in Iceberg, by looking into what statistics are used by some query engines and trying to fill the gaps (credit also goes to Denys K to lay groundwork). The motivation is to use Iceberg for the source of truth when it comes to statistics across all the engines.
> Meanwhile, there have been movements on other proposals (Restructuring col-stats, Restructuring metadata) that might overlap with mine. Let’s see how much of my proposal still holds up in light of these developments.
>
> Any feedback is appreciated!
> Gabor