You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Gábor Kaszab <ga...@apache.org> on 2025/07/07 13:21:16 UTC
Re: [DISCUSS] v4 - Improved column statistics
+1 Seems a great improvement! Let me know if I can help out with
implementation, measurements, etc.!
Regards,
Gabor Kaszab
John Zhuge <jz...@apache.org> ezt írta (időpont: 2025. jún. 5., Cs, 23:41):
> +1 Looking forward to this feature
>
> John Zhuge
>
>
> On Thu, Jun 5, 2025 at 2:22 PM Ryan Blue <rd...@gmail.com> wrote:
>
>> > I think it does not make sense to stick manifest files to Avro if we
>> break column stats into sub fields.
>>
>> This isn't necessarily true. Avro can benefit from better pushdown with
>> Eduard's approach as well by being able to skip more efficiently. With the
>> current layout, Avro stores a list of key/value pairs that are all
>> projected and put into a map. We avoid decoding the values, but each field
>> ID is decoded, then the length of the value is decoded, and finally there
>> is a put operation with an ID and value ByteBuffer pair. With the new
>> approach, we will be able to know which fields are relevant and skip
>> unprojected fields based on the file schema, which we couldn't do before.
>>
>> To skip stats for an unused field (not part of the filter), there are two
>> cases. Lower/upper bounds for types that are fixed width are skipped by
>> updating the read position. And bounds for types that are variable length
>> (strings and binary) are skipped by reading the length and skipping that
>> number of bytes.
>>
>> It turns out that actually producing the metric maps is a fairly
>> expensive operation, so being able to skip metrics more quickly even if the
>> bytes still have to be read is going to save time. That said, using a
>> columnar format is still going to be a good idea!
>>
>> On Wed, Jun 4, 2025 at 11:22 PM Gang Wu <us...@gmail.com> wrote:
>>
>>> > Together with the change which allows storing metadata in columnar
>>> formats
>>>
>>> +1 on this. I think it does not make sense to stick manifest files to
>>> Avro if we break column stats into sub fields.
>>>
>>> On Tue, Jun 3, 2025 at 7:19 PM Péter Váry <pe...@gmail.com>
>>> wrote:
>>>
>>>> I would love to see more flexibility in file stats. Together with the
>>>> change which allows storing metadata in columnar formats will open up many
>>>> new possibilities. Bloom filters in metadata which could be used for
>>>> filtering out files, HLL scratches etc....
>>>>
>>>> +1 for the change
>>>>
>>>> On Tue, Jun 3, 2025, 08:12 Szehon Ho <sz...@gmail.com> wrote:
>>>>
>>>>> +1 , excited for this one too, we've seen the current metrics maps
>>>>> blow up the memory and hope can improve that.
>>>>>
>>>>> On the Geo front, this could allow us to add supplementary metrics
>>>>> that don't conform to the geo type, like S2 Cell Ids.
>>>>>
>>>>> Thanks
>>>>> Szehon
>>>>>
>>>>> On Mon, Jun 2, 2025 at 6:14 AM Eduard Tudenhöfner <
>>>>> etudenhoefner@apache.org> wrote:
>>>>>
>>>>>> Hey everyone,
>>>>>>
>>>>>> I'm starting a thread to connect folks interested in improving the
>>>>>> existing way of collecting column-level statistics (often referred to as
>>>>>> *metrics* in the code). I've already started a proposal, which can
>>>>>> be found at https://s.apache.org/iceberg-column-stats.
>>>>>>
>>>>>> *Motivation*
>>>>>>
>>>>>> Column statistics are currently stored as a mapping of field id to
>>>>>> values across multiple columns (lower/upper bounds, value/nan/null
>>>>>> counts, sizes). This storage model has critical limitations as the
>>>>>> number of columns increases and as new types are being added to Iceberg:
>>>>>>
>>>>>> -
>>>>>>
>>>>>> Inefficient Storage due to map-based structure:
>>>>>> -
>>>>>>
>>>>>> Large memory overhead during planning/processing
>>>>>> -
>>>>>>
>>>>>> Inability to project specific stats (e.g., only
>>>>>> null_value_counts for column X)
>>>>>> -
>>>>>>
>>>>>> Type Erasure: Original logical/physical types are lost when
>>>>>> stored as binary blobs, causing:
>>>>>> -
>>>>>>
>>>>>> Lossy type inference during reads
>>>>>> - Schema evolution challenges (e.g., widening types)
>>>>>> - Rigid Schema: Stats are tied to the data_fil entry record,
>>>>>> limiting extensibility for new stats.
>>>>>>
>>>>>>
>>>>>> *Goals*
>>>>>>
>>>>>> Improve the column stats representation to allow for the following:
>>>>>>
>>>>>> -
>>>>>>
>>>>>> Projectability: Enable independent access to specific stats
>>>>>> (e.g., lower_bounds without loading upper_bounds).
>>>>>> -
>>>>>>
>>>>>> Type Preservation: Store original data types to support accurate
>>>>>> reads and schema evolution.
>>>>>> -
>>>>>>
>>>>>> Flexible/Extensible Representation: Allow per-field stats
>>>>>> structures (e.g., complex types like Geo/Variant).
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>> Eduard
>>>>>>
>>>>>
Re: [DISCUSS] v4 - Improved column statistics
Posted by Micah Kornfield <em...@gmail.com>.
>
> If we include only positive integers for IDs then the max ID is 2^31-1 =
> 2,147,483,647. The last 200 fields are reserved and this is proposing to
> skip the first 10,000 IDs for other structures in the manifest file schema.
> That can accommodate more than 10.7 million field IDs from the table space
> (= (2,147,483,647 - 10,200) / 200). I think that is a reasonable upper
> bound for the number of table columns.
I'm still not clear on what the proposal is to handle stats for reserved
columns <https://iceberg.apache.org/spec/#reserved-field-ids> [1] (I think
there was some mention in the notes but it was light on details). It seems
like it would be potentially useful to have stats for things like _row_id,
and the multiplication would overflow for these column IDs (maybe this
still yields unique column IDs though?)
> 1. The current proposal only leaves 10000+200 ids for other columns
> than stats. If in the future, we find some other feature which would
> require a manifest file column for every data column in the table, then we
> would need to change the spec.
>
> I assume we could put whatever these columns are under stats? Maybe we
just need a more generic name for the top level struct?
2. The current proposal expects every engine to share the same stats, and
> not store any "non-standard" stat in the metadata.
I think there was some discussion on whether this should be supported (I'd
lean in favor of trying to do so). I think the two concerns would be:
a. Additional implementation complexity to carry forward these columns (we
would need to prototype to see) for manifest compaction.
b. This would potentially break for writers that don't know about these
stats doing table optimizations but this seems more of concern for people
writing these stats.
Thanks,
Micah
[1] https://iceberg.apache.org/spec/#reserved-field-ids
On Wed, Jul 23, 2025 at 4:21 AM Péter Váry <pe...@gmail.com>
wrote:
> I also like the assigned field ids, my only concern is about the way of
> generating them.
>
> I have 2 main issues:
>
> 1. The current proposal only leaves 10000+200 ids for other columns
> than stats. If in the future, we find some other feature which would
> require a manifest file column for every data column in the table, then we
> would need to change the spec.
> 2. The current proposal expects every engine to share the same stats,
> and not store any "non-standard" stat in the metadata.
>
> I think 1st could be solved by simply reserving a bigger number of
> fieldIds whatever algorithm we choose.
> I understand that the 2nd is not an immediate goal at this point, but I
> don't see how we can support this in the future if we use only ids for
> identifying the stat fields.
>
>
> Steven Wu <st...@gmail.com> ezt írta (időpont: 2025. júl. 22., K,
> 20:21):
>
>> It seems reasonable to support stats for computed/calculated columns with
>> assigned field ids.
>>
>> E.g., Flink has "computed columns" for a long time.
>>
>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/#columns
>>
>> CREATE TABLE MyTable ( `user_id` BIGINT, `price` DOUBLE, `quantity` DOUBLE, `cost` AS price * quantity -- evaluate expression and supply the result to queries) WITH ( 'connector' = 'kafka' ...);
>>
>>
>> On Tue, Jul 22, 2025 at 10:56 AM Russell Spitzer <
>> russell.spitzer@gmail.com> wrote:
>>
>>> As long as enough folks are on board for assigning table field-id's to
>>> non-physical columns I have no problem with that approach.
>>>
>>> On Tue, Jul 22, 2025 at 12:34 PM Ryan Blue <rd...@gmail.com> wrote:
>>>
>>>> > I still think that the current `10000 + 200 * data_field_id` id
>>>> generation logic is very limiting. I have already seen tables with more
>>>> than 10k columns for AI space. And I don't think we would like to create
>>>> hard limits here.
>>>>
>>>> If we include only positive integers for IDs then the max ID is 2^31-1
>>>> = 2,147,483,647. The last 200 fields are reserved and this is proposing to
>>>> skip the first 10,000 IDs for other structures in the manifest file schema.
>>>> That can accommodate more than 10.7 million field IDs from the table space
>>>> (= (2,147,483,647 - 10,200) / 200). I think that is a reasonable upper
>>>> bound for the number of table columns.
>>>>
>>>> > I think this would probably a good opportunity to also reserve space
>>>> for metrics which apply to the Sort Order transformation used to create a
>>>> file.
>>>>
>>>> I've also been thinking about being able to store stats for other
>>>> expressions in this space, including partition values that are part of
>>>> Amogh's one-file commit and adaptive metadata proposal. Another use case is
>>>> keeping stats for derived values, like `to_lower(string_col)`. What I'd
>>>> propose is being able to track expressions that are assigned an ID from the
>>>> table column space, then storing the expressions stats according to the
>>>> structure proposed here. But I wouldn't over-complicate the current
>>>> proposal by adding this just yet. We can talk about extending stats to
>>>> expressions in another proposal once we have the structure and ID
>>>> assignment done.
>>>>
>>>> On Tue, Jul 22, 2025 at 9:59 AM Russell Spitzer <
>>>> russell.spitzer@gmail.com> wrote:
>>>>
>>>>> I'm also sorry I missed the discussion because I was busy trying to
>>>>> keep a nearly-3 year old occupied on a plane :)
>>>>>
>>>>> I think the proposal is pretty strong although I have one request.
>>>>> Currently, we have the ability to note the sort order which
>>>>> was applied to a data file but we have know way of knowing the
>>>>> statistics of that applied transformation for said data file.
>>>>> This has stopped us from doing a number of optimizations (like
>>>>> combining files which are adjacent based on any complex
>>>>> sort ordering). I think this would probably a good opportunity to also
>>>>> reserve space for metrics which apply to the Sort Order
>>>>> transformation used to create a file. This would still only be using
>>>>> things that are already codified in the spec but
>>>>> would make it possible for engines to use those transforms for further
>>>>> predicate pushdown or optimization
>>>>> in file compaction.
>>>>>
>>>>> A quick example
>>>>>
>>>>> Say I am using a hierarchical sort order (A, B, C)
>>>>> I could then store the max, min of this transform which would be
>>>>> independent to individual column maxes
>>>>> Say
>>>>> A : Min 1 A: Max 2
>>>>> B: Min 1 B: Max 100000
>>>>> C: Min 1 C: Max 100000
>>>>>
>>>>> A,B,C Min : (1,7000,32)
>>>>> A,B,C Max : (2, 1, 100000)
>>>>>
>>>>> In this case if I'm looking for a record 2, 2, 4, I can instantly
>>>>> reject using the sort order transform where if I was
>>>>> using the individual columns I would have to read the file.
>>>>>
>>>>> This is of course also useful if the sort order is using some kind of
>>>>> space filling curve or other clustering algorithm.
>>>>>
>>>>> Thanks for your hard work,
>>>>> Russ
>>>>>
>>>>> On Thu, Jul 17, 2025 at 5:07 AM Péter Váry <
>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>
>>>>>> Hi Team,
>>>>>> Sorry, but I was not able to join the discussion on Tuesday :(, but I
>>>>>> listened to the recording.
>>>>>>
>>>>>> A few thoughts:
>>>>>> - I still think that the current `10000 + 200 * data_field_id` id
>>>>>> generation logic is very limiting. I have already seen tables with more
>>>>>> than 10k columns for AI space. And I don't think we would like to create
>>>>>> hard limits here.
>>>>>> - If we follow store a few things in metadata (starting_id,
>>>>>> stats_types_present, columns_with_stats), then we can greatly reduce the
>>>>>> used id space, and also allow for extensibility, as the engines which
>>>>>> doesn't know a specific stats_type could just ignore them)
>>>>>>
>>>>>> Thanks Eduard and Dan for driving this!
>>>>>> Peter
>>>>>>
>>>>>> Eduard Tudenhöfner <et...@apache.org> ezt írta (időpont:
>>>>>> 2025. júl. 16., Sze, 7:53):
>>>>>>
>>>>>>> Hey everyone,
>>>>>>>
>>>>>>> We met yesterday and talked about the column stats proposal.
>>>>>>> Please find the recording here
>>>>>>> <https://drive.google.com/file/d/1WVpSg9XxipO5NzogDc7D4DMsnj9cCMlF/view?usp=sharing>
>>>>>>> and the notes here
>>>>>>> <https://docs.google.com/document/d/1s9_o_Y8js4kHVCYI2OeL0Yh3Ey0qfkuE7Mm-z-QoEfo/edit?usp=sharing>
>>>>>>> .
>>>>>>>
>>>>>>> Thanks everyone,
>>>>>>> Eduard
>>>>>>>
>>>>>>> On Tue, Jul 8, 2025 at 6:51 PM Eduard Tudenhöfner <
>>>>>>> etudenhoefner@apache.org> wrote:
>>>>>>>
>>>>>>>> Hey everyone,
>>>>>>>>
>>>>>>>> I've just added an event to the dev calendar for July 15 at 9am
>>>>>>>> (PT) to discuss the column stats proposal.
>>>>>>>>
>>>>>>>>
>>>>>>>> Eduard
>>>>>>>>
>>>>>>>> On Tue, Jul 8, 2025 at 4:09 AM Jacky Lee <qc...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> +1 for the wonderful feature. Please count me in if you need any
>>>>>>>>> help.
>>>>>>>>>
>>>>>>>>> Gábor Kaszab <ga...@apache.org> 于2025年7月7日周一 21:22写道：
>>>>>>>>> >
>>>>>>>>> > +1 Seems a great improvement! Let me know if I can help out with
>>>>>>>>> implementation, measurements, etc.!
>>>>>>>>> >
>>>>>>>>> > Regards,
>>>>>>>>> > Gabor Kaszab
>>>>>>>>> >
>>>>>>>>> > John Zhuge <jz...@apache.org> ezt írta (időpont: 2025. jún.
>>>>>>>>> 5., Cs, 23:41):
>>>>>>>>> >>
>>>>>>>>> >> +1 Looking forward to this feature
>>>>>>>>> >>
>>>>>>>>> >> John Zhuge
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> On Thu, Jun 5, 2025 at 2:22 PM Ryan Blue <rd...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>> >>>
>>>>>>>>> >>> > I think it does not make sense to stick manifest files to
>>>>>>>>> Avro if we break column stats into sub fields.
>>>>>>>>> >>>
>>>>>>>>> >>> This isn't necessarily true. Avro can benefit from better
>>>>>>>>> pushdown with Eduard's approach as well by being able to skip more
>>>>>>>>> efficiently. With the current layout, Avro stores a list of key/value pairs
>>>>>>>>> that are all projected and put into a map. We avoid decoding the values,
>>>>>>>>> but each field ID is decoded, then the length of the value is decoded, and
>>>>>>>>> finally there is a put operation with an ID and value ByteBuffer pair. With
>>>>>>>>> the new approach, we will be able to know which fields are relevant and
>>>>>>>>> skip unprojected fields based on the file schema, which we couldn't do
>>>>>>>>> before.
>>>>>>>>> >>>
>>>>>>>>> >>> To skip stats for an unused field (not part of the filter),
>>>>>>>>> there are two cases. Lower/upper bounds for types that are fixed width are
>>>>>>>>> skipped by updating the read position. And bounds for types that are
>>>>>>>>> variable length (strings and binary) are skipped by reading the length and
>>>>>>>>> skipping that number of bytes.
>>>>>>>>> >>>
>>>>>>>>> >>> It turns out that actually producing the metric maps is a
>>>>>>>>> fairly expensive operation, so being able to skip metrics more quickly even
>>>>>>>>> if the bytes still have to be read is going to save time. That said, using
>>>>>>>>> a columnar format is still going to be a good idea!
>>>>>>>>> >>>
>>>>>>>>> >>> On Wed, Jun 4, 2025 at 11:22 PM Gang Wu <us...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>> >>>>
>>>>>>>>> >>>> > Together with the change which allows storing metadata in
>>>>>>>>> columnar formats
>>>>>>>>> >>>>
>>>>>>>>> >>>> +1 on this. I think it does not make sense to stick manifest
>>>>>>>>> files to Avro if we break column stats into sub fields.
>>>>>>>>> >>>>
>>>>>>>>> >>>> On Tue, Jun 3, 2025 at 7:19 PM Péter Váry <
>>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> I would love to see more flexibility in file stats. Together
>>>>>>>>> with the change which allows storing metadata in columnar formats will open
>>>>>>>>> up many new possibilities. Bloom filters in metadata which could be used
>>>>>>>>> for filtering out files, HLL scratches etc....
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> +1 for the change
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> On Tue, Jun 3, 2025, 08:12 Szehon Ho <
>>>>>>>>> szehon.apache@gmail.com> wrote:
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> +1 , excited for this one too, we've seen the current
>>>>>>>>> metrics maps blow up the memory and hope can improve that.
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> On the Geo front, this could allow us to add supplementary
>>>>>>>>> metrics that don't conform to the geo type, like S2 Cell Ids.
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> Thanks
>>>>>>>>> >>>>>> Szehon
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> On Mon, Jun 2, 2025 at 6:14 AM Eduard Tudenhöfner <
>>>>>>>>> etudenhoefner@apache.org> wrote:
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Hey everyone,
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> I'm starting a thread to connect folks interested in
>>>>>>>>> improving the existing way of collecting column-level statistics (often
>>>>>>>>> referred to as metrics in the code). I've already started a proposal, which
>>>>>>>>> can be found at https://s.apache.org/iceberg-column-stats.
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Motivation
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Column statistics are currently stored as a mapping of
>>>>>>>>> field id to values across multiple columns (lower/upper bounds,
>>>>>>>>> value/nan/null counts, sizes). This storage model has critical limitations
>>>>>>>>> as the number of columns increases and as new types are being added to
>>>>>>>>> Iceberg:
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Inefficient Storage due to map-based structure:
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Large memory overhead during planning/processing
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Inability to project specific stats (e.g., only
>>>>>>>>> null_value_counts for column X)
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Type Erasure: Original logical/physical types are lost
>>>>>>>>> when stored as binary blobs, causing:
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Lossy type inference during reads
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Schema evolution challenges (e.g., widening types)
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Rigid Schema: Stats are tied to the data_fil entry record,
>>>>>>>>> limiting extensibility for new stats.
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Goals
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Improve the column stats representation to allow for the
>>>>>>>>> following:
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Projectability: Enable independent access to specific
>>>>>>>>> stats (e.g., lower_bounds without loading upper_bounds).
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Type Preservation: Store original data types to support
>>>>>>>>> accurate reads and schema evolution.
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Flexible/Extensible Representation: Allow per-field stats
>>>>>>>>> structures (e.g., complex types like Geo/Variant).
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Thanks
>>>>>>>>> >>>>>>> Eduard
>>>>>>>>>
>>>>>>>>
Re: [DISCUSS] v4 - Improved column statistics
Posted by Eduard Tudenhöfner <et...@apache.org>.
Hey everyone,
It's been a while since the last update but I just wanted to raise
awareness that we've been iterating on the Content Stats Spec changes in
#14234 <https://github.com/apache/iceberg/pull/14234>.
Please take a look and let me know your thoughts.
Thanks
Eduard
On Fri, Sep 19, 2025 at 9:31 AM Eduard Tudenhöfner <et...@apache.org>
wrote:
> Hey everyone,
>
> I have updated the proposal
> <https://docs.google.com/document/d/1uvbrwwAJW2TgsnoaIcwAFpjbhHkBUL5wY_24nKgtt9I/edit?tab=t.0#heading=h.hs6r9d26w1y2>
> with the following things:
>
> - removed *column_size*, since this hasn't been used anywhere in
> earlier versions. Please shout if you think we should keep this going
> forward.
> - added *avg_value_size* and *max_value_size* for avg/max value sizes
> of variable-length types (string/binary)
> - the examples in the proposal were using *1_417_000_000* as the
> starting stats ID for the reserved field ID space, but that should have
> been *2_147_000_000* because we have 200 reserved IDs * 200 stats
> types = 40k and using *2_147_000_000* leaves enough room in case we
> decide to add other ID spaces
>
> If people are ok then I think we should be able to vote on the design
> proposal so that we could get the first portions of the code
> <https://github.com/apache/iceberg/pull/13933> in, which would allow
> parallelizing downstream work on this
>
>
> Thanks
> Eduard
>
> On Wed, Aug 20, 2025 at 3:05 PM Eduard Tudenhöfner <
> etudenhoefner@apache.org> wrote:
>
>> Hey everyone,
>>
>> We met yesterday and talked about some details around the stats proposal.
>>
>> Please find the notes here
>> <https://docs.google.com/document/d/1ZK5g8_bA1Y9SQ4UA5jAREX9iNX56xLWA5vAuKpQC4L8/edit?usp=sharing>
>> and the recording here
>> <https://drive.google.com/file/d/1YIILCIhDbgu3OYlMn5KNChsYFP8rGPPX/view?usp=sharing>
>> .
>>
>> I have updated the proposal <https://s.apache.org/iceberg-column-stats>
>> with the following points:
>>
>> - added a table schema example with a detailed stats schema
>> - updated wording to make it clear that projection is always by ID
>> and the field name of a stats field should not be relied on
>> - added a table that defines current field stats types with their
>> respective offsets from the field ID of the base stats struct
>> - updated wording to make it clear that stats are calculated for
>> assigned field IDs that are
>> - defined in the table ID space (Amogh is working on a separate
>> proposal to unify ID spaces)
>> - defined in the reserved field ID
>> <https://iceberg.apache.org/spec/#reserved-field-ids> space
>> - added some examples showing table ID -> stats ID of stats struct
>> and also the stats ID of individual stats fields
>> - updated wording to explain how variant stats would look in the new
>> stats structure
>> - updated wording to make it clear that custom stats are not
>> supported and that expressions are the preferred way
>>
>> Please let me know in case I missed anything else to include.
>>
>> Thanks everyone for participating,
>>
>> Eduard
>>
>>
>>
Re: [DISCUSS] v4 - Improved column statistics
Posted by Eduard Tudenhöfner <et...@apache.org>.
Hey everyone,
I have updated the proposal
<https://docs.google.com/document/d/1uvbrwwAJW2TgsnoaIcwAFpjbhHkBUL5wY_24nKgtt9I/edit?tab=t.0#heading=h.hs6r9d26w1y2>
with the following things:
- removed *column_size*, since this hasn't been used anywhere in earlier
versions. Please shout if you think we should keep this going forward.
- added *avg_value_size* and *max_value_size* for avg/max value sizes of
variable-length types (string/binary)
- the examples in the proposal were using *1_417_000_000* as the
starting stats ID for the reserved field ID space, but that should have
been *2_147_000_000* because we have 200 reserved IDs * 200 stats types
= 40k and using *2_147_000_000* leaves enough room in case we decide to
add other ID spaces
If people are ok then I think we should be able to vote on the design
proposal so that we could get the first portions of the code
<https://github.com/apache/iceberg/pull/13933> in, which would allow
parallelizing downstream work on this
Thanks
Eduard
On Wed, Aug 20, 2025 at 3:05 PM Eduard Tudenhöfner <et...@apache.org>
wrote:
> Hey everyone,
>
> We met yesterday and talked about some details around the stats proposal.
>
> Please find the notes here
> <https://docs.google.com/document/d/1ZK5g8_bA1Y9SQ4UA5jAREX9iNX56xLWA5vAuKpQC4L8/edit?usp=sharing>
> and the recording here
> <https://drive.google.com/file/d/1YIILCIhDbgu3OYlMn5KNChsYFP8rGPPX/view?usp=sharing>
> .
>
> I have updated the proposal <https://s.apache.org/iceberg-column-stats>
> with the following points:
>
> - added a table schema example with a detailed stats schema
> - updated wording to make it clear that projection is always by ID and
> the field name of a stats field should not be relied on
> - added a table that defines current field stats types with their
> respective offsets from the field ID of the base stats struct
> - updated wording to make it clear that stats are calculated for
> assigned field IDs that are
> - defined in the table ID space (Amogh is working on a separate
> proposal to unify ID spaces)
> - defined in the reserved field ID
> <https://iceberg.apache.org/spec/#reserved-field-ids> space
> - added some examples showing table ID -> stats ID of stats struct and
> also the stats ID of individual stats fields
> - updated wording to explain how variant stats would look in the new
> stats structure
> - updated wording to make it clear that custom stats are not supported
> and that expressions are the preferred way
>
> Please let me know in case I missed anything else to include.
>
> Thanks everyone for participating,
>
> Eduard
>
>
>
Re: [DISCUSS] v4 - Improved column statistics
Posted by Eduard Tudenhöfner <et...@apache.org>.
Hey everyone,
We met yesterday and talked about some details around the stats proposal.
Please find the notes here
<https://docs.google.com/document/d/1ZK5g8_bA1Y9SQ4UA5jAREX9iNX56xLWA5vAuKpQC4L8/edit?usp=sharing>
and the recording here
<https://drive.google.com/file/d/1YIILCIhDbgu3OYlMn5KNChsYFP8rGPPX/view?usp=sharing>
.
I have updated the proposal <https://s.apache.org/iceberg-column-stats>
with the following points:
- added a table schema example with a detailed stats schema
- updated wording to make it clear that projection is always by ID and
the field name of a stats field should not be relied on
- added a table that defines current field stats types with their
respective offsets from the field ID of the base stats struct
- updated wording to make it clear that stats are calculated for
assigned field IDs that are
- defined in the table ID space (Amogh is working on a separate
proposal to unify ID spaces)
- defined in the reserved field ID
<https://iceberg.apache.org/spec/#reserved-field-ids> space
- added some examples showing table ID -> stats ID of stats struct and
also the stats ID of individual stats fields
- updated wording to explain how variant stats would look in the new
stats structure
- updated wording to make it clear that custom stats are not supported
and that expressions are the preferred way
Please let me know in case I missed anything else to include.
Thanks everyone for participating,
Eduard
Re: [DISCUSS] v4 - Improved column statistics
Posted by Eduard Tudenhöfner <et...@apache.org>.
Hey Micah,
Has the doc been updated to reflect the latest proposal in more detail? Do
> we understand what the open issues are?
>
I haven't updated the proposal itself yet, but yes we understand the open
issues (I tackled almost all of them in my POC and e.g. also made sure that
stats for Variants work properly).
Re: [DISCUSS] v4 - Improved column statistics
Posted by Micah Kornfield <em...@gmail.com>.
>
> I've just added an event to the dev calendar for August 19 at 9am (PT) to
> go over some details of the column stats proposal, such as ID ranges and
> other things.
Has the doc been updated to reflect the latest proposal in more detail? Do
we understand what the open issues are?
If we focus strictly on file-level column statistics, then partition level
> column statistics is not a concern. However, looking ahead, we likely want
> to support column statistics at the table or partition level as well. It
> would be beneficial to adopt a consistent approach to ID generation and
> handling for partition statistics too.
I don't think the current proposal is inconsistent with these different
levels of aggregation (current partition statistics files have even fewer
fields then manifests today). Is there a specific gap you are thinking of?
Thanks,
Micah
On Fri, Aug 15, 2025 at 8:53 AM Eduard Tudenhöfner <et...@apache.org>
wrote:
> Hey everyone,
>
> I've just added an event to the dev calendar for August 19 at 9am (PT) to
> go over some details of the column stats proposal, such as ID ranges and
> other things.
>
> Eduard
>
Re: [DISCUSS] v4 - Improved column statistics
Posted by Eduard Tudenhöfner <et...@apache.org>.
Hey everyone,
I've just added an event to the dev calendar for August 19 at 9am (PT) to
go over some details of the column stats proposal, such as ID ranges and
other things.
Eduard
Re: [DISCUSS] v4 - Improved column statistics
Posted by Péter Váry <pe...@gmail.com>.
If we focus strictly on file-level column statistics, then partition level
column statistics is not a concern. However, looking ahead, we likely want
to support column statistics at the table or partition level as well. It
would be beneficial to adopt a consistent approach to ID generation and
handling for partition statistics too.
Micah Kornfield <em...@gmail.com> ezt írta (időpont: 2025. júl. 24.,
Cs, 23:50):
> Hi Dan,
>
> I largely agree that expressions will be useful and would limit the need
> for "custom stats". I just wanted to probe some of what some of the points
> you made since I think there might be some important distinctions that
> might be getting glossed over.
>
>
>> The requirement would just be that you need to project all the stats for
>> an expression/sort order when copying metadata entries (though I may be
>> trivializing this and it's harder than I expect). I think the issue with a
>> low-bar to add new stats is basically the same effort as saying you need to
>> support arbitrary/unknown stats carry-over since older clients would either
>> have to handle the unknown cases or would end up dropping values.
>
>
> I think we should disambiguate two cases:
> 1. Custom stats. In this case I assume whoever is using them has a
> custom writer that will do any carry-over necessary, and won't let
> reference writers touch their table. We shouldn't require reference
> implementations to carry over these stats.
> 2. Official non-required stats. In this case I think the projection is
> entirely known because all possible stats would be enumerated for any given
> version of the spec (i.e. it is different then unknown/arbitrary stats).
> Older clients should never be writing to newer versions of the table if
> they don't understand the version of the spec that is currently used for
> the table. Manifest compaction could still occur fairly easily without
> data loss (i.e. it seems like in this scenario carrying over less used
> stats is the same effort as carrying over stats for expressions)?
>
> What couldn't occur is file compaction/adding new files, but I think we
> have the same problem with custom expressions in this regard.
>
> I'm still open to debate on this but if we need to support
>> expressions/sort-orders it feels like a good path to both handling
>> customization as well as providing a path to standardization if we find
>> specific cases that are commonly reused as expressions.
>
>
> We should probably distinguish between two types of expressions:
> 1. Scalar expressions - i.e. transform a value in a specific way (I
> thought this was the main use case of expressions). Examples:
> Timestamp->Date. String normalization/collation.
> 2. Aggregate expressions - We are transforming N values to 1 value. Note
> that stats are all aggregate expressions.
>
> If aggregates aren't in scope for expressions then I'm not sure they would
> satisfy all custom stats requirements. If they are in scope, this brings
> up the question: do we actually need a specific concept of "stats"? It
> seems all stats could just be modelled as expressions?
>
> Cheers,
> Micah
>
>
>
>
> On Thu, Jul 24, 2025 at 1:38 PM Daniel Weeks <dw...@apache.org> wrote:
>
>> Also off topic, but doesn't this just shift the burden of
>>> standardardization to expressions? This might be controversial but maybe
>>> the bar for adding a new stat type should be relatively low? They are
>>> optional anyways, we can maybe define some stats as core (implementations
>>> are incomplete if they can't produce them) and others as non-core (not
>>> required for implementations, there can be optional configuration to either
>>> block writes that require producing the stats or just drop them).
>>
>>
>> If we need to support stats for expressions/sort-orders, then we've
>> pretty much done the hard work already. The requirement would just be that
>> you need to project all the stats for an expression/sort order when copying
>> metadata entries (though I may be trivializing this and it's harder than I
>> expect). I think the issue with a low-bar to add new stats is basically
>> the same effort as saying you need to support arbitrary/unknown stats
>> carry-over since older clients would either have to handle the unknown
>> cases or would end up dropping values. I think expressions are a better
>> way to handle customization because it wouldn't require the same
>> consistency of representation/interpretation as a formally adopted stat.
>> The expression would then really fall on whatever standard we set for
>> portability which provides more flexibility (yes, it shifts the burden, but
>> we're going to have to figure that out for expressions/udfs/etc anyway).
>>
>> I'm still open to debate on this but if we need to support
>> expressions/sort-orders it feels like a good path to both handling
>> customization as well as providing a path to standardization if we find
>> specific cases that are commonly reused as expressions.
>>
>> -Dan
>>
>> On Thu, Jul 24, 2025 at 12:03 PM Micah Kornfield <em...@gmail.com>
>> wrote:
>>
>>> After having thought about it some more, my current point of view is
>>> proceeding with something as simple as possible for V4 (I tried to
>>> formalize what I think the proposed algorithm is in the original proposal
>>> doc
>>> <https://docs.google.com/document/d/1uvbrwwAJW2TgsnoaIcwAFpjbhHkBUL5wY_24nKgtt9I/edit?tab=t.0> [1]).
>>> If in the course of V4 development we find some flaw with the simple
>>> approach we can revise it (e.g. we run out of space). If something comes
>>> up after V4, we are not talking about a lot of code either way, so having a
>>> new scheme for V5+ would not be a major burden (all manifests are now
>>> written with a spec version, so detection is easy).
>>>
>>> Given the potential complications with custom stats, I think it is
>>> reasonable to allow implementations that want custom stats to use the upper
>>> bound of reserved offset range (e.g. we have 6 reserved out of 200 today,
>>> if implementations really need custom stats then they can start using
>>> offset 199, and then 198, etc). This poses a low risk of overlap in the
>>> short term, and I assume those using custom stats would have tight control
>>> over their environment anyways, so they have the ability to manage
>>> conflicts, compactions, in a way that fits them.
>>>
>>>
>>>> Another thing that both Russel and Ryan brought up is being able to
>>>> track stats for sort orders or expressions, but they don't share an id
>>>> space with field ids.
>>>
>>>
>>> Slightly off topic, but is there a reason we can't unify the field ID
>>> range for V4?
>>>
>>> I feel like it would be better to work to formalize the stats so that
>>>> they are known and easier to project, but it's also hard to get agreement
>>>> for more complicated stats (like coalitions that have very
>>>> specific character set handling), but I think using expressions in lieu of
>>>> custom stats might address all of these cases and would be more
>>>> straightforward for the copy-forward requirement.
>>>
>>>
>>> Also off topic, but doesn't this just shift the burden of
>>> standardardization to expressions? This might be controversial but maybe
>>> the bar for adding a new stat type should be relatively low? They are
>>> optional anyways, we can maybe define some stats as core (implementations
>>> are incomplete if they can't produce them) and others as non-core (not
>>> required for implementations, there can be optional configuration to either
>>> block writes that require producing the stats or just drop them).
>>>
>>> [1]
>>> https://docs.google.com/document/d/1uvbrwwAJW2TgsnoaIcwAFpjbhHkBUL5wY_24nKgtt9I/edit?tab=t.0
>>>
>>>
>>>
>>> On Thu, Jul 24, 2025 at 11:19 AM Daniel Weeks <dw...@apache.org> wrote:
>>>
>>>> The current proposal only leaves 10000+200 ids for other columns than
>>>>> stats. If in the future, we find some other feature which would require a
>>>>> manifest file column for every data column in the table, then we would need
>>>>> to change the spec.
>>>>
>>>>
>>>> I do think we might want to put an upper bound on the column stats.
>>>> Ryan calculated the upper bound of what can be represented, but I don't
>>>> think we need to accommodate 10m+ field ids and that would block the entire
>>>> id range. It might make more sense to simply put an upper bound on the
>>>> stats space (e.g. 100k or 1m fields?). This would leave plenty of space
>>>> for future evolution of the spec without having to redefine the stats range.
>>>>
>>>> Another thing that both Russel and Ryan brought up is being able to
>>>> track stats for sort orders or expressions, but they don't share an id
>>>> space with field ids. We might want to decide what the full stats space
>>>> should look like. For example:
>>>>
>>>> 8k+ sort orders
>>>> 9k+ expressions
>>>> 10+ field ids
>>>> 1m+ <unreserved>
>>>> MAX_VALUE - 200 <reserved per spec>
>>>>
>>>> Since sort orders and expressions have much lower cardinality than
>>>> field ids, we can probably have a more constrained range.
>>>>
>>>> I'm leaning against custom stats because it does increase complexity
>>>> for all writers as Micah mentioned and introduces the potential for id
>>>> space collision. It would also easily compromise the performance of
>>>> engines if other writers drop them (via compaction or just any metadata
>>>> rewrite operation). I feel like it would be better to work to formalize
>>>> the stats so that they are known and easier to project, but it's also hard
>>>> to get agreement for more complicated stats (like coalitions that have very
>>>> specific character set handling), but I think using expressions in lieu of
>>>> custom stats might address all of these cases and would be more
>>>> straightforward for the copy-forward requirement.
>>>>
>>>> -Dan
>>>>
>>>>
>>>>
>>>> On Thu, Jul 24, 2025 at 4:03 AM Eduard Tudenhöfner
>>>> <ed...@databricks.com.invalid> wrote:
>>>>
>>>>>
>>>>>
>>>>>
>>>>>> 1. The current proposal only leaves 10000+200 ids for other
>>>>>> columns than stats. If in the future, we find some other feature which
>>>>>> would require a manifest file column for every data column in the table,
>>>>>> then we would need to change the spec.
>>>>>>
>>>>>> For this I think we could start at *100,000* so that we use *100,000 +
>>>>> 200 * <fieldID>* to calculate the field ID of a given statistic.
>>>>>
>>>>>
>>>>>>
>>>>>> 1. The current proposal expects every engine to share the same
>>>>>> stats, and not store any "non-standard" stat in the metadata.
>>>>>>
>>>>>> We haven't explicitly stated it in the proposal but there were
>>>>> discussions on how to potentially support this and what implications it
>>>>> brings for readers/writers
>>>>>
>>>>>
>>>>> I'm still not clear on what the proposal is to handle stats for reserved
>>>>>> columns <https://iceberg.apache.org/spec/#reserved-field-ids> [1] (I
>>>>>> think there was some mention in the notes but it was light on details). It
>>>>>> seems like it would be potentially useful to have stats for things like
>>>>>> _row_id, and the multiplication would overflow for these column IDs (maybe
>>>>>> this still yields unique column IDs though?)
>>>>>>
>>>>>
>>>>> To handle stats for reserved columns we could start at *2,417,000,000*
>>>>> which should give us enough room to store 200 stats per metadata ID. We
>>>>> would also ensure that those ID ranges for table columns and reserved
>>>>> columns wouldn't overlap.
>>>>>
>>>>>
>>>>> I assume we could put whatever these columns are under stats? Maybe we
>>>>>> just need a more generic name for the top level struct?
>>>>>
>>>>>
>>>>> I haven't updated the proposal yet, but I think renaming
>>>>> *column_stats* to *content_stats* would make sense.
>>>>>
>>>>>
>>>>>
Re: [DISCUSS] v4 - Improved column statistics
Posted by Micah Kornfield <em...@gmail.com>.
Hi Dan,
I largely agree that expressions will be useful and would limit the need
for "custom stats". I just wanted to probe some of what some of the points
you made since I think there might be some important distinctions that
might be getting glossed over.
> The requirement would just be that you need to project all the stats for
> an expression/sort order when copying metadata entries (though I may be
> trivializing this and it's harder than I expect). I think the issue with a
> low-bar to add new stats is basically the same effort as saying you need to
> support arbitrary/unknown stats carry-over since older clients would either
> have to handle the unknown cases or would end up dropping values.
I think we should disambiguate two cases:
1. Custom stats. In this case I assume whoever is using them has a custom
writer that will do any carry-over necessary, and won't let reference
writers touch their table. We shouldn't require reference implementations
to carry over these stats.
2. Official non-required stats. In this case I think the projection is
entirely known because all possible stats would be enumerated for any given
version of the spec (i.e. it is different then unknown/arbitrary stats).
Older clients should never be writing to newer versions of the table if
they don't understand the version of the spec that is currently used for
the table. Manifest compaction could still occur fairly easily without
data loss (i.e. it seems like in this scenario carrying over less used
stats is the same effort as carrying over stats for expressions)?
What couldn't occur is file compaction/adding new files, but I think we
have the same problem with custom expressions in this regard.
I'm still open to debate on this but if we need to support
> expressions/sort-orders it feels like a good path to both handling
> customization as well as providing a path to standardization if we find
> specific cases that are commonly reused as expressions.
We should probably distinguish between two types of expressions:
1. Scalar expressions - i.e. transform a value in a specific way (I
thought this was the main use case of expressions). Examples:
Timestamp->Date. String normalization/collation.
2. Aggregate expressions - We are transforming N values to 1 value. Note
that stats are all aggregate expressions.
If aggregates aren't in scope for expressions then I'm not sure they would
satisfy all custom stats requirements. If they are in scope, this brings
up the question: do we actually need a specific concept of "stats"? It
seems all stats could just be modelled as expressions?
Cheers,
Micah
On Thu, Jul 24, 2025 at 1:38 PM Daniel Weeks <dw...@apache.org> wrote:
> Also off topic, but doesn't this just shift the burden of
>> standardardization to expressions? This might be controversial but maybe
>> the bar for adding a new stat type should be relatively low? They are
>> optional anyways, we can maybe define some stats as core (implementations
>> are incomplete if they can't produce them) and others as non-core (not
>> required for implementations, there can be optional configuration to either
>> block writes that require producing the stats or just drop them).
>
>
> If we need to support stats for expressions/sort-orders, then we've pretty
> much done the hard work already. The requirement would just be that you
> need to project all the stats for an expression/sort order when copying
> metadata entries (though I may be trivializing this and it's harder than I
> expect). I think the issue with a low-bar to add new stats is basically
> the same effort as saying you need to support arbitrary/unknown stats
> carry-over since older clients would either have to handle the unknown
> cases or would end up dropping values. I think expressions are a better
> way to handle customization because it wouldn't require the same
> consistency of representation/interpretation as a formally adopted stat.
> The expression would then really fall on whatever standard we set for
> portability which provides more flexibility (yes, it shifts the burden, but
> we're going to have to figure that out for expressions/udfs/etc anyway).
>
> I'm still open to debate on this but if we need to support
> expressions/sort-orders it feels like a good path to both handling
> customization as well as providing a path to standardization if we find
> specific cases that are commonly reused as expressions.
>
> -Dan
>
> On Thu, Jul 24, 2025 at 12:03 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
>> After having thought about it some more, my current point of view is
>> proceeding with something as simple as possible for V4 (I tried to
>> formalize what I think the proposed algorithm is in the original proposal
>> doc
>> <https://docs.google.com/document/d/1uvbrwwAJW2TgsnoaIcwAFpjbhHkBUL5wY_24nKgtt9I/edit?tab=t.0> [1]).
>> If in the course of V4 development we find some flaw with the simple
>> approach we can revise it (e.g. we run out of space). If something comes
>> up after V4, we are not talking about a lot of code either way, so having a
>> new scheme for V5+ would not be a major burden (all manifests are now
>> written with a spec version, so detection is easy).
>>
>> Given the potential complications with custom stats, I think it is
>> reasonable to allow implementations that want custom stats to use the upper
>> bound of reserved offset range (e.g. we have 6 reserved out of 200 today,
>> if implementations really need custom stats then they can start using
>> offset 199, and then 198, etc). This poses a low risk of overlap in the
>> short term, and I assume those using custom stats would have tight control
>> over their environment anyways, so they have the ability to manage
>> conflicts, compactions, in a way that fits them.
>>
>>
>>> Another thing that both Russel and Ryan brought up is being able to
>>> track stats for sort orders or expressions, but they don't share an id
>>> space with field ids.
>>
>>
>> Slightly off topic, but is there a reason we can't unify the field ID
>> range for V4?
>>
>> I feel like it would be better to work to formalize the stats so that
>>> they are known and easier to project, but it's also hard to get agreement
>>> for more complicated stats (like coalitions that have very
>>> specific character set handling), but I think using expressions in lieu of
>>> custom stats might address all of these cases and would be more
>>> straightforward for the copy-forward requirement.
>>
>>
>> Also off topic, but doesn't this just shift the burden of
>> standardardization to expressions? This might be controversial but maybe
>> the bar for adding a new stat type should be relatively low? They are
>> optional anyways, we can maybe define some stats as core (implementations
>> are incomplete if they can't produce them) and others as non-core (not
>> required for implementations, there can be optional configuration to either
>> block writes that require producing the stats or just drop them).
>>
>> [1]
>> https://docs.google.com/document/d/1uvbrwwAJW2TgsnoaIcwAFpjbhHkBUL5wY_24nKgtt9I/edit?tab=t.0
>>
>>
>>
>> On Thu, Jul 24, 2025 at 11:19 AM Daniel Weeks <dw...@apache.org> wrote:
>>
>>> The current proposal only leaves 10000+200 ids for other columns than
>>>> stats. If in the future, we find some other feature which would require a
>>>> manifest file column for every data column in the table, then we would need
>>>> to change the spec.
>>>
>>>
>>> I do think we might want to put an upper bound on the column stats.
>>> Ryan calculated the upper bound of what can be represented, but I don't
>>> think we need to accommodate 10m+ field ids and that would block the entire
>>> id range. It might make more sense to simply put an upper bound on the
>>> stats space (e.g. 100k or 1m fields?). This would leave plenty of space
>>> for future evolution of the spec without having to redefine the stats range.
>>>
>>> Another thing that both Russel and Ryan brought up is being able to
>>> track stats for sort orders or expressions, but they don't share an id
>>> space with field ids. We might want to decide what the full stats space
>>> should look like. For example:
>>>
>>> 8k+ sort orders
>>> 9k+ expressions
>>> 10+ field ids
>>> 1m+ <unreserved>
>>> MAX_VALUE - 200 <reserved per spec>
>>>
>>> Since sort orders and expressions have much lower cardinality than field
>>> ids, we can probably have a more constrained range.
>>>
>>> I'm leaning against custom stats because it does increase complexity for
>>> all writers as Micah mentioned and introduces the potential for id space
>>> collision. It would also easily compromise the performance of engines if
>>> other writers drop them (via compaction or just any metadata rewrite
>>> operation). I feel like it would be better to work to formalize the stats
>>> so that they are known and easier to project, but it's also hard to get
>>> agreement for more complicated stats (like coalitions that have very
>>> specific character set handling), but I think using expressions in lieu of
>>> custom stats might address all of these cases and would be more
>>> straightforward for the copy-forward requirement.
>>>
>>> -Dan
>>>
>>>
>>>
>>> On Thu, Jul 24, 2025 at 4:03 AM Eduard Tudenhöfner
>>> <ed...@databricks.com.invalid> wrote:
>>>
>>>>
>>>>
>>>>
>>>>> 1. The current proposal only leaves 10000+200 ids for other
>>>>> columns than stats. If in the future, we find some other feature which
>>>>> would require a manifest file column for every data column in the table,
>>>>> then we would need to change the spec.
>>>>>
>>>>> For this I think we could start at *100,000* so that we use *100,000 +
>>>> 200 * <fieldID>* to calculate the field ID of a given statistic.
>>>>
>>>>
>>>>>
>>>>> 1. The current proposal expects every engine to share the same
>>>>> stats, and not store any "non-standard" stat in the metadata.
>>>>>
>>>>> We haven't explicitly stated it in the proposal but there were
>>>> discussions on how to potentially support this and what implications it
>>>> brings for readers/writers
>>>>
>>>>
>>>> I'm still not clear on what the proposal is to handle stats for reserved
>>>>> columns <https://iceberg.apache.org/spec/#reserved-field-ids> [1] (I
>>>>> think there was some mention in the notes but it was light on details). It
>>>>> seems like it would be potentially useful to have stats for things like
>>>>> _row_id, and the multiplication would overflow for these column IDs (maybe
>>>>> this still yields unique column IDs though?)
>>>>>
>>>>
>>>> To handle stats for reserved columns we could start at *2,417,000,000*
>>>> which should give us enough room to store 200 stats per metadata ID. We
>>>> would also ensure that those ID ranges for table columns and reserved
>>>> columns wouldn't overlap.
>>>>
>>>>
>>>> I assume we could put whatever these columns are under stats? Maybe we
>>>>> just need a more generic name for the top level struct?
>>>>
>>>>
>>>> I haven't updated the proposal yet, but I think renaming *column_stats*
>>>> to *content_stats* would make sense.
>>>>
>>>>
>>>>
Re: [DISCUSS] v4 - Improved column statistics
Posted by Daniel Weeks <dw...@apache.org>.
>
> Also off topic, but doesn't this just shift the burden of
> standardardization to expressions? This might be controversial but maybe
> the bar for adding a new stat type should be relatively low? They are
> optional anyways, we can maybe define some stats as core (implementations
> are incomplete if they can't produce them) and others as non-core (not
> required for implementations, there can be optional configuration to either
> block writes that require producing the stats or just drop them).
If we need to support stats for expressions/sort-orders, then we've pretty
much done the hard work already. The requirement would just be that you
need to project all the stats for an expression/sort order when copying
metadata entries (though I may be trivializing this and it's harder than I
expect). I think the issue with a low-bar to add new stats is basically
the same effort as saying you need to support arbitrary/unknown stats
carry-over since older clients would either have to handle the unknown
cases or would end up dropping values. I think expressions are a better
way to handle customization because it wouldn't require the same
consistency of representation/interpretation as a formally adopted stat.
The expression would then really fall on whatever standard we set for
portability which provides more flexibility (yes, it shifts the burden, but
we're going to have to figure that out for expressions/udfs/etc anyway).
I'm still open to debate on this but if we need to support
expressions/sort-orders it feels like a good path to both handling
customization as well as providing a path to standardization if we find
specific cases that are commonly reused as expressions.
-Dan
On Thu, Jul 24, 2025 at 12:03 PM Micah Kornfield <em...@gmail.com>
wrote:
> After having thought about it some more, my current point of view is
> proceeding with something as simple as possible for V4 (I tried to
> formalize what I think the proposed algorithm is in the original proposal
> doc
> <https://docs.google.com/document/d/1uvbrwwAJW2TgsnoaIcwAFpjbhHkBUL5wY_24nKgtt9I/edit?tab=t.0> [1]).
> If in the course of V4 development we find some flaw with the simple
> approach we can revise it (e.g. we run out of space). If something comes
> up after V4, we are not talking about a lot of code either way, so having a
> new scheme for V5+ would not be a major burden (all manifests are now
> written with a spec version, so detection is easy).
>
> Given the potential complications with custom stats, I think it is
> reasonable to allow implementations that want custom stats to use the upper
> bound of reserved offset range (e.g. we have 6 reserved out of 200 today,
> if implementations really need custom stats then they can start using
> offset 199, and then 198, etc). This poses a low risk of overlap in the
> short term, and I assume those using custom stats would have tight control
> over their environment anyways, so they have the ability to manage
> conflicts, compactions, in a way that fits them.
>
>
>> Another thing that both Russel and Ryan brought up is being able to track
>> stats for sort orders or expressions, but they don't share an id space with
>> field ids.
>
>
> Slightly off topic, but is there a reason we can't unify the field ID
> range for V4?
>
> I feel like it would be better to work to formalize the stats so that they
>> are known and easier to project, but it's also hard to get agreement for
>> more complicated stats (like coalitions that have very specific character
>> set handling), but I think using expressions in lieu of custom stats might
>> address all of these cases and would be more straightforward for the
>> copy-forward requirement.
>
>
> Also off topic, but doesn't this just shift the burden of
> standardardization to expressions? This might be controversial but maybe
> the bar for adding a new stat type should be relatively low? They are
> optional anyways, we can maybe define some stats as core (implementations
> are incomplete if they can't produce them) and others as non-core (not
> required for implementations, there can be optional configuration to either
> block writes that require producing the stats or just drop them).
>
> [1]
> https://docs.google.com/document/d/1uvbrwwAJW2TgsnoaIcwAFpjbhHkBUL5wY_24nKgtt9I/edit?tab=t.0
>
>
>
> On Thu, Jul 24, 2025 at 11:19 AM Daniel Weeks <dw...@apache.org> wrote:
>
>> The current proposal only leaves 10000+200 ids for other columns than
>>> stats. If in the future, we find some other feature which would require a
>>> manifest file column for every data column in the table, then we would need
>>> to change the spec.
>>
>>
>> I do think we might want to put an upper bound on the column stats. Ryan
>> calculated the upper bound of what can be represented, but I don't think we
>> need to accommodate 10m+ field ids and that would block the entire id
>> range. It might make more sense to simply put an upper bound on the stats
>> space (e.g. 100k or 1m fields?). This would leave plenty of space for
>> future evolution of the spec without having to redefine the stats range.
>>
>> Another thing that both Russel and Ryan brought up is being able to track
>> stats for sort orders or expressions, but they don't share an id space with
>> field ids. We might want to decide what the full stats space should look
>> like. For example:
>>
>> 8k+ sort orders
>> 9k+ expressions
>> 10+ field ids
>> 1m+ <unreserved>
>> MAX_VALUE - 200 <reserved per spec>
>>
>> Since sort orders and expressions have much lower cardinality than field
>> ids, we can probably have a more constrained range.
>>
>> I'm leaning against custom stats because it does increase complexity for
>> all writers as Micah mentioned and introduces the potential for id space
>> collision. It would also easily compromise the performance of engines if
>> other writers drop them (via compaction or just any metadata rewrite
>> operation). I feel like it would be better to work to formalize the stats
>> so that they are known and easier to project, but it's also hard to get
>> agreement for more complicated stats (like coalitions that have very
>> specific character set handling), but I think using expressions in lieu of
>> custom stats might address all of these cases and would be more
>> straightforward for the copy-forward requirement.
>>
>> -Dan
>>
>>
>>
>> On Thu, Jul 24, 2025 at 4:03 AM Eduard Tudenhöfner
>> <ed...@databricks.com.invalid> wrote:
>>
>>>
>>>
>>>
>>>> 1. The current proposal only leaves 10000+200 ids for other columns
>>>> than stats. If in the future, we find some other feature which would
>>>> require a manifest file column for every data column in the table, then we
>>>> would need to change the spec.
>>>>
>>>> For this I think we could start at *100,000* so that we use *100,000 +
>>> 200 * <fieldID>* to calculate the field ID of a given statistic.
>>>
>>>
>>>>
>>>> 1. The current proposal expects every engine to share the same
>>>> stats, and not store any "non-standard" stat in the metadata.
>>>>
>>>> We haven't explicitly stated it in the proposal but there were
>>> discussions on how to potentially support this and what implications it
>>> brings for readers/writers
>>>
>>>
>>> I'm still not clear on what the proposal is to handle stats for reserved
>>>> columns <https://iceberg.apache.org/spec/#reserved-field-ids> [1] (I
>>>> think there was some mention in the notes but it was light on details). It
>>>> seems like it would be potentially useful to have stats for things like
>>>> _row_id, and the multiplication would overflow for these column IDs (maybe
>>>> this still yields unique column IDs though?)
>>>>
>>>
>>> To handle stats for reserved columns we could start at *2,417,000,000*
>>> which should give us enough room to store 200 stats per metadata ID. We
>>> would also ensure that those ID ranges for table columns and reserved
>>> columns wouldn't overlap.
>>>
>>>
>>> I assume we could put whatever these columns are under stats? Maybe we
>>>> just need a more generic name for the top level struct?
>>>
>>>
>>> I haven't updated the proposal yet, but I think renaming *column_stats*
>>> to *content_stats* would make sense.
>>>
>>>
>>>
Re: [DISCUSS] v4 - Improved column statistics
Posted by Micah Kornfield <em...@gmail.com>.
After having thought about it some more, my current point of view is
proceeding with something as simple as possible for V4 (I tried to
formalize what I think the proposed algorithm is in the original proposal
doc
<https://docs.google.com/document/d/1uvbrwwAJW2TgsnoaIcwAFpjbhHkBUL5wY_24nKgtt9I/edit?tab=t.0>
[1]).
If in the course of V4 development we find some flaw with the simple
approach we can revise it (e.g. we run out of space). If something comes
up after V4, we are not talking about a lot of code either way, so having a
new scheme for V5+ would not be a major burden (all manifests are now
written with a spec version, so detection is easy).
Given the potential complications with custom stats, I think it is
reasonable to allow implementations that want custom stats to use the upper
bound of reserved offset range (e.g. we have 6 reserved out of 200 today,
if implementations really need custom stats then they can start using
offset 199, and then 198, etc). This poses a low risk of overlap in the
short term, and I assume those using custom stats would have tight control
over their environment anyways, so they have the ability to manage
conflicts, compactions, in a way that fits them.
> Another thing that both Russel and Ryan brought up is being able to track
> stats for sort orders or expressions, but they don't share an id space with
> field ids.
Slightly off topic, but is there a reason we can't unify the field ID range
for V4?
I feel like it would be better to work to formalize the stats so that they
> are known and easier to project, but it's also hard to get agreement for
> more complicated stats (like coalitions that have very specific character
> set handling), but I think using expressions in lieu of custom stats might
> address all of these cases and would be more straightforward for the
> copy-forward requirement.
Also off topic, but doesn't this just shift the burden of
standardardization to expressions? This might be controversial but maybe
the bar for adding a new stat type should be relatively low? They are
optional anyways, we can maybe define some stats as core (implementations
are incomplete if they can't produce them) and others as non-core (not
required for implementations, there can be optional configuration to either
block writes that require producing the stats or just drop them).
[1]
https://docs.google.com/document/d/1uvbrwwAJW2TgsnoaIcwAFpjbhHkBUL5wY_24nKgtt9I/edit?tab=t.0
On Thu, Jul 24, 2025 at 11:19 AM Daniel Weeks <dw...@apache.org> wrote:
> The current proposal only leaves 10000+200 ids for other columns than
>> stats. If in the future, we find some other feature which would require a
>> manifest file column for every data column in the table, then we would need
>> to change the spec.
>
>
> I do think we might want to put an upper bound on the column stats. Ryan
> calculated the upper bound of what can be represented, but I don't think we
> need to accommodate 10m+ field ids and that would block the entire id
> range. It might make more sense to simply put an upper bound on the stats
> space (e.g. 100k or 1m fields?). This would leave plenty of space for
> future evolution of the spec without having to redefine the stats range.
>
> Another thing that both Russel and Ryan brought up is being able to track
> stats for sort orders or expressions, but they don't share an id space with
> field ids. We might want to decide what the full stats space should look
> like. For example:
>
> 8k+ sort orders
> 9k+ expressions
> 10+ field ids
> 1m+ <unreserved>
> MAX_VALUE - 200 <reserved per spec>
>
> Since sort orders and expressions have much lower cardinality than field
> ids, we can probably have a more constrained range.
>
> I'm leaning against custom stats because it does increase complexity for
> all writers as Micah mentioned and introduces the potential for id space
> collision. It would also easily compromise the performance of engines if
> other writers drop them (via compaction or just any metadata rewrite
> operation). I feel like it would be better to work to formalize the stats
> so that they are known and easier to project, but it's also hard to get
> agreement for more complicated stats (like coalitions that have very
> specific character set handling), but I think using expressions in lieu of
> custom stats might address all of these cases and would be more
> straightforward for the copy-forward requirement.
>
> -Dan
>
>
>
> On Thu, Jul 24, 2025 at 4:03 AM Eduard Tudenhöfner
> <ed...@databricks.com.invalid> wrote:
>
>>
>>
>>
>>> 1. The current proposal only leaves 10000+200 ids for other columns
>>> than stats. If in the future, we find some other feature which would
>>> require a manifest file column for every data column in the table, then we
>>> would need to change the spec.
>>>
>>> For this I think we could start at *100,000* so that we use *100,000 +
>> 200 * <fieldID>* to calculate the field ID of a given statistic.
>>
>>
>>>
>>> 1. The current proposal expects every engine to share the same
>>> stats, and not store any "non-standard" stat in the metadata.
>>>
>>> We haven't explicitly stated it in the proposal but there were
>> discussions on how to potentially support this and what implications it
>> brings for readers/writers
>>
>>
>> I'm still not clear on what the proposal is to handle stats for reserved
>>> columns <https://iceberg.apache.org/spec/#reserved-field-ids> [1] (I
>>> think there was some mention in the notes but it was light on details). It
>>> seems like it would be potentially useful to have stats for things like
>>> _row_id, and the multiplication would overflow for these column IDs (maybe
>>> this still yields unique column IDs though?)
>>>
>>
>> To handle stats for reserved columns we could start at *2,417,000,000*
>> which should give us enough room to store 200 stats per metadata ID. We
>> would also ensure that those ID ranges for table columns and reserved
>> columns wouldn't overlap.
>>
>>
>> I assume we could put whatever these columns are under stats? Maybe we
>>> just need a more generic name for the top level struct?
>>
>>
>> I haven't updated the proposal yet, but I think renaming *column_stats*
>> to *content_stats* would make sense.
>>
>>
>>
Re: [DISCUSS] v4 - Improved column statistics
Posted by Daniel Weeks <dw...@apache.org>.
>
> The current proposal only leaves 10000+200 ids for other columns than
> stats. If in the future, we find some other feature which would require a
> manifest file column for every data column in the table, then we would need
> to change the spec.
I do think we might want to put an upper bound on the column stats. Ryan
calculated the upper bound of what can be represented, but I don't think we
need to accommodate 10m+ field ids and that would block the entire id
range. It might make more sense to simply put an upper bound on the stats
space (e.g. 100k or 1m fields?). This would leave plenty of space for
future evolution of the spec without having to redefine the stats range.
Another thing that both Russel and Ryan brought up is being able to track
stats for sort orders or expressions, but they don't share an id space with
field ids. We might want to decide what the full stats space should look
like. For example:
8k+ sort orders
9k+ expressions
10+ field ids
1m+ <unreserved>
MAX_VALUE - 200 <reserved per spec>
Since sort orders and expressions have much lower cardinality than field
ids, we can probably have a more constrained range.
I'm leaning against custom stats because it does increase complexity for
all writers as Micah mentioned and introduces the potential for id space
collision. It would also easily compromise the performance of engines if
other writers drop them (via compaction or just any metadata rewrite
operation). I feel like it would be better to work to formalize the stats
so that they are known and easier to project, but it's also hard to get
agreement for more complicated stats (like coalitions that have very
specific character set handling), but I think using expressions in lieu of
custom stats might address all of these cases and would be more
straightforward for the copy-forward requirement.
-Dan
On Thu, Jul 24, 2025 at 4:03 AM Eduard Tudenhöfner
<ed...@databricks.com.invalid> wrote:
>
>
>
>> 1. The current proposal only leaves 10000+200 ids for other columns
>> than stats. If in the future, we find some other feature which would
>> require a manifest file column for every data column in the table, then we
>> would need to change the spec.
>>
>> For this I think we could start at *100,000* so that we use *100,000 +
> 200 * <fieldID>* to calculate the field ID of a given statistic.
>
>
>>
>> 1. The current proposal expects every engine to share the same stats,
>> and not store any "non-standard" stat in the metadata.
>>
>> We haven't explicitly stated it in the proposal but there were
> discussions on how to potentially support this and what implications it
> brings for readers/writers
>
>
> I'm still not clear on what the proposal is to handle stats for reserved
>> columns <https://iceberg.apache.org/spec/#reserved-field-ids> [1] (I
>> think there was some mention in the notes but it was light on details). It
>> seems like it would be potentially useful to have stats for things like
>> _row_id, and the multiplication would overflow for these column IDs (maybe
>> this still yields unique column IDs though?)
>>
>
> To handle stats for reserved columns we could start at *2,417,000,000*
> which should give us enough room to store 200 stats per metadata ID. We
> would also ensure that those ID ranges for table columns and reserved
> columns wouldn't overlap.
>
>
> I assume we could put whatever these columns are under stats? Maybe we
>> just need a more generic name for the top level struct?
>
>
> I haven't updated the proposal yet, but I think renaming *column_stats*
> to *content_stats* would make sense.
>
>
>
Re: [DISCUSS] v4 - Improved column statistics
Posted by Eduard Tudenhöfner <ed...@databricks.com.INVALID>.
> 1. The current proposal only leaves 10000+200 ids for other columns
> than stats. If in the future, we find some other feature which would
> require a manifest file column for every data column in the table, then we
> would need to change the spec.
>
> For this I think we could start at *100,000* so that we use *100,000 +
200 * <fieldID>* to calculate the field ID of a given statistic.
>
> 1. The current proposal expects every engine to share the same stats,
> and not store any "non-standard" stat in the metadata.
>
> We haven't explicitly stated it in the proposal but there were discussions
on how to potentially support this and what implications it brings for
readers/writers
I'm still not clear on what the proposal is to handle stats for reserved
> columns <https://iceberg.apache.org/spec/#reserved-field-ids> [1] (I
> think there was some mention in the notes but it was light on details). It
> seems like it would be potentially useful to have stats for things like
> _row_id, and the multiplication would overflow for these column IDs (maybe
> this still yields unique column IDs though?)
>
To handle stats for reserved columns we could start at *2,417,000,000*
which should give us enough room to store 200 stats per metadata ID. We
would also ensure that those ID ranges for table columns and reserved
columns wouldn't overlap.
I assume we could put whatever these columns are under stats? Maybe we just
> need a more generic name for the top level struct?
I haven't updated the proposal yet, but I think renaming *column_stats* to
*content_stats* would make sense.
Re: [DISCUSS] v4 - Improved column statistics
Posted by Péter Váry <pe...@gmail.com>.
I also like the assigned field ids, my only concern is about the way of
generating them.
I have 2 main issues:
1. The current proposal only leaves 10000+200 ids for other columns than
stats. If in the future, we find some other feature which would require a
manifest file column for every data column in the table, then we would need
to change the spec.
2. The current proposal expects every engine to share the same stats,
and not store any "non-standard" stat in the metadata.
I think 1st could be solved by simply reserving a bigger number of fieldIds
whatever algorithm we choose.
I understand that the 2nd is not an immediate goal at this point, but I
don't see how we can support this in the future if we use only ids for
identifying the stat fields.
Steven Wu <st...@gmail.com> ezt írta (időpont: 2025. júl. 22., K,
20:21):
> It seems reasonable to support stats for computed/calculated columns with
> assigned field ids.
>
> E.g., Flink has "computed columns" for a long time.
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/#columns
>
> CREATE TABLE MyTable ( `user_id` BIGINT, `price` DOUBLE, `quantity` DOUBLE, `cost` AS price * quantity -- evaluate expression and supply the result to queries) WITH ( 'connector' = 'kafka' ...);
>
>
> On Tue, Jul 22, 2025 at 10:56 AM Russell Spitzer <
> russell.spitzer@gmail.com> wrote:
>
>> As long as enough folks are on board for assigning table field-id's to
>> non-physical columns I have no problem with that approach.
>>
>> On Tue, Jul 22, 2025 at 12:34 PM Ryan Blue <rd...@gmail.com> wrote:
>>
>>> > I still think that the current `10000 + 200 * data_field_id` id
>>> generation logic is very limiting. I have already seen tables with more
>>> than 10k columns for AI space. And I don't think we would like to create
>>> hard limits here.
>>>
>>> If we include only positive integers for IDs then the max ID is 2^31-1 =
>>> 2,147,483,647. The last 200 fields are reserved and this is proposing to
>>> skip the first 10,000 IDs for other structures in the manifest file schema.
>>> That can accommodate more than 10.7 million field IDs from the table space
>>> (= (2,147,483,647 - 10,200) / 200). I think that is a reasonable upper
>>> bound for the number of table columns.
>>>
>>> > I think this would probably a good opportunity to also reserve space
>>> for metrics which apply to the Sort Order transformation used to create a
>>> file.
>>>
>>> I've also been thinking about being able to store stats for other
>>> expressions in this space, including partition values that are part of
>>> Amogh's one-file commit and adaptive metadata proposal. Another use case is
>>> keeping stats for derived values, like `to_lower(string_col)`. What I'd
>>> propose is being able to track expressions that are assigned an ID from the
>>> table column space, then storing the expressions stats according to the
>>> structure proposed here. But I wouldn't over-complicate the current
>>> proposal by adding this just yet. We can talk about extending stats to
>>> expressions in another proposal once we have the structure and ID
>>> assignment done.
>>>
>>> On Tue, Jul 22, 2025 at 9:59 AM Russell Spitzer <
>>> russell.spitzer@gmail.com> wrote:
>>>
>>>> I'm also sorry I missed the discussion because I was busy trying to
>>>> keep a nearly-3 year old occupied on a plane :)
>>>>
>>>> I think the proposal is pretty strong although I have one request.
>>>> Currently, we have the ability to note the sort order which
>>>> was applied to a data file but we have know way of knowing the
>>>> statistics of that applied transformation for said data file.
>>>> This has stopped us from doing a number of optimizations (like
>>>> combining files which are adjacent based on any complex
>>>> sort ordering). I think this would probably a good opportunity to also
>>>> reserve space for metrics which apply to the Sort Order
>>>> transformation used to create a file. This would still only be using
>>>> things that are already codified in the spec but
>>>> would make it possible for engines to use those transforms for further
>>>> predicate pushdown or optimization
>>>> in file compaction.
>>>>
>>>> A quick example
>>>>
>>>> Say I am using a hierarchical sort order (A, B, C)
>>>> I could then store the max, min of this transform which would be
>>>> independent to individual column maxes
>>>> Say
>>>> A : Min 1 A: Max 2
>>>> B: Min 1 B: Max 100000
>>>> C: Min 1 C: Max 100000
>>>>
>>>> A,B,C Min : (1,7000,32)
>>>> A,B,C Max : (2, 1, 100000)
>>>>
>>>> In this case if I'm looking for a record 2, 2, 4, I can instantly
>>>> reject using the sort order transform where if I was
>>>> using the individual columns I would have to read the file.
>>>>
>>>> This is of course also useful if the sort order is using some kind of
>>>> space filling curve or other clustering algorithm.
>>>>
>>>> Thanks for your hard work,
>>>> Russ
>>>>
>>>> On Thu, Jul 17, 2025 at 5:07 AM Péter Váry <pe...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Team,
>>>>> Sorry, but I was not able to join the discussion on Tuesday :(, but I
>>>>> listened to the recording.
>>>>>
>>>>> A few thoughts:
>>>>> - I still think that the current `10000 + 200 * data_field_id` id
>>>>> generation logic is very limiting. I have already seen tables with more
>>>>> than 10k columns for AI space. And I don't think we would like to create
>>>>> hard limits here.
>>>>> - If we follow store a few things in metadata (starting_id,
>>>>> stats_types_present, columns_with_stats), then we can greatly reduce the
>>>>> used id space, and also allow for extensibility, as the engines which
>>>>> doesn't know a specific stats_type could just ignore them)
>>>>>
>>>>> Thanks Eduard and Dan for driving this!
>>>>> Peter
>>>>>
>>>>> Eduard Tudenhöfner <et...@apache.org> ezt írta (időpont:
>>>>> 2025. júl. 16., Sze, 7:53):
>>>>>
>>>>>> Hey everyone,
>>>>>>
>>>>>> We met yesterday and talked about the column stats proposal.
>>>>>> Please find the recording here
>>>>>> <https://drive.google.com/file/d/1WVpSg9XxipO5NzogDc7D4DMsnj9cCMlF/view?usp=sharing>
>>>>>> and the notes here
>>>>>> <https://docs.google.com/document/d/1s9_o_Y8js4kHVCYI2OeL0Yh3Ey0qfkuE7Mm-z-QoEfo/edit?usp=sharing>
>>>>>> .
>>>>>>
>>>>>> Thanks everyone,
>>>>>> Eduard
>>>>>>
>>>>>> On Tue, Jul 8, 2025 at 6:51 PM Eduard Tudenhöfner <
>>>>>> etudenhoefner@apache.org> wrote:
>>>>>>
>>>>>>> Hey everyone,
>>>>>>>
>>>>>>> I've just added an event to the dev calendar for July 15 at 9am (PT)
>>>>>>> to discuss the column stats proposal.
>>>>>>>
>>>>>>>
>>>>>>> Eduard
>>>>>>>
>>>>>>> On Tue, Jul 8, 2025 at 4:09 AM Jacky Lee <qc...@gmail.com> wrote:
>>>>>>>
>>>>>>>> +1 for the wonderful feature. Please count me in if you need any
>>>>>>>> help.
>>>>>>>>
>>>>>>>> Gábor Kaszab <ga...@apache.org> 于2025年7月7日周一 21:22写道：
>>>>>>>> >
>>>>>>>> > +1 Seems a great improvement! Let me know if I can help out with
>>>>>>>> implementation, measurements, etc.!
>>>>>>>> >
>>>>>>>> > Regards,
>>>>>>>> > Gabor Kaszab
>>>>>>>> >
>>>>>>>> > John Zhuge <jz...@apache.org> ezt írta (időpont: 2025. jún. 5.,
>>>>>>>> Cs, 23:41):
>>>>>>>> >>
>>>>>>>> >> +1 Looking forward to this feature
>>>>>>>> >>
>>>>>>>> >> John Zhuge
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> On Thu, Jun 5, 2025 at 2:22 PM Ryan Blue <rd...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> >>>
>>>>>>>> >>> > I think it does not make sense to stick manifest files to
>>>>>>>> Avro if we break column stats into sub fields.
>>>>>>>> >>>
>>>>>>>> >>> This isn't necessarily true. Avro can benefit from better
>>>>>>>> pushdown with Eduard's approach as well by being able to skip more
>>>>>>>> efficiently. With the current layout, Avro stores a list of key/value pairs
>>>>>>>> that are all projected and put into a map. We avoid decoding the values,
>>>>>>>> but each field ID is decoded, then the length of the value is decoded, and
>>>>>>>> finally there is a put operation with an ID and value ByteBuffer pair. With
>>>>>>>> the new approach, we will be able to know which fields are relevant and
>>>>>>>> skip unprojected fields based on the file schema, which we couldn't do
>>>>>>>> before.
>>>>>>>> >>>
>>>>>>>> >>> To skip stats for an unused field (not part of the filter),
>>>>>>>> there are two cases. Lower/upper bounds for types that are fixed width are
>>>>>>>> skipped by updating the read position. And bounds for types that are
>>>>>>>> variable length (strings and binary) are skipped by reading the length and
>>>>>>>> skipping that number of bytes.
>>>>>>>> >>>
>>>>>>>> >>> It turns out that actually producing the metric maps is a
>>>>>>>> fairly expensive operation, so being able to skip metrics more quickly even
>>>>>>>> if the bytes still have to be read is going to save time. That said, using
>>>>>>>> a columnar format is still going to be a good idea!
>>>>>>>> >>>
>>>>>>>> >>> On Wed, Jun 4, 2025 at 11:22 PM Gang Wu <us...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> >>>>
>>>>>>>> >>>> > Together with the change which allows storing metadata in
>>>>>>>> columnar formats
>>>>>>>> >>>>
>>>>>>>> >>>> +1 on this. I think it does not make sense to stick manifest
>>>>>>>> files to Avro if we break column stats into sub fields.
>>>>>>>> >>>>
>>>>>>>> >>>> On Tue, Jun 3, 2025 at 7:19 PM Péter Váry <
>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>> >>>>>
>>>>>>>> >>>>> I would love to see more flexibility in file stats. Together
>>>>>>>> with the change which allows storing metadata in columnar formats will open
>>>>>>>> up many new possibilities. Bloom filters in metadata which could be used
>>>>>>>> for filtering out files, HLL scratches etc....
>>>>>>>> >>>>>
>>>>>>>> >>>>> +1 for the change
>>>>>>>> >>>>>
>>>>>>>> >>>>> On Tue, Jun 3, 2025, 08:12 Szehon Ho <sz...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> +1 , excited for this one too, we've seen the current
>>>>>>>> metrics maps blow up the memory and hope can improve that.
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> On the Geo front, this could allow us to add supplementary
>>>>>>>> metrics that don't conform to the geo type, like S2 Cell Ids.
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> Thanks
>>>>>>>> >>>>>> Szehon
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> On Mon, Jun 2, 2025 at 6:14 AM Eduard Tudenhöfner <
>>>>>>>> etudenhoefner@apache.org> wrote:
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Hey everyone,
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> I'm starting a thread to connect folks interested in
>>>>>>>> improving the existing way of collecting column-level statistics (often
>>>>>>>> referred to as metrics in the code). I've already started a proposal, which
>>>>>>>> can be found at https://s.apache.org/iceberg-column-stats.
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Motivation
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Column statistics are currently stored as a mapping of
>>>>>>>> field id to values across multiple columns (lower/upper bounds,
>>>>>>>> value/nan/null counts, sizes). This storage model has critical limitations
>>>>>>>> as the number of columns increases and as new types are being added to
>>>>>>>> Iceberg:
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Inefficient Storage due to map-based structure:
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Large memory overhead during planning/processing
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Inability to project specific stats (e.g., only
>>>>>>>> null_value_counts for column X)
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Type Erasure: Original logical/physical types are lost when
>>>>>>>> stored as binary blobs, causing:
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Lossy type inference during reads
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Schema evolution challenges (e.g., widening types)
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Rigid Schema: Stats are tied to the data_fil entry record,
>>>>>>>> limiting extensibility for new stats.
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Goals
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Improve the column stats representation to allow for the
>>>>>>>> following:
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Projectability: Enable independent access to specific stats
>>>>>>>> (e.g., lower_bounds without loading upper_bounds).
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Type Preservation: Store original data types to support
>>>>>>>> accurate reads and schema evolution.
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Flexible/Extensible Representation: Allow per-field stats
>>>>>>>> structures (e.g., complex types like Geo/Variant).
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Thanks
>>>>>>>> >>>>>>> Eduard
>>>>>>>>
>>>>>>>
Re: [DISCUSS] v4 - Improved column statistics
Posted by Steven Wu <st...@gmail.com>.
It seems reasonable to support stats for computed/calculated columns with
assigned field ids.
E.g., Flink has "computed columns" for a long time.
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/#columns
CREATE TABLE MyTable ( `user_id` BIGINT, `price` DOUBLE, `quantity`
DOUBLE, `cost` AS price * quantity -- evaluate expression and supply
the result to queries) WITH ( 'connector' = 'kafka' ...);
On Tue, Jul 22, 2025 at 10:56 AM Russell Spitzer <ru...@gmail.com>
wrote:
> As long as enough folks are on board for assigning table field-id's to
> non-physical columns I have no problem with that approach.
>
> On Tue, Jul 22, 2025 at 12:34 PM Ryan Blue <rd...@gmail.com> wrote:
>
>> > I still think that the current `10000 + 200 * data_field_id` id
>> generation logic is very limiting. I have already seen tables with more
>> than 10k columns for AI space. And I don't think we would like to create
>> hard limits here.
>>
>> If we include only positive integers for IDs then the max ID is 2^31-1 =
>> 2,147,483,647. The last 200 fields are reserved and this is proposing to
>> skip the first 10,000 IDs for other structures in the manifest file schema.
>> That can accommodate more than 10.7 million field IDs from the table space
>> (= (2,147,483,647 - 10,200) / 200). I think that is a reasonable upper
>> bound for the number of table columns.
>>
>> > I think this would probably a good opportunity to also reserve space
>> for metrics which apply to the Sort Order transformation used to create a
>> file.
>>
>> I've also been thinking about being able to store stats for other
>> expressions in this space, including partition values that are part of
>> Amogh's one-file commit and adaptive metadata proposal. Another use case is
>> keeping stats for derived values, like `to_lower(string_col)`. What I'd
>> propose is being able to track expressions that are assigned an ID from the
>> table column space, then storing the expressions stats according to the
>> structure proposed here. But I wouldn't over-complicate the current
>> proposal by adding this just yet. We can talk about extending stats to
>> expressions in another proposal once we have the structure and ID
>> assignment done.
>>
>> On Tue, Jul 22, 2025 at 9:59 AM Russell Spitzer <
>> russell.spitzer@gmail.com> wrote:
>>
>>> I'm also sorry I missed the discussion because I was busy trying to keep
>>> a nearly-3 year old occupied on a plane :)
>>>
>>> I think the proposal is pretty strong although I have one request.
>>> Currently, we have the ability to note the sort order which
>>> was applied to a data file but we have know way of knowing the
>>> statistics of that applied transformation for said data file.
>>> This has stopped us from doing a number of optimizations (like combining
>>> files which are adjacent based on any complex
>>> sort ordering). I think this would probably a good opportunity to also
>>> reserve space for metrics which apply to the Sort Order
>>> transformation used to create a file. This would still only be using
>>> things that are already codified in the spec but
>>> would make it possible for engines to use those transforms for further
>>> predicate pushdown or optimization
>>> in file compaction.
>>>
>>> A quick example
>>>
>>> Say I am using a hierarchical sort order (A, B, C)
>>> I could then store the max, min of this transform which would be
>>> independent to individual column maxes
>>> Say
>>> A : Min 1 A: Max 2
>>> B: Min 1 B: Max 100000
>>> C: Min 1 C: Max 100000
>>>
>>> A,B,C Min : (1,7000,32)
>>> A,B,C Max : (2, 1, 100000)
>>>
>>> In this case if I'm looking for a record 2, 2, 4, I can instantly reject
>>> using the sort order transform where if I was
>>> using the individual columns I would have to read the file.
>>>
>>> This is of course also useful if the sort order is using some kind of
>>> space filling curve or other clustering algorithm.
>>>
>>> Thanks for your hard work,
>>> Russ
>>>
>>> On Thu, Jul 17, 2025 at 5:07 AM Péter Váry <pe...@gmail.com>
>>> wrote:
>>>
>>>> Hi Team,
>>>> Sorry, but I was not able to join the discussion on Tuesday :(, but I
>>>> listened to the recording.
>>>>
>>>> A few thoughts:
>>>> - I still think that the current `10000 + 200 * data_field_id` id
>>>> generation logic is very limiting. I have already seen tables with more
>>>> than 10k columns for AI space. And I don't think we would like to create
>>>> hard limits here.
>>>> - If we follow store a few things in metadata (starting_id,
>>>> stats_types_present, columns_with_stats), then we can greatly reduce the
>>>> used id space, and also allow for extensibility, as the engines which
>>>> doesn't know a specific stats_type could just ignore them)
>>>>
>>>> Thanks Eduard and Dan for driving this!
>>>> Peter
>>>>
>>>> Eduard Tudenhöfner <et...@apache.org> ezt írta (időpont: 2025.
>>>> júl. 16., Sze, 7:53):
>>>>
>>>>> Hey everyone,
>>>>>
>>>>> We met yesterday and talked about the column stats proposal.
>>>>> Please find the recording here
>>>>> <https://drive.google.com/file/d/1WVpSg9XxipO5NzogDc7D4DMsnj9cCMlF/view?usp=sharing>
>>>>> and the notes here
>>>>> <https://docs.google.com/document/d/1s9_o_Y8js4kHVCYI2OeL0Yh3Ey0qfkuE7Mm-z-QoEfo/edit?usp=sharing>
>>>>> .
>>>>>
>>>>> Thanks everyone,
>>>>> Eduard
>>>>>
>>>>> On Tue, Jul 8, 2025 at 6:51 PM Eduard Tudenhöfner <
>>>>> etudenhoefner@apache.org> wrote:
>>>>>
>>>>>> Hey everyone,
>>>>>>
>>>>>> I've just added an event to the dev calendar for July 15 at 9am (PT)
>>>>>> to discuss the column stats proposal.
>>>>>>
>>>>>>
>>>>>> Eduard
>>>>>>
>>>>>> On Tue, Jul 8, 2025 at 4:09 AM Jacky Lee <qc...@gmail.com> wrote:
>>>>>>
>>>>>>> +1 for the wonderful feature. Please count me in if you need any
>>>>>>> help.
>>>>>>>
>>>>>>> Gábor Kaszab <ga...@apache.org> 于2025年7月7日周一 21:22写道：
>>>>>>> >
>>>>>>> > +1 Seems a great improvement! Let me know if I can help out with
>>>>>>> implementation, measurements, etc.!
>>>>>>> >
>>>>>>> > Regards,
>>>>>>> > Gabor Kaszab
>>>>>>> >
>>>>>>> > John Zhuge <jz...@apache.org> ezt írta (időpont: 2025. jún. 5.,
>>>>>>> Cs, 23:41):
>>>>>>> >>
>>>>>>> >> +1 Looking forward to this feature
>>>>>>> >>
>>>>>>> >> John Zhuge
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> On Thu, Jun 5, 2025 at 2:22 PM Ryan Blue <rd...@gmail.com>
>>>>>>> wrote:
>>>>>>> >>>
>>>>>>> >>> > I think it does not make sense to stick manifest files to Avro
>>>>>>> if we break column stats into sub fields.
>>>>>>> >>>
>>>>>>> >>> This isn't necessarily true. Avro can benefit from better
>>>>>>> pushdown with Eduard's approach as well by being able to skip more
>>>>>>> efficiently. With the current layout, Avro stores a list of key/value pairs
>>>>>>> that are all projected and put into a map. We avoid decoding the values,
>>>>>>> but each field ID is decoded, then the length of the value is decoded, and
>>>>>>> finally there is a put operation with an ID and value ByteBuffer pair. With
>>>>>>> the new approach, we will be able to know which fields are relevant and
>>>>>>> skip unprojected fields based on the file schema, which we couldn't do
>>>>>>> before.
>>>>>>> >>>
>>>>>>> >>> To skip stats for an unused field (not part of the filter),
>>>>>>> there are two cases. Lower/upper bounds for types that are fixed width are
>>>>>>> skipped by updating the read position. And bounds for types that are
>>>>>>> variable length (strings and binary) are skipped by reading the length and
>>>>>>> skipping that number of bytes.
>>>>>>> >>>
>>>>>>> >>> It turns out that actually producing the metric maps is a fairly
>>>>>>> expensive operation, so being able to skip metrics more quickly even if the
>>>>>>> bytes still have to be read is going to save time. That said, using a
>>>>>>> columnar format is still going to be a good idea!
>>>>>>> >>>
>>>>>>> >>> On Wed, Jun 4, 2025 at 11:22 PM Gang Wu <us...@gmail.com>
>>>>>>> wrote:
>>>>>>> >>>>
>>>>>>> >>>> > Together with the change which allows storing metadata in
>>>>>>> columnar formats
>>>>>>> >>>>
>>>>>>> >>>> +1 on this. I think it does not make sense to stick manifest
>>>>>>> files to Avro if we break column stats into sub fields.
>>>>>>> >>>>
>>>>>>> >>>> On Tue, Jun 3, 2025 at 7:19 PM Péter Váry <
>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>> >>>>>
>>>>>>> >>>>> I would love to see more flexibility in file stats. Together
>>>>>>> with the change which allows storing metadata in columnar formats will open
>>>>>>> up many new possibilities. Bloom filters in metadata which could be used
>>>>>>> for filtering out files, HLL scratches etc....
>>>>>>> >>>>>
>>>>>>> >>>>> +1 for the change
>>>>>>> >>>>>
>>>>>>> >>>>> On Tue, Jun 3, 2025, 08:12 Szehon Ho <sz...@gmail.com>
>>>>>>> wrote:
>>>>>>> >>>>>>
>>>>>>> >>>>>> +1 , excited for this one too, we've seen the current metrics
>>>>>>> maps blow up the memory and hope can improve that.
>>>>>>> >>>>>>
>>>>>>> >>>>>> On the Geo front, this could allow us to add supplementary
>>>>>>> metrics that don't conform to the geo type, like S2 Cell Ids.
>>>>>>> >>>>>>
>>>>>>> >>>>>> Thanks
>>>>>>> >>>>>> Szehon
>>>>>>> >>>>>>
>>>>>>> >>>>>> On Mon, Jun 2, 2025 at 6:14 AM Eduard Tudenhöfner <
>>>>>>> etudenhoefner@apache.org> wrote:
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Hey everyone,
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> I'm starting a thread to connect folks interested in
>>>>>>> improving the existing way of collecting column-level statistics (often
>>>>>>> referred to as metrics in the code). I've already started a proposal, which
>>>>>>> can be found at https://s.apache.org/iceberg-column-stats.
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Motivation
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Column statistics are currently stored as a mapping of field
>>>>>>> id to values across multiple columns (lower/upper bounds, value/nan/null
>>>>>>> counts, sizes). This storage model has critical limitations as the number
>>>>>>> of columns increases and as new types are being added to Iceberg:
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Inefficient Storage due to map-based structure:
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Large memory overhead during planning/processing
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Inability to project specific stats (e.g., only
>>>>>>> null_value_counts for column X)
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Type Erasure: Original logical/physical types are lost when
>>>>>>> stored as binary blobs, causing:
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Lossy type inference during reads
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Schema evolution challenges (e.g., widening types)
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Rigid Schema: Stats are tied to the data_fil entry record,
>>>>>>> limiting extensibility for new stats.
>>>>>>> >>>>>>>
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Goals
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Improve the column stats representation to allow for the
>>>>>>> following:
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Projectability: Enable independent access to specific stats
>>>>>>> (e.g., lower_bounds without loading upper_bounds).
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Type Preservation: Store original data types to support
>>>>>>> accurate reads and schema evolution.
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Flexible/Extensible Representation: Allow per-field stats
>>>>>>> structures (e.g., complex types like Geo/Variant).
>>>>>>> >>>>>>>
>>>>>>> >>>>>>>
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Thanks
>>>>>>> >>>>>>> Eduard
>>>>>>>
>>>>>>
Re: [DISCUSS] v4 - Improved column statistics
Posted by Russell Spitzer <ru...@gmail.com>.
As long as enough folks are on board for assigning table field-id's to
non-physical columns I have no problem with that approach.
On Tue, Jul 22, 2025 at 12:34 PM Ryan Blue <rd...@gmail.com> wrote:
> > I still think that the current `10000 + 200 * data_field_id` id
> generation logic is very limiting. I have already seen tables with more
> than 10k columns for AI space. And I don't think we would like to create
> hard limits here.
>
> If we include only positive integers for IDs then the max ID is 2^31-1 =
> 2,147,483,647. The last 200 fields are reserved and this is proposing to
> skip the first 10,000 IDs for other structures in the manifest file schema.
> That can accommodate more than 10.7 million field IDs from the table space
> (= (2,147,483,647 - 10,200) / 200). I think that is a reasonable upper
> bound for the number of table columns.
>
> > I think this would probably a good opportunity to also reserve space for
> metrics which apply to the Sort Order transformation used to create a file.
>
> I've also been thinking about being able to store stats for other
> expressions in this space, including partition values that are part of
> Amogh's one-file commit and adaptive metadata proposal. Another use case is
> keeping stats for derived values, like `to_lower(string_col)`. What I'd
> propose is being able to track expressions that are assigned an ID from the
> table column space, then storing the expressions stats according to the
> structure proposed here. But I wouldn't over-complicate the current
> proposal by adding this just yet. We can talk about extending stats to
> expressions in another proposal once we have the structure and ID
> assignment done.
>
> On Tue, Jul 22, 2025 at 9:59 AM Russell Spitzer <ru...@gmail.com>
> wrote:
>
>> I'm also sorry I missed the discussion because I was busy trying to keep
>> a nearly-3 year old occupied on a plane :)
>>
>> I think the proposal is pretty strong although I have one request.
>> Currently, we have the ability to note the sort order which
>> was applied to a data file but we have know way of knowing the statistics
>> of that applied transformation for said data file.
>> This has stopped us from doing a number of optimizations (like combining
>> files which are adjacent based on any complex
>> sort ordering). I think this would probably a good opportunity to also
>> reserve space for metrics which apply to the Sort Order
>> transformation used to create a file. This would still only be using
>> things that are already codified in the spec but
>> would make it possible for engines to use those transforms for further
>> predicate pushdown or optimization
>> in file compaction.
>>
>> A quick example
>>
>> Say I am using a hierarchical sort order (A, B, C)
>> I could then store the max, min of this transform which would be
>> independent to individual column maxes
>> Say
>> A : Min 1 A: Max 2
>> B: Min 1 B: Max 100000
>> C: Min 1 C: Max 100000
>>
>> A,B,C Min : (1,7000,32)
>> A,B,C Max : (2, 1, 100000)
>>
>> In this case if I'm looking for a record 2, 2, 4, I can instantly reject
>> using the sort order transform where if I was
>> using the individual columns I would have to read the file.
>>
>> This is of course also useful if the sort order is using some kind of
>> space filling curve or other clustering algorithm.
>>
>> Thanks for your hard work,
>> Russ
>>
>> On Thu, Jul 17, 2025 at 5:07 AM Péter Váry <pe...@gmail.com>
>> wrote:
>>
>>> Hi Team,
>>> Sorry, but I was not able to join the discussion on Tuesday :(, but I
>>> listened to the recording.
>>>
>>> A few thoughts:
>>> - I still think that the current `10000 + 200 * data_field_id` id
>>> generation logic is very limiting. I have already seen tables with more
>>> than 10k columns for AI space. And I don't think we would like to create
>>> hard limits here.
>>> - If we follow store a few things in metadata (starting_id,
>>> stats_types_present, columns_with_stats), then we can greatly reduce the
>>> used id space, and also allow for extensibility, as the engines which
>>> doesn't know a specific stats_type could just ignore them)
>>>
>>> Thanks Eduard and Dan for driving this!
>>> Peter
>>>
>>> Eduard Tudenhöfner <et...@apache.org> ezt írta (időpont: 2025.
>>> júl. 16., Sze, 7:53):
>>>
>>>> Hey everyone,
>>>>
>>>> We met yesterday and talked about the column stats proposal.
>>>> Please find the recording here
>>>> <https://drive.google.com/file/d/1WVpSg9XxipO5NzogDc7D4DMsnj9cCMlF/view?usp=sharing>
>>>> and the notes here
>>>> <https://docs.google.com/document/d/1s9_o_Y8js4kHVCYI2OeL0Yh3Ey0qfkuE7Mm-z-QoEfo/edit?usp=sharing>
>>>> .
>>>>
>>>> Thanks everyone,
>>>> Eduard
>>>>
>>>> On Tue, Jul 8, 2025 at 6:51 PM Eduard Tudenhöfner <
>>>> etudenhoefner@apache.org> wrote:
>>>>
>>>>> Hey everyone,
>>>>>
>>>>> I've just added an event to the dev calendar for July 15 at 9am (PT)
>>>>> to discuss the column stats proposal.
>>>>>
>>>>>
>>>>> Eduard
>>>>>
>>>>> On Tue, Jul 8, 2025 at 4:09 AM Jacky Lee <qc...@gmail.com> wrote:
>>>>>
>>>>>> +1 for the wonderful feature. Please count me in if you need any help.
>>>>>>
>>>>>> Gábor Kaszab <ga...@apache.org> 于2025年7月7日周一 21:22写道：
>>>>>> >
>>>>>> > +1 Seems a great improvement! Let me know if I can help out with
>>>>>> implementation, measurements, etc.!
>>>>>> >
>>>>>> > Regards,
>>>>>> > Gabor Kaszab
>>>>>> >
>>>>>> > John Zhuge <jz...@apache.org> ezt írta (időpont: 2025. jún. 5.,
>>>>>> Cs, 23:41):
>>>>>> >>
>>>>>> >> +1 Looking forward to this feature
>>>>>> >>
>>>>>> >> John Zhuge
>>>>>> >>
>>>>>> >>
>>>>>> >> On Thu, Jun 5, 2025 at 2:22 PM Ryan Blue <rd...@gmail.com> wrote:
>>>>>> >>>
>>>>>> >>> > I think it does not make sense to stick manifest files to Avro
>>>>>> if we break column stats into sub fields.
>>>>>> >>>
>>>>>> >>> This isn't necessarily true. Avro can benefit from better
>>>>>> pushdown with Eduard's approach as well by being able to skip more
>>>>>> efficiently. With the current layout, Avro stores a list of key/value pairs
>>>>>> that are all projected and put into a map. We avoid decoding the values,
>>>>>> but each field ID is decoded, then the length of the value is decoded, and
>>>>>> finally there is a put operation with an ID and value ByteBuffer pair. With
>>>>>> the new approach, we will be able to know which fields are relevant and
>>>>>> skip unprojected fields based on the file schema, which we couldn't do
>>>>>> before.
>>>>>> >>>
>>>>>> >>> To skip stats for an unused field (not part of the filter), there
>>>>>> are two cases. Lower/upper bounds for types that are fixed width are
>>>>>> skipped by updating the read position. And bounds for types that are
>>>>>> variable length (strings and binary) are skipped by reading the length and
>>>>>> skipping that number of bytes.
>>>>>> >>>
>>>>>> >>> It turns out that actually producing the metric maps is a fairly
>>>>>> expensive operation, so being able to skip metrics more quickly even if the
>>>>>> bytes still have to be read is going to save time. That said, using a
>>>>>> columnar format is still going to be a good idea!
>>>>>> >>>
>>>>>> >>> On Wed, Jun 4, 2025 at 11:22 PM Gang Wu <us...@gmail.com> wrote:
>>>>>> >>>>
>>>>>> >>>> > Together with the change which allows storing metadata in
>>>>>> columnar formats
>>>>>> >>>>
>>>>>> >>>> +1 on this. I think it does not make sense to stick manifest
>>>>>> files to Avro if we break column stats into sub fields.
>>>>>> >>>>
>>>>>> >>>> On Tue, Jun 3, 2025 at 7:19 PM Péter Váry <
>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>> >>>>>
>>>>>> >>>>> I would love to see more flexibility in file stats. Together
>>>>>> with the change which allows storing metadata in columnar formats will open
>>>>>> up many new possibilities. Bloom filters in metadata which could be used
>>>>>> for filtering out files, HLL scratches etc....
>>>>>> >>>>>
>>>>>> >>>>> +1 for the change
>>>>>> >>>>>
>>>>>> >>>>> On Tue, Jun 3, 2025, 08:12 Szehon Ho <sz...@gmail.com>
>>>>>> wrote:
>>>>>> >>>>>>
>>>>>> >>>>>> +1 , excited for this one too, we've seen the current metrics
>>>>>> maps blow up the memory and hope can improve that.
>>>>>> >>>>>>
>>>>>> >>>>>> On the Geo front, this could allow us to add supplementary
>>>>>> metrics that don't conform to the geo type, like S2 Cell Ids.
>>>>>> >>>>>>
>>>>>> >>>>>> Thanks
>>>>>> >>>>>> Szehon
>>>>>> >>>>>>
>>>>>> >>>>>> On Mon, Jun 2, 2025 at 6:14 AM Eduard Tudenhöfner <
>>>>>> etudenhoefner@apache.org> wrote:
>>>>>> >>>>>>>
>>>>>> >>>>>>> Hey everyone,
>>>>>> >>>>>>>
>>>>>> >>>>>>> I'm starting a thread to connect folks interested in
>>>>>> improving the existing way of collecting column-level statistics (often
>>>>>> referred to as metrics in the code). I've already started a proposal, which
>>>>>> can be found at https://s.apache.org/iceberg-column-stats.
>>>>>> >>>>>>>
>>>>>> >>>>>>> Motivation
>>>>>> >>>>>>>
>>>>>> >>>>>>> Column statistics are currently stored as a mapping of field
>>>>>> id to values across multiple columns (lower/upper bounds, value/nan/null
>>>>>> counts, sizes). This storage model has critical limitations as the number
>>>>>> of columns increases and as new types are being added to Iceberg:
>>>>>> >>>>>>>
>>>>>> >>>>>>> Inefficient Storage due to map-based structure:
>>>>>> >>>>>>>
>>>>>> >>>>>>> Large memory overhead during planning/processing
>>>>>> >>>>>>>
>>>>>> >>>>>>> Inability to project specific stats (e.g., only
>>>>>> null_value_counts for column X)
>>>>>> >>>>>>>
>>>>>> >>>>>>> Type Erasure: Original logical/physical types are lost when
>>>>>> stored as binary blobs, causing:
>>>>>> >>>>>>>
>>>>>> >>>>>>> Lossy type inference during reads
>>>>>> >>>>>>>
>>>>>> >>>>>>> Schema evolution challenges (e.g., widening types)
>>>>>> >>>>>>>
>>>>>> >>>>>>> Rigid Schema: Stats are tied to the data_fil entry record,
>>>>>> limiting extensibility for new stats.
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>> Goals
>>>>>> >>>>>>>
>>>>>> >>>>>>> Improve the column stats representation to allow for the
>>>>>> following:
>>>>>> >>>>>>>
>>>>>> >>>>>>> Projectability: Enable independent access to specific stats
>>>>>> (e.g., lower_bounds without loading upper_bounds).
>>>>>> >>>>>>>
>>>>>> >>>>>>> Type Preservation: Store original data types to support
>>>>>> accurate reads and schema evolution.
>>>>>> >>>>>>>
>>>>>> >>>>>>> Flexible/Extensible Representation: Allow per-field stats
>>>>>> structures (e.g., complex types like Geo/Variant).
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>> Thanks
>>>>>> >>>>>>> Eduard
>>>>>>
>>>>>
Re: [DISCUSS] v4 - Improved column statistics
Posted by Ryan Blue <rd...@gmail.com>.
> I still think that the current `10000 + 200 * data_field_id` id
generation logic is very limiting. I have already seen tables with more
than 10k columns for AI space. And I don't think we would like to create
hard limits here.
If we include only positive integers for IDs then the max ID is 2^31-1 =
2,147,483,647. The last 200 fields are reserved and this is proposing to
skip the first 10,000 IDs for other structures in the manifest file schema.
That can accommodate more than 10.7 million field IDs from the table space
(= (2,147,483,647 - 10,200) / 200). I think that is a reasonable upper
bound for the number of table columns.
> I think this would probably a good opportunity to also reserve space for
metrics which apply to the Sort Order transformation used to create a file.
I've also been thinking about being able to store stats for other
expressions in this space, including partition values that are part of
Amogh's one-file commit and adaptive metadata proposal. Another use case is
keeping stats for derived values, like `to_lower(string_col)`. What I'd
propose is being able to track expressions that are assigned an ID from the
table column space, then storing the expressions stats according to the
structure proposed here. But I wouldn't over-complicate the current
proposal by adding this just yet. We can talk about extending stats to
expressions in another proposal once we have the structure and ID
assignment done.
On Tue, Jul 22, 2025 at 9:59 AM Russell Spitzer <ru...@gmail.com>
wrote:
> I'm also sorry I missed the discussion because I was busy trying to keep a
> nearly-3 year old occupied on a plane :)
>
> I think the proposal is pretty strong although I have one request.
> Currently, we have the ability to note the sort order which
> was applied to a data file but we have know way of knowing the statistics
> of that applied transformation for said data file.
> This has stopped us from doing a number of optimizations (like combining
> files which are adjacent based on any complex
> sort ordering). I think this would probably a good opportunity to also
> reserve space for metrics which apply to the Sort Order
> transformation used to create a file. This would still only be using
> things that are already codified in the spec but
> would make it possible for engines to use those transforms for further
> predicate pushdown or optimization
> in file compaction.
>
> A quick example
>
> Say I am using a hierarchical sort order (A, B, C)
> I could then store the max, min of this transform which would be
> independent to individual column maxes
> Say
> A : Min 1 A: Max 2
> B: Min 1 B: Max 100000
> C: Min 1 C: Max 100000
>
> A,B,C Min : (1,7000,32)
> A,B,C Max : (2, 1, 100000)
>
> In this case if I'm looking for a record 2, 2, 4, I can instantly reject
> using the sort order transform where if I was
> using the individual columns I would have to read the file.
>
> This is of course also useful if the sort order is using some kind of
> space filling curve or other clustering algorithm.
>
> Thanks for your hard work,
> Russ
>
> On Thu, Jul 17, 2025 at 5:07 AM Péter Váry <pe...@gmail.com>
> wrote:
>
>> Hi Team,
>> Sorry, but I was not able to join the discussion on Tuesday :(, but I
>> listened to the recording.
>>
>> A few thoughts:
>> - I still think that the current `10000 + 200 * data_field_id` id
>> generation logic is very limiting. I have already seen tables with more
>> than 10k columns for AI space. And I don't think we would like to create
>> hard limits here.
>> - If we follow store a few things in metadata (starting_id,
>> stats_types_present, columns_with_stats), then we can greatly reduce the
>> used id space, and also allow for extensibility, as the engines which
>> doesn't know a specific stats_type could just ignore them)
>>
>> Thanks Eduard and Dan for driving this!
>> Peter
>>
>> Eduard Tudenhöfner <et...@apache.org> ezt írta (időpont: 2025.
>> júl. 16., Sze, 7:53):
>>
>>> Hey everyone,
>>>
>>> We met yesterday and talked about the column stats proposal.
>>> Please find the recording here
>>> <https://drive.google.com/file/d/1WVpSg9XxipO5NzogDc7D4DMsnj9cCMlF/view?usp=sharing>
>>> and the notes here
>>> <https://docs.google.com/document/d/1s9_o_Y8js4kHVCYI2OeL0Yh3Ey0qfkuE7Mm-z-QoEfo/edit?usp=sharing>
>>> .
>>>
>>> Thanks everyone,
>>> Eduard
>>>
>>> On Tue, Jul 8, 2025 at 6:51 PM Eduard Tudenhöfner <
>>> etudenhoefner@apache.org> wrote:
>>>
>>>> Hey everyone,
>>>>
>>>> I've just added an event to the dev calendar for July 15 at 9am (PT) to
>>>> discuss the column stats proposal.
>>>>
>>>>
>>>> Eduard
>>>>
>>>> On Tue, Jul 8, 2025 at 4:09 AM Jacky Lee <qc...@gmail.com> wrote:
>>>>
>>>>> +1 for the wonderful feature. Please count me in if you need any help.
>>>>>
>>>>> Gábor Kaszab <ga...@apache.org> 于2025年7月7日周一 21:22写道：
>>>>> >
>>>>> > +1 Seems a great improvement! Let me know if I can help out with
>>>>> implementation, measurements, etc.!
>>>>> >
>>>>> > Regards,
>>>>> > Gabor Kaszab
>>>>> >
>>>>> > John Zhuge <jz...@apache.org> ezt írta (időpont: 2025. jún. 5.,
>>>>> Cs, 23:41):
>>>>> >>
>>>>> >> +1 Looking forward to this feature
>>>>> >>
>>>>> >> John Zhuge
>>>>> >>
>>>>> >>
>>>>> >> On Thu, Jun 5, 2025 at 2:22 PM Ryan Blue <rd...@gmail.com> wrote:
>>>>> >>>
>>>>> >>> > I think it does not make sense to stick manifest files to Avro
>>>>> if we break column stats into sub fields.
>>>>> >>>
>>>>> >>> This isn't necessarily true. Avro can benefit from better pushdown
>>>>> with Eduard's approach as well by being able to skip more efficiently. With
>>>>> the current layout, Avro stores a list of key/value pairs that are all
>>>>> projected and put into a map. We avoid decoding the values, but each field
>>>>> ID is decoded, then the length of the value is decoded, and finally there
>>>>> is a put operation with an ID and value ByteBuffer pair. With the new
>>>>> approach, we will be able to know which fields are relevant and skip
>>>>> unprojected fields based on the file schema, which we couldn't do before.
>>>>> >>>
>>>>> >>> To skip stats for an unused field (not part of the filter), there
>>>>> are two cases. Lower/upper bounds for types that are fixed width are
>>>>> skipped by updating the read position. And bounds for types that are
>>>>> variable length (strings and binary) are skipped by reading the length and
>>>>> skipping that number of bytes.
>>>>> >>>
>>>>> >>> It turns out that actually producing the metric maps is a fairly
>>>>> expensive operation, so being able to skip metrics more quickly even if the
>>>>> bytes still have to be read is going to save time. That said, using a
>>>>> columnar format is still going to be a good idea!
>>>>> >>>
>>>>> >>> On Wed, Jun 4, 2025 at 11:22 PM Gang Wu <us...@gmail.com> wrote:
>>>>> >>>>
>>>>> >>>> > Together with the change which allows storing metadata in
>>>>> columnar formats
>>>>> >>>>
>>>>> >>>> +1 on this. I think it does not make sense to stick manifest
>>>>> files to Avro if we break column stats into sub fields.
>>>>> >>>>
>>>>> >>>> On Tue, Jun 3, 2025 at 7:19 PM Péter Váry <
>>>>> peter.vary.apache@gmail.com> wrote:
>>>>> >>>>>
>>>>> >>>>> I would love to see more flexibility in file stats. Together
>>>>> with the change which allows storing metadata in columnar formats will open
>>>>> up many new possibilities. Bloom filters in metadata which could be used
>>>>> for filtering out files, HLL scratches etc....
>>>>> >>>>>
>>>>> >>>>> +1 for the change
>>>>> >>>>>
>>>>> >>>>> On Tue, Jun 3, 2025, 08:12 Szehon Ho <sz...@gmail.com>
>>>>> wrote:
>>>>> >>>>>>
>>>>> >>>>>> +1 , excited for this one too, we've seen the current metrics
>>>>> maps blow up the memory and hope can improve that.
>>>>> >>>>>>
>>>>> >>>>>> On the Geo front, this could allow us to add supplementary
>>>>> metrics that don't conform to the geo type, like S2 Cell Ids.
>>>>> >>>>>>
>>>>> >>>>>> Thanks
>>>>> >>>>>> Szehon
>>>>> >>>>>>
>>>>> >>>>>> On Mon, Jun 2, 2025 at 6:14 AM Eduard Tudenhöfner <
>>>>> etudenhoefner@apache.org> wrote:
>>>>> >>>>>>>
>>>>> >>>>>>> Hey everyone,
>>>>> >>>>>>>
>>>>> >>>>>>> I'm starting a thread to connect folks interested in improving
>>>>> the existing way of collecting column-level statistics (often referred to
>>>>> as metrics in the code). I've already started a proposal, which can be
>>>>> found at https://s.apache.org/iceberg-column-stats.
>>>>> >>>>>>>
>>>>> >>>>>>> Motivation
>>>>> >>>>>>>
>>>>> >>>>>>> Column statistics are currently stored as a mapping of field
>>>>> id to values across multiple columns (lower/upper bounds, value/nan/null
>>>>> counts, sizes). This storage model has critical limitations as the number
>>>>> of columns increases and as new types are being added to Iceberg:
>>>>> >>>>>>>
>>>>> >>>>>>> Inefficient Storage due to map-based structure:
>>>>> >>>>>>>
>>>>> >>>>>>> Large memory overhead during planning/processing
>>>>> >>>>>>>
>>>>> >>>>>>> Inability to project specific stats (e.g., only
>>>>> null_value_counts for column X)
>>>>> >>>>>>>
>>>>> >>>>>>> Type Erasure: Original logical/physical types are lost when
>>>>> stored as binary blobs, causing:
>>>>> >>>>>>>
>>>>> >>>>>>> Lossy type inference during reads
>>>>> >>>>>>>
>>>>> >>>>>>> Schema evolution challenges (e.g., widening types)
>>>>> >>>>>>>
>>>>> >>>>>>> Rigid Schema: Stats are tied to the data_fil entry record,
>>>>> limiting extensibility for new stats.
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>> Goals
>>>>> >>>>>>>
>>>>> >>>>>>> Improve the column stats representation to allow for the
>>>>> following:
>>>>> >>>>>>>
>>>>> >>>>>>> Projectability: Enable independent access to specific stats
>>>>> (e.g., lower_bounds without loading upper_bounds).
>>>>> >>>>>>>
>>>>> >>>>>>> Type Preservation: Store original data types to support
>>>>> accurate reads and schema evolution.
>>>>> >>>>>>>
>>>>> >>>>>>> Flexible/Extensible Representation: Allow per-field stats
>>>>> structures (e.g., complex types like Geo/Variant).
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>> Thanks
>>>>> >>>>>>> Eduard
>>>>>
>>>>
Re: [DISCUSS] v4 - Improved column statistics
Posted by Russell Spitzer <ru...@gmail.com>.
I'm also sorry I missed the discussion because I was busy trying to keep a
nearly-3 year old occupied on a plane :)
I think the proposal is pretty strong although I have one request.
Currently, we have the ability to note the sort order which
was applied to a data file but we have know way of knowing the statistics
of that applied transformation for said data file.
This has stopped us from doing a number of optimizations (like combining
files which are adjacent based on any complex
sort ordering). I think this would probably a good opportunity to also
reserve space for metrics which apply to the Sort Order
transformation used to create a file. This would still only be using things
that are already codified in the spec but
would make it possible for engines to use those transforms for further
predicate pushdown or optimization
in file compaction.
A quick example
Say I am using a hierarchical sort order (A, B, C)
I could then store the max, min of this transform which would be
independent to individual column maxes
Say
A : Min 1 A: Max 2
B: Min 1 B: Max 100000
C: Min 1 C: Max 100000
A,B,C Min : (1,7000,32)
A,B,C Max : (2, 1, 100000)
In this case if I'm looking for a record 2, 2, 4, I can instantly reject
using the sort order transform where if I was
using the individual columns I would have to read the file.
This is of course also useful if the sort order is using some kind of space
filling curve or other clustering algorithm.
Thanks for your hard work,
Russ
On Thu, Jul 17, 2025 at 5:07 AM Péter Váry <pe...@gmail.com>
wrote:
> Hi Team,
> Sorry, but I was not able to join the discussion on Tuesday :(, but I
> listened to the recording.
>
> A few thoughts:
> - I still think that the current `10000 + 200 * data_field_id` id
> generation logic is very limiting. I have already seen tables with more
> than 10k columns for AI space. And I don't think we would like to create
> hard limits here.
> - If we follow store a few things in metadata (starting_id,
> stats_types_present, columns_with_stats), then we can greatly reduce the
> used id space, and also allow for extensibility, as the engines which
> doesn't know a specific stats_type could just ignore them)
>
> Thanks Eduard and Dan for driving this!
> Peter
>
> Eduard Tudenhöfner <et...@apache.org> ezt írta (időpont: 2025.
> júl. 16., Sze, 7:53):
>
>> Hey everyone,
>>
>> We met yesterday and talked about the column stats proposal.
>> Please find the recording here
>> <https://drive.google.com/file/d/1WVpSg9XxipO5NzogDc7D4DMsnj9cCMlF/view?usp=sharing>
>> and the notes here
>> <https://docs.google.com/document/d/1s9_o_Y8js4kHVCYI2OeL0Yh3Ey0qfkuE7Mm-z-QoEfo/edit?usp=sharing>
>> .
>>
>> Thanks everyone,
>> Eduard
>>
>> On Tue, Jul 8, 2025 at 6:51 PM Eduard Tudenhöfner <
>> etudenhoefner@apache.org> wrote:
>>
>>> Hey everyone,
>>>
>>> I've just added an event to the dev calendar for July 15 at 9am (PT) to
>>> discuss the column stats proposal.
>>>
>>>
>>> Eduard
>>>
>>> On Tue, Jul 8, 2025 at 4:09 AM Jacky Lee <qc...@gmail.com> wrote:
>>>
>>>> +1 for the wonderful feature. Please count me in if you need any help.
>>>>
>>>> Gábor Kaszab <ga...@apache.org> 于2025年7月7日周一 21:22写道：
>>>> >
>>>> > +1 Seems a great improvement! Let me know if I can help out with
>>>> implementation, measurements, etc.!
>>>> >
>>>> > Regards,
>>>> > Gabor Kaszab
>>>> >
>>>> > John Zhuge <jz...@apache.org> ezt írta (időpont: 2025. jún. 5., Cs,
>>>> 23:41):
>>>> >>
>>>> >> +1 Looking forward to this feature
>>>> >>
>>>> >> John Zhuge
>>>> >>
>>>> >>
>>>> >> On Thu, Jun 5, 2025 at 2:22 PM Ryan Blue <rd...@gmail.com> wrote:
>>>> >>>
>>>> >>> > I think it does not make sense to stick manifest files to Avro if
>>>> we break column stats into sub fields.
>>>> >>>
>>>> >>> This isn't necessarily true. Avro can benefit from better pushdown
>>>> with Eduard's approach as well by being able to skip more efficiently. With
>>>> the current layout, Avro stores a list of key/value pairs that are all
>>>> projected and put into a map. We avoid decoding the values, but each field
>>>> ID is decoded, then the length of the value is decoded, and finally there
>>>> is a put operation with an ID and value ByteBuffer pair. With the new
>>>> approach, we will be able to know which fields are relevant and skip
>>>> unprojected fields based on the file schema, which we couldn't do before.
>>>> >>>
>>>> >>> To skip stats for an unused field (not part of the filter), there
>>>> are two cases. Lower/upper bounds for types that are fixed width are
>>>> skipped by updating the read position. And bounds for types that are
>>>> variable length (strings and binary) are skipped by reading the length and
>>>> skipping that number of bytes.
>>>> >>>
>>>> >>> It turns out that actually producing the metric maps is a fairly
>>>> expensive operation, so being able to skip metrics more quickly even if the
>>>> bytes still have to be read is going to save time. That said, using a
>>>> columnar format is still going to be a good idea!
>>>> >>>
>>>> >>> On Wed, Jun 4, 2025 at 11:22 PM Gang Wu <us...@gmail.com> wrote:
>>>> >>>>
>>>> >>>> > Together with the change which allows storing metadata in
>>>> columnar formats
>>>> >>>>
>>>> >>>> +1 on this. I think it does not make sense to stick manifest files
>>>> to Avro if we break column stats into sub fields.
>>>> >>>>
>>>> >>>> On Tue, Jun 3, 2025 at 7:19 PM Péter Váry <
>>>> peter.vary.apache@gmail.com> wrote:
>>>> >>>>>
>>>> >>>>> I would love to see more flexibility in file stats. Together with
>>>> the change which allows storing metadata in columnar formats will open up
>>>> many new possibilities. Bloom filters in metadata which could be used for
>>>> filtering out files, HLL scratches etc....
>>>> >>>>>
>>>> >>>>> +1 for the change
>>>> >>>>>
>>>> >>>>> On Tue, Jun 3, 2025, 08:12 Szehon Ho <sz...@gmail.com>
>>>> wrote:
>>>> >>>>>>
>>>> >>>>>> +1 , excited for this one too, we've seen the current metrics
>>>> maps blow up the memory and hope can improve that.
>>>> >>>>>>
>>>> >>>>>> On the Geo front, this could allow us to add supplementary
>>>> metrics that don't conform to the geo type, like S2 Cell Ids.
>>>> >>>>>>
>>>> >>>>>> Thanks
>>>> >>>>>> Szehon
>>>> >>>>>>
>>>> >>>>>> On Mon, Jun 2, 2025 at 6:14 AM Eduard Tudenhöfner <
>>>> etudenhoefner@apache.org> wrote:
>>>> >>>>>>>
>>>> >>>>>>> Hey everyone,
>>>> >>>>>>>
>>>> >>>>>>> I'm starting a thread to connect folks interested in improving
>>>> the existing way of collecting column-level statistics (often referred to
>>>> as metrics in the code). I've already started a proposal, which can be
>>>> found at https://s.apache.org/iceberg-column-stats.
>>>> >>>>>>>
>>>> >>>>>>> Motivation
>>>> >>>>>>>
>>>> >>>>>>> Column statistics are currently stored as a mapping of field id
>>>> to values across multiple columns (lower/upper bounds, value/nan/null
>>>> counts, sizes). This storage model has critical limitations as the number
>>>> of columns increases and as new types are being added to Iceberg:
>>>> >>>>>>>
>>>> >>>>>>> Inefficient Storage due to map-based structure:
>>>> >>>>>>>
>>>> >>>>>>> Large memory overhead during planning/processing
>>>> >>>>>>>
>>>> >>>>>>> Inability to project specific stats (e.g., only
>>>> null_value_counts for column X)
>>>> >>>>>>>
>>>> >>>>>>> Type Erasure: Original logical/physical types are lost when
>>>> stored as binary blobs, causing:
>>>> >>>>>>>
>>>> >>>>>>> Lossy type inference during reads
>>>> >>>>>>>
>>>> >>>>>>> Schema evolution challenges (e.g., widening types)
>>>> >>>>>>>
>>>> >>>>>>> Rigid Schema: Stats are tied to the data_fil entry record,
>>>> limiting extensibility for new stats.
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>> Goals
>>>> >>>>>>>
>>>> >>>>>>> Improve the column stats representation to allow for the
>>>> following:
>>>> >>>>>>>
>>>> >>>>>>> Projectability: Enable independent access to specific stats
>>>> (e.g., lower_bounds without loading upper_bounds).
>>>> >>>>>>>
>>>> >>>>>>> Type Preservation: Store original data types to support
>>>> accurate reads and schema evolution.
>>>> >>>>>>>
>>>> >>>>>>> Flexible/Extensible Representation: Allow per-field stats
>>>> structures (e.g., complex types like Geo/Variant).
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>> Thanks
>>>> >>>>>>> Eduard
>>>>
>>>
Re: [DISCUSS] v4 - Improved column statistics
Posted by Péter Váry <pe...@gmail.com>.
Hi Team,
Sorry, but I was not able to join the discussion on Tuesday :(, but I
listened to the recording.
A few thoughts:
- I still think that the current `10000 + 200 * data_field_id` id
generation logic is very limiting. I have already seen tables with more
than 10k columns for AI space. And I don't think we would like to create
hard limits here.
- If we follow store a few things in metadata (starting_id,
stats_types_present, columns_with_stats), then we can greatly reduce the
used id space, and also allow for extensibility, as the engines which
doesn't know a specific stats_type could just ignore them)
Thanks Eduard and Dan for driving this!
Peter
Eduard Tudenhöfner <et...@apache.org> ezt írta (időpont: 2025. júl.
16., Sze, 7:53):
> Hey everyone,
>
> We met yesterday and talked about the column stats proposal.
> Please find the recording here
> <https://drive.google.com/file/d/1WVpSg9XxipO5NzogDc7D4DMsnj9cCMlF/view?usp=sharing>
> and the notes here
> <https://docs.google.com/document/d/1s9_o_Y8js4kHVCYI2OeL0Yh3Ey0qfkuE7Mm-z-QoEfo/edit?usp=sharing>
> .
>
> Thanks everyone,
> Eduard
>
> On Tue, Jul 8, 2025 at 6:51 PM Eduard Tudenhöfner <
> etudenhoefner@apache.org> wrote:
>
>> Hey everyone,
>>
>> I've just added an event to the dev calendar for July 15 at 9am (PT) to
>> discuss the column stats proposal.
>>
>>
>> Eduard
>>
>> On Tue, Jul 8, 2025 at 4:09 AM Jacky Lee <qc...@gmail.com> wrote:
>>
>>> +1 for the wonderful feature. Please count me in if you need any help.
>>>
>>> Gábor Kaszab <ga...@apache.org> 于2025年7月7日周一 21:22写道：
>>> >
>>> > +1 Seems a great improvement! Let me know if I can help out with
>>> implementation, measurements, etc.!
>>> >
>>> > Regards,
>>> > Gabor Kaszab
>>> >
>>> > John Zhuge <jz...@apache.org> ezt írta (időpont: 2025. jún. 5., Cs,
>>> 23:41):
>>> >>
>>> >> +1 Looking forward to this feature
>>> >>
>>> >> John Zhuge
>>> >>
>>> >>
>>> >> On Thu, Jun 5, 2025 at 2:22 PM Ryan Blue <rd...@gmail.com> wrote:
>>> >>>
>>> >>> > I think it does not make sense to stick manifest files to Avro if
>>> we break column stats into sub fields.
>>> >>>
>>> >>> This isn't necessarily true. Avro can benefit from better pushdown
>>> with Eduard's approach as well by being able to skip more efficiently. With
>>> the current layout, Avro stores a list of key/value pairs that are all
>>> projected and put into a map. We avoid decoding the values, but each field
>>> ID is decoded, then the length of the value is decoded, and finally there
>>> is a put operation with an ID and value ByteBuffer pair. With the new
>>> approach, we will be able to know which fields are relevant and skip
>>> unprojected fields based on the file schema, which we couldn't do before.
>>> >>>
>>> >>> To skip stats for an unused field (not part of the filter), there
>>> are two cases. Lower/upper bounds for types that are fixed width are
>>> skipped by updating the read position. And bounds for types that are
>>> variable length (strings and binary) are skipped by reading the length and
>>> skipping that number of bytes.
>>> >>>
>>> >>> It turns out that actually producing the metric maps is a fairly
>>> expensive operation, so being able to skip metrics more quickly even if the
>>> bytes still have to be read is going to save time. That said, using a
>>> columnar format is still going to be a good idea!
>>> >>>
>>> >>> On Wed, Jun 4, 2025 at 11:22 PM Gang Wu <us...@gmail.com> wrote:
>>> >>>>
>>> >>>> > Together with the change which allows storing metadata in
>>> columnar formats
>>> >>>>
>>> >>>> +1 on this. I think it does not make sense to stick manifest files
>>> to Avro if we break column stats into sub fields.
>>> >>>>
>>> >>>> On Tue, Jun 3, 2025 at 7:19 PM Péter Váry <
>>> peter.vary.apache@gmail.com> wrote:
>>> >>>>>
>>> >>>>> I would love to see more flexibility in file stats. Together with
>>> the change which allows storing metadata in columnar formats will open up
>>> many new possibilities. Bloom filters in metadata which could be used for
>>> filtering out files, HLL scratches etc....
>>> >>>>>
>>> >>>>> +1 for the change
>>> >>>>>
>>> >>>>> On Tue, Jun 3, 2025, 08:12 Szehon Ho <sz...@gmail.com>
>>> wrote:
>>> >>>>>>
>>> >>>>>> +1 , excited for this one too, we've seen the current metrics
>>> maps blow up the memory and hope can improve that.
>>> >>>>>>
>>> >>>>>> On the Geo front, this could allow us to add supplementary
>>> metrics that don't conform to the geo type, like S2 Cell Ids.
>>> >>>>>>
>>> >>>>>> Thanks
>>> >>>>>> Szehon
>>> >>>>>>
>>> >>>>>> On Mon, Jun 2, 2025 at 6:14 AM Eduard Tudenhöfner <
>>> etudenhoefner@apache.org> wrote:
>>> >>>>>>>
>>> >>>>>>> Hey everyone,
>>> >>>>>>>
>>> >>>>>>> I'm starting a thread to connect folks interested in improving
>>> the existing way of collecting column-level statistics (often referred to
>>> as metrics in the code). I've already started a proposal, which can be
>>> found at https://s.apache.org/iceberg-column-stats.
>>> >>>>>>>
>>> >>>>>>> Motivation
>>> >>>>>>>
>>> >>>>>>> Column statistics are currently stored as a mapping of field id
>>> to values across multiple columns (lower/upper bounds, value/nan/null
>>> counts, sizes). This storage model has critical limitations as the number
>>> of columns increases and as new types are being added to Iceberg:
>>> >>>>>>>
>>> >>>>>>> Inefficient Storage due to map-based structure:
>>> >>>>>>>
>>> >>>>>>> Large memory overhead during planning/processing
>>> >>>>>>>
>>> >>>>>>> Inability to project specific stats (e.g., only
>>> null_value_counts for column X)
>>> >>>>>>>
>>> >>>>>>> Type Erasure: Original logical/physical types are lost when
>>> stored as binary blobs, causing:
>>> >>>>>>>
>>> >>>>>>> Lossy type inference during reads
>>> >>>>>>>
>>> >>>>>>> Schema evolution challenges (e.g., widening types)
>>> >>>>>>>
>>> >>>>>>> Rigid Schema: Stats are tied to the data_fil entry record,
>>> limiting extensibility for new stats.
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> Goals
>>> >>>>>>>
>>> >>>>>>> Improve the column stats representation to allow for the
>>> following:
>>> >>>>>>>
>>> >>>>>>> Projectability: Enable independent access to specific stats
>>> (e.g., lower_bounds without loading upper_bounds).
>>> >>>>>>>
>>> >>>>>>> Type Preservation: Store original data types to support accurate
>>> reads and schema evolution.
>>> >>>>>>>
>>> >>>>>>> Flexible/Extensible Representation: Allow per-field stats
>>> structures (e.g., complex types like Geo/Variant).
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> Thanks
>>> >>>>>>> Eduard
>>>
>>
Re: [DISCUSS] v4 - Improved column statistics
Posted by Eduard Tudenhöfner <et...@apache.org>.
Hey everyone,
We met yesterday and talked about the column stats proposal.
Please find the recording here
<https://drive.google.com/file/d/1WVpSg9XxipO5NzogDc7D4DMsnj9cCMlF/view?usp=sharing>
and the notes here
<https://docs.google.com/document/d/1s9_o_Y8js4kHVCYI2OeL0Yh3Ey0qfkuE7Mm-z-QoEfo/edit?usp=sharing>
.
Thanks everyone,
Eduard
On Tue, Jul 8, 2025 at 6:51 PM Eduard Tudenhöfner <et...@apache.org>
wrote:
> Hey everyone,
>
> I've just added an event to the dev calendar for July 15 at 9am (PT) to
> discuss the column stats proposal.
>
>
> Eduard
>
> On Tue, Jul 8, 2025 at 4:09 AM Jacky Lee <qc...@gmail.com> wrote:
>
>> +1 for the wonderful feature. Please count me in if you need any help.
>>
>> Gábor Kaszab <ga...@apache.org> 于2025年7月7日周一 21:22写道：
>> >
>> > +1 Seems a great improvement! Let me know if I can help out with
>> implementation, measurements, etc.!
>> >
>> > Regards,
>> > Gabor Kaszab
>> >
>> > John Zhuge <jz...@apache.org> ezt írta (időpont: 2025. jún. 5., Cs,
>> 23:41):
>> >>
>> >> +1 Looking forward to this feature
>> >>
>> >> John Zhuge
>> >>
>> >>
>> >> On Thu, Jun 5, 2025 at 2:22 PM Ryan Blue <rd...@gmail.com> wrote:
>> >>>
>> >>> > I think it does not make sense to stick manifest files to Avro if
>> we break column stats into sub fields.
>> >>>
>> >>> This isn't necessarily true. Avro can benefit from better pushdown
>> with Eduard's approach as well by being able to skip more efficiently. With
>> the current layout, Avro stores a list of key/value pairs that are all
>> projected and put into a map. We avoid decoding the values, but each field
>> ID is decoded, then the length of the value is decoded, and finally there
>> is a put operation with an ID and value ByteBuffer pair. With the new
>> approach, we will be able to know which fields are relevant and skip
>> unprojected fields based on the file schema, which we couldn't do before.
>> >>>
>> >>> To skip stats for an unused field (not part of the filter), there are
>> two cases. Lower/upper bounds for types that are fixed width are skipped by
>> updating the read position. And bounds for types that are variable length
>> (strings and binary) are skipped by reading the length and skipping that
>> number of bytes.
>> >>>
>> >>> It turns out that actually producing the metric maps is a fairly
>> expensive operation, so being able to skip metrics more quickly even if the
>> bytes still have to be read is going to save time. That said, using a
>> columnar format is still going to be a good idea!
>> >>>
>> >>> On Wed, Jun 4, 2025 at 11:22 PM Gang Wu <us...@gmail.com> wrote:
>> >>>>
>> >>>> > Together with the change which allows storing metadata in columnar
>> formats
>> >>>>
>> >>>> +1 on this. I think it does not make sense to stick manifest files
>> to Avro if we break column stats into sub fields.
>> >>>>
>> >>>> On Tue, Jun 3, 2025 at 7:19 PM Péter Váry <
>> peter.vary.apache@gmail.com> wrote:
>> >>>>>
>> >>>>> I would love to see more flexibility in file stats. Together with
>> the change which allows storing metadata in columnar formats will open up
>> many new possibilities. Bloom filters in metadata which could be used for
>> filtering out files, HLL scratches etc....
>> >>>>>
>> >>>>> +1 for the change
>> >>>>>
>> >>>>> On Tue, Jun 3, 2025, 08:12 Szehon Ho <sz...@gmail.com>
>> wrote:
>> >>>>>>
>> >>>>>> +1 , excited for this one too, we've seen the current metrics maps
>> blow up the memory and hope can improve that.
>> >>>>>>
>> >>>>>> On the Geo front, this could allow us to add supplementary metrics
>> that don't conform to the geo type, like S2 Cell Ids.
>> >>>>>>
>> >>>>>> Thanks
>> >>>>>> Szehon
>> >>>>>>
>> >>>>>> On Mon, Jun 2, 2025 at 6:14 AM Eduard Tudenhöfner <
>> etudenhoefner@apache.org> wrote:
>> >>>>>>>
>> >>>>>>> Hey everyone,
>> >>>>>>>
>> >>>>>>> I'm starting a thread to connect folks interested in improving
>> the existing way of collecting column-level statistics (often referred to
>> as metrics in the code). I've already started a proposal, which can be
>> found at https://s.apache.org/iceberg-column-stats.
>> >>>>>>>
>> >>>>>>> Motivation
>> >>>>>>>
>> >>>>>>> Column statistics are currently stored as a mapping of field id
>> to values across multiple columns (lower/upper bounds, value/nan/null
>> counts, sizes). This storage model has critical limitations as the number
>> of columns increases and as new types are being added to Iceberg:
>> >>>>>>>
>> >>>>>>> Inefficient Storage due to map-based structure:
>> >>>>>>>
>> >>>>>>> Large memory overhead during planning/processing
>> >>>>>>>
>> >>>>>>> Inability to project specific stats (e.g., only null_value_counts
>> for column X)
>> >>>>>>>
>> >>>>>>> Type Erasure: Original logical/physical types are lost when
>> stored as binary blobs, causing:
>> >>>>>>>
>> >>>>>>> Lossy type inference during reads
>> >>>>>>>
>> >>>>>>> Schema evolution challenges (e.g., widening types)
>> >>>>>>>
>> >>>>>>> Rigid Schema: Stats are tied to the data_fil entry record,
>> limiting extensibility for new stats.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Goals
>> >>>>>>>
>> >>>>>>> Improve the column stats representation to allow for the
>> following:
>> >>>>>>>
>> >>>>>>> Projectability: Enable independent access to specific stats
>> (e.g., lower_bounds without loading upper_bounds).
>> >>>>>>>
>> >>>>>>> Type Preservation: Store original data types to support accurate
>> reads and schema evolution.
>> >>>>>>>
>> >>>>>>> Flexible/Extensible Representation: Allow per-field stats
>> structures (e.g., complex types like Geo/Variant).
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Thanks
>> >>>>>>> Eduard
>>
>
Re: [DISCUSS] v4 - Improved column statistics
Posted by Eduard Tudenhöfner <et...@apache.org>.
Hey everyone,
I've just added an event to the dev calendar for July 15 at 9am (PT) to
discuss the column stats proposal.
Eduard
On Tue, Jul 8, 2025 at 4:09 AM Jacky Lee <qc...@gmail.com> wrote:
> +1 for the wonderful feature. Please count me in if you need any help.
>
> Gábor Kaszab <ga...@apache.org> 于2025年7月7日周一 21:22写道：
> >
> > +1 Seems a great improvement! Let me know if I can help out with
> implementation, measurements, etc.!
> >
> > Regards,
> > Gabor Kaszab
> >
> > John Zhuge <jz...@apache.org> ezt írta (időpont: 2025. jún. 5., Cs,
> 23:41):
> >>
> >> +1 Looking forward to this feature
> >>
> >> John Zhuge
> >>
> >>
> >> On Thu, Jun 5, 2025 at 2:22 PM Ryan Blue <rd...@gmail.com> wrote:
> >>>
> >>> > I think it does not make sense to stick manifest files to Avro if we
> break column stats into sub fields.
> >>>
> >>> This isn't necessarily true. Avro can benefit from better pushdown
> with Eduard's approach as well by being able to skip more efficiently. With
> the current layout, Avro stores a list of key/value pairs that are all
> projected and put into a map. We avoid decoding the values, but each field
> ID is decoded, then the length of the value is decoded, and finally there
> is a put operation with an ID and value ByteBuffer pair. With the new
> approach, we will be able to know which fields are relevant and skip
> unprojected fields based on the file schema, which we couldn't do before.
> >>>
> >>> To skip stats for an unused field (not part of the filter), there are
> two cases. Lower/upper bounds for types that are fixed width are skipped by
> updating the read position. And bounds for types that are variable length
> (strings and binary) are skipped by reading the length and skipping that
> number of bytes.
> >>>
> >>> It turns out that actually producing the metric maps is a fairly
> expensive operation, so being able to skip metrics more quickly even if the
> bytes still have to be read is going to save time. That said, using a
> columnar format is still going to be a good idea!
> >>>
> >>> On Wed, Jun 4, 2025 at 11:22 PM Gang Wu <us...@gmail.com> wrote:
> >>>>
> >>>> > Together with the change which allows storing metadata in columnar
> formats
> >>>>
> >>>> +1 on this. I think it does not make sense to stick manifest files to
> Avro if we break column stats into sub fields.
> >>>>
> >>>> On Tue, Jun 3, 2025 at 7:19 PM Péter Váry <
> peter.vary.apache@gmail.com> wrote:
> >>>>>
> >>>>> I would love to see more flexibility in file stats. Together with
> the change which allows storing metadata in columnar formats will open up
> many new possibilities. Bloom filters in metadata which could be used for
> filtering out files, HLL scratches etc....
> >>>>>
> >>>>> +1 for the change
> >>>>>
> >>>>> On Tue, Jun 3, 2025, 08:12 Szehon Ho <sz...@gmail.com>
> wrote:
> >>>>>>
> >>>>>> +1 , excited for this one too, we've seen the current metrics maps
> blow up the memory and hope can improve that.
> >>>>>>
> >>>>>> On the Geo front, this could allow us to add supplementary metrics
> that don't conform to the geo type, like S2 Cell Ids.
> >>>>>>
> >>>>>> Thanks
> >>>>>> Szehon
> >>>>>>
> >>>>>> On Mon, Jun 2, 2025 at 6:14 AM Eduard Tudenhöfner <
> etudenhoefner@apache.org> wrote:
> >>>>>>>
> >>>>>>> Hey everyone,
> >>>>>>>
> >>>>>>> I'm starting a thread to connect folks interested in improving the
> existing way of collecting column-level statistics (often referred to as
> metrics in the code). I've already started a proposal, which can be found
> at https://s.apache.org/iceberg-column-stats.
> >>>>>>>
> >>>>>>> Motivation
> >>>>>>>
> >>>>>>> Column statistics are currently stored as a mapping of field id to
> values across multiple columns (lower/upper bounds, value/nan/null counts,
> sizes). This storage model has critical limitations as the number of
> columns increases and as new types are being added to Iceberg:
> >>>>>>>
> >>>>>>> Inefficient Storage due to map-based structure:
> >>>>>>>
> >>>>>>> Large memory overhead during planning/processing
> >>>>>>>
> >>>>>>> Inability to project specific stats (e.g., only null_value_counts
> for column X)
> >>>>>>>
> >>>>>>> Type Erasure: Original logical/physical types are lost when stored
> as binary blobs, causing:
> >>>>>>>
> >>>>>>> Lossy type inference during reads
> >>>>>>>
> >>>>>>> Schema evolution challenges (e.g., widening types)
> >>>>>>>
> >>>>>>> Rigid Schema: Stats are tied to the data_fil entry record,
> limiting extensibility for new stats.
> >>>>>>>
> >>>>>>>
> >>>>>>> Goals
> >>>>>>>
> >>>>>>> Improve the column stats representation to allow for the following:
> >>>>>>>
> >>>>>>> Projectability: Enable independent access to specific stats (e.g.,
> lower_bounds without loading upper_bounds).
> >>>>>>>
> >>>>>>> Type Preservation: Store original data types to support accurate
> reads and schema evolution.
> >>>>>>>
> >>>>>>> Flexible/Extensible Representation: Allow per-field stats
> structures (e.g., complex types like Geo/Variant).
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Thanks
> >>>>>>> Eduard
>
Re: [DISCUSS] v4 - Improved column statistics
Posted by Jacky Lee <qc...@gmail.com>.
+1 for the wonderful feature. Please count me in if you need any help.
Gábor Kaszab <ga...@apache.org> 于2025年7月7日周一 21:22写道：
>
> +1 Seems a great improvement! Let me know if I can help out with implementation, measurements, etc.!
>
> Regards,
> Gabor Kaszab
>
> John Zhuge <jz...@apache.org> ezt írta (időpont: 2025. jún. 5., Cs, 23:41):
>>
>> +1 Looking forward to this feature
>>
>> John Zhuge
>>
>>
>> On Thu, Jun 5, 2025 at 2:22 PM Ryan Blue <rd...@gmail.com> wrote:
>>>
>>> > I think it does not make sense to stick manifest files to Avro if we break column stats into sub fields.
>>>
>>> This isn't necessarily true. Avro can benefit from better pushdown with Eduard's approach as well by being able to skip more efficiently. With the current layout, Avro stores a list of key/value pairs that are all projected and put into a map. We avoid decoding the values, but each field ID is decoded, then the length of the value is decoded, and finally there is a put operation with an ID and value ByteBuffer pair. With the new approach, we will be able to know which fields are relevant and skip unprojected fields based on the file schema, which we couldn't do before.
>>>
>>> To skip stats for an unused field (not part of the filter), there are two cases. Lower/upper bounds for types that are fixed width are skipped by updating the read position. And bounds for types that are variable length (strings and binary) are skipped by reading the length and skipping that number of bytes.
>>>
>>> It turns out that actually producing the metric maps is a fairly expensive operation, so being able to skip metrics more quickly even if the bytes still have to be read is going to save time. That said, using a columnar format is still going to be a good idea!
>>>
>>> On Wed, Jun 4, 2025 at 11:22 PM Gang Wu <us...@gmail.com> wrote:
>>>>
>>>> > Together with the change which allows storing metadata in columnar formats
>>>>
>>>> +1 on this. I think it does not make sense to stick manifest files to Avro if we break column stats into sub fields.
>>>>
>>>> On Tue, Jun 3, 2025 at 7:19 PM Péter Váry <pe...@gmail.com> wrote:
>>>>>
>>>>> I would love to see more flexibility in file stats. Together with the change which allows storing metadata in columnar formats will open up many new possibilities. Bloom filters in metadata which could be used for filtering out files, HLL scratches etc....
>>>>>
>>>>> +1 for the change
>>>>>
>>>>> On Tue, Jun 3, 2025, 08:12 Szehon Ho <sz...@gmail.com> wrote:
>>>>>>
>>>>>> +1 , excited for this one too, we've seen the current metrics maps blow up the memory and hope can improve that.
>>>>>>
>>>>>> On the Geo front, this could allow us to add supplementary metrics that don't conform to the geo type, like S2 Cell Ids.
>>>>>>
>>>>>> Thanks
>>>>>> Szehon
>>>>>>
>>>>>> On Mon, Jun 2, 2025 at 6:14 AM Eduard Tudenhöfner <et...@apache.org> wrote:
>>>>>>>
>>>>>>> Hey everyone,
>>>>>>>
>>>>>>> I'm starting a thread to connect folks interested in improving the existing way of collecting column-level statistics (often referred to as metrics in the code). I've already started a proposal, which can be found at https://s.apache.org/iceberg-column-stats.
>>>>>>>
>>>>>>> Motivation
>>>>>>>
>>>>>>> Column statistics are currently stored as a mapping of field id to values across multiple columns (lower/upper bounds, value/nan/null counts, sizes). This storage model has critical limitations as the number of columns increases and as new types are being added to Iceberg:
>>>>>>>
>>>>>>> Inefficient Storage due to map-based structure:
>>>>>>>
>>>>>>> Large memory overhead during planning/processing
>>>>>>>
>>>>>>> Inability to project specific stats (e.g., only null_value_counts for column X)
>>>>>>>
>>>>>>> Type Erasure: Original logical/physical types are lost when stored as binary blobs, causing:
>>>>>>>
>>>>>>> Lossy type inference during reads
>>>>>>>
>>>>>>> Schema evolution challenges (e.g., widening types)
>>>>>>>
>>>>>>> Rigid Schema: Stats are tied to the data_fil entry record, limiting extensibility for new stats.
>>>>>>>
>>>>>>>
>>>>>>> Goals
>>>>>>>
>>>>>>> Improve the column stats representation to allow for the following:
>>>>>>>
>>>>>>> Projectability: Enable independent access to specific stats (e.g., lower_bounds without loading upper_bounds).
>>>>>>>
>>>>>>> Type Preservation: Store original data types to support accurate reads and schema evolution.
>>>>>>>
>>>>>>> Flexible/Extensible Representation: Allow per-field stats structures (e.g., complex types like Geo/Variant).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>> Eduard