V4: Block-level Pruning for Inlined Metadata (Adaptive Metadata Tree)

[vaquar khan <va...@gmail.com>]

Hi everyone,

I’ve been following the recent discussions and design documents regarding
the Adaptive Metadata Tree and Single-File Commits for the V4 Spec.

While moving to a Root Manifest structure solves the write amplification
issue on S3/GCS, I am concerned about a potential regression in Partition
Pruning efficiency for readers. Specifically, when Data Files are inlined
into the Root Manifest, we lose the explicit partition summary bounds that
existed in the V3 Manifest List.

Without a standardized way to store lightweight partition stats for these
inlined entries, query planners may be forced to scan significantly more
metadata bytes to perform the same pruning we get for free today.

*Proposal*: I propose we explicitly standardize a "Compact Partition
Summary" (possibly using Bloom Filters or compressed min/max tuples) within
the Root Manifest entry schema. This would ensure that V4 maintains the
"File Skipping" performance of V3 while gaining the write throughput of the
new tree structure.

I am drafting a short design doc outlining the schema changes and backward
compatibility implications for this.

Before I circulate the doc, has there been any consensus on how to handle
partition stats for inlined files in the combined Spitzer/Jahagirdar
proposal?

Regards,
Viquar Khan
Sr. Data Architect
https://www.linkedin.com/in/vaquar-khan-b695577/

Re: V4: Block-level Pruning for Inlined Metadata (Adaptive Metadata Tree)

[vaquar khan <va...@gmail.com>]

Thanks Amogh, Russell—appreciate the work refining the doc.

I’ve reviewed the proposal. You’re right about the Parquet stats—since the
Root Manifest is Parquet, we get those bounds for free in the footer. I’ll
drop the *PartitionSummary *struct from my proposal to align with that.

The tradeoff here is it creates a critical dependency on *writer behavior*
(layout) that didn't exist with the explicit Manifest List.

In V3, pruning worked regardless of file order. With this V4 design, if a
streaming writer just appends in commit-order (unsorted), we end up with
Row Groups spanning the entire table timeline. The min/max stats will be
too wide to prune anything, effectively forcing a full metadata scan on
every query.

Amogh, this circles back to the "nuances on writer requirements" you
mentioned.

I added a comment to the doc: we need to mandate that writers *SHOULD*
cluster/sort Root Manifest entries by Partition Key before flushing. That
ensures the implicit stats are tight enough to work without needing the
extra structs.

I have revised the document based on  Amogh, Russell review comments and
available for review .

https://docs.google.com/document/d/1mYTEK5eA6IjOc6yxRCvEBIzdbJO-rjXbr3YHtgWKNdo/edit?pli=1&tab=t.0


Regards,

Viquar khan

On Sat, 3 Jan 2026 at 20:30, vaquar khan <va...@gmail.com> wrote:

> Hi Amogh, Russell,
>
> Appreciate the feedback on the design proposal. I’ve addressed your
> comments in the doc.
>
>
> https://docs.google.com/document/d/1mYTEK5eA6IjOc6yxRCvEBIzdbJO-rjXbr3YHtgWKNdo/edit?pli=1&tab=t.0
>
> I think we’re actually quite close on the goals for V4. I’ve added some
> clarification on why I still believe the InlinedFileGroup structure is
> necessary—mostly to handle the CPU overhead at scale and the "noise" we get
> with min/max stats on bucketed data.
>
> Please review to see if we can close most of the discussion in doc before
> sync so our meeting will be short and to the point.
>
> Regards,
>
> Viquar Khan
>
>
> On Fri, 2 Jan 2026 at 16:59, Kevin Liu <ke...@apache.org> wrote:
>
>> Hey everyone,
>>
>> Thanks for the great discussion.
>>
>> My mental model of the Adaptive Metadata Tree proposal is that all the
>> existing metadata collected by Iceberg will remain the same but located in
>> a single file instead of across many metadata files.
>>
>> > Specifically, when Data Files are inlined into the Root Manifest, we
>> lose the explicit partition summary bounds that existed in the V3 Manifest
>> List.
>>
>> I don't think this information is lost. The `partition` field from the
>> manifest list is mapped to the column stats. This is called out in the "How
>> existing manifest list fields map to Proposed V4 content entry fields"
>> section [1]
>>
>> I might be missing something here since I looked over the proposal
>> briefly. Happy to chat more about it at the upcoming sync.
>>
>> Best,
>> Kevin Liu
>>
>> [1]
>> https://docs.google.com/document/d/1k4x8utgh41Sn1tr98eynDKCWq035SV_f75rtNHcerVw/edit?tab=t.0#heading=h.ol03vwxcdqw1
>>
>>
>> On Fri, Jan 2, 2026 at 12:42 PM Russell Spitzer <
>> russell.spitzer@gmail.com> wrote:
>>
>>> I also don't understand the concerns about the new tree. I think Amogh
>>> has been answering this quite well.
>>>
>>>  Imagine a buffer in the Root Manifest with 100 inlined files. Because
>>> streaming data arrives by time and not by partition, every file contains a
>>> random scatter of user_id buckets.
>>> *File 1:* contains buckets {0, 5, 14, 15} → Parquet Min/Max: [0, 15]
>>> *File 2:* contains buckets {1, 2, 8, 15} → Parquet Min/Max: [0, 15]
>>> ... and so on for all 100 files.
>>>
>>> If a reader queries for bucket(user_id) = 5, the Parquet footer stats
>>> for *every single file* will report a range of [0, 15]. Min/max pruning
>>> eliminates nothing. To determine if bucket 5 actually exists in those
>>> files, the REST server or engine must now project the column chunk and
>>> decode the dictionary/data pages for all 100 entries.
>>>
>>>  In V3, we could skip a whole manifest group using a single summary
>>> tuple (O(1) skipping). I
>>>
>>>
>>> I don't understand how in V3 we could skip a manifest with these files
>>> since the partition summaries also only stored min/max of the transformed
>>> values so we would get Bucket MinMax(0, 15) and (1, 15) respectively and
>>> both would hit for bucket = 5.
>>>
>>> Perhaps you mean something like
>>>
>>> Manifest 1: Contains buckets {2,3} - Parquet Column Min/Max (0, 15)
>>> Manifest 2: Contains buckets (4.5| - Parquet Column Min/Max (0, 15)
>>>
>>> In this case manifest 2 could be excluded in V3 and will remain
>>> excludable in V4 with this example. Much like in V3, a writer would have to
>>> choose to group datafiles with the same partition tuples together or use an
>>> after the fact compaction to reorganize datafile entries to get this
>>> behavior though.
>>>
>>> On Thu, Jan 1, 2026 at 8:58 PM Amogh Jahagirdar <2a...@gmail.com>
>>> wrote:
>>>
>>>> Sounds good to me, I left some comments
>>>> <https://docs.google.com/document/d/1mYTEK5eA6IjOc6yxRCvEBIzdbJO-rjXbr3YHtgWKNdo/edit?tab=t.0>
>>>> in the beginning of the proposal; I think this may be partially a
>>>> misunderstanding on how exactly the data files are stored in the root
>>>> manifest combined with how the tree does need to be rebalanced so that a
>>>> tenable root is maintained; that way the amount of work being done when
>>>> reading the root can be bounded, even in the presence of hot data. If there
>>>> are still doubts, let's chat about it in the sync (will schedule this and
>>>> notify in the existing single file commits thread).
>>>>
>>>> Thanks,
>>>> Amogh Jahagirdar
>>>>
>>>> On Thu, Jan 1, 2026 at 4:26 PM vaquar khan <va...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Amogh  ,
>>>>>
>>>>> I think it is best to meet and discuss this directly rather than
>>>>> continuing a long email trail, as it seems we are looking at this issue
>>>>> through two different lenses.
>>>>>
>>>>> My perspective comes from my day-to-day job helping big financial
>>>>> customers run billions of data records in production using Iceberg. My
>>>>> concerns regarding the Adaptive Metadata Tree and scanning overhead are
>>>>> rooted in the operational realities I am working on day to day  in these
>>>>> high-scale environments.
>>>>>
>>>>> I am happy to pause here and join the upcoming sync to discuss
>>>>> in detail .
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Viquar khan
>>>>>
>>>>> On Thu, 1 Jan 2026 at 15:43, Amogh Jahagirdar <2a...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> >If a reader queries for bucket(user_id) = 5, the Parquet footer
>>>>>> stats for *every single file* will report a range of [0, 15].
>>>>>> Min/max pruning eliminates nothing. To determine if bucket 5 actually
>>>>>> exists in those files, the REST server or engine must now project the
>>>>>> column chunk and decode the dictionary/data pages for all 100 entries.
>>>>>> >1. In V3, we could skip a whole manifest group using a single
>>>>>> summary tuple (O(1) skipping). In this V4 scenario, we move to O(N)
>>>>>> internal scanning. Does the spec intend to accept this linear scan cost as
>>>>>> the trade-off for write throughput, or is there a "pre-index" mechanism I’m
>>>>>> missing that avoids decoding data pages for every sub-second query?
>>>>>> >2. For a high-concurrency REST Catalog, the CPU and memory overhead
>>>>>> of performing "partial" Parquet decodes on hundreds of inlined entries per
>>>>>> request seems non-trivial. How do we ensure the catalog remains performant
>>>>>> if it has to become a "mini-query engine" just to perform basic partition
>>>>>> pruning?
>>>>>>
>>>>>> The provided example doesn't make sense to me since it contradicts
>>>>>> bucketing as a partition transform. If user_id was bucketed on, this
>>>>>> *necessarily* means that all values in a given file would have the
>>>>>> same bucket values of user_id (a file which is partitioned by something
>>>>>> must have all of the same partition values). So File 1 could *not*
>>>>>> contain a spread of buckets {0, 5, 14, 15} (and same principle with the
>>>>>> rest of the files). Of course there can be many user_ids in a file, but the
>>>>>> *bucket* they'd be in for a given file must be the same otherwise we
>>>>>> cannot say that file is bucketed.
>>>>>>
>>>>>> Let's say it wasn't bucketing, and it was some arbitrary clustering
>>>>>> function represented through expressions. We can't say it's a regression
>>>>>> because today this pruning based on column or derived stats is not even
>>>>>> possible at the root level of the metadata tree, since manifest lists only
>>>>>> have the upper/lower on partition values that exist in a given manifest. So
>>>>>> the new version should be a net improvement for planning costs.
>>>>>>
>>>>>> *In fact, even if this was possible today* this scenario we still
>>>>>> wouldn't be able say it's a regression in planning because we'd then be
>>>>>> comparing the cost of I/O and decoding N manifest entries in the Avro
>>>>>> manifest list (in today's fast append in the worst case there'd be a single
>>>>>> manifest per single write, putting aside manifest rewrites), vs the cost of
>>>>>> I/o and decoding X data files in the root buffer + M manifests (the fanout
>>>>>> from the root of the tree) in the Parquet root manifest. So if we're
>>>>>> analyzing the cost of reading the root of the metadata tree, the core of
>>>>>> that really is the cost of reading the V4 Parquet root manifest vs the cost
>>>>>> of reading the Avro manifest list. This comes down to numbers (more on this
>>>>>> later) but assuming comparable sizes and logical contents in the root, just
>>>>>> from a theoretical perspective we can see that planning cost is an
>>>>>> improvement in v4.
>>>>>>
>>>>>> >If the solution to the scan cost is to flush to leaf manifests more
>>>>>> frequently, don't we risk re-introducing the file-commit frequency issues
>>>>>> (and S3/GCS throttling) that Single-File Commits were specifically designed
>>>>>> to solve?
>>>>>>
>>>>>> Not necessarily, as the flushing to leaf manifests doesn't need to
>>>>>> happen on the ingest path, it may be a background maintenance (this choice
>>>>>> of if/how/when to rebalance is part of the whole "adaptive" part), but I do
>>>>>> think here's where we should provide numbers to better demonstrate this.
>>>>>> This is a matter of an amortized analysis (root size, expected commit
>>>>>> latency, rebalancing costs given some frequency and clustering etc) for
>>>>>> different types of workloads at different scale factors, i.e. how much of a
>>>>>> buffer for single file commits in the root is desirable, at any given point
>>>>>> in time, whereas batch workloads generally won't really care too much about
>>>>>> this.
>>>>>>
>>>>>> I think it's important to emphasize here that while one of the goals
>>>>>> is to enable the table format to do single file commits for small writes,
>>>>>> this doesn't mean that *every *write *always* has to be a single
>>>>>> file commit; there are a different set of tradeoffs that are imposed by
>>>>>> that. The proposed adaptive structure does allow for that if desired by
>>>>>> doing background maintenance, or writers can choose to incur that cost at
>>>>>> some point of their choosing.
>>>>>> Additionally, one of the other tests that was run a while back (that
>>>>>> I don't think is on the doc, but I'll update it with those details on the
>>>>>> appendix) was a simple s3 put latency test on different root sizes; I
>>>>>> believe what was tested was root sizes ranging from 3kb to 4mb, and while
>>>>>> of course latency increases, it did not increase *linearly *with
>>>>>> respect to size, and the latency differences are much smaller than one
>>>>>> would expect.
>>>>>>
>>>>>> >My proposal for the Compact Partition Summary (CPS) is essentially a
>>>>>> "fast-lane" index in the Root Manifest header. It provides group-level
>>>>>> exclusion for these "dirty" streaming buffers so we don't have to touch the
>>>>>> Parquet data pages at all unless we know the data is there.
>>>>>>
>>>>>> I'll take a look at this when I get a chance this week but I'm pretty
>>>>>> skeptical that this additional complexity at the root level really buys us
>>>>>> anything here. From a quick scan I did of the contents, parts of the
>>>>>> proposal look to be AI generated, which is fine, but the
>>>>>> assumptions/conclusions drawn by it aren't quite right imo.
>>>>>>
>>>>>> I do think this discussion has brought up a good point around
>>>>>> numbers, and we didn't get to it in the last sync but one of the topics we
>>>>>> wanted to discuss was around inline bitmap representations (which has
>>>>>> metadata footprint implications) , and relates to metadata maintenance
>>>>>> costs for DML heavy operations. I was planning on setting up another sync
>>>>>> when more people are back from holidays, and since this topic also relates
>>>>>> to manifest entry sizes, and scaling dynamics, perhaps we could discuss it
>>>>>> in the sync as well? That also gives others more time to understand and
>>>>>> comment on the proposal.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Amogh Jahagirdar
>>>>>>
>>>>>> On Thu, Jan 1, 2026 at 11:07 AM vaquar khan <va...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Amogh,
>>>>>>>
>>>>>>> Thanks for the detailed perspective. I’ve updated the document
>>>>>>> permissions—I’d love to get your specific thoughts on the schema sections.
>>>>>>>
>>>>>>> I want to dig deeper into the "Read Regression" point because I
>>>>>>> think we might be looking at different use cases. I completely agree that
>>>>>>> for batch-processed, well-sorted data, Parquet's columnar stats are a win.
>>>>>>> However, the "hard blocker" I’m seeing is in the streaming ingest tail.
>>>>>>>
>>>>>>>  Imagine a buffer in the Root Manifest with 100 inlined files.
>>>>>>> Because streaming data arrives by time and not by partition, every file
>>>>>>> contains a random scatter of user_id buckets.
>>>>>>>
>>>>>>>    -
>>>>>>>
>>>>>>>    *File 1:* contains buckets {0, 5, 14, 15} → Parquet Min/Max: [0,
>>>>>>>    15]
>>>>>>>    -
>>>>>>>
>>>>>>>    *File 2:* contains buckets {1, 2, 8, 15} → Parquet Min/Max: [0,
>>>>>>>    15]
>>>>>>>    -
>>>>>>>
>>>>>>>    ... and so on for all 100 files.
>>>>>>>
>>>>>>> If a reader queries for bucket(user_id) = 5, the Parquet footer
>>>>>>> stats for *every single file* will report a range of [0, 15].
>>>>>>> Min/max pruning eliminates nothing. To determine if bucket 5 actually
>>>>>>> exists in those files, the REST server or engine must now project the
>>>>>>> column chunk and decode the dictionary/data pages for all 100 entries.
>>>>>>>
>>>>>>> This leads to a few questions I’m struggling with regarding the
>>>>>>> current V4 vision:
>>>>>>>
>>>>>>>    1.
>>>>>>>
>>>>>>>     In V3, we could skip a whole manifest group using a single
>>>>>>>    summary tuple (O(1) skipping). In this V4 scenario, we move to O(N)
>>>>>>>    internal scanning. Does the spec intend to accept this linear scan cost as
>>>>>>>    the trade-off for write throughput, or is there a "pre-index" mechanism I’m
>>>>>>>    missing that avoids decoding data pages for every sub-second query?
>>>>>>>    2.
>>>>>>>
>>>>>>>    For a high-concurrency REST Catalog, the CPU and memory overhead
>>>>>>>    of performing "partial" Parquet decodes on hundreds of inlined entries per
>>>>>>>    request seems non-trivial. How do we ensure the catalog remains performant
>>>>>>>    if it has to become a "mini-query engine" just to perform basic partition
>>>>>>>    pruning?
>>>>>>>    3.
>>>>>>>
>>>>>>>     If the solution to the scan cost is to flush to leaf manifests
>>>>>>>    more frequently, don't we risk re-introducing the file-commit frequency
>>>>>>>    issues (and S3/GCS throttling) that Single-File Commits were specifically
>>>>>>>    designed to solve?
>>>>>>>
>>>>>>> My proposal for the Compact Partition Summary (CPS) is essentially a
>>>>>>> "fast-lane" index in the Root Manifest header. It provides group-level
>>>>>>> exclusion for these "dirty" streaming buffers so we don't have to touch the
>>>>>>> Parquet data pages at all unless we know the data is there.
>>>>>>>
>>>>>>> Does this scenario resonate with the performance goals you have for
>>>>>>> your proposal, or do you see a different way to handle the "random scatter"
>>>>>>> metadata problem?
>>>>>>> Regards,
>>>>>>> Viquar Khan
>>>>>>> Sr. Data Architect
>>>>>>> https://www.linkedin.com/in/vaquar-khan-b695577/
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Vaquar Khan
>>>>>
>>>>>
>
> --
> Regards,
> Vaquar Khan
>
>

-- 
Regards,
Vaquar Khan

Re: V4: Block-level Pruning for Inlined Metadata (Adaptive Metadata Tree)

[Innocent Djiofack <dj...@gmail.com>]

As a new community member, I have closely been following the thread and
would appreciate If I could be included in the call or if the call could
somehow be recorded for people who want to learn more about planning.

Thanks.

On Mon, Jan 5, 2026 at 10:24 AM vaquar khan <va...@gmail.com> wrote:

> Hi Amogh,
> Thanks for the response. I'd definitely appreciate the 1:1 to walk through
> the planning examples and correct my understanding of the V4 metadata
> structure.
> I want to ensure I'm fully aligned with the current design rationale. Let
> me find a common time to send you a 1-1 message to set up a call .
>
> regards,
> Viquar Khan
>
> On Mon, 5 Jan 2026 at 11:03, Amogh Jahagirdar <2a...@gmail.com> wrote:
>
>> Hey Vaquar,
>>
>> I see that the proposal changed quite a lot; even in the new proposal, I
>> think there's still some fundamental misunderstandings in the current
>> metadata structure, the proposed metadata structure (in particular how
>> stats and partitioning would be represented in the entries).
>>
>> I'm a little concerned that using community sync time to talk about this
>> wouldn't be a good use of time, especially since I think a lot of the
>> community can see that there's no clear, legitimate issue here.
>>
>> I'm happy to talk 1:1 (including anyone else that's interested) and we
>> can walk through concrete examples of how planning would work, with
>> specific manifest entry contents before V4 and after, and I think then our
>> rationale would be made a lot more clear. I'd prefer having a separate
>> conversation, rather than using broader community time since there are
>> other important topics that we've wanted to discuss that we haven't quite
>> gotten to and we ideally don't want to rehash topics that the rest of the
>> community is already aware of.
>>
>> Thanks,
>>
>> Amogh Jahagirdar
>>
>
>
> --
> Regards,
> Vaquar Khan
>
>

-- 

*DJIOFACK INNOCENT*
*"Be better than the day before!" -*
*+1 404 751 8024*

Re: V4: Block-level Pruning for Inlined Metadata (Adaptive Metadata Tree)

[vaquar khan <va...@gmail.com>]

Hi Amogh,
Thanks for the response. I'd definitely appreciate the 1:1 to walk through
the planning examples and correct my understanding of the V4 metadata
structure.
I want to ensure I'm fully aligned with the current design rationale. Let
me find a common time to send you a 1-1 message to set up a call .

regards,
Viquar Khan

On Mon, 5 Jan 2026 at 11:03, Amogh Jahagirdar <2a...@gmail.com> wrote:

> Hey Vaquar,
>
> I see that the proposal changed quite a lot; even in the new proposal, I
> think there's still some fundamental misunderstandings in the current
> metadata structure, the proposed metadata structure (in particular how
> stats and partitioning would be represented in the entries).
>
> I'm a little concerned that using community sync time to talk about this
> wouldn't be a good use of time, especially since I think a lot of the
> community can see that there's no clear, legitimate issue here.
>
> I'm happy to talk 1:1 (including anyone else that's interested) and we can
> walk through concrete examples of how planning would work, with specific
> manifest entry contents before V4 and after, and I think then our rationale
> would be made a lot more clear. I'd prefer having a separate conversation,
> rather than using broader community time since there are other important
> topics that we've wanted to discuss that we haven't quite gotten to and we
> ideally don't want to rehash topics that the rest of the community is
> already aware of.
>
> Thanks,
>
> Amogh Jahagirdar
>


-- 
Regards,
Vaquar Khan

Re: V4: Block-level Pruning for Inlined Metadata (Adaptive Metadata Tree)

[Amogh Jahagirdar <2a...@gmail.com>]

Hey Vaquar,

I see that the proposal changed quite a lot; even in the new proposal, I
think there's still some fundamental misunderstandings in the current
metadata structure, the proposed metadata structure (in particular how
stats and partitioning would be represented in the entries).

I'm a little concerned that using community sync time to talk about this
wouldn't be a good use of time, especially since I think a lot of the
community can see that there's no clear, legitimate issue here.

I'm happy to talk 1:1 (including anyone else that's interested) and we can
walk through concrete examples of how planning would work, with specific
manifest entry contents before V4 and after, and I think then our rationale
would be made a lot more clear. I'd prefer having a separate conversation,
rather than using broader community time since there are other important
topics that we've wanted to discuss that we haven't quite gotten to and we
ideally don't want to rehash topics that the rest of the community is
already aware of.

Thanks,

Amogh Jahagirdar

Re: V4: Block-level Pruning for Inlined Metadata (Adaptive Metadata Tree)

[vaquar khan <va...@gmail.com>]

Hi Amogh, Russell,

Appreciate the feedback on the design proposal. I’ve addressed your
comments in the doc.

https://docs.google.com/document/d/1mYTEK5eA6IjOc6yxRCvEBIzdbJO-rjXbr3YHtgWKNdo/edit?pli=1&tab=t.0

I think we’re actually quite close on the goals for V4. I’ve added some
clarification on why I still believe the InlinedFileGroup structure is
necessary—mostly to handle the CPU overhead at scale and the "noise" we get
with min/max stats on bucketed data.

Please review to see if we can close most of the discussion in doc before
sync so our meeting will be short and to the point.

Regards,

Viquar Khan


On Fri, 2 Jan 2026 at 16:59, Kevin Liu <ke...@apache.org> wrote:

> Hey everyone,
>
> Thanks for the great discussion.
>
> My mental model of the Adaptive Metadata Tree proposal is that all the
> existing metadata collected by Iceberg will remain the same but located in
> a single file instead of across many metadata files.
>
> > Specifically, when Data Files are inlined into the Root Manifest, we
> lose the explicit partition summary bounds that existed in the V3 Manifest
> List.
>
> I don't think this information is lost. The `partition` field from the
> manifest list is mapped to the column stats. This is called out in the "How
> existing manifest list fields map to Proposed V4 content entry fields"
> section [1]
>
> I might be missing something here since I looked over the proposal
> briefly. Happy to chat more about it at the upcoming sync.
>
> Best,
> Kevin Liu
>
> [1]
> https://docs.google.com/document/d/1k4x8utgh41Sn1tr98eynDKCWq035SV_f75rtNHcerVw/edit?tab=t.0#heading=h.ol03vwxcdqw1
>
>
> On Fri, Jan 2, 2026 at 12:42 PM Russell Spitzer <ru...@gmail.com>
> wrote:
>
>> I also don't understand the concerns about the new tree. I think Amogh
>> has been answering this quite well.
>>
>>  Imagine a buffer in the Root Manifest with 100 inlined files. Because
>> streaming data arrives by time and not by partition, every file contains a
>> random scatter of user_id buckets.
>> *File 1:* contains buckets {0, 5, 14, 15} → Parquet Min/Max: [0, 15]
>> *File 2:* contains buckets {1, 2, 8, 15} → Parquet Min/Max: [0, 15]
>> ... and so on for all 100 files.
>>
>> If a reader queries for bucket(user_id) = 5, the Parquet footer stats
>> for *every single file* will report a range of [0, 15]. Min/max pruning
>> eliminates nothing. To determine if bucket 5 actually exists in those
>> files, the REST server or engine must now project the column chunk and
>> decode the dictionary/data pages for all 100 entries.
>>
>>  In V3, we could skip a whole manifest group using a single summary tuple
>> (O(1) skipping). I
>>
>>
>> I don't understand how in V3 we could skip a manifest with these files
>> since the partition summaries also only stored min/max of the transformed
>> values so we would get Bucket MinMax(0, 15) and (1, 15) respectively and
>> both would hit for bucket = 5.
>>
>> Perhaps you mean something like
>>
>> Manifest 1: Contains buckets {2,3} - Parquet Column Min/Max (0, 15)
>> Manifest 2: Contains buckets (4.5| - Parquet Column Min/Max (0, 15)
>>
>> In this case manifest 2 could be excluded in V3 and will remain
>> excludable in V4 with this example. Much like in V3, a writer would have to
>> choose to group datafiles with the same partition tuples together or use an
>> after the fact compaction to reorganize datafile entries to get this
>> behavior though.
>>
>> On Thu, Jan 1, 2026 at 8:58 PM Amogh Jahagirdar <2a...@gmail.com> wrote:
>>
>>> Sounds good to me, I left some comments
>>> <https://docs.google.com/document/d/1mYTEK5eA6IjOc6yxRCvEBIzdbJO-rjXbr3YHtgWKNdo/edit?tab=t.0>
>>> in the beginning of the proposal; I think this may be partially a
>>> misunderstanding on how exactly the data files are stored in the root
>>> manifest combined with how the tree does need to be rebalanced so that a
>>> tenable root is maintained; that way the amount of work being done when
>>> reading the root can be bounded, even in the presence of hot data. If there
>>> are still doubts, let's chat about it in the sync (will schedule this and
>>> notify in the existing single file commits thread).
>>>
>>> Thanks,
>>> Amogh Jahagirdar
>>>
>>> On Thu, Jan 1, 2026 at 4:26 PM vaquar khan <va...@gmail.com>
>>> wrote:
>>>
>>>> Hi Amogh  ,
>>>>
>>>> I think it is best to meet and discuss this directly rather than
>>>> continuing a long email trail, as it seems we are looking at this issue
>>>> through two different lenses.
>>>>
>>>> My perspective comes from my day-to-day job helping big financial
>>>> customers run billions of data records in production using Iceberg. My
>>>> concerns regarding the Adaptive Metadata Tree and scanning overhead are
>>>> rooted in the operational realities I am working on day to day  in these
>>>> high-scale environments.
>>>>
>>>> I am happy to pause here and join the upcoming sync to discuss
>>>> in detail .
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Viquar khan
>>>>
>>>> On Thu, 1 Jan 2026 at 15:43, Amogh Jahagirdar <2a...@gmail.com> wrote:
>>>>
>>>>> >If a reader queries for bucket(user_id) = 5, the Parquet footer
>>>>> stats for *every single file* will report a range of [0, 15]. Min/max
>>>>> pruning eliminates nothing. To determine if bucket 5 actually exists in
>>>>> those files, the REST server or engine must now project the column chunk
>>>>> and decode the dictionary/data pages for all 100 entries.
>>>>> >1. In V3, we could skip a whole manifest group using a single summary
>>>>> tuple (O(1) skipping). In this V4 scenario, we move to O(N) internal
>>>>> scanning. Does the spec intend to accept this linear scan cost as the
>>>>> trade-off for write throughput, or is there a "pre-index" mechanism I’m
>>>>> missing that avoids decoding data pages for every sub-second query?
>>>>> >2. For a high-concurrency REST Catalog, the CPU and memory overhead
>>>>> of performing "partial" Parquet decodes on hundreds of inlined entries per
>>>>> request seems non-trivial. How do we ensure the catalog remains performant
>>>>> if it has to become a "mini-query engine" just to perform basic partition
>>>>> pruning?
>>>>>
>>>>> The provided example doesn't make sense to me since it contradicts
>>>>> bucketing as a partition transform. If user_id was bucketed on, this
>>>>> *necessarily* means that all values in a given file would have the
>>>>> same bucket values of user_id (a file which is partitioned by something
>>>>> must have all of the same partition values). So File 1 could *not*
>>>>> contain a spread of buckets {0, 5, 14, 15} (and same principle with the
>>>>> rest of the files). Of course there can be many user_ids in a file, but the
>>>>> *bucket* they'd be in for a given file must be the same otherwise we
>>>>> cannot say that file is bucketed.
>>>>>
>>>>> Let's say it wasn't bucketing, and it was some arbitrary clustering
>>>>> function represented through expressions. We can't say it's a regression
>>>>> because today this pruning based on column or derived stats is not even
>>>>> possible at the root level of the metadata tree, since manifest lists only
>>>>> have the upper/lower on partition values that exist in a given manifest. So
>>>>> the new version should be a net improvement for planning costs.
>>>>>
>>>>> *In fact, even if this was possible today* this scenario we still
>>>>> wouldn't be able say it's a regression in planning because we'd then be
>>>>> comparing the cost of I/O and decoding N manifest entries in the Avro
>>>>> manifest list (in today's fast append in the worst case there'd be a single
>>>>> manifest per single write, putting aside manifest rewrites), vs the cost of
>>>>> I/o and decoding X data files in the root buffer + M manifests (the fanout
>>>>> from the root of the tree) in the Parquet root manifest. So if we're
>>>>> analyzing the cost of reading the root of the metadata tree, the core of
>>>>> that really is the cost of reading the V4 Parquet root manifest vs the cost
>>>>> of reading the Avro manifest list. This comes down to numbers (more on this
>>>>> later) but assuming comparable sizes and logical contents in the root, just
>>>>> from a theoretical perspective we can see that planning cost is an
>>>>> improvement in v4.
>>>>>
>>>>> >If the solution to the scan cost is to flush to leaf manifests more
>>>>> frequently, don't we risk re-introducing the file-commit frequency issues
>>>>> (and S3/GCS throttling) that Single-File Commits were specifically designed
>>>>> to solve?
>>>>>
>>>>> Not necessarily, as the flushing to leaf manifests doesn't need to
>>>>> happen on the ingest path, it may be a background maintenance (this choice
>>>>> of if/how/when to rebalance is part of the whole "adaptive" part), but I do
>>>>> think here's where we should provide numbers to better demonstrate this.
>>>>> This is a matter of an amortized analysis (root size, expected commit
>>>>> latency, rebalancing costs given some frequency and clustering etc) for
>>>>> different types of workloads at different scale factors, i.e. how much of a
>>>>> buffer for single file commits in the root is desirable, at any given point
>>>>> in time, whereas batch workloads generally won't really care too much about
>>>>> this.
>>>>>
>>>>> I think it's important to emphasize here that while one of the goals
>>>>> is to enable the table format to do single file commits for small writes,
>>>>> this doesn't mean that *every *write *always* has to be a single file
>>>>> commit; there are a different set of tradeoffs that are imposed by that.
>>>>> The proposed adaptive structure does allow for that if desired by doing
>>>>> background maintenance, or writers can choose to incur that cost at some
>>>>> point of their choosing.
>>>>> Additionally, one of the other tests that was run a while back (that I
>>>>> don't think is on the doc, but I'll update it with those details on the
>>>>> appendix) was a simple s3 put latency test on different root sizes; I
>>>>> believe what was tested was root sizes ranging from 3kb to 4mb, and while
>>>>> of course latency increases, it did not increase *linearly *with
>>>>> respect to size, and the latency differences are much smaller than one
>>>>> would expect.
>>>>>
>>>>> >My proposal for the Compact Partition Summary (CPS) is essentially a
>>>>> "fast-lane" index in the Root Manifest header. It provides group-level
>>>>> exclusion for these "dirty" streaming buffers so we don't have to touch the
>>>>> Parquet data pages at all unless we know the data is there.
>>>>>
>>>>> I'll take a look at this when I get a chance this week but I'm pretty
>>>>> skeptical that this additional complexity at the root level really buys us
>>>>> anything here. From a quick scan I did of the contents, parts of the
>>>>> proposal look to be AI generated, which is fine, but the
>>>>> assumptions/conclusions drawn by it aren't quite right imo.
>>>>>
>>>>> I do think this discussion has brought up a good point around numbers,
>>>>> and we didn't get to it in the last sync but one of the topics we wanted to
>>>>> discuss was around inline bitmap representations (which has metadata
>>>>> footprint implications) , and relates to metadata maintenance costs for DML
>>>>> heavy operations. I was planning on setting up another sync when more
>>>>> people are back from holidays, and since this topic also relates to
>>>>> manifest entry sizes, and scaling dynamics, perhaps we could discuss it in
>>>>> the sync as well? That also gives others more time to understand and
>>>>> comment on the proposal.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Amogh Jahagirdar
>>>>>
>>>>> On Thu, Jan 1, 2026 at 11:07 AM vaquar khan <va...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Amogh,
>>>>>>
>>>>>> Thanks for the detailed perspective. I’ve updated the document
>>>>>> permissions—I’d love to get your specific thoughts on the schema sections.
>>>>>>
>>>>>> I want to dig deeper into the "Read Regression" point because I think
>>>>>> we might be looking at different use cases. I completely agree that for
>>>>>> batch-processed, well-sorted data, Parquet's columnar stats are a win.
>>>>>> However, the "hard blocker" I’m seeing is in the streaming ingest tail.
>>>>>>
>>>>>>  Imagine a buffer in the Root Manifest with 100 inlined files.
>>>>>> Because streaming data arrives by time and not by partition, every file
>>>>>> contains a random scatter of user_id buckets.
>>>>>>
>>>>>>    -
>>>>>>
>>>>>>    *File 1:* contains buckets {0, 5, 14, 15} → Parquet Min/Max: [0,
>>>>>>    15]
>>>>>>    -
>>>>>>
>>>>>>    *File 2:* contains buckets {1, 2, 8, 15} → Parquet Min/Max: [0,
>>>>>>    15]
>>>>>>    -
>>>>>>
>>>>>>    ... and so on for all 100 files.
>>>>>>
>>>>>> If a reader queries for bucket(user_id) = 5, the Parquet footer
>>>>>> stats for *every single file* will report a range of [0, 15].
>>>>>> Min/max pruning eliminates nothing. To determine if bucket 5 actually
>>>>>> exists in those files, the REST server or engine must now project the
>>>>>> column chunk and decode the dictionary/data pages for all 100 entries.
>>>>>>
>>>>>> This leads to a few questions I’m struggling with regarding the
>>>>>> current V4 vision:
>>>>>>
>>>>>>    1.
>>>>>>
>>>>>>     In V3, we could skip a whole manifest group using a single
>>>>>>    summary tuple (O(1) skipping). In this V4 scenario, we move to O(N)
>>>>>>    internal scanning. Does the spec intend to accept this linear scan cost as
>>>>>>    the trade-off for write throughput, or is there a "pre-index" mechanism I’m
>>>>>>    missing that avoids decoding data pages for every sub-second query?
>>>>>>    2.
>>>>>>
>>>>>>    For a high-concurrency REST Catalog, the CPU and memory overhead
>>>>>>    of performing "partial" Parquet decodes on hundreds of inlined entries per
>>>>>>    request seems non-trivial. How do we ensure the catalog remains performant
>>>>>>    if it has to become a "mini-query engine" just to perform basic partition
>>>>>>    pruning?
>>>>>>    3.
>>>>>>
>>>>>>     If the solution to the scan cost is to flush to leaf manifests
>>>>>>    more frequently, don't we risk re-introducing the file-commit frequency
>>>>>>    issues (and S3/GCS throttling) that Single-File Commits were specifically
>>>>>>    designed to solve?
>>>>>>
>>>>>> My proposal for the Compact Partition Summary (CPS) is essentially a
>>>>>> "fast-lane" index in the Root Manifest header. It provides group-level
>>>>>> exclusion for these "dirty" streaming buffers so we don't have to touch the
>>>>>> Parquet data pages at all unless we know the data is there.
>>>>>>
>>>>>> Does this scenario resonate with the performance goals you have for
>>>>>> your proposal, or do you see a different way to handle the "random scatter"
>>>>>> metadata problem?
>>>>>> Regards,
>>>>>> Viquar Khan
>>>>>> Sr. Data Architect
>>>>>> https://www.linkedin.com/in/vaquar-khan-b695577/
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Vaquar Khan
>>>>
>>>>

-- 
Regards,
Vaquar Khan

Re: V4: Block-level Pruning for Inlined Metadata (Adaptive Metadata Tree)

[Kevin Liu <ke...@apache.org>]

Hey everyone,

Thanks for the great discussion.

My mental model of the Adaptive Metadata Tree proposal is that all the
existing metadata collected by Iceberg will remain the same but located in
a single file instead of across many metadata files.

> Specifically, when Data Files are inlined into the Root Manifest, we lose
the explicit partition summary bounds that existed in the V3 Manifest List.

I don't think this information is lost. The `partition` field from the
manifest list is mapped to the column stats. This is called out in the "How
existing manifest list fields map to Proposed V4 content entry fields"
section [1]

I might be missing something here since I looked over the proposal briefly.
Happy to chat more about it at the upcoming sync.

Best,
Kevin Liu

[1]
https://docs.google.com/document/d/1k4x8utgh41Sn1tr98eynDKCWq035SV_f75rtNHcerVw/edit?tab=t.0#heading=h.ol03vwxcdqw1


On Fri, Jan 2, 2026 at 12:42 PM Russell Spitzer <ru...@gmail.com>
wrote:

> I also don't understand the concerns about the new tree. I think Amogh has
> been answering this quite well.
>
>  Imagine a buffer in the Root Manifest with 100 inlined files. Because
> streaming data arrives by time and not by partition, every file contains a
> random scatter of user_id buckets.
> *File 1:* contains buckets {0, 5, 14, 15} → Parquet Min/Max: [0, 15]
> *File 2:* contains buckets {1, 2, 8, 15} → Parquet Min/Max: [0, 15]
> ... and so on for all 100 files.
>
> If a reader queries for bucket(user_id) = 5, the Parquet footer stats for *every
> single file* will report a range of [0, 15]. Min/max pruning eliminates
> nothing. To determine if bucket 5 actually exists in those files, the REST
> server or engine must now project the column chunk and decode the
> dictionary/data pages for all 100 entries.
>
>  In V3, we could skip a whole manifest group using a single summary tuple
> (O(1) skipping). I
>
>
> I don't understand how in V3 we could skip a manifest with these files
> since the partition summaries also only stored min/max of the transformed
> values so we would get Bucket MinMax(0, 15) and (1, 15) respectively and
> both would hit for bucket = 5.
>
> Perhaps you mean something like
>
> Manifest 1: Contains buckets {2,3} - Parquet Column Min/Max (0, 15)
> Manifest 2: Contains buckets (4.5| - Parquet Column Min/Max (0, 15)
>
> In this case manifest 2 could be excluded in V3 and will remain excludable
> in V4 with this example. Much like in V3, a writer would have to choose to
> group datafiles with the same partition tuples together or use an after the
> fact compaction to reorganize datafile entries to get this behavior though.
>
> On Thu, Jan 1, 2026 at 8:58 PM Amogh Jahagirdar <2a...@gmail.com> wrote:
>
>> Sounds good to me, I left some comments
>> <https://docs.google.com/document/d/1mYTEK5eA6IjOc6yxRCvEBIzdbJO-rjXbr3YHtgWKNdo/edit?tab=t.0>
>> in the beginning of the proposal; I think this may be partially a
>> misunderstanding on how exactly the data files are stored in the root
>> manifest combined with how the tree does need to be rebalanced so that a
>> tenable root is maintained; that way the amount of work being done when
>> reading the root can be bounded, even in the presence of hot data. If there
>> are still doubts, let's chat about it in the sync (will schedule this and
>> notify in the existing single file commits thread).
>>
>> Thanks,
>> Amogh Jahagirdar
>>
>> On Thu, Jan 1, 2026 at 4:26 PM vaquar khan <va...@gmail.com> wrote:
>>
>>> Hi Amogh  ,
>>>
>>> I think it is best to meet and discuss this directly rather than
>>> continuing a long email trail, as it seems we are looking at this issue
>>> through two different lenses.
>>>
>>> My perspective comes from my day-to-day job helping big financial
>>> customers run billions of data records in production using Iceberg. My
>>> concerns regarding the Adaptive Metadata Tree and scanning overhead are
>>> rooted in the operational realities I am working on day to day  in these
>>> high-scale environments.
>>>
>>> I am happy to pause here and join the upcoming sync to discuss in detail
>>> .
>>>
>>>
>>> Regards,
>>>
>>> Viquar khan
>>>
>>> On Thu, 1 Jan 2026 at 15:43, Amogh Jahagirdar <2a...@gmail.com> wrote:
>>>
>>>> >If a reader queries for bucket(user_id) = 5, the Parquet footer stats
>>>> for *every single file* will report a range of [0, 15]. Min/max
>>>> pruning eliminates nothing. To determine if bucket 5 actually exists in
>>>> those files, the REST server or engine must now project the column chunk
>>>> and decode the dictionary/data pages for all 100 entries.
>>>> >1. In V3, we could skip a whole manifest group using a single summary
>>>> tuple (O(1) skipping). In this V4 scenario, we move to O(N) internal
>>>> scanning. Does the spec intend to accept this linear scan cost as the
>>>> trade-off for write throughput, or is there a "pre-index" mechanism I’m
>>>> missing that avoids decoding data pages for every sub-second query?
>>>> >2. For a high-concurrency REST Catalog, the CPU and memory overhead of
>>>> performing "partial" Parquet decodes on hundreds of inlined entries per
>>>> request seems non-trivial. How do we ensure the catalog remains performant
>>>> if it has to become a "mini-query engine" just to perform basic partition
>>>> pruning?
>>>>
>>>> The provided example doesn't make sense to me since it contradicts
>>>> bucketing as a partition transform. If user_id was bucketed on, this
>>>> *necessarily* means that all values in a given file would have the
>>>> same bucket values of user_id (a file which is partitioned by something
>>>> must have all of the same partition values). So File 1 could *not*
>>>> contain a spread of buckets {0, 5, 14, 15} (and same principle with the
>>>> rest of the files). Of course there can be many user_ids in a file, but the
>>>> *bucket* they'd be in for a given file must be the same otherwise we
>>>> cannot say that file is bucketed.
>>>>
>>>> Let's say it wasn't bucketing, and it was some arbitrary clustering
>>>> function represented through expressions. We can't say it's a regression
>>>> because today this pruning based on column or derived stats is not even
>>>> possible at the root level of the metadata tree, since manifest lists only
>>>> have the upper/lower on partition values that exist in a given manifest. So
>>>> the new version should be a net improvement for planning costs.
>>>>
>>>> *In fact, even if this was possible today* this scenario we still
>>>> wouldn't be able say it's a regression in planning because we'd then be
>>>> comparing the cost of I/O and decoding N manifest entries in the Avro
>>>> manifest list (in today's fast append in the worst case there'd be a single
>>>> manifest per single write, putting aside manifest rewrites), vs the cost of
>>>> I/o and decoding X data files in the root buffer + M manifests (the fanout
>>>> from the root of the tree) in the Parquet root manifest. So if we're
>>>> analyzing the cost of reading the root of the metadata tree, the core of
>>>> that really is the cost of reading the V4 Parquet root manifest vs the cost
>>>> of reading the Avro manifest list. This comes down to numbers (more on this
>>>> later) but assuming comparable sizes and logical contents in the root, just
>>>> from a theoretical perspective we can see that planning cost is an
>>>> improvement in v4.
>>>>
>>>> >If the solution to the scan cost is to flush to leaf manifests more
>>>> frequently, don't we risk re-introducing the file-commit frequency issues
>>>> (and S3/GCS throttling) that Single-File Commits were specifically designed
>>>> to solve?
>>>>
>>>> Not necessarily, as the flushing to leaf manifests doesn't need to
>>>> happen on the ingest path, it may be a background maintenance (this choice
>>>> of if/how/when to rebalance is part of the whole "adaptive" part), but I do
>>>> think here's where we should provide numbers to better demonstrate this.
>>>> This is a matter of an amortized analysis (root size, expected commit
>>>> latency, rebalancing costs given some frequency and clustering etc) for
>>>> different types of workloads at different scale factors, i.e. how much of a
>>>> buffer for single file commits in the root is desirable, at any given point
>>>> in time, whereas batch workloads generally won't really care too much about
>>>> this.
>>>>
>>>> I think it's important to emphasize here that while one of the goals is
>>>> to enable the table format to do single file commits for small writes, this
>>>> doesn't mean that *every *write *always* has to be a single file
>>>> commit; there are a different set of tradeoffs that are imposed by that.
>>>> The proposed adaptive structure does allow for that if desired by doing
>>>> background maintenance, or writers can choose to incur that cost at some
>>>> point of their choosing.
>>>> Additionally, one of the other tests that was run a while back (that I
>>>> don't think is on the doc, but I'll update it with those details on the
>>>> appendix) was a simple s3 put latency test on different root sizes; I
>>>> believe what was tested was root sizes ranging from 3kb to 4mb, and while
>>>> of course latency increases, it did not increase *linearly *with
>>>> respect to size, and the latency differences are much smaller than one
>>>> would expect.
>>>>
>>>> >My proposal for the Compact Partition Summary (CPS) is essentially a
>>>> "fast-lane" index in the Root Manifest header. It provides group-level
>>>> exclusion for these "dirty" streaming buffers so we don't have to touch the
>>>> Parquet data pages at all unless we know the data is there.
>>>>
>>>> I'll take a look at this when I get a chance this week but I'm pretty
>>>> skeptical that this additional complexity at the root level really buys us
>>>> anything here. From a quick scan I did of the contents, parts of the
>>>> proposal look to be AI generated, which is fine, but the
>>>> assumptions/conclusions drawn by it aren't quite right imo.
>>>>
>>>> I do think this discussion has brought up a good point around numbers,
>>>> and we didn't get to it in the last sync but one of the topics we wanted to
>>>> discuss was around inline bitmap representations (which has metadata
>>>> footprint implications) , and relates to metadata maintenance costs for DML
>>>> heavy operations. I was planning on setting up another sync when more
>>>> people are back from holidays, and since this topic also relates to
>>>> manifest entry sizes, and scaling dynamics, perhaps we could discuss it in
>>>> the sync as well? That also gives others more time to understand and
>>>> comment on the proposal.
>>>>
>>>> Thanks,
>>>>
>>>> Amogh Jahagirdar
>>>>
>>>> On Thu, Jan 1, 2026 at 11:07 AM vaquar khan <va...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Amogh,
>>>>>
>>>>> Thanks for the detailed perspective. I’ve updated the document
>>>>> permissions—I’d love to get your specific thoughts on the schema sections.
>>>>>
>>>>> I want to dig deeper into the "Read Regression" point because I think
>>>>> we might be looking at different use cases. I completely agree that for
>>>>> batch-processed, well-sorted data, Parquet's columnar stats are a win.
>>>>> However, the "hard blocker" I’m seeing is in the streaming ingest tail.
>>>>>
>>>>>  Imagine a buffer in the Root Manifest with 100 inlined files. Because
>>>>> streaming data arrives by time and not by partition, every file contains a
>>>>> random scatter of user_id buckets.
>>>>>
>>>>>    -
>>>>>
>>>>>    *File 1:* contains buckets {0, 5, 14, 15} → Parquet Min/Max: [0,
>>>>>    15]
>>>>>    -
>>>>>
>>>>>    *File 2:* contains buckets {1, 2, 8, 15} → Parquet Min/Max: [0, 15]
>>>>>    -
>>>>>
>>>>>    ... and so on for all 100 files.
>>>>>
>>>>> If a reader queries for bucket(user_id) = 5, the Parquet footer stats
>>>>> for *every single file* will report a range of [0, 15]. Min/max
>>>>> pruning eliminates nothing. To determine if bucket 5 actually exists in
>>>>> those files, the REST server or engine must now project the column chunk
>>>>> and decode the dictionary/data pages for all 100 entries.
>>>>>
>>>>> This leads to a few questions I’m struggling with regarding the
>>>>> current V4 vision:
>>>>>
>>>>>    1.
>>>>>
>>>>>     In V3, we could skip a whole manifest group using a single
>>>>>    summary tuple (O(1) skipping). In this V4 scenario, we move to O(N)
>>>>>    internal scanning. Does the spec intend to accept this linear scan cost as
>>>>>    the trade-off for write throughput, or is there a "pre-index" mechanism I’m
>>>>>    missing that avoids decoding data pages for every sub-second query?
>>>>>    2.
>>>>>
>>>>>    For a high-concurrency REST Catalog, the CPU and memory overhead
>>>>>    of performing "partial" Parquet decodes on hundreds of inlined entries per
>>>>>    request seems non-trivial. How do we ensure the catalog remains performant
>>>>>    if it has to become a "mini-query engine" just to perform basic partition
>>>>>    pruning?
>>>>>    3.
>>>>>
>>>>>     If the solution to the scan cost is to flush to leaf manifests
>>>>>    more frequently, don't we risk re-introducing the file-commit frequency
>>>>>    issues (and S3/GCS throttling) that Single-File Commits were specifically
>>>>>    designed to solve?
>>>>>
>>>>> My proposal for the Compact Partition Summary (CPS) is essentially a
>>>>> "fast-lane" index in the Root Manifest header. It provides group-level
>>>>> exclusion for these "dirty" streaming buffers so we don't have to touch the
>>>>> Parquet data pages at all unless we know the data is there.
>>>>>
>>>>> Does this scenario resonate with the performance goals you have for
>>>>> your proposal, or do you see a different way to handle the "random scatter"
>>>>> metadata problem?
>>>>> Regards,
>>>>> Viquar Khan
>>>>> Sr. Data Architect
>>>>> https://www.linkedin.com/in/vaquar-khan-b695577/
>>>>>
>>>>
>>>
>>> --
>>> Regards,
>>> Vaquar Khan
>>>
>>>

Re: V4: Block-level Pruning for Inlined Metadata (Adaptive Metadata Tree)

[Russell Spitzer <ru...@gmail.com>]

I also don't understand the concerns about the new tree. I think Amogh has
been answering this quite well.

 Imagine a buffer in the Root Manifest with 100 inlined files. Because
streaming data arrives by time and not by partition, every file contains a
random scatter of user_id buckets.
*File 1:* contains buckets {0, 5, 14, 15} → Parquet Min/Max: [0, 15]
*File 2:* contains buckets {1, 2, 8, 15} → Parquet Min/Max: [0, 15]
... and so on for all 100 files.

If a reader queries for bucket(user_id) = 5, the Parquet footer stats
for *every
single file* will report a range of [0, 15]. Min/max pruning eliminates
nothing. To determine if bucket 5 actually exists in those files, the REST
server or engine must now project the column chunk and decode the
dictionary/data pages for all 100 entries.

 In V3, we could skip a whole manifest group using a single summary tuple
(O(1) skipping). I


I don't understand how in V3 we could skip a manifest with these files
since the partition summaries also only stored min/max of the transformed
values so we would get Bucket MinMax(0, 15) and (1, 15) respectively and
both would hit for bucket = 5.

Perhaps you mean something like

Manifest 1: Contains buckets {2,3} - Parquet Column Min/Max (0, 15)
Manifest 2: Contains buckets (4.5| - Parquet Column Min/Max (0, 15)

In this case manifest 2 could be excluded in V3 and will remain excludable
in V4 with this example. Much like in V3, a writer would have to choose to
group datafiles with the same partition tuples together or use an after the
fact compaction to reorganize datafile entries to get this behavior though.

On Thu, Jan 1, 2026 at 8:58 PM Amogh Jahagirdar <2a...@gmail.com> wrote:

> Sounds good to me, I left some comments
> <https://docs.google.com/document/d/1mYTEK5eA6IjOc6yxRCvEBIzdbJO-rjXbr3YHtgWKNdo/edit?tab=t.0>
> in the beginning of the proposal; I think this may be partially a
> misunderstanding on how exactly the data files are stored in the root
> manifest combined with how the tree does need to be rebalanced so that a
> tenable root is maintained; that way the amount of work being done when
> reading the root can be bounded, even in the presence of hot data. If there
> are still doubts, let's chat about it in the sync (will schedule this and
> notify in the existing single file commits thread).
>
> Thanks,
> Amogh Jahagirdar
>
> On Thu, Jan 1, 2026 at 4:26 PM vaquar khan <va...@gmail.com> wrote:
>
>> Hi Amogh  ,
>>
>> I think it is best to meet and discuss this directly rather than
>> continuing a long email trail, as it seems we are looking at this issue
>> through two different lenses.
>>
>> My perspective comes from my day-to-day job helping big financial
>> customers run billions of data records in production using Iceberg. My
>> concerns regarding the Adaptive Metadata Tree and scanning overhead are
>> rooted in the operational realities I am working on day to day  in these
>> high-scale environments.
>>
>> I am happy to pause here and join the upcoming sync to discuss in detail .
>>
>>
>> Regards,
>>
>> Viquar khan
>>
>> On Thu, 1 Jan 2026 at 15:43, Amogh Jahagirdar <2a...@gmail.com> wrote:
>>
>>> >If a reader queries for bucket(user_id) = 5, the Parquet footer stats
>>> for *every single file* will report a range of [0, 15]. Min/max pruning
>>> eliminates nothing. To determine if bucket 5 actually exists in those
>>> files, the REST server or engine must now project the column chunk and
>>> decode the dictionary/data pages for all 100 entries.
>>> >1. In V3, we could skip a whole manifest group using a single summary
>>> tuple (O(1) skipping). In this V4 scenario, we move to O(N) internal
>>> scanning. Does the spec intend to accept this linear scan cost as the
>>> trade-off for write throughput, or is there a "pre-index" mechanism I’m
>>> missing that avoids decoding data pages for every sub-second query?
>>> >2. For a high-concurrency REST Catalog, the CPU and memory overhead of
>>> performing "partial" Parquet decodes on hundreds of inlined entries per
>>> request seems non-trivial. How do we ensure the catalog remains performant
>>> if it has to become a "mini-query engine" just to perform basic partition
>>> pruning?
>>>
>>> The provided example doesn't make sense to me since it contradicts
>>> bucketing as a partition transform. If user_id was bucketed on, this
>>> *necessarily* means that all values in a given file would have the same
>>> bucket values of user_id (a file which is partitioned by something must
>>> have all of the same partition values). So File 1 could *not* contain a
>>> spread of buckets {0, 5, 14, 15} (and same principle with the rest of the
>>> files). Of course there can be many user_ids in a file, but the *bucket*
>>> they'd be in for a given file must be the same otherwise we cannot say that
>>> file is bucketed.
>>>
>>> Let's say it wasn't bucketing, and it was some arbitrary clustering
>>> function represented through expressions. We can't say it's a regression
>>> because today this pruning based on column or derived stats is not even
>>> possible at the root level of the metadata tree, since manifest lists only
>>> have the upper/lower on partition values that exist in a given manifest. So
>>> the new version should be a net improvement for planning costs.
>>>
>>> *In fact, even if this was possible today* this scenario we still
>>> wouldn't be able say it's a regression in planning because we'd then be
>>> comparing the cost of I/O and decoding N manifest entries in the Avro
>>> manifest list (in today's fast append in the worst case there'd be a single
>>> manifest per single write, putting aside manifest rewrites), vs the cost of
>>> I/o and decoding X data files in the root buffer + M manifests (the fanout
>>> from the root of the tree) in the Parquet root manifest. So if we're
>>> analyzing the cost of reading the root of the metadata tree, the core of
>>> that really is the cost of reading the V4 Parquet root manifest vs the cost
>>> of reading the Avro manifest list. This comes down to numbers (more on this
>>> later) but assuming comparable sizes and logical contents in the root, just
>>> from a theoretical perspective we can see that planning cost is an
>>> improvement in v4.
>>>
>>> >If the solution to the scan cost is to flush to leaf manifests more
>>> frequently, don't we risk re-introducing the file-commit frequency issues
>>> (and S3/GCS throttling) that Single-File Commits were specifically designed
>>> to solve?
>>>
>>> Not necessarily, as the flushing to leaf manifests doesn't need to
>>> happen on the ingest path, it may be a background maintenance (this choice
>>> of if/how/when to rebalance is part of the whole "adaptive" part), but I do
>>> think here's where we should provide numbers to better demonstrate this.
>>> This is a matter of an amortized analysis (root size, expected commit
>>> latency, rebalancing costs given some frequency and clustering etc) for
>>> different types of workloads at different scale factors, i.e. how much of a
>>> buffer for single file commits in the root is desirable, at any given point
>>> in time, whereas batch workloads generally won't really care too much about
>>> this.
>>>
>>> I think it's important to emphasize here that while one of the goals is
>>> to enable the table format to do single file commits for small writes, this
>>> doesn't mean that *every *write *always* has to be a single file
>>> commit; there are a different set of tradeoffs that are imposed by that.
>>> The proposed adaptive structure does allow for that if desired by doing
>>> background maintenance, or writers can choose to incur that cost at some
>>> point of their choosing.
>>> Additionally, one of the other tests that was run a while back (that I
>>> don't think is on the doc, but I'll update it with those details on the
>>> appendix) was a simple s3 put latency test on different root sizes; I
>>> believe what was tested was root sizes ranging from 3kb to 4mb, and while
>>> of course latency increases, it did not increase *linearly *with
>>> respect to size, and the latency differences are much smaller than one
>>> would expect.
>>>
>>> >My proposal for the Compact Partition Summary (CPS) is essentially a
>>> "fast-lane" index in the Root Manifest header. It provides group-level
>>> exclusion for these "dirty" streaming buffers so we don't have to touch the
>>> Parquet data pages at all unless we know the data is there.
>>>
>>> I'll take a look at this when I get a chance this week but I'm pretty
>>> skeptical that this additional complexity at the root level really buys us
>>> anything here. From a quick scan I did of the contents, parts of the
>>> proposal look to be AI generated, which is fine, but the
>>> assumptions/conclusions drawn by it aren't quite right imo.
>>>
>>> I do think this discussion has brought up a good point around numbers,
>>> and we didn't get to it in the last sync but one of the topics we wanted to
>>> discuss was around inline bitmap representations (which has metadata
>>> footprint implications) , and relates to metadata maintenance costs for DML
>>> heavy operations. I was planning on setting up another sync when more
>>> people are back from holidays, and since this topic also relates to
>>> manifest entry sizes, and scaling dynamics, perhaps we could discuss it in
>>> the sync as well? That also gives others more time to understand and
>>> comment on the proposal.
>>>
>>> Thanks,
>>>
>>> Amogh Jahagirdar
>>>
>>> On Thu, Jan 1, 2026 at 11:07 AM vaquar khan <va...@gmail.com>
>>> wrote:
>>>
>>>> Hi Amogh,
>>>>
>>>> Thanks for the detailed perspective. I’ve updated the document
>>>> permissions—I’d love to get your specific thoughts on the schema sections.
>>>>
>>>> I want to dig deeper into the "Read Regression" point because I think
>>>> we might be looking at different use cases. I completely agree that for
>>>> batch-processed, well-sorted data, Parquet's columnar stats are a win.
>>>> However, the "hard blocker" I’m seeing is in the streaming ingest tail.
>>>>
>>>>  Imagine a buffer in the Root Manifest with 100 inlined files. Because
>>>> streaming data arrives by time and not by partition, every file contains a
>>>> random scatter of user_id buckets.
>>>>
>>>>    -
>>>>
>>>>    *File 1:* contains buckets {0, 5, 14, 15} → Parquet Min/Max: [0, 15]
>>>>    -
>>>>
>>>>    *File 2:* contains buckets {1, 2, 8, 15} → Parquet Min/Max: [0, 15]
>>>>    -
>>>>
>>>>    ... and so on for all 100 files.
>>>>
>>>> If a reader queries for bucket(user_id) = 5, the Parquet footer stats
>>>> for *every single file* will report a range of [0, 15]. Min/max
>>>> pruning eliminates nothing. To determine if bucket 5 actually exists in
>>>> those files, the REST server or engine must now project the column chunk
>>>> and decode the dictionary/data pages for all 100 entries.
>>>>
>>>> This leads to a few questions I’m struggling with regarding the current
>>>> V4 vision:
>>>>
>>>>    1.
>>>>
>>>>     In V3, we could skip a whole manifest group using a single summary
>>>>    tuple (O(1) skipping). In this V4 scenario, we move to O(N) internal
>>>>    scanning. Does the spec intend to accept this linear scan cost as the
>>>>    trade-off for write throughput, or is there a "pre-index" mechanism I’m
>>>>    missing that avoids decoding data pages for every sub-second query?
>>>>    2.
>>>>
>>>>    For a high-concurrency REST Catalog, the CPU and memory overhead of
>>>>    performing "partial" Parquet decodes on hundreds of inlined entries per
>>>>    request seems non-trivial. How do we ensure the catalog remains performant
>>>>    if it has to become a "mini-query engine" just to perform basic partition
>>>>    pruning?
>>>>    3.
>>>>
>>>>     If the solution to the scan cost is to flush to leaf manifests
>>>>    more frequently, don't we risk re-introducing the file-commit frequency
>>>>    issues (and S3/GCS throttling) that Single-File Commits were specifically
>>>>    designed to solve?
>>>>
>>>> My proposal for the Compact Partition Summary (CPS) is essentially a
>>>> "fast-lane" index in the Root Manifest header. It provides group-level
>>>> exclusion for these "dirty" streaming buffers so we don't have to touch the
>>>> Parquet data pages at all unless we know the data is there.
>>>>
>>>> Does this scenario resonate with the performance goals you have for
>>>> your proposal, or do you see a different way to handle the "random scatter"
>>>> metadata problem?
>>>> Regards,
>>>> Viquar Khan
>>>> Sr. Data Architect
>>>> https://www.linkedin.com/in/vaquar-khan-b695577/
>>>>
>>>
>>
>> --
>> Regards,
>> Vaquar Khan
>>
>>

Re: V4: Block-level Pruning for Inlined Metadata (Adaptive Metadata Tree)

[Amogh Jahagirdar <2a...@gmail.com>]

Sounds good to me, I left some comments
<https://docs.google.com/document/d/1mYTEK5eA6IjOc6yxRCvEBIzdbJO-rjXbr3YHtgWKNdo/edit?tab=t.0>
in the beginning of the proposal; I think this may be partially a
misunderstanding on how exactly the data files are stored in the root
manifest combined with how the tree does need to be rebalanced so that a
tenable root is maintained; that way the amount of work being done when
reading the root can be bounded, even in the presence of hot data. If there
are still doubts, let's chat about it in the sync (will schedule this and
notify in the existing single file commits thread).

Thanks,
Amogh Jahagirdar

On Thu, Jan 1, 2026 at 4:26 PM vaquar khan <va...@gmail.com> wrote:

> Hi Amogh  ,
>
> I think it is best to meet and discuss this directly rather than
> continuing a long email trail, as it seems we are looking at this issue
> through two different lenses.
>
> My perspective comes from my day-to-day job helping big financial
> customers run billions of data records in production using Iceberg. My
> concerns regarding the Adaptive Metadata Tree and scanning overhead are
> rooted in the operational realities I am working on day to day  in these
> high-scale environments.
>
> I am happy to pause here and join the upcoming sync to discuss in detail .
>
>
> Regards,
>
> Viquar khan
>
> On Thu, 1 Jan 2026 at 15:43, Amogh Jahagirdar <2a...@gmail.com> wrote:
>
>> >If a reader queries for bucket(user_id) = 5, the Parquet footer stats
>> for *every single file* will report a range of [0, 15]. Min/max pruning
>> eliminates nothing. To determine if bucket 5 actually exists in those
>> files, the REST server or engine must now project the column chunk and
>> decode the dictionary/data pages for all 100 entries.
>> >1. In V3, we could skip a whole manifest group using a single summary
>> tuple (O(1) skipping). In this V4 scenario, we move to O(N) internal
>> scanning. Does the spec intend to accept this linear scan cost as the
>> trade-off for write throughput, or is there a "pre-index" mechanism I’m
>> missing that avoids decoding data pages for every sub-second query?
>> >2. For a high-concurrency REST Catalog, the CPU and memory overhead of
>> performing "partial" Parquet decodes on hundreds of inlined entries per
>> request seems non-trivial. How do we ensure the catalog remains performant
>> if it has to become a "mini-query engine" just to perform basic partition
>> pruning?
>>
>> The provided example doesn't make sense to me since it contradicts
>> bucketing as a partition transform. If user_id was bucketed on, this
>> *necessarily* means that all values in a given file would have the same
>> bucket values of user_id (a file which is partitioned by something must
>> have all of the same partition values). So File 1 could *not* contain a
>> spread of buckets {0, 5, 14, 15} (and same principle with the rest of the
>> files). Of course there can be many user_ids in a file, but the *bucket*
>> they'd be in for a given file must be the same otherwise we cannot say that
>> file is bucketed.
>>
>> Let's say it wasn't bucketing, and it was some arbitrary clustering
>> function represented through expressions. We can't say it's a regression
>> because today this pruning based on column or derived stats is not even
>> possible at the root level of the metadata tree, since manifest lists only
>> have the upper/lower on partition values that exist in a given manifest. So
>> the new version should be a net improvement for planning costs.
>>
>> *In fact, even if this was possible today* this scenario we still
>> wouldn't be able say it's a regression in planning because we'd then be
>> comparing the cost of I/O and decoding N manifest entries in the Avro
>> manifest list (in today's fast append in the worst case there'd be a single
>> manifest per single write, putting aside manifest rewrites), vs the cost of
>> I/o and decoding X data files in the root buffer + M manifests (the fanout
>> from the root of the tree) in the Parquet root manifest. So if we're
>> analyzing the cost of reading the root of the metadata tree, the core of
>> that really is the cost of reading the V4 Parquet root manifest vs the cost
>> of reading the Avro manifest list. This comes down to numbers (more on this
>> later) but assuming comparable sizes and logical contents in the root, just
>> from a theoretical perspective we can see that planning cost is an
>> improvement in v4.
>>
>> >If the solution to the scan cost is to flush to leaf manifests more
>> frequently, don't we risk re-introducing the file-commit frequency issues
>> (and S3/GCS throttling) that Single-File Commits were specifically designed
>> to solve?
>>
>> Not necessarily, as the flushing to leaf manifests doesn't need to happen
>> on the ingest path, it may be a background maintenance (this choice of
>> if/how/when to rebalance is part of the whole "adaptive" part), but I do
>> think here's where we should provide numbers to better demonstrate this.
>> This is a matter of an amortized analysis (root size, expected commit
>> latency, rebalancing costs given some frequency and clustering etc) for
>> different types of workloads at different scale factors, i.e. how much of a
>> buffer for single file commits in the root is desirable, at any given point
>> in time, whereas batch workloads generally won't really care too much about
>> this.
>>
>> I think it's important to emphasize here that while one of the goals is
>> to enable the table format to do single file commits for small writes, this
>> doesn't mean that *every *write *always* has to be a single file commit;
>> there are a different set of tradeoffs that are imposed by that. The
>> proposed adaptive structure does allow for that if desired by doing
>> background maintenance, or writers can choose to incur that cost at some
>> point of their choosing.
>> Additionally, one of the other tests that was run a while back (that I
>> don't think is on the doc, but I'll update it with those details on the
>> appendix) was a simple s3 put latency test on different root sizes; I
>> believe what was tested was root sizes ranging from 3kb to 4mb, and while
>> of course latency increases, it did not increase *linearly *with respect
>> to size, and the latency differences are much smaller than one would expect.
>>
>> >My proposal for the Compact Partition Summary (CPS) is essentially a
>> "fast-lane" index in the Root Manifest header. It provides group-level
>> exclusion for these "dirty" streaming buffers so we don't have to touch the
>> Parquet data pages at all unless we know the data is there.
>>
>> I'll take a look at this when I get a chance this week but I'm pretty
>> skeptical that this additional complexity at the root level really buys us
>> anything here. From a quick scan I did of the contents, parts of the
>> proposal look to be AI generated, which is fine, but the
>> assumptions/conclusions drawn by it aren't quite right imo.
>>
>> I do think this discussion has brought up a good point around numbers,
>> and we didn't get to it in the last sync but one of the topics we wanted to
>> discuss was around inline bitmap representations (which has metadata
>> footprint implications) , and relates to metadata maintenance costs for DML
>> heavy operations. I was planning on setting up another sync when more
>> people are back from holidays, and since this topic also relates to
>> manifest entry sizes, and scaling dynamics, perhaps we could discuss it in
>> the sync as well? That also gives others more time to understand and
>> comment on the proposal.
>>
>> Thanks,
>>
>> Amogh Jahagirdar
>>
>> On Thu, Jan 1, 2026 at 11:07 AM vaquar khan <va...@gmail.com>
>> wrote:
>>
>>> Hi Amogh,
>>>
>>> Thanks for the detailed perspective. I’ve updated the document
>>> permissions—I’d love to get your specific thoughts on the schema sections.
>>>
>>> I want to dig deeper into the "Read Regression" point because I think we
>>> might be looking at different use cases. I completely agree that for
>>> batch-processed, well-sorted data, Parquet's columnar stats are a win.
>>> However, the "hard blocker" I’m seeing is in the streaming ingest tail.
>>>
>>>  Imagine a buffer in the Root Manifest with 100 inlined files. Because
>>> streaming data arrives by time and not by partition, every file contains a
>>> random scatter of user_id buckets.
>>>
>>>    -
>>>
>>>    *File 1:* contains buckets {0, 5, 14, 15} → Parquet Min/Max: [0, 15]
>>>    -
>>>
>>>    *File 2:* contains buckets {1, 2, 8, 15} → Parquet Min/Max: [0, 15]
>>>    -
>>>
>>>    ... and so on for all 100 files.
>>>
>>> If a reader queries for bucket(user_id) = 5, the Parquet footer stats
>>> for *every single file* will report a range of [0, 15]. Min/max pruning
>>> eliminates nothing. To determine if bucket 5 actually exists in those
>>> files, the REST server or engine must now project the column chunk and
>>> decode the dictionary/data pages for all 100 entries.
>>>
>>> This leads to a few questions I’m struggling with regarding the current
>>> V4 vision:
>>>
>>>    1.
>>>
>>>     In V3, we could skip a whole manifest group using a single summary
>>>    tuple (O(1) skipping). In this V4 scenario, we move to O(N) internal
>>>    scanning. Does the spec intend to accept this linear scan cost as the
>>>    trade-off for write throughput, or is there a "pre-index" mechanism I’m
>>>    missing that avoids decoding data pages for every sub-second query?
>>>    2.
>>>
>>>    For a high-concurrency REST Catalog, the CPU and memory overhead of
>>>    performing "partial" Parquet decodes on hundreds of inlined entries per
>>>    request seems non-trivial. How do we ensure the catalog remains performant
>>>    if it has to become a "mini-query engine" just to perform basic partition
>>>    pruning?
>>>    3.
>>>
>>>     If the solution to the scan cost is to flush to leaf manifests more
>>>    frequently, don't we risk re-introducing the file-commit frequency issues
>>>    (and S3/GCS throttling) that Single-File Commits were specifically designed
>>>    to solve?
>>>
>>> My proposal for the Compact Partition Summary (CPS) is essentially a
>>> "fast-lane" index in the Root Manifest header. It provides group-level
>>> exclusion for these "dirty" streaming buffers so we don't have to touch the
>>> Parquet data pages at all unless we know the data is there.
>>>
>>> Does this scenario resonate with the performance goals you have for your
>>> proposal, or do you see a different way to handle the "random scatter"
>>> metadata problem?
>>> Regards,
>>> Viquar Khan
>>> Sr. Data Architect
>>> https://www.linkedin.com/in/vaquar-khan-b695577/
>>>
>>
>
> --
> Regards,
> Vaquar Khan
>
>

Re: V4: Block-level Pruning for Inlined Metadata (Adaptive Metadata Tree)

[vaquar khan <va...@gmail.com>]

Hi Amogh  ,

I think it is best to meet and discuss this directly rather than continuing
a long email trail, as it seems we are looking at this issue through two
different lenses.

My perspective comes from my day-to-day job helping big financial customers
run billions of data records in production using Iceberg. My concerns
regarding the Adaptive Metadata Tree and scanning overhead are rooted in
the operational realities I am working on day to day  in these high-scale
environments.

I am happy to pause here and join the upcoming sync to discuss in detail .


Regards,

Viquar khan

On Thu, 1 Jan 2026 at 15:43, Amogh Jahagirdar <2a...@gmail.com> wrote:

> >If a reader queries for bucket(user_id) = 5, the Parquet footer stats
> for *every single file* will report a range of [0, 15]. Min/max pruning
> eliminates nothing. To determine if bucket 5 actually exists in those
> files, the REST server or engine must now project the column chunk and
> decode the dictionary/data pages for all 100 entries.
> >1. In V3, we could skip a whole manifest group using a single summary
> tuple (O(1) skipping). In this V4 scenario, we move to O(N) internal
> scanning. Does the spec intend to accept this linear scan cost as the
> trade-off for write throughput, or is there a "pre-index" mechanism I’m
> missing that avoids decoding data pages for every sub-second query?
> >2. For a high-concurrency REST Catalog, the CPU and memory overhead of
> performing "partial" Parquet decodes on hundreds of inlined entries per
> request seems non-trivial. How do we ensure the catalog remains performant
> if it has to become a "mini-query engine" just to perform basic partition
> pruning?
>
> The provided example doesn't make sense to me since it contradicts
> bucketing as a partition transform. If user_id was bucketed on, this
> *necessarily* means that all values in a given file would have the same
> bucket values of user_id (a file which is partitioned by something must
> have all of the same partition values). So File 1 could *not* contain a
> spread of buckets {0, 5, 14, 15} (and same principle with the rest of the
> files). Of course there can be many user_ids in a file, but the *bucket*
> they'd be in for a given file must be the same otherwise we cannot say that
> file is bucketed.
>
> Let's say it wasn't bucketing, and it was some arbitrary clustering
> function represented through expressions. We can't say it's a regression
> because today this pruning based on column or derived stats is not even
> possible at the root level of the metadata tree, since manifest lists only
> have the upper/lower on partition values that exist in a given manifest. So
> the new version should be a net improvement for planning costs.
>
> *In fact, even if this was possible today* this scenario we still
> wouldn't be able say it's a regression in planning because we'd then be
> comparing the cost of I/O and decoding N manifest entries in the Avro
> manifest list (in today's fast append in the worst case there'd be a single
> manifest per single write, putting aside manifest rewrites), vs the cost of
> I/o and decoding X data files in the root buffer + M manifests (the fanout
> from the root of the tree) in the Parquet root manifest. So if we're
> analyzing the cost of reading the root of the metadata tree, the core of
> that really is the cost of reading the V4 Parquet root manifest vs the cost
> of reading the Avro manifest list. This comes down to numbers (more on this
> later) but assuming comparable sizes and logical contents in the root, just
> from a theoretical perspective we can see that planning cost is an
> improvement in v4.
>
> >If the solution to the scan cost is to flush to leaf manifests more
> frequently, don't we risk re-introducing the file-commit frequency issues
> (and S3/GCS throttling) that Single-File Commits were specifically designed
> to solve?
>
> Not necessarily, as the flushing to leaf manifests doesn't need to happen
> on the ingest path, it may be a background maintenance (this choice of
> if/how/when to rebalance is part of the whole "adaptive" part), but I do
> think here's where we should provide numbers to better demonstrate this.
> This is a matter of an amortized analysis (root size, expected commit
> latency, rebalancing costs given some frequency and clustering etc) for
> different types of workloads at different scale factors, i.e. how much of a
> buffer for single file commits in the root is desirable, at any given point
> in time, whereas batch workloads generally won't really care too much about
> this.
>
> I think it's important to emphasize here that while one of the goals is to
> enable the table format to do single file commits for small writes, this
> doesn't mean that *every *write *always* has to be a single file commit;
> there are a different set of tradeoffs that are imposed by that. The
> proposed adaptive structure does allow for that if desired by doing
> background maintenance, or writers can choose to incur that cost at some
> point of their choosing.
> Additionally, one of the other tests that was run a while back (that I
> don't think is on the doc, but I'll update it with those details on the
> appendix) was a simple s3 put latency test on different root sizes; I
> believe what was tested was root sizes ranging from 3kb to 4mb, and while
> of course latency increases, it did not increase *linearly *with respect
> to size, and the latency differences are much smaller than one would expect.
>
> >My proposal for the Compact Partition Summary (CPS) is essentially a
> "fast-lane" index in the Root Manifest header. It provides group-level
> exclusion for these "dirty" streaming buffers so we don't have to touch the
> Parquet data pages at all unless we know the data is there.
>
> I'll take a look at this when I get a chance this week but I'm pretty
> skeptical that this additional complexity at the root level really buys us
> anything here. From a quick scan I did of the contents, parts of the
> proposal look to be AI generated, which is fine, but the
> assumptions/conclusions drawn by it aren't quite right imo.
>
> I do think this discussion has brought up a good point around numbers, and
> we didn't get to it in the last sync but one of the topics we wanted to
> discuss was around inline bitmap representations (which has metadata
> footprint implications) , and relates to metadata maintenance costs for DML
> heavy operations. I was planning on setting up another sync when more
> people are back from holidays, and since this topic also relates to
> manifest entry sizes, and scaling dynamics, perhaps we could discuss it in
> the sync as well? That also gives others more time to understand and
> comment on the proposal.
>
> Thanks,
>
> Amogh Jahagirdar
>
> On Thu, Jan 1, 2026 at 11:07 AM vaquar khan <va...@gmail.com> wrote:
>
>> Hi Amogh,
>>
>> Thanks for the detailed perspective. I’ve updated the document
>> permissions—I’d love to get your specific thoughts on the schema sections.
>>
>> I want to dig deeper into the "Read Regression" point because I think we
>> might be looking at different use cases. I completely agree that for
>> batch-processed, well-sorted data, Parquet's columnar stats are a win.
>> However, the "hard blocker" I’m seeing is in the streaming ingest tail.
>>
>>  Imagine a buffer in the Root Manifest with 100 inlined files. Because
>> streaming data arrives by time and not by partition, every file contains a
>> random scatter of user_id buckets.
>>
>>    -
>>
>>    *File 1:* contains buckets {0, 5, 14, 15} → Parquet Min/Max: [0, 15]
>>    -
>>
>>    *File 2:* contains buckets {1, 2, 8, 15} → Parquet Min/Max: [0, 15]
>>    -
>>
>>    ... and so on for all 100 files.
>>
>> If a reader queries for bucket(user_id) = 5, the Parquet footer stats
>> for *every single file* will report a range of [0, 15]. Min/max pruning
>> eliminates nothing. To determine if bucket 5 actually exists in those
>> files, the REST server or engine must now project the column chunk and
>> decode the dictionary/data pages for all 100 entries.
>>
>> This leads to a few questions I’m struggling with regarding the current
>> V4 vision:
>>
>>    1.
>>
>>     In V3, we could skip a whole manifest group using a single summary
>>    tuple (O(1) skipping). In this V4 scenario, we move to O(N) internal
>>    scanning. Does the spec intend to accept this linear scan cost as the
>>    trade-off for write throughput, or is there a "pre-index" mechanism I’m
>>    missing that avoids decoding data pages for every sub-second query?
>>    2.
>>
>>    For a high-concurrency REST Catalog, the CPU and memory overhead of
>>    performing "partial" Parquet decodes on hundreds of inlined entries per
>>    request seems non-trivial. How do we ensure the catalog remains performant
>>    if it has to become a "mini-query engine" just to perform basic partition
>>    pruning?
>>    3.
>>
>>     If the solution to the scan cost is to flush to leaf manifests more
>>    frequently, don't we risk re-introducing the file-commit frequency issues
>>    (and S3/GCS throttling) that Single-File Commits were specifically designed
>>    to solve?
>>
>> My proposal for the Compact Partition Summary (CPS) is essentially a
>> "fast-lane" index in the Root Manifest header. It provides group-level
>> exclusion for these "dirty" streaming buffers so we don't have to touch the
>> Parquet data pages at all unless we know the data is there.
>>
>> Does this scenario resonate with the performance goals you have for your
>> proposal, or do you see a different way to handle the "random scatter"
>> metadata problem?
>> Regards,
>> Viquar Khan
>> Sr. Data Architect
>> https://www.linkedin.com/in/vaquar-khan-b695577/
>>
>

-- 
Regards,
Vaquar Khan

Re: V4: Block-level Pruning for Inlined Metadata (Adaptive Metadata Tree)

[Amogh Jahagirdar <2a...@gmail.com>]

>If a reader queries for bucket(user_id) = 5, the Parquet footer stats for *every
single file* will report a range of [0, 15]. Min/max pruning eliminates
nothing. To determine if bucket 5 actually exists in those files, the REST
server or engine must now project the column chunk and decode the
dictionary/data pages for all 100 entries.
>1. In V3, we could skip a whole manifest group using a single summary
tuple (O(1) skipping). In this V4 scenario, we move to O(N) internal
scanning. Does the spec intend to accept this linear scan cost as the
trade-off for write throughput, or is there a "pre-index" mechanism I’m
missing that avoids decoding data pages for every sub-second query?
>2. For a high-concurrency REST Catalog, the CPU and memory overhead of
performing "partial" Parquet decodes on hundreds of inlined entries per
request seems non-trivial. How do we ensure the catalog remains performant
if it has to become a "mini-query engine" just to perform basic partition
pruning?

The provided example doesn't make sense to me since it contradicts
bucketing as a partition transform. If user_id was bucketed on, this
*necessarily* means that all values in a given file would have the same
bucket values of user_id (a file which is partitioned by something must
have all of the same partition values). So File 1 could *not* contain a
spread of buckets {0, 5, 14, 15} (and same principle with the rest of the
files). Of course there can be many user_ids in a file, but the *bucket*
they'd be in for a given file must be the same otherwise we cannot say that
file is bucketed.

Let's say it wasn't bucketing, and it was some arbitrary clustering
function represented through expressions. We can't say it's a regression
because today this pruning based on column or derived stats is not even
possible at the root level of the metadata tree, since manifest lists only
have the upper/lower on partition values that exist in a given manifest. So
the new version should be a net improvement for planning costs.

*In fact, even if this was possible today* this scenario we still wouldn't
be able say it's a regression in planning because we'd then be comparing
the cost of I/O and decoding N manifest entries in the Avro manifest list
(in today's fast append in the worst case there'd be a single manifest per
single write, putting aside manifest rewrites), vs the cost of I/o and
decoding X data files in the root buffer + M manifests (the fanout from the
root of the tree) in the Parquet root manifest. So if we're analyzing the
cost of reading the root of the metadata tree, the core of that really is
the cost of reading the V4 Parquet root manifest vs the cost of reading the
Avro manifest list. This comes down to numbers (more on this later) but
assuming comparable sizes and logical contents in the root, just from a
theoretical perspective we can see that planning cost is an improvement in
v4.

>If the solution to the scan cost is to flush to leaf manifests more
frequently, don't we risk re-introducing the file-commit frequency issues
(and S3/GCS throttling) that Single-File Commits were specifically designed
to solve?

Not necessarily, as the flushing to leaf manifests doesn't need to happen
on the ingest path, it may be a background maintenance (this choice of
if/how/when to rebalance is part of the whole "adaptive" part), but I do
think here's where we should provide numbers to better demonstrate this.
This is a matter of an amortized analysis (root size, expected commit
latency, rebalancing costs given some frequency and clustering etc) for
different types of workloads at different scale factors, i.e. how much of a
buffer for single file commits in the root is desirable, at any given point
in time, whereas batch workloads generally won't really care too much about
this.

I think it's important to emphasize here that while one of the goals is to
enable the table format to do single file commits for small writes, this
doesn't mean that *every *write *always* has to be a single file commit;
there are a different set of tradeoffs that are imposed by that. The
proposed adaptive structure does allow for that if desired by doing
background maintenance, or writers can choose to incur that cost at some
point of their choosing.
Additionally, one of the other tests that was run a while back (that I
don't think is on the doc, but I'll update it with those details on the
appendix) was a simple s3 put latency test on different root sizes; I
believe what was tested was root sizes ranging from 3kb to 4mb, and while
of course latency increases, it did not increase *linearly *with respect to
size, and the latency differences are much smaller than one would expect.

>My proposal for the Compact Partition Summary (CPS) is essentially a
"fast-lane" index in the Root Manifest header. It provides group-level
exclusion for these "dirty" streaming buffers so we don't have to touch the
Parquet data pages at all unless we know the data is there.

I'll take a look at this when I get a chance this week but I'm pretty
skeptical that this additional complexity at the root level really buys us
anything here. From a quick scan I did of the contents, parts of the
proposal look to be AI generated, which is fine, but the
assumptions/conclusions drawn by it aren't quite right imo.

I do think this discussion has brought up a good point around numbers, and
we didn't get to it in the last sync but one of the topics we wanted to
discuss was around inline bitmap representations (which has metadata
footprint implications) , and relates to metadata maintenance costs for DML
heavy operations. I was planning on setting up another sync when more
people are back from holidays, and since this topic also relates to
manifest entry sizes, and scaling dynamics, perhaps we could discuss it in
the sync as well? That also gives others more time to understand and
comment on the proposal.

Thanks,

Amogh Jahagirdar

On Thu, Jan 1, 2026 at 11:07 AM vaquar khan <va...@gmail.com> wrote:

> Hi Amogh,
>
> Thanks for the detailed perspective. I’ve updated the document
> permissions—I’d love to get your specific thoughts on the schema sections.
>
> I want to dig deeper into the "Read Regression" point because I think we
> might be looking at different use cases. I completely agree that for
> batch-processed, well-sorted data, Parquet's columnar stats are a win.
> However, the "hard blocker" I’m seeing is in the streaming ingest tail.
>
>  Imagine a buffer in the Root Manifest with 100 inlined files. Because
> streaming data arrives by time and not by partition, every file contains a
> random scatter of user_id buckets.
>
>    -
>
>    *File 1:* contains buckets {0, 5, 14, 15} → Parquet Min/Max: [0, 15]
>    -
>
>    *File 2:* contains buckets {1, 2, 8, 15} → Parquet Min/Max: [0, 15]
>    -
>
>    ... and so on for all 100 files.
>
> If a reader queries for bucket(user_id) = 5, the Parquet footer stats for *every
> single file* will report a range of [0, 15]. Min/max pruning eliminates
> nothing. To determine if bucket 5 actually exists in those files, the REST
> server or engine must now project the column chunk and decode the
> dictionary/data pages for all 100 entries.
>
> This leads to a few questions I’m struggling with regarding the current V4
> vision:
>
>    1.
>
>     In V3, we could skip a whole manifest group using a single summary
>    tuple (O(1) skipping). In this V4 scenario, we move to O(N) internal
>    scanning. Does the spec intend to accept this linear scan cost as the
>    trade-off for write throughput, or is there a "pre-index" mechanism I’m
>    missing that avoids decoding data pages for every sub-second query?
>    2.
>
>    For a high-concurrency REST Catalog, the CPU and memory overhead of
>    performing "partial" Parquet decodes on hundreds of inlined entries per
>    request seems non-trivial. How do we ensure the catalog remains performant
>    if it has to become a "mini-query engine" just to perform basic partition
>    pruning?
>    3.
>
>     If the solution to the scan cost is to flush to leaf manifests more
>    frequently, don't we risk re-introducing the file-commit frequency issues
>    (and S3/GCS throttling) that Single-File Commits were specifically designed
>    to solve?
>
> My proposal for the Compact Partition Summary (CPS) is essentially a
> "fast-lane" index in the Root Manifest header. It provides group-level
> exclusion for these "dirty" streaming buffers so we don't have to touch the
> Parquet data pages at all unless we know the data is there.
>
> Does this scenario resonate with the performance goals you have for your
> proposal, or do you see a different way to handle the "random scatter"
> metadata problem?
> Regards,
> Viquar Khan
> Sr. Data Architect
> https://www.linkedin.com/in/vaquar-khan-b695577/
>

Re: V4: Block-level Pruning for Inlined Metadata (Adaptive Metadata Tree)

[vaquar khan <va...@gmail.com>]

Hi Amogh,

Thanks for the detailed perspective. I’ve updated the document
permissions—I’d love to get your specific thoughts on the schema sections.

I want to dig deeper into the "Read Regression" point because I think we
might be looking at different use cases. I completely agree that for
batch-processed, well-sorted data, Parquet's columnar stats are a win.
However, the "hard blocker" I’m seeing is in the streaming ingest tail.

 Imagine a buffer in the Root Manifest with 100 inlined files. Because
streaming data arrives by time and not by partition, every file contains a
random scatter of user_id buckets.

   -

   *File 1:* contains buckets {0, 5, 14, 15} → Parquet Min/Max: [0, 15]
   -

   *File 2:* contains buckets {1, 2, 8, 15} → Parquet Min/Max: [0, 15]
   -

   ... and so on for all 100 files.

If a reader queries for bucket(user_id) = 5, the Parquet footer stats
for *every
single file* will report a range of [0, 15]. Min/max pruning eliminates
nothing. To determine if bucket 5 actually exists in those files, the REST
server or engine must now project the column chunk and decode the
dictionary/data pages for all 100 entries.

This leads to a few questions I’m struggling with regarding the current V4
vision:

   1.

    In V3, we could skip a whole manifest group using a single summary
   tuple (O(1) skipping). In this V4 scenario, we move to O(N) internal
   scanning. Does the spec intend to accept this linear scan cost as the
   trade-off for write throughput, or is there a "pre-index" mechanism I’m
   missing that avoids decoding data pages for every sub-second query?
   2.

   For a high-concurrency REST Catalog, the CPU and memory overhead of
   performing "partial" Parquet decodes on hundreds of inlined entries per
   request seems non-trivial. How do we ensure the catalog remains performant
   if it has to become a "mini-query engine" just to perform basic partition
   pruning?
   3.

    If the solution to the scan cost is to flush to leaf manifests more
   frequently, don't we risk re-introducing the file-commit frequency issues
   (and S3/GCS throttling) that Single-File Commits were specifically designed
   to solve?

My proposal for the Compact Partition Summary (CPS) is essentially a
"fast-lane" index in the Root Manifest header. It provides group-level
exclusion for these "dirty" streaming buffers so we don't have to touch the
Parquet data pages at all unless we know the data is there.

Does this scenario resonate with the performance goals you have for your
proposal, or do you see a different way to handle the "random scatter"
metadata problem?
Regards,
Viquar Khan
Sr. Data Architect
https://www.linkedin.com/in/vaquar-khan-b695577/

Re: V4: Block-level Pruning for Inlined Metadata (Adaptive Metadata Tree)

[Amogh Jahagirdar <2a...@gmail.com>]

Hey Vaquar,

I don't have comment permissions on the doc, so if I could get that I could
comment in more detail in each section when I get a chance.

>  As you noted, we'd need to store bucket function results in the stats to
make that work. Doing that at the file level introduces a lot of
redundancy, and more importantly, it forces the planner to deserialize
every single inlined file entry to check those stats. That linear scan is
going to be a bottleneck for REST catalogs handling heavy concurrent loads.

It's inaccurate to say that it's a regression compared to the previous
version. If you think about the root manifest in V4, it can effectively
have the same contents as what a manifest list has in the current structure
plus the ability to buffer data files and delete files for small writes. At
certain points (when the root manifest has reached some size for instance),
writers may choose to flush the buffered files into leaf manifests (and
then update the root manifest to point to those),  rebalancing the metadata
tree. Large writes essentially behave the same as today where writers would
just produce the leaf manifests, and update the root manifest (what would
previously be a manifest list).

So for planning, we go from:

Before v4: readers having to read a manifest list and pruning which
manifests to read based on partition field summary, and then pruning
data/delete files from those manifests based on partitioning/stats  to

After v4: readers read a root manifest, pruning inlined data files, delete
files, and manifests to read based on derived column stats. I think one of
the concerns is if it's somehow an additional cost to decode the inlined
entries in the root, which I will address later on.

For the bucketing case, if you think about the manifest list partition
field summary today, it's also just storing lower and upper bound for
partition values contained in a manifest (i.e. lower(bucket(16,
device_id)), upper(bucket(16, device_id)). The derived stats for a manifest
could also just be lower/upper bounds of the same bucketing values. Also
keep in mind that in V4, manifests would be Parquet files, where stats
(including derived stats) would be stored in a columnar manner, getting
compression benefits, and column pruning benefits during planning (just
need to read the stats relevant to satisfy the query).

From a logical pruning capability you can see that it's the same in terms
of filtering out which files and layers in the tree to read.

In terms of how much work we'd need to do (I/O and CPU) when reading the
root manifest and pruning compared to today's manifest list, this  just
comes down to right sizing the root and choosing when to rebalance and
flush the inline entries or not (based on I/O, decoding costs of the
current root).Again considering that the V4 manifest structure is also
Parquet, we'd expect this to be an improvement since pruning based would
just need to project the stats required, and metadata footprint from stats
(which tends to dominate data file entry sizes) should be lower compared to
today's structure. Also one of the other major goals of this new format is
to provide better cacheability of metadata, and if you keep the root
manifest right sized, you'd be able to achieve better cache hits which
would serve those higher concurrency read cases better.

Let me know if that makes sense or not,
Thanks,
Amogh Jahagirdar

On Wed, Dec 31, 2025 at 6:30 PM vaquar khan <va...@gmail.com> wrote:

> Hey Amogh,
>
> Thanks for the clarification and the video links.
>
> I agree that expression-based stats work well for monotonic transforms
> like day(ts), so we are aligned there.
>
> My concern is specifically around the non-monotonic cases (bucketing) and
> the impact on server-side planning. As you noted, we'd need to store bucket
> function results in the stats to make that work. Doing that at the file
> level introduces a lot of redundancy, and more importantly, it forces the
> planner to deserialize every single inlined file entry to check those
> stats. That linear scan is going to be a bottleneck for REST catalogs
> handling heavy concurrent loads.
>
> I’ve written up a proposal for a hybrid approach that solves this. It uses
> a "Compact Partition Summary" (effectively a virtual manifest header) for
> group-level pruning, while relying on your expression stats for the
> file-level pruning.
>
> Here is the design doc:
> https://docs.google.com/document/d/1mYTEK5eA6IjOc6yxRCvEBIzdbJO-rjXbr3YHtgWKNdo/edit?tab=t.0#heading=h.xr9w02y1yuod
> <https://docs.google.com/document/d/1mYTEK5eA6IjOc6yxRCvEBIzdbJO-rjXbr3YHtgWKNdo/edit?tab=t.0&authuser=1#heading=h.xr9w02y1yuod>
>
> This way we get the V4 write throughput but keep the O(1) pruning for the
> active tail. Let me know if this hybrid structure works for you.
>
> Regards,
> Viquar Khan
> Sr Data Architect
> https://www.linkedin.com/in/vaquar-khan-b695577/
>
> On Wed, 31 Dec 2025 at 00:24, vaquar khan <va...@gmail.com> wrote:
>
>> Thanks, Amogh. I really appreciate your quick and detailed response. I’m
>> going to watch the video now and will get back to you shortly with my
>> thoughts.
>>
>> Regards,
>> Viquar Khan
>> Sr Data Architect
>> https://www.linkedin.com/in/vaquar-khan-b695577/
>>
>> On Tue, 30 Dec 2025 at 12:59, Amogh Jahagirdar <2a...@gmail.com> wrote:
>>
>>> Meant to say *partition tuple on data files.
>>>
>>> The other part I should clarify is that the reason why the stats on
>>> expressions approach preserves pruning is the majority of transforms
>>> (except bucketing) are monotonic. So the lower/upper bounds on those stats
>>> effectively "cover" the same amount. For bucketing, we'd have to store the
>>> result of the bucketing function in stats as well, so we can also preserve
>>> the ability to do things like bucket joins.
>>>
>>> On Tue, Dec 30, 2025 at 11:46 AM Amogh Jahagirdar <2a...@gmail.com>
>>> wrote:
>>>
>>>> Hey Viquar,
>>>>
>>>> There shouldn't be a read regression here since the data files would
>>>> have columnar stats which would cover the ability to prune based on
>>>> partitions (since essentially all the partition transforms are derivations
>>>> on a source data column). There's been discussions in the sync on if we
>>>> should keep the partition tuple for manifests and there's nuances on writer
>>>> requirements if we were to completely rely on column stats, but regardless
>>>> of if the partition tuple is kept or not, from a pruning perspective we
>>>> certainly want to keep the same level pruning as we had before; that's a
>>>> critical property to preserve.
>>>>
>>>> If we model the partition transform as an expression with its own ID,
>>>> we could then have stats on that expression.  e.g. if you have a column ts,
>>>> and partitioning days(ts), there'd be an expression <http:///> in
>>>> metadata representing days(ts), and in stats for the data file there'd be a
>>>> stat entry containing lower(days(ts)) and upper(days(ts)). For a
>>>> partitioned file, the lower and upper bounds would have to be equal. For a
>>>> leaf manifest in the root, we'd have the aggregated lower/upper stats which
>>>> is effectively the same as the partition field summary that exists today.
>>>> Then in short, a reader could just run data filters and get the same
>>>> level of pruning as before. Notice that in this modeling we avoid having to
>>>> tie a manifest to a given partition spec like what happens today.
>>>>
>>>> I do think the aspect to get to more of a conclusion on is if we should
>>>> keep the partition tuple or completely rely on stats on expressions. For
>>>> reference, from a past v4 sync
>>>> <https://drive.google.com/file/d/1gv8TrR5xzqqNxek7_sTZkpbwQx1M3dhK/view?usp=sharing&t=2327> discussion
>>>> on this topic (linked to the time the discussion start). Let me know if
>>>> that makes sense!
>>>>
>>>> On Tue, Dec 30, 2025 at 10:47 AM vaquar khan <va...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> I’ve been following the recent discussions and design documents
>>>>> regarding the Adaptive Metadata Tree and Single-File Commits for the V4
>>>>> Spec.
>>>>>
>>>>> While moving to a Root Manifest structure solves the write
>>>>> amplification issue on S3/GCS, I am concerned about a potential regression
>>>>> in Partition Pruning efficiency for readers. Specifically, when Data Files
>>>>> are inlined into the Root Manifest, we lose the explicit partition summary
>>>>> bounds that existed in the V3 Manifest List.
>>>>>
>>>>> Without a standardized way to store lightweight partition stats for
>>>>> these inlined entries, query planners may be forced to scan significantly
>>>>> more metadata bytes to perform the same pruning we get for free today.
>>>>>
>>>>> *Proposal*: I propose we explicitly standardize a "Compact Partition
>>>>> Summary" (possibly using Bloom Filters or compressed min/max tuples) within
>>>>> the Root Manifest entry schema. This would ensure that V4 maintains the
>>>>> "File Skipping" performance of V3 while gaining the write throughput of the
>>>>> new tree structure.
>>>>>
>>>>> I am drafting a short design doc outlining the schema changes and
>>>>> backward compatibility implications for this.
>>>>>
>>>>> Before I circulate the doc, has there been any consensus on how to
>>>>> handle partition stats for inlined files in the combined Spitzer/Jahagirdar
>>>>> proposal?
>>>>>
>>>>> Regards,
>>>>> Viquar Khan
>>>>> Sr. Data Architect
>>>>> https://www.linkedin.com/in/vaquar-khan-b695577/
>>>>>
>>>>
>>
>> --
>> Regards,
>> Vaquar Khan
>>
>>
>
> --
> Regards,
> Vaquar Khan
>
>

Re: V4: Block-level Pruning for Inlined Metadata (Adaptive Metadata Tree)

[vaquar khan <va...@gmail.com>]

Hey Amogh,

Thanks for the clarification and the video links.

I agree that expression-based stats work well for monotonic transforms like
day(ts), so we are aligned there.

My concern is specifically around the non-monotonic cases (bucketing) and
the impact on server-side planning. As you noted, we'd need to store bucket
function results in the stats to make that work. Doing that at the file
level introduces a lot of redundancy, and more importantly, it forces the
planner to deserialize every single inlined file entry to check those
stats. That linear scan is going to be a bottleneck for REST catalogs
handling heavy concurrent loads.

I’ve written up a proposal for a hybrid approach that solves this. It uses
a "Compact Partition Summary" (effectively a virtual manifest header) for
group-level pruning, while relying on your expression stats for the
file-level pruning.

Here is the design doc:
https://docs.google.com/document/d/1mYTEK5eA6IjOc6yxRCvEBIzdbJO-rjXbr3YHtgWKNdo/edit?tab=t.0#heading=h.xr9w02y1yuod
<https://docs.google.com/document/d/1mYTEK5eA6IjOc6yxRCvEBIzdbJO-rjXbr3YHtgWKNdo/edit?tab=t.0&authuser=1#heading=h.xr9w02y1yuod>

This way we get the V4 write throughput but keep the O(1) pruning for the
active tail. Let me know if this hybrid structure works for you.

Regards,
Viquar Khan
Sr Data Architect
https://www.linkedin.com/in/vaquar-khan-b695577/

On Wed, 31 Dec 2025 at 00:24, vaquar khan <va...@gmail.com> wrote:

> Thanks, Amogh. I really appreciate your quick and detailed response. I’m
> going to watch the video now and will get back to you shortly with my
> thoughts.
>
> Regards,
> Viquar Khan
> Sr Data Architect
> https://www.linkedin.com/in/vaquar-khan-b695577/
>
> On Tue, 30 Dec 2025 at 12:59, Amogh Jahagirdar <2a...@gmail.com> wrote:
>
>> Meant to say *partition tuple on data files.
>>
>> The other part I should clarify is that the reason why the stats on
>> expressions approach preserves pruning is the majority of transforms
>> (except bucketing) are monotonic. So the lower/upper bounds on those stats
>> effectively "cover" the same amount. For bucketing, we'd have to store the
>> result of the bucketing function in stats as well, so we can also preserve
>> the ability to do things like bucket joins.
>>
>> On Tue, Dec 30, 2025 at 11:46 AM Amogh Jahagirdar <2a...@gmail.com>
>> wrote:
>>
>>> Hey Viquar,
>>>
>>> There shouldn't be a read regression here since the data files would
>>> have columnar stats which would cover the ability to prune based on
>>> partitions (since essentially all the partition transforms are derivations
>>> on a source data column). There's been discussions in the sync on if we
>>> should keep the partition tuple for manifests and there's nuances on writer
>>> requirements if we were to completely rely on column stats, but regardless
>>> of if the partition tuple is kept or not, from a pruning perspective we
>>> certainly want to keep the same level pruning as we had before; that's a
>>> critical property to preserve.
>>>
>>> If we model the partition transform as an expression with its own ID, we
>>> could then have stats on that expression.  e.g. if you have a column ts,
>>> and partitioning days(ts), there'd be an expression <http:///> in
>>> metadata representing days(ts), and in stats for the data file there'd be a
>>> stat entry containing lower(days(ts)) and upper(days(ts)). For a
>>> partitioned file, the lower and upper bounds would have to be equal. For a
>>> leaf manifest in the root, we'd have the aggregated lower/upper stats which
>>> is effectively the same as the partition field summary that exists today.
>>> Then in short, a reader could just run data filters and get the same
>>> level of pruning as before. Notice that in this modeling we avoid having to
>>> tie a manifest to a given partition spec like what happens today.
>>>
>>> I do think the aspect to get to more of a conclusion on is if we should
>>> keep the partition tuple or completely rely on stats on expressions. For
>>> reference, from a past v4 sync
>>> <https://drive.google.com/file/d/1gv8TrR5xzqqNxek7_sTZkpbwQx1M3dhK/view?usp=sharing&t=2327> discussion
>>> on this topic (linked to the time the discussion start). Let me know if
>>> that makes sense!
>>>
>>> On Tue, Dec 30, 2025 at 10:47 AM vaquar khan <va...@gmail.com>
>>> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> I’ve been following the recent discussions and design documents
>>>> regarding the Adaptive Metadata Tree and Single-File Commits for the V4
>>>> Spec.
>>>>
>>>> While moving to a Root Manifest structure solves the write
>>>> amplification issue on S3/GCS, I am concerned about a potential regression
>>>> in Partition Pruning efficiency for readers. Specifically, when Data Files
>>>> are inlined into the Root Manifest, we lose the explicit partition summary
>>>> bounds that existed in the V3 Manifest List.
>>>>
>>>> Without a standardized way to store lightweight partition stats for
>>>> these inlined entries, query planners may be forced to scan significantly
>>>> more metadata bytes to perform the same pruning we get for free today.
>>>>
>>>> *Proposal*: I propose we explicitly standardize a "Compact Partition
>>>> Summary" (possibly using Bloom Filters or compressed min/max tuples) within
>>>> the Root Manifest entry schema. This would ensure that V4 maintains the
>>>> "File Skipping" performance of V3 while gaining the write throughput of the
>>>> new tree structure.
>>>>
>>>> I am drafting a short design doc outlining the schema changes and
>>>> backward compatibility implications for this.
>>>>
>>>> Before I circulate the doc, has there been any consensus on how to
>>>> handle partition stats for inlined files in the combined Spitzer/Jahagirdar
>>>> proposal?
>>>>
>>>> Regards,
>>>> Viquar Khan
>>>> Sr. Data Architect
>>>> https://www.linkedin.com/in/vaquar-khan-b695577/
>>>>
>>>
>
> --
> Regards,
> Vaquar Khan
>
>

-- 
Regards,
Vaquar Khan

Re: V4: Block-level Pruning for Inlined Metadata (Adaptive Metadata Tree)

[vaquar khan <va...@gmail.com>]

Thanks, Amogh. I really appreciate your quick and detailed response. I’m
going to watch the video now and will get back to you shortly with my
thoughts.

Regards,
Viquar Khan
Sr Data Architect
https://www.linkedin.com/in/vaquar-khan-b695577/

On Tue, 30 Dec 2025 at 12:59, Amogh Jahagirdar <2a...@gmail.com> wrote:

> Meant to say *partition tuple on data files.
>
> The other part I should clarify is that the reason why the stats on
> expressions approach preserves pruning is the majority of transforms
> (except bucketing) are monotonic. So the lower/upper bounds on those stats
> effectively "cover" the same amount. For bucketing, we'd have to store the
> result of the bucketing function in stats as well, so we can also preserve
> the ability to do things like bucket joins.
>
> On Tue, Dec 30, 2025 at 11:46 AM Amogh Jahagirdar <2a...@gmail.com>
> wrote:
>
>> Hey Viquar,
>>
>> There shouldn't be a read regression here since the data files would have
>> columnar stats which would cover the ability to prune based on partitions
>> (since essentially all the partition transforms are derivations on a source
>> data column). There's been discussions in the sync on if we should keep the
>> partition tuple for manifests and there's nuances on writer requirements if
>> we were to completely rely on column stats, but regardless of if the
>> partition tuple is kept or not, from a pruning perspective we certainly
>> want to keep the same level pruning as we had before; that's a critical
>> property to preserve.
>>
>> If we model the partition transform as an expression with its own ID, we
>> could then have stats on that expression.  e.g. if you have a column ts,
>> and partitioning days(ts), there'd be an expression <http:///> in
>> metadata representing days(ts), and in stats for the data file there'd be a
>> stat entry containing lower(days(ts)) and upper(days(ts)). For a
>> partitioned file, the lower and upper bounds would have to be equal. For a
>> leaf manifest in the root, we'd have the aggregated lower/upper stats which
>> is effectively the same as the partition field summary that exists today.
>> Then in short, a reader could just run data filters and get the same
>> level of pruning as before. Notice that in this modeling we avoid having to
>> tie a manifest to a given partition spec like what happens today.
>>
>> I do think the aspect to get to more of a conclusion on is if we should
>> keep the partition tuple or completely rely on stats on expressions. For
>> reference, from a past v4 sync
>> <https://drive.google.com/file/d/1gv8TrR5xzqqNxek7_sTZkpbwQx1M3dhK/view?usp=sharing&t=2327> discussion
>> on this topic (linked to the time the discussion start). Let me know if
>> that makes sense!
>>
>> On Tue, Dec 30, 2025 at 10:47 AM vaquar khan <va...@gmail.com>
>> wrote:
>>
>>> Hi everyone,
>>>
>>> I’ve been following the recent discussions and design documents
>>> regarding the Adaptive Metadata Tree and Single-File Commits for the V4
>>> Spec.
>>>
>>> While moving to a Root Manifest structure solves the write amplification
>>> issue on S3/GCS, I am concerned about a potential regression in Partition
>>> Pruning efficiency for readers. Specifically, when Data Files are inlined
>>> into the Root Manifest, we lose the explicit partition summary bounds that
>>> existed in the V3 Manifest List.
>>>
>>> Without a standardized way to store lightweight partition stats for
>>> these inlined entries, query planners may be forced to scan significantly
>>> more metadata bytes to perform the same pruning we get for free today.
>>>
>>> *Proposal*: I propose we explicitly standardize a "Compact Partition
>>> Summary" (possibly using Bloom Filters or compressed min/max tuples) within
>>> the Root Manifest entry schema. This would ensure that V4 maintains the
>>> "File Skipping" performance of V3 while gaining the write throughput of the
>>> new tree structure.
>>>
>>> I am drafting a short design doc outlining the schema changes and
>>> backward compatibility implications for this.
>>>
>>> Before I circulate the doc, has there been any consensus on how to
>>> handle partition stats for inlined files in the combined Spitzer/Jahagirdar
>>> proposal?
>>>
>>> Regards,
>>> Viquar Khan
>>> Sr. Data Architect
>>> https://www.linkedin.com/in/vaquar-khan-b695577/
>>>
>>

-- 
Regards,
Vaquar Khan

Re: V4: Block-level Pruning for Inlined Metadata (Adaptive Metadata Tree)

[Amogh Jahagirdar <2a...@gmail.com>]

Meant to say *partition tuple on data files.

The other part I should clarify is that the reason why the stats on
expressions approach preserves pruning is the majority of transforms
(except bucketing) are monotonic. So the lower/upper bounds on those stats
effectively "cover" the same amount. For bucketing, we'd have to store the
result of the bucketing function in stats as well, so we can also preserve
the ability to do things like bucket joins.

On Tue, Dec 30, 2025 at 11:46 AM Amogh Jahagirdar <2a...@gmail.com> wrote:

> Hey Viquar,
>
> There shouldn't be a read regression here since the data files would have
> columnar stats which would cover the ability to prune based on partitions
> (since essentially all the partition transforms are derivations on a source
> data column). There's been discussions in the sync on if we should keep the
> partition tuple for manifests and there's nuances on writer requirements if
> we were to completely rely on column stats, but regardless of if the
> partition tuple is kept or not, from a pruning perspective we certainly
> want to keep the same level pruning as we had before; that's a critical
> property to preserve.
>
> If we model the partition transform as an expression with its own ID, we
> could then have stats on that expression.  e.g. if you have a column ts,
> and partitioning days(ts), there'd be an expression <http:///> in
> metadata representing days(ts), and in stats for the data file there'd be a
> stat entry containing lower(days(ts)) and upper(days(ts)). For a
> partitioned file, the lower and upper bounds would have to be equal. For a
> leaf manifest in the root, we'd have the aggregated lower/upper stats which
> is effectively the same as the partition field summary that exists today.
> Then in short, a reader could just run data filters and get the same level
> of pruning as before. Notice that in this modeling we avoid having to tie a
> manifest to a given partition spec like what happens today.
>
> I do think the aspect to get to more of a conclusion on is if we should
> keep the partition tuple or completely rely on stats on expressions. For
> reference, from a past v4 sync
> <https://drive.google.com/file/d/1gv8TrR5xzqqNxek7_sTZkpbwQx1M3dhK/view?usp=sharing&t=2327> discussion
> on this topic (linked to the time the discussion start). Let me know if
> that makes sense!
>
> On Tue, Dec 30, 2025 at 10:47 AM vaquar khan <va...@gmail.com>
> wrote:
>
>> Hi everyone,
>>
>> I’ve been following the recent discussions and design documents regarding
>> the Adaptive Metadata Tree and Single-File Commits for the V4 Spec.
>>
>> While moving to a Root Manifest structure solves the write amplification
>> issue on S3/GCS, I am concerned about a potential regression in Partition
>> Pruning efficiency for readers. Specifically, when Data Files are inlined
>> into the Root Manifest, we lose the explicit partition summary bounds that
>> existed in the V3 Manifest List.
>>
>> Without a standardized way to store lightweight partition stats for these
>> inlined entries, query planners may be forced to scan significantly more
>> metadata bytes to perform the same pruning we get for free today.
>>
>> *Proposal*: I propose we explicitly standardize a "Compact Partition
>> Summary" (possibly using Bloom Filters or compressed min/max tuples) within
>> the Root Manifest entry schema. This would ensure that V4 maintains the
>> "File Skipping" performance of V3 while gaining the write throughput of the
>> new tree structure.
>>
>> I am drafting a short design doc outlining the schema changes and
>> backward compatibility implications for this.
>>
>> Before I circulate the doc, has there been any consensus on how to handle
>> partition stats for inlined files in the combined Spitzer/Jahagirdar
>> proposal?
>>
>> Regards,
>> Viquar Khan
>> Sr. Data Architect
>> https://www.linkedin.com/in/vaquar-khan-b695577/
>>
>

Re: V4: Block-level Pruning for Inlined Metadata (Adaptive Metadata Tree)

[Amogh Jahagirdar <2a...@gmail.com>]

Hey Viquar,

There shouldn't be a read regression here since the data files would have
columnar stats which would cover the ability to prune based on partitions
(since essentially all the partition transforms are derivations on a source
data column). There's been discussions in the sync on if we should keep the
partition tuple for manifests and there's nuances on writer requirements if
we were to completely rely on column stats, but regardless of if the
partition tuple is kept or not, from a pruning perspective we certainly
want to keep the same level pruning as we had before; that's a critical
property to preserve.

If we model the partition transform as an expression with its own ID, we
could then have stats on that expression.  e.g. if you have a column ts,
and partitioning days(ts), there'd be an expression <http:///> in metadata
representing days(ts), and in stats for the data file there'd be a stat
entry containing lower(days(ts)) and upper(days(ts)). For a partitioned
file, the lower and upper bounds would have to be equal. For a leaf
manifest in the root, we'd have the aggregated lower/upper stats which is
effectively the same as the partition field summary that exists today.
Then in short, a reader could just run data filters and get the same level
of pruning as before. Notice that in this modeling we avoid having to tie a
manifest to a given partition spec like what happens today.

I do think the aspect to get to more of a conclusion on is if we should
keep the partition tuple or completely rely on stats on expressions. For
reference, from a past v4 sync
<https://drive.google.com/file/d/1gv8TrR5xzqqNxek7_sTZkpbwQx1M3dhK/view?usp=sharing&t=2327>
discussion
on this topic (linked to the time the discussion start). Let me know if
that makes sense!

On Tue, Dec 30, 2025 at 10:47 AM vaquar khan <va...@gmail.com> wrote:

> Hi everyone,
>
> I’ve been following the recent discussions and design documents regarding
> the Adaptive Metadata Tree and Single-File Commits for the V4 Spec.
>
> While moving to a Root Manifest structure solves the write amplification
> issue on S3/GCS, I am concerned about a potential regression in Partition
> Pruning efficiency for readers. Specifically, when Data Files are inlined
> into the Root Manifest, we lose the explicit partition summary bounds that
> existed in the V3 Manifest List.
>
> Without a standardized way to store lightweight partition stats for these
> inlined entries, query planners may be forced to scan significantly more
> metadata bytes to perform the same pruning we get for free today.
>
> *Proposal*: I propose we explicitly standardize a "Compact Partition
> Summary" (possibly using Bloom Filters or compressed min/max tuples) within
> the Root Manifest entry schema. This would ensure that V4 maintains the
> "File Skipping" performance of V3 while gaining the write throughput of the
> new tree structure.
>
> I am drafting a short design doc outlining the schema changes and backward
> compatibility implications for this.
>
> Before I circulate the doc, has there been any consensus on how to handle
> partition stats for inlined files in the combined Spitzer/Jahagirdar
> proposal?
>
> Regards,
> Viquar Khan
> Sr. Data Architect
> https://www.linkedin.com/in/vaquar-khan-b695577/
>