Re: Dedicated sync for Iceberg Index Support

[Péter Váry <pe...@gmail.com>]

Hi everyone,

Here is the recording of the Monday sync:
https://apache-iceberg.slack.com/files/U06538UFDH6/F0BTT739Y7P/impromptu_google_meet_meeting_-_aug_31_2026.mp4

Thanks, Shawn, for making it available.

A brief summary of the discussion:

   - We finalized the file naming:
      - Tracking file: contains the list of range files (similar to a
      manifest file for tables).
      - Range file: contains the index data rows (similar to a data file
      for tables).
   - We agreed not to prescribe how expression values are materialized (for
   example, Hilbert or hash transformations), leaving room for implementation
   flexibility.
   - Ryan mentioned that he is working on a v4 proposal for generating
   Field IDs for non-materialized columns. The current thinking is that index
   columns should follow the same approach.
   - We agreed to retain older versions of the index metadata JSON.
   - We agreed not to require a strict one-to-one mapping between index
   snapshots and table snapshots. An index may contain multiple snapshots
   corresponding to the same table snapshot.

Based on the discussion, I have updated the spec PR. Following Ryan's
pattern for non-materialized column definitions, the proposal now
introduces:

   - *materialized-fields* for range file columns
   - *non-materialized-fields* for tracking file statistics
   - *cluster-spec* to define which columns participate in range file
   clustering

Overall, I think we reached good alignment on the remaining design points,
and I do not see any open questions at the moment. The PR is in good shape
for review, so I would greatly appreciate any feedback or comments.

Thanks everyone for the productive discussion and continued collaboration.
Peter

Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026. aug. 18.,
K, 14:17):

> Hi everyone,
>
> Unfortunately, yesterday's sync was not recorded due to a technical issue.
> :(
>
> Here is a brief summary of the discussion and the decisions we reached:
>
>    1. We agreed to use a list of Iceberg expressions to define the
>    *index_keys*, which determine how the index is ordered.
>    2. We agreed not to restrict the functions that can be used in those
>    expressions. Engines that do not understand a particular expression can
>    simply choose not to use the index.
>    3. We agreed that index region files (leaf files) should be sorted by
>    the values generated by the index_keys.
>    4. We discussed using *included_column_ids* to define the contents and
>    schema of the index region files, and generally felt this could be a good
>    approach.
>    5. We started discussing whether the results of the index_key
>    expressions should also be stored in the index region files alongside the
>    included columns. In some cases, storing them could help readers optimize
>    lookups within a region file. In other cases, the values could be derived
>    when needed, making them redundant and potentially hurting performance by
>    increasing the size of the region files.
>
> I think we should continue the discussion around point 5, as the outcome
> may influence, or even reopen, the decisions around point 4.
>
> Thanks!
>
>
> Flavio Junqueira <fl...@gmail.com> ezt írta (időpont: 2026.
> aug. 4., K, 18:57):
>
>> Thank you for the summary, the proposal and the overall discussion. I
>> have one main question about the target and I apologize in advance if I'm
>> missing some important detail as I might not have the full context.
>>
>> I noticed that partitioning, sorting and clustering (via Hilbert) were
>> discussed in the call in this context of index transform functions. My
>> understanding is that engines currently are already pruning partitions and
>> files using partition information and table metadata like per-column
>> min-max. If that's the case, then what's it that this proposal is trying to
>> achieve by having transform functions that capture partitioning, sorting
>> and clustering when engines are already able to prune without a secondary
>> index? I can more easily see the primary key index example, but I'm not
>> entirely sure about these data layout mechanisms.
>>
>> I also didn't quite get how engines are expected to consume such an
>> index. There was a discussion about where the mapping of the input to files
>> happens, whether that's in Iceberg or the engine, and I didn't quite get
>> the recommendation. Checking Peter's proposal, I couldn't see it clearly
>> either, and if I just missed it, I'd appreciate a pointer to where I should
>> look into.
>>
>> -Flavio
>>
>> On 4 Aug 2026, at 10:52, Péter Váry <pe...@gmail.com> wrote:
>>
>> Hi everyone,
>>
>> Here is the recording from yesterday’s sync:
>> https://www.youtube.com/watch?v=Wg9orP9JAmk
>>
>> To summarize:
>> - We agreed to retain the history of index snapshots, but not the history
>> of other index properties. If readers require information that may later be
>> modified by a user, that information should be copied into the
>> corresponding snapshot.
>> - We discussed the possibility of defining index ordering using a list of
>> transform functions.
>>
>> Dan and Yingyi suggested something along the following lines:
>>
>> *"index_keys" : [*
>> *  {*
>> *    "type": "apply",*
>> *    "func-name": {*
>> *      "catalog": "iceberg_functions",*
>> *      "name": "day"*
>> *    },*
>> *    "arguments": [*
>> *      { "type": "reference", "id": 3 }*
>> *    ]*
>> *  },*
>> *  {*
>> *    "type": "apply",*
>> *    "func-name": {*
>> *      "catalog": "iceberg_functions",*
>> *      "name": "truncate"*
>> *    },*
>> *    "arguments": [*
>> *      { "type": "reference", "id": 4 },*
>> *      2*
>> *    ]*
>> *  }*
>> *]*
>>
>>
>> In this example, rows would first be ordered by day(col3) and then by
>> truncate(col4, 2). To ensure a deterministic ordering, ties would be broken
>> using the original values of columns 3 and 4. If those are still equal,
>> ordering would fall back to the file name and finally the row position.
>>
>> I spent some time thinking about this after the sync, and here are my
>> thoughts.
>>
>> Pros:
>> - Provides a high degree of flexibility.
>> - Enables expression-based indexes.
>>
>> Cons:
>> - Do we need this flexibility at this stage?
>>    - Equality deletes, the primary use cases we are targeting, do not
>> require expressions.
>>    - Most indexes I have seen in production are not expression-based.
>> - Increased complexity for engines:
>>    - Engines must parse and understand transforms to determine whether
>> using an index can improve a query. For example, is ordering by
>> day(timestamp) sufficient for a key lookup?
>>    - Complexity increases further as new UDFs are introduced.
>>    - Expressions are not pushed into scans today, so index handling
>> cannot be fully hidden behind Iceberg scan planning.
>> - Risk of fragmentation:
>>    - The additional complexity may slow engine adoption.
>>    - Adopters may choose to support only exact transform matches created
>> by them, rather than learn to reason about the general transforms.
>>    - UDFs are prone to missing implementations and behavioral differences
>> across engines. In the worst case, such differences could lead to incorrect
>> query results when indexes are used.
>>
>> Given the points above, I still prefer defining HASH, HILBERT, IDENTITY
>> as a set of standardized values combined with an explicit list of key
>> columns. This keeps adoption simple, unlocks the primary use cases we are
>> targeting today, and allows the Iceberg library to provide the transform
>> implementations directly.
>>
>> Internally, these transforms could still be represented as Iceberg
>> expressions, giving us a migration path toward future index types that may
>> require full expression support.
>>
>> For example:
>>
>> *{*
>> *  "transform": "HILBERT",*
>> *  "key-column-ids": [3, 4]*
>> *}*
>>
>>
>> I'm interested in hearing other opinions.
>>
>> Thanks,
>> Peter
>>
>> Renjie Liu <li...@gmail.com> ezt írta (időpont: 2026. júl. 28.,
>> K, 10:07):
>>
>>> Hi, Peter:
>>>
>>> Thanks for driving the proposal.
>>>
>>> >  *Hash Index Type*: Should hash-based indexes be a separate index
>>> type? They are only applicable to equality and IN predicates, and the hash
>>> algorithm must always be known. A dedicated HASH type could make these
>>> constraints explicit and distinguish hash-based indexes from ordered
>>> indexes.
>>>
>>> I'm leaning toward keeping them in scalar index type due to the
>>> similarity with others transformation functions like identity. As with the
>>> constraints, I think they are properties of transformation functions.
>>>
>>> > *Transform Function/Layout Representation*: Should transform
>>> functions be represented as an enum or a string? I lean toward strings, but
>>> I do not have a strong preference. In many cases, an index can still be
>>> used based on standard column statistics, even if a reader does not
>>> recognize a particular transform.
>>>
>>> This seems more like a problem in java implementation?
>>>
>>> Also I prefer the name transform function compared with layout
>>> representation. Layout representation makes me feel that it's not about the
>>> definition of an index function.
>>>
>>> > *Future Index Types*: How should we handle index types that are
>>> expected in the long term but fall outside the scope of this proposal (for
>>> example, VECTOR, IVF, TERM)? Should they be mentioned as planned/reserved
>>> concepts, or omitted entirely? I lean toward at least mentioning them. The
>>> rationale for introducing an index type is easier to explain when
>>> considering future index families.
>>>
>>> +1
>>>
>>> *> Index History Handling:* Most database systems treat index changes
>>> as a drop-and-recreate operation rather than an in-place update. If we
>>> follow the same approach, there is limited value in maintaining index
>>> metadata history. During index maintenance, a new metadata.json can be
>>> created and atomically swapped in, with the catalog responsible for
>>> cleaning up obsolete metadata and index data files according to its
>>> retention policy.
>>>
>>> +1, I think we only need to keep index data for different snapshots, but
>>> if an index definition changed, we should drop and recreate a new index.
>>>
>>> > *File Format Location*: Should file format be specified at the
>>> leaf-file level or only at the index level? My preference is to stay
>>> aligned with the table metadata model and avoid introducing differences
>>> unless there is a compelling requirement. This would simplify
>>> implementation and maximize code reuse.
>>>
>>> +1.
>>>
>>>
>>>
>>> On Fri, Jul 24, 2026 at 11:57 PM Péter Váry <pe...@gmail.com>
>>> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> Here are the key takeaways from Monday’s meeting (July 20):
>>>>
>>>>    - *Index Type*: An index type enum will be introduced in v1,
>>>>    initially with a single value (SCALAR). This establishes a future-proof API
>>>>    boundary while leaving room for additional index families.
>>>>    - *Strict Total Ordering*: The specification should require
>>>>    transform functions to produce values that preserve a well-defined total
>>>>    ordering across the key space. The transform function does not need to
>>>>    return a long. Instead, leaf files can store the first and last values
>>>>    according to the ordering, enabling generic filtering capabilities for
>>>>    readers.
>>>>
>>>> Open Questions:
>>>>
>>>>    - *Hash Index Type*: Should hash-based indexes be a separate index
>>>>    type? They are only applicable to equality and IN predicates, and the hash
>>>>    algorithm must always be known. A dedicated HASH type could make these
>>>>    constraints explicit and distinguish hash-based indexes from ordered
>>>>    indexes.
>>>>    - *Transform Function/Layout Representation*: Should transform
>>>>    functions be represented as an enum or a string? I lean toward strings, but
>>>>    I do not have a strong preference. In many cases, an index can still be
>>>>    used based on standard column statistics, even if a reader does not
>>>>    recognize a particular transform.
>>>>    - *Future Index Types*: How should we handle index types that are
>>>>    expected in the long term but fall outside the scope of this proposal (for
>>>>    example, VECTOR, IVF, TERM)? Should they be mentioned as planned/reserved
>>>>    concepts, or omitted entirely? I lean toward at least mentioning them. The
>>>>    rationale for introducing an index type is easier to explain when
>>>>    considering future index families.
>>>>    - *Index History Handling:* Most database systems treat index
>>>>    changes as a drop-and-recreate operation rather than an in-place update. If
>>>>    we follow the same approach, there is limited value in maintaining index
>>>>    metadata history. During index maintenance, a new metadata.json can be
>>>>    created and atomically swapped in, with the catalog responsible for
>>>>    cleaning up obsolete metadata and index data files according to its
>>>>    retention policy.
>>>>    - *Min/Max Statistics Representation*: Should transform-specific or
>>>>    sort-order min/max values be represented using the existing file statistics
>>>>    structure, or exposed through dedicated fields? My preference is to keep
>>>>    them within the statistics structure defined by the file schema while
>>>>    exposing them directly through the Java API if needed.
>>>>    - *File Format Location*: Should file format be specified at the
>>>>    leaf-file level or only at the index level? My preference is to stay
>>>>    aligned with the table metadata model and avoid introducing differences
>>>>    unless there is a compelling requirement. This would simplify
>>>>    implementation and maximize code reuse.
>>>>
>>>> I have updated the spec PR based on the decisions above:
>>>> https://github.com/apache/iceberg/pull/16961
>>>>
>>>> Thanks,
>>>> Peter
>>>>
>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont: 2026. jún. 26.,
>>>> P, 19:42):
>>>>
>>>>> Hi all, Here is the summary of this Monday's index meeting:
>>>>>
>>>>> Two blocking decisions closed:
>>>>>
>>>>>    1. Index is *not* a table— it's its own object (reuses table
>>>>>    machinery under the hood). Reasoning: requires a sort order, no column
>>>>>    updates, no partition spec, no overlapping ranges between leaves, inherits
>>>>>    base-table permissions, and has its own CREATE/DROP/UPDATE INDEX DDL.
>>>>>    2. Index is a separate catalog entity, with no pointers in table
>>>>>    metadata— has its own REST endpoints; the catalog can optionally return
>>>>>    index metadata withloadTable to avoid extra round-trips. Keeps table and
>>>>>    index updates independent/async.
>>>>>
>>>>>
>>>>> Next steps: Start writing the spec and build out the copy-on-write
>>>>> path now.
>>>>>
>>>>> Here are the draft spec:
>>>>> secondary index spec <https://github.com/apache/iceberg/pull/16961>
>>>>> irc spec <https://github.com/apache/iceberg/pull/16963>
>>>>>
>>>>> Thanks,
>>>>> Huaxin
>>>>>
>>>>> On Sat, Jun 20, 2026 at 11:11 AM huaxin gao <hu...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I built a standalone PoC to validate that the basic index structure
>>>>>> works: that we can build a PK index, convert equality deletes to position
>>>>>> deletes through it, and have every converted delete land on the correct
>>>>>> live row. I ran it up to *100M keys*.
>>>>>>
>>>>>> *Headline: the structure works.* The index builds over up to 100M
>>>>>> keys, the eq-delete → position-delete conversion resolved correctly at
>>>>>> *every* size (100% of converted deletes mapped to the right live
>>>>>> row), and the resulting position deletes are *~8× cheaper to apply* at
>>>>>> query time than the equality deletes they replace.
>>>>>>
>>>>>> Beyond correctness, the run also shows how the index’s *maintenance* cost
>>>>>> scales, comparing copy-on-write (COW, rewrite touched leaves) vs an
>>>>>> append/merge (MOR) option, under a realistic mixed CDC checkpoint (1,000
>>>>>> insert + 500 update + 500 delete), local wall-clock:
>>>>>> keys EQ baseline INDEX (COW) % of 60s (COW) INDEX (MOR) % of 60s
>>>>>> (MOR) correct
>>>>>> 5M 6 ms 6.7s 11.2% 2.2s 3.7% PASS
>>>>>> 20M 8 ms 24.2s 40.4% 6.4s 10.6% PASS
>>>>>> 50M 7 ms 51.6s 86.1% 12.2s 20.4% PASS
>>>>>> *100M* 6 ms *75.0s* 125% (BEHIND) *16.9s* 28.2% (keeps up) PASS
>>>>>>
>>>>>> COW maintenance crosses the 60 s checkpoint around 100M (75 s/cycle,
>>>>>> 125%); MOR stays at ~28% and keeps pace; the equality-delete baseline is
>>>>>> ~6 ms and flat. So the structure works, but *COW alone can’t sustain
>>>>>> scattered CDC at hundreds of millions of keys on a single writer*.
>>>>>> It’s worth allowing a merge-on-read / update-file maintenance option
>>>>>> alongside COW (or sharding the index across parallel writers).
>>>>>>
>>>>>> *Full write-up, all tables, and the in-region reality-check:* link
>>>>>> <https://docs.google.com/document/d/1G3zxbW8X0eU3UrouslZfp42bBc9CvgJGnJyDONCB4PU/edit?tab=t.0>
>>>>>>
>>>>>> Feedback welcome, especially on the spec direction (whether to allow
>>>>>> a merge-on-read / update-file maintenance option alongside COW)  and on the
>>>>>> read-side modeling.
>>>>>>
>>>>>> Thanks,
>>>>>> Huaxin
>>>>>>
>>>>>> On Tue, Jun 9, 2026 at 5:45 PM huaxin gao <hu...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Sorry,  we've skipped posting a few of the dedicated index-sync
>>>>>>> summaries to the mailing list; you can find those in the Google doc
>>>>>>> <https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?pli=1&tab=t.8041k7j2n7y3>
>>>>>>> and the Slack channel. Here's yesterday's summary:
>>>>>>>
>>>>>>> *Decided*
>>>>>>>
>>>>>>>    -
>>>>>>>
>>>>>>>    Index vs. table (what we agreed):
>>>>>>>    - Reuse table implementation/library code and a near-identical
>>>>>>>       spec — the commit path will be custom regardless, so reuse isn't the
>>>>>>>       deciding factor.
>>>>>>>       - An index is not a table from a user/API view: loading or
>>>>>>>       writing an index as a table must fail(it would violate index invariants).
>>>>>>>       - The spec forbids most table behaviors: no overlapping
>>>>>>>       files, one mandatory transform sort order, no column updates, no partition
>>>>>>>       spec.
>>>>>>>       - Delete vectors: reuse Iceberg's existing DV — benchmarks
>>>>>>>    showed no new delete format is worth introducing.
>>>>>>>    - Incremental updates: start with copy-on-write only (no update
>>>>>>>    files). For object-store-sized leaves, a full leaf rewrite is about as
>>>>>>>    cheap as maintaining an overlay update file + DV, so we'll skip the MOR
>>>>>>>    machinery for now and add it later only if benchmarks prove we need it
>>>>>>>    (likely just the very-large-leaf case).
>>>>>>>    -
>>>>>>>
>>>>>>>    Validate the spec first: build a quick, hand-wired prototype
>>>>>>>    (Parquet files structured per the spec) and benchmark it on real scales
>>>>>>>    before formalizing.
>>>>>>>
>>>>>>> *Leaning, not final*
>>>>>>>
>>>>>>>    -
>>>>>>>
>>>>>>>    Indexes are likely separate catalog objects, linked from the
>>>>>>>    table by storing just an identifier (like materialized views) and not
>>>>>>>    visible in LIST TABLES.
>>>>>>>    -
>>>>>>>
>>>>>>>    We'll need a commit path for indexes, but simpler than tables
>>>>>>>    (no stage-create).
>>>>>>>
>>>>>>> *Still open*
>>>>>>>
>>>>>>>
>>>>>>>    -
>>>>>>>
>>>>>>>    Permissions model — separate vs. inherited (action: look at what
>>>>>>>    real DBs do for index permissions).
>>>>>>>    - REST/catalog RPC design — minimize round-trips; index metadata
>>>>>>>    ideally returned with LOAD TABLE. Catalog RPC cost may dominate
>>>>>>>    Parquet IO, so this needs real design.
>>>>>>>    -
>>>>>>>
>>>>>>>    Scale modeling — target rows-per-leaf vs. leaf size vs.
>>>>>>>    metadata-file count.
>>>>>>>    -
>>>>>>>
>>>>>>>    DDL-on-index semantics (reuse table schema-update actions or
>>>>>>>    separate)
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Huaxin
>>>>>>>
>>>>>>> On Wed, Apr 22, 2026 at 8:47 AM Péter Váry <
>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> TL;DR
>>>>>>>> We still need to validate with ADLS and S3, but based on the local
>>>>>>>> tests, the MPHF approach looks more promising if we can tolerate larger
>>>>>>>> files and longer index maintenance times.
>>>>>>>>
>>>>>>>> Details:
>>>>>>>> Here are the results from the local experiments on my Mac. I
>>>>>>>> removed unnecessary statistics from the Parquet files and tested different
>>>>>>>> row group sizes:
>>>>>>>>
>>>>>>>>    - For an index file with 1M records, a row group size of 5,000
>>>>>>>>    appears to be the sweet spot.
>>>>>>>>    - For 10M records, 10,000 rows per row group works best.
>>>>>>>>
>>>>>>>> If you have additional ideas for optimizing Parquet-based indexes,
>>>>>>>> I’d be very interested to hear them.
>>>>>>>> The test code is available on this branch:
>>>>>>>> https://github.com/pvary/iceberg/tree/leaf_bench
>>>>>>>>
>>>>>>>> Best results:
>>>>>>>> *1m records/file*
>>>>>>>>
>>>>>>>>    - Parquet - 5000 row/RowGroup
>>>>>>>>       - Read: 1191 µs - 1 file open, 3 seek, 123KB read per lookup
>>>>>>>>       - Write: 1.7 s, 15 MB
>>>>>>>>    - MPHF
>>>>>>>>       - Read: 202 µs - 1 file open, 1 seek,  282KB read per lookup
>>>>>>>>       - Write: 0.8 s, 34 MB
>>>>>>>>
>>>>>>>> *10m records/file*
>>>>>>>>
>>>>>>>>    - Parquet - 10000 row/RowGroup
>>>>>>>>       - Read: 4168 µs - 1 file open, 3 seek, 395KB read per lookup
>>>>>>>>       - Write: 19.5s s, 144 MB
>>>>>>>>    - MPHF
>>>>>>>>       - Read: 1086 µs - 1 file open, 1 seek,  2.8 MB (2812KB) read
>>>>>>>>       per lookup
>>>>>>>>       - Write: 6.5 s, 34 MB, 353 MB
>>>>>>>>
>>>>>>>> Below are the full results.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *Benchmark                                      (indexType)
>>>>>>>>  (keyType)  (numRows)  Mode    Cnt            Score          Error
>>>>>>>>  UnitsInvertedIndexBenchmark.lookup                 PARQUET_1000       LONG
>>>>>>>>    1000000    ss  10000         3285.284 ±        5.138
>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_1000       LONG
>>>>>>>>    1000000    ss  10000   2522168989.000
>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_1000       LONG
>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_1000       LONG
>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_1000       LONG
>>>>>>>> 10000000    ss  10000        35449.614 ±       34.673
>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_1000       LONG
>>>>>>>>   10000000    ss  10000  24302649201.000
>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_1000       LONG
>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_1000       LONG
>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_5000       LONG
>>>>>>>>  1000000    ss  10000         1191.959 ±        4.169
>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_5000       LONG
>>>>>>>>    1000000    ss  10000   1230877229.000
>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_5000       LONG
>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_5000       LONG
>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_5000       LONG
>>>>>>>> 10000000    ss  10000         7236.447 ±       10.374
>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_5000       LONG
>>>>>>>>   10000000    ss  10000   5650715973.000
>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_5000       LONG
>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_5000       LONG
>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_10000       LONG
>>>>>>>>  1000000    ss  10000         1349.946 ±        7.834
>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_10000       LONG
>>>>>>>>    1000000    ss  10000   1730219377.000
>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_10000       LONG
>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_10000       LONG
>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_10000       LONG
>>>>>>>> 10000000    ss  10000         4168.635 ±       11.051
>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_10000       LONG
>>>>>>>>   10000000    ss  10000   3946341532.000
>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_10000       LONG
>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_10000       LONG
>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_50000       LONG
>>>>>>>>  1000000    ss  10000         4736.466 ±       38.179
>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_50000       LONG
>>>>>>>>    1000000    ss  10000   7427413541.000
>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_50000       LONG
>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_50000       LONG
>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_50000       LONG
>>>>>>>> 10000000    ss  10000         4979.031 ±       34.708
>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_50000       LONG
>>>>>>>>   10000000    ss  10000   7694887636.000
>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_50000       LONG
>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_50000       LONG
>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>> #InvertedIndexBenchmark.lookup                         MPHF       LONG
>>>>>>>>  1000000    ss  10000          202.571 ±        2.336
>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead               MPHF       LONG
>>>>>>>>    1000000    ss  10000   2821570000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams             MPHF       LONG
>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:seeks                   MPHF       LONG
>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>> #InvertedIndexBenchmark.lookup                         MPHF       LONG
>>>>>>>> 10000000    ss  10000         1086.957 ±        4.524
>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead               MPHF       LONG
>>>>>>>>   10000000    ss  10000  28119460000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams             MPHF       LONG
>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:seeks                   MPHF       LONG
>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_1000       LONG
>>>>>>>>  1000000    ss      3      1720731.014 ±   876636.004
>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_1000       LONG
>>>>>>>>    1000000    ss      3     46453317.000
>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_1000       LONG
>>>>>>>> 10000000    ss      3     18547947.876 ± 12258125.307
>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_1000       LONG
>>>>>>>>   10000000    ss      3    452655675.000
>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_5000       LONG
>>>>>>>>  1000000    ss      3      1718345.583 ±  1103928.016
>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_5000       LONG
>>>>>>>>    1000000    ss      3     44845788.000
>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_5000       LONG
>>>>>>>> 10000000    ss      3     18604229.931 ±  2668361.915
>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_5000       LONG
>>>>>>>>   10000000    ss      3    435388818.000
>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_10000       LONG
>>>>>>>>  1000000    ss      3      1761555.389 ±   535857.675
>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_10000       LONG
>>>>>>>>    1000000    ss      3     44536635.000
>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_10000       LONG
>>>>>>>> 10000000    ss      3     19501588.264 ±  2130054.558
>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_10000       LONG
>>>>>>>>   10000000    ss      3    433189623.000
>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_50000       LONG
>>>>>>>>  1000000    ss      3      1936624.889 ±  6601363.985
>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_50000       LONG
>>>>>>>>    1000000    ss      3     44264655.000
>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_50000       LONG
>>>>>>>> 10000000    ss      3     20471742.278 ± 10705206.310
>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_50000       LONG
>>>>>>>>   10000000    ss      3    431311305.000
>>>>>>>> #InvertedIndexBenchmark.write                          MPHF       LONG
>>>>>>>>  1000000    ss      3       896573.958 ±  1408024.851
>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes           MPHF       LONG
>>>>>>>>    1000000    ss      3    102846369.000
>>>>>>>> #InvertedIndexBenchmark.write                          MPHF       LONG
>>>>>>>> 10000000    ss      3      6509348.875 ± 15519975.479
>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes           MPHF       LONG
>>>>>>>>   10000000    ss      3   1058435733.000                     #*
>>>>>>>>
>>>>>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont: 2026. ápr.
>>>>>>>> 21., K, 20:53):
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> In recent secondary index sync meetings, the discussion converged
>>>>>>>>> on the need to define what an index is from first principles before
>>>>>>>>> settling on physical layout.
>>>>>>>>>
>>>>>>>>> To address that, Peter and I have drafted a requirements document
>>>>>>>>> for a key lookup index (renamed from "primary key index" to avoid implying
>>>>>>>>> uniqueness enforcement), the goal is to nail down one well-scoped index
>>>>>>>>> type first.
>>>>>>>>>
>>>>>>>>> Doc: Key Lookup Index Requirements
>>>>>>>>> <https://docs.google.com/document/d/1e0zxK-jA0LBDq8YQlQgFipTHelDFiga8lCkgDTmYub8/edit?tab=t.0#heading=h.8shrgabvl19>
>>>>>>>>>
>>>>>>>>> It covers requirements, three design options (manifest + sorted
>>>>>>>>> Parquet, hash + sorted Parquet, hash + MPHF) and open questions. We will
>>>>>>>>> add preliminary benchmark results shortly.
>>>>>>>>>
>>>>>>>>> Feedback welcome — inline in the doc, on this thread, or at the
>>>>>>>>> next index sync.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Huaxin
>>>>>>>>>
>>>>>>>>> On Mon, Apr 13, 2026 at 7:22 AM Steven Wu <st...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Do we need the special index identifier that was originally
>>>>>>>>>> proposed? A generic CatalogObjectIdentifier (with namespace and name) would
>>>>>>>>>> be consistent with all object types in the catalog. I have a discussion
>>>>>>>>>> thread on the generic identifier topic: [DISCUSS] REST Spec:
>>>>>>>>>> generic CatalogObjectIdentifier.
>>>>>>>>>>
>>>>>>>>>> Should we add an indexes array field to table metadata? It only
>>>>>>>>>> contains a list of index object identifiers. It doesn't contain any index
>>>>>>>>>> metadata which should live in the index objects. Yufei was trying to bring
>>>>>>>>>> this up at the end of the first sync. But we didn't get enough time to
>>>>>>>>>> really discuss it. It will be great to discuss this as the first agenda
>>>>>>>>>> item today.
>>>>>>>>>>
>>>>>>>>>> On Mon, Apr 13, 2026 at 3:17 AM Péter Váry <
>>>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>
>>>>>>>>>>> We had several engaging discussions at the Iceberg Summit, and
>>>>>>>>>>> it was great to finally catch up with many of you in person. We truly
>>>>>>>>>>> missed those who couldn’t attend, hopefully we’ll all meet again at the
>>>>>>>>>>> next summit.
>>>>>>>>>>>
>>>>>>>>>>> To keep the conversation going, Huaxin and I have put together
>>>>>>>>>>> the agenda for our next meeting. As a reminder, we’ll meet on *April
>>>>>>>>>>> 13th, 9:00–10:00 AM *PDT (6:00–7:00 PM CEST).
>>>>>>>>>>>
>>>>>>>>>>> Proposed agenda:
>>>>>>>>>>>
>>>>>>>>>>>    - Continue first-principles index design discussion from Mar
>>>>>>>>>>>    30
>>>>>>>>>>>       - *Index Ownership and Write Responsibility*
>>>>>>>>>>>          - Should writers be allowed to update indexes, or
>>>>>>>>>>>          - Should all index writes be handled exclusively by
>>>>>>>>>>>          the Index Maintenance process?
>>>>>>>>>>>          - If writers can update indexes then we need to define
>>>>>>>>>>>          what guarantees are required (compaction, file splitting, layout
>>>>>>>>>>>          expectations)?
>>>>>>>>>>>          - If only Index Maintenance updates indexes then we
>>>>>>>>>>>          only need to define what observable properties should be exposed to
>>>>>>>>>>>          consumers? Like:
>>>>>>>>>>>             - Expected max files for a single key
>>>>>>>>>>>             - Current max files for a single key
>>>>>>>>>>>             - Deletes allowed/present
>>>>>>>>>>>             - Sorted by
>>>>>>>>>>>             - Partitioned by
>>>>>>>>>>>          - *Specification Scope: What Belongs in the Spec?*
>>>>>>>>>>>          - Related to the ownership question above
>>>>>>>>>>>          - Light spec: Just define that the index table should
>>>>>>>>>>>          be optimized for retrieval by key columns and the index columns should be
>>>>>>>>>>>          contained in the table. This could give us more flexibility if better
>>>>>>>>>>>          organization methods come up, or
>>>>>>>>>>>          - Detailed spec: We could define the max number of
>>>>>>>>>>>          files per index to read for a single key, or even the partitioning and the
>>>>>>>>>>>          exact sort order. This could allow more use-cases for a given index, like
>>>>>>>>>>>          joins or cardinality estimations.
>>>>>>>>>>>          - I would go for light spec for the main types (PK,
>>>>>>>>>>>          Containing) and only the Index Maintenance processes should update the
>>>>>>>>>>>          Indexes, as for many use-cases the details are not important, and writers
>>>>>>>>>>>          will very rarely update the Indexes themselves.
>>>>>>>>>>>       - *Logical Placement of Indexes*
>>>>>>>>>>>          - Index as a child object of an Iceberg Table, or
>>>>>>>>>>>          - Index as a first‑class entity under
>>>>>>>>>>>          /namespace/indexes/{index}
>>>>>>>>>>>          - Based on the discussions on the summit we are
>>>>>>>>>>>          leaning in this direction. This means the index id should be unique in the
>>>>>>>>>>>          namespace but helps the catalog implementations quite a bit
>>>>>>>>>>>       - *Physical Placement of Index Data*
>>>>>>>>>>>          - I don’t think we should specify this. We should have
>>>>>>>>>>>          a base location for the index, but can rely on the catalog implementations
>>>>>>>>>>>          to decide on their own, like they do with the tables, views, udfs.
>>>>>>>>>>>       - *Iceberg Reader Based indexes* (Containing indexes and
>>>>>>>>>>>       potentially PK indexes). These are the indexes which could be read by the
>>>>>>>>>>>       existing Iceberg readers. We might decide to store the PK index similarly
>>>>>>>>>>>       to an Iceberg Table and treat it as a reader based index.
>>>>>>>>>>>          - What are the table properties/features exposed to
>>>>>>>>>>>          the readers
>>>>>>>>>>>             - Maybe just some behavioral descriptors for the
>>>>>>>>>>>             optimizer to decide if the index could be used or should be skipped, like:
>>>>>>>>>>>                - Expected max files for a single key
>>>>>>>>>>>                - max files for a single key
>>>>>>>>>>>                - Deletes allowed/present
>>>>>>>>>>>                - Sorted by
>>>>>>>>>>>                - Partitioned by
>>>>>>>>>>>             - The Tasks when reading the index based on the
>>>>>>>>>>>             filters and projection
>>>>>>>>>>>          - What are the table properties/features exposed to
>>>>>>>>>>>          the Index Maintenance. I think this could be internal to the Index
>>>>>>>>>>>          Maintenance process and might not be exposed through the spec. The Index
>>>>>>>>>>>          Maintenance process could handle this as a standard Iceberg Table and could
>>>>>>>>>>>          be based on the Table Maintenance process, but there might be some totally
>>>>>>>>>>>          different processes.
>>>>>>>>>>>       - It should be possible to add properties to an index
>>>>>>>>>>>       defined by the Index Maintenance process which could be used and updated in
>>>>>>>>>>>       the next Index Maintenance run.
>>>>>>>>>>>    - *PK index storage format benchmark results*
>>>>>>>>>>>       - Flat Parquet (baseline)
>>>>>>>>>>>       - BTree with Parquet leaves
>>>>>>>>>>>       - Vortex
>>>>>>>>>>>    - *Open items / next steps*
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Peter
>>>>>>>>>>>
>>>>>>>>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont: 2026.
>>>>>>>>>>> márc. 23., H, 3:03):
>>>>>>>>>>>
>>>>>>>>>>>> Hi everyone, I wanted to share an update on the primary key
>>>>>>>>>>>> index work.
>>>>>>>>>>>> Since there are still open questions on whether bloom filter
>>>>>>>>>>>> indexes fit in the secondary index framework or should be treated as
>>>>>>>>>>>> extended stats, I've shifted focus to the primary key index since it's a
>>>>>>>>>>>> clearer fit for the framework.
>>>>>>>>>>>> I've put together a proposal for a primary key reverse-lookup
>>>>>>>>>>>> index that maps each key to its physical location (file_path,
>>>>>>>>>>>> row_position). It enables:
>>>>>>>>>>>>
>>>>>>>>>>>>    - Scan-time file pruning for point lookups
>>>>>>>>>>>>    - Converting key-based deletes into position deletes
>>>>>>>>>>>>    (eliminating equality deletes for Flink CDC)
>>>>>>>>>>>>    - Accelerating Spark MERGE INTO by replacing full-table
>>>>>>>>>>>>    joins with direct file lookups
>>>>>>>>>>>>
>>>>>>>>>>>> Proposal:
>>>>>>>>>>>> https://docs.google.com/document/d/1HuhCZ0n2FqDh8yqQb9oEj1CPM5yXpEsMPGZno2aSf8E/edit?tab=t.0#heading=h.tbevg4q0m9
>>>>>>>>>>>> Feedback welcome!
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Huaxin
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Mar 18, 2026 at 11:42 PM Péter Váry <
>>>>>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Key takeaways from the general index discussion at the May 16
>>>>>>>>>>>>> meeting.
>>>>>>>>>>>>> Thanks to everyone who participated! The recording is
>>>>>>>>>>>>> available here: https://www.youtube.com/watch?v=btmjhtRWUCE
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - Q: Do we need to tie index types to the algorithms used
>>>>>>>>>>>>>    to access them?
>>>>>>>>>>>>>    - A: From a specification perspective, the goal is to
>>>>>>>>>>>>>    define the storage-level data layout so it can be shared across engines.
>>>>>>>>>>>>>    Engines are free to interpret and use the data as they see fit, but the
>>>>>>>>>>>>>    on-disk data layout itself must be strictly defined and interoperable.
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - Q: Should we introduce an additional abstraction layer
>>>>>>>>>>>>>    (e.g., Vector Index) with sub-types such as IVF and DiskANN?
>>>>>>>>>>>>>    - A: This is possible if we decide it is beneficial. I
>>>>>>>>>>>>>    explored potential naming, but it is not yet clear how such a layer would
>>>>>>>>>>>>>    be used in practice.
>>>>>>>>>>>>>    *Question to Yingyi Bu*: could you provide examples where
>>>>>>>>>>>>>    this additional layer would be useful? Should this abstraction be defined
>>>>>>>>>>>>>    at the spec level, or is it better handled at the engine level?
>>>>>>>>>>>>>    My initial idea was that users would create a generic
>>>>>>>>>>>>>    Vector Index and let the engine choose the concrete implementation.
>>>>>>>>>>>>>    However, this would limit user control and users likely need to specify the
>>>>>>>>>>>>>    exact index representation, which implies they must be aware of the
>>>>>>>>>>>>>    available representations.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - Q: Do we want to allow extensibility for index types?
>>>>>>>>>>>>>    - A: Yes. The intent is to support a small set of
>>>>>>>>>>>>>    well-defined index types while allowing experimentation with new ones. If a
>>>>>>>>>>>>>    new index type proves broadly useful, a follow-up proposal can standardize
>>>>>>>>>>>>>    it and incorporate it into the spec.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - Q: Do we allow multiple versions of an index for the
>>>>>>>>>>>>>    same table snapshot?
>>>>>>>>>>>>>    - A: Yes. Older index versions must be retained for
>>>>>>>>>>>>>    readers that have already started using them, while new readers should
>>>>>>>>>>>>>    automatically use the latest available version
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - Q: Do we need to use materialized views for these
>>>>>>>>>>>>>    indexes?
>>>>>>>>>>>>>    - A: No. These indexes are primarily examples, and
>>>>>>>>>>>>>    different types may require different storage methods. However, the Primary
>>>>>>>>>>>>>    Key, Containing, and parts of the IVF indexes can be structured as Iceberg
>>>>>>>>>>>>>    tables. This allows engines to read them natively; in some cases, Iceberg
>>>>>>>>>>>>>    planners can automatically redirect queries to the index table without
>>>>>>>>>>>>>    engine modifications. Furthermore, index maintenance for these tables can
>>>>>>>>>>>>>    leverage existing materialized view maintenance workflows. Other index
>>>>>>>>>>>>>    types may instead rely on Puffin files or alternative storage approaches.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - Q: How should index metadata be accessed? Should we add
>>>>>>>>>>>>>    explicit pointers for the indexes in the table metadata?
>>>>>>>>>>>>>    - A: We did not have sufficient time to fully explore and
>>>>>>>>>>>>>    conclude this topic.
>>>>>>>>>>>>>    *Question for Yufei Gu*: Did I understand correctly that
>>>>>>>>>>>>>    your main concern stems from endpoint resolution from a REST Catalog
>>>>>>>>>>>>>    perspective? Specifically, if indexes are exposed under a URI such as
>>>>>>>>>>>>>    v1/{prefix}/namespaces/{namespace}/tables/{table}/indexes/{index}, would
>>>>>>>>>>>>>    this make it more difficult for the REST Catalog to resolve and route
>>>>>>>>>>>>>    requests to the appropriate endpoint?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Suhas Jayaram Subramanya via dev <de...@iceberg.apache.org> ezt
>>>>>>>>>>>>> írta (időpont: 2026. márc. 13., P, 23:32):
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here's a proposal for native Vector Index support in Iceberg
>>>>>>>>>>>>>> tables --
>>>>>>>>>>>>>> https://docs.google.com/document/d/1KL4qLOwdqnhOcqTc0EjO1O16NV3M3c-gZCEINDWw4lA/edit?usp=sharing
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We've been working on this proposal with Peter internally at
>>>>>>>>>>>>>> Microsoft and he suggested we post it here to bring this to the community's
>>>>>>>>>>>>>> attention, ahead of the next Secondary Index Sync.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Suhas
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 2026/02/19 04:34:34 huaxin gao wrote:
>>>>>>>>>>>>>> > Hi Everyone,
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Here are the recording and notes from the Iceberg Index
>>>>>>>>>>>>>> Support Sync on
>>>>>>>>>>>>>> > 2/11.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Recording: https://www.youtube.com/watch?v=3sFfQ0A50yk
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Notes:
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.8041k7j2n7y3
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > The meeting will move to biweekly, Mondays 9–10am PST,
>>>>>>>>>>>>>> starting March 2.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Since the sync, I updated the Bloom skipping index proposal
>>>>>>>>>>>>>> > <
>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.5r5kl6k3fqwu
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > to address the discussion questions, specifically:
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > - Performance justification: when this helps
>>>>>>>>>>>>>> (high-cardinality = / IN,
>>>>>>>>>>>>>> > many data files, high object-store latency) and how it
>>>>>>>>>>>>>> differs from Parquet
>>>>>>>>>>>>>> > row-group Bloom filters (which still require opening the
>>>>>>>>>>>>>> data file).
>>>>>>>>>>>>>> > - Cost / scalability: rough sizing (Bloom blob size per
>>>>>>>>>>>>>> file, Puffin
>>>>>>>>>>>>>> > file size), the planning cost trade-off (driver index reads
>>>>>>>>>>>>>> vs executor
>>>>>>>>>>>>>> > file opens), and mitigations via caching.
>>>>>>>>>>>>>> > - Lifecycle / maintenance: incremental production as new
>>>>>>>>>>>>>> data files
>>>>>>>>>>>>>> > arrive, behavior when the index is missing/behind, and
>>>>>>>>>>>>>> sharding/compaction
>>>>>>>>>>>>>> > plus cleanup to avoid accumulating too many small Puffin
>>>>>>>>>>>>>> files over time.
>>>>>>>>>>>>>> > - Writer expectations: inline (optional) vs asynchronous
>>>>>>>>>>>>>> (primary) index
>>>>>>>>>>>>>> > creation.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > I also implemented a Spark 4.1 POC
>>>>>>>>>>>>>> > <https://github.com/apache/iceberg/pull/15311> and a local
>>>>>>>>>>>>>> benchmark to
>>>>>>>>>>>>>> > quantify both the pruning impact (plannedFiles →
>>>>>>>>>>>>>> afterBloom) and the index
>>>>>>>>>>>>>> > read overhead (statsFiles, statsBytes, bloomPayloadBytes)
>>>>>>>>>>>>>> for point
>>>>>>>>>>>>>> > predicates on high-cardinality columns. Please take a look
>>>>>>>>>>>>>> and let me know
>>>>>>>>>>>>>> > if you have any questions or feedback.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Huaxin
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > On Tue, Feb 10, 2026 at 1:43 PM huaxin gao <hu...@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > > Reminder for tomorrow's sync on Iceberg Index Support.
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > > Wednesday: Feb. 11 9:00 – 10:00am
>>>>>>>>>>>>>> > > Time zone: America/Los_Angeles
>>>>>>>>>>>>>> > > Google Meet joining info
>>>>>>>>>>>>>> > > Video call link: meet.google.com/nsp-ctyr-khk
>>>>>>>>>>>>>> > > Design doc:
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.0#heading=h.hs6r9d26w1y2
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.qouk73o4jxx7
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > > Thanks,
>>>>>>>>>>>>>> > > Huaxin
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > > On Tue, Feb 3, 2026 at 10:52 PM Péter Váry <
>>>>>>>>>>>>>> pe...@gmail.com>
>>>>>>>>>>>>>> > > wrote:
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >> Thanks Huaxin and Steven for organizing this. Looking
>>>>>>>>>>>>>> forward to meet you
>>>>>>>>>>>>>> > >> all next week!
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >> On Wed, Feb 4, 2026, 02:48 Steven Wu <st...@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>> We set up the dev calendar event with a new google meet
>>>>>>>>>>>>>> link. Please
>>>>>>>>>>>>>> > >>> ignore the link from Huaxin's original email.
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>> The dev calendar has the correct info (including the
>>>>>>>>>>>>>> new meeting link)
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>> Iceberg Index Support Sync
>>>>>>>>>>>>>> > >>> Wednesday, February 11 · 9:00 – 10:00am
>>>>>>>>>>>>>> > >>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>> > >>> Google Meet joining info
>>>>>>>>>>>>>> > >>> Video call link: https://meet.google.com/nsp-ctyr-khk
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>> On Tue, Feb 3, 2026 at 5:08 PM huaxin gao <
>>>>>>>>>>>>>> hu...@gmail.com>
>>>>>>>>>>>>>> > >>> wrote:
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>>> Sorry, I meant PST (not EST) :)
>>>>>>>>>>>>>> > >>>> Looking forward to the discussion!
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>> On Tue, Feb 3, 2026 at 4:58 PM Shawn Chang <
>>>>>>>>>>>>>> ye...@gmail.com>
>>>>>>>>>>>>>> > >>>> wrote:
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>> Hi Huaxin,
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>> Thanks for starting the sync!
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>> The meeting seems to be 9-10AM PST on the dev events
>>>>>>>>>>>>>> calendar
>>>>>>>>>>>>>> > >>>>> <
>>>>>>>>>>>>>> https://calendar.google.com/calendar/u/0?cid=MzkwNWQ0OTJmMWI0NTBiYTA3MTJmMmFlNmFmYTc2ZWI3NTdmMTNkODUyMjBjYzAzYWE0NTI3ODg1YWRjNTYyOUBncm91cC5jYWxlbmRhci5nb29nbGUuY29t
>>>>>>>>>>>>>> >,
>>>>>>>>>>>>>> > >>>>> not EST. Maybe it's a typo?
>>>>>>>>>>>>>> > >>>>> Otherwise, looking forward to the discussion!
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>> Best,
>>>>>>>>>>>>>> > >>>>> Shawn
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>> On Tue, Feb 3, 2026 at 9:18 AM huaxin gao <
>>>>>>>>>>>>>> hu...@gmail.com>
>>>>>>>>>>>>>> > >>>>> wrote:
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>>> Hi all,
>>>>>>>>>>>>>> > >>>>>> I'd like to start a dedicated sync to discuss
>>>>>>>>>>>>>> Iceberg Index support.
>>>>>>>>>>>>>> > >>>>>> Here is the existing discussion thread:
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> https://lists.apache.org/thread/fzqk3jjf0xpj5m4cfqb3v4c65p0t04ty
>>>>>>>>>>>>>> .
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>> To ground the discussion, here are the two proposals:
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>> - Peter's proposal
>>>>>>>>>>>>>> > >>>>>> <
>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.0#heading=h.hs6r9d26w1y2>
>>>>>>>>>>>>>> (overall
>>>>>>>>>>>>>> > >>>>>> index support)
>>>>>>>>>>>>>> > >>>>>> - My proposal
>>>>>>>>>>>>>> > >>>>>> <
>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.qouk73o4jxx7
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > >>>>>> (bloom filter skipping index)
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>> Time slot: Every 3 weeks, Wednesdays at 9 AM to 10
>>>>>>>>>>>>>> AM EST, starting
>>>>>>>>>>>>>> > >>>>>> next Wednesday (2/11). After FileFormat sync
>>>>>>>>>>>>>> finishes, we plan to use that
>>>>>>>>>>>>>> > >>>>>> slot and switch to every other Monday, 9 AM to 10 AM
>>>>>>>>>>>>>> EST.
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>> Meet link: https://meet.google.com/fjn-tyze-mko
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>>>>> > >>>>>> Huaxin
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>

Re: Dedicated sync for Iceberg Index Support

[Péter Váry <pe...@gmail.com>]

I had a longer discussion with Ryan and Russell at ApacheCon, and we
explored two main changes to the current spec:

   1. Store Region file *boundaries as top-level fields* in the tracking
   file schema instead of column stats. We still need to decide whether to
   reserve new field IDs or reuse existing field IDs, like the partition field
   IDs. Currently this former one is documented in the PR.
   2. *Store the actual Extract schema* directly in the index metadata
   instead of deriving it from `identity`, `materialized`, and
   `non-materialized` fields. This makes the index self-contained, at the cost
   of maintaining historical schemas and schema evolution rules. This also
   makes Derived Column support mandatory for indexes. See:
   https://docs.google.com/document/d/1u93kyIrTc8bt9VSTB4O0GM9S7zuXekqixJR6Id2FZC4


Please take a look and let me know what you think.

Thanks,
Peter


Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026. okt. 6.,
K, 13:25):

> Hi everyone,
>
> Sorry for the delay in sharing the summary. I had some issues uploading
> the recording, but it is now available:
> https://www.youtube.com/watch?v=TTLQuCNx7EI
>
> Summary
>
>    - We revisited the index region design and agreed to use
>    non-overlapping regions to optimize point lookup performance, accepting
>    higher write amplification and deferring overlay-based approaches to future
>    versions.
>    - Discussed and agreed that we stick to total row ordering using sort
>    keys (renamed from cluster keys).
>    - Kept explicit index type metadata for compatibility, with readers
>    skipping unsupported index types.
>    - Agreed that dropping indexed columns invalidates the corresponding
>    index and discussed schema evolution and type promotion handling.
>
> Next steps
>
>    - Resolve the remaining questions around index types and schema
>    evolution.
>    - Update the specification and PR to adopt the sort key terminology.
>    - Continue review and discussion of the proposal.
>
>
> The spec PR is now ready for review:
> https://github.com/apache/iceberg/pull/16961
>
> Thanks,
>  Peter
>
>
> Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026. szept.
> 15., K, 16:56):
>
>> Hi everyone,
>>
>> Here is the recording of the Monday sync:
>> https://drive.google.com/drive/folders/1iodHASJBVwNtDYEINq7k2HjogPsYQocH
>>
>> A brief summary of the discussion:
>>
>>    - Bloom Filter Index
>>       - Puffin reduces file-open overhead, while Bloom filter
>>       effectiveness appears independent of file placement.
>>       - The observed scaling issues are likely due to rate limiting.
>>       - Manifest alignment would be required for parallel planning.
>>       - A scalar index is expected to deliver similar or better
>>       performance.
>>       - We will *prioritize scalar indexes* before pursuing additional
>>       index types.
>>       - The tests are still useful to understand the decisions in the
>>       design stage.
>>    - Scalar Index Design
>>       - *Region file schema: identity, materialized, and
>>       non-materialized fields*.
>>       - Range files will use stable index-specific IDs, while
>>       expressions reference table field IDs.
>>       - Scalar indexes require both sorting and clustering; a dedicated
>>       index definition may be cleaner than reusing table sort/cluster definitions.
>>
>> We can continue the remaining discussions offline or in the next sync.
>>
>> There are only a few open questions, so the spec PR is ready for review:
>> https://github.com/apache/iceberg/pull/16961.
>>
>> Thanks,
>> Peter
>>
>>
>> Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026. szept.
>> 14., H, 9:07):
>>
>>> One correction to the previous summary: I originally wrote that *Range*
>>> files would store the index data, but I was later corrected that the
>>> decision was actually to use *Region* files.
>>>
>>> The correct index file naming is:
>>>
>>>    - *Tracking file*: contains the list of region files (analogous to a
>>>    manifest file for tables).
>>>    - *Region file*: contains the index data rows (analogous to a data
>>>    file for tables).
>>>
>>> Sorry for the confusion,
>>> Peter
>>>
>>> Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026.
>>> szept. 3., Cs, 13:18):
>>>
>>>>
>>>> Hi everyone,
>>>>
>>>> Here is the recording of the Monday sync:
>>>>
>>>> https://apache-iceberg.slack.com/files/U06538UFDH6/F0BTT739Y7P/impromptu_google_meet_meeting_-_aug_31_2026.mp4
>>>>
>>>> Thanks, Shawn, for making it available.
>>>>
>>>> A brief summary of the discussion:
>>>>
>>>>    - We finalized the file naming:
>>>>       - Tracking file: contains the list of range files (similar to a
>>>>       manifest file for tables).
>>>>       - Range file: contains the index data rows (similar to a data
>>>>       file for tables).
>>>>    - We agreed not to prescribe how expression values are materialized
>>>>    (for example, Hilbert or hash transformations), leaving room for
>>>>    implementation flexibility.
>>>>    - Ryan mentioned that he is working on a v4 proposal for generating
>>>>    Field IDs for non-materialized columns. The current thinking is that index
>>>>    columns should follow the same approach.
>>>>    - We agreed to retain older versions of the index metadata JSON.
>>>>    - We agreed not to require a strict one-to-one mapping between
>>>>    index snapshots and table snapshots. An index may contain multiple
>>>>    snapshots corresponding to the same table snapshot.
>>>>
>>>> Based on the discussion, I have updated the spec PR. Following Ryan's
>>>> pattern for non-materialized column definitions, the proposal now
>>>> introduces:
>>>>
>>>>    - *materialized-fields* for range file columns
>>>>    - *non-materialized-fields* for tracking file statistics
>>>>    - *cluster-spec* to define which columns participate in range file
>>>>    clustering
>>>>
>>>> Overall, I think we reached good alignment on the remaining design
>>>> points, and I do not see any open questions at the moment. The PR is in
>>>> good shape for review, so I would greatly appreciate any feedback or
>>>> comments.
>>>>
>>>> Thanks everyone for the productive discussion and continued
>>>> collaboration.
>>>> Peter
>>>>
>>>> Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026. aug.
>>>> 18., K, 14:17):
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> Unfortunately, yesterday's sync was not recorded due to a technical
>>>>> issue. :(
>>>>>
>>>>> Here is a brief summary of the discussion and the decisions we reached:
>>>>>
>>>>>    1. We agreed to use a list of Iceberg expressions to define the
>>>>>    *index_keys*, which determine how the index is ordered.
>>>>>    2. We agreed not to restrict the functions that can be used in
>>>>>    those expressions. Engines that do not understand a particular expression
>>>>>    can simply choose not to use the index.
>>>>>    3. We agreed that index region files (leaf files) should be sorted
>>>>>    by the values generated by the index_keys.
>>>>>    4. We discussed using *included_column_ids* to define the contents
>>>>>    and schema of the index region files, and generally felt this could be a
>>>>>    good approach.
>>>>>    5. We started discussing whether the results of the index_key
>>>>>    expressions should also be stored in the index region files alongside the
>>>>>    included columns. In some cases, storing them could help readers optimize
>>>>>    lookups within a region file. In other cases, the values could be derived
>>>>>    when needed, making them redundant and potentially hurting performance by
>>>>>    increasing the size of the region files.
>>>>>
>>>>> I think we should continue the discussion around point 5, as the
>>>>> outcome may influence, or even reopen, the decisions around point 4.
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>> Flavio Junqueira <fl...@gmail.com> ezt írta (időpont:
>>>>> 2026. aug. 4., K, 18:57):
>>>>>
>>>>>> Thank you for the summary, the proposal and the overall discussion. I
>>>>>> have one main question about the target and I apologize in advance if I'm
>>>>>> missing some important detail as I might not have the full context.
>>>>>>
>>>>>> I noticed that partitioning, sorting and clustering (via Hilbert)
>>>>>> were discussed in the call in this context of index transform functions. My
>>>>>> understanding is that engines currently are already pruning partitions and
>>>>>> files using partition information and table metadata like per-column
>>>>>> min-max. If that's the case, then what's it that this proposal is trying to
>>>>>> achieve by having transform functions that capture partitioning, sorting
>>>>>> and clustering when engines are already able to prune without a secondary
>>>>>> index? I can more easily see the primary key index example, but I'm not
>>>>>> entirely sure about these data layout mechanisms.
>>>>>>
>>>>>> I also didn't quite get how engines are expected to consume such an
>>>>>> index. There was a discussion about where the mapping of the input to files
>>>>>> happens, whether that's in Iceberg or the engine, and I didn't quite get
>>>>>> the recommendation. Checking Peter's proposal, I couldn't see it clearly
>>>>>> either, and if I just missed it, I'd appreciate a pointer to where I should
>>>>>> look into.
>>>>>>
>>>>>> -Flavio
>>>>>>
>>>>>> On 4 Aug 2026, at 10:52, Péter Váry <pe...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> Here is the recording from yesterday’s sync:
>>>>>> https://www.youtube.com/watch?v=Wg9orP9JAmk
>>>>>>
>>>>>> To summarize:
>>>>>> - We agreed to retain the history of index snapshots, but not the
>>>>>> history of other index properties. If readers require information that may
>>>>>> later be modified by a user, that information should be copied into the
>>>>>> corresponding snapshot.
>>>>>> - We discussed the possibility of defining index ordering using a
>>>>>> list of transform functions.
>>>>>>
>>>>>> Dan and Yingyi suggested something along the following lines:
>>>>>>
>>>>>> *"index_keys" : [*
>>>>>> *  {*
>>>>>> *    "type": "apply",*
>>>>>> *    "func-name": {*
>>>>>> *      "catalog": "iceberg_functions",*
>>>>>> *      "name": "day"*
>>>>>> *    },*
>>>>>> *    "arguments": [*
>>>>>> *      { "type": "reference", "id": 3 }*
>>>>>> *    ]*
>>>>>> *  },*
>>>>>> *  {*
>>>>>> *    "type": "apply",*
>>>>>> *    "func-name": {*
>>>>>> *      "catalog": "iceberg_functions",*
>>>>>> *      "name": "truncate"*
>>>>>> *    },*
>>>>>> *    "arguments": [*
>>>>>> *      { "type": "reference", "id": 4 },*
>>>>>> *      2*
>>>>>> *    ]*
>>>>>> *  }*
>>>>>> *]*
>>>>>>
>>>>>>
>>>>>> In this example, rows would first be ordered by day(col3) and then by
>>>>>> truncate(col4, 2). To ensure a deterministic ordering, ties would be broken
>>>>>> using the original values of columns 3 and 4. If those are still equal,
>>>>>> ordering would fall back to the file name and finally the row position.
>>>>>>
>>>>>> I spent some time thinking about this after the sync, and here are my
>>>>>> thoughts.
>>>>>>
>>>>>> Pros:
>>>>>> - Provides a high degree of flexibility.
>>>>>> - Enables expression-based indexes.
>>>>>>
>>>>>> Cons:
>>>>>> - Do we need this flexibility at this stage?
>>>>>>    - Equality deletes, the primary use cases we are targeting, do not
>>>>>> require expressions.
>>>>>>    - Most indexes I have seen in production are not expression-based.
>>>>>> - Increased complexity for engines:
>>>>>>    - Engines must parse and understand transforms to determine
>>>>>> whether using an index can improve a query. For example, is ordering by
>>>>>> day(timestamp) sufficient for a key lookup?
>>>>>>    - Complexity increases further as new UDFs are introduced.
>>>>>>    - Expressions are not pushed into scans today, so index handling
>>>>>> cannot be fully hidden behind Iceberg scan planning.
>>>>>> - Risk of fragmentation:
>>>>>>    - The additional complexity may slow engine adoption.
>>>>>>    - Adopters may choose to support only exact transform matches
>>>>>> created by them, rather than learn to reason about the general transforms.
>>>>>>    - UDFs are prone to missing implementations and behavioral
>>>>>> differences across engines. In the worst case, such differences could lead
>>>>>> to incorrect query results when indexes are used.
>>>>>>
>>>>>> Given the points above, I still prefer defining HASH, HILBERT,
>>>>>> IDENTITY as a set of standardized values combined with an explicit list of
>>>>>> key columns. This keeps adoption simple, unlocks the primary use cases we
>>>>>> are targeting today, and allows the Iceberg library to provide the
>>>>>> transform implementations directly.
>>>>>>
>>>>>> Internally, these transforms could still be represented as Iceberg
>>>>>> expressions, giving us a migration path toward future index types that may
>>>>>> require full expression support.
>>>>>>
>>>>>> For example:
>>>>>>
>>>>>> *{*
>>>>>> *  "transform": "HILBERT",*
>>>>>> *  "key-column-ids": [3, 4]*
>>>>>> *}*
>>>>>>
>>>>>>
>>>>>> I'm interested in hearing other opinions.
>>>>>>
>>>>>> Thanks,
>>>>>> Peter
>>>>>>
>>>>>> Renjie Liu <li...@gmail.com> ezt írta (időpont: 2026. júl.
>>>>>> 28., K, 10:07):
>>>>>>
>>>>>>> Hi, Peter:
>>>>>>>
>>>>>>> Thanks for driving the proposal.
>>>>>>>
>>>>>>> >  *Hash Index Type*: Should hash-based indexes be a separate index
>>>>>>> type? They are only applicable to equality and IN predicates, and the hash
>>>>>>> algorithm must always be known. A dedicated HASH type could make these
>>>>>>> constraints explicit and distinguish hash-based indexes from ordered
>>>>>>> indexes.
>>>>>>>
>>>>>>> I'm leaning toward keeping them in scalar index type due to the
>>>>>>> similarity with others transformation functions like identity. As with the
>>>>>>> constraints, I think they are properties of transformation functions.
>>>>>>>
>>>>>>> > *Transform Function/Layout Representation*: Should transform
>>>>>>> functions be represented as an enum or a string? I lean toward strings, but
>>>>>>> I do not have a strong preference. In many cases, an index can still be
>>>>>>> used based on standard column statistics, even if a reader does not
>>>>>>> recognize a particular transform.
>>>>>>>
>>>>>>> This seems more like a problem in java implementation?
>>>>>>>
>>>>>>> Also I prefer the name transform function compared with layout
>>>>>>> representation. Layout representation makes me feel that it's not about the
>>>>>>> definition of an index function.
>>>>>>>
>>>>>>> > *Future Index Types*: How should we handle index types that are
>>>>>>> expected in the long term but fall outside the scope of this proposal (for
>>>>>>> example, VECTOR, IVF, TERM)? Should they be mentioned as planned/reserved
>>>>>>> concepts, or omitted entirely? I lean toward at least mentioning them. The
>>>>>>> rationale for introducing an index type is easier to explain when
>>>>>>> considering future index families.
>>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>> *> Index History Handling:* Most database systems treat index
>>>>>>> changes as a drop-and-recreate operation rather than an in-place update. If
>>>>>>> we follow the same approach, there is limited value in maintaining index
>>>>>>> metadata history. During index maintenance, a new metadata.json can be
>>>>>>> created and atomically swapped in, with the catalog responsible for
>>>>>>> cleaning up obsolete metadata and index data files according to its
>>>>>>> retention policy.
>>>>>>>
>>>>>>> +1, I think we only need to keep index data for different snapshots,
>>>>>>> but if an index definition changed, we should drop and recreate a new index.
>>>>>>>
>>>>>>> > *File Format Location*: Should file format be specified at the
>>>>>>> leaf-file level or only at the index level? My preference is to stay
>>>>>>> aligned with the table metadata model and avoid introducing differences
>>>>>>> unless there is a compelling requirement. This would simplify
>>>>>>> implementation and maximize code reuse.
>>>>>>>
>>>>>>> +1.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jul 24, 2026 at 11:57 PM Péter Váry <
>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi everyone,
>>>>>>>>
>>>>>>>> Here are the key takeaways from Monday’s meeting (July 20):
>>>>>>>>
>>>>>>>>    - *Index Type*: An index type enum will be introduced in v1,
>>>>>>>>    initially with a single value (SCALAR). This establishes a future-proof API
>>>>>>>>    boundary while leaving room for additional index families.
>>>>>>>>    - *Strict Total Ordering*: The specification should require
>>>>>>>>    transform functions to produce values that preserve a well-defined total
>>>>>>>>    ordering across the key space. The transform function does not need to
>>>>>>>>    return a long. Instead, leaf files can store the first and last values
>>>>>>>>    according to the ordering, enabling generic filtering capabilities for
>>>>>>>>    readers.
>>>>>>>>
>>>>>>>> Open Questions:
>>>>>>>>
>>>>>>>>    - *Hash Index Type*: Should hash-based indexes be a separate
>>>>>>>>    index type? They are only applicable to equality and IN predicates, and the
>>>>>>>>    hash algorithm must always be known. A dedicated HASH type could make these
>>>>>>>>    constraints explicit and distinguish hash-based indexes from ordered
>>>>>>>>    indexes.
>>>>>>>>    - *Transform Function/Layout Representation*: Should transform
>>>>>>>>    functions be represented as an enum or a string? I lean toward strings, but
>>>>>>>>    I do not have a strong preference. In many cases, an index can still be
>>>>>>>>    used based on standard column statistics, even if a reader does not
>>>>>>>>    recognize a particular transform.
>>>>>>>>    - *Future Index Types*: How should we handle index types that
>>>>>>>>    are expected in the long term but fall outside the scope of this proposal
>>>>>>>>    (for example, VECTOR, IVF, TERM)? Should they be mentioned as
>>>>>>>>    planned/reserved concepts, or omitted entirely? I lean toward at least
>>>>>>>>    mentioning them. The rationale for introducing an index type is easier to
>>>>>>>>    explain when considering future index families.
>>>>>>>>    - *Index History Handling:* Most database systems treat index
>>>>>>>>    changes as a drop-and-recreate operation rather than an in-place update. If
>>>>>>>>    we follow the same approach, there is limited value in maintaining index
>>>>>>>>    metadata history. During index maintenance, a new metadata.json can be
>>>>>>>>    created and atomically swapped in, with the catalog responsible for
>>>>>>>>    cleaning up obsolete metadata and index data files according to its
>>>>>>>>    retention policy.
>>>>>>>>    - *Min/Max Statistics Representation*: Should
>>>>>>>>    transform-specific or sort-order min/max values be represented using the
>>>>>>>>    existing file statistics structure, or exposed through dedicated fields? My
>>>>>>>>    preference is to keep them within the statistics structure defined by the
>>>>>>>>    file schema while exposing them directly through the Java API if needed.
>>>>>>>>    - *File Format Location*: Should file format be specified at
>>>>>>>>    the leaf-file level or only at the index level? My preference is to stay
>>>>>>>>    aligned with the table metadata model and avoid introducing differences
>>>>>>>>    unless there is a compelling requirement. This would simplify
>>>>>>>>    implementation and maximize code reuse.
>>>>>>>>
>>>>>>>> I have updated the spec PR based on the decisions above:
>>>>>>>> https://github.com/apache/iceberg/pull/16961
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Peter
>>>>>>>>
>>>>>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont: 2026. jún.
>>>>>>>> 26., P, 19:42):
>>>>>>>>
>>>>>>>>> Hi all, Here is the summary of this Monday's index meeting:
>>>>>>>>>
>>>>>>>>> Two blocking decisions closed:
>>>>>>>>>
>>>>>>>>>    1. Index is *not* a table— it's its own object (reuses table
>>>>>>>>>    machinery under the hood). Reasoning: requires a sort order, no column
>>>>>>>>>    updates, no partition spec, no overlapping ranges between leaves, inherits
>>>>>>>>>    base-table permissions, and has its own CREATE/DROP/UPDATE INDEX DDL.
>>>>>>>>>    2. Index is a separate catalog entity, with no pointers in
>>>>>>>>>    table metadata— has its own REST endpoints; the catalog can optionally
>>>>>>>>>    return index metadata withloadTable to avoid extra round-trips. Keeps table
>>>>>>>>>    and index updates independent/async.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Next steps: Start writing the spec and build out the copy-on-write
>>>>>>>>> path now.
>>>>>>>>>
>>>>>>>>> Here are the draft spec:
>>>>>>>>> secondary index spec
>>>>>>>>> <https://github.com/apache/iceberg/pull/16961>
>>>>>>>>> irc spec <https://github.com/apache/iceberg/pull/16963>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Huaxin
>>>>>>>>>
>>>>>>>>> On Sat, Jun 20, 2026 at 11:11 AM huaxin gao <
>>>>>>>>> huaxin.gao11@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> I built a standalone PoC to validate that the basic index
>>>>>>>>>> structure works: that we can build a PK index, convert equality deletes to
>>>>>>>>>> position deletes through it, and have every converted delete land on the
>>>>>>>>>> correct live row. I ran it up to *100M keys*.
>>>>>>>>>>
>>>>>>>>>> *Headline: the structure works.* The index builds over up to
>>>>>>>>>> 100M keys, the eq-delete → position-delete conversion resolved correctly at
>>>>>>>>>> *every* size (100% of converted deletes mapped to the right live
>>>>>>>>>> row), and the resulting position deletes are *~8× cheaper to
>>>>>>>>>> apply* at query time than the equality deletes they replace.
>>>>>>>>>>
>>>>>>>>>> Beyond correctness, the run also shows how the index’s
>>>>>>>>>> *maintenance* cost scales, comparing copy-on-write (COW, rewrite
>>>>>>>>>> touched leaves) vs an append/merge (MOR) option, under a realistic mixed
>>>>>>>>>> CDC checkpoint (1,000 insert + 500 update + 500 delete), local wall-clock:
>>>>>>>>>> keys EQ baseline INDEX (COW) % of 60s (COW) INDEX (MOR) % of 60s
>>>>>>>>>> (MOR) correct
>>>>>>>>>> 5M 6 ms 6.7s 11.2% 2.2s 3.7% PASS
>>>>>>>>>> 20M 8 ms 24.2s 40.4% 6.4s 10.6% PASS
>>>>>>>>>> 50M 7 ms 51.6s 86.1% 12.2s 20.4% PASS
>>>>>>>>>> *100M* 6 ms *75.0s* 125% (BEHIND) *16.9s* 28.2% (keeps up) PASS
>>>>>>>>>>
>>>>>>>>>> COW maintenance crosses the 60 s checkpoint around 100M
>>>>>>>>>> (75 s/cycle, 125%); MOR stays at ~28% and keeps pace; the equality-delete
>>>>>>>>>> baseline is ~6 ms and flat. So the structure works, but *COW
>>>>>>>>>> alone can’t sustain scattered CDC at hundreds of millions of keys on a
>>>>>>>>>> single writer*. It’s worth allowing a merge-on-read /
>>>>>>>>>> update-file maintenance option alongside COW (or sharding the index across
>>>>>>>>>> parallel writers).
>>>>>>>>>>
>>>>>>>>>> *Full write-up, all tables, and the in-region reality-check:*
>>>>>>>>>> link
>>>>>>>>>> <https://docs.google.com/document/d/1G3zxbW8X0eU3UrouslZfp42bBc9CvgJGnJyDONCB4PU/edit?tab=t.0>
>>>>>>>>>>
>>>>>>>>>> Feedback welcome, especially on the spec direction (whether to
>>>>>>>>>> allow a merge-on-read / update-file maintenance option alongside COW)  and
>>>>>>>>>> on the read-side modeling.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Huaxin
>>>>>>>>>>
>>>>>>>>>> On Tue, Jun 9, 2026 at 5:45 PM huaxin gao <hu...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Sorry,  we've skipped posting a few of the dedicated index-sync
>>>>>>>>>>> summaries to the mailing list; you can find those in the Google
>>>>>>>>>>> doc
>>>>>>>>>>> <https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?pli=1&tab=t.8041k7j2n7y3>
>>>>>>>>>>> and the Slack channel. Here's yesterday's summary:
>>>>>>>>>>>
>>>>>>>>>>> *Decided*
>>>>>>>>>>>
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    Index vs. table (what we agreed):
>>>>>>>>>>>    - Reuse table implementation/library code and a
>>>>>>>>>>>       near-identical spec — the commit path will be custom regardless, so reuse
>>>>>>>>>>>       isn't the deciding factor.
>>>>>>>>>>>       - An index is not a table from a user/API view: loading
>>>>>>>>>>>       or writing an index as a table must fail(it would violate index invariants).
>>>>>>>>>>>       - The spec forbids most table behaviors: no overlapping
>>>>>>>>>>>       files, one mandatory transform sort order, no column updates, no partition
>>>>>>>>>>>       spec.
>>>>>>>>>>>       - Delete vectors: reuse Iceberg's existing DV —
>>>>>>>>>>>    benchmarks showed no new delete format is worth introducing.
>>>>>>>>>>>    - Incremental updates: start with copy-on-write only (no
>>>>>>>>>>>    update files). For object-store-sized leaves, a full leaf rewrite is about
>>>>>>>>>>>    as cheap as maintaining an overlay update file + DV, so we'll skip the MOR
>>>>>>>>>>>    machinery for now and add it later only if benchmarks prove we need it
>>>>>>>>>>>    (likely just the very-large-leaf case).
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    Validate the spec first: build a quick, hand-wired prototype
>>>>>>>>>>>    (Parquet files structured per the spec) and benchmark it on real scales
>>>>>>>>>>>    before formalizing.
>>>>>>>>>>>
>>>>>>>>>>> *Leaning, not final*
>>>>>>>>>>>
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    Indexes are likely separate catalog objects, linked from the
>>>>>>>>>>>    table by storing just an identifier (like materialized views) and not
>>>>>>>>>>>    visible in LIST TABLES.
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    We'll need a commit path for indexes, but simpler than
>>>>>>>>>>>    tables (no stage-create).
>>>>>>>>>>>
>>>>>>>>>>> *Still open*
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    Permissions model — separate vs. inherited (action: look at
>>>>>>>>>>>    what real DBs do for index permissions).
>>>>>>>>>>>    - REST/catalog RPC design — minimize round-trips; index
>>>>>>>>>>>    metadata ideally returned with LOAD TABLE. Catalog RPC cost
>>>>>>>>>>>    may dominate Parquet IO, so this needs real design.
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    Scale modeling — target rows-per-leaf vs. leaf size vs.
>>>>>>>>>>>    metadata-file count.
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    DDL-on-index semantics (reuse table schema-update actions or
>>>>>>>>>>>    separate)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Huaxin
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Apr 22, 2026 at 8:47 AM Péter Váry <
>>>>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>
>>>>>>>>>>>> TL;DR
>>>>>>>>>>>> We still need to validate with ADLS and S3, but based on the
>>>>>>>>>>>> local tests, the MPHF approach looks more promising if we can tolerate
>>>>>>>>>>>> larger files and longer index maintenance times.
>>>>>>>>>>>>
>>>>>>>>>>>> Details:
>>>>>>>>>>>> Here are the results from the local experiments on my Mac. I
>>>>>>>>>>>> removed unnecessary statistics from the Parquet files and tested different
>>>>>>>>>>>> row group sizes:
>>>>>>>>>>>>
>>>>>>>>>>>>    - For an index file with 1M records, a row group size of
>>>>>>>>>>>>    5,000 appears to be the sweet spot.
>>>>>>>>>>>>    - For 10M records, 10,000 rows per row group works best.
>>>>>>>>>>>>
>>>>>>>>>>>> If you have additional ideas for optimizing Parquet-based
>>>>>>>>>>>> indexes, I’d be very interested to hear them.
>>>>>>>>>>>> The test code is available on this branch:
>>>>>>>>>>>> https://github.com/pvary/iceberg/tree/leaf_bench
>>>>>>>>>>>>
>>>>>>>>>>>> Best results:
>>>>>>>>>>>> *1m records/file*
>>>>>>>>>>>>
>>>>>>>>>>>>    - Parquet - 5000 row/RowGroup
>>>>>>>>>>>>       - Read: 1191 µs - 1 file open, 3 seek, 123KB read per
>>>>>>>>>>>>       lookup
>>>>>>>>>>>>       - Write: 1.7 s, 15 MB
>>>>>>>>>>>>    - MPHF
>>>>>>>>>>>>       - Read: 202 µs - 1 file open, 1 seek,  282KB read per
>>>>>>>>>>>>       lookup
>>>>>>>>>>>>       - Write: 0.8 s, 34 MB
>>>>>>>>>>>>
>>>>>>>>>>>> *10m records/file*
>>>>>>>>>>>>
>>>>>>>>>>>>    - Parquet - 10000 row/RowGroup
>>>>>>>>>>>>       - Read: 4168 µs - 1 file open, 3 seek, 395KB read per
>>>>>>>>>>>>       lookup
>>>>>>>>>>>>       - Write: 19.5s s, 144 MB
>>>>>>>>>>>>    - MPHF
>>>>>>>>>>>>       - Read: 1086 µs - 1 file open, 1 seek,  2.8 MB (2812KB)
>>>>>>>>>>>>       read per lookup
>>>>>>>>>>>>       - Write: 6.5 s, 34 MB, 353 MB
>>>>>>>>>>>>
>>>>>>>>>>>> Below are the full results.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> *Benchmark                                      (indexType)
>>>>>>>>>>>>  (keyType)  (numRows)  Mode    Cnt            Score          Error
>>>>>>>>>>>>  UnitsInvertedIndexBenchmark.lookup                 PARQUET_1000       LONG
>>>>>>>>>>>>    1000000    ss  10000         3285.284 ±        5.138
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_1000       LONG
>>>>>>>>>>>>    1000000    ss  10000   2522168989.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_1000       LONG
>>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_1000       LONG
>>>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_1000       LONG
>>>>>>>>>>>> 10000000    ss  10000        35449.614 ±       34.673
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_1000       LONG
>>>>>>>>>>>>   10000000    ss  10000  24302649201.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_1000       LONG
>>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_1000       LONG
>>>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_5000       LONG
>>>>>>>>>>>>  1000000    ss  10000         1191.959 ±        4.169
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_5000       LONG
>>>>>>>>>>>>    1000000    ss  10000   1230877229.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_5000       LONG
>>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_5000       LONG
>>>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_5000       LONG
>>>>>>>>>>>> 10000000    ss  10000         7236.447 ±       10.374
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_5000       LONG
>>>>>>>>>>>>   10000000    ss  10000   5650715973.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_5000       LONG
>>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_5000       LONG
>>>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_10000       LONG
>>>>>>>>>>>>  1000000    ss  10000         1349.946 ±        7.834
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_10000       LONG
>>>>>>>>>>>>    1000000    ss  10000   1730219377.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_10000       LONG
>>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_10000       LONG
>>>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_10000       LONG
>>>>>>>>>>>> 10000000    ss  10000         4168.635 ±       11.051
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_10000       LONG
>>>>>>>>>>>>   10000000    ss  10000   3946341532.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_10000       LONG
>>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_10000       LONG
>>>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_50000       LONG
>>>>>>>>>>>>  1000000    ss  10000         4736.466 ±       38.179
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_50000       LONG
>>>>>>>>>>>>    1000000    ss  10000   7427413541.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_50000       LONG
>>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_50000       LONG
>>>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_50000       LONG
>>>>>>>>>>>> 10000000    ss  10000         4979.031 ±       34.708
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_50000       LONG
>>>>>>>>>>>>   10000000    ss  10000   7694887636.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_50000       LONG
>>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_50000       LONG
>>>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                         MPHF       LONG
>>>>>>>>>>>>  1000000    ss  10000          202.571 ±        2.336
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead               MPHF       LONG
>>>>>>>>>>>>    1000000    ss  10000   2821570000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams             MPHF       LONG
>>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks                   MPHF       LONG
>>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                         MPHF       LONG
>>>>>>>>>>>> 10000000    ss  10000         1086.957 ±        4.524
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead               MPHF       LONG
>>>>>>>>>>>>   10000000    ss  10000  28119460000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams             MPHF       LONG
>>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks                   MPHF       LONG
>>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_1000       LONG
>>>>>>>>>>>>  1000000    ss      3      1720731.014 ±   876636.004
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_1000       LONG
>>>>>>>>>>>>    1000000    ss      3     46453317.000
>>>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_1000       LONG
>>>>>>>>>>>> 10000000    ss      3     18547947.876 ± 12258125.307
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_1000       LONG
>>>>>>>>>>>>   10000000    ss      3    452655675.000
>>>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_5000       LONG
>>>>>>>>>>>>  1000000    ss      3      1718345.583 ±  1103928.016
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_5000       LONG
>>>>>>>>>>>>    1000000    ss      3     44845788.000
>>>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_5000       LONG
>>>>>>>>>>>> 10000000    ss      3     18604229.931 ±  2668361.915
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_5000       LONG
>>>>>>>>>>>>   10000000    ss      3    435388818.000
>>>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_10000       LONG
>>>>>>>>>>>>  1000000    ss      3      1761555.389 ±   535857.675
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_10000       LONG
>>>>>>>>>>>>    1000000    ss      3     44536635.000
>>>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_10000       LONG
>>>>>>>>>>>> 10000000    ss      3     19501588.264 ±  2130054.558
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_10000       LONG
>>>>>>>>>>>>   10000000    ss      3    433189623.000
>>>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_50000       LONG
>>>>>>>>>>>>  1000000    ss      3      1936624.889 ±  6601363.985
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_50000       LONG
>>>>>>>>>>>>    1000000    ss      3     44264655.000
>>>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_50000       LONG
>>>>>>>>>>>> 10000000    ss      3     20471742.278 ± 10705206.310
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_50000       LONG
>>>>>>>>>>>>   10000000    ss      3    431311305.000
>>>>>>>>>>>> #InvertedIndexBenchmark.write                          MPHF       LONG
>>>>>>>>>>>>  1000000    ss      3       896573.958 ±  1408024.851
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes           MPHF       LONG
>>>>>>>>>>>>    1000000    ss      3    102846369.000
>>>>>>>>>>>> #InvertedIndexBenchmark.write                          MPHF       LONG
>>>>>>>>>>>> 10000000    ss      3      6509348.875 ± 15519975.479
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes           MPHF       LONG
>>>>>>>>>>>>   10000000    ss      3   1058435733.000                     #*
>>>>>>>>>>>>
>>>>>>>>>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont: 2026.
>>>>>>>>>>>> ápr. 21., K, 20:53):
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>
>>>>>>>>>>>>> In recent secondary index sync meetings, the discussion
>>>>>>>>>>>>> converged on the need to define what an index is from first principles
>>>>>>>>>>>>> before settling on physical layout.
>>>>>>>>>>>>>
>>>>>>>>>>>>> To address that, Peter and I have drafted a requirements
>>>>>>>>>>>>> document for a key lookup index (renamed from "primary key index" to avoid
>>>>>>>>>>>>> implying uniqueness enforcement), the goal is to nail down one well-scoped
>>>>>>>>>>>>> index type first.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Doc: Key Lookup Index Requirements
>>>>>>>>>>>>> <https://docs.google.com/document/d/1e0zxK-jA0LBDq8YQlQgFipTHelDFiga8lCkgDTmYub8/edit?tab=t.0#heading=h.8shrgabvl19>
>>>>>>>>>>>>>
>>>>>>>>>>>>> It covers requirements, three design options (manifest +
>>>>>>>>>>>>> sorted Parquet, hash + sorted Parquet, hash + MPHF) and open questions. We
>>>>>>>>>>>>> will add preliminary benchmark results shortly.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Feedback welcome — inline in the doc, on this thread, or at
>>>>>>>>>>>>> the next index sync.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Huaxin
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Apr 13, 2026 at 7:22 AM Steven Wu <
>>>>>>>>>>>>> stevenz3wu@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Do we need the special index identifier that was originally
>>>>>>>>>>>>>> proposed? A generic CatalogObjectIdentifier (with namespace and name) would
>>>>>>>>>>>>>> be consistent with all object types in the catalog. I have a discussion
>>>>>>>>>>>>>> thread on the generic identifier topic: [DISCUSS] REST Spec:
>>>>>>>>>>>>>> generic CatalogObjectIdentifier.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Should we add an indexes array field to table metadata? It
>>>>>>>>>>>>>> only contains a list of index object identifiers. It doesn't contain any
>>>>>>>>>>>>>> index metadata which should live in the index objects. Yufei was trying to
>>>>>>>>>>>>>> bring this up at the end of the first sync. But we didn't get enough time
>>>>>>>>>>>>>> to really discuss it. It will be great to discuss this as the first agenda
>>>>>>>>>>>>>> item today.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Apr 13, 2026 at 3:17 AM Péter Váry <
>>>>>>>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We had several engaging discussions at the Iceberg Summit,
>>>>>>>>>>>>>>> and it was great to finally catch up with many of you in person. We truly
>>>>>>>>>>>>>>> missed those who couldn’t attend, hopefully we’ll all meet again at the
>>>>>>>>>>>>>>> next summit.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> To keep the conversation going, Huaxin and I have put
>>>>>>>>>>>>>>> together the agenda for our next meeting. As a reminder, we’ll meet on *April
>>>>>>>>>>>>>>> 13th, 9:00–10:00 AM *PDT (6:00–7:00 PM CEST).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Proposed agenda:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    - Continue first-principles index design discussion from
>>>>>>>>>>>>>>>    Mar 30
>>>>>>>>>>>>>>>       - *Index Ownership and Write Responsibility*
>>>>>>>>>>>>>>>          - Should writers be allowed to update indexes, or
>>>>>>>>>>>>>>>          - Should all index writes be handled exclusively
>>>>>>>>>>>>>>>          by the Index Maintenance process?
>>>>>>>>>>>>>>>          - If writers can update indexes then we need to
>>>>>>>>>>>>>>>          define what guarantees are required (compaction, file splitting, layout
>>>>>>>>>>>>>>>          expectations)?
>>>>>>>>>>>>>>>          - If only Index Maintenance updates indexes then
>>>>>>>>>>>>>>>          we only need to define what observable properties should be exposed to
>>>>>>>>>>>>>>>          consumers? Like:
>>>>>>>>>>>>>>>             - Expected max files for a single key
>>>>>>>>>>>>>>>             - Current max files for a single key
>>>>>>>>>>>>>>>             - Deletes allowed/present
>>>>>>>>>>>>>>>             - Sorted by
>>>>>>>>>>>>>>>             - Partitioned by
>>>>>>>>>>>>>>>          - *Specification Scope: What Belongs in the Spec?*
>>>>>>>>>>>>>>>          - Related to the ownership question above
>>>>>>>>>>>>>>>          - Light spec: Just define that the index table
>>>>>>>>>>>>>>>          should be optimized for retrieval by key columns and the index columns
>>>>>>>>>>>>>>>          should be contained in the table. This could give us more flexibility if
>>>>>>>>>>>>>>>          better organization methods come up, or
>>>>>>>>>>>>>>>          - Detailed spec: We could define the max number of
>>>>>>>>>>>>>>>          files per index to read for a single key, or even the partitioning and the
>>>>>>>>>>>>>>>          exact sort order. This could allow more use-cases for a given index, like
>>>>>>>>>>>>>>>          joins or cardinality estimations.
>>>>>>>>>>>>>>>          - I would go for light spec for the main types
>>>>>>>>>>>>>>>          (PK, Containing) and only the Index Maintenance processes should update the
>>>>>>>>>>>>>>>          Indexes, as for many use-cases the details are not important, and writers
>>>>>>>>>>>>>>>          will very rarely update the Indexes themselves.
>>>>>>>>>>>>>>>       - *Logical Placement of Indexes*
>>>>>>>>>>>>>>>          - Index as a child object of an Iceberg Table, or
>>>>>>>>>>>>>>>          - Index as a first‑class entity under
>>>>>>>>>>>>>>>          /namespace/indexes/{index}
>>>>>>>>>>>>>>>          - Based on the discussions on the summit we are
>>>>>>>>>>>>>>>          leaning in this direction. This means the index id should be unique in the
>>>>>>>>>>>>>>>          namespace but helps the catalog implementations quite a bit
>>>>>>>>>>>>>>>       - *Physical Placement of Index Data*
>>>>>>>>>>>>>>>          - I don’t think we should specify this. We should
>>>>>>>>>>>>>>>          have a base location for the index, but can rely on the catalog
>>>>>>>>>>>>>>>          implementations to decide on their own, like they do with the tables,
>>>>>>>>>>>>>>>          views, udfs.
>>>>>>>>>>>>>>>       - *Iceberg Reader Based indexes* (Containing indexes
>>>>>>>>>>>>>>>       and potentially PK indexes). These are the indexes which could be read by
>>>>>>>>>>>>>>>       the existing Iceberg readers. We might decide to store the PK index
>>>>>>>>>>>>>>>       similarly to an Iceberg Table and treat it as a reader based index.
>>>>>>>>>>>>>>>          - What are the table properties/features exposed
>>>>>>>>>>>>>>>          to the readers
>>>>>>>>>>>>>>>             - Maybe just some behavioral descriptors for
>>>>>>>>>>>>>>>             the optimizer to decide if the index could be used or should be skipped,
>>>>>>>>>>>>>>>             like:
>>>>>>>>>>>>>>>                - Expected max files for a single key
>>>>>>>>>>>>>>>                - max files for a single key
>>>>>>>>>>>>>>>                - Deletes allowed/present
>>>>>>>>>>>>>>>                - Sorted by
>>>>>>>>>>>>>>>                - Partitioned by
>>>>>>>>>>>>>>>             - The Tasks when reading the index based on the
>>>>>>>>>>>>>>>             filters and projection
>>>>>>>>>>>>>>>          - What are the table properties/features exposed
>>>>>>>>>>>>>>>          to the Index Maintenance. I think this could be internal to the Index
>>>>>>>>>>>>>>>          Maintenance process and might not be exposed through the spec. The Index
>>>>>>>>>>>>>>>          Maintenance process could handle this as a standard Iceberg Table and could
>>>>>>>>>>>>>>>          be based on the Table Maintenance process, but there might be some totally
>>>>>>>>>>>>>>>          different processes.
>>>>>>>>>>>>>>>       - It should be possible to add properties to an index
>>>>>>>>>>>>>>>       defined by the Index Maintenance process which could be used and updated in
>>>>>>>>>>>>>>>       the next Index Maintenance run.
>>>>>>>>>>>>>>>    - *PK index storage format benchmark results*
>>>>>>>>>>>>>>>       - Flat Parquet (baseline)
>>>>>>>>>>>>>>>       - BTree with Parquet leaves
>>>>>>>>>>>>>>>       - Vortex
>>>>>>>>>>>>>>>    - *Open items / next steps*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Peter
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont:
>>>>>>>>>>>>>>> 2026. márc. 23., H, 3:03):
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi everyone, I wanted to share an update on the primary key
>>>>>>>>>>>>>>>> index work.
>>>>>>>>>>>>>>>> Since there are still open questions on whether bloom
>>>>>>>>>>>>>>>> filter indexes fit in the secondary index framework or should be treated as
>>>>>>>>>>>>>>>> extended stats, I've shifted focus to the primary key index since it's a
>>>>>>>>>>>>>>>> clearer fit for the framework.
>>>>>>>>>>>>>>>> I've put together a proposal for a primary key
>>>>>>>>>>>>>>>> reverse-lookup index that maps each key to its physical location
>>>>>>>>>>>>>>>> (file_path, row_position). It enables:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - Scan-time file pruning for point lookups
>>>>>>>>>>>>>>>>    - Converting key-based deletes into position deletes
>>>>>>>>>>>>>>>>    (eliminating equality deletes for Flink CDC)
>>>>>>>>>>>>>>>>    - Accelerating Spark MERGE INTO by replacing full-table
>>>>>>>>>>>>>>>>    joins with direct file lookups
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Proposal:
>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1HuhCZ0n2FqDh8yqQb9oEj1CPM5yXpEsMPGZno2aSf8E/edit?tab=t.0#heading=h.tbevg4q0m9
>>>>>>>>>>>>>>>> Feedback welcome!
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Huaxin
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Mar 18, 2026 at 11:42 PM Péter Váry <
>>>>>>>>>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Key takeaways from the general index discussion at the May
>>>>>>>>>>>>>>>>> 16 meeting.
>>>>>>>>>>>>>>>>> Thanks to everyone who participated! The recording is
>>>>>>>>>>>>>>>>> available here:
>>>>>>>>>>>>>>>>> https://www.youtube.com/watch?v=btmjhtRWUCE
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - Q: Do we need to tie index types to the algorithms
>>>>>>>>>>>>>>>>>    used to access them?
>>>>>>>>>>>>>>>>>    - A: From a specification perspective, the goal is to
>>>>>>>>>>>>>>>>>    define the storage-level data layout so it can be shared across engines.
>>>>>>>>>>>>>>>>>    Engines are free to interpret and use the data as they see fit, but the
>>>>>>>>>>>>>>>>>    on-disk data layout itself must be strictly defined and interoperable.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - Q: Should we introduce an additional abstraction
>>>>>>>>>>>>>>>>>    layer (e.g., Vector Index) with sub-types such as IVF and DiskANN?
>>>>>>>>>>>>>>>>>    - A: This is possible if we decide it is beneficial. I
>>>>>>>>>>>>>>>>>    explored potential naming, but it is not yet clear how such a layer would
>>>>>>>>>>>>>>>>>    be used in practice.
>>>>>>>>>>>>>>>>>    *Question to Yingyi Bu*: could you provide examples
>>>>>>>>>>>>>>>>>    where this additional layer would be useful? Should this abstraction be
>>>>>>>>>>>>>>>>>    defined at the spec level, or is it better handled at the engine level?
>>>>>>>>>>>>>>>>>    My initial idea was that users would create a generic
>>>>>>>>>>>>>>>>>    Vector Index and let the engine choose the concrete implementation.
>>>>>>>>>>>>>>>>>    However, this would limit user control and users likely need to specify the
>>>>>>>>>>>>>>>>>    exact index representation, which implies they must be aware of the
>>>>>>>>>>>>>>>>>    available representations.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - Q: Do we want to allow extensibility for index types?
>>>>>>>>>>>>>>>>>    - A: Yes. The intent is to support a small set of
>>>>>>>>>>>>>>>>>    well-defined index types while allowing experimentation with new ones. If a
>>>>>>>>>>>>>>>>>    new index type proves broadly useful, a follow-up proposal can standardize
>>>>>>>>>>>>>>>>>    it and incorporate it into the spec.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - Q: Do we allow multiple versions of an index for the
>>>>>>>>>>>>>>>>>    same table snapshot?
>>>>>>>>>>>>>>>>>    - A: Yes. Older index versions must be retained for
>>>>>>>>>>>>>>>>>    readers that have already started using them, while new readers should
>>>>>>>>>>>>>>>>>    automatically use the latest available version
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - Q: Do we need to use materialized views for these
>>>>>>>>>>>>>>>>>    indexes?
>>>>>>>>>>>>>>>>>    - A: No. These indexes are primarily examples, and
>>>>>>>>>>>>>>>>>    different types may require different storage methods. However, the Primary
>>>>>>>>>>>>>>>>>    Key, Containing, and parts of the IVF indexes can be structured as Iceberg
>>>>>>>>>>>>>>>>>    tables. This allows engines to read them natively; in some cases, Iceberg
>>>>>>>>>>>>>>>>>    planners can automatically redirect queries to the index table without
>>>>>>>>>>>>>>>>>    engine modifications. Furthermore, index maintenance for these tables can
>>>>>>>>>>>>>>>>>    leverage existing materialized view maintenance workflows. Other index
>>>>>>>>>>>>>>>>>    types may instead rely on Puffin files or alternative storage approaches.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - Q: How should index metadata be accessed? Should we
>>>>>>>>>>>>>>>>>    add explicit pointers for the indexes in the table metadata?
>>>>>>>>>>>>>>>>>    - A: We did not have sufficient time to fully explore
>>>>>>>>>>>>>>>>>    and conclude this topic.
>>>>>>>>>>>>>>>>>    *Question for Yufei Gu*: Did I understand correctly
>>>>>>>>>>>>>>>>>    that your main concern stems from endpoint resolution from a REST Catalog
>>>>>>>>>>>>>>>>>    perspective? Specifically, if indexes are exposed under a URI such as
>>>>>>>>>>>>>>>>>    v1/{prefix}/namespaces/{namespace}/tables/{table}/indexes/{index}, would
>>>>>>>>>>>>>>>>>    this make it more difficult for the REST Catalog to resolve and route
>>>>>>>>>>>>>>>>>    requests to the appropriate endpoint?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Suhas Jayaram Subramanya via dev <de...@iceberg.apache.org>
>>>>>>>>>>>>>>>>> ezt írta (időpont: 2026. márc. 13., P, 23:32):
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Here's a proposal for native Vector Index support in
>>>>>>>>>>>>>>>>>> Iceberg tables --
>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1KL4qLOwdqnhOcqTc0EjO1O16NV3M3c-gZCEINDWw4lA/edit?usp=sharing
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We've been working on this proposal with Peter internally
>>>>>>>>>>>>>>>>>> at Microsoft and he suggested we post it here to bring this to the
>>>>>>>>>>>>>>>>>> community's attention, ahead of the next Secondary Index Sync.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Suhas
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 2026/02/19 04:34:34 huaxin gao wrote:
>>>>>>>>>>>>>>>>>> > Hi Everyone,
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Here are the recording and notes from the Iceberg Index
>>>>>>>>>>>>>>>>>> Support Sync on
>>>>>>>>>>>>>>>>>> > 2/11.
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Recording: https://www.youtube.com/watch?v=3sFfQ0A50yk
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Notes:
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.8041k7j2n7y3
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > The meeting will move to biweekly, Mondays 9–10am PST,
>>>>>>>>>>>>>>>>>> starting March 2.
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Since the sync, I updated the Bloom skipping index
>>>>>>>>>>>>>>>>>> proposal
>>>>>>>>>>>>>>>>>> > <
>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.5r5kl6k3fqwu
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > to address the discussion questions, specifically:
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > - Performance justification: when this helps
>>>>>>>>>>>>>>>>>> (high-cardinality = / IN,
>>>>>>>>>>>>>>>>>> > many data files, high object-store latency) and how it
>>>>>>>>>>>>>>>>>> differs from Parquet
>>>>>>>>>>>>>>>>>> > row-group Bloom filters (which still require opening
>>>>>>>>>>>>>>>>>> the data file).
>>>>>>>>>>>>>>>>>> > - Cost / scalability: rough sizing (Bloom blob size per
>>>>>>>>>>>>>>>>>> file, Puffin
>>>>>>>>>>>>>>>>>> > file size), the planning cost trade-off (driver index
>>>>>>>>>>>>>>>>>> reads vs executor
>>>>>>>>>>>>>>>>>> > file opens), and mitigations via caching.
>>>>>>>>>>>>>>>>>> > - Lifecycle / maintenance: incremental production as
>>>>>>>>>>>>>>>>>> new data files
>>>>>>>>>>>>>>>>>> > arrive, behavior when the index is missing/behind, and
>>>>>>>>>>>>>>>>>> sharding/compaction
>>>>>>>>>>>>>>>>>> > plus cleanup to avoid accumulating too many small
>>>>>>>>>>>>>>>>>> Puffin files over time.
>>>>>>>>>>>>>>>>>> > - Writer expectations: inline (optional) vs
>>>>>>>>>>>>>>>>>> asynchronous (primary) index
>>>>>>>>>>>>>>>>>> > creation.
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > I also implemented a Spark 4.1 POC
>>>>>>>>>>>>>>>>>> > <https://github.com/apache/iceberg/pull/15311> and a
>>>>>>>>>>>>>>>>>> local benchmark to
>>>>>>>>>>>>>>>>>> > quantify both the pruning impact (plannedFiles →
>>>>>>>>>>>>>>>>>> afterBloom) and the index
>>>>>>>>>>>>>>>>>> > read overhead (statsFiles, statsBytes,
>>>>>>>>>>>>>>>>>> bloomPayloadBytes) for point
>>>>>>>>>>>>>>>>>> > predicates on high-cardinality columns. Please take a
>>>>>>>>>>>>>>>>>> look and let me know
>>>>>>>>>>>>>>>>>> > if you have any questions or feedback.
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Huaxin
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > On Tue, Feb 10, 2026 at 1:43 PM huaxin gao <
>>>>>>>>>>>>>>>>>> hu...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > > Reminder for tomorrow's sync on Iceberg Index Support.
>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>> > > Wednesday: Feb. 11 9:00 – 10:00am
>>>>>>>>>>>>>>>>>> > > Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>> > > Google Meet joining info
>>>>>>>>>>>>>>>>>> > > Video call link: meet.google.com/nsp-ctyr-khk
>>>>>>>>>>>>>>>>>> > > Design doc:
>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.0#heading=h.hs6r9d26w1y2
>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.qouk73o4jxx7
>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>> > > Thanks,
>>>>>>>>>>>>>>>>>> > > Huaxin
>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>> > > On Tue, Feb 3, 2026 at 10:52 PM Péter Váry <
>>>>>>>>>>>>>>>>>> pe...@gmail.com>
>>>>>>>>>>>>>>>>>> > > wrote:
>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>> > >> Thanks Huaxin and Steven for organizing this.
>>>>>>>>>>>>>>>>>> Looking forward to meet you
>>>>>>>>>>>>>>>>>> > >> all next week!
>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>> > >> On Wed, Feb 4, 2026, 02:48 Steven Wu <
>>>>>>>>>>>>>>>>>> st...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>> > >>> We set up the dev calendar event with a new google
>>>>>>>>>>>>>>>>>> meet link. Please
>>>>>>>>>>>>>>>>>> > >>> ignore the link from Huaxin's original email.
>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>> > >>> The dev calendar has the correct info (including
>>>>>>>>>>>>>>>>>> the new meeting link)
>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>> > >>> Iceberg Index Support Sync
>>>>>>>>>>>>>>>>>> > >>> Wednesday, February 11 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>>> > >>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>> > >>> Google Meet joining info
>>>>>>>>>>>>>>>>>> > >>> Video call link:
>>>>>>>>>>>>>>>>>> https://meet.google.com/nsp-ctyr-khk
>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>> > >>> On Tue, Feb 3, 2026 at 5:08 PM huaxin gao <
>>>>>>>>>>>>>>>>>> hu...@gmail.com>
>>>>>>>>>>>>>>>>>> > >>> wrote:
>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>> > >>>> Sorry, I meant PST (not EST) :)
>>>>>>>>>>>>>>>>>> > >>>> Looking forward to the discussion!
>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>> > >>>> On Tue, Feb 3, 2026 at 4:58 PM Shawn Chang <
>>>>>>>>>>>>>>>>>> ye...@gmail.com>
>>>>>>>>>>>>>>>>>> > >>>> wrote:
>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>> > >>>>> Hi Huaxin,
>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>> > >>>>> Thanks for starting the sync!
>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>> > >>>>> The meeting seems to be 9-10AM PST on the dev
>>>>>>>>>>>>>>>>>> events calendar
>>>>>>>>>>>>>>>>>> > >>>>> <
>>>>>>>>>>>>>>>>>> https://calendar.google.com/calendar/u/0?cid=MzkwNWQ0OTJmMWI0NTBiYTA3MTJmMmFlNmFmYTc2ZWI3NTdmMTNkODUyMjBjYzAzYWE0NTI3ODg1YWRjNTYyOUBncm91cC5jYWxlbmRhci5nb29nbGUuY29t
>>>>>>>>>>>>>>>>>> >,
>>>>>>>>>>>>>>>>>> > >>>>> not EST. Maybe it's a typo?
>>>>>>>>>>>>>>>>>> > >>>>> Otherwise, looking forward to the discussion!
>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>> > >>>>> Best,
>>>>>>>>>>>>>>>>>> > >>>>> Shawn
>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>> > >>>>> On Tue, Feb 3, 2026 at 9:18 AM huaxin gao <
>>>>>>>>>>>>>>>>>> hu...@gmail.com>
>>>>>>>>>>>>>>>>>> > >>>>> wrote:
>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>> > >>>>>> Hi all,
>>>>>>>>>>>>>>>>>> > >>>>>> I'd like to start a dedicated sync to discuss
>>>>>>>>>>>>>>>>>> Iceberg Index support.
>>>>>>>>>>>>>>>>>> > >>>>>> Here is the existing discussion thread:
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> https://lists.apache.org/thread/fzqk3jjf0xpj5m4cfqb3v4c65p0t04ty
>>>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>> To ground the discussion, here are the two
>>>>>>>>>>>>>>>>>> proposals:
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>> - Peter's proposal
>>>>>>>>>>>>>>>>>> > >>>>>> <
>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.0#heading=h.hs6r9d26w1y2>
>>>>>>>>>>>>>>>>>> (overall
>>>>>>>>>>>>>>>>>> > >>>>>> index support)
>>>>>>>>>>>>>>>>>> > >>>>>> - My proposal
>>>>>>>>>>>>>>>>>> > >>>>>> <
>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.qouk73o4jxx7
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > >>>>>> (bloom filter skipping index)
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>> Time slot: Every 3 weeks, Wednesdays at 9 AM to
>>>>>>>>>>>>>>>>>> 10 AM EST, starting
>>>>>>>>>>>>>>>>>> > >>>>>> next Wednesday (2/11). After FileFormat sync
>>>>>>>>>>>>>>>>>> finishes, we plan to use that
>>>>>>>>>>>>>>>>>> > >>>>>> slot and switch to every other Monday, 9 AM to
>>>>>>>>>>>>>>>>>> 10 AM EST.
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>> Meet link: https://meet.google.com/fjn-tyze-mko
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>>>>>>>>> > >>>>>> Huaxin
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>

Re: Dedicated sync for Iceberg Index Support

[Péter Váry <pe...@gmail.com>]

Hi everyone,

Sorry for the delay in sharing the summary. I had some issues uploading the
recording, but it is now available:
https://www.youtube.com/watch?v=TTLQuCNx7EI

Summary

   - We revisited the index region design and agreed to use non-overlapping
   regions to optimize point lookup performance, accepting higher write
   amplification and deferring overlay-based approaches to future versions.
   - Discussed and agreed that we stick to total row ordering using sort
   keys (renamed from cluster keys).
   - Kept explicit index type metadata for compatibility, with readers
   skipping unsupported index types.
   - Agreed that dropping indexed columns invalidates the corresponding
   index and discussed schema evolution and type promotion handling.

Next steps

   - Resolve the remaining questions around index types and schema
   evolution.
   - Update the specification and PR to adopt the sort key terminology.
   - Continue review and discussion of the proposal.


The spec PR is now ready for review:
https://github.com/apache/iceberg/pull/16961

Thanks,
 Peter


Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026. szept.
15., K, 16:56):

> Hi everyone,
>
> Here is the recording of the Monday sync:
> https://drive.google.com/drive/folders/1iodHASJBVwNtDYEINq7k2HjogPsYQocH
>
> A brief summary of the discussion:
>
>    - Bloom Filter Index
>       - Puffin reduces file-open overhead, while Bloom filter
>       effectiveness appears independent of file placement.
>       - The observed scaling issues are likely due to rate limiting.
>       - Manifest alignment would be required for parallel planning.
>       - A scalar index is expected to deliver similar or better
>       performance.
>       - We will *prioritize scalar indexes* before pursuing additional
>       index types.
>       - The tests are still useful to understand the decisions in the
>       design stage.
>    - Scalar Index Design
>       - *Region file schema: identity, materialized, and non-materialized
>       fields*.
>       - Range files will use stable index-specific IDs, while expressions
>       reference table field IDs.
>       - Scalar indexes require both sorting and clustering; a dedicated
>       index definition may be cleaner than reusing table sort/cluster definitions.
>
> We can continue the remaining discussions offline or in the next sync.
>
> There are only a few open questions, so the spec PR is ready for review:
> https://github.com/apache/iceberg/pull/16961.
>
> Thanks,
> Peter
>
>
> Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026. szept.
> 14., H, 9:07):
>
>> One correction to the previous summary: I originally wrote that *Range*
>> files would store the index data, but I was later corrected that the
>> decision was actually to use *Region* files.
>>
>> The correct index file naming is:
>>
>>    - *Tracking file*: contains the list of region files (analogous to a
>>    manifest file for tables).
>>    - *Region file*: contains the index data rows (analogous to a data
>>    file for tables).
>>
>> Sorry for the confusion,
>> Peter
>>
>> Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026. szept.
>> 3., Cs, 13:18):
>>
>>>
>>> Hi everyone,
>>>
>>> Here is the recording of the Monday sync:
>>>
>>> https://apache-iceberg.slack.com/files/U06538UFDH6/F0BTT739Y7P/impromptu_google_meet_meeting_-_aug_31_2026.mp4
>>>
>>> Thanks, Shawn, for making it available.
>>>
>>> A brief summary of the discussion:
>>>
>>>    - We finalized the file naming:
>>>       - Tracking file: contains the list of range files (similar to a
>>>       manifest file for tables).
>>>       - Range file: contains the index data rows (similar to a data
>>>       file for tables).
>>>    - We agreed not to prescribe how expression values are materialized
>>>    (for example, Hilbert or hash transformations), leaving room for
>>>    implementation flexibility.
>>>    - Ryan mentioned that he is working on a v4 proposal for generating
>>>    Field IDs for non-materialized columns. The current thinking is that index
>>>    columns should follow the same approach.
>>>    - We agreed to retain older versions of the index metadata JSON.
>>>    - We agreed not to require a strict one-to-one mapping between index
>>>    snapshots and table snapshots. An index may contain multiple snapshots
>>>    corresponding to the same table snapshot.
>>>
>>> Based on the discussion, I have updated the spec PR. Following Ryan's
>>> pattern for non-materialized column definitions, the proposal now
>>> introduces:
>>>
>>>    - *materialized-fields* for range file columns
>>>    - *non-materialized-fields* for tracking file statistics
>>>    - *cluster-spec* to define which columns participate in range file
>>>    clustering
>>>
>>> Overall, I think we reached good alignment on the remaining design
>>> points, and I do not see any open questions at the moment. The PR is in
>>> good shape for review, so I would greatly appreciate any feedback or
>>> comments.
>>>
>>> Thanks everyone for the productive discussion and continued
>>> collaboration.
>>> Peter
>>>
>>> Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026. aug.
>>> 18., K, 14:17):
>>>
>>>> Hi everyone,
>>>>
>>>> Unfortunately, yesterday's sync was not recorded due to a technical
>>>> issue. :(
>>>>
>>>> Here is a brief summary of the discussion and the decisions we reached:
>>>>
>>>>    1. We agreed to use a list of Iceberg expressions to define the
>>>>    *index_keys*, which determine how the index is ordered.
>>>>    2. We agreed not to restrict the functions that can be used in
>>>>    those expressions. Engines that do not understand a particular expression
>>>>    can simply choose not to use the index.
>>>>    3. We agreed that index region files (leaf files) should be sorted
>>>>    by the values generated by the index_keys.
>>>>    4. We discussed using *included_column_ids* to define the contents
>>>>    and schema of the index region files, and generally felt this could be a
>>>>    good approach.
>>>>    5. We started discussing whether the results of the index_key
>>>>    expressions should also be stored in the index region files alongside the
>>>>    included columns. In some cases, storing them could help readers optimize
>>>>    lookups within a region file. In other cases, the values could be derived
>>>>    when needed, making them redundant and potentially hurting performance by
>>>>    increasing the size of the region files.
>>>>
>>>> I think we should continue the discussion around point 5, as the
>>>> outcome may influence, or even reopen, the decisions around point 4.
>>>>
>>>> Thanks!
>>>>
>>>>
>>>> Flavio Junqueira <fl...@gmail.com> ezt írta (időpont: 2026.
>>>> aug. 4., K, 18:57):
>>>>
>>>>> Thank you for the summary, the proposal and the overall discussion. I
>>>>> have one main question about the target and I apologize in advance if I'm
>>>>> missing some important detail as I might not have the full context.
>>>>>
>>>>> I noticed that partitioning, sorting and clustering (via Hilbert) were
>>>>> discussed in the call in this context of index transform functions. My
>>>>> understanding is that engines currently are already pruning partitions and
>>>>> files using partition information and table metadata like per-column
>>>>> min-max. If that's the case, then what's it that this proposal is trying to
>>>>> achieve by having transform functions that capture partitioning, sorting
>>>>> and clustering when engines are already able to prune without a secondary
>>>>> index? I can more easily see the primary key index example, but I'm not
>>>>> entirely sure about these data layout mechanisms.
>>>>>
>>>>> I also didn't quite get how engines are expected to consume such an
>>>>> index. There was a discussion about where the mapping of the input to files
>>>>> happens, whether that's in Iceberg or the engine, and I didn't quite get
>>>>> the recommendation. Checking Peter's proposal, I couldn't see it clearly
>>>>> either, and if I just missed it, I'd appreciate a pointer to where I should
>>>>> look into.
>>>>>
>>>>> -Flavio
>>>>>
>>>>> On 4 Aug 2026, at 10:52, Péter Váry <pe...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi everyone,
>>>>>
>>>>> Here is the recording from yesterday’s sync:
>>>>> https://www.youtube.com/watch?v=Wg9orP9JAmk
>>>>>
>>>>> To summarize:
>>>>> - We agreed to retain the history of index snapshots, but not the
>>>>> history of other index properties. If readers require information that may
>>>>> later be modified by a user, that information should be copied into the
>>>>> corresponding snapshot.
>>>>> - We discussed the possibility of defining index ordering using a list
>>>>> of transform functions.
>>>>>
>>>>> Dan and Yingyi suggested something along the following lines:
>>>>>
>>>>> *"index_keys" : [*
>>>>> *  {*
>>>>> *    "type": "apply",*
>>>>> *    "func-name": {*
>>>>> *      "catalog": "iceberg_functions",*
>>>>> *      "name": "day"*
>>>>> *    },*
>>>>> *    "arguments": [*
>>>>> *      { "type": "reference", "id": 3 }*
>>>>> *    ]*
>>>>> *  },*
>>>>> *  {*
>>>>> *    "type": "apply",*
>>>>> *    "func-name": {*
>>>>> *      "catalog": "iceberg_functions",*
>>>>> *      "name": "truncate"*
>>>>> *    },*
>>>>> *    "arguments": [*
>>>>> *      { "type": "reference", "id": 4 },*
>>>>> *      2*
>>>>> *    ]*
>>>>> *  }*
>>>>> *]*
>>>>>
>>>>>
>>>>> In this example, rows would first be ordered by day(col3) and then by
>>>>> truncate(col4, 2). To ensure a deterministic ordering, ties would be broken
>>>>> using the original values of columns 3 and 4. If those are still equal,
>>>>> ordering would fall back to the file name and finally the row position.
>>>>>
>>>>> I spent some time thinking about this after the sync, and here are my
>>>>> thoughts.
>>>>>
>>>>> Pros:
>>>>> - Provides a high degree of flexibility.
>>>>> - Enables expression-based indexes.
>>>>>
>>>>> Cons:
>>>>> - Do we need this flexibility at this stage?
>>>>>    - Equality deletes, the primary use cases we are targeting, do not
>>>>> require expressions.
>>>>>    - Most indexes I have seen in production are not expression-based.
>>>>> - Increased complexity for engines:
>>>>>    - Engines must parse and understand transforms to determine whether
>>>>> using an index can improve a query. For example, is ordering by
>>>>> day(timestamp) sufficient for a key lookup?
>>>>>    - Complexity increases further as new UDFs are introduced.
>>>>>    - Expressions are not pushed into scans today, so index handling
>>>>> cannot be fully hidden behind Iceberg scan planning.
>>>>> - Risk of fragmentation:
>>>>>    - The additional complexity may slow engine adoption.
>>>>>    - Adopters may choose to support only exact transform matches
>>>>> created by them, rather than learn to reason about the general transforms.
>>>>>    - UDFs are prone to missing implementations and behavioral
>>>>> differences across engines. In the worst case, such differences could lead
>>>>> to incorrect query results when indexes are used.
>>>>>
>>>>> Given the points above, I still prefer defining HASH, HILBERT,
>>>>> IDENTITY as a set of standardized values combined with an explicit list of
>>>>> key columns. This keeps adoption simple, unlocks the primary use cases we
>>>>> are targeting today, and allows the Iceberg library to provide the
>>>>> transform implementations directly.
>>>>>
>>>>> Internally, these transforms could still be represented as Iceberg
>>>>> expressions, giving us a migration path toward future index types that may
>>>>> require full expression support.
>>>>>
>>>>> For example:
>>>>>
>>>>> *{*
>>>>> *  "transform": "HILBERT",*
>>>>> *  "key-column-ids": [3, 4]*
>>>>> *}*
>>>>>
>>>>>
>>>>> I'm interested in hearing other opinions.
>>>>>
>>>>> Thanks,
>>>>> Peter
>>>>>
>>>>> Renjie Liu <li...@gmail.com> ezt írta (időpont: 2026. júl.
>>>>> 28., K, 10:07):
>>>>>
>>>>>> Hi, Peter:
>>>>>>
>>>>>> Thanks for driving the proposal.
>>>>>>
>>>>>> >  *Hash Index Type*: Should hash-based indexes be a separate index
>>>>>> type? They are only applicable to equality and IN predicates, and the hash
>>>>>> algorithm must always be known. A dedicated HASH type could make these
>>>>>> constraints explicit and distinguish hash-based indexes from ordered
>>>>>> indexes.
>>>>>>
>>>>>> I'm leaning toward keeping them in scalar index type due to the
>>>>>> similarity with others transformation functions like identity. As with the
>>>>>> constraints, I think they are properties of transformation functions.
>>>>>>
>>>>>> > *Transform Function/Layout Representation*: Should transform
>>>>>> functions be represented as an enum or a string? I lean toward strings, but
>>>>>> I do not have a strong preference. In many cases, an index can still be
>>>>>> used based on standard column statistics, even if a reader does not
>>>>>> recognize a particular transform.
>>>>>>
>>>>>> This seems more like a problem in java implementation?
>>>>>>
>>>>>> Also I prefer the name transform function compared with layout
>>>>>> representation. Layout representation makes me feel that it's not about the
>>>>>> definition of an index function.
>>>>>>
>>>>>> > *Future Index Types*: How should we handle index types that are
>>>>>> expected in the long term but fall outside the scope of this proposal (for
>>>>>> example, VECTOR, IVF, TERM)? Should they be mentioned as planned/reserved
>>>>>> concepts, or omitted entirely? I lean toward at least mentioning them. The
>>>>>> rationale for introducing an index type is easier to explain when
>>>>>> considering future index families.
>>>>>>
>>>>>> +1
>>>>>>
>>>>>> *> Index History Handling:* Most database systems treat index
>>>>>> changes as a drop-and-recreate operation rather than an in-place update. If
>>>>>> we follow the same approach, there is limited value in maintaining index
>>>>>> metadata history. During index maintenance, a new metadata.json can be
>>>>>> created and atomically swapped in, with the catalog responsible for
>>>>>> cleaning up obsolete metadata and index data files according to its
>>>>>> retention policy.
>>>>>>
>>>>>> +1, I think we only need to keep index data for different snapshots,
>>>>>> but if an index definition changed, we should drop and recreate a new index.
>>>>>>
>>>>>> > *File Format Location*: Should file format be specified at the
>>>>>> leaf-file level or only at the index level? My preference is to stay
>>>>>> aligned with the table metadata model and avoid introducing differences
>>>>>> unless there is a compelling requirement. This would simplify
>>>>>> implementation and maximize code reuse.
>>>>>>
>>>>>> +1.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 24, 2026 at 11:57 PM Péter Váry <
>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> Here are the key takeaways from Monday’s meeting (July 20):
>>>>>>>
>>>>>>>    - *Index Type*: An index type enum will be introduced in v1,
>>>>>>>    initially with a single value (SCALAR). This establishes a future-proof API
>>>>>>>    boundary while leaving room for additional index families.
>>>>>>>    - *Strict Total Ordering*: The specification should require
>>>>>>>    transform functions to produce values that preserve a well-defined total
>>>>>>>    ordering across the key space. The transform function does not need to
>>>>>>>    return a long. Instead, leaf files can store the first and last values
>>>>>>>    according to the ordering, enabling generic filtering capabilities for
>>>>>>>    readers.
>>>>>>>
>>>>>>> Open Questions:
>>>>>>>
>>>>>>>    - *Hash Index Type*: Should hash-based indexes be a separate
>>>>>>>    index type? They are only applicable to equality and IN predicates, and the
>>>>>>>    hash algorithm must always be known. A dedicated HASH type could make these
>>>>>>>    constraints explicit and distinguish hash-based indexes from ordered
>>>>>>>    indexes.
>>>>>>>    - *Transform Function/Layout Representation*: Should transform
>>>>>>>    functions be represented as an enum or a string? I lean toward strings, but
>>>>>>>    I do not have a strong preference. In many cases, an index can still be
>>>>>>>    used based on standard column statistics, even if a reader does not
>>>>>>>    recognize a particular transform.
>>>>>>>    - *Future Index Types*: How should we handle index types that
>>>>>>>    are expected in the long term but fall outside the scope of this proposal
>>>>>>>    (for example, VECTOR, IVF, TERM)? Should they be mentioned as
>>>>>>>    planned/reserved concepts, or omitted entirely? I lean toward at least
>>>>>>>    mentioning them. The rationale for introducing an index type is easier to
>>>>>>>    explain when considering future index families.
>>>>>>>    - *Index History Handling:* Most database systems treat index
>>>>>>>    changes as a drop-and-recreate operation rather than an in-place update. If
>>>>>>>    we follow the same approach, there is limited value in maintaining index
>>>>>>>    metadata history. During index maintenance, a new metadata.json can be
>>>>>>>    created and atomically swapped in, with the catalog responsible for
>>>>>>>    cleaning up obsolete metadata and index data files according to its
>>>>>>>    retention policy.
>>>>>>>    - *Min/Max Statistics Representation*: Should transform-specific
>>>>>>>    or sort-order min/max values be represented using the existing file
>>>>>>>    statistics structure, or exposed through dedicated fields? My preference is
>>>>>>>    to keep them within the statistics structure defined by the file schema
>>>>>>>    while exposing them directly through the Java API if needed.
>>>>>>>    - *File Format Location*: Should file format be specified at the
>>>>>>>    leaf-file level or only at the index level? My preference is to stay
>>>>>>>    aligned with the table metadata model and avoid introducing differences
>>>>>>>    unless there is a compelling requirement. This would simplify
>>>>>>>    implementation and maximize code reuse.
>>>>>>>
>>>>>>> I have updated the spec PR based on the decisions above:
>>>>>>> https://github.com/apache/iceberg/pull/16961
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Peter
>>>>>>>
>>>>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont: 2026. jún.
>>>>>>> 26., P, 19:42):
>>>>>>>
>>>>>>>> Hi all, Here is the summary of this Monday's index meeting:
>>>>>>>>
>>>>>>>> Two blocking decisions closed:
>>>>>>>>
>>>>>>>>    1. Index is *not* a table— it's its own object (reuses table
>>>>>>>>    machinery under the hood). Reasoning: requires a sort order, no column
>>>>>>>>    updates, no partition spec, no overlapping ranges between leaves, inherits
>>>>>>>>    base-table permissions, and has its own CREATE/DROP/UPDATE INDEX DDL.
>>>>>>>>    2. Index is a separate catalog entity, with no pointers in
>>>>>>>>    table metadata— has its own REST endpoints; the catalog can optionally
>>>>>>>>    return index metadata withloadTable to avoid extra round-trips. Keeps table
>>>>>>>>    and index updates independent/async.
>>>>>>>>
>>>>>>>>
>>>>>>>> Next steps: Start writing the spec and build out the copy-on-write
>>>>>>>> path now.
>>>>>>>>
>>>>>>>> Here are the draft spec:
>>>>>>>> secondary index spec <https://github.com/apache/iceberg/pull/16961>
>>>>>>>> irc spec <https://github.com/apache/iceberg/pull/16963>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Huaxin
>>>>>>>>
>>>>>>>> On Sat, Jun 20, 2026 at 11:11 AM huaxin gao <hu...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> I built a standalone PoC to validate that the basic index
>>>>>>>>> structure works: that we can build a PK index, convert equality deletes to
>>>>>>>>> position deletes through it, and have every converted delete land on the
>>>>>>>>> correct live row. I ran it up to *100M keys*.
>>>>>>>>>
>>>>>>>>> *Headline: the structure works.* The index builds over up to 100M
>>>>>>>>> keys, the eq-delete → position-delete conversion resolved correctly at
>>>>>>>>> *every* size (100% of converted deletes mapped to the right live
>>>>>>>>> row), and the resulting position deletes are *~8× cheaper to
>>>>>>>>> apply* at query time than the equality deletes they replace.
>>>>>>>>>
>>>>>>>>> Beyond correctness, the run also shows how the index’s
>>>>>>>>> *maintenance* cost scales, comparing copy-on-write (COW, rewrite
>>>>>>>>> touched leaves) vs an append/merge (MOR) option, under a realistic mixed
>>>>>>>>> CDC checkpoint (1,000 insert + 500 update + 500 delete), local wall-clock:
>>>>>>>>> keys EQ baseline INDEX (COW) % of 60s (COW) INDEX (MOR) % of 60s
>>>>>>>>> (MOR) correct
>>>>>>>>> 5M 6 ms 6.7s 11.2% 2.2s 3.7% PASS
>>>>>>>>> 20M 8 ms 24.2s 40.4% 6.4s 10.6% PASS
>>>>>>>>> 50M 7 ms 51.6s 86.1% 12.2s 20.4% PASS
>>>>>>>>> *100M* 6 ms *75.0s* 125% (BEHIND) *16.9s* 28.2% (keeps up) PASS
>>>>>>>>>
>>>>>>>>> COW maintenance crosses the 60 s checkpoint around 100M
>>>>>>>>> (75 s/cycle, 125%); MOR stays at ~28% and keeps pace; the equality-delete
>>>>>>>>> baseline is ~6 ms and flat. So the structure works, but *COW
>>>>>>>>> alone can’t sustain scattered CDC at hundreds of millions of keys on a
>>>>>>>>> single writer*. It’s worth allowing a merge-on-read / update-file
>>>>>>>>> maintenance option alongside COW (or sharding the index across parallel
>>>>>>>>> writers).
>>>>>>>>>
>>>>>>>>> *Full write-up, all tables, and the in-region reality-check:* link
>>>>>>>>> <https://docs.google.com/document/d/1G3zxbW8X0eU3UrouslZfp42bBc9CvgJGnJyDONCB4PU/edit?tab=t.0>
>>>>>>>>>
>>>>>>>>> Feedback welcome, especially on the spec direction (whether to
>>>>>>>>> allow a merge-on-read / update-file maintenance option alongside COW)  and
>>>>>>>>> on the read-side modeling.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Huaxin
>>>>>>>>>
>>>>>>>>> On Tue, Jun 9, 2026 at 5:45 PM huaxin gao <hu...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Sorry,  we've skipped posting a few of the dedicated index-sync
>>>>>>>>>> summaries to the mailing list; you can find those in the Google
>>>>>>>>>> doc
>>>>>>>>>> <https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?pli=1&tab=t.8041k7j2n7y3>
>>>>>>>>>> and the Slack channel. Here's yesterday's summary:
>>>>>>>>>>
>>>>>>>>>> *Decided*
>>>>>>>>>>
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    Index vs. table (what we agreed):
>>>>>>>>>>    - Reuse table implementation/library code and a
>>>>>>>>>>       near-identical spec — the commit path will be custom regardless, so reuse
>>>>>>>>>>       isn't the deciding factor.
>>>>>>>>>>       - An index is not a table from a user/API view: loading or
>>>>>>>>>>       writing an index as a table must fail(it would violate index invariants).
>>>>>>>>>>       - The spec forbids most table behaviors: no overlapping
>>>>>>>>>>       files, one mandatory transform sort order, no column updates, no partition
>>>>>>>>>>       spec.
>>>>>>>>>>       - Delete vectors: reuse Iceberg's existing DV — benchmarks
>>>>>>>>>>    showed no new delete format is worth introducing.
>>>>>>>>>>    - Incremental updates: start with copy-on-write only (no
>>>>>>>>>>    update files). For object-store-sized leaves, a full leaf rewrite is about
>>>>>>>>>>    as cheap as maintaining an overlay update file + DV, so we'll skip the MOR
>>>>>>>>>>    machinery for now and add it later only if benchmarks prove we need it
>>>>>>>>>>    (likely just the very-large-leaf case).
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    Validate the spec first: build a quick, hand-wired prototype
>>>>>>>>>>    (Parquet files structured per the spec) and benchmark it on real scales
>>>>>>>>>>    before formalizing.
>>>>>>>>>>
>>>>>>>>>> *Leaning, not final*
>>>>>>>>>>
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    Indexes are likely separate catalog objects, linked from the
>>>>>>>>>>    table by storing just an identifier (like materialized views) and not
>>>>>>>>>>    visible in LIST TABLES.
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    We'll need a commit path for indexes, but simpler than tables
>>>>>>>>>>    (no stage-create).
>>>>>>>>>>
>>>>>>>>>> *Still open*
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    Permissions model — separate vs. inherited (action: look at
>>>>>>>>>>    what real DBs do for index permissions).
>>>>>>>>>>    - REST/catalog RPC design — minimize round-trips; index
>>>>>>>>>>    metadata ideally returned with LOAD TABLE. Catalog RPC cost
>>>>>>>>>>    may dominate Parquet IO, so this needs real design.
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    Scale modeling — target rows-per-leaf vs. leaf size vs.
>>>>>>>>>>    metadata-file count.
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    DDL-on-index semantics (reuse table schema-update actions or
>>>>>>>>>>    separate)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Huaxin
>>>>>>>>>>
>>>>>>>>>> On Wed, Apr 22, 2026 at 8:47 AM Péter Váry <
>>>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi All,
>>>>>>>>>>>
>>>>>>>>>>> TL;DR
>>>>>>>>>>> We still need to validate with ADLS and S3, but based on the
>>>>>>>>>>> local tests, the MPHF approach looks more promising if we can tolerate
>>>>>>>>>>> larger files and longer index maintenance times.
>>>>>>>>>>>
>>>>>>>>>>> Details:
>>>>>>>>>>> Here are the results from the local experiments on my Mac. I
>>>>>>>>>>> removed unnecessary statistics from the Parquet files and tested different
>>>>>>>>>>> row group sizes:
>>>>>>>>>>>
>>>>>>>>>>>    - For an index file with 1M records, a row group size of
>>>>>>>>>>>    5,000 appears to be the sweet spot.
>>>>>>>>>>>    - For 10M records, 10,000 rows per row group works best.
>>>>>>>>>>>
>>>>>>>>>>> If you have additional ideas for optimizing Parquet-based
>>>>>>>>>>> indexes, I’d be very interested to hear them.
>>>>>>>>>>> The test code is available on this branch:
>>>>>>>>>>> https://github.com/pvary/iceberg/tree/leaf_bench
>>>>>>>>>>>
>>>>>>>>>>> Best results:
>>>>>>>>>>> *1m records/file*
>>>>>>>>>>>
>>>>>>>>>>>    - Parquet - 5000 row/RowGroup
>>>>>>>>>>>       - Read: 1191 µs - 1 file open, 3 seek, 123KB read per
>>>>>>>>>>>       lookup
>>>>>>>>>>>       - Write: 1.7 s, 15 MB
>>>>>>>>>>>    - MPHF
>>>>>>>>>>>       - Read: 202 µs - 1 file open, 1 seek,  282KB read per
>>>>>>>>>>>       lookup
>>>>>>>>>>>       - Write: 0.8 s, 34 MB
>>>>>>>>>>>
>>>>>>>>>>> *10m records/file*
>>>>>>>>>>>
>>>>>>>>>>>    - Parquet - 10000 row/RowGroup
>>>>>>>>>>>       - Read: 4168 µs - 1 file open, 3 seek, 395KB read per
>>>>>>>>>>>       lookup
>>>>>>>>>>>       - Write: 19.5s s, 144 MB
>>>>>>>>>>>    - MPHF
>>>>>>>>>>>       - Read: 1086 µs - 1 file open, 1 seek,  2.8 MB (2812KB)
>>>>>>>>>>>       read per lookup
>>>>>>>>>>>       - Write: 6.5 s, 34 MB, 353 MB
>>>>>>>>>>>
>>>>>>>>>>> Below are the full results.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> *Benchmark                                      (indexType)
>>>>>>>>>>>  (keyType)  (numRows)  Mode    Cnt            Score          Error
>>>>>>>>>>>  UnitsInvertedIndexBenchmark.lookup                 PARQUET_1000       LONG
>>>>>>>>>>>    1000000    ss  10000         3285.284 ±        5.138
>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_1000       LONG
>>>>>>>>>>>    1000000    ss  10000   2522168989.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_1000       LONG
>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_1000       LONG
>>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_1000       LONG
>>>>>>>>>>> 10000000    ss  10000        35449.614 ±       34.673
>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_1000       LONG
>>>>>>>>>>>   10000000    ss  10000  24302649201.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_1000       LONG
>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_1000       LONG
>>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_5000       LONG
>>>>>>>>>>>  1000000    ss  10000         1191.959 ±        4.169
>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_5000       LONG
>>>>>>>>>>>    1000000    ss  10000   1230877229.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_5000       LONG
>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_5000       LONG
>>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_5000       LONG
>>>>>>>>>>> 10000000    ss  10000         7236.447 ±       10.374
>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_5000       LONG
>>>>>>>>>>>   10000000    ss  10000   5650715973.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_5000       LONG
>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_5000       LONG
>>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_10000       LONG
>>>>>>>>>>>  1000000    ss  10000         1349.946 ±        7.834
>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_10000       LONG
>>>>>>>>>>>    1000000    ss  10000   1730219377.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_10000       LONG
>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_10000       LONG
>>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_10000       LONG
>>>>>>>>>>> 10000000    ss  10000         4168.635 ±       11.051
>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_10000       LONG
>>>>>>>>>>>   10000000    ss  10000   3946341532.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_10000       LONG
>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_10000       LONG
>>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_50000       LONG
>>>>>>>>>>>  1000000    ss  10000         4736.466 ±       38.179
>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_50000       LONG
>>>>>>>>>>>    1000000    ss  10000   7427413541.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_50000       LONG
>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_50000       LONG
>>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_50000       LONG
>>>>>>>>>>> 10000000    ss  10000         4979.031 ±       34.708
>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_50000       LONG
>>>>>>>>>>>   10000000    ss  10000   7694887636.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_50000       LONG
>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_50000       LONG
>>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup                         MPHF       LONG
>>>>>>>>>>>  1000000    ss  10000          202.571 ±        2.336
>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead               MPHF       LONG
>>>>>>>>>>>    1000000    ss  10000   2821570000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams             MPHF       LONG
>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks                   MPHF       LONG
>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup                         MPHF       LONG
>>>>>>>>>>> 10000000    ss  10000         1086.957 ±        4.524
>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead               MPHF       LONG
>>>>>>>>>>>   10000000    ss  10000  28119460000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams             MPHF       LONG
>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks                   MPHF       LONG
>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_1000       LONG
>>>>>>>>>>>  1000000    ss      3      1720731.014 ±   876636.004
>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_1000       LONG
>>>>>>>>>>>    1000000    ss      3     46453317.000
>>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_1000       LONG
>>>>>>>>>>> 10000000    ss      3     18547947.876 ± 12258125.307
>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_1000       LONG
>>>>>>>>>>>   10000000    ss      3    452655675.000
>>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_5000       LONG
>>>>>>>>>>>  1000000    ss      3      1718345.583 ±  1103928.016
>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_5000       LONG
>>>>>>>>>>>    1000000    ss      3     44845788.000
>>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_5000       LONG
>>>>>>>>>>> 10000000    ss      3     18604229.931 ±  2668361.915
>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_5000       LONG
>>>>>>>>>>>   10000000    ss      3    435388818.000
>>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_10000       LONG
>>>>>>>>>>>  1000000    ss      3      1761555.389 ±   535857.675
>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_10000       LONG
>>>>>>>>>>>    1000000    ss      3     44536635.000
>>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_10000       LONG
>>>>>>>>>>> 10000000    ss      3     19501588.264 ±  2130054.558
>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_10000       LONG
>>>>>>>>>>>   10000000    ss      3    433189623.000
>>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_50000       LONG
>>>>>>>>>>>  1000000    ss      3      1936624.889 ±  6601363.985
>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_50000       LONG
>>>>>>>>>>>    1000000    ss      3     44264655.000
>>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_50000       LONG
>>>>>>>>>>> 10000000    ss      3     20471742.278 ± 10705206.310
>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_50000       LONG
>>>>>>>>>>>   10000000    ss      3    431311305.000
>>>>>>>>>>> #InvertedIndexBenchmark.write                          MPHF       LONG
>>>>>>>>>>>  1000000    ss      3       896573.958 ±  1408024.851
>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes           MPHF       LONG
>>>>>>>>>>>    1000000    ss      3    102846369.000
>>>>>>>>>>> #InvertedIndexBenchmark.write                          MPHF       LONG
>>>>>>>>>>> 10000000    ss      3      6509348.875 ± 15519975.479
>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes           MPHF       LONG
>>>>>>>>>>>   10000000    ss      3   1058435733.000                     #*
>>>>>>>>>>>
>>>>>>>>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont: 2026.
>>>>>>>>>>> ápr. 21., K, 20:53):
>>>>>>>>>>>
>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>
>>>>>>>>>>>> In recent secondary index sync meetings, the discussion
>>>>>>>>>>>> converged on the need to define what an index is from first principles
>>>>>>>>>>>> before settling on physical layout.
>>>>>>>>>>>>
>>>>>>>>>>>> To address that, Peter and I have drafted a requirements
>>>>>>>>>>>> document for a key lookup index (renamed from "primary key index" to avoid
>>>>>>>>>>>> implying uniqueness enforcement), the goal is to nail down one well-scoped
>>>>>>>>>>>> index type first.
>>>>>>>>>>>>
>>>>>>>>>>>> Doc: Key Lookup Index Requirements
>>>>>>>>>>>> <https://docs.google.com/document/d/1e0zxK-jA0LBDq8YQlQgFipTHelDFiga8lCkgDTmYub8/edit?tab=t.0#heading=h.8shrgabvl19>
>>>>>>>>>>>>
>>>>>>>>>>>> It covers requirements, three design options (manifest + sorted
>>>>>>>>>>>> Parquet, hash + sorted Parquet, hash + MPHF) and open questions. We will
>>>>>>>>>>>> add preliminary benchmark results shortly.
>>>>>>>>>>>>
>>>>>>>>>>>> Feedback welcome — inline in the doc, on this thread, or at the
>>>>>>>>>>>> next index sync.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> Huaxin
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Apr 13, 2026 at 7:22 AM Steven Wu <st...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Do we need the special index identifier that was originally
>>>>>>>>>>>>> proposed? A generic CatalogObjectIdentifier (with namespace and name) would
>>>>>>>>>>>>> be consistent with all object types in the catalog. I have a discussion
>>>>>>>>>>>>> thread on the generic identifier topic: [DISCUSS] REST Spec:
>>>>>>>>>>>>> generic CatalogObjectIdentifier.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Should we add an indexes array field to table metadata? It
>>>>>>>>>>>>> only contains a list of index object identifiers. It doesn't contain any
>>>>>>>>>>>>> index metadata which should live in the index objects. Yufei was trying to
>>>>>>>>>>>>> bring this up at the end of the first sync. But we didn't get enough time
>>>>>>>>>>>>> to really discuss it. It will be great to discuss this as the first agenda
>>>>>>>>>>>>> item today.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Apr 13, 2026 at 3:17 AM Péter Váry <
>>>>>>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We had several engaging discussions at the Iceberg Summit,
>>>>>>>>>>>>>> and it was great to finally catch up with many of you in person. We truly
>>>>>>>>>>>>>> missed those who couldn’t attend, hopefully we’ll all meet again at the
>>>>>>>>>>>>>> next summit.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To keep the conversation going, Huaxin and I have put
>>>>>>>>>>>>>> together the agenda for our next meeting. As a reminder, we’ll meet on *April
>>>>>>>>>>>>>> 13th, 9:00–10:00 AM *PDT (6:00–7:00 PM CEST).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Proposed agenda:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - Continue first-principles index design discussion from
>>>>>>>>>>>>>>    Mar 30
>>>>>>>>>>>>>>       - *Index Ownership and Write Responsibility*
>>>>>>>>>>>>>>          - Should writers be allowed to update indexes, or
>>>>>>>>>>>>>>          - Should all index writes be handled exclusively by
>>>>>>>>>>>>>>          the Index Maintenance process?
>>>>>>>>>>>>>>          - If writers can update indexes then we need to
>>>>>>>>>>>>>>          define what guarantees are required (compaction, file splitting, layout
>>>>>>>>>>>>>>          expectations)?
>>>>>>>>>>>>>>          - If only Index Maintenance updates indexes then we
>>>>>>>>>>>>>>          only need to define what observable properties should be exposed to
>>>>>>>>>>>>>>          consumers? Like:
>>>>>>>>>>>>>>             - Expected max files for a single key
>>>>>>>>>>>>>>             - Current max files for a single key
>>>>>>>>>>>>>>             - Deletes allowed/present
>>>>>>>>>>>>>>             - Sorted by
>>>>>>>>>>>>>>             - Partitioned by
>>>>>>>>>>>>>>          - *Specification Scope: What Belongs in the Spec?*
>>>>>>>>>>>>>>          - Related to the ownership question above
>>>>>>>>>>>>>>          - Light spec: Just define that the index table
>>>>>>>>>>>>>>          should be optimized for retrieval by key columns and the index columns
>>>>>>>>>>>>>>          should be contained in the table. This could give us more flexibility if
>>>>>>>>>>>>>>          better organization methods come up, or
>>>>>>>>>>>>>>          - Detailed spec: We could define the max number of
>>>>>>>>>>>>>>          files per index to read for a single key, or even the partitioning and the
>>>>>>>>>>>>>>          exact sort order. This could allow more use-cases for a given index, like
>>>>>>>>>>>>>>          joins or cardinality estimations.
>>>>>>>>>>>>>>          - I would go for light spec for the main types (PK,
>>>>>>>>>>>>>>          Containing) and only the Index Maintenance processes should update the
>>>>>>>>>>>>>>          Indexes, as for many use-cases the details are not important, and writers
>>>>>>>>>>>>>>          will very rarely update the Indexes themselves.
>>>>>>>>>>>>>>       - *Logical Placement of Indexes*
>>>>>>>>>>>>>>          - Index as a child object of an Iceberg Table, or
>>>>>>>>>>>>>>          - Index as a first‑class entity under
>>>>>>>>>>>>>>          /namespace/indexes/{index}
>>>>>>>>>>>>>>          - Based on the discussions on the summit we are
>>>>>>>>>>>>>>          leaning in this direction. This means the index id should be unique in the
>>>>>>>>>>>>>>          namespace but helps the catalog implementations quite a bit
>>>>>>>>>>>>>>       - *Physical Placement of Index Data*
>>>>>>>>>>>>>>          - I don’t think we should specify this. We should
>>>>>>>>>>>>>>          have a base location for the index, but can rely on the catalog
>>>>>>>>>>>>>>          implementations to decide on their own, like they do with the tables,
>>>>>>>>>>>>>>          views, udfs.
>>>>>>>>>>>>>>       - *Iceberg Reader Based indexes* (Containing indexes
>>>>>>>>>>>>>>       and potentially PK indexes). These are the indexes which could be read by
>>>>>>>>>>>>>>       the existing Iceberg readers. We might decide to store the PK index
>>>>>>>>>>>>>>       similarly to an Iceberg Table and treat it as a reader based index.
>>>>>>>>>>>>>>          - What are the table properties/features exposed to
>>>>>>>>>>>>>>          the readers
>>>>>>>>>>>>>>             - Maybe just some behavioral descriptors for the
>>>>>>>>>>>>>>             optimizer to decide if the index could be used or should be skipped, like:
>>>>>>>>>>>>>>                - Expected max files for a single key
>>>>>>>>>>>>>>                - max files for a single key
>>>>>>>>>>>>>>                - Deletes allowed/present
>>>>>>>>>>>>>>                - Sorted by
>>>>>>>>>>>>>>                - Partitioned by
>>>>>>>>>>>>>>             - The Tasks when reading the index based on the
>>>>>>>>>>>>>>             filters and projection
>>>>>>>>>>>>>>          - What are the table properties/features exposed to
>>>>>>>>>>>>>>          the Index Maintenance. I think this could be internal to the Index
>>>>>>>>>>>>>>          Maintenance process and might not be exposed through the spec. The Index
>>>>>>>>>>>>>>          Maintenance process could handle this as a standard Iceberg Table and could
>>>>>>>>>>>>>>          be based on the Table Maintenance process, but there might be some totally
>>>>>>>>>>>>>>          different processes.
>>>>>>>>>>>>>>       - It should be possible to add properties to an index
>>>>>>>>>>>>>>       defined by the Index Maintenance process which could be used and updated in
>>>>>>>>>>>>>>       the next Index Maintenance run.
>>>>>>>>>>>>>>    - *PK index storage format benchmark results*
>>>>>>>>>>>>>>       - Flat Parquet (baseline)
>>>>>>>>>>>>>>       - BTree with Parquet leaves
>>>>>>>>>>>>>>       - Vortex
>>>>>>>>>>>>>>    - *Open items / next steps*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Peter
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont: 2026.
>>>>>>>>>>>>>> márc. 23., H, 3:03):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi everyone, I wanted to share an update on the primary key
>>>>>>>>>>>>>>> index work.
>>>>>>>>>>>>>>> Since there are still open questions on whether bloom filter
>>>>>>>>>>>>>>> indexes fit in the secondary index framework or should be treated as
>>>>>>>>>>>>>>> extended stats, I've shifted focus to the primary key index since it's a
>>>>>>>>>>>>>>> clearer fit for the framework.
>>>>>>>>>>>>>>> I've put together a proposal for a primary key
>>>>>>>>>>>>>>> reverse-lookup index that maps each key to its physical location
>>>>>>>>>>>>>>> (file_path, row_position). It enables:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    - Scan-time file pruning for point lookups
>>>>>>>>>>>>>>>    - Converting key-based deletes into position deletes
>>>>>>>>>>>>>>>    (eliminating equality deletes for Flink CDC)
>>>>>>>>>>>>>>>    - Accelerating Spark MERGE INTO by replacing full-table
>>>>>>>>>>>>>>>    joins with direct file lookups
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Proposal:
>>>>>>>>>>>>>>> https://docs.google.com/document/d/1HuhCZ0n2FqDh8yqQb9oEj1CPM5yXpEsMPGZno2aSf8E/edit?tab=t.0#heading=h.tbevg4q0m9
>>>>>>>>>>>>>>> Feedback welcome!
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Huaxin
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Mar 18, 2026 at 11:42 PM Péter Váry <
>>>>>>>>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Key takeaways from the general index discussion at the May
>>>>>>>>>>>>>>>> 16 meeting.
>>>>>>>>>>>>>>>> Thanks to everyone who participated! The recording is
>>>>>>>>>>>>>>>> available here: https://www.youtube.com/watch?v=btmjhtRWUCE
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - Q: Do we need to tie index types to the algorithms
>>>>>>>>>>>>>>>>    used to access them?
>>>>>>>>>>>>>>>>    - A: From a specification perspective, the goal is to
>>>>>>>>>>>>>>>>    define the storage-level data layout so it can be shared across engines.
>>>>>>>>>>>>>>>>    Engines are free to interpret and use the data as they see fit, but the
>>>>>>>>>>>>>>>>    on-disk data layout itself must be strictly defined and interoperable.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - Q: Should we introduce an additional abstraction
>>>>>>>>>>>>>>>>    layer (e.g., Vector Index) with sub-types such as IVF and DiskANN?
>>>>>>>>>>>>>>>>    - A: This is possible if we decide it is beneficial. I
>>>>>>>>>>>>>>>>    explored potential naming, but it is not yet clear how such a layer would
>>>>>>>>>>>>>>>>    be used in practice.
>>>>>>>>>>>>>>>>    *Question to Yingyi Bu*: could you provide examples
>>>>>>>>>>>>>>>>    where this additional layer would be useful? Should this abstraction be
>>>>>>>>>>>>>>>>    defined at the spec level, or is it better handled at the engine level?
>>>>>>>>>>>>>>>>    My initial idea was that users would create a generic
>>>>>>>>>>>>>>>>    Vector Index and let the engine choose the concrete implementation.
>>>>>>>>>>>>>>>>    However, this would limit user control and users likely need to specify the
>>>>>>>>>>>>>>>>    exact index representation, which implies they must be aware of the
>>>>>>>>>>>>>>>>    available representations.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - Q: Do we want to allow extensibility for index types?
>>>>>>>>>>>>>>>>    - A: Yes. The intent is to support a small set of
>>>>>>>>>>>>>>>>    well-defined index types while allowing experimentation with new ones. If a
>>>>>>>>>>>>>>>>    new index type proves broadly useful, a follow-up proposal can standardize
>>>>>>>>>>>>>>>>    it and incorporate it into the spec.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - Q: Do we allow multiple versions of an index for the
>>>>>>>>>>>>>>>>    same table snapshot?
>>>>>>>>>>>>>>>>    - A: Yes. Older index versions must be retained for
>>>>>>>>>>>>>>>>    readers that have already started using them, while new readers should
>>>>>>>>>>>>>>>>    automatically use the latest available version
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - Q: Do we need to use materialized views for these
>>>>>>>>>>>>>>>>    indexes?
>>>>>>>>>>>>>>>>    - A: No. These indexes are primarily examples, and
>>>>>>>>>>>>>>>>    different types may require different storage methods. However, the Primary
>>>>>>>>>>>>>>>>    Key, Containing, and parts of the IVF indexes can be structured as Iceberg
>>>>>>>>>>>>>>>>    tables. This allows engines to read them natively; in some cases, Iceberg
>>>>>>>>>>>>>>>>    planners can automatically redirect queries to the index table without
>>>>>>>>>>>>>>>>    engine modifications. Furthermore, index maintenance for these tables can
>>>>>>>>>>>>>>>>    leverage existing materialized view maintenance workflows. Other index
>>>>>>>>>>>>>>>>    types may instead rely on Puffin files or alternative storage approaches.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - Q: How should index metadata be accessed? Should we
>>>>>>>>>>>>>>>>    add explicit pointers for the indexes in the table metadata?
>>>>>>>>>>>>>>>>    - A: We did not have sufficient time to fully explore
>>>>>>>>>>>>>>>>    and conclude this topic.
>>>>>>>>>>>>>>>>    *Question for Yufei Gu*: Did I understand correctly
>>>>>>>>>>>>>>>>    that your main concern stems from endpoint resolution from a REST Catalog
>>>>>>>>>>>>>>>>    perspective? Specifically, if indexes are exposed under a URI such as
>>>>>>>>>>>>>>>>    v1/{prefix}/namespaces/{namespace}/tables/{table}/indexes/{index}, would
>>>>>>>>>>>>>>>>    this make it more difficult for the REST Catalog to resolve and route
>>>>>>>>>>>>>>>>    requests to the appropriate endpoint?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Suhas Jayaram Subramanya via dev <de...@iceberg.apache.org>
>>>>>>>>>>>>>>>> ezt írta (időpont: 2026. márc. 13., P, 23:32):
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Here's a proposal for native Vector Index support in
>>>>>>>>>>>>>>>>> Iceberg tables --
>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1KL4qLOwdqnhOcqTc0EjO1O16NV3M3c-gZCEINDWw4lA/edit?usp=sharing
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We've been working on this proposal with Peter internally
>>>>>>>>>>>>>>>>> at Microsoft and he suggested we post it here to bring this to the
>>>>>>>>>>>>>>>>> community's attention, ahead of the next Secondary Index Sync.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Suhas
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 2026/02/19 04:34:34 huaxin gao wrote:
>>>>>>>>>>>>>>>>> > Hi Everyone,
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Here are the recording and notes from the Iceberg Index
>>>>>>>>>>>>>>>>> Support Sync on
>>>>>>>>>>>>>>>>> > 2/11.
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Recording: https://www.youtube.com/watch?v=3sFfQ0A50yk
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Notes:
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.8041k7j2n7y3
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > The meeting will move to biweekly, Mondays 9–10am PST,
>>>>>>>>>>>>>>>>> starting March 2.
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Since the sync, I updated the Bloom skipping index
>>>>>>>>>>>>>>>>> proposal
>>>>>>>>>>>>>>>>> > <
>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.5r5kl6k3fqwu
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > to address the discussion questions, specifically:
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > - Performance justification: when this helps
>>>>>>>>>>>>>>>>> (high-cardinality = / IN,
>>>>>>>>>>>>>>>>> > many data files, high object-store latency) and how it
>>>>>>>>>>>>>>>>> differs from Parquet
>>>>>>>>>>>>>>>>> > row-group Bloom filters (which still require opening the
>>>>>>>>>>>>>>>>> data file).
>>>>>>>>>>>>>>>>> > - Cost / scalability: rough sizing (Bloom blob size per
>>>>>>>>>>>>>>>>> file, Puffin
>>>>>>>>>>>>>>>>> > file size), the planning cost trade-off (driver index
>>>>>>>>>>>>>>>>> reads vs executor
>>>>>>>>>>>>>>>>> > file opens), and mitigations via caching.
>>>>>>>>>>>>>>>>> > - Lifecycle / maintenance: incremental production as new
>>>>>>>>>>>>>>>>> data files
>>>>>>>>>>>>>>>>> > arrive, behavior when the index is missing/behind, and
>>>>>>>>>>>>>>>>> sharding/compaction
>>>>>>>>>>>>>>>>> > plus cleanup to avoid accumulating too many small Puffin
>>>>>>>>>>>>>>>>> files over time.
>>>>>>>>>>>>>>>>> > - Writer expectations: inline (optional) vs asynchronous
>>>>>>>>>>>>>>>>> (primary) index
>>>>>>>>>>>>>>>>> > creation.
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > I also implemented a Spark 4.1 POC
>>>>>>>>>>>>>>>>> > <https://github.com/apache/iceberg/pull/15311> and a
>>>>>>>>>>>>>>>>> local benchmark to
>>>>>>>>>>>>>>>>> > quantify both the pruning impact (plannedFiles →
>>>>>>>>>>>>>>>>> afterBloom) and the index
>>>>>>>>>>>>>>>>> > read overhead (statsFiles, statsBytes,
>>>>>>>>>>>>>>>>> bloomPayloadBytes) for point
>>>>>>>>>>>>>>>>> > predicates on high-cardinality columns. Please take a
>>>>>>>>>>>>>>>>> look and let me know
>>>>>>>>>>>>>>>>> > if you have any questions or feedback.
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Huaxin
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > On Tue, Feb 10, 2026 at 1:43 PM huaxin gao <
>>>>>>>>>>>>>>>>> hu...@gmail.com> wrote:
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > > Reminder for tomorrow's sync on Iceberg Index Support.
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > > Wednesday: Feb. 11 9:00 – 10:00am
>>>>>>>>>>>>>>>>> > > Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>> > > Google Meet joining info
>>>>>>>>>>>>>>>>> > > Video call link: meet.google.com/nsp-ctyr-khk
>>>>>>>>>>>>>>>>> > > Design doc:
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.0#heading=h.hs6r9d26w1y2
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.qouk73o4jxx7
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > > Thanks,
>>>>>>>>>>>>>>>>> > > Huaxin
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > > On Tue, Feb 3, 2026 at 10:52 PM Péter Váry <
>>>>>>>>>>>>>>>>> pe...@gmail.com>
>>>>>>>>>>>>>>>>> > > wrote:
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > >> Thanks Huaxin and Steven for organizing this. Looking
>>>>>>>>>>>>>>>>> forward to meet you
>>>>>>>>>>>>>>>>> > >> all next week!
>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>> > >> On Wed, Feb 4, 2026, 02:48 Steven Wu <st...@gmail.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>> > >>> We set up the dev calendar event with a new google
>>>>>>>>>>>>>>>>> meet link. Please
>>>>>>>>>>>>>>>>> > >>> ignore the link from Huaxin's original email.
>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>> > >>> The dev calendar has the correct info (including the
>>>>>>>>>>>>>>>>> new meeting link)
>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>> > >>> Iceberg Index Support Sync
>>>>>>>>>>>>>>>>> > >>> Wednesday, February 11 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>> > >>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>> > >>> Google Meet joining info
>>>>>>>>>>>>>>>>> > >>> Video call link:
>>>>>>>>>>>>>>>>> https://meet.google.com/nsp-ctyr-khk
>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>> > >>> On Tue, Feb 3, 2026 at 5:08 PM huaxin gao <
>>>>>>>>>>>>>>>>> hu...@gmail.com>
>>>>>>>>>>>>>>>>> > >>> wrote:
>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>> > >>>> Sorry, I meant PST (not EST) :)
>>>>>>>>>>>>>>>>> > >>>> Looking forward to the discussion!
>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>> > >>>> On Tue, Feb 3, 2026 at 4:58 PM Shawn Chang <
>>>>>>>>>>>>>>>>> ye...@gmail.com>
>>>>>>>>>>>>>>>>> > >>>> wrote:
>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>> > >>>>> Hi Huaxin,
>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>> > >>>>> Thanks for starting the sync!
>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>> > >>>>> The meeting seems to be 9-10AM PST on the dev
>>>>>>>>>>>>>>>>> events calendar
>>>>>>>>>>>>>>>>> > >>>>> <
>>>>>>>>>>>>>>>>> https://calendar.google.com/calendar/u/0?cid=MzkwNWQ0OTJmMWI0NTBiYTA3MTJmMmFlNmFmYTc2ZWI3NTdmMTNkODUyMjBjYzAzYWE0NTI3ODg1YWRjNTYyOUBncm91cC5jYWxlbmRhci5nb29nbGUuY29t
>>>>>>>>>>>>>>>>> >,
>>>>>>>>>>>>>>>>> > >>>>> not EST. Maybe it's a typo?
>>>>>>>>>>>>>>>>> > >>>>> Otherwise, looking forward to the discussion!
>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>> > >>>>> Best,
>>>>>>>>>>>>>>>>> > >>>>> Shawn
>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>> > >>>>> On Tue, Feb 3, 2026 at 9:18 AM huaxin gao <
>>>>>>>>>>>>>>>>> hu...@gmail.com>
>>>>>>>>>>>>>>>>> > >>>>> wrote:
>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>> > >>>>>> Hi all,
>>>>>>>>>>>>>>>>> > >>>>>> I'd like to start a dedicated sync to discuss
>>>>>>>>>>>>>>>>> Iceberg Index support.
>>>>>>>>>>>>>>>>> > >>>>>> Here is the existing discussion thread:
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> https://lists.apache.org/thread/fzqk3jjf0xpj5m4cfqb3v4c65p0t04ty
>>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> > >>>>>> To ground the discussion, here are the two
>>>>>>>>>>>>>>>>> proposals:
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> > >>>>>> - Peter's proposal
>>>>>>>>>>>>>>>>> > >>>>>> <
>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.0#heading=h.hs6r9d26w1y2>
>>>>>>>>>>>>>>>>> (overall
>>>>>>>>>>>>>>>>> > >>>>>> index support)
>>>>>>>>>>>>>>>>> > >>>>>> - My proposal
>>>>>>>>>>>>>>>>> > >>>>>> <
>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.qouk73o4jxx7
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > >>>>>> (bloom filter skipping index)
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> > >>>>>> Time slot: Every 3 weeks, Wednesdays at 9 AM to
>>>>>>>>>>>>>>>>> 10 AM EST, starting
>>>>>>>>>>>>>>>>> > >>>>>> next Wednesday (2/11). After FileFormat sync
>>>>>>>>>>>>>>>>> finishes, we plan to use that
>>>>>>>>>>>>>>>>> > >>>>>> slot and switch to every other Monday, 9 AM to 10
>>>>>>>>>>>>>>>>> AM EST.
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> > >>>>>> Meet link: https://meet.google.com/fjn-tyze-mko
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>>>>>>>> > >>>>>> Huaxin
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>

Re: Dedicated sync for Iceberg Index Support

[Daniel Weeks <dw...@apache.org>]

It seems that approaches for tracking/managing indexes are inconsistent
across implementations, but I think the proposal is best aligned with how
it would fit in context of Iceberg and the REST spec.

+1 on the proposal

On Wed, Oct 28, 2026 at 6:25 PM huaxin gao <hu...@gmail.com> wrote:

> Hi everyone,
>
> Following up on my action item from Monday's sync: here is what I found on
> how databases, catalogs, and REST APIs model, list, and name indexes.
>
> 1. Where an index lives
>
>    - Iceberg REST spec: resources that belong to one table are already
>    nested under it, for example .../tables/{table}/plan, /credentials, and
>    /metrics.
>    - Catalog and REST APIs scope indexes to the table. None of them makes
>    an index a top-level namespace object.
>       - Lance (/v1/table/{id}/index/{index_name}) and Firestore put the
>       index under the table in the URL path.
>       - Glue (DatabaseName + TableName + IndexName) and Hive Metastore
>       (get_indexes(db, tbl)) require the table to address an index.
>    - SQL databases are split.
>       - MySQL, SQL Server, Snowflake, and BigQuery make index names
>       unique per table, so you name the table to address an index (for example
>       DROP INDEX idx ON tbl in MySQL).
>       - Postgres, Oracle, and SQLite make an index a schema object (DROP
>       INDEX idx). Even Postgres always creates the index in its table's schema.
>
> 2. Listing across tables
> Most SQL databases also list indexes across tables, separately from where
> the index lives. For example, Snowflake has SHOW INDEXES IN SCHEMA /
> DATABASE / ACCOUNT, and MySQL and Postgres have system views
> (information_schema.STATISTICS and pg_indexes) that return every index in a
> schema in one query. This covers the maintenance use case: find every index
> in a namespace without going table by table.
>
> 3. Rename
> MySQL, SQL Server, Postgres, and Oracle support index rename, mainly for
> naming cleanup and schema migration tools. Glue, Lance, Firestore, and
> BigQuery don't. That supports leaving it out of the first version.
>
> Proposal:
>
>    - Nest index routes under the table:
>    /v1/{prefix}/namespaces/{namespace}/tables/{table}/indexes[/{index}]
>    - Make index names unique per table. The index identifier then
>    includes its table identifier plus the index name.
>    - Add a second, optional endpoint that lists all indexes in a
>    namespace, for maintenance: GET
>    /v1/{prefix}/namespaces/{namespace}/indexes. For each index, it returns the
>    index name and the table it belongs to. Catalogs don't have to support it.
>
> Feedback welcome.
>
> Thanks,
> Huaxin
>
> On Tue, Oct 27, 2026 at 10:00 AM huaxin gao <hu...@gmail.com>
> wrote:
>
>> Hi everyone,
>>
>> Here is a summary of yesterday's index sync. Thanks to everyone who
>> joined.
>>
>> 1. Index spec (#16961 <https://github.com/apache/iceberg/pull/16961>)
>> We went over the two questions Péter listed:
>> - Region file boundary: the per-column group max value is replaced by a
>> top-level max_ordering_key in each tracking file entry.
>> - Region file schema: the index now stores its own schema, with a mapping
>> from table fields to index fields, so index field IDs are independent of
>> the table's. Computed values are derived columns, either stored in the
>> region files (like a SQL generated column) or kept only as stats. This
>> depends on the derived columns proposal for V4, and the index keeps its
>> schema history (for example after int to long promotion).
>> There are no open issues left on the spec PR. Please review:
>> https://github.com/apache/iceberg/pull/16961
>>
>> 2. Index REST spec
>> Agreed:
>> - An index must be in the same namespace as its table. Cross-namespace
>> indexes are dropped.
>> - Rename and register are left out of the first version, to reduce
>> complexity. We can add them later if there is demand.
>>
>> Main open question: should index routes be namespace-level (as in the
>> PR), or nested under the table?
>> Prior art differs: MySQL scopes indexes to a table (DROP INDEX idx ON
>> tbl), while Postgres makes index names unique within a schema (DROP INDEX
>> idx). Lance nests indexes under the table.
>> Next step: look at how major databases and REST APIs model indexes, then
>> revisit the route design at the next sync.
>>
>> Action items
>> - Everyone: review the index spec PR #16961
>> <https://github.com/apache/iceberg/pull/16961>.
>> - Huaxin: research how MySQL, Postgres, and other databases and REST APIs
>> model, list, and name indexes, and whether rename has real use cases.
>>
>> Recording and notes:
>> https://drive.google.com/drive/folders/1iodHASJBVwNtDYEINq7k2HjogPsYQocH
>>
>> Thanks,
>> Huaxin
>>
>> On Tue, Sep 15, 2026 at 7:56 AM Péter Váry <pe...@gmail.com>
>> wrote:
>>
>>> Hi everyone,
>>>
>>> Here is the recording of the Monday sync:
>>> https://drive.google.com/drive/folders/1iodHASJBVwNtDYEINq7k2HjogPsYQocH
>>>
>>> A brief summary of the discussion:
>>>
>>>    - Bloom Filter Index
>>>       - Puffin reduces file-open overhead, while Bloom filter
>>>       effectiveness appears independent of file placement.
>>>       - The observed scaling issues are likely due to rate limiting.
>>>       - Manifest alignment would be required for parallel planning.
>>>       - A scalar index is expected to deliver similar or better
>>>       performance.
>>>       - We will *prioritize scalar indexes* before pursuing additional
>>>       index types.
>>>       - The tests are still useful to understand the decisions in the
>>>       design stage.
>>>    - Scalar Index Design
>>>       - *Region file schema: identity, materialized, and
>>>       non-materialized fields*.
>>>       - Range files will use stable index-specific IDs, while
>>>       expressions reference table field IDs.
>>>       - Scalar indexes require both sorting and clustering; a dedicated
>>>       index definition may be cleaner than reusing table sort/cluster definitions.
>>>
>>> We can continue the remaining discussions offline or in the next sync.
>>>
>>> There are only a few open questions, so the spec PR is ready for review:
>>> https://github.com/apache/iceberg/pull/16961.
>>>
>>> Thanks,
>>> Peter
>>>
>>>
>>> Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026.
>>> szept. 14., H, 9:07):
>>>
>>>> One correction to the previous summary: I originally wrote that *Range*
>>>> files would store the index data, but I was later corrected that the
>>>> decision was actually to use *Region* files.
>>>>
>>>> The correct index file naming is:
>>>>
>>>>    - *Tracking file*: contains the list of region files (analogous to
>>>>    a manifest file for tables).
>>>>    - *Region file*: contains the index data rows (analogous to a data
>>>>    file for tables).
>>>>
>>>> Sorry for the confusion,
>>>> Peter
>>>>
>>>> Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026.
>>>> szept. 3., Cs, 13:18):
>>>>
>>>>>
>>>>> Hi everyone,
>>>>>
>>>>> Here is the recording of the Monday sync:
>>>>>
>>>>> https://apache-iceberg.slack.com/files/U06538UFDH6/F0BTT739Y7P/impromptu_google_meet_meeting_-_aug_31_2026.mp4
>>>>>
>>>>> Thanks, Shawn, for making it available.
>>>>>
>>>>> A brief summary of the discussion:
>>>>>
>>>>>    - We finalized the file naming:
>>>>>       - Tracking file: contains the list of range files (similar to a
>>>>>       manifest file for tables).
>>>>>       - Range file: contains the index data rows (similar to a data
>>>>>       file for tables).
>>>>>    - We agreed not to prescribe how expression values are
>>>>>    materialized (for example, Hilbert or hash transformations), leaving room
>>>>>    for implementation flexibility.
>>>>>    - Ryan mentioned that he is working on a v4 proposal for
>>>>>    generating Field IDs for non-materialized columns. The current thinking is
>>>>>    that index columns should follow the same approach.
>>>>>    - We agreed to retain older versions of the index metadata JSON.
>>>>>    - We agreed not to require a strict one-to-one mapping between
>>>>>    index snapshots and table snapshots. An index may contain multiple
>>>>>    snapshots corresponding to the same table snapshot.
>>>>>
>>>>> Based on the discussion, I have updated the spec PR. Following Ryan's
>>>>> pattern for non-materialized column definitions, the proposal now
>>>>> introduces:
>>>>>
>>>>>    - *materialized-fields* for range file columns
>>>>>    - *non-materialized-fields* for tracking file statistics
>>>>>    - *cluster-spec* to define which columns participate in range file
>>>>>    clustering
>>>>>
>>>>> Overall, I think we reached good alignment on the remaining design
>>>>> points, and I do not see any open questions at the moment. The PR is in
>>>>> good shape for review, so I would greatly appreciate any feedback or
>>>>> comments.
>>>>>
>>>>> Thanks everyone for the productive discussion and continued
>>>>> collaboration.
>>>>> Peter
>>>>>
>>>>> Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026.
>>>>> aug. 18., K, 14:17):
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> Unfortunately, yesterday's sync was not recorded due to a technical
>>>>>> issue. :(
>>>>>>
>>>>>> Here is a brief summary of the discussion and the decisions we
>>>>>> reached:
>>>>>>
>>>>>>    1. We agreed to use a list of Iceberg expressions to define the
>>>>>>    *index_keys*, which determine how the index is ordered.
>>>>>>    2. We agreed not to restrict the functions that can be used in
>>>>>>    those expressions. Engines that do not understand a particular expression
>>>>>>    can simply choose not to use the index.
>>>>>>    3. We agreed that index region files (leaf files) should be
>>>>>>    sorted by the values generated by the index_keys.
>>>>>>    4. We discussed using *included_column_ids* to define the
>>>>>>    contents and schema of the index region files, and generally felt this
>>>>>>    could be a good approach.
>>>>>>    5. We started discussing whether the results of the index_key
>>>>>>    expressions should also be stored in the index region files alongside the
>>>>>>    included columns. In some cases, storing them could help readers optimize
>>>>>>    lookups within a region file. In other cases, the values could be derived
>>>>>>    when needed, making them redundant and potentially hurting performance by
>>>>>>    increasing the size of the region files.
>>>>>>
>>>>>> I think we should continue the discussion around point 5, as the
>>>>>> outcome may influence, or even reopen, the decisions around point 4.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>>> Flavio Junqueira <fl...@gmail.com> ezt írta (időpont:
>>>>>> 2026. aug. 4., K, 18:57):
>>>>>>
>>>>>>> Thank you for the summary, the proposal and the overall discussion.
>>>>>>> I have one main question about the target and I apologize in advance if I'm
>>>>>>> missing some important detail as I might not have the full context.
>>>>>>>
>>>>>>> I noticed that partitioning, sorting and clustering (via Hilbert)
>>>>>>> were discussed in the call in this context of index transform functions. My
>>>>>>> understanding is that engines currently are already pruning partitions and
>>>>>>> files using partition information and table metadata like per-column
>>>>>>> min-max. If that's the case, then what's it that this proposal is trying to
>>>>>>> achieve by having transform functions that capture partitioning, sorting
>>>>>>> and clustering when engines are already able to prune without a secondary
>>>>>>> index? I can more easily see the primary key index example, but I'm not
>>>>>>> entirely sure about these data layout mechanisms.
>>>>>>>
>>>>>>> I also didn't quite get how engines are expected to consume such an
>>>>>>> index. There was a discussion about where the mapping of the input to files
>>>>>>> happens, whether that's in Iceberg or the engine, and I didn't quite get
>>>>>>> the recommendation. Checking Peter's proposal, I couldn't see it clearly
>>>>>>> either, and if I just missed it, I'd appreciate a pointer to where I should
>>>>>>> look into.
>>>>>>>
>>>>>>> -Flavio
>>>>>>>
>>>>>>> On 4 Aug 2026, at 10:52, Péter Váry <pe...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> Here is the recording from yesterday’s sync:
>>>>>>> https://www.youtube.com/watch?v=Wg9orP9JAmk
>>>>>>>
>>>>>>> To summarize:
>>>>>>> - We agreed to retain the history of index snapshots, but not the
>>>>>>> history of other index properties. If readers require information that may
>>>>>>> later be modified by a user, that information should be copied into the
>>>>>>> corresponding snapshot.
>>>>>>> - We discussed the possibility of defining index ordering using a
>>>>>>> list of transform functions.
>>>>>>>
>>>>>>> Dan and Yingyi suggested something along the following lines:
>>>>>>>
>>>>>>> *"index_keys" : [*
>>>>>>> *  {*
>>>>>>> *    "type": "apply",*
>>>>>>> *    "func-name": {*
>>>>>>> *      "catalog": "iceberg_functions",*
>>>>>>> *      "name": "day"*
>>>>>>> *    },*
>>>>>>> *    "arguments": [*
>>>>>>> *      { "type": "reference", "id": 3 }*
>>>>>>> *    ]*
>>>>>>> *  },*
>>>>>>> *  {*
>>>>>>> *    "type": "apply",*
>>>>>>> *    "func-name": {*
>>>>>>> *      "catalog": "iceberg_functions",*
>>>>>>> *      "name": "truncate"*
>>>>>>> *    },*
>>>>>>> *    "arguments": [*
>>>>>>> *      { "type": "reference", "id": 4 },*
>>>>>>> *      2*
>>>>>>> *    ]*
>>>>>>> *  }*
>>>>>>> *]*
>>>>>>>
>>>>>>>
>>>>>>> In this example, rows would first be ordered by day(col3) and then
>>>>>>> by truncate(col4, 2). To ensure a deterministic ordering, ties would be
>>>>>>> broken using the original values of columns 3 and 4. If those are still
>>>>>>> equal, ordering would fall back to the file name and finally the row
>>>>>>> position.
>>>>>>>
>>>>>>> I spent some time thinking about this after the sync, and here are
>>>>>>> my thoughts.
>>>>>>>
>>>>>>> Pros:
>>>>>>> - Provides a high degree of flexibility.
>>>>>>> - Enables expression-based indexes.
>>>>>>>
>>>>>>> Cons:
>>>>>>> - Do we need this flexibility at this stage?
>>>>>>>    - Equality deletes, the primary use cases we are targeting, do
>>>>>>> not require expressions.
>>>>>>>    - Most indexes I have seen in production are not expression-based.
>>>>>>> - Increased complexity for engines:
>>>>>>>    - Engines must parse and understand transforms to determine
>>>>>>> whether using an index can improve a query. For example, is ordering by
>>>>>>> day(timestamp) sufficient for a key lookup?
>>>>>>>    - Complexity increases further as new UDFs are introduced.
>>>>>>>    - Expressions are not pushed into scans today, so index handling
>>>>>>> cannot be fully hidden behind Iceberg scan planning.
>>>>>>> - Risk of fragmentation:
>>>>>>>    - The additional complexity may slow engine adoption.
>>>>>>>    - Adopters may choose to support only exact transform matches
>>>>>>> created by them, rather than learn to reason about the general transforms.
>>>>>>>    - UDFs are prone to missing implementations and behavioral
>>>>>>> differences across engines. In the worst case, such differences could lead
>>>>>>> to incorrect query results when indexes are used.
>>>>>>>
>>>>>>> Given the points above, I still prefer defining HASH, HILBERT,
>>>>>>> IDENTITY as a set of standardized values combined with an explicit list of
>>>>>>> key columns. This keeps adoption simple, unlocks the primary use cases we
>>>>>>> are targeting today, and allows the Iceberg library to provide the
>>>>>>> transform implementations directly.
>>>>>>>
>>>>>>> Internally, these transforms could still be represented as Iceberg
>>>>>>> expressions, giving us a migration path toward future index types that may
>>>>>>> require full expression support.
>>>>>>>
>>>>>>> For example:
>>>>>>>
>>>>>>> *{*
>>>>>>> *  "transform": "HILBERT",*
>>>>>>> *  "key-column-ids": [3, 4]*
>>>>>>> *}*
>>>>>>>
>>>>>>>
>>>>>>> I'm interested in hearing other opinions.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Peter
>>>>>>>
>>>>>>> Renjie Liu <li...@gmail.com> ezt írta (időpont: 2026. júl.
>>>>>>> 28., K, 10:07):
>>>>>>>
>>>>>>>> Hi, Peter:
>>>>>>>>
>>>>>>>> Thanks for driving the proposal.
>>>>>>>>
>>>>>>>> >  *Hash Index Type*: Should hash-based indexes be a separate
>>>>>>>> index type? They are only applicable to equality and IN predicates, and the
>>>>>>>> hash algorithm must always be known. A dedicated HASH type could make these
>>>>>>>> constraints explicit and distinguish hash-based indexes from ordered
>>>>>>>> indexes.
>>>>>>>>
>>>>>>>> I'm leaning toward keeping them in scalar index type due to the
>>>>>>>> similarity with others transformation functions like identity. As with the
>>>>>>>> constraints, I think they are properties of transformation functions.
>>>>>>>>
>>>>>>>> > *Transform Function/Layout Representation*: Should transform
>>>>>>>> functions be represented as an enum or a string? I lean toward strings, but
>>>>>>>> I do not have a strong preference. In many cases, an index can still be
>>>>>>>> used based on standard column statistics, even if a reader does not
>>>>>>>> recognize a particular transform.
>>>>>>>>
>>>>>>>> This seems more like a problem in java implementation?
>>>>>>>>
>>>>>>>> Also I prefer the name transform function compared with layout
>>>>>>>> representation. Layout representation makes me feel that it's not about the
>>>>>>>> definition of an index function.
>>>>>>>>
>>>>>>>> > *Future Index Types*: How should we handle index types that are
>>>>>>>> expected in the long term but fall outside the scope of this proposal (for
>>>>>>>> example, VECTOR, IVF, TERM)? Should they be mentioned as planned/reserved
>>>>>>>> concepts, or omitted entirely? I lean toward at least mentioning them. The
>>>>>>>> rationale for introducing an index type is easier to explain when
>>>>>>>> considering future index families.
>>>>>>>>
>>>>>>>> +1
>>>>>>>>
>>>>>>>> *> Index History Handling:* Most database systems treat index
>>>>>>>> changes as a drop-and-recreate operation rather than an in-place update. If
>>>>>>>> we follow the same approach, there is limited value in maintaining index
>>>>>>>> metadata history. During index maintenance, a new metadata.json can be
>>>>>>>> created and atomically swapped in, with the catalog responsible for
>>>>>>>> cleaning up obsolete metadata and index data files according to its
>>>>>>>> retention policy.
>>>>>>>>
>>>>>>>> +1, I think we only need to keep index data for different
>>>>>>>> snapshots, but if an index definition changed, we should drop and recreate
>>>>>>>> a new index.
>>>>>>>>
>>>>>>>> > *File Format Location*: Should file format be specified at the
>>>>>>>> leaf-file level or only at the index level? My preference is to stay
>>>>>>>> aligned with the table metadata model and avoid introducing differences
>>>>>>>> unless there is a compelling requirement. This would simplify
>>>>>>>> implementation and maximize code reuse.
>>>>>>>>
>>>>>>>> +1.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jul 24, 2026 at 11:57 PM Péter Váry <
>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi everyone,
>>>>>>>>>
>>>>>>>>> Here are the key takeaways from Monday’s meeting (July 20):
>>>>>>>>>
>>>>>>>>>    - *Index Type*: An index type enum will be introduced in v1,
>>>>>>>>>    initially with a single value (SCALAR). This establishes a future-proof API
>>>>>>>>>    boundary while leaving room for additional index families.
>>>>>>>>>    - *Strict Total Ordering*: The specification should require
>>>>>>>>>    transform functions to produce values that preserve a well-defined total
>>>>>>>>>    ordering across the key space. The transform function does not need to
>>>>>>>>>    return a long. Instead, leaf files can store the first and last values
>>>>>>>>>    according to the ordering, enabling generic filtering capabilities for
>>>>>>>>>    readers.
>>>>>>>>>
>>>>>>>>> Open Questions:
>>>>>>>>>
>>>>>>>>>    - *Hash Index Type*: Should hash-based indexes be a separate
>>>>>>>>>    index type? They are only applicable to equality and IN predicates, and the
>>>>>>>>>    hash algorithm must always be known. A dedicated HASH type could make these
>>>>>>>>>    constraints explicit and distinguish hash-based indexes from ordered
>>>>>>>>>    indexes.
>>>>>>>>>    - *Transform Function/Layout Representation*: Should transform
>>>>>>>>>    functions be represented as an enum or a string? I lean toward strings, but
>>>>>>>>>    I do not have a strong preference. In many cases, an index can still be
>>>>>>>>>    used based on standard column statistics, even if a reader does not
>>>>>>>>>    recognize a particular transform.
>>>>>>>>>    - *Future Index Types*: How should we handle index types that
>>>>>>>>>    are expected in the long term but fall outside the scope of this proposal
>>>>>>>>>    (for example, VECTOR, IVF, TERM)? Should they be mentioned as
>>>>>>>>>    planned/reserved concepts, or omitted entirely? I lean toward at least
>>>>>>>>>    mentioning them. The rationale for introducing an index type is easier to
>>>>>>>>>    explain when considering future index families.
>>>>>>>>>    - *Index History Handling:* Most database systems treat index
>>>>>>>>>    changes as a drop-and-recreate operation rather than an in-place update. If
>>>>>>>>>    we follow the same approach, there is limited value in maintaining index
>>>>>>>>>    metadata history. During index maintenance, a new metadata.json can be
>>>>>>>>>    created and atomically swapped in, with the catalog responsible for
>>>>>>>>>    cleaning up obsolete metadata and index data files according to its
>>>>>>>>>    retention policy.
>>>>>>>>>    - *Min/Max Statistics Representation*: Should
>>>>>>>>>    transform-specific or sort-order min/max values be represented using the
>>>>>>>>>    existing file statistics structure, or exposed through dedicated fields? My
>>>>>>>>>    preference is to keep them within the statistics structure defined by the
>>>>>>>>>    file schema while exposing them directly through the Java API if needed.
>>>>>>>>>    - *File Format Location*: Should file format be specified at
>>>>>>>>>    the leaf-file level or only at the index level? My preference is to stay
>>>>>>>>>    aligned with the table metadata model and avoid introducing differences
>>>>>>>>>    unless there is a compelling requirement. This would simplify
>>>>>>>>>    implementation and maximize code reuse.
>>>>>>>>>
>>>>>>>>> I have updated the spec PR based on the decisions above:
>>>>>>>>> https://github.com/apache/iceberg/pull/16961
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Peter
>>>>>>>>>
>>>>>>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont: 2026. jún.
>>>>>>>>> 26., P, 19:42):
>>>>>>>>>
>>>>>>>>>> Hi all, Here is the summary of this Monday's index meeting:
>>>>>>>>>>
>>>>>>>>>> Two blocking decisions closed:
>>>>>>>>>>
>>>>>>>>>>    1. Index is *not* a table— it's its own object (reuses table
>>>>>>>>>>    machinery under the hood). Reasoning: requires a sort order, no column
>>>>>>>>>>    updates, no partition spec, no overlapping ranges between leaves, inherits
>>>>>>>>>>    base-table permissions, and has its own CREATE/DROP/UPDATE INDEX DDL.
>>>>>>>>>>    2. Index is a separate catalog entity, with no pointers in
>>>>>>>>>>    table metadata— has its own REST endpoints; the catalog can optionally
>>>>>>>>>>    return index metadata withloadTable to avoid extra round-trips. Keeps table
>>>>>>>>>>    and index updates independent/async.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Next steps: Start writing the spec and build out the
>>>>>>>>>> copy-on-write path now.
>>>>>>>>>>
>>>>>>>>>> Here are the draft spec:
>>>>>>>>>> secondary index spec
>>>>>>>>>> <https://github.com/apache/iceberg/pull/16961>
>>>>>>>>>> irc spec <https://github.com/apache/iceberg/pull/16963>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Huaxin
>>>>>>>>>>
>>>>>>>>>> On Sat, Jun 20, 2026 at 11:11 AM huaxin gao <
>>>>>>>>>> huaxin.gao11@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> I built a standalone PoC to validate that the basic index
>>>>>>>>>>> structure works: that we can build a PK index, convert equality deletes to
>>>>>>>>>>> position deletes through it, and have every converted delete land on the
>>>>>>>>>>> correct live row. I ran it up to *100M keys*.
>>>>>>>>>>>
>>>>>>>>>>> *Headline: the structure works.* The index builds over up to
>>>>>>>>>>> 100M keys, the eq-delete → position-delete conversion resolved correctly at
>>>>>>>>>>> *every* size (100% of converted deletes mapped to the right
>>>>>>>>>>> live row), and the resulting position deletes are *~8× cheaper
>>>>>>>>>>> to apply* at query time than the equality deletes they replace.
>>>>>>>>>>>
>>>>>>>>>>> Beyond correctness, the run also shows how the index’s
>>>>>>>>>>> *maintenance* cost scales, comparing copy-on-write (COW,
>>>>>>>>>>> rewrite touched leaves) vs an append/merge (MOR) option, under a realistic
>>>>>>>>>>> mixed CDC checkpoint (1,000 insert + 500 update + 500 delete), local
>>>>>>>>>>> wall-clock:
>>>>>>>>>>> keys EQ baseline INDEX (COW) % of 60s (COW) INDEX (MOR) % of
>>>>>>>>>>> 60s (MOR) correct
>>>>>>>>>>> 5M 6 ms 6.7s 11.2% 2.2s 3.7% PASS
>>>>>>>>>>> 20M 8 ms 24.2s 40.4% 6.4s 10.6% PASS
>>>>>>>>>>> 50M 7 ms 51.6s 86.1% 12.2s 20.4% PASS
>>>>>>>>>>> *100M* 6 ms *75.0s* 125% (BEHIND) *16.9s* 28.2% (keeps up) PASS
>>>>>>>>>>>
>>>>>>>>>>> COW maintenance crosses the 60 s checkpoint around 100M
>>>>>>>>>>> (75 s/cycle, 125%); MOR stays at ~28% and keeps pace; the equality-delete
>>>>>>>>>>> baseline is ~6 ms and flat. So the structure works, but *COW
>>>>>>>>>>> alone can’t sustain scattered CDC at hundreds of millions of keys on a
>>>>>>>>>>> single writer*. It’s worth allowing a merge-on-read /
>>>>>>>>>>> update-file maintenance option alongside COW (or sharding the index across
>>>>>>>>>>> parallel writers).
>>>>>>>>>>>
>>>>>>>>>>> *Full write-up, all tables, and the in-region reality-check:*
>>>>>>>>>>> link
>>>>>>>>>>> <https://docs.google.com/document/d/1G3zxbW8X0eU3UrouslZfp42bBc9CvgJGnJyDONCB4PU/edit?tab=t.0>
>>>>>>>>>>>
>>>>>>>>>>> Feedback welcome, especially on the spec direction (whether to
>>>>>>>>>>> allow a merge-on-read / update-file maintenance option alongside COW)  and
>>>>>>>>>>> on the read-side modeling.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Huaxin
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Jun 9, 2026 at 5:45 PM huaxin gao <
>>>>>>>>>>> huaxin.gao11@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Sorry,  we've skipped posting a few of the dedicated index-sync
>>>>>>>>>>>> summaries to the mailing list; you can find those in the Google
>>>>>>>>>>>> doc
>>>>>>>>>>>> <https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?pli=1&tab=t.8041k7j2n7y3>
>>>>>>>>>>>> and the Slack channel. Here's yesterday's summary:
>>>>>>>>>>>>
>>>>>>>>>>>> *Decided*
>>>>>>>>>>>>
>>>>>>>>>>>>    -
>>>>>>>>>>>>
>>>>>>>>>>>>    Index vs. table (what we agreed):
>>>>>>>>>>>>    - Reuse table implementation/library code and a
>>>>>>>>>>>>       near-identical spec — the commit path will be custom regardless, so reuse
>>>>>>>>>>>>       isn't the deciding factor.
>>>>>>>>>>>>       - An index is not a table from a user/API view: loading
>>>>>>>>>>>>       or writing an index as a table must fail(it would violate index invariants).
>>>>>>>>>>>>       - The spec forbids most table behaviors: no overlapping
>>>>>>>>>>>>       files, one mandatory transform sort order, no column updates, no partition
>>>>>>>>>>>>       spec.
>>>>>>>>>>>>       - Delete vectors: reuse Iceberg's existing DV —
>>>>>>>>>>>>    benchmarks showed no new delete format is worth introducing.
>>>>>>>>>>>>    - Incremental updates: start with copy-on-write only (no
>>>>>>>>>>>>    update files). For object-store-sized leaves, a full leaf rewrite is about
>>>>>>>>>>>>    as cheap as maintaining an overlay update file + DV, so we'll skip the MOR
>>>>>>>>>>>>    machinery for now and add it later only if benchmarks prove we need it
>>>>>>>>>>>>    (likely just the very-large-leaf case).
>>>>>>>>>>>>    -
>>>>>>>>>>>>
>>>>>>>>>>>>    Validate the spec first: build a quick, hand-wired
>>>>>>>>>>>>    prototype (Parquet files structured per the spec) and benchmark it on real
>>>>>>>>>>>>    scales before formalizing.
>>>>>>>>>>>>
>>>>>>>>>>>> *Leaning, not final*
>>>>>>>>>>>>
>>>>>>>>>>>>    -
>>>>>>>>>>>>
>>>>>>>>>>>>    Indexes are likely separate catalog objects, linked from
>>>>>>>>>>>>    the table by storing just an identifier (like materialized views) and not
>>>>>>>>>>>>    visible in LIST TABLES.
>>>>>>>>>>>>    -
>>>>>>>>>>>>
>>>>>>>>>>>>    We'll need a commit path for indexes, but simpler than
>>>>>>>>>>>>    tables (no stage-create).
>>>>>>>>>>>>
>>>>>>>>>>>> *Still open*
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>    -
>>>>>>>>>>>>
>>>>>>>>>>>>    Permissions model — separate vs. inherited (action: look at
>>>>>>>>>>>>    what real DBs do for index permissions).
>>>>>>>>>>>>    - REST/catalog RPC design — minimize round-trips; index
>>>>>>>>>>>>    metadata ideally returned with LOAD TABLE. Catalog RPC cost
>>>>>>>>>>>>    may dominate Parquet IO, so this needs real design.
>>>>>>>>>>>>    -
>>>>>>>>>>>>
>>>>>>>>>>>>    Scale modeling — target rows-per-leaf vs. leaf size vs.
>>>>>>>>>>>>    metadata-file count.
>>>>>>>>>>>>    -
>>>>>>>>>>>>
>>>>>>>>>>>>    DDL-on-index semantics (reuse table schema-update actions
>>>>>>>>>>>>    or separate)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Huaxin
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Apr 22, 2026 at 8:47 AM Péter Váry <
>>>>>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>
>>>>>>>>>>>>> TL;DR
>>>>>>>>>>>>> We still need to validate with ADLS and S3, but based on the
>>>>>>>>>>>>> local tests, the MPHF approach looks more promising if we can tolerate
>>>>>>>>>>>>> larger files and longer index maintenance times.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Details:
>>>>>>>>>>>>> Here are the results from the local experiments on my Mac. I
>>>>>>>>>>>>> removed unnecessary statistics from the Parquet files and tested different
>>>>>>>>>>>>> row group sizes:
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - For an index file with 1M records, a row group size of
>>>>>>>>>>>>>    5,000 appears to be the sweet spot.
>>>>>>>>>>>>>    - For 10M records, 10,000 rows per row group works best.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If you have additional ideas for optimizing Parquet-based
>>>>>>>>>>>>> indexes, I’d be very interested to hear them.
>>>>>>>>>>>>> The test code is available on this branch:
>>>>>>>>>>>>> https://github.com/pvary/iceberg/tree/leaf_bench
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best results:
>>>>>>>>>>>>> *1m records/file*
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - Parquet - 5000 row/RowGroup
>>>>>>>>>>>>>       - Read: 1191 µs - 1 file open, 3 seek, 123KB read per
>>>>>>>>>>>>>       lookup
>>>>>>>>>>>>>       - Write: 1.7 s, 15 MB
>>>>>>>>>>>>>    - MPHF
>>>>>>>>>>>>>       - Read: 202 µs - 1 file open, 1 seek,  282KB read per
>>>>>>>>>>>>>       lookup
>>>>>>>>>>>>>       - Write: 0.8 s, 34 MB
>>>>>>>>>>>>>
>>>>>>>>>>>>> *10m records/file*
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - Parquet - 10000 row/RowGroup
>>>>>>>>>>>>>       - Read: 4168 µs - 1 file open, 3 seek, 395KB read per
>>>>>>>>>>>>>       lookup
>>>>>>>>>>>>>       - Write: 19.5s s, 144 MB
>>>>>>>>>>>>>    - MPHF
>>>>>>>>>>>>>       - Read: 1086 µs - 1 file open, 1 seek,  2.8 MB (2812KB)
>>>>>>>>>>>>>       read per lookup
>>>>>>>>>>>>>       - Write: 6.5 s, 34 MB, 353 MB
>>>>>>>>>>>>>
>>>>>>>>>>>>> Below are the full results.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Benchmark                                      (indexType)
>>>>>>>>>>>>>  (keyType)  (numRows)  Mode    Cnt            Score          Error
>>>>>>>>>>>>>  UnitsInvertedIndexBenchmark.lookup                 PARQUET_1000       LONG
>>>>>>>>>>>>>    1000000    ss  10000         3285.284 ±        5.138
>>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_1000       LONG
>>>>>>>>>>>>>    1000000    ss  10000   2522168989.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_1000       LONG
>>>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_1000       LONG
>>>>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_1000       LONG
>>>>>>>>>>>>> 10000000    ss  10000        35449.614 ±       34.673
>>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_1000       LONG
>>>>>>>>>>>>>   10000000    ss  10000  24302649201.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_1000       LONG
>>>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_1000       LONG
>>>>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_5000       LONG
>>>>>>>>>>>>>  1000000    ss  10000         1191.959 ±        4.169
>>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_5000       LONG
>>>>>>>>>>>>>    1000000    ss  10000   1230877229.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_5000       LONG
>>>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_5000       LONG
>>>>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_5000       LONG
>>>>>>>>>>>>> 10000000    ss  10000         7236.447 ±       10.374
>>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_5000       LONG
>>>>>>>>>>>>>   10000000    ss  10000   5650715973.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_5000       LONG
>>>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_5000       LONG
>>>>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_10000       LONG
>>>>>>>>>>>>>  1000000    ss  10000         1349.946 ±        7.834
>>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_10000       LONG
>>>>>>>>>>>>>    1000000    ss  10000   1730219377.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_10000       LONG
>>>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_10000       LONG
>>>>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_10000       LONG
>>>>>>>>>>>>> 10000000    ss  10000         4168.635 ±       11.051
>>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_10000       LONG
>>>>>>>>>>>>>   10000000    ss  10000   3946341532.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_10000       LONG
>>>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_10000       LONG
>>>>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_50000       LONG
>>>>>>>>>>>>>  1000000    ss  10000         4736.466 ±       38.179
>>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_50000       LONG
>>>>>>>>>>>>>    1000000    ss  10000   7427413541.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_50000       LONG
>>>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_50000       LONG
>>>>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_50000       LONG
>>>>>>>>>>>>> 10000000    ss  10000         4979.031 ±       34.708
>>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_50000       LONG
>>>>>>>>>>>>>   10000000    ss  10000   7694887636.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_50000       LONG
>>>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_50000       LONG
>>>>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                         MPHF       LONG
>>>>>>>>>>>>>  1000000    ss  10000          202.571 ±        2.336
>>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead               MPHF       LONG
>>>>>>>>>>>>>    1000000    ss  10000   2821570000.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams             MPHF       LONG
>>>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks                   MPHF       LONG
>>>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                         MPHF       LONG
>>>>>>>>>>>>> 10000000    ss  10000         1086.957 ±        4.524
>>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead               MPHF       LONG
>>>>>>>>>>>>>   10000000    ss  10000  28119460000.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams             MPHF       LONG
>>>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks                   MPHF       LONG
>>>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_1000       LONG
>>>>>>>>>>>>>  1000000    ss      3      1720731.014 ±   876636.004
>>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_1000       LONG
>>>>>>>>>>>>>    1000000    ss      3     46453317.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_1000       LONG
>>>>>>>>>>>>> 10000000    ss      3     18547947.876 ± 12258125.307
>>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_1000       LONG
>>>>>>>>>>>>>   10000000    ss      3    452655675.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_5000       LONG
>>>>>>>>>>>>>  1000000    ss      3      1718345.583 ±  1103928.016
>>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_5000       LONG
>>>>>>>>>>>>>    1000000    ss      3     44845788.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_5000       LONG
>>>>>>>>>>>>> 10000000    ss      3     18604229.931 ±  2668361.915
>>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_5000       LONG
>>>>>>>>>>>>>   10000000    ss      3    435388818.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_10000       LONG
>>>>>>>>>>>>>  1000000    ss      3      1761555.389 ±   535857.675
>>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_10000       LONG
>>>>>>>>>>>>>    1000000    ss      3     44536635.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_10000       LONG
>>>>>>>>>>>>> 10000000    ss      3     19501588.264 ±  2130054.558
>>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_10000       LONG
>>>>>>>>>>>>>   10000000    ss      3    433189623.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_50000       LONG
>>>>>>>>>>>>>  1000000    ss      3      1936624.889 ±  6601363.985
>>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_50000       LONG
>>>>>>>>>>>>>    1000000    ss      3     44264655.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_50000       LONG
>>>>>>>>>>>>> 10000000    ss      3     20471742.278 ± 10705206.310
>>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_50000       LONG
>>>>>>>>>>>>>   10000000    ss      3    431311305.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.write                          MPHF       LONG
>>>>>>>>>>>>>  1000000    ss      3       896573.958 ±  1408024.851
>>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes           MPHF       LONG
>>>>>>>>>>>>>    1000000    ss      3    102846369.000
>>>>>>>>>>>>> #InvertedIndexBenchmark.write                          MPHF       LONG
>>>>>>>>>>>>> 10000000    ss      3      6509348.875 ± 15519975.479
>>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes           MPHF       LONG
>>>>>>>>>>>>>   10000000    ss      3   1058435733.000                     #*
>>>>>>>>>>>>>
>>>>>>>>>>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont: 2026.
>>>>>>>>>>>>> ápr. 21., K, 20:53):
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In recent secondary index sync meetings, the discussion
>>>>>>>>>>>>>> converged on the need to define what an index is from first principles
>>>>>>>>>>>>>> before settling on physical layout.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To address that, Peter and I have drafted a requirements
>>>>>>>>>>>>>> document for a key lookup index (renamed from "primary key index" to avoid
>>>>>>>>>>>>>> implying uniqueness enforcement), the goal is to nail down one well-scoped
>>>>>>>>>>>>>> index type first.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Doc: Key Lookup Index Requirements
>>>>>>>>>>>>>> <https://docs.google.com/document/d/1e0zxK-jA0LBDq8YQlQgFipTHelDFiga8lCkgDTmYub8/edit?tab=t.0#heading=h.8shrgabvl19>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It covers requirements, three design options (manifest +
>>>>>>>>>>>>>> sorted Parquet, hash + sorted Parquet, hash + MPHF) and open questions. We
>>>>>>>>>>>>>> will add preliminary benchmark results shortly.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Feedback welcome — inline in the doc, on this thread, or at
>>>>>>>>>>>>>> the next index sync.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Huaxin
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Apr 13, 2026 at 7:22 AM Steven Wu <
>>>>>>>>>>>>>> stevenz3wu@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Do we need the special index identifier that was originally
>>>>>>>>>>>>>>> proposed? A generic CatalogObjectIdentifier (with namespace and name) would
>>>>>>>>>>>>>>> be consistent with all object types in the catalog. I have a discussion
>>>>>>>>>>>>>>> thread on the generic identifier topic: [DISCUSS] REST
>>>>>>>>>>>>>>> Spec: generic CatalogObjectIdentifier.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Should we add an indexes array field to table metadata? It
>>>>>>>>>>>>>>> only contains a list of index object identifiers. It doesn't contain any
>>>>>>>>>>>>>>> index metadata which should live in the index objects. Yufei was trying to
>>>>>>>>>>>>>>> bring this up at the end of the first sync. But we didn't get enough time
>>>>>>>>>>>>>>> to really discuss it. It will be great to discuss this as the first agenda
>>>>>>>>>>>>>>> item today.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Apr 13, 2026 at 3:17 AM Péter Váry <
>>>>>>>>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We had several engaging discussions at the Iceberg Summit,
>>>>>>>>>>>>>>>> and it was great to finally catch up with many of you in person. We truly
>>>>>>>>>>>>>>>> missed those who couldn’t attend, hopefully we’ll all meet again at the
>>>>>>>>>>>>>>>> next summit.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> To keep the conversation going, Huaxin and I have put
>>>>>>>>>>>>>>>> together the agenda for our next meeting. As a reminder, we’ll meet on *April
>>>>>>>>>>>>>>>> 13th, 9:00–10:00 AM *PDT (6:00–7:00 PM CEST).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Proposed agenda:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - Continue first-principles index design discussion
>>>>>>>>>>>>>>>>    from Mar 30
>>>>>>>>>>>>>>>>       - *Index Ownership and Write Responsibility*
>>>>>>>>>>>>>>>>          - Should writers be allowed to update indexes, or
>>>>>>>>>>>>>>>>          - Should all index writes be handled exclusively
>>>>>>>>>>>>>>>>          by the Index Maintenance process?
>>>>>>>>>>>>>>>>          - If writers can update indexes then we need to
>>>>>>>>>>>>>>>>          define what guarantees are required (compaction, file splitting, layout
>>>>>>>>>>>>>>>>          expectations)?
>>>>>>>>>>>>>>>>          - If only Index Maintenance updates indexes then
>>>>>>>>>>>>>>>>          we only need to define what observable properties should be exposed to
>>>>>>>>>>>>>>>>          consumers? Like:
>>>>>>>>>>>>>>>>             - Expected max files for a single key
>>>>>>>>>>>>>>>>             - Current max files for a single key
>>>>>>>>>>>>>>>>             - Deletes allowed/present
>>>>>>>>>>>>>>>>             - Sorted by
>>>>>>>>>>>>>>>>             - Partitioned by
>>>>>>>>>>>>>>>>          - *Specification Scope: What Belongs in the Spec?*
>>>>>>>>>>>>>>>>          - Related to the ownership question above
>>>>>>>>>>>>>>>>          - Light spec: Just define that the index table
>>>>>>>>>>>>>>>>          should be optimized for retrieval by key columns and the index columns
>>>>>>>>>>>>>>>>          should be contained in the table. This could give us more flexibility if
>>>>>>>>>>>>>>>>          better organization methods come up, or
>>>>>>>>>>>>>>>>          - Detailed spec: We could define the max number
>>>>>>>>>>>>>>>>          of files per index to read for a single key, or even the partitioning and
>>>>>>>>>>>>>>>>          the exact sort order. This could allow more use-cases for a given index,
>>>>>>>>>>>>>>>>          like joins or cardinality estimations.
>>>>>>>>>>>>>>>>          - I would go for light spec for the main types
>>>>>>>>>>>>>>>>          (PK, Containing) and only the Index Maintenance processes should update the
>>>>>>>>>>>>>>>>          Indexes, as for many use-cases the details are not important, and writers
>>>>>>>>>>>>>>>>          will very rarely update the Indexes themselves.
>>>>>>>>>>>>>>>>       - *Logical Placement of Indexes*
>>>>>>>>>>>>>>>>          - Index as a child object of an Iceberg Table, or
>>>>>>>>>>>>>>>>          - Index as a first‑class entity under
>>>>>>>>>>>>>>>>          /namespace/indexes/{index}
>>>>>>>>>>>>>>>>          - Based on the discussions on the summit we are
>>>>>>>>>>>>>>>>          leaning in this direction. This means the index id should be unique in the
>>>>>>>>>>>>>>>>          namespace but helps the catalog implementations quite a bit
>>>>>>>>>>>>>>>>       - *Physical Placement of Index Data*
>>>>>>>>>>>>>>>>          - I don’t think we should specify this. We should
>>>>>>>>>>>>>>>>          have a base location for the index, but can rely on the catalog
>>>>>>>>>>>>>>>>          implementations to decide on their own, like they do with the tables,
>>>>>>>>>>>>>>>>          views, udfs.
>>>>>>>>>>>>>>>>       - *Iceberg Reader Based indexes* (Containing indexes
>>>>>>>>>>>>>>>>       and potentially PK indexes). These are the indexes which could be read by
>>>>>>>>>>>>>>>>       the existing Iceberg readers. We might decide to store the PK index
>>>>>>>>>>>>>>>>       similarly to an Iceberg Table and treat it as a reader based index.
>>>>>>>>>>>>>>>>          - What are the table properties/features exposed
>>>>>>>>>>>>>>>>          to the readers
>>>>>>>>>>>>>>>>             - Maybe just some behavioral descriptors for
>>>>>>>>>>>>>>>>             the optimizer to decide if the index could be used or should be skipped,
>>>>>>>>>>>>>>>>             like:
>>>>>>>>>>>>>>>>                - Expected max files for a single key
>>>>>>>>>>>>>>>>                - max files for a single key
>>>>>>>>>>>>>>>>                - Deletes allowed/present
>>>>>>>>>>>>>>>>                - Sorted by
>>>>>>>>>>>>>>>>                - Partitioned by
>>>>>>>>>>>>>>>>             - The Tasks when reading the index based on
>>>>>>>>>>>>>>>>             the filters and projection
>>>>>>>>>>>>>>>>          - What are the table properties/features exposed
>>>>>>>>>>>>>>>>          to the Index Maintenance. I think this could be internal to the Index
>>>>>>>>>>>>>>>>          Maintenance process and might not be exposed through the spec. The Index
>>>>>>>>>>>>>>>>          Maintenance process could handle this as a standard Iceberg Table and could
>>>>>>>>>>>>>>>>          be based on the Table Maintenance process, but there might be some totally
>>>>>>>>>>>>>>>>          different processes.
>>>>>>>>>>>>>>>>       - It should be possible to add properties to an
>>>>>>>>>>>>>>>>       index defined by the Index Maintenance process which could be used and
>>>>>>>>>>>>>>>>       updated in the next Index Maintenance run.
>>>>>>>>>>>>>>>>    - *PK index storage format benchmark results*
>>>>>>>>>>>>>>>>       - Flat Parquet (baseline)
>>>>>>>>>>>>>>>>       - BTree with Parquet leaves
>>>>>>>>>>>>>>>>       - Vortex
>>>>>>>>>>>>>>>>    - *Open items / next steps*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Peter
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont:
>>>>>>>>>>>>>>>> 2026. márc. 23., H, 3:03):
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi everyone, I wanted to share an update on the primary
>>>>>>>>>>>>>>>>> key index work.
>>>>>>>>>>>>>>>>> Since there are still open questions on whether bloom
>>>>>>>>>>>>>>>>> filter indexes fit in the secondary index framework or should be treated as
>>>>>>>>>>>>>>>>> extended stats, I've shifted focus to the primary key index since it's a
>>>>>>>>>>>>>>>>> clearer fit for the framework.
>>>>>>>>>>>>>>>>> I've put together a proposal for a primary key
>>>>>>>>>>>>>>>>> reverse-lookup index that maps each key to its physical location
>>>>>>>>>>>>>>>>> (file_path, row_position). It enables:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - Scan-time file pruning for point lookups
>>>>>>>>>>>>>>>>>    - Converting key-based deletes into position deletes
>>>>>>>>>>>>>>>>>    (eliminating equality deletes for Flink CDC)
>>>>>>>>>>>>>>>>>    - Accelerating Spark MERGE INTO by replacing
>>>>>>>>>>>>>>>>>    full-table joins with direct file lookups
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Proposal:
>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1HuhCZ0n2FqDh8yqQb9oEj1CPM5yXpEsMPGZno2aSf8E/edit?tab=t.0#heading=h.tbevg4q0m9
>>>>>>>>>>>>>>>>> Feedback welcome!
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Huaxin
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Mar 18, 2026 at 11:42 PM Péter Váry <
>>>>>>>>>>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Key takeaways from the general index discussion at the
>>>>>>>>>>>>>>>>>> May 16 meeting.
>>>>>>>>>>>>>>>>>> Thanks to everyone who participated! The recording is
>>>>>>>>>>>>>>>>>> available here:
>>>>>>>>>>>>>>>>>> https://www.youtube.com/watch?v=btmjhtRWUCE
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    - Q: Do we need to tie index types to the algorithms
>>>>>>>>>>>>>>>>>>    used to access them?
>>>>>>>>>>>>>>>>>>    - A: From a specification perspective, the goal is to
>>>>>>>>>>>>>>>>>>    define the storage-level data layout so it can be shared across engines.
>>>>>>>>>>>>>>>>>>    Engines are free to interpret and use the data as they see fit, but the
>>>>>>>>>>>>>>>>>>    on-disk data layout itself must be strictly defined and interoperable.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    - Q: Should we introduce an additional abstraction
>>>>>>>>>>>>>>>>>>    layer (e.g., Vector Index) with sub-types such as IVF and DiskANN?
>>>>>>>>>>>>>>>>>>    - A: This is possible if we decide it is beneficial.
>>>>>>>>>>>>>>>>>>    I explored potential naming, but it is not yet clear how such a layer would
>>>>>>>>>>>>>>>>>>    be used in practice.
>>>>>>>>>>>>>>>>>>    *Question to Yingyi Bu*: could you provide examples
>>>>>>>>>>>>>>>>>>    where this additional layer would be useful? Should this abstraction be
>>>>>>>>>>>>>>>>>>    defined at the spec level, or is it better handled at the engine level?
>>>>>>>>>>>>>>>>>>    My initial idea was that users would create a generic
>>>>>>>>>>>>>>>>>>    Vector Index and let the engine choose the concrete implementation.
>>>>>>>>>>>>>>>>>>    However, this would limit user control and users likely need to specify the
>>>>>>>>>>>>>>>>>>    exact index representation, which implies they must be aware of the
>>>>>>>>>>>>>>>>>>    available representations.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    - Q: Do we want to allow extensibility for index
>>>>>>>>>>>>>>>>>>    types?
>>>>>>>>>>>>>>>>>>    - A: Yes. The intent is to support a small set of
>>>>>>>>>>>>>>>>>>    well-defined index types while allowing experimentation with new ones. If a
>>>>>>>>>>>>>>>>>>    new index type proves broadly useful, a follow-up proposal can standardize
>>>>>>>>>>>>>>>>>>    it and incorporate it into the spec.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    - Q: Do we allow multiple versions of an index for
>>>>>>>>>>>>>>>>>>    the same table snapshot?
>>>>>>>>>>>>>>>>>>    - A: Yes. Older index versions must be retained for
>>>>>>>>>>>>>>>>>>    readers that have already started using them, while new readers should
>>>>>>>>>>>>>>>>>>    automatically use the latest available version
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    - Q: Do we need to use materialized views for these
>>>>>>>>>>>>>>>>>>    indexes?
>>>>>>>>>>>>>>>>>>    - A: No. These indexes are primarily examples, and
>>>>>>>>>>>>>>>>>>    different types may require different storage methods. However, the Primary
>>>>>>>>>>>>>>>>>>    Key, Containing, and parts of the IVF indexes can be structured as Iceberg
>>>>>>>>>>>>>>>>>>    tables. This allows engines to read them natively; in some cases, Iceberg
>>>>>>>>>>>>>>>>>>    planners can automatically redirect queries to the index table without
>>>>>>>>>>>>>>>>>>    engine modifications. Furthermore, index maintenance for these tables can
>>>>>>>>>>>>>>>>>>    leverage existing materialized view maintenance workflows. Other index
>>>>>>>>>>>>>>>>>>    types may instead rely on Puffin files or alternative storage approaches.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    - Q: How should index metadata be accessed? Should we
>>>>>>>>>>>>>>>>>>    add explicit pointers for the indexes in the table metadata?
>>>>>>>>>>>>>>>>>>    - A: We did not have sufficient time to fully explore
>>>>>>>>>>>>>>>>>>    and conclude this topic.
>>>>>>>>>>>>>>>>>>    *Question for Yufei Gu*: Did I understand correctly
>>>>>>>>>>>>>>>>>>    that your main concern stems from endpoint resolution from a REST Catalog
>>>>>>>>>>>>>>>>>>    perspective? Specifically, if indexes are exposed under a URI such as
>>>>>>>>>>>>>>>>>>    v1/{prefix}/namespaces/{namespace}/tables/{table}/indexes/{index}, would
>>>>>>>>>>>>>>>>>>    this make it more difficult for the REST Catalog to resolve and route
>>>>>>>>>>>>>>>>>>    requests to the appropriate endpoint?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Suhas Jayaram Subramanya via dev <de...@iceberg.apache.org>
>>>>>>>>>>>>>>>>>> ezt írta (időpont: 2026. márc. 13., P, 23:32):
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Here's a proposal for native Vector Index support in
>>>>>>>>>>>>>>>>>>> Iceberg tables --
>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1KL4qLOwdqnhOcqTc0EjO1O16NV3M3c-gZCEINDWw4lA/edit?usp=sharing
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> We've been working on this proposal with Peter
>>>>>>>>>>>>>>>>>>> internally at Microsoft and he suggested we post it here to bring this to
>>>>>>>>>>>>>>>>>>> the community's attention, ahead of the next Secondary Index Sync.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Suhas
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 2026/02/19 04:34:34 huaxin gao wrote:
>>>>>>>>>>>>>>>>>>> > Hi Everyone,
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > Here are the recording and notes from the Iceberg
>>>>>>>>>>>>>>>>>>> Index Support Sync on
>>>>>>>>>>>>>>>>>>> > 2/11.
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > Recording: https://www.youtube.com/watch?v=3sFfQ0A50yk
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > Notes:
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.8041k7j2n7y3
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > The meeting will move to biweekly, Mondays 9–10am PST,
>>>>>>>>>>>>>>>>>>> starting March 2.
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > Since the sync, I updated the Bloom skipping index
>>>>>>>>>>>>>>>>>>> proposal
>>>>>>>>>>>>>>>>>>> > <
>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.5r5kl6k3fqwu
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > to address the discussion questions, specifically:
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > - Performance justification: when this helps
>>>>>>>>>>>>>>>>>>> (high-cardinality = / IN,
>>>>>>>>>>>>>>>>>>> > many data files, high object-store latency) and how it
>>>>>>>>>>>>>>>>>>> differs from Parquet
>>>>>>>>>>>>>>>>>>> > row-group Bloom filters (which still require opening
>>>>>>>>>>>>>>>>>>> the data file).
>>>>>>>>>>>>>>>>>>> > - Cost / scalability: rough sizing (Bloom blob size
>>>>>>>>>>>>>>>>>>> per file, Puffin
>>>>>>>>>>>>>>>>>>> > file size), the planning cost trade-off (driver index
>>>>>>>>>>>>>>>>>>> reads vs executor
>>>>>>>>>>>>>>>>>>> > file opens), and mitigations via caching.
>>>>>>>>>>>>>>>>>>> > - Lifecycle / maintenance: incremental production as
>>>>>>>>>>>>>>>>>>> new data files
>>>>>>>>>>>>>>>>>>> > arrive, behavior when the index is missing/behind, and
>>>>>>>>>>>>>>>>>>> sharding/compaction
>>>>>>>>>>>>>>>>>>> > plus cleanup to avoid accumulating too many small
>>>>>>>>>>>>>>>>>>> Puffin files over time.
>>>>>>>>>>>>>>>>>>> > - Writer expectations: inline (optional) vs
>>>>>>>>>>>>>>>>>>> asynchronous (primary) index
>>>>>>>>>>>>>>>>>>> > creation.
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > I also implemented a Spark 4.1 POC
>>>>>>>>>>>>>>>>>>> > <https://github.com/apache/iceberg/pull/15311> and a
>>>>>>>>>>>>>>>>>>> local benchmark to
>>>>>>>>>>>>>>>>>>> > quantify both the pruning impact (plannedFiles →
>>>>>>>>>>>>>>>>>>> afterBloom) and the index
>>>>>>>>>>>>>>>>>>> > read overhead (statsFiles, statsBytes,
>>>>>>>>>>>>>>>>>>> bloomPayloadBytes) for point
>>>>>>>>>>>>>>>>>>> > predicates on high-cardinality columns. Please take a
>>>>>>>>>>>>>>>>>>> look and let me know
>>>>>>>>>>>>>>>>>>> > if you have any questions or feedback.
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > Huaxin
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > On Tue, Feb 10, 2026 at 1:43 PM huaxin gao <
>>>>>>>>>>>>>>>>>>> hu...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > > Reminder for tomorrow's sync on Iceberg Index
>>>>>>>>>>>>>>>>>>> Support.
>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>> > > Wednesday: Feb. 11 9:00 – 10:00am
>>>>>>>>>>>>>>>>>>> > > Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>> > > Google Meet joining info
>>>>>>>>>>>>>>>>>>> > > Video call link: meet.google.com/nsp-ctyr-khk
>>>>>>>>>>>>>>>>>>> > > Design doc:
>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.0#heading=h.hs6r9d26w1y2
>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.qouk73o4jxx7
>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>> > > Thanks,
>>>>>>>>>>>>>>>>>>> > > Huaxin
>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>> > > On Tue, Feb 3, 2026 at 10:52 PM Péter Váry <
>>>>>>>>>>>>>>>>>>> pe...@gmail.com>
>>>>>>>>>>>>>>>>>>> > > wrote:
>>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>>> > >> Thanks Huaxin and Steven for organizing this.
>>>>>>>>>>>>>>>>>>> Looking forward to meet you
>>>>>>>>>>>>>>>>>>> > >> all next week!
>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>> > >> On Wed, Feb 4, 2026, 02:48 Steven Wu <
>>>>>>>>>>>>>>>>>>> st...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>>> > >>> We set up the dev calendar event with a new google
>>>>>>>>>>>>>>>>>>> meet link. Please
>>>>>>>>>>>>>>>>>>> > >>> ignore the link from Huaxin's original email.
>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>> > >>> The dev calendar has the correct info (including
>>>>>>>>>>>>>>>>>>> the new meeting link)
>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>> > >>> Iceberg Index Support Sync
>>>>>>>>>>>>>>>>>>> > >>> Wednesday, February 11 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>>>> > >>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>>> > >>> Google Meet joining info
>>>>>>>>>>>>>>>>>>> > >>> Video call link:
>>>>>>>>>>>>>>>>>>> https://meet.google.com/nsp-ctyr-khk
>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>> > >>> On Tue, Feb 3, 2026 at 5:08 PM huaxin gao <
>>>>>>>>>>>>>>>>>>> hu...@gmail.com>
>>>>>>>>>>>>>>>>>>> > >>> wrote:
>>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>>> > >>>> Sorry, I meant PST (not EST) :)
>>>>>>>>>>>>>>>>>>> > >>>> Looking forward to the discussion!
>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>> > >>>> On Tue, Feb 3, 2026 at 4:58 PM Shawn Chang <
>>>>>>>>>>>>>>>>>>> ye...@gmail.com>
>>>>>>>>>>>>>>>>>>> > >>>> wrote:
>>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>>> > >>>>> Hi Huaxin,
>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>> > >>>>> Thanks for starting the sync!
>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>> > >>>>> The meeting seems to be 9-10AM PST on the dev
>>>>>>>>>>>>>>>>>>> events calendar
>>>>>>>>>>>>>>>>>>> > >>>>> <
>>>>>>>>>>>>>>>>>>> https://calendar.google.com/calendar/u/0?cid=MzkwNWQ0OTJmMWI0NTBiYTA3MTJmMmFlNmFmYTc2ZWI3NTdmMTNkODUyMjBjYzAzYWE0NTI3ODg1YWRjNTYyOUBncm91cC5jYWxlbmRhci5nb29nbGUuY29t
>>>>>>>>>>>>>>>>>>> >,
>>>>>>>>>>>>>>>>>>> > >>>>> not EST. Maybe it's a typo?
>>>>>>>>>>>>>>>>>>> > >>>>> Otherwise, looking forward to the discussion!
>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>> > >>>>> Best,
>>>>>>>>>>>>>>>>>>> > >>>>> Shawn
>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>> > >>>>> On Tue, Feb 3, 2026 at 9:18 AM huaxin gao <
>>>>>>>>>>>>>>>>>>> hu...@gmail.com>
>>>>>>>>>>>>>>>>>>> > >>>>> wrote:
>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>> Hi all,
>>>>>>>>>>>>>>>>>>> > >>>>>> I'd like to start a dedicated sync to discuss
>>>>>>>>>>>>>>>>>>> Iceberg Index support.
>>>>>>>>>>>>>>>>>>> > >>>>>> Here is the existing discussion thread:
>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>> https://lists.apache.org/thread/fzqk3jjf0xpj5m4cfqb3v4c65p0t04ty
>>>>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>> To ground the discussion, here are the two
>>>>>>>>>>>>>>>>>>> proposals:
>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>> - Peter's proposal
>>>>>>>>>>>>>>>>>>> > >>>>>> <
>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.0#heading=h.hs6r9d26w1y2>
>>>>>>>>>>>>>>>>>>> (overall
>>>>>>>>>>>>>>>>>>> > >>>>>> index support)
>>>>>>>>>>>>>>>>>>> > >>>>>> - My proposal
>>>>>>>>>>>>>>>>>>> > >>>>>> <
>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.qouk73o4jxx7
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > >>>>>> (bloom filter skipping index)
>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>> Time slot: Every 3 weeks, Wednesdays at 9 AM to
>>>>>>>>>>>>>>>>>>> 10 AM EST, starting
>>>>>>>>>>>>>>>>>>> > >>>>>> next Wednesday (2/11). After FileFormat sync
>>>>>>>>>>>>>>>>>>> finishes, we plan to use that
>>>>>>>>>>>>>>>>>>> > >>>>>> slot and switch to every other Monday, 9 AM to
>>>>>>>>>>>>>>>>>>> 10 AM EST.
>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>> Meet link: https://meet.google.com/fjn-tyze-mko
>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> > >>>>>> Huaxin
>>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>

Re: Dedicated sync for Iceberg Index Support

[huaxin gao <hu...@gmail.com>]

Hi everyone,

Following up on my action item from Monday's sync: here is what I found on
how databases, catalogs, and REST APIs model, list, and name indexes.

1. Where an index lives

   - Iceberg REST spec: resources that belong to one table are already
   nested under it, for example .../tables/{table}/plan, /credentials, and
   /metrics.
   - Catalog and REST APIs scope indexes to the table. None of them makes
   an index a top-level namespace object.
      - Lance (/v1/table/{id}/index/{index_name}) and Firestore put the
      index under the table in the URL path.
      - Glue (DatabaseName + TableName + IndexName) and Hive Metastore
      (get_indexes(db, tbl)) require the table to address an index.
   - SQL databases are split.
      - MySQL, SQL Server, Snowflake, and BigQuery make index names unique
      per table, so you name the table to address an index (for example DROP
      INDEX idx ON tbl in MySQL).
      - Postgres, Oracle, and SQLite make an index a schema object (DROP
      INDEX idx). Even Postgres always creates the index in its table's schema.

2. Listing across tables
Most SQL databases also list indexes across tables, separately from where
the index lives. For example, Snowflake has SHOW INDEXES IN SCHEMA /
DATABASE / ACCOUNT, and MySQL and Postgres have system views
(information_schema.STATISTICS and pg_indexes) that return every index in a
schema in one query. This covers the maintenance use case: find every index
in a namespace without going table by table.

3. Rename
MySQL, SQL Server, Postgres, and Oracle support index rename, mainly for
naming cleanup and schema migration tools. Glue, Lance, Firestore, and
BigQuery don't. That supports leaving it out of the first version.

Proposal:

   - Nest index routes under the table:
   /v1/{prefix}/namespaces/{namespace}/tables/{table}/indexes[/{index}]
   - Make index names unique per table. The index identifier then includes
   its table identifier plus the index name.
   - Add a second, optional endpoint that lists all indexes in a namespace,
   for maintenance: GET /v1/{prefix}/namespaces/{namespace}/indexes. For each
   index, it returns the index name and the table it belongs to. Catalogs
   don't have to support it.

Feedback welcome.

Thanks,
Huaxin

On Tue, Oct 27, 2026 at 10:00 AM huaxin gao <hu...@gmail.com> wrote:

> Hi everyone,
>
> Here is a summary of yesterday's index sync. Thanks to everyone who joined.
>
> 1. Index spec (#16961 <https://github.com/apache/iceberg/pull/16961>)
> We went over the two questions Péter listed:
> - Region file boundary: the per-column group max value is replaced by a
> top-level max_ordering_key in each tracking file entry.
> - Region file schema: the index now stores its own schema, with a mapping
> from table fields to index fields, so index field IDs are independent of
> the table's. Computed values are derived columns, either stored in the
> region files (like a SQL generated column) or kept only as stats. This
> depends on the derived columns proposal for V4, and the index keeps its
> schema history (for example after int to long promotion).
> There are no open issues left on the spec PR. Please review:
> https://github.com/apache/iceberg/pull/16961
>
> 2. Index REST spec
> Agreed:
> - An index must be in the same namespace as its table. Cross-namespace
> indexes are dropped.
> - Rename and register are left out of the first version, to reduce
> complexity. We can add them later if there is demand.
>
> Main open question: should index routes be namespace-level (as in the PR),
> or nested under the table?
> Prior art differs: MySQL scopes indexes to a table (DROP INDEX idx ON
> tbl), while Postgres makes index names unique within a schema (DROP INDEX
> idx). Lance nests indexes under the table.
> Next step: look at how major databases and REST APIs model indexes, then
> revisit the route design at the next sync.
>
> Action items
> - Everyone: review the index spec PR #16961
> <https://github.com/apache/iceberg/pull/16961>.
> - Huaxin: research how MySQL, Postgres, and other databases and REST APIs
> model, list, and name indexes, and whether rename has real use cases.
>
> Recording and notes:
> https://drive.google.com/drive/folders/1iodHASJBVwNtDYEINq7k2HjogPsYQocH
>
> Thanks,
> Huaxin
>
> On Tue, Sep 15, 2026 at 7:56 AM Péter Váry <pe...@gmail.com>
> wrote:
>
>> Hi everyone,
>>
>> Here is the recording of the Monday sync:
>> https://drive.google.com/drive/folders/1iodHASJBVwNtDYEINq7k2HjogPsYQocH
>>
>> A brief summary of the discussion:
>>
>>    - Bloom Filter Index
>>       - Puffin reduces file-open overhead, while Bloom filter
>>       effectiveness appears independent of file placement.
>>       - The observed scaling issues are likely due to rate limiting.
>>       - Manifest alignment would be required for parallel planning.
>>       - A scalar index is expected to deliver similar or better
>>       performance.
>>       - We will *prioritize scalar indexes* before pursuing additional
>>       index types.
>>       - The tests are still useful to understand the decisions in the
>>       design stage.
>>    - Scalar Index Design
>>       - *Region file schema: identity, materialized, and
>>       non-materialized fields*.
>>       - Range files will use stable index-specific IDs, while
>>       expressions reference table field IDs.
>>       - Scalar indexes require both sorting and clustering; a dedicated
>>       index definition may be cleaner than reusing table sort/cluster definitions.
>>
>> We can continue the remaining discussions offline or in the next sync.
>>
>> There are only a few open questions, so the spec PR is ready for review:
>> https://github.com/apache/iceberg/pull/16961.
>>
>> Thanks,
>> Peter
>>
>>
>> Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026. szept.
>> 14., H, 9:07):
>>
>>> One correction to the previous summary: I originally wrote that *Range*
>>> files would store the index data, but I was later corrected that the
>>> decision was actually to use *Region* files.
>>>
>>> The correct index file naming is:
>>>
>>>    - *Tracking file*: contains the list of region files (analogous to a
>>>    manifest file for tables).
>>>    - *Region file*: contains the index data rows (analogous to a data
>>>    file for tables).
>>>
>>> Sorry for the confusion,
>>> Peter
>>>
>>> Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026.
>>> szept. 3., Cs, 13:18):
>>>
>>>>
>>>> Hi everyone,
>>>>
>>>> Here is the recording of the Monday sync:
>>>>
>>>> https://apache-iceberg.slack.com/files/U06538UFDH6/F0BTT739Y7P/impromptu_google_meet_meeting_-_aug_31_2026.mp4
>>>>
>>>> Thanks, Shawn, for making it available.
>>>>
>>>> A brief summary of the discussion:
>>>>
>>>>    - We finalized the file naming:
>>>>       - Tracking file: contains the list of range files (similar to a
>>>>       manifest file for tables).
>>>>       - Range file: contains the index data rows (similar to a data
>>>>       file for tables).
>>>>    - We agreed not to prescribe how expression values are materialized
>>>>    (for example, Hilbert or hash transformations), leaving room for
>>>>    implementation flexibility.
>>>>    - Ryan mentioned that he is working on a v4 proposal for generating
>>>>    Field IDs for non-materialized columns. The current thinking is that index
>>>>    columns should follow the same approach.
>>>>    - We agreed to retain older versions of the index metadata JSON.
>>>>    - We agreed not to require a strict one-to-one mapping between
>>>>    index snapshots and table snapshots. An index may contain multiple
>>>>    snapshots corresponding to the same table snapshot.
>>>>
>>>> Based on the discussion, I have updated the spec PR. Following Ryan's
>>>> pattern for non-materialized column definitions, the proposal now
>>>> introduces:
>>>>
>>>>    - *materialized-fields* for range file columns
>>>>    - *non-materialized-fields* for tracking file statistics
>>>>    - *cluster-spec* to define which columns participate in range file
>>>>    clustering
>>>>
>>>> Overall, I think we reached good alignment on the remaining design
>>>> points, and I do not see any open questions at the moment. The PR is in
>>>> good shape for review, so I would greatly appreciate any feedback or
>>>> comments.
>>>>
>>>> Thanks everyone for the productive discussion and continued
>>>> collaboration.
>>>> Peter
>>>>
>>>> Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026. aug.
>>>> 18., K, 14:17):
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> Unfortunately, yesterday's sync was not recorded due to a technical
>>>>> issue. :(
>>>>>
>>>>> Here is a brief summary of the discussion and the decisions we reached:
>>>>>
>>>>>    1. We agreed to use a list of Iceberg expressions to define the
>>>>>    *index_keys*, which determine how the index is ordered.
>>>>>    2. We agreed not to restrict the functions that can be used in
>>>>>    those expressions. Engines that do not understand a particular expression
>>>>>    can simply choose not to use the index.
>>>>>    3. We agreed that index region files (leaf files) should be sorted
>>>>>    by the values generated by the index_keys.
>>>>>    4. We discussed using *included_column_ids* to define the contents
>>>>>    and schema of the index region files, and generally felt this could be a
>>>>>    good approach.
>>>>>    5. We started discussing whether the results of the index_key
>>>>>    expressions should also be stored in the index region files alongside the
>>>>>    included columns. In some cases, storing them could help readers optimize
>>>>>    lookups within a region file. In other cases, the values could be derived
>>>>>    when needed, making them redundant and potentially hurting performance by
>>>>>    increasing the size of the region files.
>>>>>
>>>>> I think we should continue the discussion around point 5, as the
>>>>> outcome may influence, or even reopen, the decisions around point 4.
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>> Flavio Junqueira <fl...@gmail.com> ezt írta (időpont:
>>>>> 2026. aug. 4., K, 18:57):
>>>>>
>>>>>> Thank you for the summary, the proposal and the overall discussion. I
>>>>>> have one main question about the target and I apologize in advance if I'm
>>>>>> missing some important detail as I might not have the full context.
>>>>>>
>>>>>> I noticed that partitioning, sorting and clustering (via Hilbert)
>>>>>> were discussed in the call in this context of index transform functions. My
>>>>>> understanding is that engines currently are already pruning partitions and
>>>>>> files using partition information and table metadata like per-column
>>>>>> min-max. If that's the case, then what's it that this proposal is trying to
>>>>>> achieve by having transform functions that capture partitioning, sorting
>>>>>> and clustering when engines are already able to prune without a secondary
>>>>>> index? I can more easily see the primary key index example, but I'm not
>>>>>> entirely sure about these data layout mechanisms.
>>>>>>
>>>>>> I also didn't quite get how engines are expected to consume such an
>>>>>> index. There was a discussion about where the mapping of the input to files
>>>>>> happens, whether that's in Iceberg or the engine, and I didn't quite get
>>>>>> the recommendation. Checking Peter's proposal, I couldn't see it clearly
>>>>>> either, and if I just missed it, I'd appreciate a pointer to where I should
>>>>>> look into.
>>>>>>
>>>>>> -Flavio
>>>>>>
>>>>>> On 4 Aug 2026, at 10:52, Péter Váry <pe...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> Here is the recording from yesterday’s sync:
>>>>>> https://www.youtube.com/watch?v=Wg9orP9JAmk
>>>>>>
>>>>>> To summarize:
>>>>>> - We agreed to retain the history of index snapshots, but not the
>>>>>> history of other index properties. If readers require information that may
>>>>>> later be modified by a user, that information should be copied into the
>>>>>> corresponding snapshot.
>>>>>> - We discussed the possibility of defining index ordering using a
>>>>>> list of transform functions.
>>>>>>
>>>>>> Dan and Yingyi suggested something along the following lines:
>>>>>>
>>>>>> *"index_keys" : [*
>>>>>> *  {*
>>>>>> *    "type": "apply",*
>>>>>> *    "func-name": {*
>>>>>> *      "catalog": "iceberg_functions",*
>>>>>> *      "name": "day"*
>>>>>> *    },*
>>>>>> *    "arguments": [*
>>>>>> *      { "type": "reference", "id": 3 }*
>>>>>> *    ]*
>>>>>> *  },*
>>>>>> *  {*
>>>>>> *    "type": "apply",*
>>>>>> *    "func-name": {*
>>>>>> *      "catalog": "iceberg_functions",*
>>>>>> *      "name": "truncate"*
>>>>>> *    },*
>>>>>> *    "arguments": [*
>>>>>> *      { "type": "reference", "id": 4 },*
>>>>>> *      2*
>>>>>> *    ]*
>>>>>> *  }*
>>>>>> *]*
>>>>>>
>>>>>>
>>>>>> In this example, rows would first be ordered by day(col3) and then by
>>>>>> truncate(col4, 2). To ensure a deterministic ordering, ties would be broken
>>>>>> using the original values of columns 3 and 4. If those are still equal,
>>>>>> ordering would fall back to the file name and finally the row position.
>>>>>>
>>>>>> I spent some time thinking about this after the sync, and here are my
>>>>>> thoughts.
>>>>>>
>>>>>> Pros:
>>>>>> - Provides a high degree of flexibility.
>>>>>> - Enables expression-based indexes.
>>>>>>
>>>>>> Cons:
>>>>>> - Do we need this flexibility at this stage?
>>>>>>    - Equality deletes, the primary use cases we are targeting, do not
>>>>>> require expressions.
>>>>>>    - Most indexes I have seen in production are not expression-based.
>>>>>> - Increased complexity for engines:
>>>>>>    - Engines must parse and understand transforms to determine
>>>>>> whether using an index can improve a query. For example, is ordering by
>>>>>> day(timestamp) sufficient for a key lookup?
>>>>>>    - Complexity increases further as new UDFs are introduced.
>>>>>>    - Expressions are not pushed into scans today, so index handling
>>>>>> cannot be fully hidden behind Iceberg scan planning.
>>>>>> - Risk of fragmentation:
>>>>>>    - The additional complexity may slow engine adoption.
>>>>>>    - Adopters may choose to support only exact transform matches
>>>>>> created by them, rather than learn to reason about the general transforms.
>>>>>>    - UDFs are prone to missing implementations and behavioral
>>>>>> differences across engines. In the worst case, such differences could lead
>>>>>> to incorrect query results when indexes are used.
>>>>>>
>>>>>> Given the points above, I still prefer defining HASH, HILBERT,
>>>>>> IDENTITY as a set of standardized values combined with an explicit list of
>>>>>> key columns. This keeps adoption simple, unlocks the primary use cases we
>>>>>> are targeting today, and allows the Iceberg library to provide the
>>>>>> transform implementations directly.
>>>>>>
>>>>>> Internally, these transforms could still be represented as Iceberg
>>>>>> expressions, giving us a migration path toward future index types that may
>>>>>> require full expression support.
>>>>>>
>>>>>> For example:
>>>>>>
>>>>>> *{*
>>>>>> *  "transform": "HILBERT",*
>>>>>> *  "key-column-ids": [3, 4]*
>>>>>> *}*
>>>>>>
>>>>>>
>>>>>> I'm interested in hearing other opinions.
>>>>>>
>>>>>> Thanks,
>>>>>> Peter
>>>>>>
>>>>>> Renjie Liu <li...@gmail.com> ezt írta (időpont: 2026. júl.
>>>>>> 28., K, 10:07):
>>>>>>
>>>>>>> Hi, Peter:
>>>>>>>
>>>>>>> Thanks for driving the proposal.
>>>>>>>
>>>>>>> >  *Hash Index Type*: Should hash-based indexes be a separate index
>>>>>>> type? They are only applicable to equality and IN predicates, and the hash
>>>>>>> algorithm must always be known. A dedicated HASH type could make these
>>>>>>> constraints explicit and distinguish hash-based indexes from ordered
>>>>>>> indexes.
>>>>>>>
>>>>>>> I'm leaning toward keeping them in scalar index type due to the
>>>>>>> similarity with others transformation functions like identity. As with the
>>>>>>> constraints, I think they are properties of transformation functions.
>>>>>>>
>>>>>>> > *Transform Function/Layout Representation*: Should transform
>>>>>>> functions be represented as an enum or a string? I lean toward strings, but
>>>>>>> I do not have a strong preference. In many cases, an index can still be
>>>>>>> used based on standard column statistics, even if a reader does not
>>>>>>> recognize a particular transform.
>>>>>>>
>>>>>>> This seems more like a problem in java implementation?
>>>>>>>
>>>>>>> Also I prefer the name transform function compared with layout
>>>>>>> representation. Layout representation makes me feel that it's not about the
>>>>>>> definition of an index function.
>>>>>>>
>>>>>>> > *Future Index Types*: How should we handle index types that are
>>>>>>> expected in the long term but fall outside the scope of this proposal (for
>>>>>>> example, VECTOR, IVF, TERM)? Should they be mentioned as planned/reserved
>>>>>>> concepts, or omitted entirely? I lean toward at least mentioning them. The
>>>>>>> rationale for introducing an index type is easier to explain when
>>>>>>> considering future index families.
>>>>>>>
>>>>>>> +1
>>>>>>>
>>>>>>> *> Index History Handling:* Most database systems treat index
>>>>>>> changes as a drop-and-recreate operation rather than an in-place update. If
>>>>>>> we follow the same approach, there is limited value in maintaining index
>>>>>>> metadata history. During index maintenance, a new metadata.json can be
>>>>>>> created and atomically swapped in, with the catalog responsible for
>>>>>>> cleaning up obsolete metadata and index data files according to its
>>>>>>> retention policy.
>>>>>>>
>>>>>>> +1, I think we only need to keep index data for different snapshots,
>>>>>>> but if an index definition changed, we should drop and recreate a new index.
>>>>>>>
>>>>>>> > *File Format Location*: Should file format be specified at the
>>>>>>> leaf-file level or only at the index level? My preference is to stay
>>>>>>> aligned with the table metadata model and avoid introducing differences
>>>>>>> unless there is a compelling requirement. This would simplify
>>>>>>> implementation and maximize code reuse.
>>>>>>>
>>>>>>> +1.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jul 24, 2026 at 11:57 PM Péter Váry <
>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi everyone,
>>>>>>>>
>>>>>>>> Here are the key takeaways from Monday’s meeting (July 20):
>>>>>>>>
>>>>>>>>    - *Index Type*: An index type enum will be introduced in v1,
>>>>>>>>    initially with a single value (SCALAR). This establishes a future-proof API
>>>>>>>>    boundary while leaving room for additional index families.
>>>>>>>>    - *Strict Total Ordering*: The specification should require
>>>>>>>>    transform functions to produce values that preserve a well-defined total
>>>>>>>>    ordering across the key space. The transform function does not need to
>>>>>>>>    return a long. Instead, leaf files can store the first and last values
>>>>>>>>    according to the ordering, enabling generic filtering capabilities for
>>>>>>>>    readers.
>>>>>>>>
>>>>>>>> Open Questions:
>>>>>>>>
>>>>>>>>    - *Hash Index Type*: Should hash-based indexes be a separate
>>>>>>>>    index type? They are only applicable to equality and IN predicates, and the
>>>>>>>>    hash algorithm must always be known. A dedicated HASH type could make these
>>>>>>>>    constraints explicit and distinguish hash-based indexes from ordered
>>>>>>>>    indexes.
>>>>>>>>    - *Transform Function/Layout Representation*: Should transform
>>>>>>>>    functions be represented as an enum or a string? I lean toward strings, but
>>>>>>>>    I do not have a strong preference. In many cases, an index can still be
>>>>>>>>    used based on standard column statistics, even if a reader does not
>>>>>>>>    recognize a particular transform.
>>>>>>>>    - *Future Index Types*: How should we handle index types that
>>>>>>>>    are expected in the long term but fall outside the scope of this proposal
>>>>>>>>    (for example, VECTOR, IVF, TERM)? Should they be mentioned as
>>>>>>>>    planned/reserved concepts, or omitted entirely? I lean toward at least
>>>>>>>>    mentioning them. The rationale for introducing an index type is easier to
>>>>>>>>    explain when considering future index families.
>>>>>>>>    - *Index History Handling:* Most database systems treat index
>>>>>>>>    changes as a drop-and-recreate operation rather than an in-place update. If
>>>>>>>>    we follow the same approach, there is limited value in maintaining index
>>>>>>>>    metadata history. During index maintenance, a new metadata.json can be
>>>>>>>>    created and atomically swapped in, with the catalog responsible for
>>>>>>>>    cleaning up obsolete metadata and index data files according to its
>>>>>>>>    retention policy.
>>>>>>>>    - *Min/Max Statistics Representation*: Should
>>>>>>>>    transform-specific or sort-order min/max values be represented using the
>>>>>>>>    existing file statistics structure, or exposed through dedicated fields? My
>>>>>>>>    preference is to keep them within the statistics structure defined by the
>>>>>>>>    file schema while exposing them directly through the Java API if needed.
>>>>>>>>    - *File Format Location*: Should file format be specified at
>>>>>>>>    the leaf-file level or only at the index level? My preference is to stay
>>>>>>>>    aligned with the table metadata model and avoid introducing differences
>>>>>>>>    unless there is a compelling requirement. This would simplify
>>>>>>>>    implementation and maximize code reuse.
>>>>>>>>
>>>>>>>> I have updated the spec PR based on the decisions above:
>>>>>>>> https://github.com/apache/iceberg/pull/16961
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Peter
>>>>>>>>
>>>>>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont: 2026. jún.
>>>>>>>> 26., P, 19:42):
>>>>>>>>
>>>>>>>>> Hi all, Here is the summary of this Monday's index meeting:
>>>>>>>>>
>>>>>>>>> Two blocking decisions closed:
>>>>>>>>>
>>>>>>>>>    1. Index is *not* a table— it's its own object (reuses table
>>>>>>>>>    machinery under the hood). Reasoning: requires a sort order, no column
>>>>>>>>>    updates, no partition spec, no overlapping ranges between leaves, inherits
>>>>>>>>>    base-table permissions, and has its own CREATE/DROP/UPDATE INDEX DDL.
>>>>>>>>>    2. Index is a separate catalog entity, with no pointers in
>>>>>>>>>    table metadata— has its own REST endpoints; the catalog can optionally
>>>>>>>>>    return index metadata withloadTable to avoid extra round-trips. Keeps table
>>>>>>>>>    and index updates independent/async.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Next steps: Start writing the spec and build out the copy-on-write
>>>>>>>>> path now.
>>>>>>>>>
>>>>>>>>> Here are the draft spec:
>>>>>>>>> secondary index spec
>>>>>>>>> <https://github.com/apache/iceberg/pull/16961>
>>>>>>>>> irc spec <https://github.com/apache/iceberg/pull/16963>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Huaxin
>>>>>>>>>
>>>>>>>>> On Sat, Jun 20, 2026 at 11:11 AM huaxin gao <
>>>>>>>>> huaxin.gao11@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> I built a standalone PoC to validate that the basic index
>>>>>>>>>> structure works: that we can build a PK index, convert equality deletes to
>>>>>>>>>> position deletes through it, and have every converted delete land on the
>>>>>>>>>> correct live row. I ran it up to *100M keys*.
>>>>>>>>>>
>>>>>>>>>> *Headline: the structure works.* The index builds over up to
>>>>>>>>>> 100M keys, the eq-delete → position-delete conversion resolved correctly at
>>>>>>>>>> *every* size (100% of converted deletes mapped to the right live
>>>>>>>>>> row), and the resulting position deletes are *~8× cheaper to
>>>>>>>>>> apply* at query time than the equality deletes they replace.
>>>>>>>>>>
>>>>>>>>>> Beyond correctness, the run also shows how the index’s
>>>>>>>>>> *maintenance* cost scales, comparing copy-on-write (COW, rewrite
>>>>>>>>>> touched leaves) vs an append/merge (MOR) option, under a realistic mixed
>>>>>>>>>> CDC checkpoint (1,000 insert + 500 update + 500 delete), local wall-clock:
>>>>>>>>>> keys EQ baseline INDEX (COW) % of 60s (COW) INDEX (MOR) % of 60s
>>>>>>>>>> (MOR) correct
>>>>>>>>>> 5M 6 ms 6.7s 11.2% 2.2s 3.7% PASS
>>>>>>>>>> 20M 8 ms 24.2s 40.4% 6.4s 10.6% PASS
>>>>>>>>>> 50M 7 ms 51.6s 86.1% 12.2s 20.4% PASS
>>>>>>>>>> *100M* 6 ms *75.0s* 125% (BEHIND) *16.9s* 28.2% (keeps up) PASS
>>>>>>>>>>
>>>>>>>>>> COW maintenance crosses the 60 s checkpoint around 100M
>>>>>>>>>> (75 s/cycle, 125%); MOR stays at ~28% and keeps pace; the equality-delete
>>>>>>>>>> baseline is ~6 ms and flat. So the structure works, but *COW
>>>>>>>>>> alone can’t sustain scattered CDC at hundreds of millions of keys on a
>>>>>>>>>> single writer*. It’s worth allowing a merge-on-read /
>>>>>>>>>> update-file maintenance option alongside COW (or sharding the index across
>>>>>>>>>> parallel writers).
>>>>>>>>>>
>>>>>>>>>> *Full write-up, all tables, and the in-region reality-check:*
>>>>>>>>>> link
>>>>>>>>>> <https://docs.google.com/document/d/1G3zxbW8X0eU3UrouslZfp42bBc9CvgJGnJyDONCB4PU/edit?tab=t.0>
>>>>>>>>>>
>>>>>>>>>> Feedback welcome, especially on the spec direction (whether to
>>>>>>>>>> allow a merge-on-read / update-file maintenance option alongside COW)  and
>>>>>>>>>> on the read-side modeling.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Huaxin
>>>>>>>>>>
>>>>>>>>>> On Tue, Jun 9, 2026 at 5:45 PM huaxin gao <hu...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Sorry,  we've skipped posting a few of the dedicated index-sync
>>>>>>>>>>> summaries to the mailing list; you can find those in the Google
>>>>>>>>>>> doc
>>>>>>>>>>> <https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?pli=1&tab=t.8041k7j2n7y3>
>>>>>>>>>>> and the Slack channel. Here's yesterday's summary:
>>>>>>>>>>>
>>>>>>>>>>> *Decided*
>>>>>>>>>>>
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    Index vs. table (what we agreed):
>>>>>>>>>>>    - Reuse table implementation/library code and a
>>>>>>>>>>>       near-identical spec — the commit path will be custom regardless, so reuse
>>>>>>>>>>>       isn't the deciding factor.
>>>>>>>>>>>       - An index is not a table from a user/API view: loading
>>>>>>>>>>>       or writing an index as a table must fail(it would violate index invariants).
>>>>>>>>>>>       - The spec forbids most table behaviors: no overlapping
>>>>>>>>>>>       files, one mandatory transform sort order, no column updates, no partition
>>>>>>>>>>>       spec.
>>>>>>>>>>>       - Delete vectors: reuse Iceberg's existing DV —
>>>>>>>>>>>    benchmarks showed no new delete format is worth introducing.
>>>>>>>>>>>    - Incremental updates: start with copy-on-write only (no
>>>>>>>>>>>    update files). For object-store-sized leaves, a full leaf rewrite is about
>>>>>>>>>>>    as cheap as maintaining an overlay update file + DV, so we'll skip the MOR
>>>>>>>>>>>    machinery for now and add it later only if benchmarks prove we need it
>>>>>>>>>>>    (likely just the very-large-leaf case).
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    Validate the spec first: build a quick, hand-wired prototype
>>>>>>>>>>>    (Parquet files structured per the spec) and benchmark it on real scales
>>>>>>>>>>>    before formalizing.
>>>>>>>>>>>
>>>>>>>>>>> *Leaning, not final*
>>>>>>>>>>>
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    Indexes are likely separate catalog objects, linked from the
>>>>>>>>>>>    table by storing just an identifier (like materialized views) and not
>>>>>>>>>>>    visible in LIST TABLES.
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    We'll need a commit path for indexes, but simpler than
>>>>>>>>>>>    tables (no stage-create).
>>>>>>>>>>>
>>>>>>>>>>> *Still open*
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    Permissions model — separate vs. inherited (action: look at
>>>>>>>>>>>    what real DBs do for index permissions).
>>>>>>>>>>>    - REST/catalog RPC design — minimize round-trips; index
>>>>>>>>>>>    metadata ideally returned with LOAD TABLE. Catalog RPC cost
>>>>>>>>>>>    may dominate Parquet IO, so this needs real design.
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    Scale modeling — target rows-per-leaf vs. leaf size vs.
>>>>>>>>>>>    metadata-file count.
>>>>>>>>>>>    -
>>>>>>>>>>>
>>>>>>>>>>>    DDL-on-index semantics (reuse table schema-update actions or
>>>>>>>>>>>    separate)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Huaxin
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Apr 22, 2026 at 8:47 AM Péter Váry <
>>>>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>
>>>>>>>>>>>> TL;DR
>>>>>>>>>>>> We still need to validate with ADLS and S3, but based on the
>>>>>>>>>>>> local tests, the MPHF approach looks more promising if we can tolerate
>>>>>>>>>>>> larger files and longer index maintenance times.
>>>>>>>>>>>>
>>>>>>>>>>>> Details:
>>>>>>>>>>>> Here are the results from the local experiments on my Mac. I
>>>>>>>>>>>> removed unnecessary statistics from the Parquet files and tested different
>>>>>>>>>>>> row group sizes:
>>>>>>>>>>>>
>>>>>>>>>>>>    - For an index file with 1M records, a row group size of
>>>>>>>>>>>>    5,000 appears to be the sweet spot.
>>>>>>>>>>>>    - For 10M records, 10,000 rows per row group works best.
>>>>>>>>>>>>
>>>>>>>>>>>> If you have additional ideas for optimizing Parquet-based
>>>>>>>>>>>> indexes, I’d be very interested to hear them.
>>>>>>>>>>>> The test code is available on this branch:
>>>>>>>>>>>> https://github.com/pvary/iceberg/tree/leaf_bench
>>>>>>>>>>>>
>>>>>>>>>>>> Best results:
>>>>>>>>>>>> *1m records/file*
>>>>>>>>>>>>
>>>>>>>>>>>>    - Parquet - 5000 row/RowGroup
>>>>>>>>>>>>       - Read: 1191 µs - 1 file open, 3 seek, 123KB read per
>>>>>>>>>>>>       lookup
>>>>>>>>>>>>       - Write: 1.7 s, 15 MB
>>>>>>>>>>>>    - MPHF
>>>>>>>>>>>>       - Read: 202 µs - 1 file open, 1 seek,  282KB read per
>>>>>>>>>>>>       lookup
>>>>>>>>>>>>       - Write: 0.8 s, 34 MB
>>>>>>>>>>>>
>>>>>>>>>>>> *10m records/file*
>>>>>>>>>>>>
>>>>>>>>>>>>    - Parquet - 10000 row/RowGroup
>>>>>>>>>>>>       - Read: 4168 µs - 1 file open, 3 seek, 395KB read per
>>>>>>>>>>>>       lookup
>>>>>>>>>>>>       - Write: 19.5s s, 144 MB
>>>>>>>>>>>>    - MPHF
>>>>>>>>>>>>       - Read: 1086 µs - 1 file open, 1 seek,  2.8 MB (2812KB)
>>>>>>>>>>>>       read per lookup
>>>>>>>>>>>>       - Write: 6.5 s, 34 MB, 353 MB
>>>>>>>>>>>>
>>>>>>>>>>>> Below are the full results.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> *Benchmark                                      (indexType)
>>>>>>>>>>>>  (keyType)  (numRows)  Mode    Cnt            Score          Error
>>>>>>>>>>>>  UnitsInvertedIndexBenchmark.lookup                 PARQUET_1000       LONG
>>>>>>>>>>>>    1000000    ss  10000         3285.284 ±        5.138
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_1000       LONG
>>>>>>>>>>>>    1000000    ss  10000   2522168989.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_1000       LONG
>>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_1000       LONG
>>>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_1000       LONG
>>>>>>>>>>>> 10000000    ss  10000        35449.614 ±       34.673
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_1000       LONG
>>>>>>>>>>>>   10000000    ss  10000  24302649201.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_1000       LONG
>>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_1000       LONG
>>>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_5000       LONG
>>>>>>>>>>>>  1000000    ss  10000         1191.959 ±        4.169
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_5000       LONG
>>>>>>>>>>>>    1000000    ss  10000   1230877229.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_5000       LONG
>>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_5000       LONG
>>>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_5000       LONG
>>>>>>>>>>>> 10000000    ss  10000         7236.447 ±       10.374
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_5000       LONG
>>>>>>>>>>>>   10000000    ss  10000   5650715973.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_5000       LONG
>>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_5000       LONG
>>>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_10000       LONG
>>>>>>>>>>>>  1000000    ss  10000         1349.946 ±        7.834
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_10000       LONG
>>>>>>>>>>>>    1000000    ss  10000   1730219377.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_10000       LONG
>>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_10000       LONG
>>>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_10000       LONG
>>>>>>>>>>>> 10000000    ss  10000         4168.635 ±       11.051
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_10000       LONG
>>>>>>>>>>>>   10000000    ss  10000   3946341532.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_10000       LONG
>>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_10000       LONG
>>>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_50000       LONG
>>>>>>>>>>>>  1000000    ss  10000         4736.466 ±       38.179
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_50000       LONG
>>>>>>>>>>>>    1000000    ss  10000   7427413541.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_50000       LONG
>>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_50000       LONG
>>>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_50000       LONG
>>>>>>>>>>>> 10000000    ss  10000         4979.031 ±       34.708
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_50000       LONG
>>>>>>>>>>>>   10000000    ss  10000   7694887636.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_50000       LONG
>>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_50000       LONG
>>>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                         MPHF       LONG
>>>>>>>>>>>>  1000000    ss  10000          202.571 ±        2.336
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead               MPHF       LONG
>>>>>>>>>>>>    1000000    ss  10000   2821570000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams             MPHF       LONG
>>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks                   MPHF       LONG
>>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup                         MPHF       LONG
>>>>>>>>>>>> 10000000    ss  10000         1086.957 ±        4.524
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead               MPHF       LONG
>>>>>>>>>>>>   10000000    ss  10000  28119460000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams             MPHF       LONG
>>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks                   MPHF       LONG
>>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_1000       LONG
>>>>>>>>>>>>  1000000    ss      3      1720731.014 ±   876636.004
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_1000       LONG
>>>>>>>>>>>>    1000000    ss      3     46453317.000
>>>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_1000       LONG
>>>>>>>>>>>> 10000000    ss      3     18547947.876 ± 12258125.307
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_1000       LONG
>>>>>>>>>>>>   10000000    ss      3    452655675.000
>>>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_5000       LONG
>>>>>>>>>>>>  1000000    ss      3      1718345.583 ±  1103928.016
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_5000       LONG
>>>>>>>>>>>>    1000000    ss      3     44845788.000
>>>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_5000       LONG
>>>>>>>>>>>> 10000000    ss      3     18604229.931 ±  2668361.915
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_5000       LONG
>>>>>>>>>>>>   10000000    ss      3    435388818.000
>>>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_10000       LONG
>>>>>>>>>>>>  1000000    ss      3      1761555.389 ±   535857.675
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_10000       LONG
>>>>>>>>>>>>    1000000    ss      3     44536635.000
>>>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_10000       LONG
>>>>>>>>>>>> 10000000    ss      3     19501588.264 ±  2130054.558
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_10000       LONG
>>>>>>>>>>>>   10000000    ss      3    433189623.000
>>>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_50000       LONG
>>>>>>>>>>>>  1000000    ss      3      1936624.889 ±  6601363.985
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_50000       LONG
>>>>>>>>>>>>    1000000    ss      3     44264655.000
>>>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_50000       LONG
>>>>>>>>>>>> 10000000    ss      3     20471742.278 ± 10705206.310
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_50000       LONG
>>>>>>>>>>>>   10000000    ss      3    431311305.000
>>>>>>>>>>>> #InvertedIndexBenchmark.write                          MPHF       LONG
>>>>>>>>>>>>  1000000    ss      3       896573.958 ±  1408024.851
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes           MPHF       LONG
>>>>>>>>>>>>    1000000    ss      3    102846369.000
>>>>>>>>>>>> #InvertedIndexBenchmark.write                          MPHF       LONG
>>>>>>>>>>>> 10000000    ss      3      6509348.875 ± 15519975.479
>>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes           MPHF       LONG
>>>>>>>>>>>>   10000000    ss      3   1058435733.000                     #*
>>>>>>>>>>>>
>>>>>>>>>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont: 2026.
>>>>>>>>>>>> ápr. 21., K, 20:53):
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>
>>>>>>>>>>>>> In recent secondary index sync meetings, the discussion
>>>>>>>>>>>>> converged on the need to define what an index is from first principles
>>>>>>>>>>>>> before settling on physical layout.
>>>>>>>>>>>>>
>>>>>>>>>>>>> To address that, Peter and I have drafted a requirements
>>>>>>>>>>>>> document for a key lookup index (renamed from "primary key index" to avoid
>>>>>>>>>>>>> implying uniqueness enforcement), the goal is to nail down one well-scoped
>>>>>>>>>>>>> index type first.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Doc: Key Lookup Index Requirements
>>>>>>>>>>>>> <https://docs.google.com/document/d/1e0zxK-jA0LBDq8YQlQgFipTHelDFiga8lCkgDTmYub8/edit?tab=t.0#heading=h.8shrgabvl19>
>>>>>>>>>>>>>
>>>>>>>>>>>>> It covers requirements, three design options (manifest +
>>>>>>>>>>>>> sorted Parquet, hash + sorted Parquet, hash + MPHF) and open questions. We
>>>>>>>>>>>>> will add preliminary benchmark results shortly.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Feedback welcome — inline in the doc, on this thread, or at
>>>>>>>>>>>>> the next index sync.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Huaxin
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Apr 13, 2026 at 7:22 AM Steven Wu <
>>>>>>>>>>>>> stevenz3wu@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Do we need the special index identifier that was originally
>>>>>>>>>>>>>> proposed? A generic CatalogObjectIdentifier (with namespace and name) would
>>>>>>>>>>>>>> be consistent with all object types in the catalog. I have a discussion
>>>>>>>>>>>>>> thread on the generic identifier topic: [DISCUSS] REST Spec:
>>>>>>>>>>>>>> generic CatalogObjectIdentifier.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Should we add an indexes array field to table metadata? It
>>>>>>>>>>>>>> only contains a list of index object identifiers. It doesn't contain any
>>>>>>>>>>>>>> index metadata which should live in the index objects. Yufei was trying to
>>>>>>>>>>>>>> bring this up at the end of the first sync. But we didn't get enough time
>>>>>>>>>>>>>> to really discuss it. It will be great to discuss this as the first agenda
>>>>>>>>>>>>>> item today.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Apr 13, 2026 at 3:17 AM Péter Váry <
>>>>>>>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We had several engaging discussions at the Iceberg Summit,
>>>>>>>>>>>>>>> and it was great to finally catch up with many of you in person. We truly
>>>>>>>>>>>>>>> missed those who couldn’t attend, hopefully we’ll all meet again at the
>>>>>>>>>>>>>>> next summit.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> To keep the conversation going, Huaxin and I have put
>>>>>>>>>>>>>>> together the agenda for our next meeting. As a reminder, we’ll meet on *April
>>>>>>>>>>>>>>> 13th, 9:00–10:00 AM *PDT (6:00–7:00 PM CEST).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Proposed agenda:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    - Continue first-principles index design discussion from
>>>>>>>>>>>>>>>    Mar 30
>>>>>>>>>>>>>>>       - *Index Ownership and Write Responsibility*
>>>>>>>>>>>>>>>          - Should writers be allowed to update indexes, or
>>>>>>>>>>>>>>>          - Should all index writes be handled exclusively
>>>>>>>>>>>>>>>          by the Index Maintenance process?
>>>>>>>>>>>>>>>          - If writers can update indexes then we need to
>>>>>>>>>>>>>>>          define what guarantees are required (compaction, file splitting, layout
>>>>>>>>>>>>>>>          expectations)?
>>>>>>>>>>>>>>>          - If only Index Maintenance updates indexes then
>>>>>>>>>>>>>>>          we only need to define what observable properties should be exposed to
>>>>>>>>>>>>>>>          consumers? Like:
>>>>>>>>>>>>>>>             - Expected max files for a single key
>>>>>>>>>>>>>>>             - Current max files for a single key
>>>>>>>>>>>>>>>             - Deletes allowed/present
>>>>>>>>>>>>>>>             - Sorted by
>>>>>>>>>>>>>>>             - Partitioned by
>>>>>>>>>>>>>>>          - *Specification Scope: What Belongs in the Spec?*
>>>>>>>>>>>>>>>          - Related to the ownership question above
>>>>>>>>>>>>>>>          - Light spec: Just define that the index table
>>>>>>>>>>>>>>>          should be optimized for retrieval by key columns and the index columns
>>>>>>>>>>>>>>>          should be contained in the table. This could give us more flexibility if
>>>>>>>>>>>>>>>          better organization methods come up, or
>>>>>>>>>>>>>>>          - Detailed spec: We could define the max number of
>>>>>>>>>>>>>>>          files per index to read for a single key, or even the partitioning and the
>>>>>>>>>>>>>>>          exact sort order. This could allow more use-cases for a given index, like
>>>>>>>>>>>>>>>          joins or cardinality estimations.
>>>>>>>>>>>>>>>          - I would go for light spec for the main types
>>>>>>>>>>>>>>>          (PK, Containing) and only the Index Maintenance processes should update the
>>>>>>>>>>>>>>>          Indexes, as for many use-cases the details are not important, and writers
>>>>>>>>>>>>>>>          will very rarely update the Indexes themselves.
>>>>>>>>>>>>>>>       - *Logical Placement of Indexes*
>>>>>>>>>>>>>>>          - Index as a child object of an Iceberg Table, or
>>>>>>>>>>>>>>>          - Index as a first‑class entity under
>>>>>>>>>>>>>>>          /namespace/indexes/{index}
>>>>>>>>>>>>>>>          - Based on the discussions on the summit we are
>>>>>>>>>>>>>>>          leaning in this direction. This means the index id should be unique in the
>>>>>>>>>>>>>>>          namespace but helps the catalog implementations quite a bit
>>>>>>>>>>>>>>>       - *Physical Placement of Index Data*
>>>>>>>>>>>>>>>          - I don’t think we should specify this. We should
>>>>>>>>>>>>>>>          have a base location for the index, but can rely on the catalog
>>>>>>>>>>>>>>>          implementations to decide on their own, like they do with the tables,
>>>>>>>>>>>>>>>          views, udfs.
>>>>>>>>>>>>>>>       - *Iceberg Reader Based indexes* (Containing indexes
>>>>>>>>>>>>>>>       and potentially PK indexes). These are the indexes which could be read by
>>>>>>>>>>>>>>>       the existing Iceberg readers. We might decide to store the PK index
>>>>>>>>>>>>>>>       similarly to an Iceberg Table and treat it as a reader based index.
>>>>>>>>>>>>>>>          - What are the table properties/features exposed
>>>>>>>>>>>>>>>          to the readers
>>>>>>>>>>>>>>>             - Maybe just some behavioral descriptors for
>>>>>>>>>>>>>>>             the optimizer to decide if the index could be used or should be skipped,
>>>>>>>>>>>>>>>             like:
>>>>>>>>>>>>>>>                - Expected max files for a single key
>>>>>>>>>>>>>>>                - max files for a single key
>>>>>>>>>>>>>>>                - Deletes allowed/present
>>>>>>>>>>>>>>>                - Sorted by
>>>>>>>>>>>>>>>                - Partitioned by
>>>>>>>>>>>>>>>             - The Tasks when reading the index based on the
>>>>>>>>>>>>>>>             filters and projection
>>>>>>>>>>>>>>>          - What are the table properties/features exposed
>>>>>>>>>>>>>>>          to the Index Maintenance. I think this could be internal to the Index
>>>>>>>>>>>>>>>          Maintenance process and might not be exposed through the spec. The Index
>>>>>>>>>>>>>>>          Maintenance process could handle this as a standard Iceberg Table and could
>>>>>>>>>>>>>>>          be based on the Table Maintenance process, but there might be some totally
>>>>>>>>>>>>>>>          different processes.
>>>>>>>>>>>>>>>       - It should be possible to add properties to an index
>>>>>>>>>>>>>>>       defined by the Index Maintenance process which could be used and updated in
>>>>>>>>>>>>>>>       the next Index Maintenance run.
>>>>>>>>>>>>>>>    - *PK index storage format benchmark results*
>>>>>>>>>>>>>>>       - Flat Parquet (baseline)
>>>>>>>>>>>>>>>       - BTree with Parquet leaves
>>>>>>>>>>>>>>>       - Vortex
>>>>>>>>>>>>>>>    - *Open items / next steps*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Peter
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont:
>>>>>>>>>>>>>>> 2026. márc. 23., H, 3:03):
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi everyone, I wanted to share an update on the primary key
>>>>>>>>>>>>>>>> index work.
>>>>>>>>>>>>>>>> Since there are still open questions on whether bloom
>>>>>>>>>>>>>>>> filter indexes fit in the secondary index framework or should be treated as
>>>>>>>>>>>>>>>> extended stats, I've shifted focus to the primary key index since it's a
>>>>>>>>>>>>>>>> clearer fit for the framework.
>>>>>>>>>>>>>>>> I've put together a proposal for a primary key
>>>>>>>>>>>>>>>> reverse-lookup index that maps each key to its physical location
>>>>>>>>>>>>>>>> (file_path, row_position). It enables:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - Scan-time file pruning for point lookups
>>>>>>>>>>>>>>>>    - Converting key-based deletes into position deletes
>>>>>>>>>>>>>>>>    (eliminating equality deletes for Flink CDC)
>>>>>>>>>>>>>>>>    - Accelerating Spark MERGE INTO by replacing full-table
>>>>>>>>>>>>>>>>    joins with direct file lookups
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Proposal:
>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1HuhCZ0n2FqDh8yqQb9oEj1CPM5yXpEsMPGZno2aSf8E/edit?tab=t.0#heading=h.tbevg4q0m9
>>>>>>>>>>>>>>>> Feedback welcome!
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Huaxin
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Mar 18, 2026 at 11:42 PM Péter Váry <
>>>>>>>>>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Key takeaways from the general index discussion at the May
>>>>>>>>>>>>>>>>> 16 meeting.
>>>>>>>>>>>>>>>>> Thanks to everyone who participated! The recording is
>>>>>>>>>>>>>>>>> available here:
>>>>>>>>>>>>>>>>> https://www.youtube.com/watch?v=btmjhtRWUCE
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - Q: Do we need to tie index types to the algorithms
>>>>>>>>>>>>>>>>>    used to access them?
>>>>>>>>>>>>>>>>>    - A: From a specification perspective, the goal is to
>>>>>>>>>>>>>>>>>    define the storage-level data layout so it can be shared across engines.
>>>>>>>>>>>>>>>>>    Engines are free to interpret and use the data as they see fit, but the
>>>>>>>>>>>>>>>>>    on-disk data layout itself must be strictly defined and interoperable.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - Q: Should we introduce an additional abstraction
>>>>>>>>>>>>>>>>>    layer (e.g., Vector Index) with sub-types such as IVF and DiskANN?
>>>>>>>>>>>>>>>>>    - A: This is possible if we decide it is beneficial. I
>>>>>>>>>>>>>>>>>    explored potential naming, but it is not yet clear how such a layer would
>>>>>>>>>>>>>>>>>    be used in practice.
>>>>>>>>>>>>>>>>>    *Question to Yingyi Bu*: could you provide examples
>>>>>>>>>>>>>>>>>    where this additional layer would be useful? Should this abstraction be
>>>>>>>>>>>>>>>>>    defined at the spec level, or is it better handled at the engine level?
>>>>>>>>>>>>>>>>>    My initial idea was that users would create a generic
>>>>>>>>>>>>>>>>>    Vector Index and let the engine choose the concrete implementation.
>>>>>>>>>>>>>>>>>    However, this would limit user control and users likely need to specify the
>>>>>>>>>>>>>>>>>    exact index representation, which implies they must be aware of the
>>>>>>>>>>>>>>>>>    available representations.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - Q: Do we want to allow extensibility for index types?
>>>>>>>>>>>>>>>>>    - A: Yes. The intent is to support a small set of
>>>>>>>>>>>>>>>>>    well-defined index types while allowing experimentation with new ones. If a
>>>>>>>>>>>>>>>>>    new index type proves broadly useful, a follow-up proposal can standardize
>>>>>>>>>>>>>>>>>    it and incorporate it into the spec.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - Q: Do we allow multiple versions of an index for the
>>>>>>>>>>>>>>>>>    same table snapshot?
>>>>>>>>>>>>>>>>>    - A: Yes. Older index versions must be retained for
>>>>>>>>>>>>>>>>>    readers that have already started using them, while new readers should
>>>>>>>>>>>>>>>>>    automatically use the latest available version
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - Q: Do we need to use materialized views for these
>>>>>>>>>>>>>>>>>    indexes?
>>>>>>>>>>>>>>>>>    - A: No. These indexes are primarily examples, and
>>>>>>>>>>>>>>>>>    different types may require different storage methods. However, the Primary
>>>>>>>>>>>>>>>>>    Key, Containing, and parts of the IVF indexes can be structured as Iceberg
>>>>>>>>>>>>>>>>>    tables. This allows engines to read them natively; in some cases, Iceberg
>>>>>>>>>>>>>>>>>    planners can automatically redirect queries to the index table without
>>>>>>>>>>>>>>>>>    engine modifications. Furthermore, index maintenance for these tables can
>>>>>>>>>>>>>>>>>    leverage existing materialized view maintenance workflows. Other index
>>>>>>>>>>>>>>>>>    types may instead rely on Puffin files or alternative storage approaches.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - Q: How should index metadata be accessed? Should we
>>>>>>>>>>>>>>>>>    add explicit pointers for the indexes in the table metadata?
>>>>>>>>>>>>>>>>>    - A: We did not have sufficient time to fully explore
>>>>>>>>>>>>>>>>>    and conclude this topic.
>>>>>>>>>>>>>>>>>    *Question for Yufei Gu*: Did I understand correctly
>>>>>>>>>>>>>>>>>    that your main concern stems from endpoint resolution from a REST Catalog
>>>>>>>>>>>>>>>>>    perspective? Specifically, if indexes are exposed under a URI such as
>>>>>>>>>>>>>>>>>    v1/{prefix}/namespaces/{namespace}/tables/{table}/indexes/{index}, would
>>>>>>>>>>>>>>>>>    this make it more difficult for the REST Catalog to resolve and route
>>>>>>>>>>>>>>>>>    requests to the appropriate endpoint?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Suhas Jayaram Subramanya via dev <de...@iceberg.apache.org>
>>>>>>>>>>>>>>>>> ezt írta (időpont: 2026. márc. 13., P, 23:32):
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Here's a proposal for native Vector Index support in
>>>>>>>>>>>>>>>>>> Iceberg tables --
>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1KL4qLOwdqnhOcqTc0EjO1O16NV3M3c-gZCEINDWw4lA/edit?usp=sharing
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We've been working on this proposal with Peter internally
>>>>>>>>>>>>>>>>>> at Microsoft and he suggested we post it here to bring this to the
>>>>>>>>>>>>>>>>>> community's attention, ahead of the next Secondary Index Sync.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Suhas
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 2026/02/19 04:34:34 huaxin gao wrote:
>>>>>>>>>>>>>>>>>> > Hi Everyone,
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Here are the recording and notes from the Iceberg Index
>>>>>>>>>>>>>>>>>> Support Sync on
>>>>>>>>>>>>>>>>>> > 2/11.
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Recording: https://www.youtube.com/watch?v=3sFfQ0A50yk
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Notes:
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.8041k7j2n7y3
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > The meeting will move to biweekly, Mondays 9–10am PST,
>>>>>>>>>>>>>>>>>> starting March 2.
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Since the sync, I updated the Bloom skipping index
>>>>>>>>>>>>>>>>>> proposal
>>>>>>>>>>>>>>>>>> > <
>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.5r5kl6k3fqwu
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > to address the discussion questions, specifically:
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > - Performance justification: when this helps
>>>>>>>>>>>>>>>>>> (high-cardinality = / IN,
>>>>>>>>>>>>>>>>>> > many data files, high object-store latency) and how it
>>>>>>>>>>>>>>>>>> differs from Parquet
>>>>>>>>>>>>>>>>>> > row-group Bloom filters (which still require opening
>>>>>>>>>>>>>>>>>> the data file).
>>>>>>>>>>>>>>>>>> > - Cost / scalability: rough sizing (Bloom blob size per
>>>>>>>>>>>>>>>>>> file, Puffin
>>>>>>>>>>>>>>>>>> > file size), the planning cost trade-off (driver index
>>>>>>>>>>>>>>>>>> reads vs executor
>>>>>>>>>>>>>>>>>> > file opens), and mitigations via caching.
>>>>>>>>>>>>>>>>>> > - Lifecycle / maintenance: incremental production as
>>>>>>>>>>>>>>>>>> new data files
>>>>>>>>>>>>>>>>>> > arrive, behavior when the index is missing/behind, and
>>>>>>>>>>>>>>>>>> sharding/compaction
>>>>>>>>>>>>>>>>>> > plus cleanup to avoid accumulating too many small
>>>>>>>>>>>>>>>>>> Puffin files over time.
>>>>>>>>>>>>>>>>>> > - Writer expectations: inline (optional) vs
>>>>>>>>>>>>>>>>>> asynchronous (primary) index
>>>>>>>>>>>>>>>>>> > creation.
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > I also implemented a Spark 4.1 POC
>>>>>>>>>>>>>>>>>> > <https://github.com/apache/iceberg/pull/15311> and a
>>>>>>>>>>>>>>>>>> local benchmark to
>>>>>>>>>>>>>>>>>> > quantify both the pruning impact (plannedFiles →
>>>>>>>>>>>>>>>>>> afterBloom) and the index
>>>>>>>>>>>>>>>>>> > read overhead (statsFiles, statsBytes,
>>>>>>>>>>>>>>>>>> bloomPayloadBytes) for point
>>>>>>>>>>>>>>>>>> > predicates on high-cardinality columns. Please take a
>>>>>>>>>>>>>>>>>> look and let me know
>>>>>>>>>>>>>>>>>> > if you have any questions or feedback.
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Huaxin
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > On Tue, Feb 10, 2026 at 1:43 PM huaxin gao <
>>>>>>>>>>>>>>>>>> hu...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > > Reminder for tomorrow's sync on Iceberg Index Support.
>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>> > > Wednesday: Feb. 11 9:00 – 10:00am
>>>>>>>>>>>>>>>>>> > > Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>> > > Google Meet joining info
>>>>>>>>>>>>>>>>>> > > Video call link: meet.google.com/nsp-ctyr-khk
>>>>>>>>>>>>>>>>>> > > Design doc:
>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.0#heading=h.hs6r9d26w1y2
>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.qouk73o4jxx7
>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>> > > Thanks,
>>>>>>>>>>>>>>>>>> > > Huaxin
>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>> > > On Tue, Feb 3, 2026 at 10:52 PM Péter Váry <
>>>>>>>>>>>>>>>>>> pe...@gmail.com>
>>>>>>>>>>>>>>>>>> > > wrote:
>>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>> > >> Thanks Huaxin and Steven for organizing this.
>>>>>>>>>>>>>>>>>> Looking forward to meet you
>>>>>>>>>>>>>>>>>> > >> all next week!
>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>> > >> On Wed, Feb 4, 2026, 02:48 Steven Wu <
>>>>>>>>>>>>>>>>>> st...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>>> > >>> We set up the dev calendar event with a new google
>>>>>>>>>>>>>>>>>> meet link. Please
>>>>>>>>>>>>>>>>>> > >>> ignore the link from Huaxin's original email.
>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>> > >>> The dev calendar has the correct info (including
>>>>>>>>>>>>>>>>>> the new meeting link)
>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>> > >>> Iceberg Index Support Sync
>>>>>>>>>>>>>>>>>> > >>> Wednesday, February 11 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>>> > >>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>>> > >>> Google Meet joining info
>>>>>>>>>>>>>>>>>> > >>> Video call link:
>>>>>>>>>>>>>>>>>> https://meet.google.com/nsp-ctyr-khk
>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>> > >>> On Tue, Feb 3, 2026 at 5:08 PM huaxin gao <
>>>>>>>>>>>>>>>>>> hu...@gmail.com>
>>>>>>>>>>>>>>>>>> > >>> wrote:
>>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>>> > >>>> Sorry, I meant PST (not EST) :)
>>>>>>>>>>>>>>>>>> > >>>> Looking forward to the discussion!
>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>> > >>>> On Tue, Feb 3, 2026 at 4:58 PM Shawn Chang <
>>>>>>>>>>>>>>>>>> ye...@gmail.com>
>>>>>>>>>>>>>>>>>> > >>>> wrote:
>>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>>> > >>>>> Hi Huaxin,
>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>> > >>>>> Thanks for starting the sync!
>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>> > >>>>> The meeting seems to be 9-10AM PST on the dev
>>>>>>>>>>>>>>>>>> events calendar
>>>>>>>>>>>>>>>>>> > >>>>> <
>>>>>>>>>>>>>>>>>> https://calendar.google.com/calendar/u/0?cid=MzkwNWQ0OTJmMWI0NTBiYTA3MTJmMmFlNmFmYTc2ZWI3NTdmMTNkODUyMjBjYzAzYWE0NTI3ODg1YWRjNTYyOUBncm91cC5jYWxlbmRhci5nb29nbGUuY29t
>>>>>>>>>>>>>>>>>> >,
>>>>>>>>>>>>>>>>>> > >>>>> not EST. Maybe it's a typo?
>>>>>>>>>>>>>>>>>> > >>>>> Otherwise, looking forward to the discussion!
>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>> > >>>>> Best,
>>>>>>>>>>>>>>>>>> > >>>>> Shawn
>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>> > >>>>> On Tue, Feb 3, 2026 at 9:18 AM huaxin gao <
>>>>>>>>>>>>>>>>>> hu...@gmail.com>
>>>>>>>>>>>>>>>>>> > >>>>> wrote:
>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>> > >>>>>> Hi all,
>>>>>>>>>>>>>>>>>> > >>>>>> I'd like to start a dedicated sync to discuss
>>>>>>>>>>>>>>>>>> Iceberg Index support.
>>>>>>>>>>>>>>>>>> > >>>>>> Here is the existing discussion thread:
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> https://lists.apache.org/thread/fzqk3jjf0xpj5m4cfqb3v4c65p0t04ty
>>>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>> To ground the discussion, here are the two
>>>>>>>>>>>>>>>>>> proposals:
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>> - Peter's proposal
>>>>>>>>>>>>>>>>>> > >>>>>> <
>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.0#heading=h.hs6r9d26w1y2>
>>>>>>>>>>>>>>>>>> (overall
>>>>>>>>>>>>>>>>>> > >>>>>> index support)
>>>>>>>>>>>>>>>>>> > >>>>>> - My proposal
>>>>>>>>>>>>>>>>>> > >>>>>> <
>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.qouk73o4jxx7
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > >>>>>> (bloom filter skipping index)
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>> Time slot: Every 3 weeks, Wednesdays at 9 AM to
>>>>>>>>>>>>>>>>>> 10 AM EST, starting
>>>>>>>>>>>>>>>>>> > >>>>>> next Wednesday (2/11). After FileFormat sync
>>>>>>>>>>>>>>>>>> finishes, we plan to use that
>>>>>>>>>>>>>>>>>> > >>>>>> slot and switch to every other Monday, 9 AM to
>>>>>>>>>>>>>>>>>> 10 AM EST.
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>> Meet link: https://meet.google.com/fjn-tyze-mko
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>>>>>>>>> > >>>>>> Huaxin
>>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>

Re: Dedicated sync for Iceberg Index Support

[huaxin gao <hu...@gmail.com>]

Hi everyone,

Here is a summary of yesterday's index sync. Thanks to everyone who joined.

1. Index spec (#16961 <https://github.com/apache/iceberg/pull/16961>)
We went over the two questions Péter listed:
- Region file boundary: the per-column group max value is replaced by a
top-level max_ordering_key in each tracking file entry.
- Region file schema: the index now stores its own schema, with a mapping
from table fields to index fields, so index field IDs are independent of
the table's. Computed values are derived columns, either stored in the
region files (like a SQL generated column) or kept only as stats. This
depends on the derived columns proposal for V4, and the index keeps its
schema history (for example after int to long promotion).
There are no open issues left on the spec PR. Please review:
https://github.com/apache/iceberg/pull/16961

2. Index REST spec
Agreed:
- An index must be in the same namespace as its table. Cross-namespace
indexes are dropped.
- Rename and register are left out of the first version, to reduce
complexity. We can add them later if there is demand.

Main open question: should index routes be namespace-level (as in the PR),
or nested under the table?
Prior art differs: MySQL scopes indexes to a table (DROP INDEX idx ON tbl),
while Postgres makes index names unique within a schema (DROP INDEX idx).
Lance nests indexes under the table.
Next step: look at how major databases and REST APIs model indexes, then
revisit the route design at the next sync.

Action items
- Everyone: review the index spec PR #16961
<https://github.com/apache/iceberg/pull/16961>.
- Huaxin: research how MySQL, Postgres, and other databases and REST APIs
model, list, and name indexes, and whether rename has real use cases.

Recording and notes:
https://drive.google.com/drive/folders/1iodHASJBVwNtDYEINq7k2HjogPsYQocH

Thanks,
Huaxin

On Tue, Sep 15, 2026 at 7:56 AM Péter Váry <pe...@gmail.com>
wrote:

> Hi everyone,
>
> Here is the recording of the Monday sync:
> https://drive.google.com/drive/folders/1iodHASJBVwNtDYEINq7k2HjogPsYQocH
>
> A brief summary of the discussion:
>
>    - Bloom Filter Index
>       - Puffin reduces file-open overhead, while Bloom filter
>       effectiveness appears independent of file placement.
>       - The observed scaling issues are likely due to rate limiting.
>       - Manifest alignment would be required for parallel planning.
>       - A scalar index is expected to deliver similar or better
>       performance.
>       - We will *prioritize scalar indexes* before pursuing additional
>       index types.
>       - The tests are still useful to understand the decisions in the
>       design stage.
>    - Scalar Index Design
>       - *Region file schema: identity, materialized, and non-materialized
>       fields*.
>       - Range files will use stable index-specific IDs, while expressions
>       reference table field IDs.
>       - Scalar indexes require both sorting and clustering; a dedicated
>       index definition may be cleaner than reusing table sort/cluster definitions.
>
> We can continue the remaining discussions offline or in the next sync.
>
> There are only a few open questions, so the spec PR is ready for review:
> https://github.com/apache/iceberg/pull/16961.
>
> Thanks,
> Peter
>
>
> Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026. szept.
> 14., H, 9:07):
>
>> One correction to the previous summary: I originally wrote that *Range*
>> files would store the index data, but I was later corrected that the
>> decision was actually to use *Region* files.
>>
>> The correct index file naming is:
>>
>>    - *Tracking file*: contains the list of region files (analogous to a
>>    manifest file for tables).
>>    - *Region file*: contains the index data rows (analogous to a data
>>    file for tables).
>>
>> Sorry for the confusion,
>> Peter
>>
>> Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026. szept.
>> 3., Cs, 13:18):
>>
>>>
>>> Hi everyone,
>>>
>>> Here is the recording of the Monday sync:
>>>
>>> https://apache-iceberg.slack.com/files/U06538UFDH6/F0BTT739Y7P/impromptu_google_meet_meeting_-_aug_31_2026.mp4
>>>
>>> Thanks, Shawn, for making it available.
>>>
>>> A brief summary of the discussion:
>>>
>>>    - We finalized the file naming:
>>>       - Tracking file: contains the list of range files (similar to a
>>>       manifest file for tables).
>>>       - Range file: contains the index data rows (similar to a data
>>>       file for tables).
>>>    - We agreed not to prescribe how expression values are materialized
>>>    (for example, Hilbert or hash transformations), leaving room for
>>>    implementation flexibility.
>>>    - Ryan mentioned that he is working on a v4 proposal for generating
>>>    Field IDs for non-materialized columns. The current thinking is that index
>>>    columns should follow the same approach.
>>>    - We agreed to retain older versions of the index metadata JSON.
>>>    - We agreed not to require a strict one-to-one mapping between index
>>>    snapshots and table snapshots. An index may contain multiple snapshots
>>>    corresponding to the same table snapshot.
>>>
>>> Based on the discussion, I have updated the spec PR. Following Ryan's
>>> pattern for non-materialized column definitions, the proposal now
>>> introduces:
>>>
>>>    - *materialized-fields* for range file columns
>>>    - *non-materialized-fields* for tracking file statistics
>>>    - *cluster-spec* to define which columns participate in range file
>>>    clustering
>>>
>>> Overall, I think we reached good alignment on the remaining design
>>> points, and I do not see any open questions at the moment. The PR is in
>>> good shape for review, so I would greatly appreciate any feedback or
>>> comments.
>>>
>>> Thanks everyone for the productive discussion and continued
>>> collaboration.
>>> Peter
>>>
>>> Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026. aug.
>>> 18., K, 14:17):
>>>
>>>> Hi everyone,
>>>>
>>>> Unfortunately, yesterday's sync was not recorded due to a technical
>>>> issue. :(
>>>>
>>>> Here is a brief summary of the discussion and the decisions we reached:
>>>>
>>>>    1. We agreed to use a list of Iceberg expressions to define the
>>>>    *index_keys*, which determine how the index is ordered.
>>>>    2. We agreed not to restrict the functions that can be used in
>>>>    those expressions. Engines that do not understand a particular expression
>>>>    can simply choose not to use the index.
>>>>    3. We agreed that index region files (leaf files) should be sorted
>>>>    by the values generated by the index_keys.
>>>>    4. We discussed using *included_column_ids* to define the contents
>>>>    and schema of the index region files, and generally felt this could be a
>>>>    good approach.
>>>>    5. We started discussing whether the results of the index_key
>>>>    expressions should also be stored in the index region files alongside the
>>>>    included columns. In some cases, storing them could help readers optimize
>>>>    lookups within a region file. In other cases, the values could be derived
>>>>    when needed, making them redundant and potentially hurting performance by
>>>>    increasing the size of the region files.
>>>>
>>>> I think we should continue the discussion around point 5, as the
>>>> outcome may influence, or even reopen, the decisions around point 4.
>>>>
>>>> Thanks!
>>>>
>>>>
>>>> Flavio Junqueira <fl...@gmail.com> ezt írta (időpont: 2026.
>>>> aug. 4., K, 18:57):
>>>>
>>>>> Thank you for the summary, the proposal and the overall discussion. I
>>>>> have one main question about the target and I apologize in advance if I'm
>>>>> missing some important detail as I might not have the full context.
>>>>>
>>>>> I noticed that partitioning, sorting and clustering (via Hilbert) were
>>>>> discussed in the call in this context of index transform functions. My
>>>>> understanding is that engines currently are already pruning partitions and
>>>>> files using partition information and table metadata like per-column
>>>>> min-max. If that's the case, then what's it that this proposal is trying to
>>>>> achieve by having transform functions that capture partitioning, sorting
>>>>> and clustering when engines are already able to prune without a secondary
>>>>> index? I can more easily see the primary key index example, but I'm not
>>>>> entirely sure about these data layout mechanisms.
>>>>>
>>>>> I also didn't quite get how engines are expected to consume such an
>>>>> index. There was a discussion about where the mapping of the input to files
>>>>> happens, whether that's in Iceberg or the engine, and I didn't quite get
>>>>> the recommendation. Checking Peter's proposal, I couldn't see it clearly
>>>>> either, and if I just missed it, I'd appreciate a pointer to where I should
>>>>> look into.
>>>>>
>>>>> -Flavio
>>>>>
>>>>> On 4 Aug 2026, at 10:52, Péter Váry <pe...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi everyone,
>>>>>
>>>>> Here is the recording from yesterday’s sync:
>>>>> https://www.youtube.com/watch?v=Wg9orP9JAmk
>>>>>
>>>>> To summarize:
>>>>> - We agreed to retain the history of index snapshots, but not the
>>>>> history of other index properties. If readers require information that may
>>>>> later be modified by a user, that information should be copied into the
>>>>> corresponding snapshot.
>>>>> - We discussed the possibility of defining index ordering using a list
>>>>> of transform functions.
>>>>>
>>>>> Dan and Yingyi suggested something along the following lines:
>>>>>
>>>>> *"index_keys" : [*
>>>>> *  {*
>>>>> *    "type": "apply",*
>>>>> *    "func-name": {*
>>>>> *      "catalog": "iceberg_functions",*
>>>>> *      "name": "day"*
>>>>> *    },*
>>>>> *    "arguments": [*
>>>>> *      { "type": "reference", "id": 3 }*
>>>>> *    ]*
>>>>> *  },*
>>>>> *  {*
>>>>> *    "type": "apply",*
>>>>> *    "func-name": {*
>>>>> *      "catalog": "iceberg_functions",*
>>>>> *      "name": "truncate"*
>>>>> *    },*
>>>>> *    "arguments": [*
>>>>> *      { "type": "reference", "id": 4 },*
>>>>> *      2*
>>>>> *    ]*
>>>>> *  }*
>>>>> *]*
>>>>>
>>>>>
>>>>> In this example, rows would first be ordered by day(col3) and then by
>>>>> truncate(col4, 2). To ensure a deterministic ordering, ties would be broken
>>>>> using the original values of columns 3 and 4. If those are still equal,
>>>>> ordering would fall back to the file name and finally the row position.
>>>>>
>>>>> I spent some time thinking about this after the sync, and here are my
>>>>> thoughts.
>>>>>
>>>>> Pros:
>>>>> - Provides a high degree of flexibility.
>>>>> - Enables expression-based indexes.
>>>>>
>>>>> Cons:
>>>>> - Do we need this flexibility at this stage?
>>>>>    - Equality deletes, the primary use cases we are targeting, do not
>>>>> require expressions.
>>>>>    - Most indexes I have seen in production are not expression-based.
>>>>> - Increased complexity for engines:
>>>>>    - Engines must parse and understand transforms to determine whether
>>>>> using an index can improve a query. For example, is ordering by
>>>>> day(timestamp) sufficient for a key lookup?
>>>>>    - Complexity increases further as new UDFs are introduced.
>>>>>    - Expressions are not pushed into scans today, so index handling
>>>>> cannot be fully hidden behind Iceberg scan planning.
>>>>> - Risk of fragmentation:
>>>>>    - The additional complexity may slow engine adoption.
>>>>>    - Adopters may choose to support only exact transform matches
>>>>> created by them, rather than learn to reason about the general transforms.
>>>>>    - UDFs are prone to missing implementations and behavioral
>>>>> differences across engines. In the worst case, such differences could lead
>>>>> to incorrect query results when indexes are used.
>>>>>
>>>>> Given the points above, I still prefer defining HASH, HILBERT,
>>>>> IDENTITY as a set of standardized values combined with an explicit list of
>>>>> key columns. This keeps adoption simple, unlocks the primary use cases we
>>>>> are targeting today, and allows the Iceberg library to provide the
>>>>> transform implementations directly.
>>>>>
>>>>> Internally, these transforms could still be represented as Iceberg
>>>>> expressions, giving us a migration path toward future index types that may
>>>>> require full expression support.
>>>>>
>>>>> For example:
>>>>>
>>>>> *{*
>>>>> *  "transform": "HILBERT",*
>>>>> *  "key-column-ids": [3, 4]*
>>>>> *}*
>>>>>
>>>>>
>>>>> I'm interested in hearing other opinions.
>>>>>
>>>>> Thanks,
>>>>> Peter
>>>>>
>>>>> Renjie Liu <li...@gmail.com> ezt írta (időpont: 2026. júl.
>>>>> 28., K, 10:07):
>>>>>
>>>>>> Hi, Peter:
>>>>>>
>>>>>> Thanks for driving the proposal.
>>>>>>
>>>>>> >  *Hash Index Type*: Should hash-based indexes be a separate index
>>>>>> type? They are only applicable to equality and IN predicates, and the hash
>>>>>> algorithm must always be known. A dedicated HASH type could make these
>>>>>> constraints explicit and distinguish hash-based indexes from ordered
>>>>>> indexes.
>>>>>>
>>>>>> I'm leaning toward keeping them in scalar index type due to the
>>>>>> similarity with others transformation functions like identity. As with the
>>>>>> constraints, I think they are properties of transformation functions.
>>>>>>
>>>>>> > *Transform Function/Layout Representation*: Should transform
>>>>>> functions be represented as an enum or a string? I lean toward strings, but
>>>>>> I do not have a strong preference. In many cases, an index can still be
>>>>>> used based on standard column statistics, even if a reader does not
>>>>>> recognize a particular transform.
>>>>>>
>>>>>> This seems more like a problem in java implementation?
>>>>>>
>>>>>> Also I prefer the name transform function compared with layout
>>>>>> representation. Layout representation makes me feel that it's not about the
>>>>>> definition of an index function.
>>>>>>
>>>>>> > *Future Index Types*: How should we handle index types that are
>>>>>> expected in the long term but fall outside the scope of this proposal (for
>>>>>> example, VECTOR, IVF, TERM)? Should they be mentioned as planned/reserved
>>>>>> concepts, or omitted entirely? I lean toward at least mentioning them. The
>>>>>> rationale for introducing an index type is easier to explain when
>>>>>> considering future index families.
>>>>>>
>>>>>> +1
>>>>>>
>>>>>> *> Index History Handling:* Most database systems treat index
>>>>>> changes as a drop-and-recreate operation rather than an in-place update. If
>>>>>> we follow the same approach, there is limited value in maintaining index
>>>>>> metadata history. During index maintenance, a new metadata.json can be
>>>>>> created and atomically swapped in, with the catalog responsible for
>>>>>> cleaning up obsolete metadata and index data files according to its
>>>>>> retention policy.
>>>>>>
>>>>>> +1, I think we only need to keep index data for different snapshots,
>>>>>> but if an index definition changed, we should drop and recreate a new index.
>>>>>>
>>>>>> > *File Format Location*: Should file format be specified at the
>>>>>> leaf-file level or only at the index level? My preference is to stay
>>>>>> aligned with the table metadata model and avoid introducing differences
>>>>>> unless there is a compelling requirement. This would simplify
>>>>>> implementation and maximize code reuse.
>>>>>>
>>>>>> +1.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 24, 2026 at 11:57 PM Péter Váry <
>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> Here are the key takeaways from Monday’s meeting (July 20):
>>>>>>>
>>>>>>>    - *Index Type*: An index type enum will be introduced in v1,
>>>>>>>    initially with a single value (SCALAR). This establishes a future-proof API
>>>>>>>    boundary while leaving room for additional index families.
>>>>>>>    - *Strict Total Ordering*: The specification should require
>>>>>>>    transform functions to produce values that preserve a well-defined total
>>>>>>>    ordering across the key space. The transform function does not need to
>>>>>>>    return a long. Instead, leaf files can store the first and last values
>>>>>>>    according to the ordering, enabling generic filtering capabilities for
>>>>>>>    readers.
>>>>>>>
>>>>>>> Open Questions:
>>>>>>>
>>>>>>>    - *Hash Index Type*: Should hash-based indexes be a separate
>>>>>>>    index type? They are only applicable to equality and IN predicates, and the
>>>>>>>    hash algorithm must always be known. A dedicated HASH type could make these
>>>>>>>    constraints explicit and distinguish hash-based indexes from ordered
>>>>>>>    indexes.
>>>>>>>    - *Transform Function/Layout Representation*: Should transform
>>>>>>>    functions be represented as an enum or a string? I lean toward strings, but
>>>>>>>    I do not have a strong preference. In many cases, an index can still be
>>>>>>>    used based on standard column statistics, even if a reader does not
>>>>>>>    recognize a particular transform.
>>>>>>>    - *Future Index Types*: How should we handle index types that
>>>>>>>    are expected in the long term but fall outside the scope of this proposal
>>>>>>>    (for example, VECTOR, IVF, TERM)? Should they be mentioned as
>>>>>>>    planned/reserved concepts, or omitted entirely? I lean toward at least
>>>>>>>    mentioning them. The rationale for introducing an index type is easier to
>>>>>>>    explain when considering future index families.
>>>>>>>    - *Index History Handling:* Most database systems treat index
>>>>>>>    changes as a drop-and-recreate operation rather than an in-place update. If
>>>>>>>    we follow the same approach, there is limited value in maintaining index
>>>>>>>    metadata history. During index maintenance, a new metadata.json can be
>>>>>>>    created and atomically swapped in, with the catalog responsible for
>>>>>>>    cleaning up obsolete metadata and index data files according to its
>>>>>>>    retention policy.
>>>>>>>    - *Min/Max Statistics Representation*: Should transform-specific
>>>>>>>    or sort-order min/max values be represented using the existing file
>>>>>>>    statistics structure, or exposed through dedicated fields? My preference is
>>>>>>>    to keep them within the statistics structure defined by the file schema
>>>>>>>    while exposing them directly through the Java API if needed.
>>>>>>>    - *File Format Location*: Should file format be specified at the
>>>>>>>    leaf-file level or only at the index level? My preference is to stay
>>>>>>>    aligned with the table metadata model and avoid introducing differences
>>>>>>>    unless there is a compelling requirement. This would simplify
>>>>>>>    implementation and maximize code reuse.
>>>>>>>
>>>>>>> I have updated the spec PR based on the decisions above:
>>>>>>> https://github.com/apache/iceberg/pull/16961
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Peter
>>>>>>>
>>>>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont: 2026. jún.
>>>>>>> 26., P, 19:42):
>>>>>>>
>>>>>>>> Hi all, Here is the summary of this Monday's index meeting:
>>>>>>>>
>>>>>>>> Two blocking decisions closed:
>>>>>>>>
>>>>>>>>    1. Index is *not* a table— it's its own object (reuses table
>>>>>>>>    machinery under the hood). Reasoning: requires a sort order, no column
>>>>>>>>    updates, no partition spec, no overlapping ranges between leaves, inherits
>>>>>>>>    base-table permissions, and has its own CREATE/DROP/UPDATE INDEX DDL.
>>>>>>>>    2. Index is a separate catalog entity, with no pointers in
>>>>>>>>    table metadata— has its own REST endpoints; the catalog can optionally
>>>>>>>>    return index metadata withloadTable to avoid extra round-trips. Keeps table
>>>>>>>>    and index updates independent/async.
>>>>>>>>
>>>>>>>>
>>>>>>>> Next steps: Start writing the spec and build out the copy-on-write
>>>>>>>> path now.
>>>>>>>>
>>>>>>>> Here are the draft spec:
>>>>>>>> secondary index spec <https://github.com/apache/iceberg/pull/16961>
>>>>>>>> irc spec <https://github.com/apache/iceberg/pull/16963>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Huaxin
>>>>>>>>
>>>>>>>> On Sat, Jun 20, 2026 at 11:11 AM huaxin gao <hu...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> I built a standalone PoC to validate that the basic index
>>>>>>>>> structure works: that we can build a PK index, convert equality deletes to
>>>>>>>>> position deletes through it, and have every converted delete land on the
>>>>>>>>> correct live row. I ran it up to *100M keys*.
>>>>>>>>>
>>>>>>>>> *Headline: the structure works.* The index builds over up to 100M
>>>>>>>>> keys, the eq-delete → position-delete conversion resolved correctly at
>>>>>>>>> *every* size (100% of converted deletes mapped to the right live
>>>>>>>>> row), and the resulting position deletes are *~8× cheaper to
>>>>>>>>> apply* at query time than the equality deletes they replace.
>>>>>>>>>
>>>>>>>>> Beyond correctness, the run also shows how the index’s
>>>>>>>>> *maintenance* cost scales, comparing copy-on-write (COW, rewrite
>>>>>>>>> touched leaves) vs an append/merge (MOR) option, under a realistic mixed
>>>>>>>>> CDC checkpoint (1,000 insert + 500 update + 500 delete), local wall-clock:
>>>>>>>>> keys EQ baseline INDEX (COW) % of 60s (COW) INDEX (MOR) % of 60s
>>>>>>>>> (MOR) correct
>>>>>>>>> 5M 6 ms 6.7s 11.2% 2.2s 3.7% PASS
>>>>>>>>> 20M 8 ms 24.2s 40.4% 6.4s 10.6% PASS
>>>>>>>>> 50M 7 ms 51.6s 86.1% 12.2s 20.4% PASS
>>>>>>>>> *100M* 6 ms *75.0s* 125% (BEHIND) *16.9s* 28.2% (keeps up) PASS
>>>>>>>>>
>>>>>>>>> COW maintenance crosses the 60 s checkpoint around 100M
>>>>>>>>> (75 s/cycle, 125%); MOR stays at ~28% and keeps pace; the equality-delete
>>>>>>>>> baseline is ~6 ms and flat. So the structure works, but *COW
>>>>>>>>> alone can’t sustain scattered CDC at hundreds of millions of keys on a
>>>>>>>>> single writer*. It’s worth allowing a merge-on-read / update-file
>>>>>>>>> maintenance option alongside COW (or sharding the index across parallel
>>>>>>>>> writers).
>>>>>>>>>
>>>>>>>>> *Full write-up, all tables, and the in-region reality-check:* link
>>>>>>>>> <https://docs.google.com/document/d/1G3zxbW8X0eU3UrouslZfp42bBc9CvgJGnJyDONCB4PU/edit?tab=t.0>
>>>>>>>>>
>>>>>>>>> Feedback welcome, especially on the spec direction (whether to
>>>>>>>>> allow a merge-on-read / update-file maintenance option alongside COW)  and
>>>>>>>>> on the read-side modeling.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Huaxin
>>>>>>>>>
>>>>>>>>> On Tue, Jun 9, 2026 at 5:45 PM huaxin gao <hu...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Sorry,  we've skipped posting a few of the dedicated index-sync
>>>>>>>>>> summaries to the mailing list; you can find those in the Google
>>>>>>>>>> doc
>>>>>>>>>> <https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?pli=1&tab=t.8041k7j2n7y3>
>>>>>>>>>> and the Slack channel. Here's yesterday's summary:
>>>>>>>>>>
>>>>>>>>>> *Decided*
>>>>>>>>>>
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    Index vs. table (what we agreed):
>>>>>>>>>>    - Reuse table implementation/library code and a
>>>>>>>>>>       near-identical spec — the commit path will be custom regardless, so reuse
>>>>>>>>>>       isn't the deciding factor.
>>>>>>>>>>       - An index is not a table from a user/API view: loading or
>>>>>>>>>>       writing an index as a table must fail(it would violate index invariants).
>>>>>>>>>>       - The spec forbids most table behaviors: no overlapping
>>>>>>>>>>       files, one mandatory transform sort order, no column updates, no partition
>>>>>>>>>>       spec.
>>>>>>>>>>       - Delete vectors: reuse Iceberg's existing DV — benchmarks
>>>>>>>>>>    showed no new delete format is worth introducing.
>>>>>>>>>>    - Incremental updates: start with copy-on-write only (no
>>>>>>>>>>    update files). For object-store-sized leaves, a full leaf rewrite is about
>>>>>>>>>>    as cheap as maintaining an overlay update file + DV, so we'll skip the MOR
>>>>>>>>>>    machinery for now and add it later only if benchmarks prove we need it
>>>>>>>>>>    (likely just the very-large-leaf case).
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    Validate the spec first: build a quick, hand-wired prototype
>>>>>>>>>>    (Parquet files structured per the spec) and benchmark it on real scales
>>>>>>>>>>    before formalizing.
>>>>>>>>>>
>>>>>>>>>> *Leaning, not final*
>>>>>>>>>>
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    Indexes are likely separate catalog objects, linked from the
>>>>>>>>>>    table by storing just an identifier (like materialized views) and not
>>>>>>>>>>    visible in LIST TABLES.
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    We'll need a commit path for indexes, but simpler than tables
>>>>>>>>>>    (no stage-create).
>>>>>>>>>>
>>>>>>>>>> *Still open*
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    Permissions model — separate vs. inherited (action: look at
>>>>>>>>>>    what real DBs do for index permissions).
>>>>>>>>>>    - REST/catalog RPC design — minimize round-trips; index
>>>>>>>>>>    metadata ideally returned with LOAD TABLE. Catalog RPC cost
>>>>>>>>>>    may dominate Parquet IO, so this needs real design.
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    Scale modeling — target rows-per-leaf vs. leaf size vs.
>>>>>>>>>>    metadata-file count.
>>>>>>>>>>    -
>>>>>>>>>>
>>>>>>>>>>    DDL-on-index semantics (reuse table schema-update actions or
>>>>>>>>>>    separate)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Huaxin
>>>>>>>>>>
>>>>>>>>>> On Wed, Apr 22, 2026 at 8:47 AM Péter Váry <
>>>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi All,
>>>>>>>>>>>
>>>>>>>>>>> TL;DR
>>>>>>>>>>> We still need to validate with ADLS and S3, but based on the
>>>>>>>>>>> local tests, the MPHF approach looks more promising if we can tolerate
>>>>>>>>>>> larger files and longer index maintenance times.
>>>>>>>>>>>
>>>>>>>>>>> Details:
>>>>>>>>>>> Here are the results from the local experiments on my Mac. I
>>>>>>>>>>> removed unnecessary statistics from the Parquet files and tested different
>>>>>>>>>>> row group sizes:
>>>>>>>>>>>
>>>>>>>>>>>    - For an index file with 1M records, a row group size of
>>>>>>>>>>>    5,000 appears to be the sweet spot.
>>>>>>>>>>>    - For 10M records, 10,000 rows per row group works best.
>>>>>>>>>>>
>>>>>>>>>>> If you have additional ideas for optimizing Parquet-based
>>>>>>>>>>> indexes, I’d be very interested to hear them.
>>>>>>>>>>> The test code is available on this branch:
>>>>>>>>>>> https://github.com/pvary/iceberg/tree/leaf_bench
>>>>>>>>>>>
>>>>>>>>>>> Best results:
>>>>>>>>>>> *1m records/file*
>>>>>>>>>>>
>>>>>>>>>>>    - Parquet - 5000 row/RowGroup
>>>>>>>>>>>       - Read: 1191 µs - 1 file open, 3 seek, 123KB read per
>>>>>>>>>>>       lookup
>>>>>>>>>>>       - Write: 1.7 s, 15 MB
>>>>>>>>>>>    - MPHF
>>>>>>>>>>>       - Read: 202 µs - 1 file open, 1 seek,  282KB read per
>>>>>>>>>>>       lookup
>>>>>>>>>>>       - Write: 0.8 s, 34 MB
>>>>>>>>>>>
>>>>>>>>>>> *10m records/file*
>>>>>>>>>>>
>>>>>>>>>>>    - Parquet - 10000 row/RowGroup
>>>>>>>>>>>       - Read: 4168 µs - 1 file open, 3 seek, 395KB read per
>>>>>>>>>>>       lookup
>>>>>>>>>>>       - Write: 19.5s s, 144 MB
>>>>>>>>>>>    - MPHF
>>>>>>>>>>>       - Read: 1086 µs - 1 file open, 1 seek,  2.8 MB (2812KB)
>>>>>>>>>>>       read per lookup
>>>>>>>>>>>       - Write: 6.5 s, 34 MB, 353 MB
>>>>>>>>>>>
>>>>>>>>>>> Below are the full results.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> *Benchmark                                      (indexType)
>>>>>>>>>>>  (keyType)  (numRows)  Mode    Cnt            Score          Error
>>>>>>>>>>>  UnitsInvertedIndexBenchmark.lookup                 PARQUET_1000       LONG
>>>>>>>>>>>    1000000    ss  10000         3285.284 ±        5.138
>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_1000       LONG
>>>>>>>>>>>    1000000    ss  10000   2522168989.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_1000       LONG
>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_1000       LONG
>>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_1000       LONG
>>>>>>>>>>> 10000000    ss  10000        35449.614 ±       34.673
>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_1000       LONG
>>>>>>>>>>>   10000000    ss  10000  24302649201.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_1000       LONG
>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_1000       LONG
>>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_5000       LONG
>>>>>>>>>>>  1000000    ss  10000         1191.959 ±        4.169
>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_5000       LONG
>>>>>>>>>>>    1000000    ss  10000   1230877229.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_5000       LONG
>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_5000       LONG
>>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_5000       LONG
>>>>>>>>>>> 10000000    ss  10000         7236.447 ±       10.374
>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_5000       LONG
>>>>>>>>>>>   10000000    ss  10000   5650715973.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_5000       LONG
>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_5000       LONG
>>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_10000       LONG
>>>>>>>>>>>  1000000    ss  10000         1349.946 ±        7.834
>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_10000       LONG
>>>>>>>>>>>    1000000    ss  10000   1730219377.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_10000       LONG
>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_10000       LONG
>>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_10000       LONG
>>>>>>>>>>> 10000000    ss  10000         4168.635 ±       11.051
>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_10000       LONG
>>>>>>>>>>>   10000000    ss  10000   3946341532.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_10000       LONG
>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_10000       LONG
>>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_50000       LONG
>>>>>>>>>>>  1000000    ss  10000         4736.466 ±       38.179
>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_50000       LONG
>>>>>>>>>>>    1000000    ss  10000   7427413541.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_50000       LONG
>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_50000       LONG
>>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_50000       LONG
>>>>>>>>>>> 10000000    ss  10000         4979.031 ±       34.708
>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_50000       LONG
>>>>>>>>>>>   10000000    ss  10000   7694887636.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_50000       LONG
>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_50000       LONG
>>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup                         MPHF       LONG
>>>>>>>>>>>  1000000    ss  10000          202.571 ±        2.336
>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead               MPHF       LONG
>>>>>>>>>>>    1000000    ss  10000   2821570000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams             MPHF       LONG
>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks                   MPHF       LONG
>>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup                         MPHF       LONG
>>>>>>>>>>> 10000000    ss  10000         1086.957 ±        4.524
>>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead               MPHF       LONG
>>>>>>>>>>>   10000000    ss  10000  28119460000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams             MPHF       LONG
>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks                   MPHF       LONG
>>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_1000       LONG
>>>>>>>>>>>  1000000    ss      3      1720731.014 ±   876636.004
>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_1000       LONG
>>>>>>>>>>>    1000000    ss      3     46453317.000
>>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_1000       LONG
>>>>>>>>>>> 10000000    ss      3     18547947.876 ± 12258125.307
>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_1000       LONG
>>>>>>>>>>>   10000000    ss      3    452655675.000
>>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_5000       LONG
>>>>>>>>>>>  1000000    ss      3      1718345.583 ±  1103928.016
>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_5000       LONG
>>>>>>>>>>>    1000000    ss      3     44845788.000
>>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_5000       LONG
>>>>>>>>>>> 10000000    ss      3     18604229.931 ±  2668361.915
>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_5000       LONG
>>>>>>>>>>>   10000000    ss      3    435388818.000
>>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_10000       LONG
>>>>>>>>>>>  1000000    ss      3      1761555.389 ±   535857.675
>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_10000       LONG
>>>>>>>>>>>    1000000    ss      3     44536635.000
>>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_10000       LONG
>>>>>>>>>>> 10000000    ss      3     19501588.264 ±  2130054.558
>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_10000       LONG
>>>>>>>>>>>   10000000    ss      3    433189623.000
>>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_50000       LONG
>>>>>>>>>>>  1000000    ss      3      1936624.889 ±  6601363.985
>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_50000       LONG
>>>>>>>>>>>    1000000    ss      3     44264655.000
>>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_50000       LONG
>>>>>>>>>>> 10000000    ss      3     20471742.278 ± 10705206.310
>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_50000       LONG
>>>>>>>>>>>   10000000    ss      3    431311305.000
>>>>>>>>>>> #InvertedIndexBenchmark.write                          MPHF       LONG
>>>>>>>>>>>  1000000    ss      3       896573.958 ±  1408024.851
>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes           MPHF       LONG
>>>>>>>>>>>    1000000    ss      3    102846369.000
>>>>>>>>>>> #InvertedIndexBenchmark.write                          MPHF       LONG
>>>>>>>>>>> 10000000    ss      3      6509348.875 ± 15519975.479
>>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes           MPHF       LONG
>>>>>>>>>>>   10000000    ss      3   1058435733.000                     #*
>>>>>>>>>>>
>>>>>>>>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont: 2026.
>>>>>>>>>>> ápr. 21., K, 20:53):
>>>>>>>>>>>
>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>
>>>>>>>>>>>> In recent secondary index sync meetings, the discussion
>>>>>>>>>>>> converged on the need to define what an index is from first principles
>>>>>>>>>>>> before settling on physical layout.
>>>>>>>>>>>>
>>>>>>>>>>>> To address that, Peter and I have drafted a requirements
>>>>>>>>>>>> document for a key lookup index (renamed from "primary key index" to avoid
>>>>>>>>>>>> implying uniqueness enforcement), the goal is to nail down one well-scoped
>>>>>>>>>>>> index type first.
>>>>>>>>>>>>
>>>>>>>>>>>> Doc: Key Lookup Index Requirements
>>>>>>>>>>>> <https://docs.google.com/document/d/1e0zxK-jA0LBDq8YQlQgFipTHelDFiga8lCkgDTmYub8/edit?tab=t.0#heading=h.8shrgabvl19>
>>>>>>>>>>>>
>>>>>>>>>>>> It covers requirements, three design options (manifest + sorted
>>>>>>>>>>>> Parquet, hash + sorted Parquet, hash + MPHF) and open questions. We will
>>>>>>>>>>>> add preliminary benchmark results shortly.
>>>>>>>>>>>>
>>>>>>>>>>>> Feedback welcome — inline in the doc, on this thread, or at the
>>>>>>>>>>>> next index sync.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> Huaxin
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Apr 13, 2026 at 7:22 AM Steven Wu <st...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Do we need the special index identifier that was originally
>>>>>>>>>>>>> proposed? A generic CatalogObjectIdentifier (with namespace and name) would
>>>>>>>>>>>>> be consistent with all object types in the catalog. I have a discussion
>>>>>>>>>>>>> thread on the generic identifier topic: [DISCUSS] REST Spec:
>>>>>>>>>>>>> generic CatalogObjectIdentifier.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Should we add an indexes array field to table metadata? It
>>>>>>>>>>>>> only contains a list of index object identifiers. It doesn't contain any
>>>>>>>>>>>>> index metadata which should live in the index objects. Yufei was trying to
>>>>>>>>>>>>> bring this up at the end of the first sync. But we didn't get enough time
>>>>>>>>>>>>> to really discuss it. It will be great to discuss this as the first agenda
>>>>>>>>>>>>> item today.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Apr 13, 2026 at 3:17 AM Péter Váry <
>>>>>>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We had several engaging discussions at the Iceberg Summit,
>>>>>>>>>>>>>> and it was great to finally catch up with many of you in person. We truly
>>>>>>>>>>>>>> missed those who couldn’t attend, hopefully we’ll all meet again at the
>>>>>>>>>>>>>> next summit.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To keep the conversation going, Huaxin and I have put
>>>>>>>>>>>>>> together the agenda for our next meeting. As a reminder, we’ll meet on *April
>>>>>>>>>>>>>> 13th, 9:00–10:00 AM *PDT (6:00–7:00 PM CEST).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Proposed agenda:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - Continue first-principles index design discussion from
>>>>>>>>>>>>>>    Mar 30
>>>>>>>>>>>>>>       - *Index Ownership and Write Responsibility*
>>>>>>>>>>>>>>          - Should writers be allowed to update indexes, or
>>>>>>>>>>>>>>          - Should all index writes be handled exclusively by
>>>>>>>>>>>>>>          the Index Maintenance process?
>>>>>>>>>>>>>>          - If writers can update indexes then we need to
>>>>>>>>>>>>>>          define what guarantees are required (compaction, file splitting, layout
>>>>>>>>>>>>>>          expectations)?
>>>>>>>>>>>>>>          - If only Index Maintenance updates indexes then we
>>>>>>>>>>>>>>          only need to define what observable properties should be exposed to
>>>>>>>>>>>>>>          consumers? Like:
>>>>>>>>>>>>>>             - Expected max files for a single key
>>>>>>>>>>>>>>             - Current max files for a single key
>>>>>>>>>>>>>>             - Deletes allowed/present
>>>>>>>>>>>>>>             - Sorted by
>>>>>>>>>>>>>>             - Partitioned by
>>>>>>>>>>>>>>          - *Specification Scope: What Belongs in the Spec?*
>>>>>>>>>>>>>>          - Related to the ownership question above
>>>>>>>>>>>>>>          - Light spec: Just define that the index table
>>>>>>>>>>>>>>          should be optimized for retrieval by key columns and the index columns
>>>>>>>>>>>>>>          should be contained in the table. This could give us more flexibility if
>>>>>>>>>>>>>>          better organization methods come up, or
>>>>>>>>>>>>>>          - Detailed spec: We could define the max number of
>>>>>>>>>>>>>>          files per index to read for a single key, or even the partitioning and the
>>>>>>>>>>>>>>          exact sort order. This could allow more use-cases for a given index, like
>>>>>>>>>>>>>>          joins or cardinality estimations.
>>>>>>>>>>>>>>          - I would go for light spec for the main types (PK,
>>>>>>>>>>>>>>          Containing) and only the Index Maintenance processes should update the
>>>>>>>>>>>>>>          Indexes, as for many use-cases the details are not important, and writers
>>>>>>>>>>>>>>          will very rarely update the Indexes themselves.
>>>>>>>>>>>>>>       - *Logical Placement of Indexes*
>>>>>>>>>>>>>>          - Index as a child object of an Iceberg Table, or
>>>>>>>>>>>>>>          - Index as a first‑class entity under
>>>>>>>>>>>>>>          /namespace/indexes/{index}
>>>>>>>>>>>>>>          - Based on the discussions on the summit we are
>>>>>>>>>>>>>>          leaning in this direction. This means the index id should be unique in the
>>>>>>>>>>>>>>          namespace but helps the catalog implementations quite a bit
>>>>>>>>>>>>>>       - *Physical Placement of Index Data*
>>>>>>>>>>>>>>          - I don’t think we should specify this. We should
>>>>>>>>>>>>>>          have a base location for the index, but can rely on the catalog
>>>>>>>>>>>>>>          implementations to decide on their own, like they do with the tables,
>>>>>>>>>>>>>>          views, udfs.
>>>>>>>>>>>>>>       - *Iceberg Reader Based indexes* (Containing indexes
>>>>>>>>>>>>>>       and potentially PK indexes). These are the indexes which could be read by
>>>>>>>>>>>>>>       the existing Iceberg readers. We might decide to store the PK index
>>>>>>>>>>>>>>       similarly to an Iceberg Table and treat it as a reader based index.
>>>>>>>>>>>>>>          - What are the table properties/features exposed to
>>>>>>>>>>>>>>          the readers
>>>>>>>>>>>>>>             - Maybe just some behavioral descriptors for the
>>>>>>>>>>>>>>             optimizer to decide if the index could be used or should be skipped, like:
>>>>>>>>>>>>>>                - Expected max files for a single key
>>>>>>>>>>>>>>                - max files for a single key
>>>>>>>>>>>>>>                - Deletes allowed/present
>>>>>>>>>>>>>>                - Sorted by
>>>>>>>>>>>>>>                - Partitioned by
>>>>>>>>>>>>>>             - The Tasks when reading the index based on the
>>>>>>>>>>>>>>             filters and projection
>>>>>>>>>>>>>>          - What are the table properties/features exposed to
>>>>>>>>>>>>>>          the Index Maintenance. I think this could be internal to the Index
>>>>>>>>>>>>>>          Maintenance process and might not be exposed through the spec. The Index
>>>>>>>>>>>>>>          Maintenance process could handle this as a standard Iceberg Table and could
>>>>>>>>>>>>>>          be based on the Table Maintenance process, but there might be some totally
>>>>>>>>>>>>>>          different processes.
>>>>>>>>>>>>>>       - It should be possible to add properties to an index
>>>>>>>>>>>>>>       defined by the Index Maintenance process which could be used and updated in
>>>>>>>>>>>>>>       the next Index Maintenance run.
>>>>>>>>>>>>>>    - *PK index storage format benchmark results*
>>>>>>>>>>>>>>       - Flat Parquet (baseline)
>>>>>>>>>>>>>>       - BTree with Parquet leaves
>>>>>>>>>>>>>>       - Vortex
>>>>>>>>>>>>>>    - *Open items / next steps*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Peter
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont: 2026.
>>>>>>>>>>>>>> márc. 23., H, 3:03):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi everyone, I wanted to share an update on the primary key
>>>>>>>>>>>>>>> index work.
>>>>>>>>>>>>>>> Since there are still open questions on whether bloom filter
>>>>>>>>>>>>>>> indexes fit in the secondary index framework or should be treated as
>>>>>>>>>>>>>>> extended stats, I've shifted focus to the primary key index since it's a
>>>>>>>>>>>>>>> clearer fit for the framework.
>>>>>>>>>>>>>>> I've put together a proposal for a primary key
>>>>>>>>>>>>>>> reverse-lookup index that maps each key to its physical location
>>>>>>>>>>>>>>> (file_path, row_position). It enables:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    - Scan-time file pruning for point lookups
>>>>>>>>>>>>>>>    - Converting key-based deletes into position deletes
>>>>>>>>>>>>>>>    (eliminating equality deletes for Flink CDC)
>>>>>>>>>>>>>>>    - Accelerating Spark MERGE INTO by replacing full-table
>>>>>>>>>>>>>>>    joins with direct file lookups
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Proposal:
>>>>>>>>>>>>>>> https://docs.google.com/document/d/1HuhCZ0n2FqDh8yqQb9oEj1CPM5yXpEsMPGZno2aSf8E/edit?tab=t.0#heading=h.tbevg4q0m9
>>>>>>>>>>>>>>> Feedback welcome!
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Huaxin
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Mar 18, 2026 at 11:42 PM Péter Váry <
>>>>>>>>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Key takeaways from the general index discussion at the May
>>>>>>>>>>>>>>>> 16 meeting.
>>>>>>>>>>>>>>>> Thanks to everyone who participated! The recording is
>>>>>>>>>>>>>>>> available here: https://www.youtube.com/watch?v=btmjhtRWUCE
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - Q: Do we need to tie index types to the algorithms
>>>>>>>>>>>>>>>>    used to access them?
>>>>>>>>>>>>>>>>    - A: From a specification perspective, the goal is to
>>>>>>>>>>>>>>>>    define the storage-level data layout so it can be shared across engines.
>>>>>>>>>>>>>>>>    Engines are free to interpret and use the data as they see fit, but the
>>>>>>>>>>>>>>>>    on-disk data layout itself must be strictly defined and interoperable.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - Q: Should we introduce an additional abstraction
>>>>>>>>>>>>>>>>    layer (e.g., Vector Index) with sub-types such as IVF and DiskANN?
>>>>>>>>>>>>>>>>    - A: This is possible if we decide it is beneficial. I
>>>>>>>>>>>>>>>>    explored potential naming, but it is not yet clear how such a layer would
>>>>>>>>>>>>>>>>    be used in practice.
>>>>>>>>>>>>>>>>    *Question to Yingyi Bu*: could you provide examples
>>>>>>>>>>>>>>>>    where this additional layer would be useful? Should this abstraction be
>>>>>>>>>>>>>>>>    defined at the spec level, or is it better handled at the engine level?
>>>>>>>>>>>>>>>>    My initial idea was that users would create a generic
>>>>>>>>>>>>>>>>    Vector Index and let the engine choose the concrete implementation.
>>>>>>>>>>>>>>>>    However, this would limit user control and users likely need to specify the
>>>>>>>>>>>>>>>>    exact index representation, which implies they must be aware of the
>>>>>>>>>>>>>>>>    available representations.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - Q: Do we want to allow extensibility for index types?
>>>>>>>>>>>>>>>>    - A: Yes. The intent is to support a small set of
>>>>>>>>>>>>>>>>    well-defined index types while allowing experimentation with new ones. If a
>>>>>>>>>>>>>>>>    new index type proves broadly useful, a follow-up proposal can standardize
>>>>>>>>>>>>>>>>    it and incorporate it into the spec.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - Q: Do we allow multiple versions of an index for the
>>>>>>>>>>>>>>>>    same table snapshot?
>>>>>>>>>>>>>>>>    - A: Yes. Older index versions must be retained for
>>>>>>>>>>>>>>>>    readers that have already started using them, while new readers should
>>>>>>>>>>>>>>>>    automatically use the latest available version
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - Q: Do we need to use materialized views for these
>>>>>>>>>>>>>>>>    indexes?
>>>>>>>>>>>>>>>>    - A: No. These indexes are primarily examples, and
>>>>>>>>>>>>>>>>    different types may require different storage methods. However, the Primary
>>>>>>>>>>>>>>>>    Key, Containing, and parts of the IVF indexes can be structured as Iceberg
>>>>>>>>>>>>>>>>    tables. This allows engines to read them natively; in some cases, Iceberg
>>>>>>>>>>>>>>>>    planners can automatically redirect queries to the index table without
>>>>>>>>>>>>>>>>    engine modifications. Furthermore, index maintenance for these tables can
>>>>>>>>>>>>>>>>    leverage existing materialized view maintenance workflows. Other index
>>>>>>>>>>>>>>>>    types may instead rely on Puffin files or alternative storage approaches.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - Q: How should index metadata be accessed? Should we
>>>>>>>>>>>>>>>>    add explicit pointers for the indexes in the table metadata?
>>>>>>>>>>>>>>>>    - A: We did not have sufficient time to fully explore
>>>>>>>>>>>>>>>>    and conclude this topic.
>>>>>>>>>>>>>>>>    *Question for Yufei Gu*: Did I understand correctly
>>>>>>>>>>>>>>>>    that your main concern stems from endpoint resolution from a REST Catalog
>>>>>>>>>>>>>>>>    perspective? Specifically, if indexes are exposed under a URI such as
>>>>>>>>>>>>>>>>    v1/{prefix}/namespaces/{namespace}/tables/{table}/indexes/{index}, would
>>>>>>>>>>>>>>>>    this make it more difficult for the REST Catalog to resolve and route
>>>>>>>>>>>>>>>>    requests to the appropriate endpoint?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Suhas Jayaram Subramanya via dev <de...@iceberg.apache.org>
>>>>>>>>>>>>>>>> ezt írta (időpont: 2026. márc. 13., P, 23:32):
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Here's a proposal for native Vector Index support in
>>>>>>>>>>>>>>>>> Iceberg tables --
>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1KL4qLOwdqnhOcqTc0EjO1O16NV3M3c-gZCEINDWw4lA/edit?usp=sharing
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We've been working on this proposal with Peter internally
>>>>>>>>>>>>>>>>> at Microsoft and he suggested we post it here to bring this to the
>>>>>>>>>>>>>>>>> community's attention, ahead of the next Secondary Index Sync.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Suhas
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 2026/02/19 04:34:34 huaxin gao wrote:
>>>>>>>>>>>>>>>>> > Hi Everyone,
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Here are the recording and notes from the Iceberg Index
>>>>>>>>>>>>>>>>> Support Sync on
>>>>>>>>>>>>>>>>> > 2/11.
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Recording: https://www.youtube.com/watch?v=3sFfQ0A50yk
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Notes:
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.8041k7j2n7y3
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > The meeting will move to biweekly, Mondays 9–10am PST,
>>>>>>>>>>>>>>>>> starting March 2.
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Since the sync, I updated the Bloom skipping index
>>>>>>>>>>>>>>>>> proposal
>>>>>>>>>>>>>>>>> > <
>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.5r5kl6k3fqwu
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > to address the discussion questions, specifically:
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > - Performance justification: when this helps
>>>>>>>>>>>>>>>>> (high-cardinality = / IN,
>>>>>>>>>>>>>>>>> > many data files, high object-store latency) and how it
>>>>>>>>>>>>>>>>> differs from Parquet
>>>>>>>>>>>>>>>>> > row-group Bloom filters (which still require opening the
>>>>>>>>>>>>>>>>> data file).
>>>>>>>>>>>>>>>>> > - Cost / scalability: rough sizing (Bloom blob size per
>>>>>>>>>>>>>>>>> file, Puffin
>>>>>>>>>>>>>>>>> > file size), the planning cost trade-off (driver index
>>>>>>>>>>>>>>>>> reads vs executor
>>>>>>>>>>>>>>>>> > file opens), and mitigations via caching.
>>>>>>>>>>>>>>>>> > - Lifecycle / maintenance: incremental production as new
>>>>>>>>>>>>>>>>> data files
>>>>>>>>>>>>>>>>> > arrive, behavior when the index is missing/behind, and
>>>>>>>>>>>>>>>>> sharding/compaction
>>>>>>>>>>>>>>>>> > plus cleanup to avoid accumulating too many small Puffin
>>>>>>>>>>>>>>>>> files over time.
>>>>>>>>>>>>>>>>> > - Writer expectations: inline (optional) vs asynchronous
>>>>>>>>>>>>>>>>> (primary) index
>>>>>>>>>>>>>>>>> > creation.
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > I also implemented a Spark 4.1 POC
>>>>>>>>>>>>>>>>> > <https://github.com/apache/iceberg/pull/15311> and a
>>>>>>>>>>>>>>>>> local benchmark to
>>>>>>>>>>>>>>>>> > quantify both the pruning impact (plannedFiles →
>>>>>>>>>>>>>>>>> afterBloom) and the index
>>>>>>>>>>>>>>>>> > read overhead (statsFiles, statsBytes,
>>>>>>>>>>>>>>>>> bloomPayloadBytes) for point
>>>>>>>>>>>>>>>>> > predicates on high-cardinality columns. Please take a
>>>>>>>>>>>>>>>>> look and let me know
>>>>>>>>>>>>>>>>> > if you have any questions or feedback.
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Huaxin
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > On Tue, Feb 10, 2026 at 1:43 PM huaxin gao <
>>>>>>>>>>>>>>>>> hu...@gmail.com> wrote:
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > > Reminder for tomorrow's sync on Iceberg Index Support.
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > > Wednesday: Feb. 11 9:00 – 10:00am
>>>>>>>>>>>>>>>>> > > Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>> > > Google Meet joining info
>>>>>>>>>>>>>>>>> > > Video call link: meet.google.com/nsp-ctyr-khk
>>>>>>>>>>>>>>>>> > > Design doc:
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.0#heading=h.hs6r9d26w1y2
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.qouk73o4jxx7
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > > Thanks,
>>>>>>>>>>>>>>>>> > > Huaxin
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > > On Tue, Feb 3, 2026 at 10:52 PM Péter Váry <
>>>>>>>>>>>>>>>>> pe...@gmail.com>
>>>>>>>>>>>>>>>>> > > wrote:
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > >> Thanks Huaxin and Steven for organizing this. Looking
>>>>>>>>>>>>>>>>> forward to meet you
>>>>>>>>>>>>>>>>> > >> all next week!
>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>> > >> On Wed, Feb 4, 2026, 02:48 Steven Wu <st...@gmail.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>>> > >>> We set up the dev calendar event with a new google
>>>>>>>>>>>>>>>>> meet link. Please
>>>>>>>>>>>>>>>>> > >>> ignore the link from Huaxin's original email.
>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>> > >>> The dev calendar has the correct info (including the
>>>>>>>>>>>>>>>>> new meeting link)
>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>> > >>> Iceberg Index Support Sync
>>>>>>>>>>>>>>>>> > >>> Wednesday, February 11 · 9:00 – 10:00am
>>>>>>>>>>>>>>>>> > >>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>>> > >>> Google Meet joining info
>>>>>>>>>>>>>>>>> > >>> Video call link:
>>>>>>>>>>>>>>>>> https://meet.google.com/nsp-ctyr-khk
>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>> > >>> On Tue, Feb 3, 2026 at 5:08 PM huaxin gao <
>>>>>>>>>>>>>>>>> hu...@gmail.com>
>>>>>>>>>>>>>>>>> > >>> wrote:
>>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>>> > >>>> Sorry, I meant PST (not EST) :)
>>>>>>>>>>>>>>>>> > >>>> Looking forward to the discussion!
>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>> > >>>> On Tue, Feb 3, 2026 at 4:58 PM Shawn Chang <
>>>>>>>>>>>>>>>>> ye...@gmail.com>
>>>>>>>>>>>>>>>>> > >>>> wrote:
>>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>>> > >>>>> Hi Huaxin,
>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>> > >>>>> Thanks for starting the sync!
>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>> > >>>>> The meeting seems to be 9-10AM PST on the dev
>>>>>>>>>>>>>>>>> events calendar
>>>>>>>>>>>>>>>>> > >>>>> <
>>>>>>>>>>>>>>>>> https://calendar.google.com/calendar/u/0?cid=MzkwNWQ0OTJmMWI0NTBiYTA3MTJmMmFlNmFmYTc2ZWI3NTdmMTNkODUyMjBjYzAzYWE0NTI3ODg1YWRjNTYyOUBncm91cC5jYWxlbmRhci5nb29nbGUuY29t
>>>>>>>>>>>>>>>>> >,
>>>>>>>>>>>>>>>>> > >>>>> not EST. Maybe it's a typo?
>>>>>>>>>>>>>>>>> > >>>>> Otherwise, looking forward to the discussion!
>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>> > >>>>> Best,
>>>>>>>>>>>>>>>>> > >>>>> Shawn
>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>> > >>>>> On Tue, Feb 3, 2026 at 9:18 AM huaxin gao <
>>>>>>>>>>>>>>>>> hu...@gmail.com>
>>>>>>>>>>>>>>>>> > >>>>> wrote:
>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>> > >>>>>> Hi all,
>>>>>>>>>>>>>>>>> > >>>>>> I'd like to start a dedicated sync to discuss
>>>>>>>>>>>>>>>>> Iceberg Index support.
>>>>>>>>>>>>>>>>> > >>>>>> Here is the existing discussion thread:
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> https://lists.apache.org/thread/fzqk3jjf0xpj5m4cfqb3v4c65p0t04ty
>>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> > >>>>>> To ground the discussion, here are the two
>>>>>>>>>>>>>>>>> proposals:
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> > >>>>>> - Peter's proposal
>>>>>>>>>>>>>>>>> > >>>>>> <
>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.0#heading=h.hs6r9d26w1y2>
>>>>>>>>>>>>>>>>> (overall
>>>>>>>>>>>>>>>>> > >>>>>> index support)
>>>>>>>>>>>>>>>>> > >>>>>> - My proposal
>>>>>>>>>>>>>>>>> > >>>>>> <
>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.qouk73o4jxx7
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > >>>>>> (bloom filter skipping index)
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> > >>>>>> Time slot: Every 3 weeks, Wednesdays at 9 AM to
>>>>>>>>>>>>>>>>> 10 AM EST, starting
>>>>>>>>>>>>>>>>> > >>>>>> next Wednesday (2/11). After FileFormat sync
>>>>>>>>>>>>>>>>> finishes, we plan to use that
>>>>>>>>>>>>>>>>> > >>>>>> slot and switch to every other Monday, 9 AM to 10
>>>>>>>>>>>>>>>>> AM EST.
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> > >>>>>> Meet link: https://meet.google.com/fjn-tyze-mko
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>>>>>>>> > >>>>>> Huaxin
>>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>

Re: Dedicated sync for Iceberg Index Support

[Péter Váry <pe...@gmail.com>]

Hi everyone,

Here is the recording of the Monday sync:
https://drive.google.com/drive/folders/1iodHASJBVwNtDYEINq7k2HjogPsYQocH

A brief summary of the discussion:

   - Bloom Filter Index
      - Puffin reduces file-open overhead, while Bloom filter effectiveness
      appears independent of file placement.
      - The observed scaling issues are likely due to rate limiting.
      - Manifest alignment would be required for parallel planning.
      - A scalar index is expected to deliver similar or better performance.
      - We will *prioritize scalar indexes* before pursuing additional
      index types.
      - The tests are still useful to understand the decisions in the
      design stage.
   - Scalar Index Design
      - *Region file schema: identity, materialized, and non-materialized
      fields*.
      - Range files will use stable index-specific IDs, while expressions
      reference table field IDs.
      - Scalar indexes require both sorting and clustering; a dedicated
      index definition may be cleaner than reusing table sort/cluster
definitions.

We can continue the remaining discussions offline or in the next sync.

There are only a few open questions, so the spec PR is ready for review:
https://github.com/apache/iceberg/pull/16961.

Thanks,
Peter


Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026. szept.
14., H, 9:07):

> One correction to the previous summary: I originally wrote that *Range*
> files would store the index data, but I was later corrected that the
> decision was actually to use *Region* files.
>
> The correct index file naming is:
>
>    - *Tracking file*: contains the list of region files (analogous to a
>    manifest file for tables).
>    - *Region file*: contains the index data rows (analogous to a data
>    file for tables).
>
> Sorry for the confusion,
> Peter
>
> Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026. szept.
> 3., Cs, 13:18):
>
>>
>> Hi everyone,
>>
>> Here is the recording of the Monday sync:
>>
>> https://apache-iceberg.slack.com/files/U06538UFDH6/F0BTT739Y7P/impromptu_google_meet_meeting_-_aug_31_2026.mp4
>>
>> Thanks, Shawn, for making it available.
>>
>> A brief summary of the discussion:
>>
>>    - We finalized the file naming:
>>       - Tracking file: contains the list of range files (similar to a
>>       manifest file for tables).
>>       - Range file: contains the index data rows (similar to a data file
>>       for tables).
>>    - We agreed not to prescribe how expression values are materialized
>>    (for example, Hilbert or hash transformations), leaving room for
>>    implementation flexibility.
>>    - Ryan mentioned that he is working on a v4 proposal for generating
>>    Field IDs for non-materialized columns. The current thinking is that index
>>    columns should follow the same approach.
>>    - We agreed to retain older versions of the index metadata JSON.
>>    - We agreed not to require a strict one-to-one mapping between index
>>    snapshots and table snapshots. An index may contain multiple snapshots
>>    corresponding to the same table snapshot.
>>
>> Based on the discussion, I have updated the spec PR. Following Ryan's
>> pattern for non-materialized column definitions, the proposal now
>> introduces:
>>
>>    - *materialized-fields* for range file columns
>>    - *non-materialized-fields* for tracking file statistics
>>    - *cluster-spec* to define which columns participate in range file
>>    clustering
>>
>> Overall, I think we reached good alignment on the remaining design
>> points, and I do not see any open questions at the moment. The PR is in
>> good shape for review, so I would greatly appreciate any feedback or
>> comments.
>>
>> Thanks everyone for the productive discussion and continued collaboration.
>> Peter
>>
>> Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026. aug.
>> 18., K, 14:17):
>>
>>> Hi everyone,
>>>
>>> Unfortunately, yesterday's sync was not recorded due to a technical
>>> issue. :(
>>>
>>> Here is a brief summary of the discussion and the decisions we reached:
>>>
>>>    1. We agreed to use a list of Iceberg expressions to define the
>>>    *index_keys*, which determine how the index is ordered.
>>>    2. We agreed not to restrict the functions that can be used in those
>>>    expressions. Engines that do not understand a particular expression can
>>>    simply choose not to use the index.
>>>    3. We agreed that index region files (leaf files) should be sorted
>>>    by the values generated by the index_keys.
>>>    4. We discussed using *included_column_ids* to define the contents
>>>    and schema of the index region files, and generally felt this could be a
>>>    good approach.
>>>    5. We started discussing whether the results of the index_key
>>>    expressions should also be stored in the index region files alongside the
>>>    included columns. In some cases, storing them could help readers optimize
>>>    lookups within a region file. In other cases, the values could be derived
>>>    when needed, making them redundant and potentially hurting performance by
>>>    increasing the size of the region files.
>>>
>>> I think we should continue the discussion around point 5, as the outcome
>>> may influence, or even reopen, the decisions around point 4.
>>>
>>> Thanks!
>>>
>>>
>>> Flavio Junqueira <fl...@gmail.com> ezt írta (időpont: 2026.
>>> aug. 4., K, 18:57):
>>>
>>>> Thank you for the summary, the proposal and the overall discussion. I
>>>> have one main question about the target and I apologize in advance if I'm
>>>> missing some important detail as I might not have the full context.
>>>>
>>>> I noticed that partitioning, sorting and clustering (via Hilbert) were
>>>> discussed in the call in this context of index transform functions. My
>>>> understanding is that engines currently are already pruning partitions and
>>>> files using partition information and table metadata like per-column
>>>> min-max. If that's the case, then what's it that this proposal is trying to
>>>> achieve by having transform functions that capture partitioning, sorting
>>>> and clustering when engines are already able to prune without a secondary
>>>> index? I can more easily see the primary key index example, but I'm not
>>>> entirely sure about these data layout mechanisms.
>>>>
>>>> I also didn't quite get how engines are expected to consume such an
>>>> index. There was a discussion about where the mapping of the input to files
>>>> happens, whether that's in Iceberg or the engine, and I didn't quite get
>>>> the recommendation. Checking Peter's proposal, I couldn't see it clearly
>>>> either, and if I just missed it, I'd appreciate a pointer to where I should
>>>> look into.
>>>>
>>>> -Flavio
>>>>
>>>> On 4 Aug 2026, at 10:52, Péter Váry <pe...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi everyone,
>>>>
>>>> Here is the recording from yesterday’s sync:
>>>> https://www.youtube.com/watch?v=Wg9orP9JAmk
>>>>
>>>> To summarize:
>>>> - We agreed to retain the history of index snapshots, but not the
>>>> history of other index properties. If readers require information that may
>>>> later be modified by a user, that information should be copied into the
>>>> corresponding snapshot.
>>>> - We discussed the possibility of defining index ordering using a list
>>>> of transform functions.
>>>>
>>>> Dan and Yingyi suggested something along the following lines:
>>>>
>>>> *"index_keys" : [*
>>>> *  {*
>>>> *    "type": "apply",*
>>>> *    "func-name": {*
>>>> *      "catalog": "iceberg_functions",*
>>>> *      "name": "day"*
>>>> *    },*
>>>> *    "arguments": [*
>>>> *      { "type": "reference", "id": 3 }*
>>>> *    ]*
>>>> *  },*
>>>> *  {*
>>>> *    "type": "apply",*
>>>> *    "func-name": {*
>>>> *      "catalog": "iceberg_functions",*
>>>> *      "name": "truncate"*
>>>> *    },*
>>>> *    "arguments": [*
>>>> *      { "type": "reference", "id": 4 },*
>>>> *      2*
>>>> *    ]*
>>>> *  }*
>>>> *]*
>>>>
>>>>
>>>> In this example, rows would first be ordered by day(col3) and then by
>>>> truncate(col4, 2). To ensure a deterministic ordering, ties would be broken
>>>> using the original values of columns 3 and 4. If those are still equal,
>>>> ordering would fall back to the file name and finally the row position.
>>>>
>>>> I spent some time thinking about this after the sync, and here are my
>>>> thoughts.
>>>>
>>>> Pros:
>>>> - Provides a high degree of flexibility.
>>>> - Enables expression-based indexes.
>>>>
>>>> Cons:
>>>> - Do we need this flexibility at this stage?
>>>>    - Equality deletes, the primary use cases we are targeting, do not
>>>> require expressions.
>>>>    - Most indexes I have seen in production are not expression-based.
>>>> - Increased complexity for engines:
>>>>    - Engines must parse and understand transforms to determine whether
>>>> using an index can improve a query. For example, is ordering by
>>>> day(timestamp) sufficient for a key lookup?
>>>>    - Complexity increases further as new UDFs are introduced.
>>>>    - Expressions are not pushed into scans today, so index handling
>>>> cannot be fully hidden behind Iceberg scan planning.
>>>> - Risk of fragmentation:
>>>>    - The additional complexity may slow engine adoption.
>>>>    - Adopters may choose to support only exact transform matches
>>>> created by them, rather than learn to reason about the general transforms.
>>>>    - UDFs are prone to missing implementations and behavioral
>>>> differences across engines. In the worst case, such differences could lead
>>>> to incorrect query results when indexes are used.
>>>>
>>>> Given the points above, I still prefer defining HASH, HILBERT, IDENTITY
>>>> as a set of standardized values combined with an explicit list of key
>>>> columns. This keeps adoption simple, unlocks the primary use cases we are
>>>> targeting today, and allows the Iceberg library to provide the transform
>>>> implementations directly.
>>>>
>>>> Internally, these transforms could still be represented as Iceberg
>>>> expressions, giving us a migration path toward future index types that may
>>>> require full expression support.
>>>>
>>>> For example:
>>>>
>>>> *{*
>>>> *  "transform": "HILBERT",*
>>>> *  "key-column-ids": [3, 4]*
>>>> *}*
>>>>
>>>>
>>>> I'm interested in hearing other opinions.
>>>>
>>>> Thanks,
>>>> Peter
>>>>
>>>> Renjie Liu <li...@gmail.com> ezt írta (időpont: 2026. júl.
>>>> 28., K, 10:07):
>>>>
>>>>> Hi, Peter:
>>>>>
>>>>> Thanks for driving the proposal.
>>>>>
>>>>> >  *Hash Index Type*: Should hash-based indexes be a separate index
>>>>> type? They are only applicable to equality and IN predicates, and the hash
>>>>> algorithm must always be known. A dedicated HASH type could make these
>>>>> constraints explicit and distinguish hash-based indexes from ordered
>>>>> indexes.
>>>>>
>>>>> I'm leaning toward keeping them in scalar index type due to the
>>>>> similarity with others transformation functions like identity. As with the
>>>>> constraints, I think they are properties of transformation functions.
>>>>>
>>>>> > *Transform Function/Layout Representation*: Should transform
>>>>> functions be represented as an enum or a string? I lean toward strings, but
>>>>> I do not have a strong preference. In many cases, an index can still be
>>>>> used based on standard column statistics, even if a reader does not
>>>>> recognize a particular transform.
>>>>>
>>>>> This seems more like a problem in java implementation?
>>>>>
>>>>> Also I prefer the name transform function compared with layout
>>>>> representation. Layout representation makes me feel that it's not about the
>>>>> definition of an index function.
>>>>>
>>>>> > *Future Index Types*: How should we handle index types that are
>>>>> expected in the long term but fall outside the scope of this proposal (for
>>>>> example, VECTOR, IVF, TERM)? Should they be mentioned as planned/reserved
>>>>> concepts, or omitted entirely? I lean toward at least mentioning them. The
>>>>> rationale for introducing an index type is easier to explain when
>>>>> considering future index families.
>>>>>
>>>>> +1
>>>>>
>>>>> *> Index History Handling:* Most database systems treat index changes
>>>>> as a drop-and-recreate operation rather than an in-place update. If we
>>>>> follow the same approach, there is limited value in maintaining index
>>>>> metadata history. During index maintenance, a new metadata.json can be
>>>>> created and atomically swapped in, with the catalog responsible for
>>>>> cleaning up obsolete metadata and index data files according to its
>>>>> retention policy.
>>>>>
>>>>> +1, I think we only need to keep index data for different snapshots,
>>>>> but if an index definition changed, we should drop and recreate a new index.
>>>>>
>>>>> > *File Format Location*: Should file format be specified at the
>>>>> leaf-file level or only at the index level? My preference is to stay
>>>>> aligned with the table metadata model and avoid introducing differences
>>>>> unless there is a compelling requirement. This would simplify
>>>>> implementation and maximize code reuse.
>>>>>
>>>>> +1.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Jul 24, 2026 at 11:57 PM Péter Váry <
>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> Here are the key takeaways from Monday’s meeting (July 20):
>>>>>>
>>>>>>    - *Index Type*: An index type enum will be introduced in v1,
>>>>>>    initially with a single value (SCALAR). This establishes a future-proof API
>>>>>>    boundary while leaving room for additional index families.
>>>>>>    - *Strict Total Ordering*: The specification should require
>>>>>>    transform functions to produce values that preserve a well-defined total
>>>>>>    ordering across the key space. The transform function does not need to
>>>>>>    return a long. Instead, leaf files can store the first and last values
>>>>>>    according to the ordering, enabling generic filtering capabilities for
>>>>>>    readers.
>>>>>>
>>>>>> Open Questions:
>>>>>>
>>>>>>    - *Hash Index Type*: Should hash-based indexes be a separate
>>>>>>    index type? They are only applicable to equality and IN predicates, and the
>>>>>>    hash algorithm must always be known. A dedicated HASH type could make these
>>>>>>    constraints explicit and distinguish hash-based indexes from ordered
>>>>>>    indexes.
>>>>>>    - *Transform Function/Layout Representation*: Should transform
>>>>>>    functions be represented as an enum or a string? I lean toward strings, but
>>>>>>    I do not have a strong preference. In many cases, an index can still be
>>>>>>    used based on standard column statistics, even if a reader does not
>>>>>>    recognize a particular transform.
>>>>>>    - *Future Index Types*: How should we handle index types that are
>>>>>>    expected in the long term but fall outside the scope of this proposal (for
>>>>>>    example, VECTOR, IVF, TERM)? Should they be mentioned as planned/reserved
>>>>>>    concepts, or omitted entirely? I lean toward at least mentioning them. The
>>>>>>    rationale for introducing an index type is easier to explain when
>>>>>>    considering future index families.
>>>>>>    - *Index History Handling:* Most database systems treat index
>>>>>>    changes as a drop-and-recreate operation rather than an in-place update. If
>>>>>>    we follow the same approach, there is limited value in maintaining index
>>>>>>    metadata history. During index maintenance, a new metadata.json can be
>>>>>>    created and atomically swapped in, with the catalog responsible for
>>>>>>    cleaning up obsolete metadata and index data files according to its
>>>>>>    retention policy.
>>>>>>    - *Min/Max Statistics Representation*: Should transform-specific
>>>>>>    or sort-order min/max values be represented using the existing file
>>>>>>    statistics structure, or exposed through dedicated fields? My preference is
>>>>>>    to keep them within the statistics structure defined by the file schema
>>>>>>    while exposing them directly through the Java API if needed.
>>>>>>    - *File Format Location*: Should file format be specified at the
>>>>>>    leaf-file level or only at the index level? My preference is to stay
>>>>>>    aligned with the table metadata model and avoid introducing differences
>>>>>>    unless there is a compelling requirement. This would simplify
>>>>>>    implementation and maximize code reuse.
>>>>>>
>>>>>> I have updated the spec PR based on the decisions above:
>>>>>> https://github.com/apache/iceberg/pull/16961
>>>>>>
>>>>>> Thanks,
>>>>>> Peter
>>>>>>
>>>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont: 2026. jún.
>>>>>> 26., P, 19:42):
>>>>>>
>>>>>>> Hi all, Here is the summary of this Monday's index meeting:
>>>>>>>
>>>>>>> Two blocking decisions closed:
>>>>>>>
>>>>>>>    1. Index is *not* a table— it's its own object (reuses table
>>>>>>>    machinery under the hood). Reasoning: requires a sort order, no column
>>>>>>>    updates, no partition spec, no overlapping ranges between leaves, inherits
>>>>>>>    base-table permissions, and has its own CREATE/DROP/UPDATE INDEX DDL.
>>>>>>>    2. Index is a separate catalog entity, with no pointers in table
>>>>>>>    metadata— has its own REST endpoints; the catalog can optionally return
>>>>>>>    index metadata withloadTable to avoid extra round-trips. Keeps table and
>>>>>>>    index updates independent/async.
>>>>>>>
>>>>>>>
>>>>>>> Next steps: Start writing the spec and build out the copy-on-write
>>>>>>> path now.
>>>>>>>
>>>>>>> Here are the draft spec:
>>>>>>> secondary index spec <https://github.com/apache/iceberg/pull/16961>
>>>>>>> irc spec <https://github.com/apache/iceberg/pull/16963>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Huaxin
>>>>>>>
>>>>>>> On Sat, Jun 20, 2026 at 11:11 AM huaxin gao <hu...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I built a standalone PoC to validate that the basic index structure
>>>>>>>> works: that we can build a PK index, convert equality deletes to position
>>>>>>>> deletes through it, and have every converted delete land on the correct
>>>>>>>> live row. I ran it up to *100M keys*.
>>>>>>>>
>>>>>>>> *Headline: the structure works.* The index builds over up to 100M
>>>>>>>> keys, the eq-delete → position-delete conversion resolved correctly at
>>>>>>>> *every* size (100% of converted deletes mapped to the right live
>>>>>>>> row), and the resulting position deletes are *~8× cheaper to apply* at
>>>>>>>> query time than the equality deletes they replace.
>>>>>>>>
>>>>>>>> Beyond correctness, the run also shows how the index’s
>>>>>>>> *maintenance* cost scales, comparing copy-on-write (COW, rewrite
>>>>>>>> touched leaves) vs an append/merge (MOR) option, under a realistic mixed
>>>>>>>> CDC checkpoint (1,000 insert + 500 update + 500 delete), local wall-clock:
>>>>>>>> keys EQ baseline INDEX (COW) % of 60s (COW) INDEX (MOR) % of 60s
>>>>>>>> (MOR) correct
>>>>>>>> 5M 6 ms 6.7s 11.2% 2.2s 3.7% PASS
>>>>>>>> 20M 8 ms 24.2s 40.4% 6.4s 10.6% PASS
>>>>>>>> 50M 7 ms 51.6s 86.1% 12.2s 20.4% PASS
>>>>>>>> *100M* 6 ms *75.0s* 125% (BEHIND) *16.9s* 28.2% (keeps up) PASS
>>>>>>>>
>>>>>>>> COW maintenance crosses the 60 s checkpoint around 100M
>>>>>>>> (75 s/cycle, 125%); MOR stays at ~28% and keeps pace; the equality-delete
>>>>>>>> baseline is ~6 ms and flat. So the structure works, but *COW alone
>>>>>>>> can’t sustain scattered CDC at hundreds of millions of keys on a single
>>>>>>>> writer*. It’s worth allowing a merge-on-read / update-file
>>>>>>>> maintenance option alongside COW (or sharding the index across parallel
>>>>>>>> writers).
>>>>>>>>
>>>>>>>> *Full write-up, all tables, and the in-region reality-check:* link
>>>>>>>> <https://docs.google.com/document/d/1G3zxbW8X0eU3UrouslZfp42bBc9CvgJGnJyDONCB4PU/edit?tab=t.0>
>>>>>>>>
>>>>>>>> Feedback welcome, especially on the spec direction (whether to
>>>>>>>> allow a merge-on-read / update-file maintenance option alongside COW)  and
>>>>>>>> on the read-side modeling.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Huaxin
>>>>>>>>
>>>>>>>> On Tue, Jun 9, 2026 at 5:45 PM huaxin gao <hu...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Sorry,  we've skipped posting a few of the dedicated index-sync
>>>>>>>>> summaries to the mailing list; you can find those in the Google
>>>>>>>>> doc
>>>>>>>>> <https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?pli=1&tab=t.8041k7j2n7y3>
>>>>>>>>> and the Slack channel. Here's yesterday's summary:
>>>>>>>>>
>>>>>>>>> *Decided*
>>>>>>>>>
>>>>>>>>>    -
>>>>>>>>>
>>>>>>>>>    Index vs. table (what we agreed):
>>>>>>>>>    - Reuse table implementation/library code and a near-identical
>>>>>>>>>       spec — the commit path will be custom regardless, so reuse isn't the
>>>>>>>>>       deciding factor.
>>>>>>>>>       - An index is not a table from a user/API view: loading or
>>>>>>>>>       writing an index as a table must fail(it would violate index invariants).
>>>>>>>>>       - The spec forbids most table behaviors: no overlapping
>>>>>>>>>       files, one mandatory transform sort order, no column updates, no partition
>>>>>>>>>       spec.
>>>>>>>>>       - Delete vectors: reuse Iceberg's existing DV — benchmarks
>>>>>>>>>    showed no new delete format is worth introducing.
>>>>>>>>>    - Incremental updates: start with copy-on-write only (no
>>>>>>>>>    update files). For object-store-sized leaves, a full leaf rewrite is about
>>>>>>>>>    as cheap as maintaining an overlay update file + DV, so we'll skip the MOR
>>>>>>>>>    machinery for now and add it later only if benchmarks prove we need it
>>>>>>>>>    (likely just the very-large-leaf case).
>>>>>>>>>    -
>>>>>>>>>
>>>>>>>>>    Validate the spec first: build a quick, hand-wired prototype
>>>>>>>>>    (Parquet files structured per the spec) and benchmark it on real scales
>>>>>>>>>    before formalizing.
>>>>>>>>>
>>>>>>>>> *Leaning, not final*
>>>>>>>>>
>>>>>>>>>    -
>>>>>>>>>
>>>>>>>>>    Indexes are likely separate catalog objects, linked from the
>>>>>>>>>    table by storing just an identifier (like materialized views) and not
>>>>>>>>>    visible in LIST TABLES.
>>>>>>>>>    -
>>>>>>>>>
>>>>>>>>>    We'll need a commit path for indexes, but simpler than tables
>>>>>>>>>    (no stage-create).
>>>>>>>>>
>>>>>>>>> *Still open*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>    -
>>>>>>>>>
>>>>>>>>>    Permissions model — separate vs. inherited (action: look at
>>>>>>>>>    what real DBs do for index permissions).
>>>>>>>>>    - REST/catalog RPC design — minimize round-trips; index
>>>>>>>>>    metadata ideally returned with LOAD TABLE. Catalog RPC cost
>>>>>>>>>    may dominate Parquet IO, so this needs real design.
>>>>>>>>>    -
>>>>>>>>>
>>>>>>>>>    Scale modeling — target rows-per-leaf vs. leaf size vs.
>>>>>>>>>    metadata-file count.
>>>>>>>>>    -
>>>>>>>>>
>>>>>>>>>    DDL-on-index semantics (reuse table schema-update actions or
>>>>>>>>>    separate)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Huaxin
>>>>>>>>>
>>>>>>>>> On Wed, Apr 22, 2026 at 8:47 AM Péter Váry <
>>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi All,
>>>>>>>>>>
>>>>>>>>>> TL;DR
>>>>>>>>>> We still need to validate with ADLS and S3, but based on the
>>>>>>>>>> local tests, the MPHF approach looks more promising if we can tolerate
>>>>>>>>>> larger files and longer index maintenance times.
>>>>>>>>>>
>>>>>>>>>> Details:
>>>>>>>>>> Here are the results from the local experiments on my Mac. I
>>>>>>>>>> removed unnecessary statistics from the Parquet files and tested different
>>>>>>>>>> row group sizes:
>>>>>>>>>>
>>>>>>>>>>    - For an index file with 1M records, a row group size of
>>>>>>>>>>    5,000 appears to be the sweet spot.
>>>>>>>>>>    - For 10M records, 10,000 rows per row group works best.
>>>>>>>>>>
>>>>>>>>>> If you have additional ideas for optimizing Parquet-based
>>>>>>>>>> indexes, I’d be very interested to hear them.
>>>>>>>>>> The test code is available on this branch:
>>>>>>>>>> https://github.com/pvary/iceberg/tree/leaf_bench
>>>>>>>>>>
>>>>>>>>>> Best results:
>>>>>>>>>> *1m records/file*
>>>>>>>>>>
>>>>>>>>>>    - Parquet - 5000 row/RowGroup
>>>>>>>>>>       - Read: 1191 µs - 1 file open, 3 seek, 123KB read per
>>>>>>>>>>       lookup
>>>>>>>>>>       - Write: 1.7 s, 15 MB
>>>>>>>>>>    - MPHF
>>>>>>>>>>       - Read: 202 µs - 1 file open, 1 seek,  282KB read per
>>>>>>>>>>       lookup
>>>>>>>>>>       - Write: 0.8 s, 34 MB
>>>>>>>>>>
>>>>>>>>>> *10m records/file*
>>>>>>>>>>
>>>>>>>>>>    - Parquet - 10000 row/RowGroup
>>>>>>>>>>       - Read: 4168 µs - 1 file open, 3 seek, 395KB read per
>>>>>>>>>>       lookup
>>>>>>>>>>       - Write: 19.5s s, 144 MB
>>>>>>>>>>    - MPHF
>>>>>>>>>>       - Read: 1086 µs - 1 file open, 1 seek,  2.8 MB (2812KB)
>>>>>>>>>>       read per lookup
>>>>>>>>>>       - Write: 6.5 s, 34 MB, 353 MB
>>>>>>>>>>
>>>>>>>>>> Below are the full results.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *Benchmark                                      (indexType)
>>>>>>>>>>  (keyType)  (numRows)  Mode    Cnt            Score          Error
>>>>>>>>>>  UnitsInvertedIndexBenchmark.lookup                 PARQUET_1000       LONG
>>>>>>>>>>    1000000    ss  10000         3285.284 ±        5.138
>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_1000       LONG
>>>>>>>>>>    1000000    ss  10000   2522168989.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_1000       LONG
>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_1000       LONG
>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_1000       LONG
>>>>>>>>>> 10000000    ss  10000        35449.614 ±       34.673
>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_1000       LONG
>>>>>>>>>>   10000000    ss  10000  24302649201.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_1000       LONG
>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_1000       LONG
>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_5000       LONG
>>>>>>>>>>  1000000    ss  10000         1191.959 ±        4.169
>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_5000       LONG
>>>>>>>>>>    1000000    ss  10000   1230877229.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_5000       LONG
>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_5000       LONG
>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_5000       LONG
>>>>>>>>>> 10000000    ss  10000         7236.447 ±       10.374
>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_5000       LONG
>>>>>>>>>>   10000000    ss  10000   5650715973.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_5000       LONG
>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_5000       LONG
>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_10000       LONG
>>>>>>>>>>  1000000    ss  10000         1349.946 ±        7.834
>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_10000       LONG
>>>>>>>>>>    1000000    ss  10000   1730219377.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_10000       LONG
>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_10000       LONG
>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_10000       LONG
>>>>>>>>>> 10000000    ss  10000         4168.635 ±       11.051
>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_10000       LONG
>>>>>>>>>>   10000000    ss  10000   3946341532.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_10000       LONG
>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_10000       LONG
>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_50000       LONG
>>>>>>>>>>  1000000    ss  10000         4736.466 ±       38.179
>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_50000       LONG
>>>>>>>>>>    1000000    ss  10000   7427413541.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_50000       LONG
>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_50000       LONG
>>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_50000       LONG
>>>>>>>>>> 10000000    ss  10000         4979.031 ±       34.708
>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_50000       LONG
>>>>>>>>>>   10000000    ss  10000   7694887636.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_50000       LONG
>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_50000       LONG
>>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup                         MPHF       LONG
>>>>>>>>>>  1000000    ss  10000          202.571 ±        2.336
>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead               MPHF       LONG
>>>>>>>>>>    1000000    ss  10000   2821570000.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams             MPHF       LONG
>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks                   MPHF       LONG
>>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup                         MPHF       LONG
>>>>>>>>>> 10000000    ss  10000         1086.957 ±        4.524
>>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead               MPHF       LONG
>>>>>>>>>>   10000000    ss  10000  28119460000.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams             MPHF       LONG
>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks                   MPHF       LONG
>>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_1000       LONG
>>>>>>>>>>  1000000    ss      3      1720731.014 ±   876636.004
>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_1000       LONG
>>>>>>>>>>    1000000    ss      3     46453317.000
>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_1000       LONG
>>>>>>>>>> 10000000    ss      3     18547947.876 ± 12258125.307
>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_1000       LONG
>>>>>>>>>>   10000000    ss      3    452655675.000
>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_5000       LONG
>>>>>>>>>>  1000000    ss      3      1718345.583 ±  1103928.016
>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_5000       LONG
>>>>>>>>>>    1000000    ss      3     44845788.000
>>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_5000       LONG
>>>>>>>>>> 10000000    ss      3     18604229.931 ±  2668361.915
>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_5000       LONG
>>>>>>>>>>   10000000    ss      3    435388818.000
>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_10000       LONG
>>>>>>>>>>  1000000    ss      3      1761555.389 ±   535857.675
>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_10000       LONG
>>>>>>>>>>    1000000    ss      3     44536635.000
>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_10000       LONG
>>>>>>>>>> 10000000    ss      3     19501588.264 ±  2130054.558
>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_10000       LONG
>>>>>>>>>>   10000000    ss      3    433189623.000
>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_50000       LONG
>>>>>>>>>>  1000000    ss      3      1936624.889 ±  6601363.985
>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_50000       LONG
>>>>>>>>>>    1000000    ss      3     44264655.000
>>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_50000       LONG
>>>>>>>>>> 10000000    ss      3     20471742.278 ± 10705206.310
>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_50000       LONG
>>>>>>>>>>   10000000    ss      3    431311305.000
>>>>>>>>>> #InvertedIndexBenchmark.write                          MPHF       LONG
>>>>>>>>>>  1000000    ss      3       896573.958 ±  1408024.851
>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes           MPHF       LONG
>>>>>>>>>>    1000000    ss      3    102846369.000
>>>>>>>>>> #InvertedIndexBenchmark.write                          MPHF       LONG
>>>>>>>>>> 10000000    ss      3      6509348.875 ± 15519975.479
>>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes           MPHF       LONG
>>>>>>>>>>   10000000    ss      3   1058435733.000                     #*
>>>>>>>>>>
>>>>>>>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont: 2026.
>>>>>>>>>> ápr. 21., K, 20:53):
>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> In recent secondary index sync meetings, the discussion
>>>>>>>>>>> converged on the need to define what an index is from first principles
>>>>>>>>>>> before settling on physical layout.
>>>>>>>>>>>
>>>>>>>>>>> To address that, Peter and I have drafted a requirements
>>>>>>>>>>> document for a key lookup index (renamed from "primary key index" to avoid
>>>>>>>>>>> implying uniqueness enforcement), the goal is to nail down one well-scoped
>>>>>>>>>>> index type first.
>>>>>>>>>>>
>>>>>>>>>>> Doc: Key Lookup Index Requirements
>>>>>>>>>>> <https://docs.google.com/document/d/1e0zxK-jA0LBDq8YQlQgFipTHelDFiga8lCkgDTmYub8/edit?tab=t.0#heading=h.8shrgabvl19>
>>>>>>>>>>>
>>>>>>>>>>> It covers requirements, three design options (manifest + sorted
>>>>>>>>>>> Parquet, hash + sorted Parquet, hash + MPHF) and open questions. We will
>>>>>>>>>>> add preliminary benchmark results shortly.
>>>>>>>>>>>
>>>>>>>>>>> Feedback welcome — inline in the doc, on this thread, or at the
>>>>>>>>>>> next index sync.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>> Huaxin
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Apr 13, 2026 at 7:22 AM Steven Wu <st...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Do we need the special index identifier that was originally
>>>>>>>>>>>> proposed? A generic CatalogObjectIdentifier (with namespace and name) would
>>>>>>>>>>>> be consistent with all object types in the catalog. I have a discussion
>>>>>>>>>>>> thread on the generic identifier topic: [DISCUSS] REST Spec:
>>>>>>>>>>>> generic CatalogObjectIdentifier.
>>>>>>>>>>>>
>>>>>>>>>>>> Should we add an indexes array field to table metadata? It
>>>>>>>>>>>> only contains a list of index object identifiers. It doesn't contain any
>>>>>>>>>>>> index metadata which should live in the index objects. Yufei was trying to
>>>>>>>>>>>> bring this up at the end of the first sync. But we didn't get enough time
>>>>>>>>>>>> to really discuss it. It will be great to discuss this as the first agenda
>>>>>>>>>>>> item today.
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Apr 13, 2026 at 3:17 AM Péter Váry <
>>>>>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>
>>>>>>>>>>>>> We had several engaging discussions at the Iceberg Summit, and
>>>>>>>>>>>>> it was great to finally catch up with many of you in person. We truly
>>>>>>>>>>>>> missed those who couldn’t attend, hopefully we’ll all meet again at the
>>>>>>>>>>>>> next summit.
>>>>>>>>>>>>>
>>>>>>>>>>>>> To keep the conversation going, Huaxin and I have put together
>>>>>>>>>>>>> the agenda for our next meeting. As a reminder, we’ll meet on *April
>>>>>>>>>>>>> 13th, 9:00–10:00 AM *PDT (6:00–7:00 PM CEST).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Proposed agenda:
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - Continue first-principles index design discussion from
>>>>>>>>>>>>>    Mar 30
>>>>>>>>>>>>>       - *Index Ownership and Write Responsibility*
>>>>>>>>>>>>>          - Should writers be allowed to update indexes, or
>>>>>>>>>>>>>          - Should all index writes be handled exclusively by
>>>>>>>>>>>>>          the Index Maintenance process?
>>>>>>>>>>>>>          - If writers can update indexes then we need to
>>>>>>>>>>>>>          define what guarantees are required (compaction, file splitting, layout
>>>>>>>>>>>>>          expectations)?
>>>>>>>>>>>>>          - If only Index Maintenance updates indexes then we
>>>>>>>>>>>>>          only need to define what observable properties should be exposed to
>>>>>>>>>>>>>          consumers? Like:
>>>>>>>>>>>>>             - Expected max files for a single key
>>>>>>>>>>>>>             - Current max files for a single key
>>>>>>>>>>>>>             - Deletes allowed/present
>>>>>>>>>>>>>             - Sorted by
>>>>>>>>>>>>>             - Partitioned by
>>>>>>>>>>>>>          - *Specification Scope: What Belongs in the Spec?*
>>>>>>>>>>>>>          - Related to the ownership question above
>>>>>>>>>>>>>          - Light spec: Just define that the index table
>>>>>>>>>>>>>          should be optimized for retrieval by key columns and the index columns
>>>>>>>>>>>>>          should be contained in the table. This could give us more flexibility if
>>>>>>>>>>>>>          better organization methods come up, or
>>>>>>>>>>>>>          - Detailed spec: We could define the max number of
>>>>>>>>>>>>>          files per index to read for a single key, or even the partitioning and the
>>>>>>>>>>>>>          exact sort order. This could allow more use-cases for a given index, like
>>>>>>>>>>>>>          joins or cardinality estimations.
>>>>>>>>>>>>>          - I would go for light spec for the main types (PK,
>>>>>>>>>>>>>          Containing) and only the Index Maintenance processes should update the
>>>>>>>>>>>>>          Indexes, as for many use-cases the details are not important, and writers
>>>>>>>>>>>>>          will very rarely update the Indexes themselves.
>>>>>>>>>>>>>       - *Logical Placement of Indexes*
>>>>>>>>>>>>>          - Index as a child object of an Iceberg Table, or
>>>>>>>>>>>>>          - Index as a first‑class entity under
>>>>>>>>>>>>>          /namespace/indexes/{index}
>>>>>>>>>>>>>          - Based on the discussions on the summit we are
>>>>>>>>>>>>>          leaning in this direction. This means the index id should be unique in the
>>>>>>>>>>>>>          namespace but helps the catalog implementations quite a bit
>>>>>>>>>>>>>       - *Physical Placement of Index Data*
>>>>>>>>>>>>>          - I don’t think we should specify this. We should
>>>>>>>>>>>>>          have a base location for the index, but can rely on the catalog
>>>>>>>>>>>>>          implementations to decide on their own, like they do with the tables,
>>>>>>>>>>>>>          views, udfs.
>>>>>>>>>>>>>       - *Iceberg Reader Based indexes* (Containing indexes
>>>>>>>>>>>>>       and potentially PK indexes). These are the indexes which could be read by
>>>>>>>>>>>>>       the existing Iceberg readers. We might decide to store the PK index
>>>>>>>>>>>>>       similarly to an Iceberg Table and treat it as a reader based index.
>>>>>>>>>>>>>          - What are the table properties/features exposed to
>>>>>>>>>>>>>          the readers
>>>>>>>>>>>>>             - Maybe just some behavioral descriptors for the
>>>>>>>>>>>>>             optimizer to decide if the index could be used or should be skipped, like:
>>>>>>>>>>>>>                - Expected max files for a single key
>>>>>>>>>>>>>                - max files for a single key
>>>>>>>>>>>>>                - Deletes allowed/present
>>>>>>>>>>>>>                - Sorted by
>>>>>>>>>>>>>                - Partitioned by
>>>>>>>>>>>>>             - The Tasks when reading the index based on the
>>>>>>>>>>>>>             filters and projection
>>>>>>>>>>>>>          - What are the table properties/features exposed to
>>>>>>>>>>>>>          the Index Maintenance. I think this could be internal to the Index
>>>>>>>>>>>>>          Maintenance process and might not be exposed through the spec. The Index
>>>>>>>>>>>>>          Maintenance process could handle this as a standard Iceberg Table and could
>>>>>>>>>>>>>          be based on the Table Maintenance process, but there might be some totally
>>>>>>>>>>>>>          different processes.
>>>>>>>>>>>>>       - It should be possible to add properties to an index
>>>>>>>>>>>>>       defined by the Index Maintenance process which could be used and updated in
>>>>>>>>>>>>>       the next Index Maintenance run.
>>>>>>>>>>>>>    - *PK index storage format benchmark results*
>>>>>>>>>>>>>       - Flat Parquet (baseline)
>>>>>>>>>>>>>       - BTree with Parquet leaves
>>>>>>>>>>>>>       - Vortex
>>>>>>>>>>>>>    - *Open items / next steps*
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Peter
>>>>>>>>>>>>>
>>>>>>>>>>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont: 2026.
>>>>>>>>>>>>> márc. 23., H, 3:03):
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi everyone, I wanted to share an update on the primary key
>>>>>>>>>>>>>> index work.
>>>>>>>>>>>>>> Since there are still open questions on whether bloom filter
>>>>>>>>>>>>>> indexes fit in the secondary index framework or should be treated as
>>>>>>>>>>>>>> extended stats, I've shifted focus to the primary key index since it's a
>>>>>>>>>>>>>> clearer fit for the framework.
>>>>>>>>>>>>>> I've put together a proposal for a primary key reverse-lookup
>>>>>>>>>>>>>> index that maps each key to its physical location (file_path,
>>>>>>>>>>>>>> row_position). It enables:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - Scan-time file pruning for point lookups
>>>>>>>>>>>>>>    - Converting key-based deletes into position deletes
>>>>>>>>>>>>>>    (eliminating equality deletes for Flink CDC)
>>>>>>>>>>>>>>    - Accelerating Spark MERGE INTO by replacing full-table
>>>>>>>>>>>>>>    joins with direct file lookups
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Proposal:
>>>>>>>>>>>>>> https://docs.google.com/document/d/1HuhCZ0n2FqDh8yqQb9oEj1CPM5yXpEsMPGZno2aSf8E/edit?tab=t.0#heading=h.tbevg4q0m9
>>>>>>>>>>>>>> Feedback welcome!
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Huaxin
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Mar 18, 2026 at 11:42 PM Péter Váry <
>>>>>>>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Key takeaways from the general index discussion at the May
>>>>>>>>>>>>>>> 16 meeting.
>>>>>>>>>>>>>>> Thanks to everyone who participated! The recording is
>>>>>>>>>>>>>>> available here: https://www.youtube.com/watch?v=btmjhtRWUCE
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    - Q: Do we need to tie index types to the algorithms
>>>>>>>>>>>>>>>    used to access them?
>>>>>>>>>>>>>>>    - A: From a specification perspective, the goal is to
>>>>>>>>>>>>>>>    define the storage-level data layout so it can be shared across engines.
>>>>>>>>>>>>>>>    Engines are free to interpret and use the data as they see fit, but the
>>>>>>>>>>>>>>>    on-disk data layout itself must be strictly defined and interoperable.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    - Q: Should we introduce an additional abstraction layer
>>>>>>>>>>>>>>>    (e.g., Vector Index) with sub-types such as IVF and DiskANN?
>>>>>>>>>>>>>>>    - A: This is possible if we decide it is beneficial. I
>>>>>>>>>>>>>>>    explored potential naming, but it is not yet clear how such a layer would
>>>>>>>>>>>>>>>    be used in practice.
>>>>>>>>>>>>>>>    *Question to Yingyi Bu*: could you provide examples
>>>>>>>>>>>>>>>    where this additional layer would be useful? Should this abstraction be
>>>>>>>>>>>>>>>    defined at the spec level, or is it better handled at the engine level?
>>>>>>>>>>>>>>>    My initial idea was that users would create a generic
>>>>>>>>>>>>>>>    Vector Index and let the engine choose the concrete implementation.
>>>>>>>>>>>>>>>    However, this would limit user control and users likely need to specify the
>>>>>>>>>>>>>>>    exact index representation, which implies they must be aware of the
>>>>>>>>>>>>>>>    available representations.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    - Q: Do we want to allow extensibility for index types?
>>>>>>>>>>>>>>>    - A: Yes. The intent is to support a small set of
>>>>>>>>>>>>>>>    well-defined index types while allowing experimentation with new ones. If a
>>>>>>>>>>>>>>>    new index type proves broadly useful, a follow-up proposal can standardize
>>>>>>>>>>>>>>>    it and incorporate it into the spec.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    - Q: Do we allow multiple versions of an index for the
>>>>>>>>>>>>>>>    same table snapshot?
>>>>>>>>>>>>>>>    - A: Yes. Older index versions must be retained for
>>>>>>>>>>>>>>>    readers that have already started using them, while new readers should
>>>>>>>>>>>>>>>    automatically use the latest available version
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    - Q: Do we need to use materialized views for these
>>>>>>>>>>>>>>>    indexes?
>>>>>>>>>>>>>>>    - A: No. These indexes are primarily examples, and
>>>>>>>>>>>>>>>    different types may require different storage methods. However, the Primary
>>>>>>>>>>>>>>>    Key, Containing, and parts of the IVF indexes can be structured as Iceberg
>>>>>>>>>>>>>>>    tables. This allows engines to read them natively; in some cases, Iceberg
>>>>>>>>>>>>>>>    planners can automatically redirect queries to the index table without
>>>>>>>>>>>>>>>    engine modifications. Furthermore, index maintenance for these tables can
>>>>>>>>>>>>>>>    leverage existing materialized view maintenance workflows. Other index
>>>>>>>>>>>>>>>    types may instead rely on Puffin files or alternative storage approaches.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    - Q: How should index metadata be accessed? Should we
>>>>>>>>>>>>>>>    add explicit pointers for the indexes in the table metadata?
>>>>>>>>>>>>>>>    - A: We did not have sufficient time to fully explore
>>>>>>>>>>>>>>>    and conclude this topic.
>>>>>>>>>>>>>>>    *Question for Yufei Gu*: Did I understand correctly that
>>>>>>>>>>>>>>>    your main concern stems from endpoint resolution from a REST Catalog
>>>>>>>>>>>>>>>    perspective? Specifically, if indexes are exposed under a URI such as
>>>>>>>>>>>>>>>    v1/{prefix}/namespaces/{namespace}/tables/{table}/indexes/{index}, would
>>>>>>>>>>>>>>>    this make it more difficult for the REST Catalog to resolve and route
>>>>>>>>>>>>>>>    requests to the appropriate endpoint?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Suhas Jayaram Subramanya via dev <de...@iceberg.apache.org>
>>>>>>>>>>>>>>> ezt írta (időpont: 2026. márc. 13., P, 23:32):
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Here's a proposal for native Vector Index support in
>>>>>>>>>>>>>>>> Iceberg tables --
>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1KL4qLOwdqnhOcqTc0EjO1O16NV3M3c-gZCEINDWw4lA/edit?usp=sharing
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We've been working on this proposal with Peter internally
>>>>>>>>>>>>>>>> at Microsoft and he suggested we post it here to bring this to the
>>>>>>>>>>>>>>>> community's attention, ahead of the next Secondary Index Sync.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Suhas
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 2026/02/19 04:34:34 huaxin gao wrote:
>>>>>>>>>>>>>>>> > Hi Everyone,
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > Here are the recording and notes from the Iceberg Index
>>>>>>>>>>>>>>>> Support Sync on
>>>>>>>>>>>>>>>> > 2/11.
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > Recording: https://www.youtube.com/watch?v=3sFfQ0A50yk
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > Notes:
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.8041k7j2n7y3
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > The meeting will move to biweekly, Mondays 9–10am PST,
>>>>>>>>>>>>>>>> starting March 2.
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > Since the sync, I updated the Bloom skipping index
>>>>>>>>>>>>>>>> proposal
>>>>>>>>>>>>>>>> > <
>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.5r5kl6k3fqwu
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > to address the discussion questions, specifically:
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > - Performance justification: when this helps
>>>>>>>>>>>>>>>> (high-cardinality = / IN,
>>>>>>>>>>>>>>>> > many data files, high object-store latency) and how it
>>>>>>>>>>>>>>>> differs from Parquet
>>>>>>>>>>>>>>>> > row-group Bloom filters (which still require opening the
>>>>>>>>>>>>>>>> data file).
>>>>>>>>>>>>>>>> > - Cost / scalability: rough sizing (Bloom blob size per
>>>>>>>>>>>>>>>> file, Puffin
>>>>>>>>>>>>>>>> > file size), the planning cost trade-off (driver index
>>>>>>>>>>>>>>>> reads vs executor
>>>>>>>>>>>>>>>> > file opens), and mitigations via caching.
>>>>>>>>>>>>>>>> > - Lifecycle / maintenance: incremental production as new
>>>>>>>>>>>>>>>> data files
>>>>>>>>>>>>>>>> > arrive, behavior when the index is missing/behind, and
>>>>>>>>>>>>>>>> sharding/compaction
>>>>>>>>>>>>>>>> > plus cleanup to avoid accumulating too many small Puffin
>>>>>>>>>>>>>>>> files over time.
>>>>>>>>>>>>>>>> > - Writer expectations: inline (optional) vs asynchronous
>>>>>>>>>>>>>>>> (primary) index
>>>>>>>>>>>>>>>> > creation.
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > I also implemented a Spark 4.1 POC
>>>>>>>>>>>>>>>> > <https://github.com/apache/iceberg/pull/15311> and a
>>>>>>>>>>>>>>>> local benchmark to
>>>>>>>>>>>>>>>> > quantify both the pruning impact (plannedFiles →
>>>>>>>>>>>>>>>> afterBloom) and the index
>>>>>>>>>>>>>>>> > read overhead (statsFiles, statsBytes, bloomPayloadBytes)
>>>>>>>>>>>>>>>> for point
>>>>>>>>>>>>>>>> > predicates on high-cardinality columns. Please take a
>>>>>>>>>>>>>>>> look and let me know
>>>>>>>>>>>>>>>> > if you have any questions or feedback.
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > Huaxin
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > On Tue, Feb 10, 2026 at 1:43 PM huaxin gao <
>>>>>>>>>>>>>>>> hu...@gmail.com> wrote:
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > > Reminder for tomorrow's sync on Iceberg Index Support.
>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>> > > Wednesday: Feb. 11 9:00 – 10:00am
>>>>>>>>>>>>>>>> > > Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>> > > Google Meet joining info
>>>>>>>>>>>>>>>> > > Video call link: meet.google.com/nsp-ctyr-khk
>>>>>>>>>>>>>>>> > > Design doc:
>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.0#heading=h.hs6r9d26w1y2
>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.qouk73o4jxx7
>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>> > > Thanks,
>>>>>>>>>>>>>>>> > > Huaxin
>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>> > > On Tue, Feb 3, 2026 at 10:52 PM Péter Váry <
>>>>>>>>>>>>>>>> pe...@gmail.com>
>>>>>>>>>>>>>>>> > > wrote:
>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>> > >> Thanks Huaxin and Steven for organizing this. Looking
>>>>>>>>>>>>>>>> forward to meet you
>>>>>>>>>>>>>>>> > >> all next week!
>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>> > >> On Wed, Feb 4, 2026, 02:48 Steven Wu <st...@gmail.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>>> > >>> We set up the dev calendar event with a new google
>>>>>>>>>>>>>>>> meet link. Please
>>>>>>>>>>>>>>>> > >>> ignore the link from Huaxin's original email.
>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>> > >>> The dev calendar has the correct info (including the
>>>>>>>>>>>>>>>> new meeting link)
>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>> > >>> Iceberg Index Support Sync
>>>>>>>>>>>>>>>> > >>> Wednesday, February 11 · 9:00 – 10:00am
>>>>>>>>>>>>>>>> > >>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>>> > >>> Google Meet joining info
>>>>>>>>>>>>>>>> > >>> Video call link: https://meet.google.com/nsp-ctyr-khk
>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>> > >>> On Tue, Feb 3, 2026 at 5:08 PM huaxin gao <
>>>>>>>>>>>>>>>> hu...@gmail.com>
>>>>>>>>>>>>>>>> > >>> wrote:
>>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>>> > >>>> Sorry, I meant PST (not EST) :)
>>>>>>>>>>>>>>>> > >>>> Looking forward to the discussion!
>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>> > >>>> On Tue, Feb 3, 2026 at 4:58 PM Shawn Chang <
>>>>>>>>>>>>>>>> ye...@gmail.com>
>>>>>>>>>>>>>>>> > >>>> wrote:
>>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>>> > >>>>> Hi Huaxin,
>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>> > >>>>> Thanks for starting the sync!
>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>> > >>>>> The meeting seems to be 9-10AM PST on the dev
>>>>>>>>>>>>>>>> events calendar
>>>>>>>>>>>>>>>> > >>>>> <
>>>>>>>>>>>>>>>> https://calendar.google.com/calendar/u/0?cid=MzkwNWQ0OTJmMWI0NTBiYTA3MTJmMmFlNmFmYTc2ZWI3NTdmMTNkODUyMjBjYzAzYWE0NTI3ODg1YWRjNTYyOUBncm91cC5jYWxlbmRhci5nb29nbGUuY29t
>>>>>>>>>>>>>>>> >,
>>>>>>>>>>>>>>>> > >>>>> not EST. Maybe it's a typo?
>>>>>>>>>>>>>>>> > >>>>> Otherwise, looking forward to the discussion!
>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>> > >>>>> Best,
>>>>>>>>>>>>>>>> > >>>>> Shawn
>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>> > >>>>> On Tue, Feb 3, 2026 at 9:18 AM huaxin gao <
>>>>>>>>>>>>>>>> hu...@gmail.com>
>>>>>>>>>>>>>>>> > >>>>> wrote:
>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>> > >>>>>> Hi all,
>>>>>>>>>>>>>>>> > >>>>>> I'd like to start a dedicated sync to discuss
>>>>>>>>>>>>>>>> Iceberg Index support.
>>>>>>>>>>>>>>>> > >>>>>> Here is the existing discussion thread:
>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>> https://lists.apache.org/thread/fzqk3jjf0xpj5m4cfqb3v4c65p0t04ty
>>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>> > >>>>>> To ground the discussion, here are the two
>>>>>>>>>>>>>>>> proposals:
>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>> > >>>>>> - Peter's proposal
>>>>>>>>>>>>>>>> > >>>>>> <
>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.0#heading=h.hs6r9d26w1y2>
>>>>>>>>>>>>>>>> (overall
>>>>>>>>>>>>>>>> > >>>>>> index support)
>>>>>>>>>>>>>>>> > >>>>>> - My proposal
>>>>>>>>>>>>>>>> > >>>>>> <
>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.qouk73o4jxx7
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > >>>>>> (bloom filter skipping index)
>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>> > >>>>>> Time slot: Every 3 weeks, Wednesdays at 9 AM to 10
>>>>>>>>>>>>>>>> AM EST, starting
>>>>>>>>>>>>>>>> > >>>>>> next Wednesday (2/11). After FileFormat sync
>>>>>>>>>>>>>>>> finishes, we plan to use that
>>>>>>>>>>>>>>>> > >>>>>> slot and switch to every other Monday, 9 AM to 10
>>>>>>>>>>>>>>>> AM EST.
>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>> > >>>>>> Meet link: https://meet.google.com/fjn-tyze-mko
>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>>>>>>> > >>>>>> Huaxin
>>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>

Re: Dedicated sync for Iceberg Index Support

[Péter Váry <pe...@gmail.com>]

One correction to the previous summary: I originally wrote that *Range*
files would store the index data, but I was later corrected that the
decision was actually to use *Region* files.

The correct index file naming is:

   - *Tracking file*: contains the list of region files (analogous to a
   manifest file for tables).
   - *Region file*: contains the index data rows (analogous to a data file
   for tables).

Sorry for the confusion,
Peter

Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026. szept.
3., Cs, 13:18):

>
> Hi everyone,
>
> Here is the recording of the Monday sync:
>
> https://apache-iceberg.slack.com/files/U06538UFDH6/F0BTT739Y7P/impromptu_google_meet_meeting_-_aug_31_2026.mp4
>
> Thanks, Shawn, for making it available.
>
> A brief summary of the discussion:
>
>    - We finalized the file naming:
>       - Tracking file: contains the list of range files (similar to a
>       manifest file for tables).
>       - Range file: contains the index data rows (similar to a data file
>       for tables).
>    - We agreed not to prescribe how expression values are materialized
>    (for example, Hilbert or hash transformations), leaving room for
>    implementation flexibility.
>    - Ryan mentioned that he is working on a v4 proposal for generating
>    Field IDs for non-materialized columns. The current thinking is that index
>    columns should follow the same approach.
>    - We agreed to retain older versions of the index metadata JSON.
>    - We agreed not to require a strict one-to-one mapping between index
>    snapshots and table snapshots. An index may contain multiple snapshots
>    corresponding to the same table snapshot.
>
> Based on the discussion, I have updated the spec PR. Following Ryan's
> pattern for non-materialized column definitions, the proposal now
> introduces:
>
>    - *materialized-fields* for range file columns
>    - *non-materialized-fields* for tracking file statistics
>    - *cluster-spec* to define which columns participate in range file
>    clustering
>
> Overall, I think we reached good alignment on the remaining design points,
> and I do not see any open questions at the moment. The PR is in good shape
> for review, so I would greatly appreciate any feedback or comments.
>
> Thanks everyone for the productive discussion and continued collaboration.
> Peter
>
> Péter Váry <pe...@gmail.com> ezt írta (időpont: 2026. aug.
> 18., K, 14:17):
>
>> Hi everyone,
>>
>> Unfortunately, yesterday's sync was not recorded due to a technical
>> issue. :(
>>
>> Here is a brief summary of the discussion and the decisions we reached:
>>
>>    1. We agreed to use a list of Iceberg expressions to define the
>>    *index_keys*, which determine how the index is ordered.
>>    2. We agreed not to restrict the functions that can be used in those
>>    expressions. Engines that do not understand a particular expression can
>>    simply choose not to use the index.
>>    3. We agreed that index region files (leaf files) should be sorted by
>>    the values generated by the index_keys.
>>    4. We discussed using *included_column_ids* to define the contents
>>    and schema of the index region files, and generally felt this could be a
>>    good approach.
>>    5. We started discussing whether the results of the index_key
>>    expressions should also be stored in the index region files alongside the
>>    included columns. In some cases, storing them could help readers optimize
>>    lookups within a region file. In other cases, the values could be derived
>>    when needed, making them redundant and potentially hurting performance by
>>    increasing the size of the region files.
>>
>> I think we should continue the discussion around point 5, as the outcome
>> may influence, or even reopen, the decisions around point 4.
>>
>> Thanks!
>>
>>
>> Flavio Junqueira <fl...@gmail.com> ezt írta (időpont: 2026.
>> aug. 4., K, 18:57):
>>
>>> Thank you for the summary, the proposal and the overall discussion. I
>>> have one main question about the target and I apologize in advance if I'm
>>> missing some important detail as I might not have the full context.
>>>
>>> I noticed that partitioning, sorting and clustering (via Hilbert) were
>>> discussed in the call in this context of index transform functions. My
>>> understanding is that engines currently are already pruning partitions and
>>> files using partition information and table metadata like per-column
>>> min-max. If that's the case, then what's it that this proposal is trying to
>>> achieve by having transform functions that capture partitioning, sorting
>>> and clustering when engines are already able to prune without a secondary
>>> index? I can more easily see the primary key index example, but I'm not
>>> entirely sure about these data layout mechanisms.
>>>
>>> I also didn't quite get how engines are expected to consume such an
>>> index. There was a discussion about where the mapping of the input to files
>>> happens, whether that's in Iceberg or the engine, and I didn't quite get
>>> the recommendation. Checking Peter's proposal, I couldn't see it clearly
>>> either, and if I just missed it, I'd appreciate a pointer to where I should
>>> look into.
>>>
>>> -Flavio
>>>
>>> On 4 Aug 2026, at 10:52, Péter Váry <pe...@gmail.com> wrote:
>>>
>>> Hi everyone,
>>>
>>> Here is the recording from yesterday’s sync:
>>> https://www.youtube.com/watch?v=Wg9orP9JAmk
>>>
>>> To summarize:
>>> - We agreed to retain the history of index snapshots, but not the
>>> history of other index properties. If readers require information that may
>>> later be modified by a user, that information should be copied into the
>>> corresponding snapshot.
>>> - We discussed the possibility of defining index ordering using a list
>>> of transform functions.
>>>
>>> Dan and Yingyi suggested something along the following lines:
>>>
>>> *"index_keys" : [*
>>> *  {*
>>> *    "type": "apply",*
>>> *    "func-name": {*
>>> *      "catalog": "iceberg_functions",*
>>> *      "name": "day"*
>>> *    },*
>>> *    "arguments": [*
>>> *      { "type": "reference", "id": 3 }*
>>> *    ]*
>>> *  },*
>>> *  {*
>>> *    "type": "apply",*
>>> *    "func-name": {*
>>> *      "catalog": "iceberg_functions",*
>>> *      "name": "truncate"*
>>> *    },*
>>> *    "arguments": [*
>>> *      { "type": "reference", "id": 4 },*
>>> *      2*
>>> *    ]*
>>> *  }*
>>> *]*
>>>
>>>
>>> In this example, rows would first be ordered by day(col3) and then by
>>> truncate(col4, 2). To ensure a deterministic ordering, ties would be broken
>>> using the original values of columns 3 and 4. If those are still equal,
>>> ordering would fall back to the file name and finally the row position.
>>>
>>> I spent some time thinking about this after the sync, and here are my
>>> thoughts.
>>>
>>> Pros:
>>> - Provides a high degree of flexibility.
>>> - Enables expression-based indexes.
>>>
>>> Cons:
>>> - Do we need this flexibility at this stage?
>>>    - Equality deletes, the primary use cases we are targeting, do not
>>> require expressions.
>>>    - Most indexes I have seen in production are not expression-based.
>>> - Increased complexity for engines:
>>>    - Engines must parse and understand transforms to determine whether
>>> using an index can improve a query. For example, is ordering by
>>> day(timestamp) sufficient for a key lookup?
>>>    - Complexity increases further as new UDFs are introduced.
>>>    - Expressions are not pushed into scans today, so index handling
>>> cannot be fully hidden behind Iceberg scan planning.
>>> - Risk of fragmentation:
>>>    - The additional complexity may slow engine adoption.
>>>    - Adopters may choose to support only exact transform matches created
>>> by them, rather than learn to reason about the general transforms.
>>>    - UDFs are prone to missing implementations and behavioral
>>> differences across engines. In the worst case, such differences could lead
>>> to incorrect query results when indexes are used.
>>>
>>> Given the points above, I still prefer defining HASH, HILBERT, IDENTITY
>>> as a set of standardized values combined with an explicit list of key
>>> columns. This keeps adoption simple, unlocks the primary use cases we are
>>> targeting today, and allows the Iceberg library to provide the transform
>>> implementations directly.
>>>
>>> Internally, these transforms could still be represented as Iceberg
>>> expressions, giving us a migration path toward future index types that may
>>> require full expression support.
>>>
>>> For example:
>>>
>>> *{*
>>> *  "transform": "HILBERT",*
>>> *  "key-column-ids": [3, 4]*
>>> *}*
>>>
>>>
>>> I'm interested in hearing other opinions.
>>>
>>> Thanks,
>>> Peter
>>>
>>> Renjie Liu <li...@gmail.com> ezt írta (időpont: 2026. júl. 28.,
>>> K, 10:07):
>>>
>>>> Hi, Peter:
>>>>
>>>> Thanks for driving the proposal.
>>>>
>>>> >  *Hash Index Type*: Should hash-based indexes be a separate index
>>>> type? They are only applicable to equality and IN predicates, and the hash
>>>> algorithm must always be known. A dedicated HASH type could make these
>>>> constraints explicit and distinguish hash-based indexes from ordered
>>>> indexes.
>>>>
>>>> I'm leaning toward keeping them in scalar index type due to the
>>>> similarity with others transformation functions like identity. As with the
>>>> constraints, I think they are properties of transformation functions.
>>>>
>>>> > *Transform Function/Layout Representation*: Should transform
>>>> functions be represented as an enum or a string? I lean toward strings, but
>>>> I do not have a strong preference. In many cases, an index can still be
>>>> used based on standard column statistics, even if a reader does not
>>>> recognize a particular transform.
>>>>
>>>> This seems more like a problem in java implementation?
>>>>
>>>> Also I prefer the name transform function compared with layout
>>>> representation. Layout representation makes me feel that it's not about the
>>>> definition of an index function.
>>>>
>>>> > *Future Index Types*: How should we handle index types that are
>>>> expected in the long term but fall outside the scope of this proposal (for
>>>> example, VECTOR, IVF, TERM)? Should they be mentioned as planned/reserved
>>>> concepts, or omitted entirely? I lean toward at least mentioning them. The
>>>> rationale for introducing an index type is easier to explain when
>>>> considering future index families.
>>>>
>>>> +1
>>>>
>>>> *> Index History Handling:* Most database systems treat index changes
>>>> as a drop-and-recreate operation rather than an in-place update. If we
>>>> follow the same approach, there is limited value in maintaining index
>>>> metadata history. During index maintenance, a new metadata.json can be
>>>> created and atomically swapped in, with the catalog responsible for
>>>> cleaning up obsolete metadata and index data files according to its
>>>> retention policy.
>>>>
>>>> +1, I think we only need to keep index data for different snapshots,
>>>> but if an index definition changed, we should drop and recreate a new index.
>>>>
>>>> > *File Format Location*: Should file format be specified at the
>>>> leaf-file level or only at the index level? My preference is to stay
>>>> aligned with the table metadata model and avoid introducing differences
>>>> unless there is a compelling requirement. This would simplify
>>>> implementation and maximize code reuse.
>>>>
>>>> +1.
>>>>
>>>>
>>>>
>>>> On Fri, Jul 24, 2026 at 11:57 PM Péter Váry <
>>>> peter.vary.apache@gmail.com> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> Here are the key takeaways from Monday’s meeting (July 20):
>>>>>
>>>>>    - *Index Type*: An index type enum will be introduced in v1,
>>>>>    initially with a single value (SCALAR). This establishes a future-proof API
>>>>>    boundary while leaving room for additional index families.
>>>>>    - *Strict Total Ordering*: The specification should require
>>>>>    transform functions to produce values that preserve a well-defined total
>>>>>    ordering across the key space. The transform function does not need to
>>>>>    return a long. Instead, leaf files can store the first and last values
>>>>>    according to the ordering, enabling generic filtering capabilities for
>>>>>    readers.
>>>>>
>>>>> Open Questions:
>>>>>
>>>>>    - *Hash Index Type*: Should hash-based indexes be a separate index
>>>>>    type? They are only applicable to equality and IN predicates, and the hash
>>>>>    algorithm must always be known. A dedicated HASH type could make these
>>>>>    constraints explicit and distinguish hash-based indexes from ordered
>>>>>    indexes.
>>>>>    - *Transform Function/Layout Representation*: Should transform
>>>>>    functions be represented as an enum or a string? I lean toward strings, but
>>>>>    I do not have a strong preference. In many cases, an index can still be
>>>>>    used based on standard column statistics, even if a reader does not
>>>>>    recognize a particular transform.
>>>>>    - *Future Index Types*: How should we handle index types that are
>>>>>    expected in the long term but fall outside the scope of this proposal (for
>>>>>    example, VECTOR, IVF, TERM)? Should they be mentioned as planned/reserved
>>>>>    concepts, or omitted entirely? I lean toward at least mentioning them. The
>>>>>    rationale for introducing an index type is easier to explain when
>>>>>    considering future index families.
>>>>>    - *Index History Handling:* Most database systems treat index
>>>>>    changes as a drop-and-recreate operation rather than an in-place update. If
>>>>>    we follow the same approach, there is limited value in maintaining index
>>>>>    metadata history. During index maintenance, a new metadata.json can be
>>>>>    created and atomically swapped in, with the catalog responsible for
>>>>>    cleaning up obsolete metadata and index data files according to its
>>>>>    retention policy.
>>>>>    - *Min/Max Statistics Representation*: Should transform-specific
>>>>>    or sort-order min/max values be represented using the existing file
>>>>>    statistics structure, or exposed through dedicated fields? My preference is
>>>>>    to keep them within the statistics structure defined by the file schema
>>>>>    while exposing them directly through the Java API if needed.
>>>>>    - *File Format Location*: Should file format be specified at the
>>>>>    leaf-file level or only at the index level? My preference is to stay
>>>>>    aligned with the table metadata model and avoid introducing differences
>>>>>    unless there is a compelling requirement. This would simplify
>>>>>    implementation and maximize code reuse.
>>>>>
>>>>> I have updated the spec PR based on the decisions above:
>>>>> https://github.com/apache/iceberg/pull/16961
>>>>>
>>>>> Thanks,
>>>>> Peter
>>>>>
>>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont: 2026. jún.
>>>>> 26., P, 19:42):
>>>>>
>>>>>> Hi all, Here is the summary of this Monday's index meeting:
>>>>>>
>>>>>> Two blocking decisions closed:
>>>>>>
>>>>>>    1. Index is *not* a table— it's its own object (reuses table
>>>>>>    machinery under the hood). Reasoning: requires a sort order, no column
>>>>>>    updates, no partition spec, no overlapping ranges between leaves, inherits
>>>>>>    base-table permissions, and has its own CREATE/DROP/UPDATE INDEX DDL.
>>>>>>    2. Index is a separate catalog entity, with no pointers in table
>>>>>>    metadata— has its own REST endpoints; the catalog can optionally return
>>>>>>    index metadata withloadTable to avoid extra round-trips. Keeps table and
>>>>>>    index updates independent/async.
>>>>>>
>>>>>>
>>>>>> Next steps: Start writing the spec and build out the copy-on-write
>>>>>> path now.
>>>>>>
>>>>>> Here are the draft spec:
>>>>>> secondary index spec <https://github.com/apache/iceberg/pull/16961>
>>>>>> irc spec <https://github.com/apache/iceberg/pull/16963>
>>>>>>
>>>>>> Thanks,
>>>>>> Huaxin
>>>>>>
>>>>>> On Sat, Jun 20, 2026 at 11:11 AM huaxin gao <hu...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I built a standalone PoC to validate that the basic index structure
>>>>>>> works: that we can build a PK index, convert equality deletes to position
>>>>>>> deletes through it, and have every converted delete land on the correct
>>>>>>> live row. I ran it up to *100M keys*.
>>>>>>>
>>>>>>> *Headline: the structure works.* The index builds over up to 100M
>>>>>>> keys, the eq-delete → position-delete conversion resolved correctly at
>>>>>>> *every* size (100% of converted deletes mapped to the right live
>>>>>>> row), and the resulting position deletes are *~8× cheaper to apply* at
>>>>>>> query time than the equality deletes they replace.
>>>>>>>
>>>>>>> Beyond correctness, the run also shows how the index’s *maintenance* cost
>>>>>>> scales, comparing copy-on-write (COW, rewrite touched leaves) vs an
>>>>>>> append/merge (MOR) option, under a realistic mixed CDC checkpoint (1,000
>>>>>>> insert + 500 update + 500 delete), local wall-clock:
>>>>>>> keys EQ baseline INDEX (COW) % of 60s (COW) INDEX (MOR) % of 60s
>>>>>>> (MOR) correct
>>>>>>> 5M 6 ms 6.7s 11.2% 2.2s 3.7% PASS
>>>>>>> 20M 8 ms 24.2s 40.4% 6.4s 10.6% PASS
>>>>>>> 50M 7 ms 51.6s 86.1% 12.2s 20.4% PASS
>>>>>>> *100M* 6 ms *75.0s* 125% (BEHIND) *16.9s* 28.2% (keeps up) PASS
>>>>>>>
>>>>>>> COW maintenance crosses the 60 s checkpoint around 100M (75 s/cycle,
>>>>>>> 125%); MOR stays at ~28% and keeps pace; the equality-delete baseline is
>>>>>>> ~6 ms and flat. So the structure works, but *COW alone can’t
>>>>>>> sustain scattered CDC at hundreds of millions of keys on a single writer*.
>>>>>>> It’s worth allowing a merge-on-read / update-file maintenance option
>>>>>>> alongside COW (or sharding the index across parallel writers).
>>>>>>>
>>>>>>> *Full write-up, all tables, and the in-region reality-check:* link
>>>>>>> <https://docs.google.com/document/d/1G3zxbW8X0eU3UrouslZfp42bBc9CvgJGnJyDONCB4PU/edit?tab=t.0>
>>>>>>>
>>>>>>> Feedback welcome, especially on the spec direction (whether to allow
>>>>>>> a merge-on-read / update-file maintenance option alongside COW)  and on the
>>>>>>> read-side modeling.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Huaxin
>>>>>>>
>>>>>>> On Tue, Jun 9, 2026 at 5:45 PM huaxin gao <hu...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Sorry,  we've skipped posting a few of the dedicated index-sync
>>>>>>>> summaries to the mailing list; you can find those in the Google doc
>>>>>>>> <https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?pli=1&tab=t.8041k7j2n7y3>
>>>>>>>> and the Slack channel. Here's yesterday's summary:
>>>>>>>>
>>>>>>>> *Decided*
>>>>>>>>
>>>>>>>>    -
>>>>>>>>
>>>>>>>>    Index vs. table (what we agreed):
>>>>>>>>    - Reuse table implementation/library code and a near-identical
>>>>>>>>       spec — the commit path will be custom regardless, so reuse isn't the
>>>>>>>>       deciding factor.
>>>>>>>>       - An index is not a table from a user/API view: loading or
>>>>>>>>       writing an index as a table must fail(it would violate index invariants).
>>>>>>>>       - The spec forbids most table behaviors: no overlapping
>>>>>>>>       files, one mandatory transform sort order, no column updates, no partition
>>>>>>>>       spec.
>>>>>>>>       - Delete vectors: reuse Iceberg's existing DV — benchmarks
>>>>>>>>    showed no new delete format is worth introducing.
>>>>>>>>    - Incremental updates: start with copy-on-write only (no update
>>>>>>>>    files). For object-store-sized leaves, a full leaf rewrite is about as
>>>>>>>>    cheap as maintaining an overlay update file + DV, so we'll skip the MOR
>>>>>>>>    machinery for now and add it later only if benchmarks prove we need it
>>>>>>>>    (likely just the very-large-leaf case).
>>>>>>>>    -
>>>>>>>>
>>>>>>>>    Validate the spec first: build a quick, hand-wired prototype
>>>>>>>>    (Parquet files structured per the spec) and benchmark it on real scales
>>>>>>>>    before formalizing.
>>>>>>>>
>>>>>>>> *Leaning, not final*
>>>>>>>>
>>>>>>>>    -
>>>>>>>>
>>>>>>>>    Indexes are likely separate catalog objects, linked from the
>>>>>>>>    table by storing just an identifier (like materialized views) and not
>>>>>>>>    visible in LIST TABLES.
>>>>>>>>    -
>>>>>>>>
>>>>>>>>    We'll need a commit path for indexes, but simpler than tables
>>>>>>>>    (no stage-create).
>>>>>>>>
>>>>>>>> *Still open*
>>>>>>>>
>>>>>>>>
>>>>>>>>    -
>>>>>>>>
>>>>>>>>    Permissions model — separate vs. inherited (action: look at
>>>>>>>>    what real DBs do for index permissions).
>>>>>>>>    - REST/catalog RPC design — minimize round-trips; index
>>>>>>>>    metadata ideally returned with LOAD TABLE. Catalog RPC cost may
>>>>>>>>    dominate Parquet IO, so this needs real design.
>>>>>>>>    -
>>>>>>>>
>>>>>>>>    Scale modeling — target rows-per-leaf vs. leaf size vs.
>>>>>>>>    metadata-file count.
>>>>>>>>    -
>>>>>>>>
>>>>>>>>    DDL-on-index semantics (reuse table schema-update actions or
>>>>>>>>    separate)
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Huaxin
>>>>>>>>
>>>>>>>> On Wed, Apr 22, 2026 at 8:47 AM Péter Váry <
>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> TL;DR
>>>>>>>>> We still need to validate with ADLS and S3, but based on the local
>>>>>>>>> tests, the MPHF approach looks more promising if we can tolerate larger
>>>>>>>>> files and longer index maintenance times.
>>>>>>>>>
>>>>>>>>> Details:
>>>>>>>>> Here are the results from the local experiments on my Mac. I
>>>>>>>>> removed unnecessary statistics from the Parquet files and tested different
>>>>>>>>> row group sizes:
>>>>>>>>>
>>>>>>>>>    - For an index file with 1M records, a row group size of 5,000
>>>>>>>>>    appears to be the sweet spot.
>>>>>>>>>    - For 10M records, 10,000 rows per row group works best.
>>>>>>>>>
>>>>>>>>> If you have additional ideas for optimizing Parquet-based indexes,
>>>>>>>>> I’d be very interested to hear them.
>>>>>>>>> The test code is available on this branch:
>>>>>>>>> https://github.com/pvary/iceberg/tree/leaf_bench
>>>>>>>>>
>>>>>>>>> Best results:
>>>>>>>>> *1m records/file*
>>>>>>>>>
>>>>>>>>>    - Parquet - 5000 row/RowGroup
>>>>>>>>>       - Read: 1191 µs - 1 file open, 3 seek, 123KB read per lookup
>>>>>>>>>       - Write: 1.7 s, 15 MB
>>>>>>>>>    - MPHF
>>>>>>>>>       - Read: 202 µs - 1 file open, 1 seek,  282KB read per lookup
>>>>>>>>>       - Write: 0.8 s, 34 MB
>>>>>>>>>
>>>>>>>>> *10m records/file*
>>>>>>>>>
>>>>>>>>>    - Parquet - 10000 row/RowGroup
>>>>>>>>>       - Read: 4168 µs - 1 file open, 3 seek, 395KB read per lookup
>>>>>>>>>       - Write: 19.5s s, 144 MB
>>>>>>>>>    - MPHF
>>>>>>>>>       - Read: 1086 µs - 1 file open, 1 seek,  2.8 MB (2812KB)
>>>>>>>>>       read per lookup
>>>>>>>>>       - Write: 6.5 s, 34 MB, 353 MB
>>>>>>>>>
>>>>>>>>> Below are the full results.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *Benchmark                                      (indexType)
>>>>>>>>>  (keyType)  (numRows)  Mode    Cnt            Score          Error
>>>>>>>>>  UnitsInvertedIndexBenchmark.lookup                 PARQUET_1000       LONG
>>>>>>>>>    1000000    ss  10000         3285.284 ±        5.138
>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_1000       LONG
>>>>>>>>>    1000000    ss  10000   2522168989.000
>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_1000       LONG
>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_1000       LONG
>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_1000       LONG
>>>>>>>>> 10000000    ss  10000        35449.614 ±       34.673
>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_1000       LONG
>>>>>>>>>   10000000    ss  10000  24302649201.000
>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_1000       LONG
>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_1000       LONG
>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_5000       LONG
>>>>>>>>>  1000000    ss  10000         1191.959 ±        4.169
>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_5000       LONG
>>>>>>>>>    1000000    ss  10000   1230877229.000
>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_5000       LONG
>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_5000       LONG
>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>> #InvertedIndexBenchmark.lookup                 PARQUET_5000       LONG
>>>>>>>>> 10000000    ss  10000         7236.447 ±       10.374
>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead       PARQUET_5000       LONG
>>>>>>>>>   10000000    ss  10000   5650715973.000
>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams     PARQUET_5000       LONG
>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks           PARQUET_5000       LONG
>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_10000       LONG
>>>>>>>>>  1000000    ss  10000         1349.946 ±        7.834
>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_10000       LONG
>>>>>>>>>    1000000    ss  10000   1730219377.000
>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_10000       LONG
>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_10000       LONG
>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_10000       LONG
>>>>>>>>> 10000000    ss  10000         4168.635 ±       11.051
>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_10000       LONG
>>>>>>>>>   10000000    ss  10000   3946341532.000
>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_10000       LONG
>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_10000       LONG
>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_50000       LONG
>>>>>>>>>  1000000    ss  10000         4736.466 ±       38.179
>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_50000       LONG
>>>>>>>>>    1000000    ss  10000   7427413541.000
>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_50000       LONG
>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_50000       LONG
>>>>>>>>>  1000000    ss  10000        30000.000
>>>>>>>>> #InvertedIndexBenchmark.lookup                PARQUET_50000       LONG
>>>>>>>>> 10000000    ss  10000         4979.031 ±       34.708
>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead      PARQUET_50000       LONG
>>>>>>>>>   10000000    ss  10000   7694887636.000
>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams    PARQUET_50000       LONG
>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks          PARQUET_50000       LONG
>>>>>>>>> 10000000    ss  10000        30000.000
>>>>>>>>> #InvertedIndexBenchmark.lookup                         MPHF       LONG
>>>>>>>>>  1000000    ss  10000          202.571 ±        2.336
>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead               MPHF       LONG
>>>>>>>>>    1000000    ss  10000   2821570000.000
>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams             MPHF       LONG
>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks                   MPHF       LONG
>>>>>>>>>  1000000    ss  10000        10000.000
>>>>>>>>> #InvertedIndexBenchmark.lookup                         MPHF       LONG
>>>>>>>>> 10000000    ss  10000         1086.957 ±        4.524
>>>>>>>>>  us/opInvertedIndexBenchmark.lookup:bytesRead               MPHF       LONG
>>>>>>>>>   10000000    ss  10000  28119460000.000
>>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams             MPHF       LONG
>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>> #InvertedIndexBenchmark.lookup:seeks                   MPHF       LONG
>>>>>>>>> 10000000    ss  10000        10000.000
>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_1000       LONG
>>>>>>>>>  1000000    ss      3      1720731.014 ±   876636.004
>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_1000       LONG
>>>>>>>>>    1000000    ss      3     46453317.000
>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_1000       LONG
>>>>>>>>> 10000000    ss      3     18547947.876 ± 12258125.307
>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_1000       LONG
>>>>>>>>>   10000000    ss      3    452655675.000
>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_5000       LONG
>>>>>>>>>  1000000    ss      3      1718345.583 ±  1103928.016
>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_5000       LONG
>>>>>>>>>    1000000    ss      3     44845788.000
>>>>>>>>> #InvertedIndexBenchmark.write                  PARQUET_5000       LONG
>>>>>>>>> 10000000    ss      3     18604229.931 ±  2668361.915
>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes   PARQUET_5000       LONG
>>>>>>>>>   10000000    ss      3    435388818.000
>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_10000       LONG
>>>>>>>>>  1000000    ss      3      1761555.389 ±   535857.675
>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_10000       LONG
>>>>>>>>>    1000000    ss      3     44536635.000
>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_10000       LONG
>>>>>>>>> 10000000    ss      3     19501588.264 ±  2130054.558
>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_10000       LONG
>>>>>>>>>   10000000    ss      3    433189623.000
>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_50000       LONG
>>>>>>>>>  1000000    ss      3      1936624.889 ±  6601363.985
>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_50000       LONG
>>>>>>>>>    1000000    ss      3     44264655.000
>>>>>>>>> #InvertedIndexBenchmark.write                 PARQUET_50000       LONG
>>>>>>>>> 10000000    ss      3     20471742.278 ± 10705206.310
>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes  PARQUET_50000       LONG
>>>>>>>>>   10000000    ss      3    431311305.000
>>>>>>>>> #InvertedIndexBenchmark.write                          MPHF       LONG
>>>>>>>>>  1000000    ss      3       896573.958 ±  1408024.851
>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes           MPHF       LONG
>>>>>>>>>    1000000    ss      3    102846369.000
>>>>>>>>> #InvertedIndexBenchmark.write                          MPHF       LONG
>>>>>>>>> 10000000    ss      3      6509348.875 ± 15519975.479
>>>>>>>>>  us/opInvertedIndexBenchmark.write:indexFileBytes           MPHF       LONG
>>>>>>>>>   10000000    ss      3   1058435733.000                     #*
>>>>>>>>>
>>>>>>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont: 2026. ápr.
>>>>>>>>> 21., K, 20:53):
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> In recent secondary index sync meetings, the discussion converged
>>>>>>>>>> on the need to define what an index is from first principles before
>>>>>>>>>> settling on physical layout.
>>>>>>>>>>
>>>>>>>>>> To address that, Peter and I have drafted a requirements document
>>>>>>>>>> for a key lookup index (renamed from "primary key index" to avoid implying
>>>>>>>>>> uniqueness enforcement), the goal is to nail down one well-scoped index
>>>>>>>>>> type first.
>>>>>>>>>>
>>>>>>>>>> Doc: Key Lookup Index Requirements
>>>>>>>>>> <https://docs.google.com/document/d/1e0zxK-jA0LBDq8YQlQgFipTHelDFiga8lCkgDTmYub8/edit?tab=t.0#heading=h.8shrgabvl19>
>>>>>>>>>>
>>>>>>>>>> It covers requirements, three design options (manifest + sorted
>>>>>>>>>> Parquet, hash + sorted Parquet, hash + MPHF) and open questions. We will
>>>>>>>>>> add preliminary benchmark results shortly.
>>>>>>>>>>
>>>>>>>>>> Feedback welcome — inline in the doc, on this thread, or at the
>>>>>>>>>> next index sync.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Huaxin
>>>>>>>>>>
>>>>>>>>>> On Mon, Apr 13, 2026 at 7:22 AM Steven Wu <st...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Do we need the special index identifier that was originally
>>>>>>>>>>> proposed? A generic CatalogObjectIdentifier (with namespace and name) would
>>>>>>>>>>> be consistent with all object types in the catalog. I have a discussion
>>>>>>>>>>> thread on the generic identifier topic: [DISCUSS] REST Spec:
>>>>>>>>>>> generic CatalogObjectIdentifier.
>>>>>>>>>>>
>>>>>>>>>>> Should we add an indexes array field to table metadata? It only
>>>>>>>>>>> contains a list of index object identifiers. It doesn't contain any index
>>>>>>>>>>> metadata which should live in the index objects. Yufei was trying to bring
>>>>>>>>>>> this up at the end of the first sync. But we didn't get enough time to
>>>>>>>>>>> really discuss it. It will be great to discuss this as the first agenda
>>>>>>>>>>> item today.
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Apr 13, 2026 at 3:17 AM Péter Váry <
>>>>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>
>>>>>>>>>>>> We had several engaging discussions at the Iceberg Summit, and
>>>>>>>>>>>> it was great to finally catch up with many of you in person. We truly
>>>>>>>>>>>> missed those who couldn’t attend, hopefully we’ll all meet again at the
>>>>>>>>>>>> next summit.
>>>>>>>>>>>>
>>>>>>>>>>>> To keep the conversation going, Huaxin and I have put together
>>>>>>>>>>>> the agenda for our next meeting. As a reminder, we’ll meet on *April
>>>>>>>>>>>> 13th, 9:00–10:00 AM *PDT (6:00–7:00 PM CEST).
>>>>>>>>>>>>
>>>>>>>>>>>> Proposed agenda:
>>>>>>>>>>>>
>>>>>>>>>>>>    - Continue first-principles index design discussion from
>>>>>>>>>>>>    Mar 30
>>>>>>>>>>>>       - *Index Ownership and Write Responsibility*
>>>>>>>>>>>>          - Should writers be allowed to update indexes, or
>>>>>>>>>>>>          - Should all index writes be handled exclusively by
>>>>>>>>>>>>          the Index Maintenance process?
>>>>>>>>>>>>          - If writers can update indexes then we need to
>>>>>>>>>>>>          define what guarantees are required (compaction, file splitting, layout
>>>>>>>>>>>>          expectations)?
>>>>>>>>>>>>          - If only Index Maintenance updates indexes then we
>>>>>>>>>>>>          only need to define what observable properties should be exposed to
>>>>>>>>>>>>          consumers? Like:
>>>>>>>>>>>>             - Expected max files for a single key
>>>>>>>>>>>>             - Current max files for a single key
>>>>>>>>>>>>             - Deletes allowed/present
>>>>>>>>>>>>             - Sorted by
>>>>>>>>>>>>             - Partitioned by
>>>>>>>>>>>>          - *Specification Scope: What Belongs in the Spec?*
>>>>>>>>>>>>          - Related to the ownership question above
>>>>>>>>>>>>          - Light spec: Just define that the index table should
>>>>>>>>>>>>          be optimized for retrieval by key columns and the index columns should be
>>>>>>>>>>>>          contained in the table. This could give us more flexibility if better
>>>>>>>>>>>>          organization methods come up, or
>>>>>>>>>>>>          - Detailed spec: We could define the max number of
>>>>>>>>>>>>          files per index to read for a single key, or even the partitioning and the
>>>>>>>>>>>>          exact sort order. This could allow more use-cases for a given index, like
>>>>>>>>>>>>          joins or cardinality estimations.
>>>>>>>>>>>>          - I would go for light spec for the main types (PK,
>>>>>>>>>>>>          Containing) and only the Index Maintenance processes should update the
>>>>>>>>>>>>          Indexes, as for many use-cases the details are not important, and writers
>>>>>>>>>>>>          will very rarely update the Indexes themselves.
>>>>>>>>>>>>       - *Logical Placement of Indexes*
>>>>>>>>>>>>          - Index as a child object of an Iceberg Table, or
>>>>>>>>>>>>          - Index as a first‑class entity under
>>>>>>>>>>>>          /namespace/indexes/{index}
>>>>>>>>>>>>          - Based on the discussions on the summit we are
>>>>>>>>>>>>          leaning in this direction. This means the index id should be unique in the
>>>>>>>>>>>>          namespace but helps the catalog implementations quite a bit
>>>>>>>>>>>>       - *Physical Placement of Index Data*
>>>>>>>>>>>>          - I don’t think we should specify this. We should
>>>>>>>>>>>>          have a base location for the index, but can rely on the catalog
>>>>>>>>>>>>          implementations to decide on their own, like they do with the tables,
>>>>>>>>>>>>          views, udfs.
>>>>>>>>>>>>       - *Iceberg Reader Based indexes* (Containing indexes and
>>>>>>>>>>>>       potentially PK indexes). These are the indexes which could be read by the
>>>>>>>>>>>>       existing Iceberg readers. We might decide to store the PK index similarly
>>>>>>>>>>>>       to an Iceberg Table and treat it as a reader based index.
>>>>>>>>>>>>          - What are the table properties/features exposed to
>>>>>>>>>>>>          the readers
>>>>>>>>>>>>             - Maybe just some behavioral descriptors for the
>>>>>>>>>>>>             optimizer to decide if the index could be used or should be skipped, like:
>>>>>>>>>>>>                - Expected max files for a single key
>>>>>>>>>>>>                - max files for a single key
>>>>>>>>>>>>                - Deletes allowed/present
>>>>>>>>>>>>                - Sorted by
>>>>>>>>>>>>                - Partitioned by
>>>>>>>>>>>>             - The Tasks when reading the index based on the
>>>>>>>>>>>>             filters and projection
>>>>>>>>>>>>          - What are the table properties/features exposed to
>>>>>>>>>>>>          the Index Maintenance. I think this could be internal to the Index
>>>>>>>>>>>>          Maintenance process and might not be exposed through the spec. The Index
>>>>>>>>>>>>          Maintenance process could handle this as a standard Iceberg Table and could
>>>>>>>>>>>>          be based on the Table Maintenance process, but there might be some totally
>>>>>>>>>>>>          different processes.
>>>>>>>>>>>>       - It should be possible to add properties to an index
>>>>>>>>>>>>       defined by the Index Maintenance process which could be used and updated in
>>>>>>>>>>>>       the next Index Maintenance run.
>>>>>>>>>>>>    - *PK index storage format benchmark results*
>>>>>>>>>>>>       - Flat Parquet (baseline)
>>>>>>>>>>>>       - BTree with Parquet leaves
>>>>>>>>>>>>       - Vortex
>>>>>>>>>>>>    - *Open items / next steps*
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Peter
>>>>>>>>>>>>
>>>>>>>>>>>> huaxin gao <hu...@gmail.com> ezt írta (időpont: 2026.
>>>>>>>>>>>> márc. 23., H, 3:03):
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi everyone, I wanted to share an update on the primary key
>>>>>>>>>>>>> index work.
>>>>>>>>>>>>> Since there are still open questions on whether bloom filter
>>>>>>>>>>>>> indexes fit in the secondary index framework or should be treated as
>>>>>>>>>>>>> extended stats, I've shifted focus to the primary key index since it's a
>>>>>>>>>>>>> clearer fit for the framework.
>>>>>>>>>>>>> I've put together a proposal for a primary key reverse-lookup
>>>>>>>>>>>>> index that maps each key to its physical location (file_path,
>>>>>>>>>>>>> row_position). It enables:
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - Scan-time file pruning for point lookups
>>>>>>>>>>>>>    - Converting key-based deletes into position deletes
>>>>>>>>>>>>>    (eliminating equality deletes for Flink CDC)
>>>>>>>>>>>>>    - Accelerating Spark MERGE INTO by replacing full-table
>>>>>>>>>>>>>    joins with direct file lookups
>>>>>>>>>>>>>
>>>>>>>>>>>>> Proposal:
>>>>>>>>>>>>> https://docs.google.com/document/d/1HuhCZ0n2FqDh8yqQb9oEj1CPM5yXpEsMPGZno2aSf8E/edit?tab=t.0#heading=h.tbevg4q0m9
>>>>>>>>>>>>> Feedback welcome!
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Huaxin
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Mar 18, 2026 at 11:42 PM Péter Váry <
>>>>>>>>>>>>> peter.vary.apache@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Key takeaways from the general index discussion at the May 16
>>>>>>>>>>>>>> meeting.
>>>>>>>>>>>>>> Thanks to everyone who participated! The recording is
>>>>>>>>>>>>>> available here: https://www.youtube.com/watch?v=btmjhtRWUCE
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - Q: Do we need to tie index types to the algorithms used
>>>>>>>>>>>>>>    to access them?
>>>>>>>>>>>>>>    - A: From a specification perspective, the goal is to
>>>>>>>>>>>>>>    define the storage-level data layout so it can be shared across engines.
>>>>>>>>>>>>>>    Engines are free to interpret and use the data as they see fit, but the
>>>>>>>>>>>>>>    on-disk data layout itself must be strictly defined and interoperable.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - Q: Should we introduce an additional abstraction layer
>>>>>>>>>>>>>>    (e.g., Vector Index) with sub-types such as IVF and DiskANN?
>>>>>>>>>>>>>>    - A: This is possible if we decide it is beneficial. I
>>>>>>>>>>>>>>    explored potential naming, but it is not yet clear how such a layer would
>>>>>>>>>>>>>>    be used in practice.
>>>>>>>>>>>>>>    *Question to Yingyi Bu*: could you provide examples where
>>>>>>>>>>>>>>    this additional layer would be useful? Should this abstraction be defined
>>>>>>>>>>>>>>    at the spec level, or is it better handled at the engine level?
>>>>>>>>>>>>>>    My initial idea was that users would create a generic
>>>>>>>>>>>>>>    Vector Index and let the engine choose the concrete implementation.
>>>>>>>>>>>>>>    However, this would limit user control and users likely need to specify the
>>>>>>>>>>>>>>    exact index representation, which implies they must be aware of the
>>>>>>>>>>>>>>    available representations.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - Q: Do we want to allow extensibility for index types?
>>>>>>>>>>>>>>    - A: Yes. The intent is to support a small set of
>>>>>>>>>>>>>>    well-defined index types while allowing experimentation with new ones. If a
>>>>>>>>>>>>>>    new index type proves broadly useful, a follow-up proposal can standardize
>>>>>>>>>>>>>>    it and incorporate it into the spec.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - Q: Do we allow multiple versions of an index for the
>>>>>>>>>>>>>>    same table snapshot?
>>>>>>>>>>>>>>    - A: Yes. Older index versions must be retained for
>>>>>>>>>>>>>>    readers that have already started using them, while new readers should
>>>>>>>>>>>>>>    automatically use the latest available version
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - Q: Do we need to use materialized views for these
>>>>>>>>>>>>>>    indexes?
>>>>>>>>>>>>>>    - A: No. These indexes are primarily examples, and
>>>>>>>>>>>>>>    different types may require different storage methods. However, the Primary
>>>>>>>>>>>>>>    Key, Containing, and parts of the IVF indexes can be structured as Iceberg
>>>>>>>>>>>>>>    tables. This allows engines to read them natively; in some cases, Iceberg
>>>>>>>>>>>>>>    planners can automatically redirect queries to the index table without
>>>>>>>>>>>>>>    engine modifications. Furthermore, index maintenance for these tables can
>>>>>>>>>>>>>>    leverage existing materialized view maintenance workflows. Other index
>>>>>>>>>>>>>>    types may instead rely on Puffin files or alternative storage approaches.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - Q: How should index metadata be accessed? Should we add
>>>>>>>>>>>>>>    explicit pointers for the indexes in the table metadata?
>>>>>>>>>>>>>>    - A: We did not have sufficient time to fully explore and
>>>>>>>>>>>>>>    conclude this topic.
>>>>>>>>>>>>>>    *Question for Yufei Gu*: Did I understand correctly that
>>>>>>>>>>>>>>    your main concern stems from endpoint resolution from a REST Catalog
>>>>>>>>>>>>>>    perspective? Specifically, if indexes are exposed under a URI such as
>>>>>>>>>>>>>>    v1/{prefix}/namespaces/{namespace}/tables/{table}/indexes/{index}, would
>>>>>>>>>>>>>>    this make it more difficult for the REST Catalog to resolve and route
>>>>>>>>>>>>>>    requests to the appropriate endpoint?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Suhas Jayaram Subramanya via dev <de...@iceberg.apache.org>
>>>>>>>>>>>>>> ezt írta (időpont: 2026. márc. 13., P, 23:32):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Here's a proposal for native Vector Index support in Iceberg
>>>>>>>>>>>>>>> tables --
>>>>>>>>>>>>>>> https://docs.google.com/document/d/1KL4qLOwdqnhOcqTc0EjO1O16NV3M3c-gZCEINDWw4lA/edit?usp=sharing
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We've been working on this proposal with Peter internally at
>>>>>>>>>>>>>>> Microsoft and he suggested we post it here to bring this to the community's
>>>>>>>>>>>>>>> attention, ahead of the next Secondary Index Sync.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Suhas
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 2026/02/19 04:34:34 huaxin gao wrote:
>>>>>>>>>>>>>>> > Hi Everyone,
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Here are the recording and notes from the Iceberg Index
>>>>>>>>>>>>>>> Support Sync on
>>>>>>>>>>>>>>> > 2/11.
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Recording: https://www.youtube.com/watch?v=3sFfQ0A50yk
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Notes:
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.8041k7j2n7y3
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > The meeting will move to biweekly, Mondays 9–10am PST,
>>>>>>>>>>>>>>> starting March 2.
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Since the sync, I updated the Bloom skipping index proposal
>>>>>>>>>>>>>>> > <
>>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.5r5kl6k3fqwu
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > to address the discussion questions, specifically:
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > - Performance justification: when this helps
>>>>>>>>>>>>>>> (high-cardinality = / IN,
>>>>>>>>>>>>>>> > many data files, high object-store latency) and how it
>>>>>>>>>>>>>>> differs from Parquet
>>>>>>>>>>>>>>> > row-group Bloom filters (which still require opening the
>>>>>>>>>>>>>>> data file).
>>>>>>>>>>>>>>> > - Cost / scalability: rough sizing (Bloom blob size per
>>>>>>>>>>>>>>> file, Puffin
>>>>>>>>>>>>>>> > file size), the planning cost trade-off (driver index
>>>>>>>>>>>>>>> reads vs executor
>>>>>>>>>>>>>>> > file opens), and mitigations via caching.
>>>>>>>>>>>>>>> > - Lifecycle / maintenance: incremental production as new
>>>>>>>>>>>>>>> data files
>>>>>>>>>>>>>>> > arrive, behavior when the index is missing/behind, and
>>>>>>>>>>>>>>> sharding/compaction
>>>>>>>>>>>>>>> > plus cleanup to avoid accumulating too many small Puffin
>>>>>>>>>>>>>>> files over time.
>>>>>>>>>>>>>>> > - Writer expectations: inline (optional) vs asynchronous
>>>>>>>>>>>>>>> (primary) index
>>>>>>>>>>>>>>> > creation.
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > I also implemented a Spark 4.1 POC
>>>>>>>>>>>>>>> > <https://github.com/apache/iceberg/pull/15311> and a
>>>>>>>>>>>>>>> local benchmark to
>>>>>>>>>>>>>>> > quantify both the pruning impact (plannedFiles →
>>>>>>>>>>>>>>> afterBloom) and the index
>>>>>>>>>>>>>>> > read overhead (statsFiles, statsBytes, bloomPayloadBytes)
>>>>>>>>>>>>>>> for point
>>>>>>>>>>>>>>> > predicates on high-cardinality columns. Please take a look
>>>>>>>>>>>>>>> and let me know
>>>>>>>>>>>>>>> > if you have any questions or feedback.
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Huaxin
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > On Tue, Feb 10, 2026 at 1:43 PM huaxin gao <
>>>>>>>>>>>>>>> hu...@gmail.com> wrote:
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > > Reminder for tomorrow's sync on Iceberg Index Support.
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > > Wednesday: Feb. 11 9:00 – 10:00am
>>>>>>>>>>>>>>> > > Time zone: America/Los_Angeles
>>>>>>>>>>>>>>> > > Google Meet joining info
>>>>>>>>>>>>>>> > > Video call link: meet.google.com/nsp-ctyr-khk
>>>>>>>>>>>>>>> > > Design doc:
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.0#heading=h.hs6r9d26w1y2
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.qouk73o4jxx7
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > > Thanks,
>>>>>>>>>>>>>>> > > Huaxin
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > > On Tue, Feb 3, 2026 at 10:52 PM Péter Váry <
>>>>>>>>>>>>>>> pe...@gmail.com>
>>>>>>>>>>>>>>> > > wrote:
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >> Thanks Huaxin and Steven for organizing this. Looking
>>>>>>>>>>>>>>> forward to meet you
>>>>>>>>>>>>>>> > >> all next week!
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >> On Wed, Feb 4, 2026, 02:48 Steven Wu <st...@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>>> > >>> We set up the dev calendar event with a new google
>>>>>>>>>>>>>>> meet link. Please
>>>>>>>>>>>>>>> > >>> ignore the link from Huaxin's original email.
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>> The dev calendar has the correct info (including the
>>>>>>>>>>>>>>> new meeting link)
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>> Iceberg Index Support Sync
>>>>>>>>>>>>>>> > >>> Wednesday, February 11 · 9:00 – 10:00am
>>>>>>>>>>>>>>> > >>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>>> > >>> Google Meet joining info
>>>>>>>>>>>>>>> > >>> Video call link: https://meet.google.com/nsp-ctyr-khk
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>> On Tue, Feb 3, 2026 at 5:08 PM huaxin gao <
>>>>>>>>>>>>>>> hu...@gmail.com>
>>>>>>>>>>>>>>> > >>> wrote:
>>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>>> > >>>> Sorry, I meant PST (not EST) :)
>>>>>>>>>>>>>>> > >>>> Looking forward to the discussion!
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>> On Tue, Feb 3, 2026 at 4:58 PM Shawn Chang <
>>>>>>>>>>>>>>> ye...@gmail.com>
>>>>>>>>>>>>>>> > >>>> wrote:
>>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>>> > >>>>> Hi Huaxin,
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>> Thanks for starting the sync!
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>> The meeting seems to be 9-10AM PST on the dev events
>>>>>>>>>>>>>>> calendar
>>>>>>>>>>>>>>> > >>>>> <
>>>>>>>>>>>>>>> https://calendar.google.com/calendar/u/0?cid=MzkwNWQ0OTJmMWI0NTBiYTA3MTJmMmFlNmFmYTc2ZWI3NTdmMTNkODUyMjBjYzAzYWE0NTI3ODg1YWRjNTYyOUBncm91cC5jYWxlbmRhci5nb29nbGUuY29t
>>>>>>>>>>>>>>> >,
>>>>>>>>>>>>>>> > >>>>> not EST. Maybe it's a typo?
>>>>>>>>>>>>>>> > >>>>> Otherwise, looking forward to the discussion!
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>> Best,
>>>>>>>>>>>>>>> > >>>>> Shawn
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>> On Tue, Feb 3, 2026 at 9:18 AM huaxin gao <
>>>>>>>>>>>>>>> hu...@gmail.com>
>>>>>>>>>>>>>>> > >>>>> wrote:
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> > >>>>>> Hi all,
>>>>>>>>>>>>>>> > >>>>>> I'd like to start a dedicated sync to discuss
>>>>>>>>>>>>>>> Iceberg Index support.
>>>>>>>>>>>>>>> > >>>>>> Here is the existing discussion thread:
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> https://lists.apache.org/thread/fzqk3jjf0xpj5m4cfqb3v4c65p0t04ty
>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>> To ground the discussion, here are the two
>>>>>>>>>>>>>>> proposals:
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>> - Peter's proposal
>>>>>>>>>>>>>>> > >>>>>> <
>>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.0#heading=h.hs6r9d26w1y2>
>>>>>>>>>>>>>>> (overall
>>>>>>>>>>>>>>> > >>>>>> index support)
>>>>>>>>>>>>>>> > >>>>>> - My proposal
>>>>>>>>>>>>>>> > >>>>>> <
>>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.qouk73o4jxx7
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > >>>>>> (bloom filter skipping index)
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>> Time slot: Every 3 weeks, Wednesdays at 9 AM to 10
>>>>>>>>>>>>>>> AM EST, starting
>>>>>>>>>>>>>>> > >>>>>> next Wednesday (2/11). After FileFormat sync
>>>>>>>>>>>>>>> finishes, we plan to use that
>>>>>>>>>>>>>>> > >>>>>> slot and switch to every other Monday, 9 AM to 10
>>>>>>>>>>>>>>> AM EST.
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>> Meet link: https://meet.google.com/fjn-tyze-mko
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>>>>>> > >>>>>> Huaxin
>>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>