You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@fluss.apache.org by "Liting Liu (litiliu) via dev" <de...@fluss.apache.org> on 2026/06/30 09:37:06 UTC
回复: [DISCUSS] Draft design: Support upgrading existing Iceberg tables to Fluss tables
Hi all,
Thanks a lot for the detailed review comments. I updated the Iceberg create_from_lake draft based on the first round of feedback:
https://docs.google.com/document/d/1REmpQ2r5W12OvW2CU9tNjDWjpkFJegiTZnE1IZbBH_U/edit?tab=t.0
The main updates are:
1. Partition mapping
Iceberg transform partition support is treated as a follow-up dependency on the proposed Fluss implicit partition design:
https://docs.google.com/document/d/1wDOkB-MuLrn3QYqL0fMlr1rj-lq891UbBcqXJSzCNcU/edit?tab=t.0
If that proposal is accepted, supported transforms such as day(ts) can be mapped to Fluss virtual partition keys. Until then, transform partitions are rejected by create_from_lake with a clear validation error. For partition spec evolution, Fluss only maps the latest/current Iceberg partition spec into Fluss metadata, while historical data reading still goes through the Iceberg reader for the captured snapshot.
2. Bucket configuration
For Iceberg lake primary-key tables, Fluss uses the Iceberg bucketing strategy. A compatible bucket(N, ...) transform provides the Fluss bucket count. If the Iceberg table has no bucket transform, users must specify bucket.num for the Fluss primary-key table.
3. Initial readable LakeSnapshot
Both log-table and primary-key paths now register an initial readable LakeSnapshot using the captured Iceberg snapshot ID plus Fluss bucket offsets. This makes the union-read boundary explicit.
4. Delete handling and deletion vectors
The boundary remains the Iceberg reader. Fluss does not implement custom equality-delete, position-delete, or deletion-vector merging.
5. Write freeze
The source Iceberg table is expected to be externally frozen during create_from_lake. Fluss captures the current Iceberg snapshot ID and re-checks it before exposing the created Fluss table/partition as ready.
6. Identifier-field duplicates
Iceberg identifier fields are advisory. In v1, Fluss does not add a full uniqueness validation during bootstrap. KV rebuild follows Fluss upsert semantics, and duplicate identifier values are not guaranteed to produce a deterministic retained row.
7. Unsupported types and empty tables
Unsupported Iceberg column types are rejected during metadata/schema mapping before bootstrap starts. Tables without a current Iceberg snapshot are also rejected in v1.
Please take another look when you have time.
Best,
Liting
________________________________
发件人: Mehul Batra <me...@gmail.com>
发送时间: 2026年6月26日 19:23
收件人: dev@fluss.apache.org <de...@fluss.apache.org>
主题: Re: [DISCUSS] Draft design: Support upgrading existing Iceberg tables to Fluss tables
Hi Liting,
Thanks for the FIP, and agreed that building Iceberg support on top of
FIP-25 is a great starting point.
In addition to the partition and bucket questions above, a few more from my
side:
1: Delete handling vs deletion vectors. The draft routes equality/position
deletes through the Iceberg reader and rejects tables the reader can't
resolve. On our side we're planning to move Iceberg PK toward deletion
vectors and v3 rather than delete files. Would the bootstrap reader treat a
v3 table with DVs the same way, i.e. just read snapshot-visible rows
through the reader, or does DV support shift the reject boundary at all?
2: Write freeze enforcement. The flow assumes writes to the source Iceberg
table are stopped and bootstrap reads a fixed snapshot. Is that freeze
enforced or advisory? For example, if a commit lands on the source table
after we capture the snapshot but before bootstrap finishes, do we detect
it, or are those rows silently missed?
3: Identifier-field uniqueness. Iceberg identifier fields are advisory and
don't enforce uniqueness. If two rows share the same identifier values in
the source, what does the KV rebuild do, last-writer-wins overwrite or
detect and reject? It would be good to state the expected behavior
explicitly.
4: v3 type mapping. As we move toward v3, types like Variant and nanosecond
timestamps come into scope. Are there source column types we can't map to
Fluss yet, and if so do we reject at create_from_lake time or fail
mid-bootstrap?
Looking forward to your reply.
Best Regards,
Mehul Batra
On Fri, Jun 19, 2026 at 9:04 PM Zhe Wang <ja...@gmail.com> wrote:
> Thanks for drafting this. The overall direction of keeping this as a delta
> design over FIP-25 looks reasonable to me.
>
> A few questions came to mind:
>
> 1. For the log-table path, should we still capture and persist the source
> Iceberg snapshot ID even though no KV bootstrap is required? It could serve
> as the historical read boundary and make the no-catch-up behavior explicit.
>
> 2. For mapping Iceberg identifier fields to Fluss primary keys, can we make
> the validation and failure semantics more explicit? Iceberg identifier
> fields are not enforced uniqueness constraints, so it would be good to
> define what happens if duplicate identifier values exist in the source
> snapshot.
>
> 3. Should the design explicitly reject unsupported Iceberg format versions
> or delete encodings, for example v3 deletion vectors, unless the selected
> Iceberg reader can guarantee correct snapshot-visible rows?
>
> I agree that delete-file handling should stay inside the Iceberg reader
> boundary. Fluss should not implement custom equality/position delete
> merging unless the Iceberg reader cannot provide correct snapshot-visible
> rows, in which case rejecting the table sounds safer for the first version.
>
>
> Best regards,
> Zhe Wang
>
>
> Yuxia Luo <yu...@apache.org> 于2026年6月17日周三 10:44写道：
>
> > Hi, Liting
> > Thanks a lot for this FIP, and apologies for the late reply.
> > A note first: the content, interfaces, and implementation of FIP-25 may
> > still change, but the overall direction should stay roughly the same.
> > Building Iceberg support on top of FIP-25 is a great starting point.
> >
> > I have a couple of small questions:
> > 1: Partition mapping. Iceberg supports transform-based partitioning, e.g.
> > day(ts). How do we plan to map this onto Fluss partitions? It'd be good
> to
> > clarify how transform partitions (and possibly partition spec evolution)
> > are handled.
> >
> > 2: Bucket configuration. For an Iceberg primary-key table, there isn't an
> > explicit bucket concept like in Paimon. But a Fluss primary-key table
> needs
> > a bucket count. Wha would the bucket count be for the resulting Fluss
> > primary-key table?
> >
> > Looking forward to your reply.
> >
> > On 2026/05/21 06:18:47 "Liting Liu (litiliu) via dev" wrote:
> > > Hi community,
> > >
> > > I drafted a design for supporting create_from_lake on existing Iceberg
> > tables:
> > >
> > >
> >
> https://docs.google.com/document/d/1REmpQ2r5W12OvW2CU9tNjDWjpkFJegiTZnE1IZbBH_U/edit?tab=t.0
> > >
> > > The proposal follows the existing Paimon create_from_lake design as
> much
> > as possible, and only introduces Iceberg-specific deltas where needed.
> > >
> > > The main difference is how the target Fluss table type is derived from
> > Iceberg metadata:
> > >
> > > * Iceberg format v1 tables are mapped to Fluss log tables.
> > > * Iceberg format v2 tables without identifier fields are mapped to
> > Fluss log tables.
> > > * Iceberg format v2 tables with non-empty identifier fields are
> > mapped to Fluss primary-key tables and reuse the existing bootstrap flow.
> > >
> > > For primary-key tables, the design reuses the same coordinator
> > lifecycle, single holdPartition model, Tiering Service bootstrap
> pipeline,
> > snapshot commit flow, partition activation, and recovery behavior from
> the
> > Paimon design. The Iceberg-specific part is mainly metadata extraction
> and
> > using the Iceberg reader to read snapshot-visible rows during bootstrap.
> > >
> > > Could you please help review the draft and share feedback, especially
> on:
> > >
> > > * Whether the table type mapping based on Iceberg format version
> and
> > identifier fields is reasonable.
> > > * Whether relying on the Iceberg reader for equality/position
> delete
> > handling during bootstrap is the right boundary.
> > > * Whether there are any Iceberg-specific cases missing from the
> > current design.
> > >
> > > Thanks
> > >
> >
>
Re: 回复: [DISCUSS] Draft design: Support upgrading existing Iceberg tables to Fluss tables
Posted by "Liting Liu (litiliu) via dev" <de...@fluss.apache.org>.
Sure, please feel free to share your insight, it’s designed to built above the FIP-25: Support In-Place Promotion of Existing Paimon Tables to Datalake-Enabled Fluss Tables.
From: Giannis Polyzos <ip...@gmail.com>
Date: Sunday, October 18, 2026 at 19:44
To: dev@fluss.apache.org <de...@fluss.apache.org>
Cc: Muhammet Orazov <mo...@morazow.com>; Liting Liu (litiliu) <li...@cisco.com>
Subject: Re: 回复: [DISCUSS] Draft design: Support upgrading existing Iceberg tables to Fluss tables
Hello guy,
Is this still active? Because it’s something we would be interested in.
Best,
Giannis
On Mon, 6 Jul 2026 at 1:12 PM, Liting Liu (litiliu) via dev <de...@fluss.apache.org>> wrote:
Hello,
Thanks for the review. I have addressed the comments in the latest draft:
https://docs.google.com/document/d/1REmpQ2r5W12OvW2CU9tNjDWjpkFJegiTZnE1IZbBH_U/edit?tab=t.0
1.
The ambiguous delete-encoding validation was removed. Format-version compatibility is checked before table creation, while delete semantics are handled by the Iceberg reader during bootstrap.
2.
Nested identifier fields are rejected during Coordinator-side validation. Only top-level identifier fields are supported.
3.
Two bucket-mapping examples were added, covering both day(ts), bucket(16, id) and day(ts) without an Iceberg bucket transform.
Thanks,
Liting
________________________________
发件人: Muhammet Orazov via dev <de...@fluss.apache.org>>
发送时间: 2026年7月1日 21:21
收件人: dev@fluss.apache.org<ma...@fluss.apache.org> <de...@fluss.apache.org>>
抄送: Muhammet Orazov <mo...@morazow.com>>
主题: Re: 回复: [DISCUSS] Draft design: Support upgrading existing Iceberg tables to Fluss tables
Hello Liting,
Thanks for the design draft!
I have couple questions/clarifications on top of the latest addressed
version.
1)
> If the selected Iceberg reader cannot guarantee correct
> snapshot-visible rows for a table that contains equality deletes,
> position deletes, deletion vectors, or any other delete encoding,
> create_from_lake rejects the table with a clear validation error.
For me this is not clear, if above check happens on validation phase or
discovered once the selected reader is running.
2)
> Iceberg tables with non-empty identifier fields are mapped to Fluss
> primary-key tables and then follow the same bootstrap pipeline as
> FIP-25.
The Iceberg identifier fields may be nested in structs, how would we map
these identifier fields? I assume it will be rejected.
3)
For the Bucket Configuration section, it would help to add some
examples.
Iceberg with combined partitioning: `day(ts), bucket(16, id)` and
identifier field `id`, how this will be transformed into Fluss PK table?
Maybe good idea to add some examples, given iceberg partitioning
combinations to the Fluss tables mapping.
Re: 回复: [DISCUSS] Draft design: Support upgrading existing Iceberg tables to Fluss tables
Posted by Giannis Polyzos <ip...@gmail.com>.
Hello guy,
Is this still active? Because it’s something we would be interested in.
Best,
Giannis
On Mon, 6 Jul 2026 at 1:12 PM, Liting Liu (litiliu) via dev <
dev@fluss.apache.org> wrote:
> Hello,
> Thanks for the review. I have addressed the comments in the latest draft:
>
> https://docs.google.com/document/d/1REmpQ2r5W12OvW2CU9tNjDWjpkFJegiTZnE1IZbBH_U/edit?tab=t.0
>
> 1.
> The ambiguous delete-encoding validation was removed. Format-version
> compatibility is checked before table creation, while delete semantics are
> handled by the Iceberg reader during bootstrap.
> 2.
> Nested identifier fields are rejected during Coordinator-side validation.
> Only top-level identifier fields are supported.
> 3.
> Two bucket-mapping examples were added, covering both day(ts), bucket(16,
> id) and day(ts) without an Iceberg bucket transform.
>
> Thanks,
> Liting
>
> ________________________________
> 发件人: Muhammet Orazov via dev <de...@fluss.apache.org>
> 发送时间: 2026年7月1日 21:21
> 收件人: dev@fluss.apache.org <de...@fluss.apache.org>
> 抄送: Muhammet Orazov <mo...@morazow.com>
> 主题: Re: 回复: [DISCUSS] Draft design: Support upgrading existing Iceberg
> tables to Fluss tables
>
> Hello Liting,
>
> Thanks for the design draft!
>
> I have couple questions/clarifications on top of the latest addressed
> version.
>
> 1)
>
> > If the selected Iceberg reader cannot guarantee correct
> > snapshot-visible rows for a table that contains equality deletes,
> > position deletes, deletion vectors, or any other delete encoding,
> > create_from_lake rejects the table with a clear validation error.
>
> For me this is not clear, if above check happens on validation phase or
> discovered once the selected reader is running.
>
> 2)
>
> > Iceberg tables with non-empty identifier fields are mapped to Fluss
> > primary-key tables and then follow the same bootstrap pipeline as
> > FIP-25.
>
> The Iceberg identifier fields may be nested in structs, how would we map
> these identifier fields? I assume it will be rejected.
>
> 3)
>
> For the Bucket Configuration section, it would help to add some
> examples.
>
> Iceberg with combined partitioning: `day(ts), bucket(16, id)` and
> identifier field `id`, how this will be transformed into Fluss PK table?
> Maybe good idea to add some examples, given iceberg partitioning
> combinations to the Fluss tables mapping.
>
>
>
>
回复: 回复: [DISCUSS] Draft design: Support upgrading existing Iceberg tables to Fluss tables
Posted by "Liting Liu (litiliu) via dev" <de...@fluss.apache.org>.
Hello,
Thanks for the review. I have addressed the comments in the latest draft:
https://docs.google.com/document/d/1REmpQ2r5W12OvW2CU9tNjDWjpkFJegiTZnE1IZbBH_U/edit?tab=t.0
1.
The ambiguous delete-encoding validation was removed. Format-version compatibility is checked before table creation, while delete semantics are handled by the Iceberg reader during bootstrap.
2.
Nested identifier fields are rejected during Coordinator-side validation. Only top-level identifier fields are supported.
3.
Two bucket-mapping examples were added, covering both day(ts), bucket(16, id) and day(ts) without an Iceberg bucket transform.
Thanks,
Liting
________________________________
发件人: Muhammet Orazov via dev <de...@fluss.apache.org>
发送时间: 2026年7月1日 21:21
收件人: dev@fluss.apache.org <de...@fluss.apache.org>
抄送: Muhammet Orazov <mo...@morazow.com>
主题: Re: 回复: [DISCUSS] Draft design: Support upgrading existing Iceberg tables to Fluss tables
Hello Liting,
Thanks for the design draft!
I have couple questions/clarifications on top of the latest addressed
version.
1)
> If the selected Iceberg reader cannot guarantee correct
> snapshot-visible rows for a table that contains equality deletes,
> position deletes, deletion vectors, or any other delete encoding,
> create_from_lake rejects the table with a clear validation error.
For me this is not clear, if above check happens on validation phase or
discovered once the selected reader is running.
2)
> Iceberg tables with non-empty identifier fields are mapped to Fluss
> primary-key tables and then follow the same bootstrap pipeline as
> FIP-25.
The Iceberg identifier fields may be nested in structs, how would we map
these identifier fields? I assume it will be rejected.
3)
For the Bucket Configuration section, it would help to add some
examples.
Iceberg with combined partitioning: `day(ts), bucket(16, id)` and
identifier field `id`, how this will be transformed into Fluss PK table?
Maybe good idea to add some examples, given iceberg partitioning
combinations to the Fluss tables mapping.
Re: 回复: [DISCUSS] Draft design: Support upgrading existing Iceberg tables to Fluss tables
Posted by Muhammet Orazov via dev <de...@fluss.apache.org>.
Hello Liting,
Thanks for the design draft!
I have couple questions/clarifications on top of the latest addressed
version.
1)
> If the selected Iceberg reader cannot guarantee correct
> snapshot-visible rows for a table that contains equality deletes,
> position deletes, deletion vectors, or any other delete encoding,
> create_from_lake rejects the table with a clear validation error.
For me this is not clear, if above check happens on validation phase or
discovered once the selected reader is running.
2)
> Iceberg tables with non-empty identifier fields are mapped to Fluss
> primary-key tables and then follow the same bootstrap pipeline as
> FIP-25.
The Iceberg identifier fields may be nested in structs, how would we map
these identifier fields? I assume it will be rejected.
3)
For the Bucket Configuration section, it would help to add some
examples.
Iceberg with combined partitioning: `day(ts), bucket(16, id)` and
identifier field `id`, how this will be transformed into Fluss PK table?
Maybe good idea to add some examples, given iceberg partitioning
combinations to the Fluss tables mapping.