You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@fluss.apache.org by yuxia <lu...@alumni.sjtu.edu.cn> on 2026/07/06 09:12:09 UTC

[DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary Key Table Union Read over Paimon

Hi all,

I would like to start a discussion for FIP-47: Introduce Deletion Vectors to accelerate Primary Key Table Union Read over Paimon.

The main problem this FIP tries to solve is the read cost of primary key table union read over Paimon. Today, union read needs to deduplicate data across the Fluss hot layer and the Paimon historical layer. For primary key tables, this currently requires a full sort-merge on the read path, so the cost grows with the table size. FIP-47 proposes introducing deletion vectors to record which rows have been updated or deleted, so union read can skip stale rows through lightweight bitmap filtering while preserving exactly-once semantics.

FIP link:
https://cwiki.apache.org/confluence/spaces/FLUSS/pages/438009916/FIP-47+Introduce+Deletion+Vectors+to+accelerate+Primary+Key+Table+Union+Read+over+Paimon

I also have a PoC branch here:
https://github.com/luoyuxia/fluss/tree/paimon-dv-support

I ran a simple benchmark to evaluate Flink batch union read performance over a Paimon table. The benchmark used a single bucket with 10 million rows.

The result is:

- Without deletion vector: 13.44s
- With deletion vector: 1.39s

This is about a 10x improvement in this benchmark.

For people who are more familiar with Iceberg than Paimon, you can also refer to this design document:
https://docs.google.com/document/d/10phRucA3LJA5z_PIeTTpR7I_o0CgZkKuffA8Ot-XBeU/edit?tab=t.0#heading=h.ylmcf7juvl7v

That document discusses how deletion vectors can be integrated with Iceberg. The overall architecture and core idea are shared with the Paimon integration, without major conceptual differences. So if you are not very familiar with Paimon but know Iceberg well, the Iceberg design may help explain the motivation and mechanism.

Looking forward to your feedback.

Best regards,
Yuxia

Re: [DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary Key Table Union Read over Paimon

Posted by "Liting Liu (litiliu) via dev" <de...@fluss.apache.org>.

Yuxia, Thanks for the clarification. Our use case is Amoro’s partition-level data expiration (data-expire.level=partition) on an Iceberg table partitioned by day.

When all data files in a daily partition are considered expired, Amoro commits a new Iceberg snapshot that logically removes those files without replacement files. This operation does not generate Fluss -D changelog records.

The removed data files are not deleted immediately because they may still be referenced by retained historical snapshots. They are physically deleted later when those snapshots expire,  provided that no retained branch or tag still references them..

From: yuxia <lu...@alumni.sjtu.edu.cn>
Date: Wednesday, July 22, 2026 at 18:02
To: dev <de...@fluss.apache.org>
Subject: Re: [DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary Key Table Union Read over Paimon

Hi, Liting.
Thanks for raising these concerns. I would like to clarify the exact expiration scenario in the first point.

By “expire historical data,” do you mean ordinary Iceberg snapshot expiration, or an operation that removes live data from the current table snapshot, such as partition/data expiration?

For ordinary snapshot expiration, we do not expect orphan RowPosIndex entries. Snapshot expiration only removes historical snapshots and files that are no longer referenced by any retained snapshot. Files referenced by the current DV-readable snapshot must be retained. If files are rewritten by compaction, the surviving RowIds are scanned from the new files and their RowPosIndex entries are overwritten with the new positions.

If Amoro instead commits a new snapshot that logically removes data files without replacement―for example, by expiring a partition or applying a data-retention policy―then your concern is valid. Those RowIds would not appear in any new data file, and no new SST mapping would overwrite their old RowPosIndex entries. Could you clarify whether this is partition-level expiration, file-level data expiration, or another Amoro operation? It would help us define the appropriate support boundary.

Regarding FIP-27, its goal is to remove mandatory system columns from regular Fluss lake tables so that all tables do not incur schema pollution and downstream compatibility impact. DV-enabled tables are an explicit, opt-in case: the DV mechanism requires internal row identity and bucket information to rebuild the mapping from RowId to physical file position. Therefore, while normal lake tables should no longer require these additional columns, a table with DV explicitly enabled would still carry the internal columns required by the DV protocol.

Best regards,
Yuxia

----- 原始邮件 -----
发件人: "Leonard Xu" <xb...@gmail.com>
收件人: "dev" <de...@fluss.apache.org>
发送时间: 星期三, 2026年 7 月 08日 下午 6:02:29
主题: Re: [DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary Key Table Union Read over Paimon

Hi Yuxia.

Thanks for sharing the FIP.  The overall direction looks very promising to me. Using RowId + RowPosIndex + LakeDv/LogDv to avoid full sort-merge during primary-key union read makes sense, and the two-phase prepare/switch protocol also looks like the right direction to reduce the stale-snapshot window.

I have several comments that I think are worth clarifying in the FIP before moving forward:

RowPosIndex cleanup when files are removed externally

The design explains how LakeDv entries are cleaned when compaction replaces files. However, I think we also need to clarify what happens when files are removed by external operations, such as snapshot expiration, partition expiration, rewrite, or data lifecycle management.

RowPosIndex is keyed by RowId and points to a physical file position. If the underlying Paimon data file is removed outside the normal compaction path, the RowPosIndex entry may become an orphan entry. Later updates/deletes may still hit this stale RowPosIndex entry and generate LakeDv markers for a file that no longer exists.

Could the FIP clarify whether such external file-removal operations are forbidden for Fluss-managed Paimon tables, or whether we need an additional reverse mapping from fileId to RowIds so RowPosIndex can be cleaned by removed file?

Compatibility with FIP-27 and system column pollution

The design introduces a new __rowid column in Paimon files. Since FIP-27 is trying to remove mandatory system columns from Fluss lake tables, I think the FIP should clarify how __rowid fits into that direction.

In particular:

Is __rowid a hidden/internal column or a normal physical column?
Will external engines see it in SELECT *?
What happens if the user already has a column named __rowid?
How does schema evolution handle this internal column?
Without a clear answer, this may introduce schema pollution or compatibility issues for downstream engines.

Merge engine compatibility should be narrowed or clarified

The FIP says all merge engines are compatible under FULL changelog mode. I think this may be too optimistic.

For example, FIRST_ROW has special semantics and does not really support normal UPDATE/DELETE behavior in Fluss today. But the DV design relies heavily on -U / -D carrying the old RowId, so the old version can be located and masked. For merge engines with special update/delete semantics, it is not obvious that this assumption always holds.

I suggest narrowing the first version to the default DEDUPLICATE primary-key table, or adding a detailed correctness explanation and test plan for each supported merge engine.

RowPos SST file lifecycle and garbage collection

The FIP defines remote RowPos SST files under rowPos/{snapshotId}/..., which are required for TabletServer prepare and recovery. However, the retention and cleanup policy is not very clear yet.

If these files are deleted too early, TabletServer recovery may fail because it cannot re-ingest RowPos SSTs. If they are never deleted, remote storage may grow indefinitely.

Could the FIP define the retention rule for RowPos SSTs, and explain how it is coordinated with:

DvRocksDB checkpoints,
current/latest DV-readable snapshot,
pending readable rounds,
Paimon snapshot expiration?
LakeDv returned to union read should be filtered by planned files

During union read, the Paimon scan already knows which data files will be read. The FIP mentions that LakeDv entries may be filtered by planned file paths, but I think this should be a required behavior instead of an optional optimization.

Otherwise, for delete-heavy tables, returning the whole bucket’s LakeDv may introduce large network transfer, bitmap deserialization, and memory overhead. The read cost may simply move from sort-merge to DV transfer.

I suggest making GetLakeDvSnapshot return only LakeDv entries for the files planned by the current Paimon scan.

Overall, I think the FIP is heading in a good direction and the expected performance improvement is very valuable. The main areas I would like to see clarified are around lifecycle management, compatibility, and keeping the read path lightweight.

Best,
Leonard

Re: [DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary Key Table Union Read over Paimon

Posted by Muhammet Orazov via dev <de...@fluss.apache.org>.

Hello Yuxia,

I sent an email to your FIP-47 discussion in mailing list. But I it is 
not delivered since.

In any case, please find my findings below:

---
Hello Yuxia,

Thanks for the FIP! Please find my findings below.

1) The __rowid Introduction

Does __rowid actually differ from the existing __offset system column?

Every Fluss-tiered Paimon table already gets __bucket, __offset, and 
__timestamp appended, PK tables included. The __offset is filled per row 
from `logRecord.logOffset()`. So after DEDUPLICATE merge, the surviving 
row's __offset is the offset of the +I/+U that produced that version. 
That's similar to the FIP's definition of __rowid.

Can we reuse __offset instead? If not, the FIP should say why, because 
right now it looks like we're paying 8 bytes per row on every DV table 
for a column we already have.

2) New Format Change

I think the new format change is visible to clients than `@Internal` 
suggestion.

KV values don't stay on the server. SnapshotFilesReader in fluss-client 
reads raw KV snapshot files and decodes `[schemaId][BinaryRow]` 
directly, so prepending 8 bytes changes a format that ships to clients.

The compatibility section says `opt-in, no migration`, maybe we can 
update it with what an older client does when it reads a DV-enabled 
table. Does it fail on a version check, or does it silently misparse the 
row as if the RowId bytes were column data? We can extend the the FIP to 
answer this with a concrete protocol or version gate, not just the 
opt-in flag.

3) Validating Correctness Invariants

I think this was one of the points from Leonard's comments.

 From the FIP, I understood that we have two assumptions: Fluss stays the 
only writer, and compaction preserves __rowid.

Nothing enforces either one, because a Paimon table is openly writable 
and Spark, Trino, or Flink can write to it at any time. A single 
external INSERT puts rows with null __rowid into the table; an external 
rewrite that drops the column loses the mapping entirely. Either way the 
index is corrupted and nothing in the FIP would notice.

I don't think documenting these would be enough. Maybe we can extend the 
FIP to specify a detection mechanism. One option: during the file scan, 
check for null __rowid values, and if any turn up, halt and alert.



Best,
Muhammet

Re: [DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary Key Table Union Read over Paimon

Posted by Mehul Batra <me...@gmail.com>.

Hi Yuxia,

Thanks for the detailed FIP and for all your hard work to push this mammoth
effort. I have a few additional questions after reading through the design.

*1. Should the validation also check the lake format?*

The creation-time validation checks primary key tables,
table.datalake.enabled, and FULL changelog mode, but I did not see a
validation for the lake format itself.

The design seems to assume Paimon throughout, for example COMPACT
snapshots, DELETE records, native deletion vectors, and
paimon.deletion-vectors.enabled.

Would it make sense to also reject tables whose lake format is not Paimon
when table.datalake.deletion-vectors.enabled = true? That way, if support
for Iceberg or other lake formats is added later, they can simply be added
to the allowlist instead of leaving the current behavior undefined.

*2. How is the __rowid preservation contract enforced?*

The FIP says external engines may compact or rewrite files as long as they
preserve __rowid, but this currently reads as a convention rather than
something that is validated.

For example:

   -

   What happens if an external engine inserts rows without a valid __rowid?
   -

   What happens if a rewrite accidentally drops or corrupts __rowid values?

Would it be worth adding a validation during the Phase A scan that fails
the round if any data file contains null or invalid __rowid values? That
would at least turn silent corruption into a visible failure.

*3. What happens if compaction never runs?*

The readable baseline only advances on COMPACT snapshots. If compaction is
disabled, misconfigured, or simply delayed for a long time, it seems that
PendingDeletes would continue to grow, LogDv could not be cleaned up, and
changelog retention would also be blocked.

Should the FIP define any operational guidance here, such as expected
compaction frequency, metrics to watch, or backpressure when the system
falls too far behind?

*4. Is there an escape hatch?*

If DV state is ever found to be corrupted, or a production issue is
discovered after deployment, is there a way to temporarily fall back to the
existing sort-merge based union read path?

Having a repair or safety mode could be valuable while the issue is
investigated, even if it comes with lower performance.

Overall, I think the direction looks very promising. Thanks again for
putting together the proposal.

Best,
Mehul Batra

On Wed, Jul 22, 2026 at 4:59 PM yuxia <lu...@alumni.sjtu.edu.cn> wrote:

> Please disregard my previous email. It was sent by mistake. Sorry for the
> confusion.
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "yuxia" <lu...@alumni.sjtu.edu.cn>
> 收件人: "dev" <de...@fluss.apache.org>
> 发送时间: 星期三, 2026年 7 月 22日 下午 6:08:26
> 主题: Re: [DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary
> Key Table Union Read over Paimon
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "yuxia" <lu...@alumni.sjtu.edu.cn>
> 收件人: "dev" <de...@fluss.apache.org>
> 发送时间: 星期三, 2026年 7 月 22日 下午 5:58:30
> 主题: Re: [DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary
> Key Table Union Read over Paimon
>
> Hi, Liting.
> Thanks for raising these concerns. I would like to clarify the exact
> expiration scenario in the first point.
>
> By “expire historical data,” do you mean ordinary Iceberg snapshot
> expiration, or an operation that removes live data from the current table
> snapshot, such as partition/data expiration?
>
> For ordinary snapshot expiration, we do not expect orphan RowPosIndex
> entries. Snapshot expiration only removes historical snapshots and files
> that are no longer referenced by any retained snapshot. Files referenced by
> the current DV-readable snapshot must be retained. If files are rewritten
> by compaction, the surviving RowIds are scanned from the new files and
> their RowPosIndex entries are overwritten with the new positions.
>
> If Amoro instead commits a new snapshot that logically removes data files
> without replacement—for example, by expiring a partition or applying a
> data-retention policy—then your concern is valid. Those RowIds would not
> appear in any new data file, and no new SST mapping would overwrite their
> old RowPosIndex entries. Could you clarify whether this is partition-level
> expiration, file-level data expiration, or another Amoro operation? It
> would help us define the appropriate support boundary.
>
> Regarding FIP-27, its goal is to remove mandatory system columns from
> regular Fluss lake tables so that all tables do not incur schema pollution
> and downstream compatibility impact. DV-enabled tables are an explicit,
> opt-in case: the DV mechanism requires internal row identity and bucket
> information to rebuild the mapping from RowId to physical file position.
> Therefore, while normal lake tables should no longer require these
> additional columns, a table with DV explicitly enabled would still carry
> the internal columns required by the DV protocol.
>
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "Leonard Xu" <xb...@gmail.com>
> 收件人: "dev" <de...@fluss.apache.org>
> 发送时间: 星期三, 2026年 7 月 08日 下午 6:02:29
> 主题: Re: [DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary
> Key Table Union Read over Paimon
>
> Hi Yuxia.
>
> Thanks for sharing the FIP.  The overall direction looks very promising to
> me. Using RowId + RowPosIndex + LakeDv/LogDv to avoid full sort-merge
> during primary-key union read makes sense, and the two-phase prepare/switch
> protocol also looks like the right direction to reduce the stale-snapshot
> window.
>
> I have several comments that I think are worth clarifying in the FIP
> before moving forward:
>
> RowPosIndex cleanup when files are removed externally
>
> The design explains how LakeDv entries are cleaned when compaction
> replaces files. However, I think we also need to clarify what happens when
> files are removed by external operations, such as snapshot expiration,
> partition expiration, rewrite, or data lifecycle management.
>
> RowPosIndex is keyed by RowId and points to a physical file position. If
> the underlying Paimon data file is removed outside the normal compaction
> path, the RowPosIndex entry may become an orphan entry. Later
> updates/deletes may still hit this stale RowPosIndex entry and generate
> LakeDv markers for a file that no longer exists.
>
> Could the FIP clarify whether such external file-removal operations are
> forbidden for Fluss-managed Paimon tables, or whether we need an additional
> reverse mapping from fileId to RowIds so RowPosIndex can be cleaned by
> removed file?
>
> Compatibility with FIP-27 and system column pollution
>
> The design introduces a new __rowid column in Paimon files. Since FIP-27
> is trying to remove mandatory system columns from Fluss lake tables, I
> think the FIP should clarify how __rowid fits into that direction.
>
> In particular:
>
> Is __rowid a hidden/internal column or a normal physical column?
> Will external engines see it in SELECT *?
> What happens if the user already has a column named __rowid?
> How does schema evolution handle this internal column?
> Without a clear answer, this may introduce schema pollution or
> compatibility issues for downstream engines.
>
> Merge engine compatibility should be narrowed or clarified
>
> The FIP says all merge engines are compatible under FULL changelog mode. I
> think this may be too optimistic.
>
> For example, FIRST_ROW has special semantics and does not really support
> normal UPDATE/DELETE behavior in Fluss today. But the DV design relies
> heavily on -U / -D carrying the old RowId, so the old version can be
> located and masked. For merge engines with special update/delete semantics,
> it is not obvious that this assumption always holds.
>
> I suggest narrowing the first version to the default DEDUPLICATE
> primary-key table, or adding a detailed correctness explanation and test
> plan for each supported merge engine.
>
> RowPos SST file lifecycle and garbage collection
>
> The FIP defines remote RowPos SST files under rowPos/{snapshotId}/...,
> which are required for TabletServer prepare and recovery. However, the
> retention and cleanup policy is not very clear yet.
>
> If these files are deleted too early, TabletServer recovery may fail
> because it cannot re-ingest RowPos SSTs. If they are never deleted, remote
> storage may grow indefinitely.
>
> Could the FIP define the retention rule for RowPos SSTs, and explain how
> it is coordinated with:
>
> DvRocksDB checkpoints,
> current/latest DV-readable snapshot,
> pending readable rounds,
> Paimon snapshot expiration?
> LakeDv returned to union read should be filtered by planned files
>
> During union read, the Paimon scan already knows which data files will be
> read. The FIP mentions that LakeDv entries may be filtered by planned file
> paths, but I think this should be a required behavior instead of an
> optional optimization.
>
> Otherwise, for delete-heavy tables, returning the whole bucket’s LakeDv
> may introduce large network transfer, bitmap deserialization, and memory
> overhead. The read cost may simply move from sort-merge to DV transfer.
>
> I suggest making GetLakeDvSnapshot return only LakeDv entries for the
> files planned by the current Paimon scan.
>
> Overall, I think the FIP is heading in a good direction and the expected
> performance improvement is very valuable. The main areas I would like to
> see clarified are around lifecycle management, compatibility, and keeping
> the read path lightweight.
>
> Best,
> Leonard
>

Re: [DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary Key Table Union Read over Paimon

Posted by yuxia <lu...@alumni.sjtu.edu.cn>.

Please disregard my previous email. It was sent by mistake. Sorry for the confusion.

Best regards,
Yuxia

----- 原始邮件 -----
发件人: "yuxia" <lu...@alumni.sjtu.edu.cn>
收件人: "dev" <de...@fluss.apache.org>
发送时间: 星期三, 2026年 7 月 22日 下午 6:08:26
主题: Re: [DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary Key Table Union Read over Paimon

Best regards,
Yuxia

----- 原始邮件 -----
发件人: "yuxia" <lu...@alumni.sjtu.edu.cn>
收件人: "dev" <de...@fluss.apache.org>
发送时间: 星期三, 2026年 7 月 22日 下午 5:58:30
主题: Re: [DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary Key Table Union Read over Paimon

Hi, Liting.
Thanks for raising these concerns. I would like to clarify the exact expiration scenario in the first point.

By “expire historical data,” do you mean ordinary Iceberg snapshot expiration, or an operation that removes live data from the current table snapshot, such as partition/data expiration?

For ordinary snapshot expiration, we do not expect orphan RowPosIndex entries. Snapshot expiration only removes historical snapshots and files that are no longer referenced by any retained snapshot. Files referenced by the current DV-readable snapshot must be retained. If files are rewritten by compaction, the surviving RowIds are scanned from the new files and their RowPosIndex entries are overwritten with the new positions.

If Amoro instead commits a new snapshot that logically removes data files without replacement—for example, by expiring a partition or applying a data-retention policy—then your concern is valid. Those RowIds would not appear in any new data file, and no new SST mapping would overwrite their old RowPosIndex entries. Could you clarify whether this is partition-level expiration, file-level data expiration, or another Amoro operation? It would help us define the appropriate support boundary.

Regarding FIP-27, its goal is to remove mandatory system columns from regular Fluss lake tables so that all tables do not incur schema pollution and downstream compatibility impact. DV-enabled tables are an explicit, opt-in case: the DV mechanism requires internal row identity and bucket information to rebuild the mapping from RowId to physical file position. Therefore, while normal lake tables should no longer require these additional columns, a table with DV explicitly enabled would still carry the internal columns required by the DV protocol.

Best regards,
Yuxia

----- 原始邮件 -----
发件人: "Leonard Xu" <xb...@gmail.com>
收件人: "dev" <de...@fluss.apache.org>
发送时间: 星期三, 2026年 7 月 08日 下午 6:02:29
主题: Re: [DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary Key Table Union Read over Paimon

Hi Yuxia.

Thanks for sharing the FIP. The overall direction looks very promising to me. Using RowId + RowPosIndex + LakeDv/LogDv to avoid full sort-merge during primary-key union read makes sense, and the two-phase prepare/switch protocol also looks like the right direction to reduce the stale-snapshot window.

I have several comments that I think are worth clarifying in the FIP before moving forward:

RowPosIndex cleanup when files are removed externally

The design explains how LakeDv entries are cleaned when compaction replaces files. However, I think we also need to clarify what happens when files are removed by external operations, such as snapshot expiration, partition expiration, rewrite, or data lifecycle management.

RowPosIndex is keyed by RowId and points to a physical file position. If the underlying Paimon data file is removed outside the normal compaction path, the RowPosIndex entry may become an orphan entry. Later updates/deletes may still hit this stale RowPosIndex entry and generate LakeDv markers for a file that no longer exists.

Could the FIP clarify whether such external file-removal operations are forbidden for Fluss-managed Paimon tables, or whether we need an additional reverse mapping from fileId to RowIds so RowPosIndex can be cleaned by removed file?

Compatibility with FIP-27 and system column pollution

The design introduces a new __rowid column in Paimon files. Since FIP-27 is trying to remove mandatory system columns from Fluss lake tables, I think the FIP should clarify how __rowid fits into that direction.

In particular:

Is __rowid a hidden/internal column or a normal physical column?
Will external engines see it in SELECT *?
What happens if the user already has a column named __rowid?
How does schema evolution handle this internal column?
Without a clear answer, this may introduce schema pollution or compatibility issues for downstream engines.

Merge engine compatibility should be narrowed or clarified

The FIP says all merge engines are compatible under FULL changelog mode. I think this may be too optimistic.

For example, FIRST_ROW has special semantics and does not really support normal UPDATE/DELETE behavior in Fluss today. But the DV design relies heavily on -U / -D carrying the old RowId, so the old version can be located and masked. For merge engines with special update/delete semantics, it is not obvious that this assumption always holds.

I suggest narrowing the first version to the default DEDUPLICATE primary-key table, or adding a detailed correctness explanation and test plan for each supported merge engine.

RowPos SST file lifecycle and garbage collection

The FIP defines remote RowPos SST files under rowPos/{snapshotId}/..., which are required for TabletServer prepare and recovery. However, the retention and cleanup policy is not very clear yet.

If these files are deleted too early, TabletServer recovery may fail because it cannot re-ingest RowPos SSTs. If they are never deleted, remote storage may grow indefinitely.

Could the FIP define the retention rule for RowPos SSTs, and explain how it is coordinated with:

DvRocksDB checkpoints,
current/latest DV-readable snapshot,
pending readable rounds,
Paimon snapshot expiration?
LakeDv returned to union read should be filtered by planned files

During union read, the Paimon scan already knows which data files will be read. The FIP mentions that LakeDv entries may be filtered by planned file paths, but I think this should be a required behavior instead of an optional optimization.

Otherwise, for delete-heavy tables, returning the whole bucket’s LakeDv may introduce large network transfer, bitmap deserialization, and memory overhead. The read cost may simply move from sort-merge to DV transfer.

I suggest making GetLakeDvSnapshot return only LakeDv entries for the files planned by the current Paimon scan.

Overall, I think the FIP is heading in a good direction and the expected performance improvement is very valuable. The main areas I would like to see clarified are around lifecycle management, compatibility, and keeping the read path lightweight.

Best,
Leonard

Re: [DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary Key Table Union Read over Paimon

Posted by yuxia <lu...@alumni.sjtu.edu.cn>.

Best regards,
Yuxia

Hi, Liting.
Thanks for raising these concerns. I would like to clarify the exact expiration scenario in the first point.

By “expire historical data,” do you mean ordinary Iceberg snapshot expiration, or an operation that removes live data from the current table snapshot, such as partition/data expiration?

Best regards,
Yuxia

Hi Yuxia.

I have several comments that I think are worth clarifying in the FIP before moving forward:

RowPosIndex cleanup when files are removed externally

Compatibility with FIP-27 and system column pollution

In particular:

Merge engine compatibility should be narrowed or clarified

The FIP says all merge engines are compatible under FULL changelog mode. I think this may be too optimistic.

I suggest narrowing the first version to the default DEDUPLICATE primary-key table, or adding a detailed correctness explanation and test plan for each supported merge engine.

RowPos SST file lifecycle and garbage collection

The FIP defines remote RowPos SST files under rowPos/{snapshotId}/..., which are required for TabletServer prepare and recovery. However, the retention and cleanup policy is not very clear yet.

If these files are deleted too early, TabletServer recovery may fail because it cannot re-ingest RowPos SSTs. If they are never deleted, remote storage may grow indefinitely.

Could the FIP define the retention rule for RowPos SSTs, and explain how it is coordinated with:

DvRocksDB checkpoints,
current/latest DV-readable snapshot,
pending readable rounds,
Paimon snapshot expiration?
LakeDv returned to union read should be filtered by planned files

I suggest making GetLakeDvSnapshot return only LakeDv entries for the files planned by the current Paimon scan.

Best,
Leonard

Re: [DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary Key Table Union Read over Paimon

Posted by yuxia <lu...@alumni.sjtu.edu.cn>.

Hi, Liting.
Thanks for raising these concerns. I would like to clarify the exact expiration scenario in the first point.

By “expire historical data,” do you mean ordinary Iceberg snapshot expiration, or an operation that removes live data from the current table snapshot, such as partition/data expiration?

Best regards,
Yuxia

Hi Yuxia.

I have several comments that I think are worth clarifying in the FIP before moving forward:

RowPosIndex cleanup when files are removed externally

Compatibility with FIP-27 and system column pollution

In particular:

Merge engine compatibility should be narrowed or clarified

The FIP says all merge engines are compatible under FULL changelog mode. I think this may be too optimistic.

I suggest narrowing the first version to the default DEDUPLICATE primary-key table, or adding a detailed correctness explanation and test plan for each supported merge engine.

RowPos SST file lifecycle and garbage collection

The FIP defines remote RowPos SST files under rowPos/{snapshotId}/..., which are required for TabletServer prepare and recovery. However, the retention and cleanup policy is not very clear yet.

If these files are deleted too early, TabletServer recovery may fail because it cannot re-ingest RowPos SSTs. If they are never deleted, remote storage may grow indefinitely.

Could the FIP define the retention rule for RowPos SSTs, and explain how it is coordinated with:

DvRocksDB checkpoints,
current/latest DV-readable snapshot,
pending readable rounds,
Paimon snapshot expiration?
LakeDv returned to union read should be filtered by planned files

I suggest making GetLakeDvSnapshot return only LakeDv entries for the files planned by the current Paimon scan.

Best,
Leonard

Re: [DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary Key Table Union Read over Paimon

Posted by Leonard Xu <xb...@gmail.com>.

Hi Yuxia.

I have several comments that I think are worth clarifying in the FIP before moving forward:

RowPosIndex cleanup when files are removed externally

Compatibility with FIP-27 and system column pollution

In particular:

Merge engine compatibility should be narrowed or clarified

The FIP says all merge engines are compatible under FULL changelog mode. I think this may be too optimistic.

I suggest narrowing the first version to the default DEDUPLICATE primary-key table, or adding a detailed correctness explanation and test plan for each supported merge engine.

RowPos SST file lifecycle and garbage collection

The FIP defines remote RowPos SST files under rowPos/{snapshotId}/..., which are required for TabletServer prepare and recovery. However, the retention and cleanup policy is not very clear yet.

If these files are deleted too early, TabletServer recovery may fail because it cannot re-ingest RowPos SSTs. If they are never deleted, remote storage may grow indefinitely.

Could the FIP define the retention rule for RowPos SSTs, and explain how it is coordinated with:

DvRocksDB checkpoints,
current/latest DV-readable snapshot,
pending readable rounds,
Paimon snapshot expiration?
LakeDv returned to union read should be filtered by planned files

I suggest making GetLakeDvSnapshot return only LakeDv entries for the files planned by the current Paimon scan.

Best,
Leonard

答复: FIP-47: Introduce Deletion Vectors to accelerate Primary Key Table Union Read over Paimon

Posted by "Liting Liu (litiliu) via dev" <de...@fluss.apache.org>.

Thanks for sharing the design. I have two concerns:

1. Possible Orphan entries in RowPosIndex

We use Amoro to periodically expire historical data by deleting Iceberg data files and committing a new snapshot. These deletions do not generate Fluss -D changelog records. During the next tiering commit, Fluss can detect previously unprocessed external Iceberg snapshots and derive the removed data files from their snapshot changes. However, RowPosIndex is keyed by RowId and cannot be cleaned using only the removed fileId. Since data expiration does not produce replacement files, there is also no new SST mapping to overwrite the old entries. As a result, orphan RowPosIndex entries may accumulate indefinitely.

2. FIP-27<https://cwiki.apache.org/confluence/spaces/FLUSS/pages/406619416/FIP-27+Remove+Mandatory+System+Columns+From+Fluss+Lake+Tables> proposes removing mandatory system columns to avoid schema pollution and downstream impact. However, this DV design requires __rowid and __bucket in Iceberg data files.

Could the FIP clarify how schema compatibility with FIP-27 will be handled?

Best regards,
Liting

发件人: yuxia <lu...@alumni.sjtu.edu.cn>
日期: 星期一, 2026年7月6日 17:14
收件人: dev <de...@fluss.apache.org>
主题: [DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary Key Table Union Read over Paimon

Hi all,

I would like to start a discussion for FIP-47: Introduce Deletion Vectors to accelerate Primary Key Table Union Read over Paimon.

FIP link:
https://cwiki.apache.org/confluence/spaces/FLUSS/pages/438009916/FIP-47+Introduce+Deletion+Vectors+to+accelerate+Primary+Key+Table+Union+Read+over+Paimon

I also have a PoC branch here:
https://github.com/luoyuxia/fluss/tree/paimon-dv-support

I ran a simple benchmark to evaluate Flink batch union read performance over a Paimon table. The benchmark used a single bucket with 10 million rows.

The result is:

- Without deletion vector: 13.44s
- With deletion vector: 1.39s

This is about a 10x improvement in this benchmark.

Looking forward to your feedback.

Best regards,
Yuxia