You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "luoyuxia (via GitHub)" <gi...@apache.org> on 2026/10/20 03:38:07 UTC
[I] [Bug] [lake/paimon] Union reads can fail after snapshot expiration [fluss]
luoyuxia opened a new issue, #4750:
URL: https://github.com/apache/fluss/issues/4750
### Search before asking
- [x] I searched the issues and found no issue specifically tracking retention protection for Paimon files still needed by an active union read.
Related: #3933 addressed checkpointing lake-phase completion after the lake data was already consumed. This issue concerns files that are still needed by an unfinished lake read or its recovery.
### Fluss version
main (development)
The upstream read path was inspected at commit `6be68ddde48a5798b8b2891dd909415f31407652`. The local planner/reader check described below used Paimon 2.1.0.
### Please describe the bug 🐞
Fluss union reads can fail with file-not-found errors when Paimon snapshot expiration deletes files referenced by lake splits that have already been planned but have not finished reading. This can affect batch union reads and the initial lake-reading phase of streaming union reads, including recovery from a checkpoint that still needs those splits.
The current path selects a readable lake snapshot and its corresponding Fluss offsets, then uses a Paimon batch scan to plan the lake splits:
- [LakeSplitGenerator selects the snapshot and plans the lake splits](https://github.com/apache/fluss/blob/6be68ddde48a5798b8b2891dd909415f31407652/fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/lake/LakeSplitGenerator.java#L85-L108).
- [PaimonSplitPlanner uses `newScan()` with `scan.snapshot-id`](https://github.com/apache/fluss/blob/6be68ddde48a5798b8b2891dd909415f31407652/fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/source/PaimonSplitPlanner.java#L68-L110).
This path does not itself register and maintain a Paimon consumer to protect the selected snapshot throughout reading and recovery. Merely setting `consumer-id` does not make the Paimon batch scan maintain such a consumer.
#### Triggering sequence
1. A lake-enabled Fluss table has a readable Paimon snapshot `S`.
2. A union read selects `S` and plans splits referencing its files. Delay an unread split, for example through slow reading or scheduling, or checkpoint while the lake phase is incomplete.
3. Tiering continues. Updates and compaction make some files needed by `S` obsolete in newer snapshots.
4. Paimon expires `S` under its normal retention policy and removes the obsolete files. This is easier to trigger with short snapshot retention or a long-running lake scan.
5. The reader opens an unread split, or restores an unfinished split from the checkpoint. Opening the referenced file fails because it has been deleted.
Compaction makes files obsolete; the subsequent expiration cleanup is what removes files still referenced by the earlier read plan.
#### Local validation
A small local check exercised the Fluss `PaimonLakeSource` planner and reader directly, rather than a complete Flink union-read job:
1. Create a Paimon table with snapshot minimum/maximum retention set to `1` and time retention set to `0 ms`.
2. Write a row and plan Fluss lake splits for snapshot `1`.
3. Overwrite the table with a different row and run snapshot expiration.
Without protection, the earliest remaining snapshot became `2` and the old split's data file no longer existed. Registering a Paimon consumer with `nextSnapshot = 1` before planning retained snapshot `1` and its file, and the same Fluss reader successfully read the original row.
**Expected behavior:** Normal Paimon snapshot expiration must respect files still required by the union read while its read protection is valid, including unfinished splits needed for recovery.
**Actual behavior:** Planning a lake split does not establish Fluss-managed retention protection, so its referenced files can be removed before they are read.
### Solution
Add lifecycle-managed retention protection for Paimon union reads. One possible implementation is a Fluss-managed Paimon consumer:
- Register an independent consumer for each active query/table/branch before planning and assigning splits for snapshot `S`.
- Hold its `nextSnapshot` at `S` while any lake splits still require that snapshot. Paimon expiration uses the consumer position as an exclusive upper bound: setting it to `S` protects `S` and later snapshots. Setting it to `S + 1` would allow `S` to expire. Snapshot protection also requires `consumer.changelog-only = false`.
- Refresh the consumer during long reads, with a configurable expiration time for abandoned jobs. Preserve or re-establish protection safely during recovery.
- Release protection only when the lake files are no longer needed by the running job or its supported recovery state. Finishing one reader or closing a failed source is insufficient.
- Handle snapshot-selection/registration races without mixing a lake snapshot with Fluss offsets from another snapshot. Fail clearly if protection cannot be established.
Add regression coverage for expiration during a delayed lake read, recovery of unfinished lake splits, independent concurrent readers, and cleanup after completion or abandonment.
Paimon's temporary read-protection tags are another possible mechanism. However, a fixed tag TTL must cover the full read/recovery duration, and the native best-effort tag-creation path can continue reading after tag creation fails. Increasing snapshot retention is also a workaround, but does not tie protection to the actual reader lifecycle.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [lake/paimon] Union reads can fail after snapshot expiration [fluss]
Posted by "Jackeyzhe (via GitHub)" <gi...@apache.org>.
Jackeyzhe commented on issue #4750:
URL: https://github.com/apache/fluss/issues/4750#issuecomment-6398279743
From code inspection, Paimon's DataSplit already carries a snapshot ID, and Fluss preserves it
when serializing lake splits. I haven't reproduced the failure yet. Would protecting the selected
snapshot for the lifetime of unfinished lake splits, including recovery, be the direction you had in mind?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org