You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "beryllw (via GitHub)" <gi...@apache.org> on 2026/10/15 10:37:38 UTC
[I] [Bug] Historical Paimon tiering repeatedly scans manifests [fluss]
beryllw opened a new issue, #4656:
URL: https://github.com/apache/fluss/issues/4656
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Fluss version
main (development)
### Please describe the bug π
A Fluss `__historical__` bucket can contain records belonging to many original date partitions. During Paimon tiering, `MergeTreeWriter` restores the target partition from each record. The first write to each `(partition, bucket)` initializes a separate Paimon writer and triggers another manifest scan.
Consequently, a small historical split can spend most of its time repeatedly reading metadata. In one observed split, 1,572 records triggered 64 restore scans, taking approximately 57 seconds out of 70 seconds of total split processing time.
Locally reproduced with Paimon 2.0.0 and Java 11.
**Reproduction**
1. Enable Paimon tiering and historical partition writes for a partitioned primary-key table.
2. Populate and tier multiple date partitions, then let their Fluss partitions expire.
3. Write updates targeting those original partitions through the historical partition.
4. Process a historical tiering split containing records for multiple original partitions in the same bucket.
**Actual behavior**
Manifest discovery is repeated for each target partition, causing high tiering latency even with low data volume.
**Expected behavior**
Target partitions within the same historical split should share bucket-level manifest discovery.
### Solution
For historical fixed-bucket primary-key writers, scan one snapshot for the target bucket and group active file entries by partition. Reuse these entries when restoring individual partition writers.
Keep the cache scoped to the current split, bound the number of prefetched entries, and retain native Paimon restoration when the limit is exceeded or additional indexes are required.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Historical Paimon tiering repeatedly scans manifests [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on issue #4656:
URL: https://github.com/apache/fluss/issues/4656#issuecomment-6228097886
I checked apache/paimon#8256 against this issue. It can remove the specific repeated *writer-initialization* scans, but only for write-only primary-key tables.
With `write.sequence-number-init-mode=snapshot`, Paimon's `KeyValueFileStoreWrite` skips `scanExistingFileMetas(partition, bucket)` when `write-only=true` and the latest snapshot already contains `sequence.generation.max-sequence-number`. That should eliminate the `restoreFiles -> plan()` calls caused by opening many `(partition, bucket)` writers in a historical split. On an existing table without the property, writers still restore old files; the first commit bootstraps the property from manifest metadata, so the benefit starts with subsequent writers after that commit.
In Fluss, `PaimonLakeWriter` sets `write-only = !table.datalake.auto-compaction`. Therefore this optimization does **not** apply to the compaction-enabled runs reported above: they retain the original restore path. Native manifest caching remains the applicable mitigation for those runs. Using this PR's path would mean running tiering without automatic compaction and accepting the corresponding file-maintenance tradeoff.
Fluss `main` currently pins Paimon 2.1.0, whose release source contains #8256; the exact Paimon binary in the reported deployment is still unverified. The Paimon option can be passed through a Fluss table property as `paimon.write.sequence-number-init-mode=snapshot`. I would treat #8256 as a conditional solution for write-only deployments, not as a full resolution of this issue while compaction remains enabled. A useful check would compare historical-split planning calls and end-to-end duration after the bootstrap commit, and verify update/delete reads remain correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Historical Paimon tiering repeatedly scans manifests [fluss]
Posted by "beryllw (via GitHub)" <gi...@apache.org>.
beryllw commented on issue #4656:
URL: https://github.com/apache/fluss/issues/4656#issuecomment-6212013908
Follow-up from parameter-only verification: native manifest caching substantially reduced historical-split planning, but removing the admission threshold alone was insufficient.
The runs used the same artifact URI and parallelism, with compaction kept enabled. Restart catch-up rounds were excluded.
| Admission threshold | Cache budget / TM process memory | Typical plan | Accumulated planning per split | Full GC increments |
| --- | --- | ---: | ---: | ---: |
| Default 1 MiB | Default 128 MiB / 4 GiB | 924 ms | 59.879 s | β |
| 8 MiB | Default 128 MiB / 4 GiB | 705 ms | 46.914 s | β |
| Unrestricted | Default 128 MiB / 4 GiB | 823 ms | 55.620 s | 45 in 4m58s |
| Unrestricted | 512 MiB / 5 GiB | 6 ms | 2.935 s | 0 in 6m45s |
| Unrestricted | 256 MiB / 4 GiB | 516 ms | 31.303 s | 34 in 4m50s |
| Unrestricted | 512 MiB / 4 GiB | 5 ms | 2.317 s | 5 in 6m56s |
βTypical planβ is the median of per-split plan-duration medians; accumulated planning is also a per-split median. GC counts are increments across the same 20 TMs within each run; βββ means not included in this comparison. Snapshots and input volumes differed, so this is an observational comparison, not an equal-load benchmark.
`Unrestricted` means `cache.manifest.small-file-threshold="9223372036854775807 b"`; the budget is `cache.manifest.small-file-memory`. Explicit cache arguments were verified on the running jobs. Default values come from the inspected dependency; `soft-values` was not explicitly changed.
Key findings:
- **Admission does not guarantee retention.** With unrestricted admission and the default budget, thread samples taken after 13β50 completed plans still showed `ObjectsCache.read -> ManifestEntryCache.createSegments -> Avro/object-store reads`.
- **Increasing the budget helped with TM memory held constant.** Changing 256 to 512 MiB while retaining 4 GiB TM process memory restored fast subsequent plans. The latest five stable rounds contained 15 historical splits and 885 plans after excluding each split's first plan: median **5 ms**, P95 **13 ms**. One split had 57 plans: first **1,823 ms**, remaining 56 combined **461 ms**. The same four older manifests were still selected by 55β56 plans, supporting reuse across target partitions rather than an entirely different manifest set.
- **Full GC improved but remains a source of tail latency.** A **354.685 ms** Full GC overlapped a **389 ms** plan; the next four plans each took **4 ms**. Another **273.998 ms** Full GC overlapped a **302 ms** plan, followed by 3β4 ms plans. These were runtime `Ergonomics` events, distinct from startup metadata GC.
This supports native cache reuse without a custom restore path or disabling compaction. It does **not** establish 512 MiB as a universal/minimum budget, or distinguish capacity eviction from soft-reference collection. Physical manifest sizes, decoded cache weights and eviction causes remain unmeasured. Admission compares file size, while capacity accounts for decoded cache contents; the total budget should not simply be reused as the admission threshold.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org