[I] [tiering] Slow manifest compaction during lake commit blocks other tables' tiering progress [fluss]

["beryllw (via GitHub)" <gi...@apache.org>]

beryllw opened a new issue, #3259:
URL: https://github.com/apache/fluss/issues/3259

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
   
   
   ### Description
   
   ## Problem
   The TieringCommitOperator currently runs with parallelism=1 and maxParallelism=1. All table commits are processed sequentially in a single thread. When a table's lake commit triggers manifest compaction (e.g., in Paimon), it can take a significant amount of time, during which all other tables' commits are blocked.This causes data freshness degradation for unrelated tables, potentially triggering alerts even though those tables have data ready to commit.
   
   ## Current Behavior
   The commit flow in TieringCommitOperator.processElement() is synchronous:
   1. Collect all bucket write results for a table
   2. Call LakeCommitter.toCommittable()
   3. Call LakeCommitter.commit() — this is the bottleneck, as it may trigger manifest compaction
   4. Call FlussTableLakeSnapshotCommitter.commit() to notify Fluss
   
   Since the operator is single-threaded with parallelism=1, step 3 for one table blocks steps 1-4 for all other tables.
   
   ### Willingness to contribute
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org