[I] [lake/tiering] Reduce split queue and commit collection overhead [fluss]

["beryllw (via GitHub)" <gi...@apache.org>]

beryllw opened a new issue, #4455:
URL: https://github.com/apache/fluss/issues/4455

   ## Description
   
   Two bookkeeping paths perform quadratic work when a tiering round contains many splits:
   
   - `TieringSourceEnumerator.assignSplits()` repeatedly calls `remove(0)` on an `ArrayList`, shifting the remaining entries on every assignment.
   - `TieringCommitOperator.collectTableAllBucketWriteResult()` scans all results collected so far and rebuilds a set and a list for every new result. Collecting N results therefore takes O(N²) work.
   
   Use a deque for pending splits and an incremental per-table result accumulator with an expected count and bucket deduplication. Preserve assignment order, failover cleanup, duplicate-bucket rejection, and inconsistent-result-count validation. Commit only after all expected results arrive.
   
   Verify equivalent behavior for normal completion, duplicate results, inconsistent counts, and failure cleanup. Queue removal and result accumulation should each take O(N) total work per round. Split prefetching and changes to reader resource lifecycles are outside this task.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [lake/tiering] Reduce split queue and commit collection overhead [fluss]

["luoyuxia (via GitHub)" <gi...@apache.org>]

luoyuxia closed issue #4455: [lake/tiering] Reduce split queue and commit collection overhead
URL: https://github.com/apache/fluss/issues/4455


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org