You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "prashantwason (via GitHub)" <gi...@apache.org> on 2026/03/29 01:09:20 UTC

[I] [HUDI-18xxx] Flink table version upgrade blocks Pekko dispatcher thread causing heartbeat timeout [hudi]

prashantwason opened a new issue, #18410:
URL: https://github.com/apache/hudi/issues/18410

   ## Description
   
   When a Flink streaming job performs a Hudi table version upgrade (e.g., v7 → v8), the `StreamWriteOperatorCoordinator.start()` method runs `tryUpgrade()` synchronously on the Pekko dispatcher thread. For tables with many archived timeline actions, the LSM timeline migration in `SevenToEightUpgradeHandler.upgradeToLSMTimeline()` performs extensive remote storage I/O that can block the dispatcher thread for 90+ seconds.
   
   This blocks the Pekko dispatcher from processing RPC messages including heartbeat responses, causing the ResourceManager to disconnect the JobManager due to heartbeat timeout. The resulting application shutdown terminates filesystem thread pools, which causes `RejectedExecutionException` in the still-running upgrade code.
   
   ## Root Cause
   
   Two issues compound:
   
   1. **`StreamWriteOperatorCoordinator.start()` runs upgrade synchronously on dispatcher thread**: The `tryUpgrade()` → `UpgradeDowngrade.run()` → `SevenToEightUpgradeHandler.upgrade()` chain runs entirely on the Pekko dispatcher thread, blocking heartbeat processing.
   
   2. **`upgradeToLSMTimeline()` has excessive I/O**: Uses small batch size (default 10) and calls `compactAndClean()` after every batch. With hundreds of archived actions, this creates ~50+ batches × ~5 remote storage operations each = 250+ storage operations, each taking ~2 seconds on cloud storage.
   
   ## Error Chain
   
   ```
   StreamWriteOperatorCoordinator.start() [dispatcher thread]
     → tryUpgrade() [blocks dispatcher for 90+ seconds]
       → upgradeToLSMTimeline() [heavy I/O]
     → Pekko dispatcher blocked, can't process heartbeat RPCs
       → ResourceManager heartbeat timeout → disconnect JobManager
         → Application shutdown → CFS ThreadPoolExecutor terminated
           → RejectedExecutionException in upgrade code
   ```
   
   ## Fix
   
   1. **Run upgrade on executor thread**: Move `tryUpgrade()`, `initMetadataTable()`, and `restoreEvents()` to the coordinator's single-threaded FIFO executor, freeing the dispatcher thread for heartbeats. Since all event handling also goes through this executor, the FIFO ordering guarantees the upgrade completes before any events are processed.
   
   2. **Optimize LSM timeline migration**: Use a larger batch size (500 instead of 10) and call `compactAndClean()` once at the end instead of after every batch, reducing remote storage operations from 250+ to ~6.
   
   ## Impact
   
   This affects any Flink streaming job that upgrades a Hudi table with a non-trivial archived timeline. The failure is non-deterministic and depends on timeline size and storage latency.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [HUDI-18xxx] Flink table version upgrade blocks Pekko dispatcher thread causing heartbeat timeout [hudi]

Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on issue #18410:
URL: https://github.com/apache/hudi/issues/18410#issuecomment-5355710519

   "Thanks for reporting this.
   
   This looks addressed by #19052 (*perf: Add dedicated batch size config for LSM timeline migration on u…*, merged 2026-06-24).
   
   Note this merged after 1.2.0, so it is on `master` and will ship in the next release rather than being available in 1.2.0.
   
   Closing on that basis - please reopen or comment if you still see the behaviour on a build containing that change."


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [HUDI-18xxx] Flink table version upgrade blocks Pekko dispatcher thread causing heartbeat timeout [hudi]

Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 closed issue #18410: [HUDI-18xxx] Flink table version upgrade blocks Pekko dispatcher thread causing heartbeat timeout
URL: https://github.com/apache/hudi/issues/18410


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org