You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@celeborn.apache.org by "senthh (via GitHub)" <gi...@apache.org> on 2026/10/09 10:48:04 UTC

[PR] [CELEBORN-2472] Unregister Celeborn application from MR AppMaster on shutdown [celeborn]

senthh opened a new pull request, #3864:
URL: https://github.com/apache/celeborn/pull/3864

   ## [CELEBORN-2472] Unregister Celeborn application from MR AppMaster on shutdown
   
   **Problem:** `MRAppMasterWithCeleborn` creates a `LifecycleManager` and registers the application with the Celeborn master, but never stops it or unregisters on shutdown. The Celeborn master only frees the application (and the worker's shuffle files / slots) via heartbeat expiry (`appHeartbeatTimeoutMs`, default 120s), leaking worker disk after every MR job.
   
   **Fix:** store the `LifecycleManager` as a field and stop it from `serviceStop()` (not `stop()`), so it runs exactly once even though both `MRAppMaster#shutDownJob()` and the JVM shutdown hook can reach `stop()`. `LifecycleManager.stop() -> heartbeater.stop() -> unregisterApplication()` sends `ApplicationLost`, releasing slots and shuffle files immediately. Mirrors the Tez client `CelebornDagAppMaster#serviceStop()` and matches every other engine client (Spark, Flink).
   
   **Files:** `client-mr/mr/src/main/java/org/apache/celeborn/mapreduce/v2/app/MRAppMasterWithCeleborn.java` (+27/−1).
   
   **Validation:** compiles clean; identical to the pattern already used by other engine clients.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [CELEBORN-2472] Unregister Celeborn application from MR AppMaster on shutdown [celeborn]

Posted by "zaynt4606 (via GitHub)" <gi...@apache.org>.

zaynt4606 commented on PR #3864:
URL: https://github.com/apache/celeborn/pull/3864#issuecomment-6136631170

   Thanks @senthh for the contribution!
   Merged to main(v1.0.0). [a649f32db](https://github.com/apache/celeborn/commit/a649f32db5c6c655e3dbde5414e4e9471ab445ce)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [CELEBORN-2472] Unregister Celeborn application from MR AppMaster on shutdown [celeborn]

Posted by "zaynt4606 (via GitHub)" <gi...@apache.org>.

zaynt4606 closed pull request #3864: [CELEBORN-2472] Unregister Celeborn application from MR AppMaster on shutdown
URL: https://github.com/apache/celeborn/pull/3864


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org