[jira] [Assigned] (YARN-12003) Router does not shut down its scheduled executor (SubClusterCleaner) on service stop

["Jose Luis López (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YARN-12003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jose Luis López reassigned YARN-12003:
--------------------------------------

    Assignee: Jose Luis López

> Router does not shut down its scheduled executor (SubClusterCleaner) on service stop
> ------------------------------------------------------------------------------------
>
>                 Key: YARN-12003
>                 URL: https://issues.apache.org/jira/browse/YARN-12003
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: router
>            Reporter: Jose Luis López
>            Assignee: Jose Luis López
>            Priority: Minor
>              Labels: pull-request-available
>
> {\{Router#serviceStart}} schedules \{{SubClusterCleaner}} on \{{scheduledExecutorService}} at a fixed rate. This is on by default (\{{yarn.router.deregister.subcluster.enabled}}) and runs every 60s (\{{yarn.router.subcluster.cleaner.interval.time}}).
> {\{Router#serviceStop}} never shuts that executor down. As a result, the cleaner keeps running after the Router has stopped. It keeps calling \{{FederationStateStoreFacade#getSubClusters}} and \{{#deregisterSubCluster}} on a state store that nothing maintains any more, and that may already be closed. The executor's thread is not a daemon thread, so it also outlives the service.
> This matters when a Router is stopped inside a JVM that keeps running: unit tests, embedded or mock federation setups (see YARN-11984), or a stop/start in the same process.
> h3. Proposed fix
> * In \{{Router#serviceStop}}, before stopping the child services, call \{{HadoopExecutors#shutdown(scheduledExecutorService, LOG, 5, SECONDS)}}.
> * This is a graceful shutdown with a bounded wait. It waits up to 5s for an in-flight cleaner run to finish, then calls \{{shutdownNow()}} and waits up to 5s more, so stopping takes at most 10s longer.
> * We don't call \{{shutdownNow()}} straight away, so that a cleaner scan already in progress can finish before the state store it is reading is closed.
> * Add a \{{@VisibleForTesting}} getter for the executor and a test (\{{TestRouter#testServiceStopShutsDownScheduledExecutor}}) that checks the executor is terminated once the Router has stopped.
> This was split out of YARN-11984. Its federation mock Router closes the state store after the Router stops, so it depends on the cleaner having stopped. This fix does not depend on the other pieces split out of YARN-11984.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org