You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2024/02/23 03:50:00 UTC

[jira] [Commented] (YARN-11626) Optimization of the safeDelete operation in ZKRMStateStore

    [ https://issues.apache.org/jira/browse/YARN-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819896#comment-17819896 ] 

ASF GitHub Bot commented on YARN-11626:
---------------------------------------

XbaoWu opened a new pull request, #6577:
URL: https://github.com/apache/hadoop/pull/6577

   <!--
     Thanks for sending a pull request!
       1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
       2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
   -->
   
   ### Description of PR
   
   For more information about this PR, please refer to the following issue:
   [YARN-11626](https://issues.apache.org/jira/browse/YARN-11626) Optimization of the safeDelete operation in ZKRMStateStore
   
   ### How was this patch tested?
   
   add TestCheckRemoveZKNodeRMStateStore.testSafeDeleteZKNode()
   
   ### For code changes:
   
   - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
   - [x] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
   - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
   
   




> Optimization of the safeDelete operation in ZKRMStateStore
> ----------------------------------------------------------
>
>                 Key: YARN-11626
>                 URL: https://issues.apache.org/jira/browse/YARN-11626
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager
>    Affects Versions: 3.0.0-alpha4, 3.1.1, 3.3.0
>            Reporter: wangzhihui
>            Priority: Minor
>
> h1. Description 
>  * We can be observed that removing app info started at 06:17:20, but the NoNodeException was received at 06:17:35. 
>  * During the 15s interval, Curator was retrying the metadata operation. Due to the non-idempotent nature of the Zookeeper deletion operation, in one of the retry attempts, the metadata operation was successful but no response was received. In the next retry it resulted in a NoNodeException, triggering the STATE_STORE_FENCED event and ultimately causing the current ResourceManager to switch to standby .
> {code:java}
> 2023-10-28 06:17:20,359 INFO  recovery.RMStateStore (RMStateStore.java:transition(333)) - Removing info for app: application_1697410508608_140368
> 2023-10-28 06:17:20,359 INFO  resourcemanager.RMAppManager (RMAppManager.java:checkAppNumCompletedLimit(303)) - Application should be expired, max number of completed apps kept in memory met: maxCompletedAppsInMemory = 1000, removing app application_1697410508608_140368 from memory:
> 2023-10-28 06:17:35,665 ERROR recovery.RMStateStore (RMStateStore.java:transition(337)) - Error removing app: application_1697410508608_140368
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
>         at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> 2023-10-28 06:17:35,666 INFO  recovery.RMStateStore (RMStateStore.java:handleStoreEvent(1147)) - RMStateStore state change from ACTIVE to FENCED
> 2023-10-28 06:17:35,666 ERROR resourcemanager.ResourceManager (ResourceManager.java:handle(898)) - Received RMFatalEvent of type STATE_STORE_FENCED, caused by org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
> 2023-10-28 06:17:35,666 INFO  resourcemanager.ResourceManager (ResourceManager.java:transitionToStandby(1309)) - Transitioning to standby state
>  {code}
> h1. Solution
> The NoNodeException clearly indicates that the Znode no longer exists, so we can safely ignore this exception to avoid triggering a larger impact on the cluster caused by ResourceManager failover.
> h1. Other
> We also need to discuss and optimize the same issues in safeCreate.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org