[jira] [Commented] (YARN-1680) availableResources sent to applicationMaster in heartbeat should exclude blacklistedNodes free memory.

["Chen He (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14301703#comment-14301703 ] 

Chen He commented on YARN-1680:
-------------------------------

Hi [~cwelch], thank you for the comments. 

{quote}Unfortunately, I don't think that this can be solved with checks during addition and removal - I believe that we will need to keep a persistent picture of all blacklisted nodes for an application regardless of their cluster state because the two can vary independently and changes after a blacklist request may invalidate things{quote}

Yes, I agree. However, as [~jlowe] suggested, just simply fix the headroom calculation without introducing new mechanism or facts can help some clusters that are not using label scheduling. Can we leave the new feature and fact introduction in YARN-2848, and just fix the headroom calculation here? 

{quote} (for example, cluster blacklists just before app blacklists, the app blacklist request is discarded, the cluster reinstates but the app still cannot use the node for reasons different from the nodes cluster availability - we will still include that node in headroom incorrectly...).{quote}

Do you mean this scenario:
Cluster just add Node A into its blacklist. A millisecond later, app requests so but suddenly discards it. (But why it is discarded? according to my proposed idea?)

I worte all possible conditions here:
1. Cluster add A to blacklist, app never requests so, then, we just remove A's resource from clusterResource, this has been covered in my previous patch;
2. Cluster add A to blacklist, app adds so, according to previous solution, A's resource can be subtract from clusterResource twice, it is incorrect, we need only subtract it once. 
3. Cluster does not add A, app adds A, it is normal case. Not a problem.
4. Cluster add A to blacklist, app did so, but a millisecond later Cluster removes A (a node becomes healthy suddenly?), we only need to subtract A's resource once,  
5. Cluster does not add A, app adds A, but during the headroom calculation, Cluster add A, We may get incorrect headroom anyway, but we can finally get a correct headroom in the next heartbeat. 

Please let me know if there is any scenario that I did not cover. 

> availableResources sent to applicationMaster in heartbeat should exclude blacklistedNodes free memory.
> ------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-1680
>                 URL: https://issues.apache.org/jira/browse/YARN-1680
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>    Affects Versions: 2.2.0, 2.3.0
>         Environment: SuSE 11 SP2 + Hadoop-2.3 
>            Reporter: Rohith
>            Assignee: Chen He
>         Attachments: YARN-1680-WIP.patch, YARN-1680-v2.patch, YARN-1680-v2.patch, YARN-1680.patch
>
>
> There are 4 NodeManagers with 8GB each.Total cluster capacity is 32GB.Cluster slow start is set to 1.
> Job is running reducer task occupied 29GB of cluster.One NodeManager(NM-4) is become unstable(3 Map got killed), MRAppMaster blacklisted unstable NodeManager(NM-4). All reducer task are running in cluster now.
> MRAppMaster does not preempt the reducers because for Reducer preemption calculation, headRoom is considering blacklisted nodes memory. This makes jobs to hang forever(ResourceManager does not assing any new containers on blacklisted nodes but returns availableResouce considers cluster free memory). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)