You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-dev@hadoop.apache.org by "Jason Lowe (JIRA)" <ji...@apache.org> on 2017/05/11 15:53:04 UTC

[jira] [Created] (HADOOP-14412) HostsFileReader#getHostDetails is very expensive on large clusters

Jason Lowe created HADOOP-14412:
-----------------------------------

             Summary: HostsFileReader#getHostDetails is very expensive on large clusters
                 Key: HADOOP-14412
                 URL: https://issues.apache.org/jira/browse/HADOOP-14412
             Project: Hadoop Common
          Issue Type: Bug
          Components: util
    Affects Versions: 2.8.0
            Reporter: Jason Lowe
            Assignee: Jason Lowe


After upgrading one of our large clusters to 2.8 we noticed many IPC server threads of the resourcemanager spending time in NodesListManager#isValidNode which in turn was calling HostsFileReader#getHostDetails.  The latter is creating complete copies of the include and exclude sets for every node heartbeat, and these sets are not small due to the size of the cluster.  These copies are causing multiple resizes of the underlying HashSets being filled and creating lots of garbage.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org