You are viewing a plain text version of this content. The canonical link for it is here.

Posted to yarn-dev@hadoop.apache.org by "Sandy Ryza (JIRA)" <ji...@apache.org> on 2013/10/10 23:14:45 UTC

[jira] [Created] (YARN-1291) RM INFO logs limit scheduling speed

Sandy Ryza created YARN-1291:
--------------------------------

             Summary: RM INFO logs limit scheduling speed
                 Key: YARN-1291
                 URL: https://issues.apache.org/jira/browse/YARN-1291
             Project: Hadoop YARN
          Issue Type: Bug
          Components: resourcemanager
    Affects Versions: 2.2.0
            Reporter: Sandy Ryza
            Assignee: Sandy Ryza


I've been running some microbenchmarks to see how fast the Fair Scheduler can fill up a cluster and found its performance is significantly hampered by logging.

I tested with 500 (mock) nodes, and found that:
* Taking out fair scheduler INFO logs on the critical path brought down the latency from 14000 ms to 6000 ms
* Taking out the INFO that RMContainerImpl logs when a container transitions brought it down from 6000 ms to 4000 ms
* Taking out RMAuditLogger logs brought it down from 4000 ms to 1700 ms



--
This message was sent by Atlassian JIRA
(v6.1#6144)