[jira] [Created] (MAPREDUCE-6970) archive-logs tool should throttle container requests

["Robert Kanter (JIRA)" <ji...@apache.org>]

Robert Kanter created MAPREDUCE-6970:
----------------------------------------

             Summary: archive-logs tool should throttle container requests
                 Key: MAPREDUCE-6970
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6970
             Project: Hadoop Map/Reduce
          Issue Type: Bug
    Affects Versions: 3.0.0-alpha1, 2.8.0
            Reporter: Robert Kanter


The {{mapred archive-logs}} command currently has no way to throttle the number of requested containers.  For example, we recently saw a busy cluster where the tool hadn't been run for a while and there were about 20,000 apps to process.  This meant that the tool tried to request 20,000 containers and got a ton of GC and then OOM trying to handle that.

This problem can be mitigated by setting {{-maxEligibleApps}} to a more reasonable value, but doing so would require running the tool multiple times; plus, the default value is {{-1}} (all).

We should add a way to throttle the max number of concurrently running containers that the tool manages.  Something like {{-concurrency <n>}} where it would only allow up to {{n}} containers at a time.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org