[jira] Updated: (HADOOP-4766) Hadoop performance degrades significantly as more and more jobs complete

["Amar Kamat (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-4766:
-------------------------------

    Attachment: HADOOP-4766-v2.4.patch

Attaching a patch that cleans up completed jobs as and when the jobtracker runs low on memory. Jobtracker is assumed to run low on memory if its memory usage crosses the predefined limit passed using {{mapred.jobtracker.maximum.usable.memory.percent}}. 

Ran 7 back to back (sleep) jobs of 100,000 tasks each on a jobtracker with 2GB heap memory and the result is as follows :
|no|job-runtime|
|1|17mins|
|2|17mins|
|3|18mins|
|4|17mins|
|5|17mins|
|6|18mins|
|7|17mins|

Also (manually) killed few jobs to check if they are cleaned up.

Result of _test-patch_ is as follows

{noformat}
[exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 8 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
{noformat}

> Hadoop performance degrades significantly as more and more jobs complete
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-4766
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4766
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.18.2, 0.19.0
>            Reporter: Runping Qi
>            Assignee: Amar Kamat
>            Priority: Blocker
>             Fix For: 0.18.3, 0.19.1, 0.20.0
>
>         Attachments: HADOOP-4766-v1.patch, HADOOP-4766-v2.4.patch, map_scheduling_rate.txt
>
>
> When I ran the gridmix 2 benchmark load on a fresh cluster of 500 nodes with hadoop trunk, 
> the gridmix load, consisting of 202 map/reduce jobs of various sizes, completed in 32 minutes. 
> Then I ran the same set of the jobs on the same cluster, yhey completed in 43 minutes.
> When I ran them the third times, it took (almost) forever --- the job tracker became non-responsive.
> The job  tracker's heap size was set to 2GB. 
> The cluster is configured to keep up to 500 jobs in memory.
> The job tracker kept one cpu busy all the time. Look like it was due to GC.
> I believe the release 0.18/0.19 have the similar behavior.
> I believe 0.18 and 0.18 also have the similar behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.