[jira] [Created] (MAPREDUCE-7336) MapTask leaks sort buffer into merge increasing GC pressure in merge

["Scott Oaks (Jira)" <ji...@apache.org>]

Scott Oaks created MAPREDUCE-7336:
-------------------------------------

             Summary: MapTask leaks sort buffer into merge increasing GC pressure in merge
                 Key: MAPREDUCE-7336
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7336
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: performance
    Affects Versions: 3.2.2
            Reporter: Scott Oaks


When running with a large value for io.sort.mb, we see lots of GC pressure (and often GC failures) during the merge phase as the array of spill records are allocated. This is because the kvbuffer allocated during the sort phase is still allocated and consuming memory.

This doesn't seem to be the intention; line 1539 of MapTask.java reads:
{code:java}
// release sort buffer before the merge
kvbuffer = null;{code}
But this isn't the only place that has a reference to kvbuffer; line 1002 wraps it into kvmeta:
{code:java}
kvmeta = ByteBUffer.wrap(kvbuffer){code}
The code needs to deference kvmeta so that kvbuffer can be garbage collected as desired.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org