You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-issues@hadoop.apache.org by "jaehoon ko (JIRA)" <ji...@apache.org> on 2014/06/27 03:18:24 UTC

[jira] [Updated] (MAPREDUCE-5947) Map phase merge can better utilize memory

     [ https://issues.apache.org/jira/browse/MAPREDUCE-5947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

jaehoon ko updated MAPREDUCE-5947:
----------------------------------

    Component/s: task

> Map phase merge can better utilize memory
> -----------------------------------------
>
>                 Key: MAPREDUCE-5947
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5947
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: performance, task
>    Affects Versions: 2.4.0
>            Reporter: jaehoon ko
>            Assignee: jaehoon ko
>              Labels: newbie, performance
>
> Map phase merge reads spills from disk and writes intermediate results back to disk, and so on. I think it is possible to use memory to store intermediate results, thereby reducing disk IO. Because kvbuffer is nullified right before merge, we have at least io.sort.mb amount of heap available.
> MAPREDUCE-4511 can be considered as an effort to utilize memory better through read ahead, but number of disk IO is unchanged.
> Please give me your thoughts. I'd like to take up this issue.



--
This message was sent by Atlassian JIRA
(v6.2#6252)