You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Xiao Kang (JIRA)" <ji...@apache.org> on 2010/06/23 17:25:52 UTC

[jira] Created: (MAPREDUCE-1891) Disk io of shuffle phase may decrease if shuffled data not decompressed

Disk io of shuffle phase may decrease if shuffled data not decompressed
-----------------------------------------------------------------------

                 Key: MAPREDUCE-1891
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1891
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: tasktracker
            Reporter: Xiao Kang


Current shuffled data is decompressed immediately. It may use much less memory for shuffle buffer if the data is not decompressed until merging to disk.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-1891) Disk io of shuffle phase may decrease if shuffled data not decompressed

Posted by "Xiao Kang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881719#action_12881719 ] 

Xiao Kang commented on MAPREDUCE-1891:
--------------------------------------

Compression of intermidate result of map output  is used to decrease the local disk io and network bandwith usage. However, in shuffle phase, shuffled data is decompressed immediately and more memeoy is used. If the shuffled data is keeped compressed until merging to disk, less memory is used for the same data set. Then more shuffled data can fit in the same shuffle buffer and less disk merge is needed.

Any suggestions?

> Disk io of shuffle phase may decrease if shuffled data not decompressed
> -----------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1891
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1891
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: tasktracker
>            Reporter: Xiao Kang
>
> Current shuffled data is decompressed immediately. It may use much less memory for shuffle buffer if the data is not decompressed until merging to disk.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.