You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-dev@hadoop.apache.org by "Devaraj Das (JIRA)" <ji...@apache.org> on 2008/07/28 06:57:31 UTC

[jira] Commented: (HADOOP-3841) merge phase runs out of disk space

    [ https://issues.apache.org/jira/browse/HADOOP-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12617360#action_12617360 ] 

Devaraj Das commented on HADOOP-3841:
-------------------------------------

If compression is on, the merge output would be compressed. The situation is better in 0.18 with compression working well. 

At this point of time, the only other option is to have a smaller io.sort.factor but this, in general, might increase the disk IO for the map output records, affecting performance in a negative way.

> merge phase runs out of disk space
> ----------------------------------
>
>                 Key: HADOOP-3841
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3841
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.17.1
>            Reporter: Christian Kunz
>
> We observe that reduce tasks run out of disk space during merging (after fetching all map output) although there would be enough space if the framework did not try to generate too large merge files.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.