You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Christian Kunz (JIRA)" <ji...@apache.org> on 2008/10/01 03:07:44 UTC

[jira] Commented: (HADOOP-3841) merge phase runs out of disk space

    [ https://issues.apache.org/jira/browse/HADOOP-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635922#action_12635922 ] 

Christian Kunz commented on HADOOP-3841:
----------------------------------------

To get beyond this bottleneck, for such reduces requiring a lot disk space for merging, we deleted all map outputs on such nodes, getting back a lot of space. On one of these nodes with about 280GB reduce input we observed that one of the merged files was 75GB (a single file!)

> merge phase runs out of disk space
> ----------------------------------
>
>                 Key: HADOOP-3841
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3841
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.17.2
>            Reporter: Christian Kunz
>
> We observe that reduce tasks run out of disk space during merging (after fetching all map output) although there would be enough space if the framework did not try to generate too large merge files.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.