You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Todd Lipcon (Created) (JIRA)" <ji...@apache.org> on 2011/10/25 22:14:35 UTC
[jira] [Created] (MAPREDUCE-3267) MR2 reduce tasks showing >100%
complete
MR2 reduce tasks showing >100% complete
---------------------------------------
Key: MAPREDUCE-3267
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3267
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, task
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Priority: Critical
My job is currently showing >100% reduce completion. Some reduce tasks are much higher than 100% complete. they appear to be in the "last merge pass" stage
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3267) MR2 reduce tasks showing >100%
complete
Posted by "Andrew Hancock (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259647#comment-13259647 ]
Andrew Hancock commented on MAPREDUCE-3267:
-------------------------------------------
I am experiencing this bug in version 0.22 as well.
It happens any time I turn on the following config in my job:
config.setFloat("mapreduce.reduce.input.buffer.percent", 0.1f);
If I remove this setting, the bug goes away.
When this option is turned on, the following symptoms are exhibited:
1. During the reduce phase many of the reduce tasks will quickly read > 100%. The Job will then report 100% complete.
2. I have on occasion seen it then go back below 100% on the overall progress. It will oscillate between different numbers in that case.
3. The job appears to complete normally in spite of the progress reporting issues.
> MR2 reduce tasks showing >100% complete
> ---------------------------------------
>
> Key: MAPREDUCE-3267
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3267
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, task
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ravi Prakash
>
> My job is currently showing >100% reduce completion. Some reduce tasks are much higher than 100% complete. they appear to be in the "last merge pass" stage
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3267) MR2 reduce tasks showing >100%
complete
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-3267:
-------------------------------------------
Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.23.1)
> MR2 reduce tasks showing >100% complete
> ---------------------------------------
>
> Key: MAPREDUCE-3267
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3267
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, task
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ravi Prakash
>
> My job is currently showing >100% reduce completion. Some reduce tasks are much higher than 100% complete. they appear to be in the "last merge pass" stage
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3267) MR2 reduce tasks showing >100%
complete
Posted by "Ravi Prakash (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173563#comment-13173563 ]
Ravi Prakash commented on MAPREDUCE-3267:
-----------------------------------------
Hi Todd!
Any clues on how this may be reproduced?
> MR2 reduce tasks showing >100% complete
> ---------------------------------------
>
> Key: MAPREDUCE-3267
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3267
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, task
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
>
> My job is currently showing >100% reduce completion. Some reduce tasks are much higher than 100% complete. they appear to be in the "last merge pass" stage
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3267) MR2 reduce tasks showing >100%
complete
Posted by "Andrew Hancock (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Hancock updated MAPREDUCE-3267:
--------------------------------------
Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 3.0.0, 2.0.0, 0.23.3)
Affects Version/s: 0.22.0
> MR2 reduce tasks showing >100% complete
> ---------------------------------------
>
> Key: MAPREDUCE-3267
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3267
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, task
> Affects Versions: 0.22.0, 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ravi Prakash
>
> My job is currently showing >100% reduce completion. Some reduce tasks are much higher than 100% complete. they appear to be in the "last merge pass" stage
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3267) MR2 reduce tasks showing >100%
complete
Posted by "Ravi Prakash (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Prakash reassigned MAPREDUCE-3267:
---------------------------------------
Assignee: Ravi Prakash
> MR2 reduce tasks showing >100% complete
> ---------------------------------------
>
> Key: MAPREDUCE-3267
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3267
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, task
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ravi Prakash
>
> My job is currently showing >100% reduce completion. Some reduce tasks are much higher than 100% complete. they appear to be in the "last merge pass" stage
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3267) MR2 reduce tasks showing >100%
complete
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173576#comment-13173576 ]
Todd Lipcon commented on MAPREDUCE-3267:
----------------------------------------
This was on 0.23.0 before release, so might be fixed by now. I do think I was using intermediate compression and maybe some of the "in-memory merge" config options. I _think_ this was the config I used for the job:
{code}
<property>
<name>io.sort.mb</name>
<value>650</value>
</property>
<property>
<name>mapreduce.map.sort.spill.percent</name>
<value>0.98</value>
</property>
<property>
<name>mapreduce.reduce.shuffle.input.buffer.percent</name>
<value>0.8</value>
</property>
<property>
<name>mapreduce.reduce.input.buffer.percent</name>
<value>0.8</value>
</property>
<property>
<name>io.sort.factor</name>
<value>100</value>
</property>
{code}
and it was a terasort
> MR2 reduce tasks showing >100% complete
> ---------------------------------------
>
> Key: MAPREDUCE-3267
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3267
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, task
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Ravi Prakash
>
> My job is currently showing >100% reduce completion. Some reduce tasks are much higher than 100% complete. they appear to be in the "last merge pass" stage
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira