You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Thomas Graves (JIRA)" <ji...@apache.org> on 2011/05/20 16:06:47 UTC
[jira] [Created] (MAPREDUCE-2524) Backport trunk heuristics for
failing maps when we get fetch failures retrieving map output during
shuffle
Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle
----------------------------------------------------------------------------------------------------------
Key: MAPREDUCE-2524
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2524
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: tasktracker
Affects Versions: 0.20.204.0
Reporter: Thomas Graves
Assignee: Thomas Graves
Priority: Minor
Fix For: 0.20.205.0
The heuristics for failing maps when we get map output fetch failures during the shuffle is pretty conservative in 20. Backport the heuristics from trunk which are more aggressive, simpler, and configurable.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2524) Backport trunk heuristics for
failing maps when we get fetch failures retrieving map output during
shuffle
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated MAPREDUCE-2524:
-------------------------------------
Fix Version/s: 0.20.204.0
> Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle
> ----------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2524
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2524
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Minor
> Fix For: 0.20.204.0, 0.20.205.0
>
> Attachments: MAPREDUCE2524-20security-v2.patch, MAPREDUCE2524-patch-20security.txt, MAPREDUCE2524-patch-20security.txt
>
>
> The heuristics for failing maps when we get map output fetch failures during the shuffle is pretty conservative in 20. Backport the heuristics from trunk which are more aggressive, simpler, and configurable.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2524) Backport trunk heuristics for
failing maps when we get fetch failures retrieving map output during
shuffle
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated MAPREDUCE-2524:
-------------------------------------
Attachment: MAPREDUCE2524-patch-20security.txt
reattaching wit license granted.
> Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle
> ----------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2524
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2524
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Minor
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE2524-patch-20security.txt, MAPREDUCE2524-patch-20security.txt
>
>
> The heuristics for failing maps when we get map output fetch failures during the shuffle is pretty conservative in 20. Backport the heuristics from trunk which are more aggressive, simpler, and configurable.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2524) Backport trunk heuristics for
failing maps when we get fetch failures retrieving map output during
shuffle
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated MAPREDUCE-2524:
-------------------------------------
Release Note: Added a new configuration option: mapreduce.reduce.shuffle.maxfetchfailures, and removed a no longer used option: mapred.reduce.copy.backoff.
Status: Patch Available (was: Open)
> Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle
> ----------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2524
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2524
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Minor
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE2524-20security-v2.patch, MAPREDUCE2524-patch-20security.txt, MAPREDUCE2524-patch-20security.txt
>
>
> The heuristics for failing maps when we get map output fetch failures during the shuffle is pretty conservative in 20. Backport the heuristics from trunk which are more aggressive, simpler, and configurable.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2524) Backport trunk heuristics for
failing maps when we get fetch failures retrieving map output during
shuffle
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated MAPREDUCE-2524:
-------------------------------------
Attachment: MAPREDUCE2524-patch-20security.txt
patch for the branch-0.20-security.
> Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle
> ----------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2524
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2524
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Minor
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE2524-patch-20security.txt
>
>
> The heuristics for failing maps when we get map output fetch failures during the shuffle is pretty conservative in 20. Backport the heuristics from trunk which are more aggressive, simpler, and configurable.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2524) Backport trunk heuristics for
failing maps when we get fetch failures retrieving map output during
shuffle
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039987#comment-13039987 ]
Hadoop QA commented on MAPREDUCE-2524:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12480599/MAPREDUCE2524-20security-v2.patch
against trunk revision 1127444.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/313//console
This message is automatically generated.
> Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle
> ----------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2524
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2524
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Minor
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE2524-20security-v2.patch, MAPREDUCE2524-patch-20security.txt, MAPREDUCE2524-patch-20security.txt
>
>
> The heuristics for failing maps when we get map output fetch failures during the shuffle is pretty conservative in 20. Backport the heuristics from trunk which are more aggressive, simpler, and configurable.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2524) Backport trunk heuristics for
failing maps when we get fetch failures retrieving map output during
shuffle
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038174#comment-13038174 ]
Thomas Graves commented on MAPREDUCE-2524:
------------------------------------------
The javadoc and eclipse failures existed before/without these changes.
[exec] -1 overall. [exec]
[exec] +1 @author. The patch does not contain any @author tags. [exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages. [exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories. [exec] [exec]
> Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle
> ----------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2524
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2524
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Minor
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE2524-patch-20security.txt
>
>
> The heuristics for failing maps when we get map output fetch failures during the shuffle is pretty conservative in 20. Backport the heuristics from trunk which are more aggressive, simpler, and configurable.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2524) Backport trunk heuristics for
failing maps when we get fetch failures retrieving map output during
shuffle
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-2524:
-------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
+1
I committed this. Thanks, Tom!
> Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle
> ----------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2524
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2524
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Minor
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE2524-20security-v2.patch, MAPREDUCE2524-patch-20security.txt, MAPREDUCE2524-patch-20security.txt
>
>
> The heuristics for failing maps when we get map output fetch failures during the shuffle is pretty conservative in 20. Backport the heuristics from trunk which are more aggressive, simpler, and configurable.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2524) Backport trunk heuristics for
failing maps when we get fetch failures retrieving map output during
shuffle
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039988#comment-13039988 ]
Thomas Graves commented on MAPREDUCE-2524:
------------------------------------------
Note, this failed because patch is for branch-0.20-security
> Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle
> ----------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2524
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2524
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Minor
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE2524-20security-v2.patch, MAPREDUCE2524-patch-20security.txt, MAPREDUCE2524-patch-20security.txt
>
>
> The heuristics for failing maps when we get map output fetch failures during the shuffle is pretty conservative in 20. Backport the heuristics from trunk which are more aggressive, simpler, and configurable.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2524) Backport trunk heuristics for
failing maps when we get fetch failures retrieving map output during
shuffle
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated MAPREDUCE-2524:
-------------------------------------
Attachment: MAPREDUCE2524-20security-v2.patch
upload new patch with changes in mapred-default.xml
> Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle
> ----------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2524
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2524
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Minor
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE2524-20security-v2.patch, MAPREDUCE2524-patch-20security.txt, MAPREDUCE2524-patch-20security.txt
>
>
> The heuristics for failing maps when we get map output fetch failures during the shuffle is pretty conservative in 20. Backport the heuristics from trunk which are more aggressive, simpler, and configurable.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2524) Backport trunk heuristics for
failing maps when we get fetch failures retrieving map output during
shuffle
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated MAPREDUCE-2524:
-------------------------------------
Fix Version/s: (was: 0.20.205.0)
> Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle
> ----------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2524
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2524
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Affects Versions: 0.20.204.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Minor
> Fix For: 0.20.204.0
>
> Attachments: MAPREDUCE2524-20security-v2.patch, MAPREDUCE2524-patch-20security.txt, MAPREDUCE2524-patch-20security.txt
>
>
> The heuristics for failing maps when we get map output fetch failures during the shuffle is pretty conservative in 20. Backport the heuristics from trunk which are more aggressive, simpler, and configurable.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira