You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Arun C Murthy (JIRA)" <ji...@apache.org> on 2010/02/22 18:44:30 UTC
[jira] Created: (MAPREDUCE-1521) Protection against incorrectly
configured reduces
Protection against incorrectly configured reduces
-------------------------------------------------
Key: MAPREDUCE-1521
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobtracker
Reporter: Arun C Murthy
Assignee: Arun C Murthy
Fix For: 0.22.0
We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
Proposal: Come up with heuristics/configs to fail such jobs early.
Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1521) Protection against incorrectly
configured reduces
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836750#action_12836750 ]
Arun C Murthy commented on MAPREDUCE-1521:
------------------------------------------
Some thoughts:
# Naive approach: Limit on inputBytes/#reduces - this doesn't handle blow-up or filtering of inputs
# Track #mapOutputBytes at JT and a limit on mapOutputBytes/#reduces - this still doesn't fail early i.e. lots of maps have to complete
Maybe a hybrid? Possibly option1 with a flag which says '#reduces is small, and i really, really want it so'?
> Protection against incorrectly configured reduces
> -------------------------------------------------
>
> Key: MAPREDUCE-1521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Priority: Critical
> Fix For: 0.22.0
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1521) Protection against incorrectly
configured reduces
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1521:
-------------------------------------
Attachment: MAPREDUCE-1521-0.20-yahoo.patch
this patch adds a test case.
> Protection against incorrectly configured reduces
> -------------------------------------------------
>
> Key: MAPREDUCE-1521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Arun C Murthy
> Assignee: Mahadev konar
> Priority: Critical
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1521) Protection against incorrectly
configured reduces
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1521:
-------------------------------------
Attachment: MAPREDUCE-1521-0.20-yahoo.patch
minor change to test case.
> Protection against incorrectly configured reduces
> -------------------------------------------------
>
> Key: MAPREDUCE-1521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Arun C Murthy
> Assignee: Mahadev konar
> Priority: Critical
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1521) Protection against incorrectly
configured reduces
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-1521:
-------------------------------------
Priority: Critical (was: Major)
> Protection against incorrectly configured reduces
> -------------------------------------------------
>
> Key: MAPREDUCE-1521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Priority: Critical
> Fix For: 0.22.0
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1521) Protection against incorrectly
configured reduces
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1521:
-------------------------------------
Attachment: MAPREDUCE-1521-0.20-yahoo.patch
modified patch so that the limit is a job limit than a cluster wide limit on the jobtracker. minor chnages to the code.
> Protection against incorrectly configured reduces
> -------------------------------------------------
>
> Key: MAPREDUCE-1521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Arun C Murthy
> Assignee: Mahadev konar
> Priority: Critical
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1521) Protection against incorrectly
configured reduces
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886079#action_12886079 ]
Mahadev konar commented on MAPREDUCE-1521:
------------------------------------------
I think Hongs idea seems reasonable.. and also todd is a valid point of disabling this feature by default.
> Protection against incorrectly configured reduces
> -------------------------------------------------
>
> Key: MAPREDUCE-1521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Arun C Murthy
> Assignee: Mahadev konar
> Priority: Critical
> Fix For: 0.22.0
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1521) Protection against incorrectly
configured reduces
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1521:
-------------------------------------
Status: Patch Available (was: Open)
> Protection against incorrectly configured reduces
> -------------------------------------------------
>
> Key: MAPREDUCE-1521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Arun C Murthy
> Assignee: Mahadev konar
> Priority: Critical
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-trunk.patch
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1521) Protection against incorrectly
configured reduces
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1521:
-------------------------------------
Attachment: MAPREDUCE-1521-trunk.patch
this patch is for trunk.
> Protection against incorrectly configured reduces
> -------------------------------------------------
>
> Key: MAPREDUCE-1521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Arun C Murthy
> Assignee: Mahadev konar
> Priority: Critical
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-trunk.patch
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-1521) Protection against incorrectly
configured reduces
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar reassigned MAPREDUCE-1521:
----------------------------------------
Assignee: Mahadev konar (was: Arun C Murthy)
> Protection against incorrectly configured reduces
> -------------------------------------------------
>
> Key: MAPREDUCE-1521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Arun C Murthy
> Assignee: Mahadev konar
> Priority: Critical
> Fix For: 0.22.0
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1521) Protection against incorrectly
configured reduces
Posted by "Alex Loddengaard (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886550#action_12886550 ]
Alex Loddengaard commented on MAPREDUCE-1521:
---------------------------------------------
I didn't review the patch, but I'm a strong +1 for the idea. Anything that gives admins more control is probably a good thing. I expect Allen will agree with me :).
Alex
> Protection against incorrectly configured reduces
> -------------------------------------------------
>
> Key: MAPREDUCE-1521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Arun C Murthy
> Assignee: Mahadev konar
> Priority: Critical
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1521-0.20-yahoo.patch
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1521) Protection against incorrectly
configured reduces
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836902#action_12836902 ]
Todd Lipcon commented on MAPREDUCE-1521:
----------------------------------------
This makes me just a little bit nervous - the case I'm worried about is when a job is working fine in production and then starts failing at 3am some morning because the data volume increased just a little bit over the threshold.
Could we default this behavior off and only turn it on for clusters where the operators prefer it?
> Protection against incorrectly configured reduces
> -------------------------------------------------
>
> Key: MAPREDUCE-1521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Priority: Critical
> Fix For: 0.22.0
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1521) Protection against incorrectly
configured reduces
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836883#action_12836883 ]
Allen Wittenauer commented on MAPREDUCE-1521:
---------------------------------------------
+10000
A common conversation:
Ops: "You have too many/too few reduces."
User: "How many should I have?"
Ops: "Uhhh...."
> Protection against incorrectly configured reduces
> -------------------------------------------------
>
> Key: MAPREDUCE-1521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Priority: Critical
> Fix For: 0.22.0
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1521) Protection against incorrectly
configured reduces
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837019#action_12837019 ]
Hong Tang commented on MAPREDUCE-1521:
--------------------------------------
We need to limit the amount of map output bytes or reduce input bytes as they must fit on a single physical node. We could then fail the *job* if a particular map or reduce violates such limits - assuming that reexecuting the map/reduce would lead to the same kind of violation.
> Protection against incorrectly configured reduces
> -------------------------------------------------
>
> Key: MAPREDUCE-1521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Priority: Critical
> Fix For: 0.22.0
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1521) Protection against incorrectly
configured reduces
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1521:
-------------------------------------
Attachment: MAPREDUCE-1521-0.20-yahoo.patch
this patch adds a limit to amount of data that a reduce receives. If the estimated input data to the reduce is greater than the configured value at the jobtracker, the jobtracker fials the job. The configuration value is set to 0 (switched off) y default.
This patch is for yahoo hadoop security branch. I will upload the patch for trunk soon.
> Protection against incorrectly configured reduces
> -------------------------------------------------
>
> Key: MAPREDUCE-1521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Arun C Murthy
> Assignee: Mahadev konar
> Priority: Critical
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1521-0.20-yahoo.patch
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1521) Protection against incorrectly
configured reduces
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-1521:
-------------------------------------
Attachment: MAPREDUCE-1521-0.20-yahoo.patch
this patch adds some diagnostic information for the users on why the job failed. I am still adding a junit test for this patch. again this patch is for yahoo 0.20. Will upload a patch for trunk soon.
> Protection against incorrectly configured reduces
> -------------------------------------------------
>
> Key: MAPREDUCE-1521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Arun C Murthy
> Assignee: Mahadev konar
> Priority: Critical
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1521) Protection against incorrectly
configured reduces
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836765#action_12836765 ]
Luke Lu commented on MAPREDUCE-1521:
------------------------------------
2a. Track output bytes per map and compute an extrapolated mapOutputBytes from a few randomly selected map output, so only a few maps needs to complete. We can have an expert mode to bypass the heuristic.
> Protection against incorrectly configured reduces
> -------------------------------------------------
>
> Key: MAPREDUCE-1521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Priority: Critical
> Fix For: 0.22.0
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.