You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Arun C Murthy (JIRA)" <ji...@apache.org> on 2010/02/22 18:44:30 UTC

[jira] Created: (MAPREDUCE-1521) Protection against incorrectly configured reduces

Protection against incorrectly configured reduces
-------------------------------------------------

                 Key: MAPREDUCE-1521
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: jobtracker
            Reporter: Arun C Murthy
            Assignee: Arun C Murthy
             Fix For: 0.22.0


We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.

This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.

Proposal: Come up with heuristics/configs to fail such jobs early. 

Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1521) Protection against incorrectly configured reduces

Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836750#action_12836750 ] 

Arun C Murthy commented on MAPREDUCE-1521:
------------------------------------------

Some thoughts:

# Naive approach: Limit on inputBytes/#reduces - this doesn't handle blow-up or filtering of inputs
# Track #mapOutputBytes at JT and a limit on mapOutputBytes/#reduces - this still doesn't fail early i.e. lots of maps have to complete

Maybe a hybrid? Possibly option1 with a flag which says '#reduces is small, and i really, really want it so'?

> Protection against incorrectly configured reduces
> -------------------------------------------------
>
>                 Key: MAPREDUCE-1521
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>            Priority: Critical
>             Fix For: 0.22.0
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early. 
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1521) Protection against incorrectly configured reduces

Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mahadev konar updated MAPREDUCE-1521:
-------------------------------------

    Attachment: MAPREDUCE-1521-0.20-yahoo.patch

this patch adds a test case.

> Protection against incorrectly configured reduces
> -------------------------------------------------
>
>                 Key: MAPREDUCE-1521
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>            Reporter: Arun C Murthy
>            Assignee: Mahadev konar
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early. 
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1521) Protection against incorrectly configured reduces

Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mahadev konar updated MAPREDUCE-1521:
-------------------------------------

    Attachment: MAPREDUCE-1521-0.20-yahoo.patch

minor change to test case.

> Protection against incorrectly configured reduces
> -------------------------------------------------
>
>                 Key: MAPREDUCE-1521
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>            Reporter: Arun C Murthy
>            Assignee: Mahadev konar
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early. 
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1521) Protection against incorrectly configured reduces

Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy updated MAPREDUCE-1521:
-------------------------------------

    Priority: Critical  (was: Major)

> Protection against incorrectly configured reduces
> -------------------------------------------------
>
>                 Key: MAPREDUCE-1521
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>            Priority: Critical
>             Fix For: 0.22.0
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early. 
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1521) Protection against incorrectly configured reduces

Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mahadev konar updated MAPREDUCE-1521:
-------------------------------------

    Attachment: MAPREDUCE-1521-0.20-yahoo.patch

modified patch so that the limit is a job limit than a cluster wide limit on the jobtracker. minor chnages to the code.

> Protection against incorrectly configured reduces
> -------------------------------------------------
>
>                 Key: MAPREDUCE-1521
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>            Reporter: Arun C Murthy
>            Assignee: Mahadev konar
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early. 
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1521) Protection against incorrectly configured reduces

Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886079#action_12886079 ] 

Mahadev konar commented on MAPREDUCE-1521:
------------------------------------------

I think Hongs idea seems reasonable.. and also todd is a valid point of disabling this feature by default.


> Protection against incorrectly configured reduces
> -------------------------------------------------
>
>                 Key: MAPREDUCE-1521
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>            Reporter: Arun C Murthy
>            Assignee: Mahadev konar
>            Priority: Critical
>             Fix For: 0.22.0
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early. 
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1521) Protection against incorrectly configured reduces

Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mahadev konar updated MAPREDUCE-1521:
-------------------------------------

    Status: Patch Available  (was: Open)

> Protection against incorrectly configured reduces
> -------------------------------------------------
>
>                 Key: MAPREDUCE-1521
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>            Reporter: Arun C Murthy
>            Assignee: Mahadev konar
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-trunk.patch
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early. 
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1521) Protection against incorrectly configured reduces

Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mahadev konar updated MAPREDUCE-1521:
-------------------------------------

    Attachment: MAPREDUCE-1521-trunk.patch

this patch is for trunk.

> Protection against incorrectly configured reduces
> -------------------------------------------------
>
>                 Key: MAPREDUCE-1521
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>            Reporter: Arun C Murthy
>            Assignee: Mahadev konar
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-trunk.patch
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early. 
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Assigned: (MAPREDUCE-1521) Protection against incorrectly configured reduces

Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mahadev konar reassigned MAPREDUCE-1521:
----------------------------------------

    Assignee: Mahadev konar  (was: Arun C Murthy)

> Protection against incorrectly configured reduces
> -------------------------------------------------
>
>                 Key: MAPREDUCE-1521
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>            Reporter: Arun C Murthy
>            Assignee: Mahadev konar
>            Priority: Critical
>             Fix For: 0.22.0
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early. 
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1521) Protection against incorrectly configured reduces

Posted by "Alex Loddengaard (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886550#action_12886550 ] 

Alex Loddengaard commented on MAPREDUCE-1521:
---------------------------------------------

I didn't review the patch, but I'm a strong +1 for the idea.  Anything that gives admins more control is probably a good thing.  I expect Allen will agree with me :).

Alex

> Protection against incorrectly configured reduces
> -------------------------------------------------
>
>                 Key: MAPREDUCE-1521
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>            Reporter: Arun C Murthy
>            Assignee: Mahadev konar
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: MAPREDUCE-1521-0.20-yahoo.patch
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early. 
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1521) Protection against incorrectly configured reduces

Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836902#action_12836902 ] 

Todd Lipcon commented on MAPREDUCE-1521:
----------------------------------------

This makes me just a little bit nervous - the case I'm worried about is when a job is working fine in production and then starts failing at 3am some morning because the data volume increased just a little bit over the threshold.

Could we default this behavior off and only turn it on for clusters where the operators prefer it?

> Protection against incorrectly configured reduces
> -------------------------------------------------
>
>                 Key: MAPREDUCE-1521
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>            Priority: Critical
>             Fix For: 0.22.0
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early. 
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1521) Protection against incorrectly configured reduces

Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836883#action_12836883 ] 

Allen Wittenauer commented on MAPREDUCE-1521:
---------------------------------------------

+10000

A common conversation:

Ops: "You have too many/too few reduces."

User: "How many should I have?"

Ops: "Uhhh...."



> Protection against incorrectly configured reduces
> -------------------------------------------------
>
>                 Key: MAPREDUCE-1521
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>            Priority: Critical
>             Fix For: 0.22.0
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early. 
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1521) Protection against incorrectly configured reduces

Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837019#action_12837019 ] 

Hong Tang commented on MAPREDUCE-1521:
--------------------------------------

We need to limit the amount of map output bytes or reduce input bytes as they must fit on a single physical node. We could then fail the *job* if a particular map or reduce violates such limits - assuming that reexecuting the map/reduce would lead to the same kind of violation.

> Protection against incorrectly configured reduces
> -------------------------------------------------
>
>                 Key: MAPREDUCE-1521
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>            Priority: Critical
>             Fix For: 0.22.0
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early. 
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1521) Protection against incorrectly configured reduces

Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mahadev konar updated MAPREDUCE-1521:
-------------------------------------

    Attachment: MAPREDUCE-1521-0.20-yahoo.patch

this patch adds a limit to amount of data that a reduce receives. If the estimated input data to the reduce is greater than the configured value at the jobtracker, the jobtracker fials the job. The configuration value is set to 0 (switched off) y default.

This patch is for yahoo hadoop security branch. I will upload the patch for trunk soon.

> Protection against incorrectly configured reduces
> -------------------------------------------------
>
>                 Key: MAPREDUCE-1521
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>            Reporter: Arun C Murthy
>            Assignee: Mahadev konar
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: MAPREDUCE-1521-0.20-yahoo.patch
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early. 
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1521) Protection against incorrectly configured reduces

Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mahadev konar updated MAPREDUCE-1521:
-------------------------------------

    Attachment: MAPREDUCE-1521-0.20-yahoo.patch

this patch adds some diagnostic information for the users on why the job failed.  I am still adding a junit test for this patch. again this patch is for yahoo 0.20. Will upload a patch for trunk soon.


> Protection against incorrectly configured reduces
> -------------------------------------------------
>
>                 Key: MAPREDUCE-1521
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>            Reporter: Arun C Murthy
>            Assignee: Mahadev konar
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: MAPREDUCE-1521-0.20-yahoo.patch, MAPREDUCE-1521-0.20-yahoo.patch
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early. 
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1521) Protection against incorrectly configured reduces

Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836765#action_12836765 ] 

Luke Lu commented on MAPREDUCE-1521:
------------------------------------

2a. Track output bytes per map and compute an extrapolated mapOutputBytes from a few randomly selected map output, so only a few maps needs to complete. We can have an expert mode to bypass the heuristic.

> Protection against incorrectly configured reduces
> -------------------------------------------------
>
>                 Key: MAPREDUCE-1521
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1521
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>            Priority: Critical
>             Fix For: 0.22.0
>
>
> We've seen a fair number of instances where naive users process huge data-sets (>10TB) with badly mis-configured #reduces e.g. 1 reduce.
> This is a significant problem on large clusters since it takes each attempt of the reduce a long time to shuffle and then run into problems such as local disk-space etc. Then it takes 4 such attempts.
> Proposal: Come up with heuristics/configs to fail such jobs early. 
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.