You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2007/05/07 23:37:15 UTC

[jira] Created: (HADOOP-1336) turn on speculative execution by defaul

turn on speculative execution by defaul
---------------------------------------

                 Key: HADOOP-1336
                 URL: https://issues.apache.org/jira/browse/HADOOP-1336
             Project: Hadoop
          Issue Type: Task
          Components: mapred
    Affects Versions: 0.12.3
            Reporter: Owen O'Malley
         Assigned To: Owen O'Malley
             Fix For: 0.14.0


Now that speculative execution is working again, we should enable it by default.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1336) turn on speculative execution by defaul

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12503913 ] 

Hadoop QA commented on HADOOP-1336:
-----------------------------------

Integrated in Hadoop-Nightly #119 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/119/])

> turn on speculative execution by defaul
> ---------------------------------------
>
>                 Key: HADOOP-1336
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1336
>             Project: Hadoop
>          Issue Type: Task
>          Components: mapred
>    Affects Versions: 0.12.3
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.14.0
>
>         Attachments: spec-exec.patch
>
>
> Now that speculative execution is working again, we should enable it by default.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1336) turn on speculative execution by defaul

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-1336:
---------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I just committed this.  Thanks, Owen!

> turn on speculative execution by defaul
> ---------------------------------------
>
>                 Key: HADOOP-1336
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1336
>             Project: Hadoop
>          Issue Type: Task
>          Components: mapred
>    Affects Versions: 0.12.3
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.14.0
>
>         Attachments: spec-exec.patch
>
>
> Now that speculative execution is working again, we should enable it by default.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1336) turn on speculative execution by defaul

Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-1336:
----------------------------------

    Status: Patch Available  (was: Open)

> turn on speculative execution by defaul
> ---------------------------------------
>
>                 Key: HADOOP-1336
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1336
>             Project: Hadoop
>          Issue Type: Task
>          Components: mapred
>    Affects Versions: 0.12.3
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.14.0
>
>         Attachments: spec-exec.patch
>
>
> Now that speculative execution is working again, we should enable it by default.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1336) turn on speculative execution by defaul

Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12552074 ] 

Runping Qi commented on HADOOP-1336:
------------------------------------


Speculative execution does not work well without the patch for hadoop-1281.

With that fix, the speculative should well, and should be on by default. It does no harm
to normal jobs that do not involve side effects such as  writing files to dfs.
The users of these jobs can choose to turn it off.


> turn on speculative execution by defaul
> ---------------------------------------
>
>                 Key: HADOOP-1336
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1336
>             Project: Hadoop
>          Issue Type: Task
>          Components: mapred
>    Affects Versions: 0.12.3
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.14.0
>
>         Attachments: spec-exec.patch
>
>
> Now that speculative execution is working again, we should enable it by default.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1336) turn on speculative execution by defaul

Posted by "Marco Nicosia (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marco Nicosia updated HADOOP-1336:
----------------------------------


My whole point is that users do not know that they need to choose to turn that option off. Why do that, instead of letting the users who want to enable the feature choose to turn that option ON?


> turn on speculative execution by defaul
> ---------------------------------------
>
>                 Key: HADOOP-1336
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1336
>             Project: Hadoop
>          Issue Type: Task
>          Components: mapred
>    Affects Versions: 0.12.3
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.14.0
>
>         Attachments: spec-exec.patch
>
>
> Now that speculative execution is working again, we should enable it by default.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1336) turn on speculative execution by defaul

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12500522 ] 

Hadoop QA commented on HADOOP-1336:
-----------------------------------

+1

http://issues.apache.org/jira/secure/attachment/12358665/spec-exec.patch applied and successfully tested against trunk revision r543222.

Test results:   http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/222/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/222/console

> turn on speculative execution by defaul
> ---------------------------------------
>
>                 Key: HADOOP-1336
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1336
>             Project: Hadoop
>          Issue Type: Task
>          Components: mapred
>    Affects Versions: 0.12.3
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.14.0
>
>         Attachments: spec-exec.patch
>
>
> Now that speculative execution is working again, we should enable it by default.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1336) turn on speculative execution by defaul

Posted by "Marco Nicosia (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marco Nicosia updated HADOOP-1336:
----------------------------------


At the risk of stirring up closed issues, are we sure this is the correct choice? We've been keeping speculative execution off on our clusters, and recently accidentally turned it on. Drama ensued. The way I see it, there are two scenarios for users who don't know about speculative execution. Either it's off by default, and it's no problem to anyone, or it's on by default, and a subset are bewildered by Hadoop's "odd" behavior. This is a support burden.

The only reason to have it on by default (I think?) is to improve performance across the board. But this is by far not the only solution to affect performance, so we are doing significant user auditing regardless. If our less-efficient users can take advantage of speculative execution for their jobs, they will. I think, since some knowledge of speculative execution is good before enabling it, it's better used as part of the optimization process, rather than making it a default of the framework.


> turn on speculative execution by defaul
> ---------------------------------------
>
>                 Key: HADOOP-1336
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1336
>             Project: Hadoop
>          Issue Type: Task
>          Components: mapred
>    Affects Versions: 0.12.3
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.14.0
>
>         Attachments: spec-exec.patch
>
>
> Now that speculative execution is working again, we should enable it by default.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-1336) turn on speculative execution by defaul

Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-1336:
----------------------------------

    Attachment: spec-exec.patch

> turn on speculative execution by defaul
> ---------------------------------------
>
>                 Key: HADOOP-1336
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1336
>             Project: Hadoop
>          Issue Type: Task
>          Components: mapred
>    Affects Versions: 0.12.3
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.14.0
>
>         Attachments: spec-exec.patch
>
>
> Now that speculative execution is working again, we should enable it by default.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-1336) turn on speculative execution by defaul

Posted by "Michael Bieniosek (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12552577 ] 

Michael Bieniosek commented on HADOOP-1336:
-------------------------------------------

-1

I agree with Marco -- I don't think speculative execution is something the average user wants by default.

> turn on speculative execution by defaul
> ---------------------------------------
>
>                 Key: HADOOP-1336
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1336
>             Project: Hadoop
>          Issue Type: Task
>          Components: mapred
>    Affects Versions: 0.12.3
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.14.0
>
>         Attachments: spec-exec.patch
>
>
> Now that speculative execution is working again, we should enable it by default.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.