You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Srikanth Kakani (JIRA)" <ji...@apache.org> on 2007/10/31 00:54:50 UTC
[jira] Created: (HADOOP-2131) Speculative execution should be
allowed for reducers only
Speculative execution should be allowed for reducers only
---------------------------------------------------------
Key: HADOOP-2131
URL: https://issues.apache.org/jira/browse/HADOOP-2131
Project: Hadoop
Issue Type: Improvement
Components: mapred
Environment: Hadoop job, map fetches data from external systems
Reporter: Srikanth Kakani
Priority: Critical
Fix For: 0.15.0
Consider hadoop jobs where maps fetch data from external systems, and emit the data. The reducers in this are identity reducers. The data processed by these jobs is huge. There could be slow nodes in this cluster and some of the reducers run twice as slow as their counterparts. This could result in a long tail. Speculative execution would help greatly in such cases. However given the current hadoop, we have to select speculative execution for both maps and reducers. In this case hurting the map performance as they are fetching data from external systems thereby overloading the external systems.
Speculative execution only on reducers would be a great way to solve this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2131) Speculative execution should be
allowed for reducers only
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2131:
---------------------------------------------
Attachment: patch-2131.txt
> Speculative execution should be allowed for reducers only
> ---------------------------------------------------------
>
> Key: HADOOP-2131
> URL: https://issues.apache.org/jira/browse/HADOOP-2131
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Environment: Hadoop job, map fetches data from external systems
> Reporter: Srikanth Kakani
> Assignee: Amareshwari Sri Ramadasu
> Priority: Critical
> Fix For: 0.16.0
>
> Attachments: patch-2131.txt, patch-2131.txt
>
>
> Consider hadoop jobs where maps fetch data from external systems, and emit the data. The reducers in this are identity reducers. The data processed by these jobs is huge. There could be slow nodes in this cluster and some of the reducers run twice as slow as their counterparts. This could result in a long tail. Speculative execution would help greatly in such cases. However given the current hadoop, we have to select speculative execution for both maps and reducers. In this case hurting the map performance as they are fetching data from external systems thereby overloading the external systems.
> Speculative execution only on reducers would be a great way to solve this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2131) Speculative execution should be
allowed for reducers only
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-2131:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Amareshwari!
> Speculative execution should be allowed for reducers only
> ---------------------------------------------------------
>
> Key: HADOOP-2131
> URL: https://issues.apache.org/jira/browse/HADOOP-2131
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Environment: Hadoop job, map fetches data from external systems
> Reporter: Srikanth Kakani
> Assignee: Amareshwari Sri Ramadasu
> Priority: Critical
> Fix For: 0.16.0
>
> Attachments: patch-2131.txt, patch-2131.txt, patch-2131.txt
>
>
> Consider hadoop jobs where maps fetch data from external systems, and emit the data. The reducers in this are identity reducers. The data processed by these jobs is huge. There could be slow nodes in this cluster and some of the reducers run twice as slow as their counterparts. This could result in a long tail. Speculative execution would help greatly in such cases. However given the current hadoop, we have to select speculative execution for both maps and reducers. In this case hurting the map performance as they are fetching data from external systems thereby overloading the external systems.
> Speculative execution only on reducers would be a great way to solve this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2131) Speculative execution should be
allowed for reducers only
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557519#action_12557519 ]
Hadoop QA commented on HADOOP-2131:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12372739/patch-2131.txt
against trunk revision r610603.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1533/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1533/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1533/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1533/console
This message is automatically generated.
> Speculative execution should be allowed for reducers only
> ---------------------------------------------------------
>
> Key: HADOOP-2131
> URL: https://issues.apache.org/jira/browse/HADOOP-2131
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Environment: Hadoop job, map fetches data from external systems
> Reporter: Srikanth Kakani
> Assignee: Amareshwari Sri Ramadasu
> Priority: Critical
> Fix For: 0.16.0
>
> Attachments: patch-2131.txt, patch-2131.txt, patch-2131.txt
>
>
> Consider hadoop jobs where maps fetch data from external systems, and emit the data. The reducers in this are identity reducers. The data processed by these jobs is huge. There could be slow nodes in this cluster and some of the reducers run twice as slow as their counterparts. This could result in a long tail. Speculative execution would help greatly in such cases. However given the current hadoop, we have to select speculative execution for both maps and reducers. In this case hurting the map performance as they are fetching data from external systems thereby overloading the external systems.
> Speculative execution only on reducers would be a great way to solve this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2131) Speculative execution should be
allowed for reducers only
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-2131:
----------------------------------
Status: Open (was: Patch Available)
Need to update this patch to reflect recent changes to trunk...
> Speculative execution should be allowed for reducers only
> ---------------------------------------------------------
>
> Key: HADOOP-2131
> URL: https://issues.apache.org/jira/browse/HADOOP-2131
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Environment: Hadoop job, map fetches data from external systems
> Reporter: Srikanth Kakani
> Assignee: Amareshwari Sri Ramadasu
> Priority: Critical
> Fix For: 0.16.0
>
> Attachments: patch-2131.txt, patch-2131.txt
>
>
> Consider hadoop jobs where maps fetch data from external systems, and emit the data. The reducers in this are identity reducers. The data processed by these jobs is huge. There could be slow nodes in this cluster and some of the reducers run twice as slow as their counterparts. This could result in a long tail. Speculative execution would help greatly in such cases. However given the current hadoop, we have to select speculative execution for both maps and reducers. In this case hurting the map performance as they are fetching data from external systems thereby overloading the external systems.
> Speculative execution only on reducers would be a great way to solve this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2131) Speculative execution should be
allowed for reducers only
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2131:
---------------------------------------------
Attachment: patch-2131.txt
> Speculative execution should be allowed for reducers only
> ---------------------------------------------------------
>
> Key: HADOOP-2131
> URL: https://issues.apache.org/jira/browse/HADOOP-2131
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Environment: Hadoop job, map fetches data from external systems
> Reporter: Srikanth Kakani
> Priority: Critical
> Fix For: 0.16.0
>
> Attachments: patch-2131.txt
>
>
> Consider hadoop jobs where maps fetch data from external systems, and emit the data. The reducers in this are identity reducers. The data processed by these jobs is huge. There could be slow nodes in this cluster and some of the reducers run twice as slow as their counterparts. This could result in a long tail. Speculative execution would help greatly in such cases. However given the current hadoop, we have to select speculative execution for both maps and reducers. In this case hurting the map performance as they are fetching data from external systems thereby overloading the external systems.
> Speculative execution only on reducers would be a great way to solve this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2131) Speculative execution should be
allowed for reducers only
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu reassigned HADOOP-2131:
------------------------------------------------
Assignee: Amareshwari Sri Ramadasu
> Speculative execution should be allowed for reducers only
> ---------------------------------------------------------
>
> Key: HADOOP-2131
> URL: https://issues.apache.org/jira/browse/HADOOP-2131
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Environment: Hadoop job, map fetches data from external systems
> Reporter: Srikanth Kakani
> Assignee: Amareshwari Sri Ramadasu
> Priority: Critical
> Fix For: 0.16.0
>
> Attachments: patch-2131.txt
>
>
> Consider hadoop jobs where maps fetch data from external systems, and emit the data. The reducers in this are identity reducers. The data processed by these jobs is huge. There could be slow nodes in this cluster and some of the reducers run twice as slow as their counterparts. This could result in a long tail. Speculative execution would help greatly in such cases. However given the current hadoop, we have to select speculative execution for both maps and reducers. In this case hurting the map performance as they are fetching data from external systems thereby overloading the external systems.
> Speculative execution only on reducers would be a great way to solve this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2131) Speculative execution should be
allowed for reducers only
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-2131:
----------------------------------
Status: Open (was: Patch Available)
Please go ahead and deprecate the old {{mapred.speculative.execution}} config in favour of the new ones which should be set to *true* in hadoop-default.xml.
For 0.16.0 we should let {{mapred.speculative.execution}} over-ride the new ones since it means folks actually went ahead and cared about the non-default value and hence set it in their hadoop-site.xml.
> Speculative execution should be allowed for reducers only
> ---------------------------------------------------------
>
> Key: HADOOP-2131
> URL: https://issues.apache.org/jira/browse/HADOOP-2131
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Environment: Hadoop job, map fetches data from external systems
> Reporter: Srikanth Kakani
> Assignee: Amareshwari Sri Ramadasu
> Priority: Critical
> Fix For: 0.16.0
>
> Attachments: patch-2131.txt
>
>
> Consider hadoop jobs where maps fetch data from external systems, and emit the data. The reducers in this are identity reducers. The data processed by these jobs is huge. There could be slow nodes in this cluster and some of the reducers run twice as slow as their counterparts. This could result in a long tail. Speculative execution would help greatly in such cases. However given the current hadoop, we have to select speculative execution for both maps and reducers. In this case hurting the map performance as they are fetching data from external systems thereby overloading the external systems.
> Speculative execution only on reducers would be a great way to solve this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2131) Speculative execution should be
allowed for reducers only
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12556558#action_12556558 ]
Hadoop QA commented on HADOOP-2131:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12372604/patch-2131.txt
against trunk revision .
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1498/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1498/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1498/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1498/console
This message is automatically generated.
> Speculative execution should be allowed for reducers only
> ---------------------------------------------------------
>
> Key: HADOOP-2131
> URL: https://issues.apache.org/jira/browse/HADOOP-2131
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Environment: Hadoop job, map fetches data from external systems
> Reporter: Srikanth Kakani
> Assignee: Amareshwari Sri Ramadasu
> Priority: Critical
> Fix For: 0.16.0
>
> Attachments: patch-2131.txt
>
>
> Consider hadoop jobs where maps fetch data from external systems, and emit the data. The reducers in this are identity reducers. The data processed by these jobs is huge. There could be slow nodes in this cluster and some of the reducers run twice as slow as their counterparts. This could result in a long tail. Speculative execution would help greatly in such cases. However given the current hadoop, we have to select speculative execution for both maps and reducers. In this case hurting the map performance as they are fetching data from external systems thereby overloading the external systems.
> Speculative execution only on reducers would be a great way to solve this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2131) Speculative execution should be
allowed for reducers only
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2131:
---------------------------------------------
Status: Patch Available (was: Open)
Submiting after deprecating mapred.speculative.execution
> Speculative execution should be allowed for reducers only
> ---------------------------------------------------------
>
> Key: HADOOP-2131
> URL: https://issues.apache.org/jira/browse/HADOOP-2131
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Environment: Hadoop job, map fetches data from external systems
> Reporter: Srikanth Kakani
> Assignee: Amareshwari Sri Ramadasu
> Priority: Critical
> Fix For: 0.16.0
>
> Attachments: patch-2131.txt, patch-2131.txt
>
>
> Consider hadoop jobs where maps fetch data from external systems, and emit the data. The reducers in this are identity reducers. The data processed by these jobs is huge. There could be slow nodes in this cluster and some of the reducers run twice as slow as their counterparts. This could result in a long tail. Speculative execution would help greatly in such cases. However given the current hadoop, we have to select speculative execution for both maps and reducers. In this case hurting the map performance as they are fetching data from external systems thereby overloading the external systems.
> Speculative execution only on reducers would be a great way to solve this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-2131) Speculative execution
should be allowed for reducers only
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12556508#action_12556508 ]
amareshwari edited comment on HADOOP-2131 at 1/7/08 9:16 PM:
--------------------------------------------------------------------------
Added mapred.map.tasks.speculative.execution and mapred.reduce.tasks.speculative.execution to enable speculative execution seperately for map and reduce tasks respectively.
was (Author: amareshwari):
Added mapred.map.task.speculative.execution and mapred.reduce.task.speculative.execution to enable speculative execution seperately for map and reduce tasks respectively.
> Speculative execution should be allowed for reducers only
> ---------------------------------------------------------
>
> Key: HADOOP-2131
> URL: https://issues.apache.org/jira/browse/HADOOP-2131
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Environment: Hadoop job, map fetches data from external systems
> Reporter: Srikanth Kakani
> Assignee: Amareshwari Sri Ramadasu
> Priority: Critical
> Fix For: 0.16.0
>
> Attachments: patch-2131.txt
>
>
> Consider hadoop jobs where maps fetch data from external systems, and emit the data. The reducers in this are identity reducers. The data processed by these jobs is huge. There could be slow nodes in this cluster and some of the reducers run twice as slow as their counterparts. This could result in a long tail. Speculative execution would help greatly in such cases. However given the current hadoop, we have to select speculative execution for both maps and reducers. In this case hurting the map performance as they are fetching data from external systems thereby overloading the external systems.
> Speculative execution only on reducers would be a great way to solve this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2131) Speculative execution should be
allowed for reducers only
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2131:
----------------------------------
Fix Version/s: (was: 0.15.0)
0.16.0
> Speculative execution should be allowed for reducers only
> ---------------------------------------------------------
>
> Key: HADOOP-2131
> URL: https://issues.apache.org/jira/browse/HADOOP-2131
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Environment: Hadoop job, map fetches data from external systems
> Reporter: Srikanth Kakani
> Priority: Critical
> Fix For: 0.16.0
>
>
> Consider hadoop jobs where maps fetch data from external systems, and emit the data. The reducers in this are identity reducers. The data processed by these jobs is huge. There could be slow nodes in this cluster and some of the reducers run twice as slow as their counterparts. This could result in a long tail. Speculative execution would help greatly in such cases. However given the current hadoop, we have to select speculative execution for both maps and reducers. In this case hurting the map performance as they are fetching data from external systems thereby overloading the external systems.
> Speculative execution only on reducers would be a great way to solve this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2131) Speculative execution should be
allowed for reducers only
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-2131:
----------------------------------
Status: Patch Available (was: Open)
> Speculative execution should be allowed for reducers only
> ---------------------------------------------------------
>
> Key: HADOOP-2131
> URL: https://issues.apache.org/jira/browse/HADOOP-2131
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Environment: Hadoop job, map fetches data from external systems
> Reporter: Srikanth Kakani
> Assignee: Amareshwari Sri Ramadasu
> Priority: Critical
> Fix For: 0.16.0
>
> Attachments: patch-2131.txt, patch-2131.txt, patch-2131.txt
>
>
> Consider hadoop jobs where maps fetch data from external systems, and emit the data. The reducers in this are identity reducers. The data processed by these jobs is huge. There could be slow nodes in this cluster and some of the reducers run twice as slow as their counterparts. This could result in a long tail. Speculative execution would help greatly in such cases. However given the current hadoop, we have to select speculative execution for both maps and reducers. In this case hurting the map performance as they are fetching data from external systems thereby overloading the external systems.
> Speculative execution only on reducers would be a great way to solve this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2131) Speculative execution should be
allowed for reducers only
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558234#action_12558234 ]
Hudson commented on HADOOP-2131:
--------------------------------
Integrated in Hadoop-Nightly #363 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/363/])
> Speculative execution should be allowed for reducers only
> ---------------------------------------------------------
>
> Key: HADOOP-2131
> URL: https://issues.apache.org/jira/browse/HADOOP-2131
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Environment: Hadoop job, map fetches data from external systems
> Reporter: Srikanth Kakani
> Assignee: Amareshwari Sri Ramadasu
> Priority: Critical
> Fix For: 0.16.0
>
> Attachments: patch-2131.txt, patch-2131.txt, patch-2131.txt
>
>
> Consider hadoop jobs where maps fetch data from external systems, and emit the data. The reducers in this are identity reducers. The data processed by these jobs is huge. There could be slow nodes in this cluster and some of the reducers run twice as slow as their counterparts. This could result in a long tail. Speculative execution would help greatly in such cases. However given the current hadoop, we have to select speculative execution for both maps and reducers. In this case hurting the map performance as they are fetching data from external systems thereby overloading the external systems.
> Speculative execution only on reducers would be a great way to solve this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2131) Speculative execution should be
allowed for reducers only
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2131:
---------------------------------------------
Status: Patch Available (was: Open)
Added mapred.map.task.speculative.execution and mapred.reduce.task.speculative.execution to enable speculative execution seperately for map and reduce tasks respectively.
> Speculative execution should be allowed for reducers only
> ---------------------------------------------------------
>
> Key: HADOOP-2131
> URL: https://issues.apache.org/jira/browse/HADOOP-2131
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Environment: Hadoop job, map fetches data from external systems
> Reporter: Srikanth Kakani
> Priority: Critical
> Fix For: 0.16.0
>
> Attachments: patch-2131.txt
>
>
> Consider hadoop jobs where maps fetch data from external systems, and emit the data. The reducers in this are identity reducers. The data processed by these jobs is huge. There could be slow nodes in this cluster and some of the reducers run twice as slow as their counterparts. This could result in a long tail. Speculative execution would help greatly in such cases. However given the current hadoop, we have to select speculative execution for both maps and reducers. In this case hurting the map performance as they are fetching data from external systems thereby overloading the external systems.
> Speculative execution only on reducers would be a great way to solve this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2131) Speculative execution should be
allowed for reducers only
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-2131:
----------------------------------
Attachment: patch-2131.txt
Attaching an updated patch( with a couple of minor javadoc fixes) on Amareshwari's behalf so that Hudson can pick this up right-away...
> Speculative execution should be allowed for reducers only
> ---------------------------------------------------------
>
> Key: HADOOP-2131
> URL: https://issues.apache.org/jira/browse/HADOOP-2131
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Environment: Hadoop job, map fetches data from external systems
> Reporter: Srikanth Kakani
> Assignee: Amareshwari Sri Ramadasu
> Priority: Critical
> Fix For: 0.16.0
>
> Attachments: patch-2131.txt, patch-2131.txt, patch-2131.txt
>
>
> Consider hadoop jobs where maps fetch data from external systems, and emit the data. The reducers in this are identity reducers. The data processed by these jobs is huge. There could be slow nodes in this cluster and some of the reducers run twice as slow as their counterparts. This could result in a long tail. Speculative execution would help greatly in such cases. However given the current hadoop, we have to select speculative execution for both maps and reducers. In this case hurting the map performance as they are fetching data from external systems thereby overloading the external systems.
> Speculative execution only on reducers would be a great way to solve this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.