You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Devaraj Das (JIRA)" <ji...@apache.org> on 2008/04/22 12:26:21 UTC
[jira] Created: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
---------------------------------------------------------------------------------------------
Key: HADOOP-3297
URL: https://issues.apache.org/jira/browse/HADOOP-3297
Project: Hadoop Core
Issue Type: Improvement
Components: mapred
Reporter: Devaraj Das
Assignee: Devaraj Das
Fix For: 0.18.0
Certain things like poll frequency, number of events fetched in one go, etc. This would affect the task->tasktracker and the tasktracker->jobtracker protocols.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595788#action_12595788 ]
Runping Qi commented on HADOOP-3297:
------------------------------------
This jira was created because of hadoop-3323
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
> Attachments: 3297.patch, 3297.patch
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594868#action_12594868 ]
Devaraj Das commented on HADOOP-3297:
-------------------------------------
Yes, I ran large jobs with 250000 maps and 400 reducers on 250 nodes and saw no performance issues.
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
> Attachments: 3297.patch, 3297.patch
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594506#action_12594506 ]
Hadoop QA commented on HADOOP-3297:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12381308/3297.patch
against trunk revision 653638.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 2 new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2406/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2406/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2406/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2406/console
This message is automatically generated.
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
> Attachments: 3297.patch
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595788#action_12595788 ]
runping edited comment on HADOOP-3297 at 5/9/08 11:09 PM:
-------------------------------------------------------------
This jira was created because of hadoop-3327
was (Author: runping):
This jira was created because of hadoop-3323
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
> Attachments: 3297.patch, 3297.patch
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595207#action_12595207 ]
Hudson commented on HADOOP-3297:
--------------------------------
Integrated in Hadoop-trunk #484 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/484/])
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
> Attachments: 3297.patch, 3297.patch
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-3297:
--------------------------------
Attachment: 3297.patch
Fixed findbugs warnings. The test failure is not related to the patch.
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
> Attachments: 3297.patch, 3297.patch
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-3297:
--------------------------------
Status: Open (was: Patch Available)
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
> Attachments: 3297.patch, 3297.patch
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-3297:
--------------------------------
Description: Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs. (was: Certain things like poll frequency, number of events fetched in one go, etc. This would affect the task->tasktracker and the tasktracker->jobtracker protocols.)
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12592816#action_12592816 ]
Devaraj Das commented on HADOOP-3297:
-------------------------------------
An interesting observation regarding using the ramfs. I guess i should raise a separate jira but let me put it here anyway -
I had a job (loadgen from hadoop-test) consisting of 2500 maps and 1 reducer. The ramfs size was 300MB and io.sort.factor was 100. The cluster had 20 nodes. Each map generated 5 MB of data. The amount of time it took to complete the job was 45 minutes (with the above changes). The number of files that missed the ramfs and ended up on disk was ~2000.
I ran the same job (with exactly the same config) with the reducer throttled - if a ramfs merge is on, it would wait for that to complete before fetching anything new. This basically results in all files ending up in the ramfs. The job ran in 30 minutes.
So although I didn't notice any significant performance gain for this job with the shuffle protocol changes as proposed in my last comment but in general it looks like this is going to be true - for a given job, if we have a faster shuffle, more files get created on the disk, and depending on the number/size of map outputs for the job, this might adversely affects the final merge, thereby affecting the overall runtime of the job.
I will see if the above behavior can be modelled.
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-3297:
--------------------------------
Attachment: 3297.patch
I ran a benchmark (loadgen) with the attached patch. Here are the details:
1) Num maps - 10000
2) Size of each map output - 1KB
3) Size of cluster - 80 nodes
4) Num reducers - 1
With the patch, the run took ~7 minutes. On trunk, the same job took ~11 minutes.
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
> Attachments: 3297.patch
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-3297:
--------------------------------
Status: Patch Available (was: Open)
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
> Attachments: 3297.patch
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594472#action_12594472 ]
Mahadev konar commented on HADOOP-3297:
---------------------------------------
agreed.
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
> Attachments: 3297.patch
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594470#action_12594470 ]
Devaraj Das commented on HADOOP-3297:
-------------------------------------
No it doesn't. That should be outside the scope of this one.
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
> Attachments: 3297.patch
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12595787#action_12595787 ]
Runping Qi commented on HADOOP-3297:
------------------------------------
This patch should be in release 17.
Without it, the shuffling phase will be painfully slow.
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
> Attachments: 3297.patch, 3297.patch
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12592806#action_12592806 ]
Devaraj Das commented on HADOOP-3297:
-------------------------------------
Here is a proposal after a discussion with Sameer:
1) The TaskTracker polls the JobTracker asking for 500 task completion events. If it gets the full payload, it immediately asks for another bunch of 500 and so on. When it gets less than 500, it switches to current behavior - sleep for a fixed amount of time (heartbeat interval). A small number of events per RPC would ensure that each RPC takes a lesser amount of time although the number of RPCs would be more.
2) The Task asks for 10000 events at a time every second from the TaskTracker.
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594962#action_12594962 ]
Mahadev konar commented on HADOOP-3297:
---------------------------------------
thats great...
+1 for commit.
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
> Attachments: 3297.patch, 3297.patch
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594633#action_12594633 ]
Hadoop QA commented on HADOOP-3297:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12381496/3297.patch
against trunk revision 653749.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2412/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2412/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2412/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2412/console
This message is automatically generated.
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
> Attachments: 3297.patch, 3297.patch
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594467#action_12594467 ]
Mahadev konar commented on HADOOP-3297:
---------------------------------------
this patch does not include waiting on the reducer fetch if the memory fs is full ? or does it ?
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
> Attachments: 3297.patch
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-3297:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Devaraj!
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
> Attachments: 3297.patch, 3297.patch
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated HADOOP-3297:
----------------------------------
Hadoop Flags: [Reviewed]
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
> Attachments: 3297.patch, 3297.patch
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-3297:
--------------------------------
Status: Patch Available (was: Open)
Retrying hudson
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
> Attachments: 3297.patch, 3297.patch
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12593778#action_12593778 ]
Runping Qi commented on HADOOP-3297:
------------------------------------
How long did the map phase take?
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
> Attachments: 3297.patch
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12593781#action_12593781 ]
Devaraj Das commented on HADOOP-3297:
-------------------------------------
The map phase took roughly 3 minutes
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
> Attachments: 3297.patch
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594808#action_12594808 ]
Mahadev konar commented on HADOOP-3297:
---------------------------------------
the patch looks good... the only concern I have is that if we want to check and see that it does not degrade perofrmace and lead to more problems at the jobtracker or otherwise.
We could try running sort with this patch or some map reduce job where we have a huge number of mappers say 100,000 and 500 reducers or something.
This is just to verify that the task tracker being agressive in fetching the mapoutputs does not degrade performance. The maps are short lived, so its not hard to imagine a situation that all the reduces start bombarding the jobtracker with requests for maps at the same time asking for more. We should check to see if the jobtracker can handle the load and the performance in such a situation does not degrade.
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
> Attachments: 3297.patch, 3297.patch
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3297) The way in which
ReduceTask/TaskTracker gets completion events during shuffle can be
improved
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12592842#action_12592842 ]
Runping Qi commented on HADOOP-3297:
------------------------------------
Under what condition fetched map outputs will end up on disk directly?
If a segment is very large, it makes sense to write it out on disk directly.
If it is one or the last few, it makes sense too. Otherwise, a fetched segment
should get into in-mem file system. If the in-mem file is full, the fetcher should wait.
This is related to hadoop-2095. They should be considered together.
> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
>
> Certain things like poll frequency, number of events fetched in one go, etc. can probably be improved to improve the shuffle performance. This would affect the task-->tasktracker and the tasktracker-->jobtracker shuffle related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.