You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Scott Chen (JIRA)" <ji...@apache.org> on 2010/05/06 21:14:48 UTC
[jira] Created: (MAPREDUCE-1761) FairScheduler should allow
separate configuration of node and rack locality wait time
FairScheduler should allow separate configuration of node and rack locality wait time
-------------------------------------------------------------------------------------
Key: MAPREDUCE-1761
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1761
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 0.22.0
Reporter: Scott Chen
Assignee: Scott Chen
Fix For: 0.22.0
It would be nice that we can separately assign rack locality wait time.
In our use case, we would set node locality wait to zero and wait only rack locality.
I propose that we add two parameters
mapred.fairscheduler.locality.delay.nodetorack
mapred.fairscheduler.locality.delay.racktoany
This allows specifying the wait time on each stage.
And we can use
mapred.fairscheduler.locality.delay
as the default value of the above fields so that this is backward compatible.
Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1761) FairScheduler should allow
separate configuration of node and rack locality wait time
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1761:
----------------------------------
Attachment: MAPREDUCE-1761.txt
> FairScheduler should allow separate configuration of node and rack locality wait time
> -------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1761
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1761
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1761.txt
>
>
> It would be nice that we can separately assign rack locality wait time.
> In our use case, we would set node locality wait to zero and wait only rack locality.
> I propose that we add two parameters
> mapred.fairscheduler.locality.delay.nodetorack
> mapred.fairscheduler.locality.delay.racktoany
> This allows specifying the wait time on each stage.
> And we can use
> mapred.fairscheduler.locality.delay
> as the default value of the above fields so that this is backward compatible.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1761) FairScheduler should allow
separate configuration of node and rack locality wait time
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated MAPREDUCE-1761:
----------------------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Resolution: Fixed
I just committed this. Thanks Scott!
> FairScheduler should allow separate configuration of node and rack locality wait time
> -------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1761
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1761
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1761-v1.1.txt, MAPREDUCE-1761-v1.2.txt, MAPREDUCE-1761.txt
>
>
> It would be nice that we can separately assign rack locality wait time.
> In our use case, we would set node locality wait to zero and wait only rack locality.
> I propose that we add two parameters
> mapred.fairscheduler.locality.delay.nodetorack
> mapred.fairscheduler.locality.delay.racktoany
> This allows specifying the wait time on each stage.
> And we can use
> mapred.fairscheduler.locality.delay
> as the default value of the above fields so that this is backward compatible.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1761) FairScheduler should allow
separate configuration of node and rack locality wait time
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866286#action_12866286 ]
Scott Chen commented on MAPREDUCE-1761:
---------------------------------------
Added two parameters in the patch
mapred.fairscheduler.locality.delay.nodetorack
mapred.fairscheduler.locality.delay.racktoany
This will allow us to configure the wait time separately.
> FairScheduler should allow separate configuration of node and rack locality wait time
> -------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1761
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1761
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1761.txt
>
>
> It would be nice that we can separately assign rack locality wait time.
> In our use case, we would set node locality wait to zero and wait only rack locality.
> I propose that we add two parameters
> mapred.fairscheduler.locality.delay.nodetorack
> mapred.fairscheduler.locality.delay.racktoany
> This allows specifying the wait time on each stage.
> And we can use
> mapred.fairscheduler.locality.delay
> as the default value of the above fields so that this is backward compatible.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1761) FairScheduler should allow
separate configuration of node and rack locality wait time
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866354#action_12866354 ]
Matei Zaharia commented on MAPREDUCE-1761:
------------------------------------------
This looks good, but I have some minor suggestions on the naming to make it less verbose:
* Change the variable names from localityDelayNodeLocal and localityDelayRackLocal to nodeLocalityDelay and rackLocalityDelay.
* Change the config params to mapred.fairscheduler.locality.delay.node and mapred.fairscheduler.locality.delay.rack
> FairScheduler should allow separate configuration of node and rack locality wait time
> -------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1761
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1761
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1761-v1.1.txt, MAPREDUCE-1761.txt
>
>
> It would be nice that we can separately assign rack locality wait time.
> In our use case, we would set node locality wait to zero and wait only rack locality.
> I propose that we add two parameters
> mapred.fairscheduler.locality.delay.nodetorack
> mapred.fairscheduler.locality.delay.racktoany
> This allows specifying the wait time on each stage.
> And we can use
> mapred.fairscheduler.locality.delay
> as the default value of the above fields so that this is backward compatible.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1761) FairScheduler should allow
separate configuration of node and rack locality wait time
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867299#action_12867299 ]
Matei Zaharia commented on MAPREDUCE-1761:
------------------------------------------
Looks good, thanks!
> FairScheduler should allow separate configuration of node and rack locality wait time
> -------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1761
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1761
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1761-v1.1.txt, MAPREDUCE-1761-v1.2.txt, MAPREDUCE-1761.txt
>
>
> It would be nice that we can separately assign rack locality wait time.
> In our use case, we would set node locality wait to zero and wait only rack locality.
> I propose that we add two parameters
> mapred.fairscheduler.locality.delay.nodetorack
> mapred.fairscheduler.locality.delay.racktoany
> This allows specifying the wait time on each stage.
> And we can use
> mapred.fairscheduler.locality.delay
> as the default value of the above fields so that this is backward compatible.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1761) FairScheduler should allow
separate configuration of node and rack locality wait time
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866453#action_12866453 ]
Hadoop QA commented on MAPREDUCE-1761:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12444258/MAPREDUCE-1761-v1.2.txt
against trunk revision 943039.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/538/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/538/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/538/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/538/console
This message is automatically generated.
> FairScheduler should allow separate configuration of node and rack locality wait time
> -------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1761
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1761
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1761-v1.1.txt, MAPREDUCE-1761-v1.2.txt, MAPREDUCE-1761.txt
>
>
> It would be nice that we can separately assign rack locality wait time.
> In our use case, we would set node locality wait to zero and wait only rack locality.
> I propose that we add two parameters
> mapred.fairscheduler.locality.delay.nodetorack
> mapred.fairscheduler.locality.delay.racktoany
> This allows specifying the wait time on each stage.
> And we can use
> mapred.fairscheduler.locality.delay
> as the default value of the above fields so that this is backward compatible.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1761) FairScheduler should allow
separate configuration of node and rack locality wait time
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1761:
----------------------------------
Attachment: MAPREDUCE-1761-v1.1.txt
dhruba suggested some name changes.
So I have changed the two parameters to
mapred.fairscheduler.locality.delay.nodelocal
mapred.fairscheduler.locality.delay.racklocal
And changed some minor comments.
> FairScheduler should allow separate configuration of node and rack locality wait time
> -------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1761
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1761
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1761-v1.1.txt, MAPREDUCE-1761.txt
>
>
> It would be nice that we can separately assign rack locality wait time.
> In our use case, we would set node locality wait to zero and wait only rack locality.
> I propose that we add two parameters
> mapred.fairscheduler.locality.delay.nodetorack
> mapred.fairscheduler.locality.delay.racktoany
> This allows specifying the wait time on each stage.
> And we can use
> mapred.fairscheduler.locality.delay
> as the default value of the above fields so that this is backward compatible.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1761) FairScheduler should allow
separate configuration of node and rack locality wait time
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1761:
----------------------------------
Attachment: MAPREDUCE-1761-v1.2.txt
Changed the params and variables according to Matei's suggestion. Thanks, Matei.
> FairScheduler should allow separate configuration of node and rack locality wait time
> -------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1761
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1761
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1761-v1.1.txt, MAPREDUCE-1761-v1.2.txt, MAPREDUCE-1761.txt
>
>
> It would be nice that we can separately assign rack locality wait time.
> In our use case, we would set node locality wait to zero and wait only rack locality.
> I propose that we add two parameters
> mapred.fairscheduler.locality.delay.nodetorack
> mapred.fairscheduler.locality.delay.racktoany
> This allows specifying the wait time on each stage.
> And we can use
> mapred.fairscheduler.locality.delay
> as the default value of the above fields so that this is backward compatible.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1761) FairScheduler should allow
separate configuration of node and rack locality wait time
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1761:
----------------------------------
Attachment: (was: MAPREDUCE-1761-v1.2.txt)
> FairScheduler should allow separate configuration of node and rack locality wait time
> -------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1761
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1761
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1761-v1.1.txt, MAPREDUCE-1761-v1.2.txt, MAPREDUCE-1761.txt
>
>
> It would be nice that we can separately assign rack locality wait time.
> In our use case, we would set node locality wait to zero and wait only rack locality.
> I propose that we add two parameters
> mapred.fairscheduler.locality.delay.nodetorack
> mapred.fairscheduler.locality.delay.racktoany
> This allows specifying the wait time on each stage.
> And we can use
> mapred.fairscheduler.locality.delay
> as the default value of the above fields so that this is backward compatible.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1761) FairScheduler should allow
separate configuration of node and rack locality wait time
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866700#action_12866700 ]
Scott Chen commented on MAPREDUCE-1761:
---------------------------------------
Thanks, Dhruba and Matei!
> FairScheduler should allow separate configuration of node and rack locality wait time
> -------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1761
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1761
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1761-v1.1.txt, MAPREDUCE-1761-v1.2.txt, MAPREDUCE-1761.txt
>
>
> It would be nice that we can separately assign rack locality wait time.
> In our use case, we would set node locality wait to zero and wait only rack locality.
> I propose that we add two parameters
> mapred.fairscheduler.locality.delay.nodetorack
> mapred.fairscheduler.locality.delay.racktoany
> This allows specifying the wait time on each stage.
> And we can use
> mapred.fairscheduler.locality.delay
> as the default value of the above fields so that this is backward compatible.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1761) FairScheduler should allow
separate configuration of node and rack locality wait time
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1761:
----------------------------------
Status: Patch Available (was: Open)
> FairScheduler should allow separate configuration of node and rack locality wait time
> -------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1761
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1761
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1761-v1.1.txt, MAPREDUCE-1761-v1.2.txt, MAPREDUCE-1761.txt
>
>
> It would be nice that we can separately assign rack locality wait time.
> In our use case, we would set node locality wait to zero and wait only rack locality.
> I propose that we add two parameters
> mapred.fairscheduler.locality.delay.nodetorack
> mapred.fairscheduler.locality.delay.racktoany
> This allows specifying the wait time on each stage.
> And we can use
> mapred.fairscheduler.locality.delay
> as the default value of the above fields so that this is backward compatible.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1761) FairScheduler should allow
separate configuration of node and rack locality wait time
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866360#action_12866360 ]
dhruba borthakur commented on MAPREDUCE-1761:
---------------------------------------------
Matie's suggestion looks good to me as well.
> FairScheduler should allow separate configuration of node and rack locality wait time
> -------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1761
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1761
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1761-v1.1.txt, MAPREDUCE-1761.txt
>
>
> It would be nice that we can separately assign rack locality wait time.
> In our use case, we would set node locality wait to zero and wait only rack locality.
> I propose that we add two parameters
> mapred.fairscheduler.locality.delay.nodetorack
> mapred.fairscheduler.locality.delay.racktoany
> This allows specifying the wait time on each stage.
> And we can use
> mapred.fairscheduler.locality.delay
> as the default value of the above fields so that this is backward compatible.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1761) FairScheduler should allow
separate configuration of node and rack locality wait time
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1761:
----------------------------------
Status: Patch Available (was: Open)
> FairScheduler should allow separate configuration of node and rack locality wait time
> -------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1761
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1761
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1761-v1.1.txt, MAPREDUCE-1761.txt
>
>
> It would be nice that we can separately assign rack locality wait time.
> In our use case, we would set node locality wait to zero and wait only rack locality.
> I propose that we add two parameters
> mapred.fairscheduler.locality.delay.nodetorack
> mapred.fairscheduler.locality.delay.racktoany
> This allows specifying the wait time on each stage.
> And we can use
> mapred.fairscheduler.locality.delay
> as the default value of the above fields so that this is backward compatible.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1761) FairScheduler should allow
separate configuration of node and rack locality wait time
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1761:
----------------------------------
Status: Open (was: Patch Available)
> FairScheduler should allow separate configuration of node and rack locality wait time
> -------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1761
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1761
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1761-v1.1.txt, MAPREDUCE-1761.txt
>
>
> It would be nice that we can separately assign rack locality wait time.
> In our use case, we would set node locality wait to zero and wait only rack locality.
> I propose that we add two parameters
> mapred.fairscheduler.locality.delay.nodetorack
> mapred.fairscheduler.locality.delay.racktoany
> This allows specifying the wait time on each stage.
> And we can use
> mapred.fairscheduler.locality.delay
> as the default value of the above fields so that this is backward compatible.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1761) FairScheduler should allow
separate configuration of node and rack locality wait time
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866333#action_12866333 ]
dhruba borthakur commented on MAPREDUCE-1761:
---------------------------------------------
+1 Code looks good.
> FairScheduler should allow separate configuration of node and rack locality wait time
> -------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1761
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1761
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1761-v1.1.txt, MAPREDUCE-1761.txt
>
>
> It would be nice that we can separately assign rack locality wait time.
> In our use case, we would set node locality wait to zero and wait only rack locality.
> I propose that we add two parameters
> mapred.fairscheduler.locality.delay.nodetorack
> mapred.fairscheduler.locality.delay.racktoany
> This allows specifying the wait time on each stage.
> And we can use
> mapred.fairscheduler.locality.delay
> as the default value of the above fields so that this is backward compatible.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1761) FairScheduler should allow
separate configuration of node and rack locality wait time
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Scott Chen updated MAPREDUCE-1761:
----------------------------------
Attachment: MAPREDUCE-1761-v1.2.txt
> FairScheduler should allow separate configuration of node and rack locality wait time
> -------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1761
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1761
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1761-v1.1.txt, MAPREDUCE-1761-v1.2.txt, MAPREDUCE-1761.txt
>
>
> It would be nice that we can separately assign rack locality wait time.
> In our use case, we would set node locality wait to zero and wait only rack locality.
> I propose that we add two parameters
> mapred.fairscheduler.locality.delay.nodetorack
> mapred.fairscheduler.locality.delay.racktoany
> This allows specifying the wait time on each stage.
> And we can use
> mapred.fairscheduler.locality.delay
> as the default value of the above fields so that this is backward compatible.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1761) FairScheduler should allow
separate configuration of node and rack locality wait time
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866420#action_12866420 ]
Hadoop QA commented on MAPREDUCE-1761:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12444243/MAPREDUCE-1761-v1.1.txt
against trunk revision 943039.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/181/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/181/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/181/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/181/console
This message is automatically generated.
> FairScheduler should allow separate configuration of node and rack locality wait time
> -------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1761
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1761
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1761-v1.1.txt, MAPREDUCE-1761-v1.2.txt, MAPREDUCE-1761.txt
>
>
> It would be nice that we can separately assign rack locality wait time.
> In our use case, we would set node locality wait to zero and wait only rack locality.
> I propose that we add two parameters
> mapred.fairscheduler.locality.delay.nodetorack
> mapred.fairscheduler.locality.delay.racktoany
> This allows specifying the wait time on each stage.
> And we can use
> mapred.fairscheduler.locality.delay
> as the default value of the above fields so that this is backward compatible.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.