You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2009/10/19 19:59:59 UTC
[jira] Created: (MAPREDUCE-1120) JobClient poll intervals should be
job configurations, not cluster configurations
JobClient poll intervals should be job configurations, not cluster configurations
---------------------------------------------------------------------------------
Key: MAPREDUCE-1120
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1120
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 0.22.0
Reporter: Todd Lipcon
Priority: Minor
Job.waitForCompletion gets the poll interval from the Cluster object's configuration rather than its own Job configuration. This is counter-intuitive - Chris and I both made this same mistake working on MAPREDUCE-64, and Aaron agrees as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1120) JobClient poll intervals should be
job configurations, not cluster configurations
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-1120:
-----------------------------------
Attachment: mapreduce-1120.txt
Trivial patch. To test, I ran TestTaskFail which uses this functionality, and verified that it's still polling quickly instead of sleeping (that testcase was setting the config in both the job and the cluster)
> JobClient poll intervals should be job configurations, not cluster configurations
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1120
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1120
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Priority: Minor
> Attachments: mapreduce-1120.txt
>
>
> Job.waitForCompletion gets the poll interval from the Cluster object's configuration rather than its own Job configuration. This is counter-intuitive - Chris and I both made this same mistake working on MAPREDUCE-64, and Aaron agrees as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1120) JobClient poll intervals should
be job configurations, not cluster configurations
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841614#action_12841614 ]
Todd Lipcon commented on MAPREDUCE-1120:
----------------------------------------
bq. If polling intervals are job-level configuration parameters, Job. getCompletionPollInterval(conf) and Job.getProgressPollInterval(conf) should be not static methods and should not take configuration as the parameter. The methods should read the values from Job's conf directly.
OK. Do we need to maintain compatibility on these static functions, since they're a public API? (eg mark the static ones deprecated, then make non-static ones that forward to the static ones for now)
> JobClient poll intervals should be job configurations, not cluster configurations
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1120
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1120
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mapreduce-1120.txt
>
>
> Job.waitForCompletion gets the poll interval from the Cluster object's configuration rather than its own Job configuration. This is counter-intuitive - Chris and I both made this same mistake working on MAPREDUCE-64, and Aaron agrees as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1120) JobClient poll intervals should be
job configurations, not cluster configurations
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1120:
-------------------------------------
Fix Version/s: 0.21.0
Affects Version/s: (was: 0.22.0)
0.21.0
Status: Open (was: Patch Available)
bq. Do we need to maintain compatibility on these static functions, since they're a public API?
I'd say no. They were added in 0.21, which hasn't been released. It can be patched in both.
Maintaining the separation between "progress" and "completion" polling intervals offers very little in functionality. Any objections to collapsing these tunables into a single set method on the {{Job}} instance?
> JobClient poll intervals should be job configurations, not cluster configurations
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1120
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1120
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 0.21.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: mapreduce-1120.txt
>
>
> Job.waitForCompletion gets the poll interval from the Cluster object's configuration rather than its own Job configuration. This is counter-intuitive - Chris and I both made this same mistake working on MAPREDUCE-64, and Aaron agrees as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1120) JobClient poll intervals should
be job configurations, not cluster configurations
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775247#action_12775247 ]
Aaron Kimball commented on MAPREDUCE-1120:
------------------------------------------
Based on my understanding of the different conf objects, +1.
> JobClient poll intervals should be job configurations, not cluster configurations
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1120
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1120
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mapreduce-1120.txt
>
>
> Job.waitForCompletion gets the poll interval from the Cluster object's configuration rather than its own Job configuration. This is counter-intuitive - Chris and I both made this same mistake working on MAPREDUCE-64, and Aaron agrees as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1120) JobClient poll intervals should
be job configurations, not cluster configurations
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848514#action_12848514 ]
Todd Lipcon commented on MAPREDUCE-1120:
----------------------------------------
Seems fine by me, Chris. Thanks for taking a look.
> JobClient poll intervals should be job configurations, not cluster configurations
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1120
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1120
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 0.21.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: mapreduce-1120.txt
>
>
> Job.waitForCompletion gets the poll interval from the Cluster object's configuration rather than its own Job configuration. This is counter-intuitive - Chris and I both made this same mistake working on MAPREDUCE-64, and Aaron agrees as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1120) JobClient poll intervals should be
job configurations, not cluster configurations
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-1120:
-----------------------------------
Assignee: Todd Lipcon
Status: Patch Available (was: Open)
> JobClient poll intervals should be job configurations, not cluster configurations
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1120
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1120
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mapreduce-1120.txt
>
>
> Job.waitForCompletion gets the poll interval from the Cluster object's configuration rather than its own Job configuration. This is counter-intuitive - Chris and I both made this same mistake working on MAPREDUCE-64, and Aaron agrees as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1120) JobClient poll intervals should
be job configurations, not cluster configurations
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767616#action_12767616 ]
Hadoop QA commented on MAPREDUCE-1120:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12422585/mapreduce-1120.txt
against trunk revision 826767.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/81/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/81/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/81/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/81/console
This message is automatically generated.
> JobClient poll intervals should be job configurations, not cluster configurations
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1120
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1120
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mapreduce-1120.txt
>
>
> Job.waitForCompletion gets the poll interval from the Cluster object's configuration rather than its own Job configuration. This is counter-intuitive - Chris and I both made this same mistake working on MAPREDUCE-64, and Aaron agrees as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1120) JobClient poll intervals should
be job configurations, not cluster configurations
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840581#action_12840581 ]
Amareshwari Sriramadasu commented on MAPREDUCE-1120:
----------------------------------------------------
If polling intervals are job-level configuration parameters, Job. getCompletionPollInterval(conf) and Job.getProgressPollInterval(conf) should be not static methods and should not take configuration as the parameter. The methods should read the values from Job's conf directly.
Shall we add setters also in Job to be consistent with other getters and setters?
> JobClient poll intervals should be job configurations, not cluster configurations
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1120
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1120
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mapreduce-1120.txt
>
>
> Job.waitForCompletion gets the poll interval from the Cluster object's configuration rather than its own Job configuration. This is counter-intuitive - Chris and I both made this same mistake working on MAPREDUCE-64, and Aaron agrees as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.