You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Shrinivas Joshi (JIRA)" <ji...@apache.org> on 2012/06/28 21:01:46 UTC
[jira] [Created] (MAPREDUCE-4381) Make PROGRESS_INTERVAL of
org.apache.hadoop.mapred.Task a tunable
Shrinivas Joshi created MAPREDUCE-4381:
------------------------------------------
Summary: Make PROGRESS_INTERVAL of org.apache.hadoop.mapred.Task a tunable
Key: MAPREDUCE-4381
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4381
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: task, tasktracker
Reporter: Shrinivas Joshi
Priority: Minor
Currently PROGRESS_INTERVAL is a hard-coded value and is set to 3000 msec. We tried making it a tunable and experimented with different values. In some cases setting it to a smaller value like 1000 msec helps significantly improve performance of short running jobs such as piEstimator. This is because the task threads do not end up blocking for as many as 3 seconds for their last progress update event. We also noticed close to 14% improvement on Mahout KMeans iteration jobs which take more than 5 minutes on the test cluster that we are using. Please let me know if this seems to be a good idea. I have an initial patch that I have attached here. This is based on branch-1 tree. It may need some rework on MRv2 based branches I think. Also note that I have not changed the variable naming style for PROGRESS_INTERVAL even though it is not a public static final anymore. I can revise the patch if there are no objections to this idea.
Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4381) Make PROGRESS_INTERVAL of
org.apache.hadoop.mapred.Task a tunable
Posted by "Shrinivas Joshi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405403#comment-13405403 ]
Shrinivas Joshi commented on MAPREDUCE-4381:
--------------------------------------------
Thanks for the code review.
I agree about the possibility of creating scalability issues. Setting progress interval to a very small value may lead to excessive status update events. Can we address this by setting a lower bound requirement on the value of progress interval that the user can set? If so, how does 500 milliseconds sound as the lower bound?
I will address your comments in the 1st and 2nd bullet above in the revised version of this patch along with other changes.
As you may have seen I have included a short description of the new property in src/mapred/mapred-default.xml file. Is there any other more appropriate file/location where this needs to be documented?
Since this patch only makes progress_interval a tunable, would it suffice to test whether the value returned by JobConf matches the one set in mapred-site.xml?
> Make PROGRESS_INTERVAL of org.apache.hadoop.mapred.Task a tunable
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-4381
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4381
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task, tasktracker
> Reporter: Shrinivas Joshi
> Priority: Minor
> Attachments: progress_interval.patch
>
>
> Currently PROGRESS_INTERVAL is a hard-coded value and is set to 3000 msec. We tried making it a tunable and experimented with different values. In some cases setting it to a smaller value like 1000 msec helps significantly improve performance of short running jobs such as piEstimator. This is because the task threads do not end up blocking for as many as 3 seconds for their last progress update event. We also noticed close to 14% improvement on Mahout KMeans iteration jobs which take more than 5 minutes on the test cluster that we are using. Please let me know if this seems to be a good idea. I have an initial patch that I have attached here. This is based on branch-1 tree. It may need some rework on MRv2 based branches I think. Also note that I have not changed the variable naming style for PROGRESS_INTERVAL even though it is not a public static final anymore. I can revise the patch if there are no objections to this idea.
> Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4381) Make PROGRESS_INTERVAL of
org.apache.hadoop.mapred.Task a tunable
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408488#comment-13408488 ]
Luke Lu commented on MAPREDUCE-4381:
------------------------------------
MAPREDUCE-4400 makes the configuration unnecessary. I agree with Steve that exposing this config can cause more issues than it can solve.
> Make PROGRESS_INTERVAL of org.apache.hadoop.mapred.Task a tunable
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-4381
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4381
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task, tasktracker
> Reporter: Shrinivas Joshi
> Priority: Minor
> Attachments: MAPREDUCE-4381-branch-1.patch, progress_interval.patch
>
>
> Currently PROGRESS_INTERVAL is a hard-coded value and is set to 3000 msec. We tried making it a tunable and experimented with different values. In some cases setting it to a smaller value like 1000 msec helps significantly improve performance of short running jobs such as piEstimator. This is because the task threads do not end up blocking for as many as 3 seconds for their last progress update event. We also noticed close to 14% improvement on Mahout KMeans iteration jobs which take more than 5 minutes on the test cluster that we are using. Please let me know if this seems to be a good idea. I have an initial patch that I have attached here. This is based on branch-1 tree. It may need some rework on MRv2 based branches I think. Also note that I have not changed the variable naming style for PROGRESS_INTERVAL even though it is not a public static final anymore. I can revise the patch if there are no objections to this idea.
> Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4381) Make PROGRESS_INTERVAL of
org.apache.hadoop.mapred.Task a tunable
Posted by "Shrinivas Joshi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shrinivas Joshi updated MAPREDUCE-4381:
---------------------------------------
Attachment: MAPREDUCE-4381-branch-1.patch
> Make PROGRESS_INTERVAL of org.apache.hadoop.mapred.Task a tunable
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-4381
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4381
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task, tasktracker
> Reporter: Shrinivas Joshi
> Priority: Minor
> Attachments: MAPREDUCE-4381-branch-1.patch, progress_interval.patch
>
>
> Currently PROGRESS_INTERVAL is a hard-coded value and is set to 3000 msec. We tried making it a tunable and experimented with different values. In some cases setting it to a smaller value like 1000 msec helps significantly improve performance of short running jobs such as piEstimator. This is because the task threads do not end up blocking for as many as 3 seconds for their last progress update event. We also noticed close to 14% improvement on Mahout KMeans iteration jobs which take more than 5 minutes on the test cluster that we are using. Please let me know if this seems to be a good idea. I have an initial patch that I have attached here. This is based on branch-1 tree. It may need some rework on MRv2 based branches I think. Also note that I have not changed the variable naming style for PROGRESS_INTERVAL even though it is not a public static final anymore. I can revise the patch if there are no objections to this idea.
> Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4381) Make PROGRESS_INTERVAL of
org.apache.hadoop.mapred.Task a tunable
Posted by "Shrinivas Joshi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shrinivas Joshi updated MAPREDUCE-4381:
---------------------------------------
Attachment: progress_interval.patch
> Make PROGRESS_INTERVAL of org.apache.hadoop.mapred.Task a tunable
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-4381
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4381
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task, tasktracker
> Reporter: Shrinivas Joshi
> Priority: Minor
> Attachments: progress_interval.patch
>
>
> Currently PROGRESS_INTERVAL is a hard-coded value and is set to 3000 msec. We tried making it a tunable and experimented with different values. In some cases setting it to a smaller value like 1000 msec helps significantly improve performance of short running jobs such as piEstimator. This is because the task threads do not end up blocking for as many as 3 seconds for their last progress update event. We also noticed close to 14% improvement on Mahout KMeans iteration jobs which take more than 5 minutes on the test cluster that we are using. Please let me know if this seems to be a good idea. I have an initial patch that I have attached here. This is based on branch-1 tree. It may need some rework on MRv2 based branches I think. Also note that I have not changed the variable naming style for PROGRESS_INTERVAL even though it is not a public static final anymore. I can revise the patch if there are no objections to this idea.
> Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4381) Make PROGRESS_INTERVAL of
org.apache.hadoop.mapred.Task a tunable
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404425#comment-13404425 ]
Steve Loughran commented on MAPREDUCE-4381:
-------------------------------------------
I can see the value in this, though I worry that per-job tuning may cause some people to submit tasks that trigger scalability issues -someone else will have to comment on that.
Independent of that,
#yes, the value field name should be switched to the standard form of a variable; being private there's no compatibility problems.
#the parameter name should be extracted into a static final field of Task, along with the default value, for ease of setup from code.
#it needs to be documented somewhere
#How do you think this should be tested?
> Make PROGRESS_INTERVAL of org.apache.hadoop.mapred.Task a tunable
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-4381
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4381
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task, tasktracker
> Reporter: Shrinivas Joshi
> Priority: Minor
> Attachments: progress_interval.patch
>
>
> Currently PROGRESS_INTERVAL is a hard-coded value and is set to 3000 msec. We tried making it a tunable and experimented with different values. In some cases setting it to a smaller value like 1000 msec helps significantly improve performance of short running jobs such as piEstimator. This is because the task threads do not end up blocking for as many as 3 seconds for their last progress update event. We also noticed close to 14% improvement on Mahout KMeans iteration jobs which take more than 5 minutes on the test cluster that we are using. Please let me know if this seems to be a good idea. I have an initial patch that I have attached here. This is based on branch-1 tree. It may need some rework on MRv2 based branches I think. Also note that I have not changed the variable naming style for PROGRESS_INTERVAL even though it is not a public static final anymore. I can revise the patch if there are no objections to this idea.
> Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4381) Make PROGRESS_INTERVAL of
org.apache.hadoop.mapred.Task a tunable
Posted by "Shrinivas Joshi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13407480#comment-13407480 ]
Shrinivas Joshi commented on MAPREDUCE-4381:
--------------------------------------------
I have attached a revised version of this patch which addresses some of the code review comments from Steve. Specifically changes in this version are:
* Renamed patch file name to match official contribution patch naming conventions
* Used appropriate naming style for PROGRESS_INTERVAL variable
* Used more appropriate name for the progress interval property
* The new property is now read using existing JobConf.getInt method instead of introducing a new query method
I have not thought this through completely, but would it be possible to implement a test case similar to TestCombineOutputCollector for this feature?
> Make PROGRESS_INTERVAL of org.apache.hadoop.mapred.Task a tunable
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-4381
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4381
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task, tasktracker
> Reporter: Shrinivas Joshi
> Priority: Minor
> Attachments: MAPREDUCE-4381-branch-1.patch, progress_interval.patch
>
>
> Currently PROGRESS_INTERVAL is a hard-coded value and is set to 3000 msec. We tried making it a tunable and experimented with different values. In some cases setting it to a smaller value like 1000 msec helps significantly improve performance of short running jobs such as piEstimator. This is because the task threads do not end up blocking for as many as 3 seconds for their last progress update event. We also noticed close to 14% improvement on Mahout KMeans iteration jobs which take more than 5 minutes on the test cluster that we are using. Please let me know if this seems to be a good idea. I have an initial patch that I have attached here. This is based on branch-1 tree. It may need some rework on MRv2 based branches I think. Also note that I have not changed the variable naming style for PROGRESS_INTERVAL even though it is not a public static final anymore. I can revise the patch if there are no objections to this idea.
> Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira