You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Hong Tang (JIRA)" <ji...@apache.org> on 2010/07/13 02:11:51 UTC
[jira] Created: (MAPREDUCE-1936) [gridmix3] Make Gridmix3 more
customizable.
[gridmix3] Make Gridmix3 more customizable.
-------------------------------------------
Key: MAPREDUCE-1936
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1936
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: contrib/gridmix
Reporter: Hong Tang
I'd like to make gridmix3 more customizable. Specifically, the proposed customizations include:
- add (random) location information for each sleep map task.
- make the parameters used in stress submission load throttling configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1936) [gridmix3] Make Gridmix3 more
customizable.
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890823#action_12890823 ]
Chris Douglas commented on MAPREDUCE-1936:
------------------------------------------
{noformat}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 5 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{noformat}
> [gridmix3] Make Gridmix3 more customizable.
> -------------------------------------------
>
> Key: MAPREDUCE-1936
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1936
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/gridmix
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mr-1936-20100715.patch, mr-1936-20100720.patch, mr-1936-yhadoop-20.1xx.patch
>
>
> I'd like to make gridmix3 more customizable. Specifically, the proposed customizations include:
> - add (random) location information for each sleep map task.
> - make the parameters used in stress submission load throttling configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1936) [gridmix3] Make Gridmix3 more
customizable.
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-1936:
---------------------------------
Attachment: mr-1936-20100715.patch
> [gridmix3] Make Gridmix3 more customizable.
> -------------------------------------------
>
> Key: MAPREDUCE-1936
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1936
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/gridmix
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mr-1936-20100715.patch, mr-1936-yhadoop-20.1xx.patch
>
>
> I'd like to make gridmix3 more customizable. Specifically, the proposed customizations include:
> - add (random) location information for each sleep map task.
> - make the parameters used in stress submission load throttling configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1936) [gridmix3] Make Gridmix3 more
customizable.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926040#action_12926040 ]
Hudson commented on MAPREDUCE-1936:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/])
> [gridmix3] Make Gridmix3 more customizable.
> -------------------------------------------
>
> Key: MAPREDUCE-1936
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1936
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/gridmix
> Reporter: Hong Tang
> Assignee: Hong Tang
> Fix For: 0.22.0
>
> Attachments: mr-1936-20100715.patch, mr-1936-20100720.patch, mr-1936-delta-20.1xx.patch, mr-1936-yhadoop-20.1xx.patch
>
>
> I'd like to make gridmix3 more customizable. Specifically, the proposed customizations include:
> - add (random) location information for each sleep map task.
> - make the parameters used in stress submission load throttling configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1936) [gridmix3] Make Gridmix3 more
customizable.
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1936:
-------------------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Fix Version/s: 0.22.0
Resolution: Fixed
+1
I committed this. Thanks, Hong!
> [gridmix3] Make Gridmix3 more customizable.
> -------------------------------------------
>
> Key: MAPREDUCE-1936
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1936
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/gridmix
> Reporter: Hong Tang
> Assignee: Hong Tang
> Fix For: 0.22.0
>
> Attachments: mr-1936-20100715.patch, mr-1936-20100720.patch, mr-1936-yhadoop-20.1xx.patch
>
>
> I'd like to make gridmix3 more customizable. Specifically, the proposed customizations include:
> - add (random) location information for each sleep map task.
> - make the parameters used in stress submission load throttling configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1936) [gridmix3] Make Gridmix3 more
customizable.
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-1936:
---------------------------------
Attachment: mr-1936-yhadoop-20.1xx.patch
Patch for yhadoop-20.1xx branch. Not to be committed.
> [gridmix3] Make Gridmix3 more customizable.
> -------------------------------------------
>
> Key: MAPREDUCE-1936
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1936
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/gridmix
> Reporter: Hong Tang
> Attachments: mr-1936-yhadoop-20.1xx.patch
>
>
> I'd like to make gridmix3 more customizable. Specifically, the proposed customizations include:
> - add (random) location information for each sleep map task.
> - make the parameters used in stress submission load throttling configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1936) [gridmix3] Make Gridmix3 more
customizable.
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-1936:
---------------------------------
Status: Patch Available (was: Open)
New patch that addresses Chris's comments. The usage output now looks like the following:
{noformat}
Usage: gridmix [-generate <MiB>] [-users URI] [-Dname=value ...] <iopath> <trace>
e.g. gridmix -generate 100m foo -
Configuration parameters:
General parameters:
gridmix.output.directory : Output directory
gridmix.client.submit.threads : Submitting threads
gridmix.client.pending.queue.depth : Queued job desc
gridmix.user.resolve.class : User resolution class
gridmix.job.type : Job types (LOADJOB|SLEEPJOB)
Parameters related to job submission:
gridmix.job-submission.default-queue : Default queue
gridmix.job-submission.use-queue-in-trace : Enable/disable using queues in trace
gridmix.job-submission.policy : Job submission policy (REPLAY|STRESS|SERIAL)
Parameters specific for LOADJOB:
gridmix.key.fraction : Key fraction of rec
Parameters specific for SLEEPJOB:
gridmix.sleep.maptask-only : Whether to ignore reduce tasks
gridmix.sleep.fake-locations : Number of fake locations for map tasks
gridmix.sleep.max-map-time : Maximum map task runtime in mili-sec
gridmix.sleep.max-reduce-time : Maximum reduce task runtime in mili-sec (merge+reduce)
Parameters specific for STRESS submission throttling policy:
gridmix.throttle.jobs-to-tracker-ratio : jobs vs task-tracker ratio
gridmix.throttle.maps.task-to-slot-ratio : maps vs map-slot ratio
gridmix.throttle.reduces.task-to-slot-ratio : reduces vs reduce-slot ratio
gridmix.throttle.maps.max-slot-share-per-job : map-slot share per job
gridmix.throttle.reducess.max-slot-share-per-job : reduce-slot share per job
{noformat}
> [gridmix3] Make Gridmix3 more customizable.
> -------------------------------------------
>
> Key: MAPREDUCE-1936
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1936
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/gridmix
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mr-1936-20100715.patch, mr-1936-20100720.patch, mr-1936-yhadoop-20.1xx.patch
>
>
> I'd like to make gridmix3 more customizable. Specifically, the proposed customizations include:
> - add (random) location information for each sleep map task.
> - make the parameters used in stress submission load throttling configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1936) [gridmix3] Make Gridmix3 more
customizable.
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-1936:
---------------------------------
Attachment: mr-1936-20100720.patch
> [gridmix3] Make Gridmix3 more customizable.
> -------------------------------------------
>
> Key: MAPREDUCE-1936
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1936
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/gridmix
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mr-1936-20100715.patch, mr-1936-20100720.patch, mr-1936-yhadoop-20.1xx.patch
>
>
> I'd like to make gridmix3 more customizable. Specifically, the proposed customizations include:
> - add (random) location information for each sleep map task.
> - make the parameters used in stress submission load throttling configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1936) [gridmix3] Make Gridmix3 more
customizable.
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-1936:
---------------------------------
Attachment: mr-1936-delta-20.1xx.patch
Delta patch incorporating changes reflecting Chris's comments.
> [gridmix3] Make Gridmix3 more customizable.
> -------------------------------------------
>
> Key: MAPREDUCE-1936
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1936
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/gridmix
> Reporter: Hong Tang
> Assignee: Hong Tang
> Fix For: 0.22.0
>
> Attachments: mr-1936-20100715.patch, mr-1936-20100720.patch, mr-1936-delta-20.1xx.patch, mr-1936-yhadoop-20.1xx.patch
>
>
> I'd like to make gridmix3 more customizable. Specifically, the proposed customizations include:
> - add (random) location information for each sleep map task.
> - make the parameters used in stress submission load throttling configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1936) [gridmix3] Make Gridmix3 more
customizable.
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-1936:
---------------------------------
Status: Patch Available (was: Open)
> [gridmix3] Make Gridmix3 more customizable.
> -------------------------------------------
>
> Key: MAPREDUCE-1936
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1936
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/gridmix
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mr-1936-20100715.patch, mr-1936-yhadoop-20.1xx.patch
>
>
> I'd like to make gridmix3 more customizable. Specifically, the proposed customizations include:
> - add (random) location information for each sleep map task.
> - make the parameters used in stress submission load throttling configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1936) [gridmix3] Make Gridmix3 more
customizable.
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890501#action_12890501 ]
Chris Douglas commented on MAPREDUCE-1936:
------------------------------------------
Small nits:
* Now that the {{\*IndexMapper}} classes are in {{RandomAlgorithms}}, they can be removed from {{FilePool}}
* In the usage, it could be clearer which parameters belong to the submission mode and job type
* {{mapMaxSleepTime}} and {{reduceMaxSleepTime}} can be final
Other than these, this looks good
> [gridmix3] Make Gridmix3 more customizable.
> -------------------------------------------
>
> Key: MAPREDUCE-1936
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1936
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/gridmix
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mr-1936-20100715.patch, mr-1936-yhadoop-20.1xx.patch
>
>
> I'd like to make gridmix3 more customizable. Specifically, the proposed customizations include:
> - add (random) location information for each sleep map task.
> - make the parameters used in stress submission load throttling configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1936) [gridmix3] Make Gridmix3 more
customizable.
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889005#action_12889005 ]
Hong Tang commented on MAPREDUCE-1936:
--------------------------------------
test-patch passes on my local machine:
{noformat}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 5 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{noformat}
> [gridmix3] Make Gridmix3 more customizable.
> -------------------------------------------
>
> Key: MAPREDUCE-1936
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1936
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/gridmix
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mr-1936-20100715.patch, mr-1936-yhadoop-20.1xx.patch
>
>
> I'd like to make gridmix3 more customizable. Specifically, the proposed customizations include:
> - add (random) location information for each sleep map task.
> - make the parameters used in stress submission load throttling configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1936) [gridmix3] Make Gridmix3 more
customizable.
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-1936:
---------------------------------
Status: Open (was: Patch Available)
> [gridmix3] Make Gridmix3 more customizable.
> -------------------------------------------
>
> Key: MAPREDUCE-1936
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1936
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/gridmix
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mr-1936-20100715.patch, mr-1936-20100720.patch, mr-1936-yhadoop-20.1xx.patch
>
>
> I'd like to make gridmix3 more customizable. Specifically, the proposed customizations include:
> - add (random) location information for each sleep map task.
> - make the parameters used in stress submission load throttling configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-1936) [gridmix3] Make Gridmix3 more
customizable.
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang reassigned MAPREDUCE-1936:
------------------------------------
Assignee: Hong Tang
> [gridmix3] Make Gridmix3 more customizable.
> -------------------------------------------
>
> Key: MAPREDUCE-1936
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1936
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/gridmix
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mr-1936-20100715.patch, mr-1936-yhadoop-20.1xx.patch
>
>
> I'd like to make gridmix3 more customizable. Specifically, the proposed customizations include:
> - add (random) location information for each sleep map task.
> - make the parameters used in stress submission load throttling configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1936) [gridmix3] Make Gridmix3 more
customizable.
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887609#action_12887609 ]
Hong Tang commented on MAPREDUCE-1936:
--------------------------------------
The patch also incorporates the changes in MAPREDUCE-1833.
> [gridmix3] Make Gridmix3 more customizable.
> -------------------------------------------
>
> Key: MAPREDUCE-1936
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1936
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/gridmix
> Reporter: Hong Tang
> Attachments: mr-1936-yhadoop-20.1xx.patch
>
>
> I'd like to make gridmix3 more customizable. Specifically, the proposed customizations include:
> - add (random) location information for each sleep map task.
> - make the parameters used in stress submission load throttling configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.