[jira] [Commented] (MAPREDUCE-3925) [Gridmix] Gridmix stress mode should be queue aware

["Amar Kamat (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217043#comment-13217043 ] 

Amar Kamat commented on MAPREDUCE-3925:
---------------------------------------

One way of doing this would be to pre-process the trace and break it down to 'n' sub-traces i.e 1 trace per queue. Similar to how the stress mode stresses the entire cluster, Gridmix can now run 'n' stress threads (one thread per queue) such that each of them will consume a sub-trace and stress the corresponding queue. This way, if Gridmix can make sure that each of the queues is sufficiently loaded, then we can safely assume that the cluster is sufficiently busy.

Note-1: If the queue-level sub-trace is not capable of stressing the queue (due to small jobs or few jobs), then other stress threads should make sure they overload their queues sufficiently to use up the free slots.
Note-2: For single queue, Gridmix should default to the current behavior.

Thoughts?
                
> [Gridmix] Gridmix stress mode should be queue aware
> ---------------------------------------------------
>
>                 Key: MAPREDUCE-3925
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3925
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/gridmix
>    Affects Versions: 0.24.0
>            Reporter: Amar Kamat
>              Labels: gridmix, multi-queue, stress
>             Fix For: 0.24.0
>
>
> Currently, the Gridmix stress mode submits jobs in the same order as seen in the trace. When Gridmix is configured to run with multiple queues, the stress mode might end up queuing lot of jobs in a single queue without really stressing the entire cluster. The goal is to make sure that each queue is loaded thus keeping the entire cluster busy. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira