[jira] [Updated] (MAPREDUCE-3787) [Gridmix] Improve STRESS mode

["Amar Kamat (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAPREDUCE-3787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated MAPREDUCE-3787:
----------------------------------

    Attachment: MAPREDUCE-3787-v1.9.patch

Attaching a patch that improves the Gridmix STRESS logic. Changes are as follows:
# StressJobFactory: 
   ## Update and submission happens in one thread. Earlier, the job submission and status-polling/load-calculation happened in separate thread. Slower status polling resulted in throttling of job submission.
   ## No RPC calls are made in stress mode. The Stress mode relies on the status cached by JobMonitor. 
   ## Load calculation logic optimized to return early if the load is considerably lower or considerably higher. 
      ### If the total number of tasks submitted (without considering their progress) is lower than the max permissible limit, the cluster is considered underloaded. 
      ### If at anytime the actual load (considering the task progress) is higher than the permissible limit, the cluster is considered overloaded without iterating over the remaining jobs.
   ## Maintains a list of blacklisted jobs that the will be ignored.
# JobMonitor:
   ## This is now multi-threaded. Total number of threads can be specified using the 'gridmix.polling.threads' config key.
   ## The sleep interval is now made configurable. The sleep time can be set using 'gridmix.polling.delay'.
   ## Invokes statistics for lost jobs. Not doing this results into missing status-update events.
# Statistics:
   ## JobStats now caches job-status along with a) Running job handle, b) total number of map tasks and c) total number of reduce tasks. Other modules should rely on these cached status objects instead of polling for job status individually.
   ## Statistics maintains a count of total number of submitted maps and reduces.
# Execution Summarizer:
   ## Now reports "Lost Jobs". These are the jobs for which the status is unavailable. 
# JobFactory:
   ## Ignores reduce only jobs. Currently, Gridmix is incapable of handling reduce only jobs.

Modified {{TestGridmixSummary}} to test summary on lost jobs. Added {{TestGridmixStatistics}} to test the {{Statistics.java}} changes.

With this patch, following things are timed and reported:
1. Time taken to submit a job
2. Time taken to build job splits
3. Time taken to get the job's status

test-patch and ant-tests passed.
                
> [Gridmix] Improve STRESS mode
> -----------------------------
>
>                 Key: MAPREDUCE-3787
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3787
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/gridmix
>    Affects Versions: 0.24.0
>            Reporter: Amar Kamat
>            Assignee: Amar Kamat
>              Labels: gridmix, stress
>             Fix For: 0.23.1
>
>         Attachments: MAPREDUCE-3787-v1.9.patch
>
>
> Gridmix STRESS mode can be improved as follows:
> 1. The sleep time in JobMonitor can be reduced and/or made configurable
> 2. Map and reduce load calculation in StressJobFactory can be done in one loop
> 3. Updating the overload status from the job submitter thread (inline)
> 4. Optimizations to avoid un-necessary progress check (which inturn would result into delay)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira