[jira] Updated: (HADOOP-3326) ReduceTask should not sleep for 200 ms while waiting for merge to finish

["Sharad Agarwal (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal updated HADOOP-3326:
-----------------------------------

    Attachment: 3326_1.patch

Attaching a patch. It does following:-
1. LocalFSMerger and InMemFSMergeThread threads are spawned only once in fetchOutputs() method. The thread gets notified when something is ready for merge. The merge happens when thresholds are met.

2. Instead of sleeping (in loop) to wait for finishing the merger threads, now it does join. 

3. The in memory merge code was duplicate at 2 places, now the common code has been moved to doInMemMerge() method.

> ReduceTask should not sleep for 200 ms while waiting for merge to finish
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-3326
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3326
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Owen O'Malley
>            Assignee: Sharad Agarwal
>         Attachments: 3326_1.patch
>
>
> Currently the merge code in Reduce task does:
> {code}
>             // Wait for the on-disk merge to complete
>             while (localFSMergeInProgress) {
>               Thread.sleep(200);
>             }
>             
>             //wait for an ongoing merge (if it is in flight) to complete
>             while (mergeInProgress) {
>               Thread.sleep(200);
>             }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.