[jira] Updated: (HADOOP-248) locating map outputs via random probing is inefficient

["Devaraj Das (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-248:
-------------------------------

    Attachment: 248-initial7.patch

The patch does the following:
1) Modifies the protocol version for InterTrackerProtocol since there is a major change there in the way map output is fetched. The method locateMapOutputs has been removed.
2) The getTaskCompletion method in both InterTrackerProtocol and JobSubmission has been changed to take an extra argument - the max no. of events we want to fetch from the JobTracker.
3) Two more fields are added in TaskCompletionEvents - the real-ID portion of the taskId string (for e.g., if the taskId is task_0001_m_000003_0, the real-ID is 3 within the job), and another boolean field to indicate whether the task is a map or not. By the way, these could have done on the Reduce side also by parsing the taskId string, but I think this is a more general way of doing it and it also is in line with the thought of having "ID objects" in the future.
4) Only 10 events are fetched at a time by the JobClient
5) A new value OBSELETE has been added to TaskCompletionEvent.Status to signify lost tasks (which were earlier reported as SUCCEEDED). For this, whenever a FAILED/KILLED TaskStatus is got, it is checked whether a SUCCEEDED was earlier reported for the same taskId, and if so that event is marked as OBSELETE.
6) The number of events probed by the ReduceTaskRunner at any time is equal to max(5*numCopiers, 50).

Feedback appreciated..

> locating map outputs via random probing is inefficient
> ------------------------------------------------------
>
>                 Key: HADOOP-248
>                 URL: https://issues.apache.org/jira/browse/HADOOP-248
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.2.1
>            Reporter: Owen O'Malley
>         Assigned To: Devaraj Das
>         Attachments: 248-initial7.patch
>
>
> Currently the ReduceTaskRunner polls the JobTracker for a random list of map tasks asking for their output locations. It would be better if the JobTracker kept an ordered log and the interface was changed to:
> class MapLocationResults {
>    public int getTimestamp();
>    public MapOutputLocation[] getLocations();
> }
> interface InterTrackerProtocol {
>   ...
>   MapLocationResults locateMapOutputs(int prevTimestamp);
> } 
> with the intention that each time a ReduceTaskRunner calls locateMapOutputs, it passes back the "timestamp" that it got from the previous result. That way, reduces can easily find the new MapOutputs. This should help the "ramp up" when the maps first start finishing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.