[jira] Commented: (HADOOP-1651) Some improvements in progress reporting

["Vivek Ratan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12515338 ] 

Vivek Ratan commented on HADOOP-1651:
-------------------------------------

IMHO, some of these suggestions are appropriate, while others may not be worth the effort. 

Since Tasks and a TastTracker (TT) run on the same machine, we don't care about network issues. All we need to do is detect if the TaskTracker is dead (or hung) or if it is running, but really really slowly. In the former case, you want to kill the task. In the latter case, you may want to wait a bit to see if the TT speeds up to normal. 

The current logic of sleeping for a bit and trying 3 times before giving up, is good enough to detect whether the TT is dead or just sluggish. If you're worried that we decide too quickly that the TT is dead (when it is not), then we should go ahead and have the progress reporting thread sleep for 3 secs instead of 1 (your first suggestion). If after 3 tries, we still fail connecting to the TT, we should die (we've given the TT 9 seconds; that seems like plenty of time). 

So it makes sense for #1 (set interval to 3 secs) and #3 (always sleep for a bit before trying again). #2 is a valid bug and needs to be fixed. #4 also makes sense, and the idle timeout should at least be a little more than the sleep interval. However, I don't think we need to implement the suggestions in your second comment. Keep the code simple. There's no difference between pinging the TT and sending it a progress report, in order to detect whether the TT is dead. If you're worried that we determine too quickly that the TT is dead, increase the interval from 3 secs to more, or retry more than 3 times, or determine why the TT is so loaded. I would recommend against adding additional code which will essentially do the same thing but in a more complicated manner. 

As far as details go, I'd modify the code of startCommunicationThread() (in Task.java) to be as follows: 

while (1) {
  sleep();
  try {
    call sendProgress() or ping();
  }
  catch() {
    set progressFlag to whatever it was at beginning of try block;
    decrement retry count; quit if we've retried enough
  }
}

Only changes are: 
- move sleep() to the beginning so we always sleep irrespective of whether there was a communication exception or not (your suggestion #3)
- in the catch block, set ProgressFlag to what it was (to fix suggestion #2)

Just my 2 bits. 



> Some improvements in progress reporting
> ---------------------------------------
>
>                 Key: HADOOP-1651
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1651
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.15.0
>
>         Attachments: 1651.patch
>
>
> Some improvements that can be done:
> 1) Progress reporting interval can be made slightly large. It is currently 1 second. Propose to make it 3 seconds to reduce the load on the TaskTracker.
> 2) Progress reports can potentially be missed. In the loop, if the first attempt at reporting a progress doesn't go through, it is not retried. The next communication will be a 'ping'. 
> 3) If there is an exception while reporting progress or doing ping, the client should sleep for sometime before retrying.
> 4) The TaskUmbilicalProtocol client can always stay connected to the server. Currently, the default idle timeout on the IPC client is set to 1000 msec (this means that the client will disconnect if the connection has been idle for 1000 msec). This might lead to unnecessary tearing-down/setting-up of connections for the TaskUmbilicalProtocol and can be avoided by having a high idle timeout for this protocol. The idea behind having the idle timeout was to not hold on to server connections unnecessarily and hence be more scalable when there are 1000s of clients, especially applicable to those protocols involving the JT and the NameNode.  We don't run into scalability issues with TaskUmbilical protocol since it is limited to a few Tasks and the corresponding TaskTracker.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.