[jira] [Created] (MAPREDUCE-6224) resolve the hosts in DNSToSwitchMapping before inter tracker server start to avoid IPC timeout in Task Tracker heartbeat

["zhihai xu (JIRA)" <ji...@apache.org>]

zhihai xu created MAPREDUCE-6224:
------------------------------------

             Summary: resolve the hosts in DNSToSwitchMapping before inter tracker server start to avoid IPC timeout in Task Tracker heartbeat
                 Key: MAPREDUCE-6224
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6224
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: mrv1
            Reporter: zhihai xu
            Assignee: zhihai xu


Resolve the hosts to fill up the cache in CachedDNSToSwitchMapping before inter tracker server start to avoid IPC timeout in Task Tracker heartbeat.
We saw IPC timeout happen in Task Tracker heartbeat for a large MR1 cluster which use topology script(ShellCommandExecutor) to resolve the Network Topology for Task Tracker host in ScriptBasedMapping.
The reason is 
Right after inter tracker server start in Job Tracker, Job Tracker receive a lots HeartBeat from the Task Tracker. 
heartbeat function call resolveAndAddToTopology to resolve the Network Topology for Task Tracker host in ScriptBasedMapping which implement CachedDNSToSwitchMapping.
ScriptBasedMapping#resolve will check whether the host is in the cache,
If the host is not in the cache, it will run topology script to get the host's Network Topology using ShellCommandExecutor. Normally running topology script is time consuming, which may cause the IPC time if too many heartbeat happened at the same time for a large MR1 cluster.
The solution is to resolve the Network Topology for all hosts in the hosts list from HostsFileReader before receive any heartbeat from Task Tracker, so the cache in ScriptBasedMapping will be filled up, and when heartbeat call resolveAndAddToTopology, it will get the result from the cache instead of running topology script.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)