Task failing due to failed on connection exception

[Joey Andres <ye...@gmail.com>]

I'm trying to setup hadoop cluster for version 2.8.2 with two slaves.  
Whenever I run:

hadoop jar 
$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar 
teragen -Dmapreduce.job.maps=1000 10t random-data

I get the following error for one of the slave:

17/10/29 00:02:51 INFO mapreduce.Job: Task Id : 
attempt_1509256119340_0001_m_000445_1, Status : FAILED
Container launch failed for container_1509256119340_0001_01_000682 : 
java.net.ConnectException: Call From linux-01/127.0.0.1 to 
localhost:37411 failed on connection exception: 
java.net.ConnectException: Connection refused; For more details see:  
http://wiki.apache.org/hadoop/ConnectionRefused

Going to that slave's yarn-hadoop-nodemanager-linux-02.log, I get:

2017-10-28 23:48:42,719 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Node ID assigned is : localhost:37411
2017-10-28 23:48:42,725 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at linux-01.local/192.168.1.1:8031
2017-10-28 23:48:42,762 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out 0 NM container statuses: []
2017-10-28 23:48:42,767 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registering with RM using containers :[]
2017-10-28 23:48:43,181 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager: Rolling master-key for container-tokens, got key with id -410082181
2017-10-28 23:48:43,187 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM: Rolling master-key for container-tokens, got key with id -1296212863
2017-10-28 23:48:43,188 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered with ResourceManager as localhost:37411 with total resource of <memory:8192, vCores:8>
2017-10-28 23:48:43,188 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying ContainerManager to unblock new container-requests
2017-10-28 23:53:46,156 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS event
2017-10-28 23:53:46,156 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS event
2017-10-28 23:53:47,156 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS event
2017-10-28 23:53:48,159 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS event
2017-10-28 23:53:49,162 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS event
2017-10-28 23:53:50,164 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS event
2017-10-28 23:53:51,169 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS event
2017-10-28 23:53:53,174 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS event
2017-10-28 23:56:48,588 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS event
2017-10-28 23:56:50,592 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS event
2017-10-28 23:56:51,595 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS event
2017-10-28 23:56:52,598 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: couldn't find app application_1509256119340_0001 while processing FINISH_CONTAINERS event

In it, you can see "Node ID assigned is : localhost:37411", which 
matches the complain from the mapreduce job.

I don't know why the resourcemanager, in my linux-01 won't call slave 
linux-02, but instead call th enon-existent localhost:37411... I'm very 
confused.

I can clarify and/or provide more info if needed.

Cheers,
Joey Andres