Tasktracker denied communication with jobtracker

[Virajith Jalaparti <vi...@gmail.com>]

Hi,

I am trying to setup a hadoop cluster with 7nodes with the master node also
functioning as a slave node (i.e. runs a datanode and a tasktracker along
with the namenode and jobtracker deamons). I am able to get HDFS working.
However when I try starting the tasktrackers (bin/start-mapred.sh), I get
the errors similar to the following logged in the jobtrackers log:

org.apache.hadoop.mapred.DisallowedTaskTrackerException: Tasktracker denied
communication with jobtracker: tracker_pc405.emulab.net:localhost/
127.0.0.1:46511
        at
org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2498)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)


I am not able to figure out why this happening. I think it is not a network
issue since HDFS is able to run just fine. This website
http://www.docjar.com/docs/api/org/apache/hadoop/mapred/DisallowedTaskTrackerException.htmlsays
that this is because "tasktracker tries to register or communicate
with
the jobtracker when it does not appear on the list of included nodes, or has
been specifically excluded". However, I have included all the tasktracker
nodes in the file which is read using the "mapred.hosts" parameter.  (These
are the same nodes as the HDFS nodes and hence shouldn't be a problem)
Can anyone help me out with this?

Thanks a lot,
Virajith

RE: Tasktracker denied communication with jobtracker

[Devaraj K <de...@huawei.com>]

Hi Virajith,

            This exception will be thrown when the host name present in the
file which is the value for the property "mapred.hosts.exclude". If you
don't mention any thing for "mapred.hosts" and "mapred.hosts.exclude"
properties or the mentioned files don't contain any hosts, job tracker
allows to connect any task tracker. If you include the host name in
"mapred.hosts" and also in "mapred.hosts.exclude", it will not allow to
connect the task tracker and it throws the below exception.

If you want to allow all the task trackers , don't mention any thing for the
both properties. If you want exclude specific hosts and allow the remaining
hosts to connect then add those host names exclude
file("mapred.hosts.exclude" file) and don't add anything for the include
list. If you want to include specific hosts, the add those host names in
"mapred.hosts" property file and don't add anything for the exclude list.

"mapred.hosts" and "mapred.hosts.exclude" are for hadoop 0.20.x versions.
For the later versions need to update these
"mapreduce.jobtracker.hosts.exclude.filename" and
"mapreduce.jobtracker.hosts.filename".



Devaraj K 

 

  _____  

From: Virajith Jalaparti [mailto:virajith.j@gmail.com] 
Sent: Tuesday, June 21, 2011 7:49 PM
To: mapreduce-user@hadoop.apache.org
Subject: Tasktracker denied communication with jobtracker

 

Hi,

I am trying to setup a hadoop cluster with 7nodes with the master node also
functioning as a slave node (i.e. runs a datanode and a tasktracker along
with the namenode and jobtracker deamons). I am able to get HDFS working.
However when I try starting the tasktrackers (bin/start-mapred.sh), I get
the errors similar to the following logged in the jobtrackers log:

org.apache.hadoop.mapred.DisallowedTaskTrackerException: Tasktracker denied
communication with jobtracker:
tracker_pc405.emulab.net:localhost/127.0.0.1:46511
        at
org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2498)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)


I am not able to figure out why this happening. I think it is not a network
issue since HDFS is able to run just fine. This website
http://www.docjar.com/docs/api/org/apache/hadoop/mapred/DisallowedTaskTracke
rException.html says that this is because "tasktracker tries to register or
communicate with the jobtracker when it does not appear on the list of
included nodes, or has been specifically excluded". However, I have included
all the tasktracker nodes in the file which is read using the "mapred.hosts"
parameter.  (These are the same nodes as the HDFS nodes and hence shouldn't
be a problem)
Can anyone help me out with this?

Thanks a lot,
Virajith