You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by John Lilley <jo...@redpoint.net> on 2014/03/03 14:19:48 UTC
ResourceManager crash on deleted NM node back from the dead
We had a DN/NM node that went offline for a while and been removed from the cluster via Ambari without decommissioning (because it was offline).
When the node came back up, its NM attempted connection to the RM.
Later the RM failed with this exception (dokken is the errant node):
2014-03-03 05:53:28,721 FATAL resourcemanager.ResourceManager (ResourceManager.java:run(449)) - Error in handling event type NODE_UPDATE to the scheduler
java.lang.IllegalArgumentException: java.net.UnknownHostException: dokken.office.datalever.com
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:418)
at org.apache.hadoop.yarn.server.utils.BuilderUtils.newContainerToken(BuilderUtils.java:247)
at org.apache.hadoop.yarn.server.resourcemanager.security.RMContainerTokenSecretManager.createContainerToken(RMContainerTokenSecretManager.java:195)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainerToken(LeafQueue.java:1297)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1345)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1211)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1170)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:871)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:645)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:559)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:690)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:734)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:86)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:440)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.UnknownHostException: dokken.office.datalever.com
... 15 more
2014-03-03 05:53:28,723 INFO resourcemanager.ResourceManager (ResourceManager.java:run(453)) - Exiting, bbye..
I can see our error in how this node was removed while not being truly dead, but I think it shouldn't crash the RM.
Forgive my ignorance ... do you know if this is a known problem?
FYI:
# hadoop version
Hadoop 2.2.0.2.0.6.0-76
Subversion git@github.com:hortonworks/hadoop.git -r 8656b1cfad13b03b29e98cad042626205e7a1c86
Compiled by jenkins on 2013-10-18T00:19Z
Compiled with protoc 2.5.0
>From source with checksum d23ee1d271c6ac5bd27de664146be2
This command was run using /usr/lib/hadoop/hadoop-common-2.2.0.2.0.6.0-76.jar
Thanks
John
Re: ResourceManager crash on deleted NM node back from the dead
Posted by Jian He <jh...@hortonworks.com>.
Hi, I believe this is recently fixed in
https://issues.apache.org/jira/browse/YARN-713, and will be part of 2.4.0
release.
Jian
On Mon, Mar 3, 2014 at 5:19 AM, John Lilley <jo...@redpoint.net>wrote:
> We had a DN/NM node that went offline for a while and been removed from
> the cluster via Ambari without decommissioning (because it was offline).
>
> When the node came back up, its NM attempted connection to the RM.
>
> Later the RM failed with this exception (dokken is the errant node):
>
>
>
> 2014-03-03 05:53:28,721 FATAL resourcemanager.ResourceManager
> (ResourceManager.java:run(449)) - Error in handling event type NODE_UPDATE
> to the scheduler
>
> java.lang.IllegalArgumentException: java.net.UnknownHostException:
> dokken.office.datalever.com
>
> at
> org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:418)
>
> at
> org.apache.hadoop.yarn.server.utils.BuilderUtils.newContainerToken(BuilderUtils.java:247)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.RMContainerTokenSecretManager.createContainerToken(RMContainerTokenSecretManager.java:195)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainerToken(LeafQueue.java:1297)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1345)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1211)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1170)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:871)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:645)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:559)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:690)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:734)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:86)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:440)
>
> at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.net.UnknownHostException: dokken.office.datalever.com
>
> ... 15 more
>
> 2014-03-03 05:53:28,723 INFO resourcemanager.ResourceManager
> (ResourceManager.java:run(453)) - Exiting, bbye..
>
>
>
> I can see our error in how this node was removed while not being truly
> dead, but I think it shouldn't crash the RM.
>
> Forgive my ignorance ... do you know if this is a known problem?
>
>
>
> FYI:
>
> # hadoop version
>
> Hadoop 2.2.0.2.0.6.0-76
>
> Subversion git@github.com:hortonworks/hadoop.git -r
> 8656b1cfad13b03b29e98cad042626205e7a1c86
>
> Compiled by jenkins on 2013-10-18T00:19Z
>
> Compiled with protoc 2.5.0
>
> From source with checksum d23ee1d271c6ac5bd27de664146be2
>
> This command was run using
> /usr/lib/hadoop/hadoop-common-2.2.0.2.0.6.0-76.jar
>
>
>
> *Thanks*
>
> *John*
>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: ResourceManager crash on deleted NM node back from the dead
Posted by Jian He <jh...@hortonworks.com>.
Hi, I believe this is recently fixed in
https://issues.apache.org/jira/browse/YARN-713, and will be part of 2.4.0
release.
Jian
On Mon, Mar 3, 2014 at 5:19 AM, John Lilley <jo...@redpoint.net>wrote:
> We had a DN/NM node that went offline for a while and been removed from
> the cluster via Ambari without decommissioning (because it was offline).
>
> When the node came back up, its NM attempted connection to the RM.
>
> Later the RM failed with this exception (dokken is the errant node):
>
>
>
> 2014-03-03 05:53:28,721 FATAL resourcemanager.ResourceManager
> (ResourceManager.java:run(449)) - Error in handling event type NODE_UPDATE
> to the scheduler
>
> java.lang.IllegalArgumentException: java.net.UnknownHostException:
> dokken.office.datalever.com
>
> at
> org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:418)
>
> at
> org.apache.hadoop.yarn.server.utils.BuilderUtils.newContainerToken(BuilderUtils.java:247)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.RMContainerTokenSecretManager.createContainerToken(RMContainerTokenSecretManager.java:195)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainerToken(LeafQueue.java:1297)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1345)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1211)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1170)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:871)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:645)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:559)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:690)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:734)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:86)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:440)
>
> at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.net.UnknownHostException: dokken.office.datalever.com
>
> ... 15 more
>
> 2014-03-03 05:53:28,723 INFO resourcemanager.ResourceManager
> (ResourceManager.java:run(453)) - Exiting, bbye..
>
>
>
> I can see our error in how this node was removed while not being truly
> dead, but I think it shouldn't crash the RM.
>
> Forgive my ignorance ... do you know if this is a known problem?
>
>
>
> FYI:
>
> # hadoop version
>
> Hadoop 2.2.0.2.0.6.0-76
>
> Subversion git@github.com:hortonworks/hadoop.git -r
> 8656b1cfad13b03b29e98cad042626205e7a1c86
>
> Compiled by jenkins on 2013-10-18T00:19Z
>
> Compiled with protoc 2.5.0
>
> From source with checksum d23ee1d271c6ac5bd27de664146be2
>
> This command was run using
> /usr/lib/hadoop/hadoop-common-2.2.0.2.0.6.0-76.jar
>
>
>
> *Thanks*
>
> *John*
>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: ResourceManager crash on deleted NM node back from the dead
Posted by Jian He <jh...@hortonworks.com>.
Hi, I believe this is recently fixed in
https://issues.apache.org/jira/browse/YARN-713, and will be part of 2.4.0
release.
Jian
On Mon, Mar 3, 2014 at 5:19 AM, John Lilley <jo...@redpoint.net>wrote:
> We had a DN/NM node that went offline for a while and been removed from
> the cluster via Ambari without decommissioning (because it was offline).
>
> When the node came back up, its NM attempted connection to the RM.
>
> Later the RM failed with this exception (dokken is the errant node):
>
>
>
> 2014-03-03 05:53:28,721 FATAL resourcemanager.ResourceManager
> (ResourceManager.java:run(449)) - Error in handling event type NODE_UPDATE
> to the scheduler
>
> java.lang.IllegalArgumentException: java.net.UnknownHostException:
> dokken.office.datalever.com
>
> at
> org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:418)
>
> at
> org.apache.hadoop.yarn.server.utils.BuilderUtils.newContainerToken(BuilderUtils.java:247)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.RMContainerTokenSecretManager.createContainerToken(RMContainerTokenSecretManager.java:195)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainerToken(LeafQueue.java:1297)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1345)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1211)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1170)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:871)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:645)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:559)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:690)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:734)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:86)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:440)
>
> at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.net.UnknownHostException: dokken.office.datalever.com
>
> ... 15 more
>
> 2014-03-03 05:53:28,723 INFO resourcemanager.ResourceManager
> (ResourceManager.java:run(453)) - Exiting, bbye..
>
>
>
> I can see our error in how this node was removed while not being truly
> dead, but I think it shouldn't crash the RM.
>
> Forgive my ignorance ... do you know if this is a known problem?
>
>
>
> FYI:
>
> # hadoop version
>
> Hadoop 2.2.0.2.0.6.0-76
>
> Subversion git@github.com:hortonworks/hadoop.git -r
> 8656b1cfad13b03b29e98cad042626205e7a1c86
>
> Compiled by jenkins on 2013-10-18T00:19Z
>
> Compiled with protoc 2.5.0
>
> From source with checksum d23ee1d271c6ac5bd27de664146be2
>
> This command was run using
> /usr/lib/hadoop/hadoop-common-2.2.0.2.0.6.0-76.jar
>
>
>
> *Thanks*
>
> *John*
>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: ResourceManager crash on deleted NM node back from the dead
Posted by Jian He <jh...@hortonworks.com>.
Hi, I believe this is recently fixed in
https://issues.apache.org/jira/browse/YARN-713, and will be part of 2.4.0
release.
Jian
On Mon, Mar 3, 2014 at 5:19 AM, John Lilley <jo...@redpoint.net>wrote:
> We had a DN/NM node that went offline for a while and been removed from
> the cluster via Ambari without decommissioning (because it was offline).
>
> When the node came back up, its NM attempted connection to the RM.
>
> Later the RM failed with this exception (dokken is the errant node):
>
>
>
> 2014-03-03 05:53:28,721 FATAL resourcemanager.ResourceManager
> (ResourceManager.java:run(449)) - Error in handling event type NODE_UPDATE
> to the scheduler
>
> java.lang.IllegalArgumentException: java.net.UnknownHostException:
> dokken.office.datalever.com
>
> at
> org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:418)
>
> at
> org.apache.hadoop.yarn.server.utils.BuilderUtils.newContainerToken(BuilderUtils.java:247)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.RMContainerTokenSecretManager.createContainerToken(RMContainerTokenSecretManager.java:195)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainerToken(LeafQueue.java:1297)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1345)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1211)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1170)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:871)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:645)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:559)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:690)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:734)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:86)
>
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:440)
>
> at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.net.UnknownHostException: dokken.office.datalever.com
>
> ... 15 more
>
> 2014-03-03 05:53:28,723 INFO resourcemanager.ResourceManager
> (ResourceManager.java:run(453)) - Exiting, bbye..
>
>
>
> I can see our error in how this node was removed while not being truly
> dead, but I think it shouldn't crash the RM.
>
> Forgive my ignorance ... do you know if this is a known problem?
>
>
>
> FYI:
>
> # hadoop version
>
> Hadoop 2.2.0.2.0.6.0-76
>
> Subversion git@github.com:hortonworks/hadoop.git -r
> 8656b1cfad13b03b29e98cad042626205e7a1c86
>
> Compiled by jenkins on 2013-10-18T00:19Z
>
> Compiled with protoc 2.5.0
>
> From source with checksum d23ee1d271c6ac5bd27de664146be2
>
> This command was run using
> /usr/lib/hadoop/hadoop-common-2.2.0.2.0.6.0-76.jar
>
>
>
> *Thanks*
>
> *John*
>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.