You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Karthik Kambatla (JIRA)" <ji...@apache.org> on 2015/11/03 19:41:27 UTC
[jira] [Updated] (YARN-3112) AM restart and keep containers from
previous attempts, then new container launch failed
[ https://issues.apache.org/jira/browse/YARN-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Kambatla updated YARN-3112:
-----------------------------------
Description:
This error is very similar to YARN-1795, YARN-1839, but i have check the solution of those jira, the patches are already included in my version. I think this error is caused by the different NMTokens between old and new appattempts. New AM has inherited the old tokens from previous AM according to my configuration (keepContainers=true), so the token for new containers are replaced by the old one in the NMTokenCache.
{noformat}
206 2015-01-29 10:04:49,603 ERROR [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Container launch failed for container_1422546145900_0001_02_000002 : org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent for ixk02:47625
207 › at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProt ocolProxy.java:256)
208 › at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.<init>(ContainerManagementProtoc olProxy.java:246)
209 › at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:132)
210 › at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:401)
211 › at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138)
212 › at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:367)
213 › at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
214 › at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
215 › at java.lang.Thread.run(Thread.java:722)
{noformat}
was:
This error is very similar to YARN-1795, YARN-1839, but i have check the solution of those jira, the patches are already included in my version. I think this error is caused by the different NMTokens between old and new appattempts. New AM has inherited the old tokens from previous AM according to my configuration (keepContainers=true), so the token for new containers are replaced by the old one in the NMTokenCache.
206 2015-01-29 10:04:49,603 ERROR [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Container launch failed for container_1422546145900_0001_02_000002 : org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent for ixk02:47625
207 › at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProt ocolProxy.java:256)
208 › at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.<init>(ContainerManagementProtoc olProxy.java:246)
209 › at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:132)
210 › at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:401)
211 › at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138)
212 › at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:367)
213 › at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
214 › at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
215 › at java.lang.Thread.run(Thread.java:722)
> AM restart and keep containers from previous attempts, then new container launch failed
> ---------------------------------------------------------------------------------------
>
> Key: YARN-3112
> URL: https://issues.apache.org/jira/browse/YARN-3112
> Project: Hadoop YARN
> Issue Type: Bug
> Components: applications, resourcemanager
> Affects Versions: 2.6.0
> Environment: in real linux cluster
> Reporter: Jack Chen
>
> This error is very similar to YARN-1795, YARN-1839, but i have check the solution of those jira, the patches are already included in my version. I think this error is caused by the different NMTokens between old and new appattempts. New AM has inherited the old tokens from previous AM according to my configuration (keepContainers=true), so the token for new containers are replaced by the old one in the NMTokenCache.
> {noformat}
> 206 2015-01-29 10:04:49,603 ERROR [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Container launch failed for container_1422546145900_0001_02_000002 : org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent for ixk02:47625
> 207 › at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProt ocolProxy.java:256)
> 208 › at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.<init>(ContainerManagementProtoc olProxy.java:246)
> 209 › at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:132)
> 210 › at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:401)
> 211 › at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138)
> 212 › at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:367)
> 213 › at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 214 › at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 215 › at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)