You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "Abhishek Tiwari (JIRA)" <ji...@apache.org> on 2017/07/19 07:41:01 UTC

[jira] [Created] (GOBBLIN-124) Gobblin Yarn fails due to Invalid AMRMToken

Abhishek Tiwari created GOBBLIN-124:
---------------------------------------

             Summary: Gobblin Yarn fails due to Invalid AMRMToken
                 Key: GOBBLIN-124
                 URL: https://issues.apache.org/jira/browse/GOBBLIN-124
             Project: Apache Gobblin
          Issue Type: Bug
            Reporter: Joel Baranick


After running a while, the app_master in gobblin yarn fails due to Invalid AMRMToken.  Without #828, any ongoing jobs may be dead in the water.  I believe this is related to [YARN-3103](https://issues.apache.org/jira/browse/YARN-3103).
### The original error

```
2016-03-10 22:54:02 UTC ERROR [AMRM Heartbeater thread] org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread  - Exception on heartbeat
org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid AMRMToken from appattempt_1457131141434_0018_000001
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
    at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
    at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79)
    at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy20.allocate(Unknown Source)
    at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:278)
    at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:224)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): Invalid AMRMToken from appattempt_1457131141434_0018_000001
    at org.apache.hadoop.ipc.Client.call(Client.java:1471)
    at org.apache.hadoop.ipc.Client.call(Client.java:1402)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    at com.sun.proxy.$Proxy19.allocate(Unknown Source)
    at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
    ... 8 more
2016-03-10 22:54:02 UTC INFO  [AMRM Callback Handler Thread] org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread  - Interrupted while waiting for queue
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274)
2016-03-10 22:54:02 UTC ERROR [AMRM Callback Handler Thread] org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread  - Stopping callback due to: 
org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid AMRMToken from appattempt_1457131141434_0018_000001
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
    at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
    at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79)
    at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy20.allocate(Unknown Source)
    at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:278)
    at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:224)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): Invalid AMRMToken from appattempt_1457131141434_0018_000001
    at org.apache.hadoop.ipc.Client.call(Client.java:1471)
    at org.apache.hadoop.ipc.Client.call(Client.java:1402)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    at com.sun.proxy.$Proxy19.allocate(Unknown Source)
    at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
    ... 8 more
2016-03-10 22:54:02 UTC ERROR [AMRM Callback Handler Thread] gobblin.yarn.YarnService$AMRMClientCallbackHandler  - Received error: org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid AMRMToken from appattempt_1457131141434_0018_000001
org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid AMRMToken from appattempt_1457131141434_0018_000001
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
    at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
    at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79)
    at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy20.allocate(Unknown Source)
    at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:278)
    at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:224)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): Invalid AMRMToken from appattempt_1457131141434_0018_000001
    at org.apache.hadoop.ipc.Client.call(Client.java:1471)
    at org.apache.hadoop.ipc.Client.call(Client.java:1402)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    at com.sun.proxy.$Proxy19.allocate(Unknown Source)
    at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
    ... 8 more
```
### And later this happens:

```
2016-03-10 22:54:02 UTC INFO  [YarnService STOPPING] gobblin.yarn.YarnService  - All of the containers have been stopped
2016-03-10 22:54:02 UTC WARN  [YarnService STOPPING] org.apache.hadoop.ipc.Client$Connection$1  - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): Invalid AMRMToken from appattempt_1457131141434_0018_000001
2016-03-10 22:54:02 UTC ERROR [YarnService STOPPING] gobblin.yarn.YarnService  - Failed to unregister the ApplicationMaster
org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid AMRMToken from appattempt_1457131141434_0018_000001
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
    at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
    at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.finishApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:94)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy20.finishApplicationMaster(Unknown Source)
    at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.unregisterApplicationMaster(AMRMClientImpl.java:378)
    at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.unregisterApplicationMaster(AMRMClientAsyncImpl.java:157)
    at gobblin.yarn.YarnService.shutDown(YarnService.java:268)
    at com.google.common.util.concurrent.AbstractIdleService$2$2.run(AbstractIdleService.java:69)
    at com.google.common.util.concurrent.Callables$3.run(Callables.java:93)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): Invalid AMRMToken from appattempt_1457131141434_0018_000001
    at org.apache.hadoop.ipc.Client.call(Client.java:1471)
    at org.apache.hadoop.ipc.Client.call(Client.java:1402)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    at com.sun.proxy.$Proxy19.finishApplicationMaster(Unknown Source)
    at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.finishApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:91)
    ... 13 more
```
### @sahilTakiar, and then a timeout shutting down the new `ServiceBasedAppLauncher`

```
2016-03-10 22:54:07 UTC ERROR [AMRM Callback Handler Thread] gobblin.runtime.app.ServiceBasedAppLauncher  - Timeout in stopping the service manager
java.util.concurrent.TimeoutException: Timeout waiting for the services to stop.
    at com.google.common.util.concurrent.ServiceManager.awaitStopped(ServiceManager.java:338)
    at gobblin.runtime.app.ServiceBasedAppLauncher.stop(ServiceBasedAppLauncher.java:136)
    at gobblin.yarn.GobblinApplicationMaster.stop(GobblinApplicationMaster.java:170)
    at gobblin.yarn.GobblinApplicationMaster.handleApplicationMasterShutdownRequest(GobblinApplicationMaster.java:246)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:74)
    at com.google.common.eventbus.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:47)
    at com.google.common.eventbus.EventBus.dispatch(EventBus.java:314)
    at com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:296)
    at com.google.common.eventbus.EventBus.post(EventBus.java:267)
    at gobblin.yarn.YarnService$AMRMClientCallbackHandler.onError(YarnService.java:598)
```

 
*Github Url* : https://github.com/linkedin/gobblin/issues/831 
*Github Reporter* : [~jbaranick] 
*Github Created At* : 2016-03-11T00:30:39Z 
*Github Updated At* : 2017-02-02T19:48:00Z 
h3. Comments 
----
[~stakiar] wrote on 2016-03-14T21:21:42Z : Do you hit this with version `2.3.0`? Any chance you can upgrade to `2.6.1` and see if it is still there?

I don't think we have seen this issue before, but I would have to investigate with a few other people to see if it came up and then was fixed internally.

It doesn't seem like this is an exception Gobblin can recover from. Is there any way you can backport the patch available in YARN-3103 to your current Hadoop version?
 
 
*Github Url* : https://github.com/linkedin/gobblin/issues/831#issuecomment-196526319 
----
[~stakiar] wrote on 2016-03-14T21:23:33Z : As for the timeout in `ServiceBasedAppLauncher` my guess is that this exception causes `YARNService` to not shutdown properly, which cases the timeout. The logs from `ServiceBasedAppLauncher` aren't very helpful in confirming this. Logging the state of each running `Service` if a `TimeoutException` occurs would be helpful here.
 
 
*Github Url* : https://github.com/linkedin/gobblin/issues/831#issuecomment-196526809 
----
[~jbaranick] wrote on 2016-03-14T21:25:53Z : @sahilTakiar I've opened a bug on Qubole to backport this fix to their version of 2.6.0.
 
 
*Github Url* : https://github.com/linkedin/gobblin/issues/831#issuecomment-196527466 
----
[~jbaranick] wrote on 2017-02-02T19:48:00Z : Qubole has backported this fix, but we still get a crash caused by this at least once per day. 
 
*Github Url* : https://github.com/linkedin/gobblin/issues/831#issuecomment-277062505



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)