You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Robert Chu <al...@gmail.com> on 2014/04/25 06:22:54 UTC
Question about YARN security tokens
Hi Everyone,
I'm new to YARN and was trying to write a simple YARN application that
starts up an ApplicationMaster that starts up an in-process Jetty server
just a simple test (this application requests no further resource
containers). An attempt is made to register as an application with the
ResourceManager while starting the ApplicationMaster process but this fails
with an exception related to security tokens when using the MiniYARNCluster:
Exception in thread "main"
org.apache.hadoop.security.token.SecretManager$InvalidToken: Password not
found for ApplicationAttempt appattempt_1398290928870_0001_000002
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy8.registerApplicationMaster(Unknown Source)
at
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.registerApplicationMaster(AMRMClientImpl.java:196)
at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.registerApplicationMaster(AMRMClientAsyncImpl.java:138)
at
org.kiji.scoring.server.yarn.YarnServiceMaster.start(YarnServiceMaster.java:228)
at
org.kiji.scoring.server.yarn.YarnServiceMaster.main(YarnServiceMaster.java:333)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Password not found for ApplicationAttempt
appattempt_1398290928870_0001_000002
at org.apache.hadoop.ipc.Client.call(Client.java:1409)
at org.apache.hadoop.ipc.Client.call(Client.java:1362)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy7.registerApplicationMaster(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106)
... 11 more
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Upon getting this stack trace, I attempted to find
documentation/information/examples about security tokens but was unable to
find anything of note.
Am I doing something dumb here and is this not related to security tokens?
If not, is there documentation or good examples around how to use security
tokens while launching an application?
Thanks!
Robert Chu
Re: Question about YARN security tokens
Posted by Robert Chu <al...@gmail.com>.
Thanks for the extra information!
For reference, the code that I was running was basically the code that can
be found here: https://github.com/hortonworks/simple-yarn-app. The issue
that I was having was when running from within intellij (aka without the
classpath bits that 'hadoop classpath' adds) against either a
psuedo-distributed-mode cluster or against the MiniYARNCluster. Running
applications with the correct hadoop classpath seems to have made any
SecretManager errors go away.
Robert
Robert Chu
On Fri, Apr 25, 2014 at 4:53 AM, Oleg Zhurakousky <
oleg.zhurakousky@gmail.com> wrote:
> Without looking at the code its hard to say. Perhaps looking at a working
> code will put you in the right direction.
> For example, here is the DistributedShell from Hadoop (only few classes)
> https://github.com/apache/hadoop-common/tree/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell
> That is what I used a a model to dig into some of the YARN internals as I
> am playing around with ideas on how to simplify YARN API
> https://github.com/olegz/yaya, so you can use that code as well to see if
> you can get some more information from it.
>
> Cheers
> Oleg
>
>
> On Fri, Apr 25, 2014 at 12:22 AM, Robert Chu <al...@gmail.com> wrote:
>
>> Hi Everyone,
>>
>> I'm new to YARN and was trying to write a simple YARN application that
>> starts up an ApplicationMaster that starts up an in-process Jetty server
>> just a simple test (this application requests no further resource
>> containers). An attempt is made to register as an application with the
>> ResourceManager while starting the ApplicationMaster process but this fails
>> with an exception related to security tokens when using the MiniYARNCluster:
>>
>> Exception in thread "main"
>> org.apache.hadoop.security.token.SecretManager$InvalidToken: Password not
>> found for ApplicationAttempt appattempt_1398290928870_0001_000002
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>> at
>> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
>> at
>> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
>> at
>> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>> at com.sun.proxy.$Proxy8.registerApplicationMaster(Unknown Source)
>> at
>> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.registerApplicationMaster(AMRMClientImpl.java:196)
>> at
>> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.registerApplicationMaster(AMRMClientAsyncImpl.java:138)
>> at
>> org.kiji.scoring.server.yarn.YarnServiceMaster.start(YarnServiceMaster.java:228)
>> at
>> org.kiji.scoring.server.yarn.YarnServiceMaster.main(YarnServiceMaster.java:333)
>> Caused by:
>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
>> Password not found for ApplicationAttempt
>> appattempt_1398290928870_0001_000002
>> at org.apache.hadoop.ipc.Client.call(Client.java:1409)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1362)
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>> at com.sun.proxy.$Proxy7.registerApplicationMaster(Unknown Source)
>> at
>> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106)
>> ... 11 more
>>
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
>> at org.apache.hadoop.util.Shell.run(Shell.java:418)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> Upon getting this stack trace, I attempted to find
>> documentation/information/examples about security tokens but was unable to
>> find anything of note.
>>
>> Am I doing something dumb here and is this not related to security
>> tokens? If not, is there documentation or good examples around how to use
>> security tokens while launching an application?
>>
>> Thanks!
>>
>> Robert Chu
>>
>
>
Re: Question about YARN security tokens
Posted by Robert Chu <al...@gmail.com>.
Thanks for the extra information!
For reference, the code that I was running was basically the code that can
be found here: https://github.com/hortonworks/simple-yarn-app. The issue
that I was having was when running from within intellij (aka without the
classpath bits that 'hadoop classpath' adds) against either a
psuedo-distributed-mode cluster or against the MiniYARNCluster. Running
applications with the correct hadoop classpath seems to have made any
SecretManager errors go away.
Robert
Robert Chu
On Fri, Apr 25, 2014 at 4:53 AM, Oleg Zhurakousky <
oleg.zhurakousky@gmail.com> wrote:
> Without looking at the code its hard to say. Perhaps looking at a working
> code will put you in the right direction.
> For example, here is the DistributedShell from Hadoop (only few classes)
> https://github.com/apache/hadoop-common/tree/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell
> That is what I used a a model to dig into some of the YARN internals as I
> am playing around with ideas on how to simplify YARN API
> https://github.com/olegz/yaya, so you can use that code as well to see if
> you can get some more information from it.
>
> Cheers
> Oleg
>
>
> On Fri, Apr 25, 2014 at 12:22 AM, Robert Chu <al...@gmail.com> wrote:
>
>> Hi Everyone,
>>
>> I'm new to YARN and was trying to write a simple YARN application that
>> starts up an ApplicationMaster that starts up an in-process Jetty server
>> just a simple test (this application requests no further resource
>> containers). An attempt is made to register as an application with the
>> ResourceManager while starting the ApplicationMaster process but this fails
>> with an exception related to security tokens when using the MiniYARNCluster:
>>
>> Exception in thread "main"
>> org.apache.hadoop.security.token.SecretManager$InvalidToken: Password not
>> found for ApplicationAttempt appattempt_1398290928870_0001_000002
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>> at
>> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
>> at
>> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
>> at
>> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>> at com.sun.proxy.$Proxy8.registerApplicationMaster(Unknown Source)
>> at
>> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.registerApplicationMaster(AMRMClientImpl.java:196)
>> at
>> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.registerApplicationMaster(AMRMClientAsyncImpl.java:138)
>> at
>> org.kiji.scoring.server.yarn.YarnServiceMaster.start(YarnServiceMaster.java:228)
>> at
>> org.kiji.scoring.server.yarn.YarnServiceMaster.main(YarnServiceMaster.java:333)
>> Caused by:
>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
>> Password not found for ApplicationAttempt
>> appattempt_1398290928870_0001_000002
>> at org.apache.hadoop.ipc.Client.call(Client.java:1409)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1362)
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>> at com.sun.proxy.$Proxy7.registerApplicationMaster(Unknown Source)
>> at
>> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106)
>> ... 11 more
>>
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
>> at org.apache.hadoop.util.Shell.run(Shell.java:418)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> Upon getting this stack trace, I attempted to find
>> documentation/information/examples about security tokens but was unable to
>> find anything of note.
>>
>> Am I doing something dumb here and is this not related to security
>> tokens? If not, is there documentation or good examples around how to use
>> security tokens while launching an application?
>>
>> Thanks!
>>
>> Robert Chu
>>
>
>
Re: Question about YARN security tokens
Posted by Robert Chu <al...@gmail.com>.
Thanks for the extra information!
For reference, the code that I was running was basically the code that can
be found here: https://github.com/hortonworks/simple-yarn-app. The issue
that I was having was when running from within intellij (aka without the
classpath bits that 'hadoop classpath' adds) against either a
psuedo-distributed-mode cluster or against the MiniYARNCluster. Running
applications with the correct hadoop classpath seems to have made any
SecretManager errors go away.
Robert
Robert Chu
On Fri, Apr 25, 2014 at 4:53 AM, Oleg Zhurakousky <
oleg.zhurakousky@gmail.com> wrote:
> Without looking at the code its hard to say. Perhaps looking at a working
> code will put you in the right direction.
> For example, here is the DistributedShell from Hadoop (only few classes)
> https://github.com/apache/hadoop-common/tree/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell
> That is what I used a a model to dig into some of the YARN internals as I
> am playing around with ideas on how to simplify YARN API
> https://github.com/olegz/yaya, so you can use that code as well to see if
> you can get some more information from it.
>
> Cheers
> Oleg
>
>
> On Fri, Apr 25, 2014 at 12:22 AM, Robert Chu <al...@gmail.com> wrote:
>
>> Hi Everyone,
>>
>> I'm new to YARN and was trying to write a simple YARN application that
>> starts up an ApplicationMaster that starts up an in-process Jetty server
>> just a simple test (this application requests no further resource
>> containers). An attempt is made to register as an application with the
>> ResourceManager while starting the ApplicationMaster process but this fails
>> with an exception related to security tokens when using the MiniYARNCluster:
>>
>> Exception in thread "main"
>> org.apache.hadoop.security.token.SecretManager$InvalidToken: Password not
>> found for ApplicationAttempt appattempt_1398290928870_0001_000002
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>> at
>> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
>> at
>> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
>> at
>> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>> at com.sun.proxy.$Proxy8.registerApplicationMaster(Unknown Source)
>> at
>> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.registerApplicationMaster(AMRMClientImpl.java:196)
>> at
>> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.registerApplicationMaster(AMRMClientAsyncImpl.java:138)
>> at
>> org.kiji.scoring.server.yarn.YarnServiceMaster.start(YarnServiceMaster.java:228)
>> at
>> org.kiji.scoring.server.yarn.YarnServiceMaster.main(YarnServiceMaster.java:333)
>> Caused by:
>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
>> Password not found for ApplicationAttempt
>> appattempt_1398290928870_0001_000002
>> at org.apache.hadoop.ipc.Client.call(Client.java:1409)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1362)
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>> at com.sun.proxy.$Proxy7.registerApplicationMaster(Unknown Source)
>> at
>> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106)
>> ... 11 more
>>
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
>> at org.apache.hadoop.util.Shell.run(Shell.java:418)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> Upon getting this stack trace, I attempted to find
>> documentation/information/examples about security tokens but was unable to
>> find anything of note.
>>
>> Am I doing something dumb here and is this not related to security
>> tokens? If not, is there documentation or good examples around how to use
>> security tokens while launching an application?
>>
>> Thanks!
>>
>> Robert Chu
>>
>
>
Re: Question about YARN security tokens
Posted by Robert Chu <al...@gmail.com>.
Thanks for the extra information!
For reference, the code that I was running was basically the code that can
be found here: https://github.com/hortonworks/simple-yarn-app. The issue
that I was having was when running from within intellij (aka without the
classpath bits that 'hadoop classpath' adds) against either a
psuedo-distributed-mode cluster or against the MiniYARNCluster. Running
applications with the correct hadoop classpath seems to have made any
SecretManager errors go away.
Robert
Robert Chu
On Fri, Apr 25, 2014 at 4:53 AM, Oleg Zhurakousky <
oleg.zhurakousky@gmail.com> wrote:
> Without looking at the code its hard to say. Perhaps looking at a working
> code will put you in the right direction.
> For example, here is the DistributedShell from Hadoop (only few classes)
> https://github.com/apache/hadoop-common/tree/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell
> That is what I used a a model to dig into some of the YARN internals as I
> am playing around with ideas on how to simplify YARN API
> https://github.com/olegz/yaya, so you can use that code as well to see if
> you can get some more information from it.
>
> Cheers
> Oleg
>
>
> On Fri, Apr 25, 2014 at 12:22 AM, Robert Chu <al...@gmail.com> wrote:
>
>> Hi Everyone,
>>
>> I'm new to YARN and was trying to write a simple YARN application that
>> starts up an ApplicationMaster that starts up an in-process Jetty server
>> just a simple test (this application requests no further resource
>> containers). An attempt is made to register as an application with the
>> ResourceManager while starting the ApplicationMaster process but this fails
>> with an exception related to security tokens when using the MiniYARNCluster:
>>
>> Exception in thread "main"
>> org.apache.hadoop.security.token.SecretManager$InvalidToken: Password not
>> found for ApplicationAttempt appattempt_1398290928870_0001_000002
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>> at
>> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
>> at
>> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
>> at
>> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>> at com.sun.proxy.$Proxy8.registerApplicationMaster(Unknown Source)
>> at
>> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.registerApplicationMaster(AMRMClientImpl.java:196)
>> at
>> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.registerApplicationMaster(AMRMClientAsyncImpl.java:138)
>> at
>> org.kiji.scoring.server.yarn.YarnServiceMaster.start(YarnServiceMaster.java:228)
>> at
>> org.kiji.scoring.server.yarn.YarnServiceMaster.main(YarnServiceMaster.java:333)
>> Caused by:
>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
>> Password not found for ApplicationAttempt
>> appattempt_1398290928870_0001_000002
>> at org.apache.hadoop.ipc.Client.call(Client.java:1409)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1362)
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>> at com.sun.proxy.$Proxy7.registerApplicationMaster(Unknown Source)
>> at
>> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106)
>> ... 11 more
>>
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
>> at org.apache.hadoop.util.Shell.run(Shell.java:418)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> Upon getting this stack trace, I attempted to find
>> documentation/information/examples about security tokens but was unable to
>> find anything of note.
>>
>> Am I doing something dumb here and is this not related to security
>> tokens? If not, is there documentation or good examples around how to use
>> security tokens while launching an application?
>>
>> Thanks!
>>
>> Robert Chu
>>
>
>
Re: Question about YARN security tokens
Posted by Oleg Zhurakousky <ol...@gmail.com>.
Without looking at the code its hard to say. Perhaps looking at a working
code will put you in the right direction.
For example, here is the DistributedShell from Hadoop (only few classes)
https://github.com/apache/hadoop-common/tree/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell
That is what I used a a model to dig into some of the YARN internals as I
am playing around with ideas on how to simplify YARN API
https://github.com/olegz/yaya, so you can use that code as well to see if
you can get some more information from it.
Cheers
Oleg
On Fri, Apr 25, 2014 at 12:22 AM, Robert Chu <al...@gmail.com> wrote:
> Hi Everyone,
>
> I'm new to YARN and was trying to write a simple YARN application that
> starts up an ApplicationMaster that starts up an in-process Jetty server
> just a simple test (this application requests no further resource
> containers). An attempt is made to register as an application with the
> ResourceManager while starting the ApplicationMaster process but this fails
> with an exception related to security tokens when using the MiniYARNCluster:
>
> Exception in thread "main"
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Password not
> found for ApplicationAttempt appattempt_1398290928870_0001_000002
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy8.registerApplicationMaster(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.registerApplicationMaster(AMRMClientImpl.java:196)
> at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.registerApplicationMaster(AMRMClientAsyncImpl.java:138)
> at
> org.kiji.scoring.server.yarn.YarnServiceMaster.start(YarnServiceMaster.java:228)
> at
> org.kiji.scoring.server.yarn.YarnServiceMaster.main(YarnServiceMaster.java:333)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> Password not found for ApplicationAttempt
> appattempt_1398290928870_0001_000002
> at org.apache.hadoop.ipc.Client.call(Client.java:1409)
> at org.apache.hadoop.ipc.Client.call(Client.java:1362)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy7.registerApplicationMaster(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106)
> ... 11 more
>
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
> at org.apache.hadoop.util.Shell.run(Shell.java:418)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
>
> Upon getting this stack trace, I attempted to find
> documentation/information/examples about security tokens but was unable to
> find anything of note.
>
> Am I doing something dumb here and is this not related to security tokens?
> If not, is there documentation or good examples around how to use security
> tokens while launching an application?
>
> Thanks!
>
> Robert Chu
>
Re: Question about YARN security tokens
Posted by Oleg Zhurakousky <ol...@gmail.com>.
Without looking at the code its hard to say. Perhaps looking at a working
code will put you in the right direction.
For example, here is the DistributedShell from Hadoop (only few classes)
https://github.com/apache/hadoop-common/tree/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell
That is what I used a a model to dig into some of the YARN internals as I
am playing around with ideas on how to simplify YARN API
https://github.com/olegz/yaya, so you can use that code as well to see if
you can get some more information from it.
Cheers
Oleg
On Fri, Apr 25, 2014 at 12:22 AM, Robert Chu <al...@gmail.com> wrote:
> Hi Everyone,
>
> I'm new to YARN and was trying to write a simple YARN application that
> starts up an ApplicationMaster that starts up an in-process Jetty server
> just a simple test (this application requests no further resource
> containers). An attempt is made to register as an application with the
> ResourceManager while starting the ApplicationMaster process but this fails
> with an exception related to security tokens when using the MiniYARNCluster:
>
> Exception in thread "main"
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Password not
> found for ApplicationAttempt appattempt_1398290928870_0001_000002
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy8.registerApplicationMaster(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.registerApplicationMaster(AMRMClientImpl.java:196)
> at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.registerApplicationMaster(AMRMClientAsyncImpl.java:138)
> at
> org.kiji.scoring.server.yarn.YarnServiceMaster.start(YarnServiceMaster.java:228)
> at
> org.kiji.scoring.server.yarn.YarnServiceMaster.main(YarnServiceMaster.java:333)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> Password not found for ApplicationAttempt
> appattempt_1398290928870_0001_000002
> at org.apache.hadoop.ipc.Client.call(Client.java:1409)
> at org.apache.hadoop.ipc.Client.call(Client.java:1362)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy7.registerApplicationMaster(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106)
> ... 11 more
>
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
> at org.apache.hadoop.util.Shell.run(Shell.java:418)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
>
> Upon getting this stack trace, I attempted to find
> documentation/information/examples about security tokens but was unable to
> find anything of note.
>
> Am I doing something dumb here and is this not related to security tokens?
> If not, is there documentation or good examples around how to use security
> tokens while launching an application?
>
> Thanks!
>
> Robert Chu
>
Re: Question about YARN security tokens
Posted by Oleg Zhurakousky <ol...@gmail.com>.
Without looking at the code its hard to say. Perhaps looking at a working
code will put you in the right direction.
For example, here is the DistributedShell from Hadoop (only few classes)
https://github.com/apache/hadoop-common/tree/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell
That is what I used a a model to dig into some of the YARN internals as I
am playing around with ideas on how to simplify YARN API
https://github.com/olegz/yaya, so you can use that code as well to see if
you can get some more information from it.
Cheers
Oleg
On Fri, Apr 25, 2014 at 12:22 AM, Robert Chu <al...@gmail.com> wrote:
> Hi Everyone,
>
> I'm new to YARN and was trying to write a simple YARN application that
> starts up an ApplicationMaster that starts up an in-process Jetty server
> just a simple test (this application requests no further resource
> containers). An attempt is made to register as an application with the
> ResourceManager while starting the ApplicationMaster process but this fails
> with an exception related to security tokens when using the MiniYARNCluster:
>
> Exception in thread "main"
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Password not
> found for ApplicationAttempt appattempt_1398290928870_0001_000002
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy8.registerApplicationMaster(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.registerApplicationMaster(AMRMClientImpl.java:196)
> at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.registerApplicationMaster(AMRMClientAsyncImpl.java:138)
> at
> org.kiji.scoring.server.yarn.YarnServiceMaster.start(YarnServiceMaster.java:228)
> at
> org.kiji.scoring.server.yarn.YarnServiceMaster.main(YarnServiceMaster.java:333)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> Password not found for ApplicationAttempt
> appattempt_1398290928870_0001_000002
> at org.apache.hadoop.ipc.Client.call(Client.java:1409)
> at org.apache.hadoop.ipc.Client.call(Client.java:1362)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy7.registerApplicationMaster(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106)
> ... 11 more
>
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
> at org.apache.hadoop.util.Shell.run(Shell.java:418)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
>
> Upon getting this stack trace, I attempted to find
> documentation/information/examples about security tokens but was unable to
> find anything of note.
>
> Am I doing something dumb here and is this not related to security tokens?
> If not, is there documentation or good examples around how to use security
> tokens while launching an application?
>
> Thanks!
>
> Robert Chu
>
Re: Question about YARN security tokens
Posted by Oleg Zhurakousky <ol...@gmail.com>.
Without looking at the code its hard to say. Perhaps looking at a working
code will put you in the right direction.
For example, here is the DistributedShell from Hadoop (only few classes)
https://github.com/apache/hadoop-common/tree/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell
That is what I used a a model to dig into some of the YARN internals as I
am playing around with ideas on how to simplify YARN API
https://github.com/olegz/yaya, so you can use that code as well to see if
you can get some more information from it.
Cheers
Oleg
On Fri, Apr 25, 2014 at 12:22 AM, Robert Chu <al...@gmail.com> wrote:
> Hi Everyone,
>
> I'm new to YARN and was trying to write a simple YARN application that
> starts up an ApplicationMaster that starts up an in-process Jetty server
> just a simple test (this application requests no further resource
> containers). An attempt is made to register as an application with the
> ResourceManager while starting the ApplicationMaster process but this fails
> with an exception related to security tokens when using the MiniYARNCluster:
>
> Exception in thread "main"
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Password not
> found for ApplicationAttempt appattempt_1398290928870_0001_000002
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy8.registerApplicationMaster(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.registerApplicationMaster(AMRMClientImpl.java:196)
> at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.registerApplicationMaster(AMRMClientAsyncImpl.java:138)
> at
> org.kiji.scoring.server.yarn.YarnServiceMaster.start(YarnServiceMaster.java:228)
> at
> org.kiji.scoring.server.yarn.YarnServiceMaster.main(YarnServiceMaster.java:333)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> Password not found for ApplicationAttempt
> appattempt_1398290928870_0001_000002
> at org.apache.hadoop.ipc.Client.call(Client.java:1409)
> at org.apache.hadoop.ipc.Client.call(Client.java:1362)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy7.registerApplicationMaster(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106)
> ... 11 more
>
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
> at org.apache.hadoop.util.Shell.run(Shell.java:418)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
>
> Upon getting this stack trace, I attempted to find
> documentation/information/examples about security tokens but was unable to
> find anything of note.
>
> Am I doing something dumb here and is this not related to security tokens?
> If not, is there documentation or good examples around how to use security
> tokens while launching an application?
>
> Thanks!
>
> Robert Chu
>