You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Jasson Chenwei <yn...@gmail.com> on 2017/06/13 18:44:36 UTC
Mapreduce application failed on Distributed scheduler on YARN.
hi all
I have set up a distributed scheduler using a new feature in Hadoop-3.0. My
Hadoop version is hadoop-3.0.0-appha3. I have enabled the
opportunistic container and distributed scheduler in
yarn-site.xml following the guide. But wordcount application master failed
to launch as follow:
*2017-06-13 12:34:11,036 INFO [main] org.eclipse.jetty.server.Server:
Started @5116ms*
*2017-06-13 12:34:11,036 INFO [main] org.apache.hadoop.yarn.webapp.WebApps:
Web app mapreduce started at 45559*
*2017-06-13 12:34:11,039 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE
job_1497050650910_0004*
*2017-06-13 12:34:11,041 INFO [main]
org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class
java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000 scheduler:
class org.apache.hadoop.ipc.DefaultRpcScheduler*
*2017-06-13 12:34:11,042 INFO [Socket Reader #1 for port 36026]
org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 36026*
*2017-06-13 12:34:11,045 INFO [IPC Server Responder]
org.apache.hadoop.ipc.Server: IPC Server Responder: starting*
*2017-06-13 12:34:11,045 INFO [IPC Server listener on 36026]
org.apache.hadoop.ipc.Server: IPC Server listener on 36026: starting*
*2017-06-13 12:34:11,083 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
nodeBlacklistingEnabled:true*
*2017-06-13 12:34:11,083 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
maxTaskFailuresPerNode is 3*
*2017-06-13 12:34:11,083 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
blacklistDisablePercent is 33*
*2017-06-13 12:34:11,090 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 0% of the
mappers will be scheduled using OPPORTUNISTIC containers*
*2017-06-13 12:34:11,132 INFO [main] org.apache.hadoop.yarn.client.RMProxy:
Connecting to ResourceManager at localhost/127.0.0.1:8030
<http://127.0.0.1:8030>*
*2017-06-13 12:34:11,193 WARN [main] org.apache.hadoop.ipc.Client:
Exception encountered while connecting to the server :
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Invalid AMRMToken from appattempt_1497050650910_0004_000002*
*2017-06-13 12:34:11,203 ERROR [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Exception while
registering*
*org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
AMRMToken from appattempt_1497050650910_0004_000002*
* at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)*
* at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)*
* at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)*
* at java.lang.reflect.Constructor.newInstance(Constructor.java:423)*
* at
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)*
* at
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)*
* at
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)*
* at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)*
* at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*
* at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)*
* at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*
* at java.lang.reflect.Method.invoke(Method.java:498)*
* at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398)*
* at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)*
* at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)*
* at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)*
* at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335)*
* at com.sun.proxy.$Proxy86.registerApplicationMaster(Unknown Source)*
* at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:161)*
* at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:121)*
* at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)*
* at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)*
* at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:966)*
* at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)*
* at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)*
* at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1263)*
* at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)*
* at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1716)*
* at java.security.AccessController.doPrivileged(Native Method)*
* at javax.security.auth.Subject.doAs(Subject.java:422)*
* at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)*
* at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1712)*
* at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1643)*
*Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Invalid AMRMToken from appattempt_1497050650910_0004_000002*
* at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1481)*
* at org.apache.hadoop.ipc.Client.call(Client.java:1427)*
* at org.apache.hadoop.ipc.Client.call(Client.java:1337)*
* at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)*
* at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:1)*
* at com.sun.proxy.$Proxy85.registerApplicationMaster(Unknown Source)*
* at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:107)*
* ... 25 more*
*2017-06-13 12:34:11,207 INFO [main]
org.apache.hadoop.service.AbstractService: Service RMCommunicator failed in
state STARTED; cause:
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
AMRMToken from appattempt_1497050650910_0004_000002*
*org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
AMRMToken from appattempt_1497050650910_0004_000002*
* at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:177)*
* at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:121)*
* at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)*
* at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)*
* at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:966)*
* at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)*
* at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)*
* at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1263)*
* at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)*
* at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1716)*
* at java.security.AccessController.doPrivileged(Native Method)*
* at javax.security.auth.Subject.doAs(Subject.java:422)*
* at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)*
* at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1712)*
* at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1643)*
*Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken:
Invalid AMRMToken from appattempt_1497050650910_0004_000002*
* at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)*
* at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)*
* at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)*
* at java.lang.reflect.Constructor.newInstance(Constructor.java:423)*
* at
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)*
* at
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)*
* at
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)*
* at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)*
* at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*
* at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)*
* at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*
* at java.lang.reflect.Method.invoke(Method.java:498)*
* at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398)*
* at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)*
* at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)*
* at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)*
* at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335)*
* at com.sun.proxy.$Proxy86.registerApplicationMaster(Unknown Source)*
* at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:161)*
* ... 14 more*
Is there any configuration missing that I have a AMRMToken erros?
I appreciate your help!
Thanks!
Wei Chen
Re: Mapreduce application failed on Distributed scheduler on YARN.
Posted by Arun Suresh <ar...@gmail.com>.
Hello Jasson
We probably have to update the documentation, but when using distributed
scheduling, the mapreduce task must be submitted with the following extra
configuration:
-Dyarn.resourcemanager.scheduler.address=127.0.0.1:8049
Essentially, the AM should talk to the DistributedScheduler interceptor
running on the local NM.
Let me know if that works.
Cheers
-Arun
On Wed, Jun 14, 2017 at 12:18 AM, Konstantinos Karanasos <
kkaranasos@gmail.com> wrote:
> Hi Wei,
>
> Can you please share your yarn-site.xml and the command you used for
> running wordcount?
>
> Thanks,
> Konstantinos
>
> On Tue, Jun 13, 2017 at 11:44 Jasson Chenwei <yn...@gmail.com>
> wrote:
>
> > hi all
> >
> > I have set up a distributed scheduler using a new feature in Hadoop-3.0.
> My
> > Hadoop version is hadoop-3.0.0-appha3. I have enabled the
> > opportunistic container and distributed scheduler in
> > yarn-site.xml following the guide. But wordcount application master
> failed
> > to launch as follow:
> >
> > *2017-06-13 12:34:11,036 INFO [main] org.eclipse.jetty.server.Server:
> > Started @5116ms*
> > *2017-06-13 12:34:11,036 INFO [main] org.apache.hadoop.yarn.webapp.
> WebApps:
> > Web app mapreduce started at 45559*
> > *2017-06-13 12:34:11,039 INFO [AsyncDispatcher event handler]
> > org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator:
> JOB_CREATE
> > job_1497050650910_0004*
> > *2017-06-13 12:34:11,041 INFO [main]
> > org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class
> > java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000 scheduler:
> > class org.apache.hadoop.ipc.DefaultRpcScheduler*
> > *2017-06-13 12:34:11,042 INFO [Socket Reader #1 for port 36026]
> > org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 36026*
> > *2017-06-13 12:34:11,045 INFO [IPC Server Responder]
> > org.apache.hadoop.ipc.Server: IPC Server Responder: starting*
> > *2017-06-13 12:34:11,045 INFO [IPC Server listener on 36026]
> > org.apache.hadoop.ipc.Server: IPC Server listener on 36026: starting*
> > *2017-06-13 12:34:11,083 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> > nodeBlacklistingEnabled:true*
> > *2017-06-13 12:34:11,083 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> > maxTaskFailuresPerNode is 3*
> > *2017-06-13 12:34:11,083 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> > blacklistDisablePercent is 33*
> > *2017-06-13 12:34:11,090 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 0% of the
> > mappers will be scheduled using OPPORTUNISTIC containers*
> > *2017-06-13 12:34:11,132 INFO [main] org.apache.hadoop.yarn.client.
> RMProxy:
> > Connecting to ResourceManager at localhost/127.0.0.1:8030
> > <http://127.0.0.1:8030>*
> > *2017-06-13 12:34:11,193 WARN [main] org.apache.hadoop.ipc.Client:
> > Exception encountered while connecting to the server :
> >
> > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.
> SecretManager$InvalidToken):
> > Invalid AMRMToken from appattempt_1497050650910_0004_000002*
> > *2017-06-13 12:34:11,203 ERROR [main]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Exception while
> > registering*
> > *org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> > AMRMToken from appattempt_1497050650910_0004_000002*
> > * at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)*
> > * at
> >
> > sun.reflect.NativeConstructorAccessorImpl.newInstance(
> NativeConstructorAccessorImpl.java:62)*
> > * at
> >
> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> DelegatingConstructorAccessorImpl.java:45)*
> > * at java.lang.reflect.Constructor.newInstance(Constructor.java:423)*
> > * at
> > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(
> RPCUtil.java:53)*
> > * at
> > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(
> RPCUtil.java:80)*
> > * at
> >
> > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(
> RPCUtil.java:119)*
> > * at
> >
> > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli
> entImpl.registerApplicationMaster(ApplicationMasterProtocolPBCli
> entImpl.java:109)*
> > * at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*
> > * at
> >
> > sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)*
> > * at
> >
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)*
> > * at java.lang.reflect.Method.invoke(Method.java:498)*
> > * at
> >
> > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(
> RetryInvocationHandler.java:398)*
> > * at
> >
> > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(
> RetryInvocationHandler.java:163)*
> > * at
> >
> > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.
> invoke(RetryInvocationHandler.java:155)*
> > * at
> >
> > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(
> RetryInvocationHandler.java:95)*
> > * at
> >
> > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(
> RetryInvocationHandler.java:335)*
> > * at com.sun.proxy.$Proxy86.registerApplicationMaster(Unknown Source)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.
> register(RMCommunicator.java:161)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.
> serviceStart(RMCommunicator.java:121)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(
> RMContainerAllocator.java:280)*
> > * at
> > org.apache.hadoop.service.AbstractService.start(
> AbstractService.java:193)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.
> serviceStart(MRAppMaster.java:966)*
> > * at
> > org.apache.hadoop.service.AbstractService.start(
> AbstractService.java:193)*
> > * at
> >
> > org.apache.hadoop.service.CompositeService.serviceStart(
> CompositeService.java:120)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.
> serviceStart(MRAppMaster.java:1263)*
> > * at
> > org.apache.hadoop.service.AbstractService.start(
> AbstractService.java:193)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(
> MRAppMaster.java:1716)*
> > * at java.security.AccessController.doPrivileged(Native Method)*
> > * at javax.security.auth.Subject.doAs(Subject.java:422)*
> > * at
> >
> > org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1857)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(
> MRAppMaster.java:1712)*
> > * at
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(
> MRAppMaster.java:1643)*
> > *Caused by:
> >
> > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.
> SecretManager$InvalidToken):
> > Invalid AMRMToken from appattempt_1497050650910_0004_000002*
> > * at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1481)*
> > * at org.apache.hadoop.ipc.Client.call(Client.java:1427)*
> > * at org.apache.hadoop.ipc.Client.call(Client.java:1337)*
> > * at
> >
> > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.
> invoke(ProtobufRpcEngine.java:227)*
> > * at
> >
> > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.
> invoke(ProtobufRpcEngine.java:1)*
> > * at com.sun.proxy.$Proxy85.registerApplicationMaster(Unknown Source)*
> > * at
> >
> > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli
> entImpl.registerApplicationMaster(ApplicationMasterProtocolPBCli
> entImpl.java:107)*
> > * ... 25 more*
> > *2017-06-13 12:34:11,207 INFO [main]
> > org.apache.hadoop.service.AbstractService: Service RMCommunicator
> failed in
> > state STARTED; cause:
> > org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> > org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> > AMRMToken from appattempt_1497050650910_0004_000002*
> > *org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> > org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> > AMRMToken from appattempt_1497050650910_0004_000002*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.
> register(RMCommunicator.java:177)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.
> serviceStart(RMCommunicator.java:121)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(
> RMContainerAllocator.java:280)*
> > * at
> > org.apache.hadoop.service.AbstractService.start(
> AbstractService.java:193)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.
> serviceStart(MRAppMaster.java:966)*
> > * at
> > org.apache.hadoop.service.AbstractService.start(
> AbstractService.java:193)*
> > * at
> >
> > org.apache.hadoop.service.CompositeService.serviceStart(
> CompositeService.java:120)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.
> serviceStart(MRAppMaster.java:1263)*
> > * at
> > org.apache.hadoop.service.AbstractService.start(
> AbstractService.java:193)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(
> MRAppMaster.java:1716)*
> > * at java.security.AccessController.doPrivileged(Native Method)*
> > * at javax.security.auth.Subject.doAs(Subject.java:422)*
> > * at
> >
> > org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1857)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(
> MRAppMaster.java:1712)*
> > * at
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(
> MRAppMaster.java:1643)*
> > *Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken:
> > Invalid AMRMToken from appattempt_1497050650910_0004_000002*
> > * at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)*
> > * at
> >
> > sun.reflect.NativeConstructorAccessorImpl.newInstance(
> NativeConstructorAccessorImpl.java:62)*
> > * at
> >
> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> DelegatingConstructorAccessorImpl.java:45)*
> > * at java.lang.reflect.Constructor.newInstance(Constructor.java:423)*
> > * at
> > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(
> RPCUtil.java:53)*
> > * at
> > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(
> RPCUtil.java:80)*
> > * at
> >
> > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(
> RPCUtil.java:119)*
> > * at
> >
> > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli
> entImpl.registerApplicationMaster(ApplicationMasterProtocolPBCli
> entImpl.java:109)*
> > * at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*
> > * at
> >
> > sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)*
> > * at
> >
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)*
> > * at java.lang.reflect.Method.invoke(Method.java:498)*
> > * at
> >
> > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(
> RetryInvocationHandler.java:398)*
> > * at
> >
> > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(
> RetryInvocationHandler.java:163)*
> > * at
> >
> > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.
> invoke(RetryInvocationHandler.java:155)*
> > * at
> >
> > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(
> RetryInvocationHandler.java:95)*
> > * at
> >
> > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(
> RetryInvocationHandler.java:335)*
> > * at com.sun.proxy.$Proxy86.registerApplicationMaster(Unknown Source)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.
> register(RMCommunicator.java:161)*
> > * ... 14 more*
> >
> >
> >
> > Is there any configuration missing that I have a AMRMToken erros?
> >
> > I appreciate your help!
> >
> > Thanks!
> >
> >
> > Wei Chen
> >
> --
> Konstantinos
>
Re: Mapreduce application failed on Distributed scheduler on YARN.
Posted by Arun Suresh <ar...@gmail.com>.
Hello Jasson
We probably have to update the documentation, but when using distributed
scheduling, the mapreduce task must be submitted with the following extra
configuration:
-Dyarn.resourcemanager.scheduler.address=127.0.0.1:8049
Essentially, the AM should talk to the DistributedScheduler interceptor
running on the local NM.
Let me know if that works.
Cheers
-Arun
On Wed, Jun 14, 2017 at 12:18 AM, Konstantinos Karanasos <
kkaranasos@gmail.com> wrote:
> Hi Wei,
>
> Can you please share your yarn-site.xml and the command you used for
> running wordcount?
>
> Thanks,
> Konstantinos
>
> On Tue, Jun 13, 2017 at 11:44 Jasson Chenwei <yn...@gmail.com>
> wrote:
>
> > hi all
> >
> > I have set up a distributed scheduler using a new feature in Hadoop-3.0.
> My
> > Hadoop version is hadoop-3.0.0-appha3. I have enabled the
> > opportunistic container and distributed scheduler in
> > yarn-site.xml following the guide. But wordcount application master
> failed
> > to launch as follow:
> >
> > *2017-06-13 12:34:11,036 INFO [main] org.eclipse.jetty.server.Server:
> > Started @5116ms*
> > *2017-06-13 12:34:11,036 INFO [main] org.apache.hadoop.yarn.webapp.
> WebApps:
> > Web app mapreduce started at 45559*
> > *2017-06-13 12:34:11,039 INFO [AsyncDispatcher event handler]
> > org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator:
> JOB_CREATE
> > job_1497050650910_0004*
> > *2017-06-13 12:34:11,041 INFO [main]
> > org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class
> > java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000 scheduler:
> > class org.apache.hadoop.ipc.DefaultRpcScheduler*
> > *2017-06-13 12:34:11,042 INFO [Socket Reader #1 for port 36026]
> > org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 36026*
> > *2017-06-13 12:34:11,045 INFO [IPC Server Responder]
> > org.apache.hadoop.ipc.Server: IPC Server Responder: starting*
> > *2017-06-13 12:34:11,045 INFO [IPC Server listener on 36026]
> > org.apache.hadoop.ipc.Server: IPC Server listener on 36026: starting*
> > *2017-06-13 12:34:11,083 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> > nodeBlacklistingEnabled:true*
> > *2017-06-13 12:34:11,083 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> > maxTaskFailuresPerNode is 3*
> > *2017-06-13 12:34:11,083 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> > blacklistDisablePercent is 33*
> > *2017-06-13 12:34:11,090 INFO [main]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 0% of the
> > mappers will be scheduled using OPPORTUNISTIC containers*
> > *2017-06-13 12:34:11,132 INFO [main] org.apache.hadoop.yarn.client.
> RMProxy:
> > Connecting to ResourceManager at localhost/127.0.0.1:8030
> > <http://127.0.0.1:8030>*
> > *2017-06-13 12:34:11,193 WARN [main] org.apache.hadoop.ipc.Client:
> > Exception encountered while connecting to the server :
> >
> > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.
> SecretManager$InvalidToken):
> > Invalid AMRMToken from appattempt_1497050650910_0004_000002*
> > *2017-06-13 12:34:11,203 ERROR [main]
> > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Exception while
> > registering*
> > *org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> > AMRMToken from appattempt_1497050650910_0004_000002*
> > * at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)*
> > * at
> >
> > sun.reflect.NativeConstructorAccessorImpl.newInstance(
> NativeConstructorAccessorImpl.java:62)*
> > * at
> >
> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> DelegatingConstructorAccessorImpl.java:45)*
> > * at java.lang.reflect.Constructor.newInstance(Constructor.java:423)*
> > * at
> > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(
> RPCUtil.java:53)*
> > * at
> > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(
> RPCUtil.java:80)*
> > * at
> >
> > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(
> RPCUtil.java:119)*
> > * at
> >
> > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli
> entImpl.registerApplicationMaster(ApplicationMasterProtocolPBCli
> entImpl.java:109)*
> > * at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*
> > * at
> >
> > sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)*
> > * at
> >
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)*
> > * at java.lang.reflect.Method.invoke(Method.java:498)*
> > * at
> >
> > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(
> RetryInvocationHandler.java:398)*
> > * at
> >
> > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(
> RetryInvocationHandler.java:163)*
> > * at
> >
> > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.
> invoke(RetryInvocationHandler.java:155)*
> > * at
> >
> > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(
> RetryInvocationHandler.java:95)*
> > * at
> >
> > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(
> RetryInvocationHandler.java:335)*
> > * at com.sun.proxy.$Proxy86.registerApplicationMaster(Unknown Source)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.
> register(RMCommunicator.java:161)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.
> serviceStart(RMCommunicator.java:121)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(
> RMContainerAllocator.java:280)*
> > * at
> > org.apache.hadoop.service.AbstractService.start(
> AbstractService.java:193)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.
> serviceStart(MRAppMaster.java:966)*
> > * at
> > org.apache.hadoop.service.AbstractService.start(
> AbstractService.java:193)*
> > * at
> >
> > org.apache.hadoop.service.CompositeService.serviceStart(
> CompositeService.java:120)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.
> serviceStart(MRAppMaster.java:1263)*
> > * at
> > org.apache.hadoop.service.AbstractService.start(
> AbstractService.java:193)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(
> MRAppMaster.java:1716)*
> > * at java.security.AccessController.doPrivileged(Native Method)*
> > * at javax.security.auth.Subject.doAs(Subject.java:422)*
> > * at
> >
> > org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1857)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(
> MRAppMaster.java:1712)*
> > * at
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(
> MRAppMaster.java:1643)*
> > *Caused by:
> >
> > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.
> SecretManager$InvalidToken):
> > Invalid AMRMToken from appattempt_1497050650910_0004_000002*
> > * at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1481)*
> > * at org.apache.hadoop.ipc.Client.call(Client.java:1427)*
> > * at org.apache.hadoop.ipc.Client.call(Client.java:1337)*
> > * at
> >
> > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.
> invoke(ProtobufRpcEngine.java:227)*
> > * at
> >
> > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.
> invoke(ProtobufRpcEngine.java:1)*
> > * at com.sun.proxy.$Proxy85.registerApplicationMaster(Unknown Source)*
> > * at
> >
> > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli
> entImpl.registerApplicationMaster(ApplicationMasterProtocolPBCli
> entImpl.java:107)*
> > * ... 25 more*
> > *2017-06-13 12:34:11,207 INFO [main]
> > org.apache.hadoop.service.AbstractService: Service RMCommunicator
> failed in
> > state STARTED; cause:
> > org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> > org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> > AMRMToken from appattempt_1497050650910_0004_000002*
> > *org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> > org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> > AMRMToken from appattempt_1497050650910_0004_000002*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.
> register(RMCommunicator.java:177)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.
> serviceStart(RMCommunicator.java:121)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(
> RMContainerAllocator.java:280)*
> > * at
> > org.apache.hadoop.service.AbstractService.start(
> AbstractService.java:193)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.
> serviceStart(MRAppMaster.java:966)*
> > * at
> > org.apache.hadoop.service.AbstractService.start(
> AbstractService.java:193)*
> > * at
> >
> > org.apache.hadoop.service.CompositeService.serviceStart(
> CompositeService.java:120)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.
> serviceStart(MRAppMaster.java:1263)*
> > * at
> > org.apache.hadoop.service.AbstractService.start(
> AbstractService.java:193)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(
> MRAppMaster.java:1716)*
> > * at java.security.AccessController.doPrivileged(Native Method)*
> > * at javax.security.auth.Subject.doAs(Subject.java:422)*
> > * at
> >
> > org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1857)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(
> MRAppMaster.java:1712)*
> > * at
> > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(
> MRAppMaster.java:1643)*
> > *Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken:
> > Invalid AMRMToken from appattempt_1497050650910_0004_000002*
> > * at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)*
> > * at
> >
> > sun.reflect.NativeConstructorAccessorImpl.newInstance(
> NativeConstructorAccessorImpl.java:62)*
> > * at
> >
> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> DelegatingConstructorAccessorImpl.java:45)*
> > * at java.lang.reflect.Constructor.newInstance(Constructor.java:423)*
> > * at
> > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(
> RPCUtil.java:53)*
> > * at
> > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(
> RPCUtil.java:80)*
> > * at
> >
> > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(
> RPCUtil.java:119)*
> > * at
> >
> > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli
> entImpl.registerApplicationMaster(ApplicationMasterProtocolPBCli
> entImpl.java:109)*
> > * at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*
> > * at
> >
> > sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)*
> > * at
> >
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)*
> > * at java.lang.reflect.Method.invoke(Method.java:498)*
> > * at
> >
> > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(
> RetryInvocationHandler.java:398)*
> > * at
> >
> > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(
> RetryInvocationHandler.java:163)*
> > * at
> >
> > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.
> invoke(RetryInvocationHandler.java:155)*
> > * at
> >
> > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(
> RetryInvocationHandler.java:95)*
> > * at
> >
> > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(
> RetryInvocationHandler.java:335)*
> > * at com.sun.proxy.$Proxy86.registerApplicationMaster(Unknown Source)*
> > * at
> >
> > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.
> register(RMCommunicator.java:161)*
> > * ... 14 more*
> >
> >
> >
> > Is there any configuration missing that I have a AMRMToken erros?
> >
> > I appreciate your help!
> >
> > Thanks!
> >
> >
> > Wei Chen
> >
> --
> Konstantinos
>
Re: Mapreduce application failed on Distributed scheduler on YARN.
Posted by Konstantinos Karanasos <kk...@gmail.com>.
Hi Wei,
Can you please share your yarn-site.xml and the command you used for
running wordcount?
Thanks,
Konstantinos
On Tue, Jun 13, 2017 at 11:44 Jasson Chenwei <yn...@gmail.com>
wrote:
> hi all
>
> I have set up a distributed scheduler using a new feature in Hadoop-3.0. My
> Hadoop version is hadoop-3.0.0-appha3. I have enabled the
> opportunistic container and distributed scheduler in
> yarn-site.xml following the guide. But wordcount application master failed
> to launch as follow:
>
> *2017-06-13 12:34:11,036 INFO [main] org.eclipse.jetty.server.Server:
> Started @5116ms*
> *2017-06-13 12:34:11,036 INFO [main] org.apache.hadoop.yarn.webapp.WebApps:
> Web app mapreduce started at 45559*
> *2017-06-13 12:34:11,039 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE
> job_1497050650910_0004*
> *2017-06-13 12:34:11,041 INFO [main]
> org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class
> java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000 scheduler:
> class org.apache.hadoop.ipc.DefaultRpcScheduler*
> *2017-06-13 12:34:11,042 INFO [Socket Reader #1 for port 36026]
> org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 36026*
> *2017-06-13 12:34:11,045 INFO [IPC Server Responder]
> org.apache.hadoop.ipc.Server: IPC Server Responder: starting*
> *2017-06-13 12:34:11,045 INFO [IPC Server listener on 36026]
> org.apache.hadoop.ipc.Server: IPC Server listener on 36026: starting*
> *2017-06-13 12:34:11,083 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> nodeBlacklistingEnabled:true*
> *2017-06-13 12:34:11,083 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> maxTaskFailuresPerNode is 3*
> *2017-06-13 12:34:11,083 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> blacklistDisablePercent is 33*
> *2017-06-13 12:34:11,090 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 0% of the
> mappers will be scheduled using OPPORTUNISTIC containers*
> *2017-06-13 12:34:11,132 INFO [main] org.apache.hadoop.yarn.client.RMProxy:
> Connecting to ResourceManager at localhost/127.0.0.1:8030
> <http://127.0.0.1:8030>*
> *2017-06-13 12:34:11,193 WARN [main] org.apache.hadoop.ipc.Client:
> Exception encountered while connecting to the server :
>
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1497050650910_0004_000002*
> *2017-06-13 12:34:11,203 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Exception while
> registering*
> *org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> AMRMToken from appattempt_1497050650910_0004_000002*
> * at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)*
> * at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)*
> * at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)*
> * at java.lang.reflect.Constructor.newInstance(Constructor.java:423)*
> * at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)*
> * at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)*
> * at
>
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)*
> * at
>
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)*
> * at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*
> * at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)*
> * at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*
> * at java.lang.reflect.Method.invoke(Method.java:498)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335)*
> * at com.sun.proxy.$Proxy86.registerApplicationMaster(Unknown Source)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:161)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:121)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)*
> * at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:966)*
> * at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)*
> * at
>
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1263)*
> * at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1716)*
> * at java.security.AccessController.doPrivileged(Native Method)*
> * at javax.security.auth.Subject.doAs(Subject.java:422)*
> * at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1712)*
> * at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1643)*
> *Caused by:
>
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1497050650910_0004_000002*
> * at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1481)*
> * at org.apache.hadoop.ipc.Client.call(Client.java:1427)*
> * at org.apache.hadoop.ipc.Client.call(Client.java:1337)*
> * at
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)*
> * at
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:1)*
> * at com.sun.proxy.$Proxy85.registerApplicationMaster(Unknown Source)*
> * at
>
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:107)*
> * ... 25 more*
> *2017-06-13 12:34:11,207 INFO [main]
> org.apache.hadoop.service.AbstractService: Service RMCommunicator failed in
> state STARTED; cause:
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> AMRMToken from appattempt_1497050650910_0004_000002*
> *org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> AMRMToken from appattempt_1497050650910_0004_000002*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:177)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:121)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)*
> * at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:966)*
> * at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)*
> * at
>
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1263)*
> * at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1716)*
> * at java.security.AccessController.doPrivileged(Native Method)*
> * at javax.security.auth.Subject.doAs(Subject.java:422)*
> * at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1712)*
> * at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1643)*
> *Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken:
> Invalid AMRMToken from appattempt_1497050650910_0004_000002*
> * at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)*
> * at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)*
> * at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)*
> * at java.lang.reflect.Constructor.newInstance(Constructor.java:423)*
> * at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)*
> * at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)*
> * at
>
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)*
> * at
>
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)*
> * at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*
> * at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)*
> * at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*
> * at java.lang.reflect.Method.invoke(Method.java:498)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335)*
> * at com.sun.proxy.$Proxy86.registerApplicationMaster(Unknown Source)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:161)*
> * ... 14 more*
>
>
>
> Is there any configuration missing that I have a AMRMToken erros?
>
> I appreciate your help!
>
> Thanks!
>
>
> Wei Chen
>
--
Konstantinos
Re: Mapreduce application failed on Distributed scheduler on YARN.
Posted by Konstantinos Karanasos <kk...@gmail.com>.
Hi Wei,
Can you please share your yarn-site.xml and the command you used for
running wordcount?
Thanks,
Konstantinos
On Tue, Jun 13, 2017 at 11:44 Jasson Chenwei <yn...@gmail.com>
wrote:
> hi all
>
> I have set up a distributed scheduler using a new feature in Hadoop-3.0. My
> Hadoop version is hadoop-3.0.0-appha3. I have enabled the
> opportunistic container and distributed scheduler in
> yarn-site.xml following the guide. But wordcount application master failed
> to launch as follow:
>
> *2017-06-13 12:34:11,036 INFO [main] org.eclipse.jetty.server.Server:
> Started @5116ms*
> *2017-06-13 12:34:11,036 INFO [main] org.apache.hadoop.yarn.webapp.WebApps:
> Web app mapreduce started at 45559*
> *2017-06-13 12:34:11,039 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE
> job_1497050650910_0004*
> *2017-06-13 12:34:11,041 INFO [main]
> org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class
> java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000 scheduler:
> class org.apache.hadoop.ipc.DefaultRpcScheduler*
> *2017-06-13 12:34:11,042 INFO [Socket Reader #1 for port 36026]
> org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 36026*
> *2017-06-13 12:34:11,045 INFO [IPC Server Responder]
> org.apache.hadoop.ipc.Server: IPC Server Responder: starting*
> *2017-06-13 12:34:11,045 INFO [IPC Server listener on 36026]
> org.apache.hadoop.ipc.Server: IPC Server listener on 36026: starting*
> *2017-06-13 12:34:11,083 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> nodeBlacklistingEnabled:true*
> *2017-06-13 12:34:11,083 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> maxTaskFailuresPerNode is 3*
> *2017-06-13 12:34:11,083 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> blacklistDisablePercent is 33*
> *2017-06-13 12:34:11,090 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 0% of the
> mappers will be scheduled using OPPORTUNISTIC containers*
> *2017-06-13 12:34:11,132 INFO [main] org.apache.hadoop.yarn.client.RMProxy:
> Connecting to ResourceManager at localhost/127.0.0.1:8030
> <http://127.0.0.1:8030>*
> *2017-06-13 12:34:11,193 WARN [main] org.apache.hadoop.ipc.Client:
> Exception encountered while connecting to the server :
>
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1497050650910_0004_000002*
> *2017-06-13 12:34:11,203 ERROR [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Exception while
> registering*
> *org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> AMRMToken from appattempt_1497050650910_0004_000002*
> * at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)*
> * at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)*
> * at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)*
> * at java.lang.reflect.Constructor.newInstance(Constructor.java:423)*
> * at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)*
> * at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)*
> * at
>
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)*
> * at
>
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)*
> * at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*
> * at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)*
> * at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*
> * at java.lang.reflect.Method.invoke(Method.java:498)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335)*
> * at com.sun.proxy.$Proxy86.registerApplicationMaster(Unknown Source)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:161)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:121)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)*
> * at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:966)*
> * at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)*
> * at
>
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1263)*
> * at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1716)*
> * at java.security.AccessController.doPrivileged(Native Method)*
> * at javax.security.auth.Subject.doAs(Subject.java:422)*
> * at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1712)*
> * at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1643)*
> *Caused by:
>
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> Invalid AMRMToken from appattempt_1497050650910_0004_000002*
> * at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1481)*
> * at org.apache.hadoop.ipc.Client.call(Client.java:1427)*
> * at org.apache.hadoop.ipc.Client.call(Client.java:1337)*
> * at
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)*
> * at
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:1)*
> * at com.sun.proxy.$Proxy85.registerApplicationMaster(Unknown Source)*
> * at
>
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:107)*
> * ... 25 more*
> *2017-06-13 12:34:11,207 INFO [main]
> org.apache.hadoop.service.AbstractService: Service RMCommunicator failed in
> state STARTED; cause:
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> AMRMToken from appattempt_1497050650910_0004_000002*
> *org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid
> AMRMToken from appattempt_1497050650910_0004_000002*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:177)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:121)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)*
> * at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:966)*
> * at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)*
> * at
>
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1263)*
> * at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1716)*
> * at java.security.AccessController.doPrivileged(Native Method)*
> * at javax.security.auth.Subject.doAs(Subject.java:422)*
> * at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1712)*
> * at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1643)*
> *Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken:
> Invalid AMRMToken from appattempt_1497050650910_0004_000002*
> * at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)*
> * at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)*
> * at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)*
> * at java.lang.reflect.Constructor.newInstance(Constructor.java:423)*
> * at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)*
> * at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)*
> * at
>
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)*
> * at
>
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)*
> * at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*
> * at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)*
> * at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)*
> * at java.lang.reflect.Method.invoke(Method.java:498)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)*
> * at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335)*
> * at com.sun.proxy.$Proxy86.registerApplicationMaster(Unknown Source)*
> * at
>
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:161)*
> * ... 14 more*
>
>
>
> Is there any configuration missing that I have a AMRMToken erros?
>
> I appreciate your help!
>
> Thanks!
>
>
> Wei Chen
>
--
Konstantinos