MiniYarnCluster multiple nodemanager support

[Shawna Qian <sh...@yahoo-inc.com>]

Hello:

I am trying to extend the MiniYarnCluster to support Multiple NodeManager.  but I am getting this address already in use for ContainerManger registration.  Seems like container manager is using a fixed address 0.0.0.0:4344.  Should this be made configurable?


Also, I am not sure if this makes it is impossible to support multiple nodemanager in the miniyarncluster for a single machine?

Thx
Shawna


Exception in thread "Thread-94" org.apache.hadoop.yarn.YarnException: Failed to Start org.apache.hadoop.yarn.server.nodemanager.NodeManager
        at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:80)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.start(NodeManager.java:146)
        at org.apache.hadoop.yarn.server.MyMiniYARNCluster$NewThread.run(MyMiniYARNCluster.java:170)
Caused by: org.apache.hadoop.yarn.YarnException: Failed to Start org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
        at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:80)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.start(ContainerManagerImpl.java:219)
        at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
        ... 2 more
Caused by: org.apache.hadoop.yarn.YarnException: java.net.BindException: Problem binding to /0.0.0.0:4344 : Address already in use
        at org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:125)
        at org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYhttps://sp1-webmail.corp.yahoo.com/owa/?ae=Item&t=IPM.Note&s=Draft&a=Open#arnProtoRPC.java:68)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.createServer(ResourceLocalizationService.java:245)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.start(ResourceLocalizationService.java:228)
        at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
        ... 4 more
Caused by: java.net.BindException: Problem binding to /0.0.0.0:4344 : Address already in use
        at org.apache.hadoop.ipc.Server.bind(Server.java:231)
        at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:320)
        at org.apache.hadoop.ipc.Server.<init>(Server.java:1597)
        at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:576)
        at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.<init>(ProtoOverHadoopRpcEngine.java:314)
        at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine.getServer(ProtoOverHadoopRpcEngine.java:383)
        at org.apache.hadoop.ipc.RPC.getServer(RPC.java:550)
        at org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.createServer(RpcServerFactoryPBImpl.java:155)
        at org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:118)
        ... 8 more
Caused by: java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
        at org.apache.hadoop.ipc.Server.bind(Server.java:229)
        ... 16 more
Exception in thread "Thread-87" java.lang.NullPointerException
        at org.apache.hadoop.yarn.event.AsyncDispatcher.stop(AsyncDispatcher.java:93)
        at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.stop(NodeManager.java:151)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager$1.run(NodeManager.java:129)