You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Edward J. Yoon" <ed...@apache.org> on 2015/04/28 02:08:42 UTC
Bug in Netty-based RPC
I tried to run BSP job using netty-based RPC instead of message
bundle, but I received "too many open files".
--
attempt_201504280858_0001_000017_0: 15/04/28 08:28:17 INFO
ipc.AsyncClient: AsyncClient startup
attempt_201504280858_0001_000017_0: 15/04/28 08:28:21 ERROR
bsp.BSPTask: Error running bsp setup and bsp function.
attempt_201504280858_0001_000017_0: java.lang.IllegalStateException:
failed to create a child event loop
attempt_201504280858_0001_000017_0: at
io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
attempt_201504280858_0001_000017_0: at
io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
attempt_201504280858_0001_000017_0: at
io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:61)
attempt_201504280858_0001_000017_0: at
io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:52)
attempt_201504280858_0001_000017_0: at
io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:44)
attempt_201504280858_0001_000017_0: at
io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:36)
attempt_201504280858_0001_000017_0: at
org.apache.hama.ipc.AsyncClient$Connection.<init>(AsyncClient.java:189)
attempt_201504280858_0001_000017_0: at
org.apache.hama.ipc.AsyncClient.getConnection(AsyncClient.java:989)
attempt_201504280858_0001_000017_0: at
org.apache.hama.ipc.AsyncClient.call(AsyncClient.java:838)
attempt_201504280858_0001_000017_0: at
org.apache.hama.ipc.AsyncRPC$Invoker.invoke(AsyncRPC.java:261)
attempt_201504280858_0001_000017_0: at
com.sun.proxy.$Proxy14.getProtocolVersion(Unknown Source)
attempt_201504280858_0001_000017_0: at
org.apache.hama.ipc.AsyncRPC.checkVersion(AsyncRPC.java:524)
attempt_201504280858_0001_000017_0: at
org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:509)
attempt_201504280858_0001_000017_0: at
org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:477)
attempt_201504280858_0001_000017_0: at
org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:435)
attempt_201504280858_0001_000017_0: at
org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:545)
attempt_201504280858_0001_000017_0: at
org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.getBSPPeerConnection(HamaAsyncMessageManagerImpl.java:155)
attempt_201504280858_0001_000017_0: at
org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.transfer(HamaAsyncMessageManagerImpl.java:203)
attempt_201504280858_0001_000017_0: at
org.apache.hama.bsp.BSPPeerImpl.sendDirectly(BSPPeerImpl.java:382)
attempt_201504280858_0001_000017_0: at
org.apache.hama.bsp.BSPPeerImpl.send(BSPPeerImpl.java:364)
attempt_201504280858_0001_000017_0: at
org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:467)
attempt_201504280858_0001_000017_0: at
org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:128)
attempt_201504280858_0001_000017_0: at
org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170)
attempt_201504280858_0001_000017_0: at
org.apache.hama.bsp.BSPTask.run(BSPTask.java:144)
attempt_201504280858_0001_000017_0: at
org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255)
attempt_201504280858_0001_000017_0: Caused by:
io.netty.channel.ChannelException: failed to open a new selector
attempt_201504280858_0001_000017_0: at
io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:128)
attempt_201504280858_0001_000017_0: at
io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:120)
attempt_201504280858_0001_000017_0: at
io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:87)
attempt_201504280858_0001_000017_0: at
io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
attempt_201504280858_0001_000017_0: ... 24 more
attempt_201504280858_0001_000017_0: Caused by: java.io.IOException:
Too many open files
attempt_201504280858_0001_000017_0: at sun.nio.ch.IOUtil.makePipe(Native Method)
attempt_201504280858_0001_000017_0: at
sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65)
attempt_201504280858_0001_000017_0: at
sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
attempt_201504280858_0001_000017_0: at
io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:126)
attempt_201504280858_0001_000017_0: ... 27 more
attempt_201504280858_0001_000017_0: 15/04/28 08:28:21 INFO
ipc.AsyncServer: AsyncServer gracefully shutdown
--
Best Regards, Edward J. Yoon
RE: Bug in Netty-based RPC
Posted by "Edward J. Yoon" <ed...@samsung.com>.
Simply reinstalled Ubuntu and now it works fine. :D
> Did you ever find the cause?
As I attached the link, it look like a issue on Netty side.
BTW, I feel like I need to explain my experiment here. Few weeks ago, I
committed HAMA-954 so that you can turn on/off the message bundle behavior. If
you set hama.bsp.messenger.bundle to true, the messenger system package the
messages into single BSPMessageBundle object and send it at once. Meanwhile,
if you set hama.bsp.messenger.bundle to false, the system send a message one
by one asynchronously. I think the advantage of netty's non-blocking RPC can
be maximized in this case (I couldn't test it yet).
--
Best Regards, Edward J. Yoon
-----Original Message-----
From: ByungSeok Min [mailto:byeongseok.min@gmail.com]
Sent: Friday, May 08, 2015 5:15 PM
To: dev@hama.apache.org
Subject: Re: Bug in Netty-based RPC
Hi~ Mr Yoon.
Is your machine ok?
Did you ever find the cause?
* My Work in progress is going to be finished well.
* I'll be back soon.
Best Regards!
2015-04-28 12:58 GMT+09:00 Edward J. Yoon <ed...@apache.org>:
> I just found this
>
> https://github.com/netty/netty/commit/dcf78961724abf97fea9e768cca02ba65146b324
>
> On Tue, Apr 28, 2015 at 11:39 AM, Chia-Hung Lin <cl...@googlemail.com>
> wrote:
> > Have you checked limit.conf?
> >
> > From the message it looks like the files opened at underlying system
> > exceed its default limit.
> >
> > On 28 April 2015 at 08:08, Edward J. Yoon <ed...@apache.org> wrote:
> >> I tried to run BSP job using netty-based RPC instead of message
> >> bundle, but I received "too many open files".
> >>
> >> --
> >>
> >> attempt_201504280858_0001_000017_0: 15/04/28 08:28:17 INFO
> >> ipc.AsyncClient: AsyncClient startup
> >> attempt_201504280858_0001_000017_0: 15/04/28 08:28:21 ERROR
> >> bsp.BSPTask: Error running bsp setup and bsp function.
> >> attempt_201504280858_0001_000017_0: java.lang.IllegalStateException:
> >> failed to create a child event loop
> >> attempt_201504280858_0001_000017_0: at
> >>
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
> >> attempt_201504280858_0001_000017_0: at
> >>
> io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
> >> attempt_201504280858_0001_000017_0: at
> >> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:61)
> >> attempt_201504280858_0001_000017_0: at
> >> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:52)
> >> attempt_201504280858_0001_000017_0: at
> >> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:44)
> >> attempt_201504280858_0001_000017_0: at
> >> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:36)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.ipc.AsyncClient$Connection.<init>(AsyncClient.java:189)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.ipc.AsyncClient.getConnection(AsyncClient.java:989)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.ipc.AsyncClient.call(AsyncClient.java:838)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.ipc.AsyncRPC$Invoker.invoke(AsyncRPC.java:261)
> >> attempt_201504280858_0001_000017_0: at
> >> com.sun.proxy.$Proxy14.getProtocolVersion(Unknown Source)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.ipc.AsyncRPC.checkVersion(AsyncRPC.java:524)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:509)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:477)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:435)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:545)
> >> attempt_201504280858_0001_000017_0: at
> >>
> org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.getBSPPeerConnection(HamaAsyncMessageManagerImpl.java:155)
> >> attempt_201504280858_0001_000017_0: at
> >>
> org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.transfer(HamaAsyncMessageManagerImpl.java:203)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.bsp.BSPPeerImpl.sendDirectly(BSPPeerImpl.java:382)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.bsp.BSPPeerImpl.send(BSPPeerImpl.java:364)
> >> attempt_201504280858_0001_000017_0: at
> >>
> org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:467)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:128)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.bsp.BSPTask.run(BSPTask.java:144)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255)
> >> attempt_201504280858_0001_000017_0: Caused by:
> >> io.netty.channel.ChannelException: failed to open a new selector
> >> attempt_201504280858_0001_000017_0: at
> >> io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:128)
> >> attempt_201504280858_0001_000017_0: at
> >> io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:120)
> >> attempt_201504280858_0001_000017_0: at
> >>
> io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:87)
> >> attempt_201504280858_0001_000017_0: at
> >>
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
> >> attempt_201504280858_0001_000017_0: ... 24 more
> >> attempt_201504280858_0001_000017_0: Caused by: java.io.IOException:
> >> Too many open files
> >> attempt_201504280858_0001_000017_0: at
> sun.nio.ch.IOUtil.makePipe(Native Method)
> >> attempt_201504280858_0001_000017_0: at
> >> sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65)
> >> attempt_201504280858_0001_000017_0: at
> >>
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
> >> attempt_201504280858_0001_000017_0: at
> >> io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:126)
> >> attempt_201504280858_0001_000017_0: ... 27 more
> >> attempt_201504280858_0001_000017_0: 15/04/28 08:28:21 INFO
> >> ipc.AsyncServer: AsyncServer gracefully shutdown
> >>
> >>
> >> --
> >> Best Regards, Edward J. Yoon
>
>
>
> --
> Best Regards, Edward J. Yoon
>
Re: Bug in Netty-based RPC
Posted by ByungSeok Min <by...@gmail.com>.
Hi~ Mr Yoon.
Is your machine ok?
Did you ever find the cause?
* My Work in progress is going to be finished well.
* I'll be back soon.
Best Regards!
2015-04-28 12:58 GMT+09:00 Edward J. Yoon <ed...@apache.org>:
> I just found this
>
> https://github.com/netty/netty/commit/dcf78961724abf97fea9e768cca02ba65146b324
>
> On Tue, Apr 28, 2015 at 11:39 AM, Chia-Hung Lin <cl...@googlemail.com>
> wrote:
> > Have you checked limit.conf?
> >
> > From the message it looks like the files opened at underlying system
> > exceed its default limit.
> >
> > On 28 April 2015 at 08:08, Edward J. Yoon <ed...@apache.org> wrote:
> >> I tried to run BSP job using netty-based RPC instead of message
> >> bundle, but I received "too many open files".
> >>
> >> --
> >>
> >> attempt_201504280858_0001_000017_0: 15/04/28 08:28:17 INFO
> >> ipc.AsyncClient: AsyncClient startup
> >> attempt_201504280858_0001_000017_0: 15/04/28 08:28:21 ERROR
> >> bsp.BSPTask: Error running bsp setup and bsp function.
> >> attempt_201504280858_0001_000017_0: java.lang.IllegalStateException:
> >> failed to create a child event loop
> >> attempt_201504280858_0001_000017_0: at
> >>
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
> >> attempt_201504280858_0001_000017_0: at
> >>
> io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
> >> attempt_201504280858_0001_000017_0: at
> >> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:61)
> >> attempt_201504280858_0001_000017_0: at
> >> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:52)
> >> attempt_201504280858_0001_000017_0: at
> >> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:44)
> >> attempt_201504280858_0001_000017_0: at
> >> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:36)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.ipc.AsyncClient$Connection.<init>(AsyncClient.java:189)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.ipc.AsyncClient.getConnection(AsyncClient.java:989)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.ipc.AsyncClient.call(AsyncClient.java:838)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.ipc.AsyncRPC$Invoker.invoke(AsyncRPC.java:261)
> >> attempt_201504280858_0001_000017_0: at
> >> com.sun.proxy.$Proxy14.getProtocolVersion(Unknown Source)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.ipc.AsyncRPC.checkVersion(AsyncRPC.java:524)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:509)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:477)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:435)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:545)
> >> attempt_201504280858_0001_000017_0: at
> >>
> org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.getBSPPeerConnection(HamaAsyncMessageManagerImpl.java:155)
> >> attempt_201504280858_0001_000017_0: at
> >>
> org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.transfer(HamaAsyncMessageManagerImpl.java:203)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.bsp.BSPPeerImpl.sendDirectly(BSPPeerImpl.java:382)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.bsp.BSPPeerImpl.send(BSPPeerImpl.java:364)
> >> attempt_201504280858_0001_000017_0: at
> >>
> org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:467)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:128)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.bsp.BSPTask.run(BSPTask.java:144)
> >> attempt_201504280858_0001_000017_0: at
> >> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255)
> >> attempt_201504280858_0001_000017_0: Caused by:
> >> io.netty.channel.ChannelException: failed to open a new selector
> >> attempt_201504280858_0001_000017_0: at
> >> io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:128)
> >> attempt_201504280858_0001_000017_0: at
> >> io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:120)
> >> attempt_201504280858_0001_000017_0: at
> >>
> io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:87)
> >> attempt_201504280858_0001_000017_0: at
> >>
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
> >> attempt_201504280858_0001_000017_0: ... 24 more
> >> attempt_201504280858_0001_000017_0: Caused by: java.io.IOException:
> >> Too many open files
> >> attempt_201504280858_0001_000017_0: at
> sun.nio.ch.IOUtil.makePipe(Native Method)
> >> attempt_201504280858_0001_000017_0: at
> >> sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65)
> >> attempt_201504280858_0001_000017_0: at
> >>
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
> >> attempt_201504280858_0001_000017_0: at
> >> io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:126)
> >> attempt_201504280858_0001_000017_0: ... 27 more
> >> attempt_201504280858_0001_000017_0: 15/04/28 08:28:21 INFO
> >> ipc.AsyncServer: AsyncServer gracefully shutdown
> >>
> >>
> >> --
> >> Best Regards, Edward J. Yoon
>
>
>
> --
> Best Regards, Edward J. Yoon
>
Re: Bug in Netty-based RPC
Posted by "Edward J. Yoon" <ed...@apache.org>.
wow, my machine has been crashed.
On Tue, Apr 28, 2015 at 12:58 PM, Edward J. Yoon <ed...@apache.org> wrote:
> I just found this
> https://github.com/netty/netty/commit/dcf78961724abf97fea9e768cca02ba65146b324
>
> On Tue, Apr 28, 2015 at 11:39 AM, Chia-Hung Lin <cl...@googlemail.com> wrote:
>> Have you checked limit.conf?
>>
>> From the message it looks like the files opened at underlying system
>> exceed its default limit.
>>
>> On 28 April 2015 at 08:08, Edward J. Yoon <ed...@apache.org> wrote:
>>> I tried to run BSP job using netty-based RPC instead of message
>>> bundle, but I received "too many open files".
>>>
>>> --
>>>
>>> attempt_201504280858_0001_000017_0: 15/04/28 08:28:17 INFO
>>> ipc.AsyncClient: AsyncClient startup
>>> attempt_201504280858_0001_000017_0: 15/04/28 08:28:21 ERROR
>>> bsp.BSPTask: Error running bsp setup and bsp function.
>>> attempt_201504280858_0001_000017_0: java.lang.IllegalStateException:
>>> failed to create a child event loop
>>> attempt_201504280858_0001_000017_0: at
>>> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
>>> attempt_201504280858_0001_000017_0: at
>>> io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
>>> attempt_201504280858_0001_000017_0: at
>>> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:61)
>>> attempt_201504280858_0001_000017_0: at
>>> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:52)
>>> attempt_201504280858_0001_000017_0: at
>>> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:44)
>>> attempt_201504280858_0001_000017_0: at
>>> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:36)
>>> attempt_201504280858_0001_000017_0: at
>>> org.apache.hama.ipc.AsyncClient$Connection.<init>(AsyncClient.java:189)
>>> attempt_201504280858_0001_000017_0: at
>>> org.apache.hama.ipc.AsyncClient.getConnection(AsyncClient.java:989)
>>> attempt_201504280858_0001_000017_0: at
>>> org.apache.hama.ipc.AsyncClient.call(AsyncClient.java:838)
>>> attempt_201504280858_0001_000017_0: at
>>> org.apache.hama.ipc.AsyncRPC$Invoker.invoke(AsyncRPC.java:261)
>>> attempt_201504280858_0001_000017_0: at
>>> com.sun.proxy.$Proxy14.getProtocolVersion(Unknown Source)
>>> attempt_201504280858_0001_000017_0: at
>>> org.apache.hama.ipc.AsyncRPC.checkVersion(AsyncRPC.java:524)
>>> attempt_201504280858_0001_000017_0: at
>>> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:509)
>>> attempt_201504280858_0001_000017_0: at
>>> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:477)
>>> attempt_201504280858_0001_000017_0: at
>>> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:435)
>>> attempt_201504280858_0001_000017_0: at
>>> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:545)
>>> attempt_201504280858_0001_000017_0: at
>>> org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.getBSPPeerConnection(HamaAsyncMessageManagerImpl.java:155)
>>> attempt_201504280858_0001_000017_0: at
>>> org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.transfer(HamaAsyncMessageManagerImpl.java:203)
>>> attempt_201504280858_0001_000017_0: at
>>> org.apache.hama.bsp.BSPPeerImpl.sendDirectly(BSPPeerImpl.java:382)
>>> attempt_201504280858_0001_000017_0: at
>>> org.apache.hama.bsp.BSPPeerImpl.send(BSPPeerImpl.java:364)
>>> attempt_201504280858_0001_000017_0: at
>>> org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:467)
>>> attempt_201504280858_0001_000017_0: at
>>> org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:128)
>>> attempt_201504280858_0001_000017_0: at
>>> org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170)
>>> attempt_201504280858_0001_000017_0: at
>>> org.apache.hama.bsp.BSPTask.run(BSPTask.java:144)
>>> attempt_201504280858_0001_000017_0: at
>>> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255)
>>> attempt_201504280858_0001_000017_0: Caused by:
>>> io.netty.channel.ChannelException: failed to open a new selector
>>> attempt_201504280858_0001_000017_0: at
>>> io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:128)
>>> attempt_201504280858_0001_000017_0: at
>>> io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:120)
>>> attempt_201504280858_0001_000017_0: at
>>> io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:87)
>>> attempt_201504280858_0001_000017_0: at
>>> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
>>> attempt_201504280858_0001_000017_0: ... 24 more
>>> attempt_201504280858_0001_000017_0: Caused by: java.io.IOException:
>>> Too many open files
>>> attempt_201504280858_0001_000017_0: at sun.nio.ch.IOUtil.makePipe(Native Method)
>>> attempt_201504280858_0001_000017_0: at
>>> sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65)
>>> attempt_201504280858_0001_000017_0: at
>>> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
>>> attempt_201504280858_0001_000017_0: at
>>> io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:126)
>>> attempt_201504280858_0001_000017_0: ... 27 more
>>> attempt_201504280858_0001_000017_0: 15/04/28 08:28:21 INFO
>>> ipc.AsyncServer: AsyncServer gracefully shutdown
>>>
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>
>
>
> --
> Best Regards, Edward J. Yoon
--
Best Regards, Edward J. Yoon
Re: Bug in Netty-based RPC
Posted by "Edward J. Yoon" <ed...@apache.org>.
I just found this
https://github.com/netty/netty/commit/dcf78961724abf97fea9e768cca02ba65146b324
On Tue, Apr 28, 2015 at 11:39 AM, Chia-Hung Lin <cl...@googlemail.com> wrote:
> Have you checked limit.conf?
>
> From the message it looks like the files opened at underlying system
> exceed its default limit.
>
> On 28 April 2015 at 08:08, Edward J. Yoon <ed...@apache.org> wrote:
>> I tried to run BSP job using netty-based RPC instead of message
>> bundle, but I received "too many open files".
>>
>> --
>>
>> attempt_201504280858_0001_000017_0: 15/04/28 08:28:17 INFO
>> ipc.AsyncClient: AsyncClient startup
>> attempt_201504280858_0001_000017_0: 15/04/28 08:28:21 ERROR
>> bsp.BSPTask: Error running bsp setup and bsp function.
>> attempt_201504280858_0001_000017_0: java.lang.IllegalStateException:
>> failed to create a child event loop
>> attempt_201504280858_0001_000017_0: at
>> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
>> attempt_201504280858_0001_000017_0: at
>> io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
>> attempt_201504280858_0001_000017_0: at
>> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:61)
>> attempt_201504280858_0001_000017_0: at
>> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:52)
>> attempt_201504280858_0001_000017_0: at
>> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:44)
>> attempt_201504280858_0001_000017_0: at
>> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:36)
>> attempt_201504280858_0001_000017_0: at
>> org.apache.hama.ipc.AsyncClient$Connection.<init>(AsyncClient.java:189)
>> attempt_201504280858_0001_000017_0: at
>> org.apache.hama.ipc.AsyncClient.getConnection(AsyncClient.java:989)
>> attempt_201504280858_0001_000017_0: at
>> org.apache.hama.ipc.AsyncClient.call(AsyncClient.java:838)
>> attempt_201504280858_0001_000017_0: at
>> org.apache.hama.ipc.AsyncRPC$Invoker.invoke(AsyncRPC.java:261)
>> attempt_201504280858_0001_000017_0: at
>> com.sun.proxy.$Proxy14.getProtocolVersion(Unknown Source)
>> attempt_201504280858_0001_000017_0: at
>> org.apache.hama.ipc.AsyncRPC.checkVersion(AsyncRPC.java:524)
>> attempt_201504280858_0001_000017_0: at
>> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:509)
>> attempt_201504280858_0001_000017_0: at
>> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:477)
>> attempt_201504280858_0001_000017_0: at
>> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:435)
>> attempt_201504280858_0001_000017_0: at
>> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:545)
>> attempt_201504280858_0001_000017_0: at
>> org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.getBSPPeerConnection(HamaAsyncMessageManagerImpl.java:155)
>> attempt_201504280858_0001_000017_0: at
>> org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.transfer(HamaAsyncMessageManagerImpl.java:203)
>> attempt_201504280858_0001_000017_0: at
>> org.apache.hama.bsp.BSPPeerImpl.sendDirectly(BSPPeerImpl.java:382)
>> attempt_201504280858_0001_000017_0: at
>> org.apache.hama.bsp.BSPPeerImpl.send(BSPPeerImpl.java:364)
>> attempt_201504280858_0001_000017_0: at
>> org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:467)
>> attempt_201504280858_0001_000017_0: at
>> org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:128)
>> attempt_201504280858_0001_000017_0: at
>> org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170)
>> attempt_201504280858_0001_000017_0: at
>> org.apache.hama.bsp.BSPTask.run(BSPTask.java:144)
>> attempt_201504280858_0001_000017_0: at
>> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255)
>> attempt_201504280858_0001_000017_0: Caused by:
>> io.netty.channel.ChannelException: failed to open a new selector
>> attempt_201504280858_0001_000017_0: at
>> io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:128)
>> attempt_201504280858_0001_000017_0: at
>> io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:120)
>> attempt_201504280858_0001_000017_0: at
>> io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:87)
>> attempt_201504280858_0001_000017_0: at
>> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
>> attempt_201504280858_0001_000017_0: ... 24 more
>> attempt_201504280858_0001_000017_0: Caused by: java.io.IOException:
>> Too many open files
>> attempt_201504280858_0001_000017_0: at sun.nio.ch.IOUtil.makePipe(Native Method)
>> attempt_201504280858_0001_000017_0: at
>> sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65)
>> attempt_201504280858_0001_000017_0: at
>> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
>> attempt_201504280858_0001_000017_0: at
>> io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:126)
>> attempt_201504280858_0001_000017_0: ... 27 more
>> attempt_201504280858_0001_000017_0: 15/04/28 08:28:21 INFO
>> ipc.AsyncServer: AsyncServer gracefully shutdown
>>
>>
>> --
>> Best Regards, Edward J. Yoon
--
Best Regards, Edward J. Yoon
Re: Bug in Netty-based RPC
Posted by Chia-Hung Lin <cl...@googlemail.com>.
Have you checked limit.conf?
>From the message it looks like the files opened at underlying system
exceed its default limit.
On 28 April 2015 at 08:08, Edward J. Yoon <ed...@apache.org> wrote:
> I tried to run BSP job using netty-based RPC instead of message
> bundle, but I received "too many open files".
>
> --
>
> attempt_201504280858_0001_000017_0: 15/04/28 08:28:17 INFO
> ipc.AsyncClient: AsyncClient startup
> attempt_201504280858_0001_000017_0: 15/04/28 08:28:21 ERROR
> bsp.BSPTask: Error running bsp setup and bsp function.
> attempt_201504280858_0001_000017_0: java.lang.IllegalStateException:
> failed to create a child event loop
> attempt_201504280858_0001_000017_0: at
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
> attempt_201504280858_0001_000017_0: at
> io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
> attempt_201504280858_0001_000017_0: at
> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:61)
> attempt_201504280858_0001_000017_0: at
> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:52)
> attempt_201504280858_0001_000017_0: at
> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:44)
> attempt_201504280858_0001_000017_0: at
> io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:36)
> attempt_201504280858_0001_000017_0: at
> org.apache.hama.ipc.AsyncClient$Connection.<init>(AsyncClient.java:189)
> attempt_201504280858_0001_000017_0: at
> org.apache.hama.ipc.AsyncClient.getConnection(AsyncClient.java:989)
> attempt_201504280858_0001_000017_0: at
> org.apache.hama.ipc.AsyncClient.call(AsyncClient.java:838)
> attempt_201504280858_0001_000017_0: at
> org.apache.hama.ipc.AsyncRPC$Invoker.invoke(AsyncRPC.java:261)
> attempt_201504280858_0001_000017_0: at
> com.sun.proxy.$Proxy14.getProtocolVersion(Unknown Source)
> attempt_201504280858_0001_000017_0: at
> org.apache.hama.ipc.AsyncRPC.checkVersion(AsyncRPC.java:524)
> attempt_201504280858_0001_000017_0: at
> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:509)
> attempt_201504280858_0001_000017_0: at
> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:477)
> attempt_201504280858_0001_000017_0: at
> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:435)
> attempt_201504280858_0001_000017_0: at
> org.apache.hama.ipc.AsyncRPC.getProxy(AsyncRPC.java:545)
> attempt_201504280858_0001_000017_0: at
> org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.getBSPPeerConnection(HamaAsyncMessageManagerImpl.java:155)
> attempt_201504280858_0001_000017_0: at
> org.apache.hama.bsp.message.HamaAsyncMessageManagerImpl.transfer(HamaAsyncMessageManagerImpl.java:203)
> attempt_201504280858_0001_000017_0: at
> org.apache.hama.bsp.BSPPeerImpl.sendDirectly(BSPPeerImpl.java:382)
> attempt_201504280858_0001_000017_0: at
> org.apache.hama.bsp.BSPPeerImpl.send(BSPPeerImpl.java:364)
> attempt_201504280858_0001_000017_0: at
> org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:467)
> attempt_201504280858_0001_000017_0: at
> org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:128)
> attempt_201504280858_0001_000017_0: at
> org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170)
> attempt_201504280858_0001_000017_0: at
> org.apache.hama.bsp.BSPTask.run(BSPTask.java:144)
> attempt_201504280858_0001_000017_0: at
> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1255)
> attempt_201504280858_0001_000017_0: Caused by:
> io.netty.channel.ChannelException: failed to open a new selector
> attempt_201504280858_0001_000017_0: at
> io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:128)
> attempt_201504280858_0001_000017_0: at
> io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:120)
> attempt_201504280858_0001_000017_0: at
> io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:87)
> attempt_201504280858_0001_000017_0: at
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
> attempt_201504280858_0001_000017_0: ... 24 more
> attempt_201504280858_0001_000017_0: Caused by: java.io.IOException:
> Too many open files
> attempt_201504280858_0001_000017_0: at sun.nio.ch.IOUtil.makePipe(Native Method)
> attempt_201504280858_0001_000017_0: at
> sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65)
> attempt_201504280858_0001_000017_0: at
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
> attempt_201504280858_0001_000017_0: at
> io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:126)
> attempt_201504280858_0001_000017_0: ... 27 more
> attempt_201504280858_0001_000017_0: 15/04/28 08:28:21 INFO
> ipc.AsyncServer: AsyncServer gracefully shutdown
>
>
> --
> Best Regards, Edward J. Yoon