You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by 李家宏 <jh...@gmail.com> on 2014/03/05 03:56:35 UTC
Netty Client failed to create a selector : too many open files
Hi, all
I'm running a topology on storm cluster of 0.9.0.1 with netty as transport
layer, this error occurs :
Netty client failed to create a selector due to* too many open files
exception*, the worker continuously halting with initialization error.
I checked the ulimit -n(> 130000) which is much bigger than currently
opened fds (sudo lsof | grep java | wc -l) which is about 6000 at most.
By the way,this topology works fine with storm cluster of 0.8.0.
What's the problem?
Thanks
--
======================================================
Gvain
Email: jh.li.em@gmail.com
Re: Netty Client failed to create a selector : too many open files
Posted by 李家宏 <jh...@gmail.com>.
Hi, Andy
Is this pull request :
https://github.com/nathanmarz/storm/pull/693
related to what you've mentioned about netty's too many threads problem ?
Regards.
2014-03-28 13:22 GMT+08:00 李家宏 <jh...@gmail.com>:
> Red Hat Enterprise Linux Server release 6.2 (Santiago)
>
>
> 2014-03-28 12:55 GMT+08:00 Andy Feng <an...@gmail.com>:
>
> What’s your OS?
>>
>> On 3/27/14, 9:42 PM, "李家宏" <jh...@gmail.com> wrote:
>>
>> >Andy,
>> >
>> >Sorry for the late reply. Here is the stack trace:
>> >
>> >2014-03-04 20:24:14 b.s.m.TransportFactory [INFO] Storm peer transport
>> >plugin:backtype.storm.messaging.netty.Context
>> > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
>> > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
>> > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
>> > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
>> > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
>> > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
>> > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
>> > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [2]
>> > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
>> > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
>> > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
>> > 2014-03-04 20:24:14 b.s.d.worker [ERROR] Error on initialization of
>> >server mk-worker
>> > org.jboss.netty.channel.ChannelException: Failed to create a selector.
>> > at
>>
>> >org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(Abstra
>> >ctNioSelector.java:337)
>> > ~[netty-3.6.3.Final.jar:na]
>> > at
>>
>> >org.jboss.netty.channel.socket.nio.AbstractNioSelector.(AbstractNioSelecto
>> >r.java:95)
>> >~[netty-3.6.3.Final.jar:na]
>> > at
>>
>> >org.jboss.netty.channel.socket.nio.AbstractNioWorker.(AbstractNioWorker.ja
>> >va:51)
>> >~[netty-3.6.3.Final.jar:na]
>> > at org.jboss.netty.channel.socket.nio.NioWorker.(NioWorker.java:45)
>> >~[netty-3.6.3.Final.jar:na]
>> > at
>>
>> >org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPoo
>> >l.java:45)
>> >~[netty-3.6.3.Final.jar:na]
>> > at
>>
>> >org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPoo
>> >l.java:28)
>> >~[netty-3.6.3.Final.jar:na]
>> > at
>>
>> >org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(Abstrac
>> >tNioWorkerPool.java:99)
>> > ~[netty-3.6.3.Final.jar:na]
>> > at
>>
>> >org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioW
>> >orkerPool.java:69)
>> > ~[netty-3.6.3.Final.jar:na]
>> > at
>> >org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:39)
>> >~[netty-3.6.3.Final.jar:na]
>> > at
>> >org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:33)
>> >~[netty-3.6.3.Final.jar:na]
>> > at
>>
>> >org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.(NioClien
>> >tSocketChannelFactory.java:152)
>> > ~[netty-3.6.3.Final.jar:na]
>> > at
>>
>> >org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.(NioClien
>> >tSocketChannelFactory.java:134)
>> > ~[netty-3.6.3.Final.jar:na]
>> > at backtype.storm.messaging.netty.Client.(Client.java:54)
>> >~[storm-netty-0.9.0.1.jar:na]
>> > at backtype.storm.messaging.netty.Context.connect(Context.java:36)
>> >~[storm-netty-0.9.0.1.jar:na]
>> > at
>>
>> >backtype.storm.daemon.worker$mk_refresh_connections$this__5827$iter__5834_
>> >_5838$fn__5839.invoke(worker.clj:250)
>> > ~[storm-core-0.9.0.1.jar:na]
>> > at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.4.0.jar:na]
>> > at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.4.0.jar:na]
>> > at clojure.lang.Cons.next(Cons.java:39) ~[clojure-1.4.0.jar:na]
>> > at clojure.lang.RT.next(RT.java:587) ~[clojure-1.4.0.jar:na]
>> > at clojure.core$next.invoke(core.clj:64) ~[clojure-1.4.0.jar:na]
>> > at clojure.core$dorun.invoke(core.clj:2726) ~[clojure-1.4.0.jar:na]
>> > at clojure.core$doall.invoke(core.clj:2741) ~[clojure-1.4.0.jar:na]
>> > at
>>
>> >backtype.storm.daemon.worker$mk_refresh_connections$this__5827.invoke(work
>> >er.clj:244)
>> >~[storm-core-0.9.0.1.jar:na]
>> > at
>>
>> >backtype.storm.daemon.worker$fn__5882$exec_fn__1229__auto____5883.invoke(w
>> >orker.clj:357)
>> > ~[storm-core-0.9.0.1.jar:na]
>> > at clojure.lang.AFn.applyToHelper(AFn.java:185) [clojure-1.4.0.jar:na]
>> > at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na]
>> > at clojure.core$apply.invoke(core.clj:601) ~[clojure-1.4.0.jar:na]
>> > at
>>
>> >backtype.storm.daemon.worker$fn__5882$mk_worker__5938.doInvoke(worker.clj:
>> >329)
>> >[storm-core-0.9.0.1.jar:na]
>> > at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.4.0.jar:na]
>> > at backtype.storm.daemon.worker$_main.invoke(worker.clj:439)
>> >[storm-core-0.9.0.1.jar:na]
>> > at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.4.0.jar:na]
>> > at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na]
>> > at backtype.storm.daemon.worker.main(Unknown Source)
>> >[storm-core-0.9.0.1.jar:na]
>> >
>> > * Caused by: java.io.IOException: Too many open files*
>> >
>> > at sun.nio.ch.IOUtil.initPipe(Native Method) ~[na:1.6.0_38]
>> > at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:49)
>> >~[na:1.6.0_38]
>> > at
>>
>> >sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:1
>> >8)
>> >~[na:1.6.0_38]
>> > at java.nio.channels.Selector.open(Selector.java:209) ~[na:1.6.0_38]
>> > at
>>
>> >org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(Abstra
>> >ctNioSelector.java:335)
>> > ~[netty-3.6.3.Final.jar:na]
>> > ... 32 common frames omitted
>> > 2014-03-04 20:24:14 b.s.util [INFO] Halting process: ("Error on
>> >initialization")
>>
>> >--------------------------------------------------------------------------
>> >------------------------------------------
>> >
>> >Regards
>> >
>> >
>> >2014-03-12 3:25 GMT+08:00 Andrew Feng <af...@yahoo-inc.com>:
>> >
>> >> Gavin:
>> >>
>> >> In our production, we have seen situation that Netty implementation
>> uses
>> >> too many threads.
>> >> It uses a dedicated thread for each remote bolt worker. We have a fix
>> to
>> >> avoid that problem.
>> >>
>> >>
>> >> Our problem could be different, since we did not face
>> >> too-many-file-descriptor issue.
>> >> Can you post the stack trace?
>> >>
>> >> Andy
>> >>
>> >> On 3/5/14 1:26 AM, "李家宏" <jh...@gmail.com> wrote:
>> >>
>> >> >Also, the topology works fine on storm cluster of 0.9.0.1 with zmq as
>> >> >message transport layer.
>> >> >I guess netty must be the problem.
>> >> >
>> >> >Regards
>> >> >
>> >> >
>> >> >2014-03-05 10:56 GMT+08:00 李家宏 <jh...@gmail.com>:
>> >> >
>> >> >> Hi, all
>> >> >> I'm running a topology on storm cluster of 0.9.0.1 with netty as
>> >> >>transport
>> >> >> layer, this error occurs :
>> >> >> Netty client failed to create a selector due to* too many open files
>> >> >> exception*, the worker continuously halting with initialization
>> >>error.
>> >> >>
>> >> >> I checked the ulimit -n(> 130000) which is much bigger than
>> currently
>> >> >> opened fds (sudo lsof | grep java | wc -l) which is about 6000 at
>> >>most.
>> >> >>
>> >> >> By the way,this topology works fine with storm cluster of 0.8.0.
>> >> >>
>> >> >> What's the problem?
>> >> >>
>> >> >> Thanks
>> >> >>
>> >> >> --
>> >> >>
>> >> >> ======================================================
>> >> >>
>> >> >> Gvain
>> >> >>
>> >> >> Email: jh.li.em@gmail.com
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> >--
>> >> >
>> >> >======================================================
>> >> >
>> >> >Gvain
>> >> >
>> >> >Email: jh.li.em@gmail.com
>> >>
>> >>
>> >
>> >
>> >--
>> >
>> >======================================================
>> >
>> >Gvain
>> >
>> >Email: jh.li.em@gmail.com
>>
>>
>>
>
>
> --
>
> ======================================================
>
> Gvain
>
> Email: jh.li.em@gmail.com
>
--
======================================================
Gvain
Email: jh.li.em@gmail.com
Re: Netty Client failed to create a selector : too many open files
Posted by 李家宏 <jh...@gmail.com>.
Red Hat Enterprise Linux Server release 6.2 (Santiago)
2014-03-28 12:55 GMT+08:00 Andy Feng <an...@gmail.com>:
> What’s your OS?
>
> On 3/27/14, 9:42 PM, "李家宏" <jh...@gmail.com> wrote:
>
> >Andy,
> >
> >Sorry for the late reply. Here is the stack trace:
> >
> >2014-03-04 20:24:14 b.s.m.TransportFactory [INFO] Storm peer transport
> >plugin:backtype.storm.messaging.netty.Context
> > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [2]
> > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> > 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> > 2014-03-04 20:24:14 b.s.d.worker [ERROR] Error on initialization of
> >server mk-worker
> > org.jboss.netty.channel.ChannelException: Failed to create a selector.
> > at
> >org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(Abstra
> >ctNioSelector.java:337)
> > ~[netty-3.6.3.Final.jar:na]
> > at
> >org.jboss.netty.channel.socket.nio.AbstractNioSelector.(AbstractNioSelecto
> >r.java:95)
> >~[netty-3.6.3.Final.jar:na]
> > at
> >org.jboss.netty.channel.socket.nio.AbstractNioWorker.(AbstractNioWorker.ja
> >va:51)
> >~[netty-3.6.3.Final.jar:na]
> > at org.jboss.netty.channel.socket.nio.NioWorker.(NioWorker.java:45)
> >~[netty-3.6.3.Final.jar:na]
> > at
> >org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPoo
> >l.java:45)
> >~[netty-3.6.3.Final.jar:na]
> > at
> >org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPoo
> >l.java:28)
> >~[netty-3.6.3.Final.jar:na]
> > at
> >org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(Abstrac
> >tNioWorkerPool.java:99)
> > ~[netty-3.6.3.Final.jar:na]
> > at
> >org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioW
> >orkerPool.java:69)
> > ~[netty-3.6.3.Final.jar:na]
> > at
> >org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:39)
> >~[netty-3.6.3.Final.jar:na]
> > at
> >org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:33)
> >~[netty-3.6.3.Final.jar:na]
> > at
> >org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.(NioClien
> >tSocketChannelFactory.java:152)
> > ~[netty-3.6.3.Final.jar:na]
> > at
> >org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.(NioClien
> >tSocketChannelFactory.java:134)
> > ~[netty-3.6.3.Final.jar:na]
> > at backtype.storm.messaging.netty.Client.(Client.java:54)
> >~[storm-netty-0.9.0.1.jar:na]
> > at backtype.storm.messaging.netty.Context.connect(Context.java:36)
> >~[storm-netty-0.9.0.1.jar:na]
> > at
> >backtype.storm.daemon.worker$mk_refresh_connections$this__5827$iter__5834_
> >_5838$fn__5839.invoke(worker.clj:250)
> > ~[storm-core-0.9.0.1.jar:na]
> > at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.4.0.jar:na]
> > at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.4.0.jar:na]
> > at clojure.lang.Cons.next(Cons.java:39) ~[clojure-1.4.0.jar:na]
> > at clojure.lang.RT.next(RT.java:587) ~[clojure-1.4.0.jar:na]
> > at clojure.core$next.invoke(core.clj:64) ~[clojure-1.4.0.jar:na]
> > at clojure.core$dorun.invoke(core.clj:2726) ~[clojure-1.4.0.jar:na]
> > at clojure.core$doall.invoke(core.clj:2741) ~[clojure-1.4.0.jar:na]
> > at
> >backtype.storm.daemon.worker$mk_refresh_connections$this__5827.invoke(work
> >er.clj:244)
> >~[storm-core-0.9.0.1.jar:na]
> > at
> >backtype.storm.daemon.worker$fn__5882$exec_fn__1229__auto____5883.invoke(w
> >orker.clj:357)
> > ~[storm-core-0.9.0.1.jar:na]
> > at clojure.lang.AFn.applyToHelper(AFn.java:185) [clojure-1.4.0.jar:na]
> > at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na]
> > at clojure.core$apply.invoke(core.clj:601) ~[clojure-1.4.0.jar:na]
> > at
> >backtype.storm.daemon.worker$fn__5882$mk_worker__5938.doInvoke(worker.clj:
> >329)
> >[storm-core-0.9.0.1.jar:na]
> > at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.4.0.jar:na]
> > at backtype.storm.daemon.worker$_main.invoke(worker.clj:439)
> >[storm-core-0.9.0.1.jar:na]
> > at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.4.0.jar:na]
> > at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na]
> > at backtype.storm.daemon.worker.main(Unknown Source)
> >[storm-core-0.9.0.1.jar:na]
> >
> > * Caused by: java.io.IOException: Too many open files*
> >
> > at sun.nio.ch.IOUtil.initPipe(Native Method) ~[na:1.6.0_38]
> > at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:49)
> >~[na:1.6.0_38]
> > at
> >sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:1
> >8)
> >~[na:1.6.0_38]
> > at java.nio.channels.Selector.open(Selector.java:209) ~[na:1.6.0_38]
> > at
> >org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(Abstra
> >ctNioSelector.java:335)
> > ~[netty-3.6.3.Final.jar:na]
> > ... 32 common frames omitted
> > 2014-03-04 20:24:14 b.s.util [INFO] Halting process: ("Error on
> >initialization")
> >--------------------------------------------------------------------------
> >------------------------------------------
> >
> >Regards
> >
> >
> >2014-03-12 3:25 GMT+08:00 Andrew Feng <af...@yahoo-inc.com>:
> >
> >> Gavin:
> >>
> >> In our production, we have seen situation that Netty implementation uses
> >> too many threads.
> >> It uses a dedicated thread for each remote bolt worker. We have a fix to
> >> avoid that problem.
> >>
> >>
> >> Our problem could be different, since we did not face
> >> too-many-file-descriptor issue.
> >> Can you post the stack trace?
> >>
> >> Andy
> >>
> >> On 3/5/14 1:26 AM, "李家宏" <jh...@gmail.com> wrote:
> >>
> >> >Also, the topology works fine on storm cluster of 0.9.0.1 with zmq as
> >> >message transport layer.
> >> >I guess netty must be the problem.
> >> >
> >> >Regards
> >> >
> >> >
> >> >2014-03-05 10:56 GMT+08:00 李家宏 <jh...@gmail.com>:
> >> >
> >> >> Hi, all
> >> >> I'm running a topology on storm cluster of 0.9.0.1 with netty as
> >> >>transport
> >> >> layer, this error occurs :
> >> >> Netty client failed to create a selector due to* too many open files
> >> >> exception*, the worker continuously halting with initialization
> >>error.
> >> >>
> >> >> I checked the ulimit -n(> 130000) which is much bigger than currently
> >> >> opened fds (sudo lsof | grep java | wc -l) which is about 6000 at
> >>most.
> >> >>
> >> >> By the way,this topology works fine with storm cluster of 0.8.0.
> >> >>
> >> >> What's the problem?
> >> >>
> >> >> Thanks
> >> >>
> >> >> --
> >> >>
> >> >> ======================================================
> >> >>
> >> >> Gvain
> >> >>
> >> >> Email: jh.li.em@gmail.com
> >> >>
> >> >
> >> >
> >> >
> >> >--
> >> >
> >> >======================================================
> >> >
> >> >Gvain
> >> >
> >> >Email: jh.li.em@gmail.com
> >>
> >>
> >
> >
> >--
> >
> >======================================================
> >
> >Gvain
> >
> >Email: jh.li.em@gmail.com
>
>
>
--
======================================================
Gvain
Email: jh.li.em@gmail.com
Re: Netty Client failed to create a selector : too many open files
Posted by Andy Feng <an...@gmail.com>.
What’s your OS?
On 3/27/14, 9:42 PM, "李家宏" <jh...@gmail.com> wrote:
>Andy,
>
>Sorry for the late reply. Here is the stack trace:
>
>2014-03-04 20:24:14 b.s.m.TransportFactory [INFO] Storm peer transport
>plugin:backtype.storm.messaging.netty.Context
> 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [2]
> 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
> 2014-03-04 20:24:14 b.s.d.worker [ERROR] Error on initialization of
>server mk-worker
> org.jboss.netty.channel.ChannelException: Failed to create a selector.
> at
>org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(Abstra
>ctNioSelector.java:337)
> ~[netty-3.6.3.Final.jar:na]
> at
>org.jboss.netty.channel.socket.nio.AbstractNioSelector.(AbstractNioSelecto
>r.java:95)
>~[netty-3.6.3.Final.jar:na]
> at
>org.jboss.netty.channel.socket.nio.AbstractNioWorker.(AbstractNioWorker.ja
>va:51)
>~[netty-3.6.3.Final.jar:na]
> at org.jboss.netty.channel.socket.nio.NioWorker.(NioWorker.java:45)
>~[netty-3.6.3.Final.jar:na]
> at
>org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPoo
>l.java:45)
>~[netty-3.6.3.Final.jar:na]
> at
>org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPoo
>l.java:28)
>~[netty-3.6.3.Final.jar:na]
> at
>org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(Abstrac
>tNioWorkerPool.java:99)
> ~[netty-3.6.3.Final.jar:na]
> at
>org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioW
>orkerPool.java:69)
> ~[netty-3.6.3.Final.jar:na]
> at
>org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:39)
>~[netty-3.6.3.Final.jar:na]
> at
>org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:33)
>~[netty-3.6.3.Final.jar:na]
> at
>org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.(NioClien
>tSocketChannelFactory.java:152)
> ~[netty-3.6.3.Final.jar:na]
> at
>org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.(NioClien
>tSocketChannelFactory.java:134)
> ~[netty-3.6.3.Final.jar:na]
> at backtype.storm.messaging.netty.Client.(Client.java:54)
>~[storm-netty-0.9.0.1.jar:na]
> at backtype.storm.messaging.netty.Context.connect(Context.java:36)
>~[storm-netty-0.9.0.1.jar:na]
> at
>backtype.storm.daemon.worker$mk_refresh_connections$this__5827$iter__5834_
>_5838$fn__5839.invoke(worker.clj:250)
> ~[storm-core-0.9.0.1.jar:na]
> at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.4.0.jar:na]
> at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.4.0.jar:na]
> at clojure.lang.Cons.next(Cons.java:39) ~[clojure-1.4.0.jar:na]
> at clojure.lang.RT.next(RT.java:587) ~[clojure-1.4.0.jar:na]
> at clojure.core$next.invoke(core.clj:64) ~[clojure-1.4.0.jar:na]
> at clojure.core$dorun.invoke(core.clj:2726) ~[clojure-1.4.0.jar:na]
> at clojure.core$doall.invoke(core.clj:2741) ~[clojure-1.4.0.jar:na]
> at
>backtype.storm.daemon.worker$mk_refresh_connections$this__5827.invoke(work
>er.clj:244)
>~[storm-core-0.9.0.1.jar:na]
> at
>backtype.storm.daemon.worker$fn__5882$exec_fn__1229__auto____5883.invoke(w
>orker.clj:357)
> ~[storm-core-0.9.0.1.jar:na]
> at clojure.lang.AFn.applyToHelper(AFn.java:185) [clojure-1.4.0.jar:na]
> at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na]
> at clojure.core$apply.invoke(core.clj:601) ~[clojure-1.4.0.jar:na]
> at
>backtype.storm.daemon.worker$fn__5882$mk_worker__5938.doInvoke(worker.clj:
>329)
>[storm-core-0.9.0.1.jar:na]
> at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.4.0.jar:na]
> at backtype.storm.daemon.worker$_main.invoke(worker.clj:439)
>[storm-core-0.9.0.1.jar:na]
> at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.4.0.jar:na]
> at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na]
> at backtype.storm.daemon.worker.main(Unknown Source)
>[storm-core-0.9.0.1.jar:na]
>
> * Caused by: java.io.IOException: Too many open files*
>
> at sun.nio.ch.IOUtil.initPipe(Native Method) ~[na:1.6.0_38]
> at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:49)
>~[na:1.6.0_38]
> at
>sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:1
>8)
>~[na:1.6.0_38]
> at java.nio.channels.Selector.open(Selector.java:209) ~[na:1.6.0_38]
> at
>org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(Abstra
>ctNioSelector.java:335)
> ~[netty-3.6.3.Final.jar:na]
> ... 32 common frames omitted
> 2014-03-04 20:24:14 b.s.util [INFO] Halting process: ("Error on
>initialization")
>--------------------------------------------------------------------------
>------------------------------------------
>
>Regards
>
>
>2014-03-12 3:25 GMT+08:00 Andrew Feng <af...@yahoo-inc.com>:
>
>> Gavin:
>>
>> In our production, we have seen situation that Netty implementation uses
>> too many threads.
>> It uses a dedicated thread for each remote bolt worker. We have a fix to
>> avoid that problem.
>>
>>
>> Our problem could be different, since we did not face
>> too-many-file-descriptor issue.
>> Can you post the stack trace?
>>
>> Andy
>>
>> On 3/5/14 1:26 AM, "李家宏" <jh...@gmail.com> wrote:
>>
>> >Also, the topology works fine on storm cluster of 0.9.0.1 with zmq as
>> >message transport layer.
>> >I guess netty must be the problem.
>> >
>> >Regards
>> >
>> >
>> >2014-03-05 10:56 GMT+08:00 李家宏 <jh...@gmail.com>:
>> >
>> >> Hi, all
>> >> I'm running a topology on storm cluster of 0.9.0.1 with netty as
>> >>transport
>> >> layer, this error occurs :
>> >> Netty client failed to create a selector due to* too many open files
>> >> exception*, the worker continuously halting with initialization
>>error.
>> >>
>> >> I checked the ulimit -n(> 130000) which is much bigger than currently
>> >> opened fds (sudo lsof | grep java | wc -l) which is about 6000 at
>>most.
>> >>
>> >> By the way,this topology works fine with storm cluster of 0.8.0.
>> >>
>> >> What's the problem?
>> >>
>> >> Thanks
>> >>
>> >> --
>> >>
>> >> ======================================================
>> >>
>> >> Gvain
>> >>
>> >> Email: jh.li.em@gmail.com
>> >>
>> >
>> >
>> >
>> >--
>> >
>> >======================================================
>> >
>> >Gvain
>> >
>> >Email: jh.li.em@gmail.com
>>
>>
>
>
>--
>
>======================================================
>
>Gvain
>
>Email: jh.li.em@gmail.com
Re: Netty Client failed to create a selector : too many open files
Posted by 李家宏 <jh...@gmail.com>.
Andy,
Sorry for the late reply. Here is the stack trace:
2014-03-04 20:24:14 b.s.m.TransportFactory [INFO] Storm peer transport
plugin:backtype.storm.messaging.netty.Context
2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [2]
2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1]
2014-03-04 20:24:14 b.s.d.worker [ERROR] Error on initialization of
server mk-worker
org.jboss.netty.channel.ChannelException: Failed to create a selector.
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:337)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.(AbstractNioSelector.java:95)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.(AbstractNioWorker.java:51)
~[netty-3.6.3.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.(NioWorker.java:45)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(AbstractNioWorkerPool.java:99)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:69)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:39)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:33)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.(NioClientSocketChannelFactory.java:152)
~[netty-3.6.3.Final.jar:na]
at
org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.(NioClientSocketChannelFactory.java:134)
~[netty-3.6.3.Final.jar:na]
at backtype.storm.messaging.netty.Client.(Client.java:54)
~[storm-netty-0.9.0.1.jar:na]
at backtype.storm.messaging.netty.Context.connect(Context.java:36)
~[storm-netty-0.9.0.1.jar:na]
at
backtype.storm.daemon.worker$mk_refresh_connections$this__5827$iter__5834__5838$fn__5839.invoke(worker.clj:250)
~[storm-core-0.9.0.1.jar:na]
at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.4.0.jar:na]
at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.4.0.jar:na]
at clojure.lang.Cons.next(Cons.java:39) ~[clojure-1.4.0.jar:na]
at clojure.lang.RT.next(RT.java:587) ~[clojure-1.4.0.jar:na]
at clojure.core$next.invoke(core.clj:64) ~[clojure-1.4.0.jar:na]
at clojure.core$dorun.invoke(core.clj:2726) ~[clojure-1.4.0.jar:na]
at clojure.core$doall.invoke(core.clj:2741) ~[clojure-1.4.0.jar:na]
at
backtype.storm.daemon.worker$mk_refresh_connections$this__5827.invoke(worker.clj:244)
~[storm-core-0.9.0.1.jar:na]
at
backtype.storm.daemon.worker$fn__5882$exec_fn__1229__auto____5883.invoke(worker.clj:357)
~[storm-core-0.9.0.1.jar:na]
at clojure.lang.AFn.applyToHelper(AFn.java:185) [clojure-1.4.0.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na]
at clojure.core$apply.invoke(core.clj:601) ~[clojure-1.4.0.jar:na]
at
backtype.storm.daemon.worker$fn__5882$mk_worker__5938.doInvoke(worker.clj:329)
[storm-core-0.9.0.1.jar:na]
at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.4.0.jar:na]
at backtype.storm.daemon.worker$_main.invoke(worker.clj:439)
[storm-core-0.9.0.1.jar:na]
at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.4.0.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na]
at backtype.storm.daemon.worker.main(Unknown Source)
[storm-core-0.9.0.1.jar:na]
* Caused by: java.io.IOException: Too many open files*
at sun.nio.ch.IOUtil.initPipe(Native Method) ~[na:1.6.0_38]
at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:49)
~[na:1.6.0_38]
at
sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
~[na:1.6.0_38]
at java.nio.channels.Selector.open(Selector.java:209) ~[na:1.6.0_38]
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:335)
~[netty-3.6.3.Final.jar:na]
... 32 common frames omitted
2014-03-04 20:24:14 b.s.util [INFO] Halting process: ("Error on
initialization")
--------------------------------------------------------------------------------------------------------------------
Regards
2014-03-12 3:25 GMT+08:00 Andrew Feng <af...@yahoo-inc.com>:
> Gavin:
>
> In our production, we have seen situation that Netty implementation uses
> too many threads.
> It uses a dedicated thread for each remote bolt worker. We have a fix to
> avoid that problem.
>
>
> Our problem could be different, since we did not face
> too-many-file-descriptor issue.
> Can you post the stack trace?
>
> Andy
>
> On 3/5/14 1:26 AM, "李家宏" <jh...@gmail.com> wrote:
>
> >Also, the topology works fine on storm cluster of 0.9.0.1 with zmq as
> >message transport layer.
> >I guess netty must be the problem.
> >
> >Regards
> >
> >
> >2014-03-05 10:56 GMT+08:00 李家宏 <jh...@gmail.com>:
> >
> >> Hi, all
> >> I'm running a topology on storm cluster of 0.9.0.1 with netty as
> >>transport
> >> layer, this error occurs :
> >> Netty client failed to create a selector due to* too many open files
> >> exception*, the worker continuously halting with initialization error.
> >>
> >> I checked the ulimit -n(> 130000) which is much bigger than currently
> >> opened fds (sudo lsof | grep java | wc -l) which is about 6000 at most.
> >>
> >> By the way,this topology works fine with storm cluster of 0.8.0.
> >>
> >> What's the problem?
> >>
> >> Thanks
> >>
> >> --
> >>
> >> ======================================================
> >>
> >> Gvain
> >>
> >> Email: jh.li.em@gmail.com
> >>
> >
> >
> >
> >--
> >
> >======================================================
> >
> >Gvain
> >
> >Email: jh.li.em@gmail.com
>
>
--
======================================================
Gvain
Email: jh.li.em@gmail.com
Re: Netty Client failed to create a selector : too many open files
Posted by Andrew Feng <af...@yahoo-inc.com>.
Gavin:
In our production, we have seen situation that Netty implementation uses
too many threads.
It uses a dedicated thread for each remote bolt worker. We have a fix to
avoid that problem.
Our problem could be different, since we did not face
too-many-file-descriptor issue.
Can you post the stack trace?
Andy
On 3/5/14 1:26 AM, "李家宏" <jh...@gmail.com> wrote:
>Also, the topology works fine on storm cluster of 0.9.0.1 with zmq as
>message transport layer.
>I guess netty must be the problem.
>
>Regards
>
>
>2014-03-05 10:56 GMT+08:00 李家宏 <jh...@gmail.com>:
>
>> Hi, all
>> I'm running a topology on storm cluster of 0.9.0.1 with netty as
>>transport
>> layer, this error occurs :
>> Netty client failed to create a selector due to* too many open files
>> exception*, the worker continuously halting with initialization error.
>>
>> I checked the ulimit -n(> 130000) which is much bigger than currently
>> opened fds (sudo lsof | grep java | wc -l) which is about 6000 at most.
>>
>> By the way,this topology works fine with storm cluster of 0.8.0.
>>
>> What's the problem?
>>
>> Thanks
>>
>> --
>>
>> ======================================================
>>
>> Gvain
>>
>> Email: jh.li.em@gmail.com
>>
>
>
>
>--
>
>======================================================
>
>Gvain
>
>Email: jh.li.em@gmail.com
Re: Netty Client failed to create a selector : too many open files
Posted by 李家宏 <jh...@gmail.com>.
hi , Andy Feng
Is there any jira links to this problem ?
And could u please explain the problem in simple words ?
Thanks again
2014-03-05 23:03 GMT+08:00 Andrew Feng <af...@yahoo-inc.com>:
> This is a known problem for large topology using netty. We will post a
> pull request very soon.
>
> Andy Feng
>
> Sent from my iPhone
>
> > On Mar 5, 2014, at 1:27 AM, "李家宏" <jh...@gmail.com> wrote:
> >
> > Also, the topology works fine on storm cluster of 0.9.0.1 with zmq as
> > message transport layer.
> > I guess netty must be the problem.
> >
> > Regards
> >
> >
> > 2014-03-05 10:56 GMT+08:00 李家宏 <jh...@gmail.com>:
> >
> >> Hi, all
> >> I'm running a topology on storm cluster of 0.9.0.1 with netty as
> transport
> >> layer, this error occurs :
> >> Netty client failed to create a selector due to* too many open files
> >> exception*, the worker continuously halting with initialization error.
> >>
> >> I checked the ulimit -n(> 130000) which is much bigger than currently
> >> opened fds (sudo lsof | grep java | wc -l) which is about 6000 at most.
> >>
> >> By the way,this topology works fine with storm cluster of 0.8.0.
> >>
> >> What's the problem?
> >>
> >> Thanks
> >>
> >> --
> >>
> >> ======================================================
> >>
> >> Gvain
> >>
> >> Email: jh.li.em@gmail.com
> >
> >
> >
> > --
> >
> > ======================================================
> >
> > Gvain
> >
> > Email: jh.li.em@gmail.com
>
--
======================================================
Gvain
Email: jh.li.em@gmail.com
Re: Netty Client failed to create a selector : too many open files
Posted by Andrew Feng <af...@yahoo-inc.com>.
This is a known problem for large topology using netty. We will post a pull request very soon.
Andy Feng
Sent from my iPhone
> On Mar 5, 2014, at 1:27 AM, "李家宏" <jh...@gmail.com> wrote:
>
> Also, the topology works fine on storm cluster of 0.9.0.1 with zmq as
> message transport layer.
> I guess netty must be the problem.
>
> Regards
>
>
> 2014-03-05 10:56 GMT+08:00 李家宏 <jh...@gmail.com>:
>
>> Hi, all
>> I'm running a topology on storm cluster of 0.9.0.1 with netty as transport
>> layer, this error occurs :
>> Netty client failed to create a selector due to* too many open files
>> exception*, the worker continuously halting with initialization error.
>>
>> I checked the ulimit -n(> 130000) which is much bigger than currently
>> opened fds (sudo lsof | grep java | wc -l) which is about 6000 at most.
>>
>> By the way,this topology works fine with storm cluster of 0.8.0.
>>
>> What's the problem?
>>
>> Thanks
>>
>> --
>>
>> ======================================================
>>
>> Gvain
>>
>> Email: jh.li.em@gmail.com
>
>
>
> --
>
> ======================================================
>
> Gvain
>
> Email: jh.li.em@gmail.com
Re: Netty Client failed to create a selector : too many open files
Posted by 李家宏 <jh...@gmail.com>.
Also, the topology works fine on storm cluster of 0.9.0.1 with zmq as
message transport layer.
I guess netty must be the problem.
Regards
2014-03-05 10:56 GMT+08:00 李家宏 <jh...@gmail.com>:
> Hi, all
> I'm running a topology on storm cluster of 0.9.0.1 with netty as transport
> layer, this error occurs :
> Netty client failed to create a selector due to* too many open files
> exception*, the worker continuously halting with initialization error.
>
> I checked the ulimit -n(> 130000) which is much bigger than currently
> opened fds (sudo lsof | grep java | wc -l) which is about 6000 at most.
>
> By the way,this topology works fine with storm cluster of 0.8.0.
>
> What's the problem?
>
> Thanks
>
> --
>
> ======================================================
>
> Gvain
>
> Email: jh.li.em@gmail.com
>
--
======================================================
Gvain
Email: jh.li.em@gmail.com
Re: Netty Client failed to create a selector : too many open files
Posted by 李家宏 <jh...@gmail.com>.
Also, the topology works fine on storm cluster of 0.9.0.1 with zmq as
message transport layer.
I guess netty must be the problem.
Regards
2014-03-05 10:56 GMT+08:00 李家宏 <jh...@gmail.com>:
> Hi, all
> I'm running a topology on storm cluster of 0.9.0.1 with netty as transport
> layer, this error occurs :
> Netty client failed to create a selector due to* too many open files
> exception*, the worker continuously halting with initialization error.
>
> I checked the ulimit -n(> 130000) which is much bigger than currently
> opened fds (sudo lsof | grep java | wc -l) which is about 6000 at most.
>
> By the way,this topology works fine with storm cluster of 0.8.0.
>
> What's the problem?
>
> Thanks
>
> --
>
> ======================================================
>
> Gvain
>
> Email: jh.li.em@gmail.com
>
--
======================================================
Gvain
Email: jh.li.em@gmail.com