You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Neelesh <ne...@gmail.com> on 2014/06/09 23:44:15 UTC
Client is being closed, and does not take requests any more
Hi all,
I've deployed storm 0.9.1 on EC2, with KafkaSpout and every time a
message is received from kafka, the worker dies with the below stacktrace.
Ive patched my 0.9.1 with https://issues.apache.org/jira/browse/STORM-187.
Im using netty, with the below config on supervisor. Has anyone run into
this issue?
storm.messaging.transport: "backtype.storm.messaging.netty.Context"
supervisor.worker.timeout.secs: 5
storm.messaging.netty.server_worker_threads: 1
storm.messaging.netty.client_worker_threads: 1
storm.messaging.netty.buffer_size: 67108864 #5MB buffer
storm.messaging.netty.max_retries: 5
storm.messaging.netty.max_wait_ms: 10000
storm.messaging.netty.min_wait_ms: 5000
2014-06-07 06:43:27 b.s.d.task [INFO] Emitting: processMunchkin
__ack_ack [2264770826798969496 7102317654394246142]
2014-06-07 06:43:27 b.s.util [ERROR] Async loop died!
java.lang.RuntimeException: java.lang.RuntimeException: Client is
being closed, and does not take requests any more
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:107)
~[storm-core-0.9.1.0-incubating.jar:0.9.1.0-incubating]
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:78)
~[storm-core-0.9.1.0-incubating.jar:0.9.1.0-incubating]
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:77)
~[storm-core-0.9.1.0-incubating.jar:0.9.1.0-incubating]
at backtype.storm.disruptor$consume_loop_STAR_$fn__1577.invoke(disruptor.clj:89)
~[na:na]
at backtype.storm.util$async_loop$fn__384.invoke(util.clj:433) ~[na:na]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_55]
Caused by: java.lang.RuntimeException: Client is being closed, and
does not take requests any more
at backtype.storm.messaging.netty.Client.send(Client.java:125)
~[storm-core-0.9.1.0-incubating.jar:0.9.1.0-incubating]
at backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__4398$fn__4399.invoke(worker.clj:319)
~[na:na]
at backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__4398.invoke(worker.clj:308)
~[na:na]
at backtype.storm.disruptor$clojure_handler$reify__1560.onEvent(disruptor.clj:58)
~[na:na]
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:104)
~[storm-core-0.9.1.0-incubating.jar:0.9.1.0-incubating]
... 6 common frames omitted
2014-06-07 06:43:27 b.s.util [INFO] Halting process: ("Async loop died!")
Re: Client is being closed, and does not take requests any more
Posted by Xing Yong <xy...@gmail.com>.
set larger ‘storm.messaging.netty.max_wait_ms’ ，for example， 4000ms
2014-06-10 8:35 GMT+08:00 Sean Zhong <cl...@gmail.com>:
> Neelesh,
>
> Can you post full worker log on this?
>
> Sean
>
> On Tue, Jun 10, 2014 at 5:44 AM, Neelesh <ne...@gmail.com> wrote:
>
>> Client is being closed, and does not take requests any more
>
>
>
>
Re: Client is being closed, and does not take requests any more
Posted by Xing Yong <xy...@gmail.com>.
@Neelesh，thanks，i will make some try on 0.9.2.
2014-06-11 5:41 GMT+08:00 Neelesh <ne...@gmail.com>:
> Sean, Unfortunately, I did not retain the full log and upgraded to the
> master branch (0.9.2). That seems to have fixed the issue. Xing Yong, the
> setting you mentioned is already at 10000ms. Not sure that would help. I
> saw that "Reconnecting..." retry log messages went away with 0.9.2 ,
> without any configuration change
>
> thanks!
>
>
> On Mon, Jun 9, 2014 at 5:35 PM, Sean Zhong <cl...@gmail.com> wrote:
>
>> Neelesh,
>>
>> Can you post full worker log on this?
>>
>> Sean
>>
>> On Tue, Jun 10, 2014 at 5:44 AM, Neelesh <ne...@gmail.com> wrote:
>>
>>> Client is being closed, and does not take requests any more
>>
>>
>>
>>
>
Re: Client is being closed, and does not take requests any more
Posted by Neelesh <ne...@gmail.com>.
Sean, Unfortunately, I did not retain the full log and upgraded to the
master branch (0.9.2). That seems to have fixed the issue. Xing Yong, the
setting you mentioned is already at 10000ms. Not sure that would help. I
saw that "Reconnecting..." retry log messages went away with 0.9.2 ,
without any configuration change
thanks!
On Mon, Jun 9, 2014 at 5:35 PM, Sean Zhong <cl...@gmail.com> wrote:
> Neelesh,
>
> Can you post full worker log on this?
>
> Sean
>
> On Tue, Jun 10, 2014 at 5:44 AM, Neelesh <ne...@gmail.com> wrote:
>
>> Client is being closed, and does not take requests any more
>
>
>
>
Re: Client is being closed, and does not take requests any more
Posted by Sean Zhong <cl...@gmail.com>.
Neelesh,
Can you post full worker log on this?
Sean
On Tue, Jun 10, 2014 at 5:44 AM, Neelesh <ne...@gmail.com> wrote:
> Client is being closed, and does not take requests any more