You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@storm.apache.org by Suparno Datta <su...@gmail.com> on 2014/07/10 16:38:57 UTC

b.s.m.n.Client [INFO] Reconnect

Hi,

 I am using storm 0.9.1-incubating on a single machine cluster to run a
simple twitter hashtag extractor.  I am using the Storm-twitter-workshop
which i found to be extremely useful.

https://github.com/kantega/storm-twitter-workshop

I have used this program before with storm 0.8.1 and it ran like a charm. I
might mention that was on a server machine with quite 2 quad xeon
processors.

This time i am trying it on my laptop( i5 , 8GB). But i am constantly
getting this error in the worker log files

2014-07-10 13:01:47 b.s.m.n.Client [INFO] Reconnect ... [24]
2014-07-10 13:01:58 b.s.m.n.Client [INFO] Reconnect ... [25]
2014-07-10 13:02:09 b.s.m.n.Client [INFO] Reconnect ... [26]
2014-07-10 13:02:19 STDIO [ERROR] Jul 10, 2014 1:02:19 PM
org.jboss.netty.channel.DefaultChannelPipeline
WARNING: An exception was thrown by a user handler while handling an
exception event ([id: 0x563f7062] EXCEPTION: java.net.ConnectException:
connection timed out)
java.lang.IllegalArgumentException: timeout value is negative
at java.lang.Thread.sleep(Native Method)
at backtype.storm.messaging.netty.Client.reconnect(Client.java:94)
at
backtype.storm.messaging.netty.StormClientHandler.exceptionCaught(StormClientHandler.java:118)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:140)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:82)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)


Now here comes the stranges part. If i declare just one instance of the
hashtag extractor Bolt it fails to get anything but for more that 1 it does
manage to get me a few hashtags though with quite high latency. Another
strange part related to this machine is if i declare more than 2
supervisor.slots.ports the program doesnt even launch any more showing some
initialitzation error.

Sorry if i blabbered a lot about the hardware and stuff. But somehow to me
it seemed quite related to the problem. Any sort of help will be really
useful.

Thanks,

Suparno

Re: b.s.m.n.Client [INFO] Reconnect

Posted by Andrew Montalenti <an...@parsely.com>.

fyi, you won't be able to change to zeromq in 0.9.2 from Netty unless you
use this updated storm-0mq module that I forked:

https://github.com/Parsely/storm-0mq/blob/master/README.md

I also added a detailed README with instructions. We'll get these changes
merged into ptgoetz's repo eventually so it is all in one place.

The truth is, though, there shouldn't be a good reason to use zeromq in
0.9.2 over Netty.

I built that module because I thought Netty was being slow, but it was just
a red herring for another issue.
On Jul 10, 2014 1:15 PM, "Harsha" <st...@harsha.io> wrote:

>  Storm 0mq package is here https://github.com/ptgoetz/storm-0mq . You
> need to add that package in STORM_HOME/lib
> and add this config to storm.yaml
>
> storm.messaging.transport: "backtype.storm.messaging.zmq"
>
>
> On Thu, Jul 10, 2014, at 10:02 AM, Suparno Datta wrote:
>
> anyone here knows how to switch to zmq from netty? Just wanted to check
> that once before going down to 0.8.1.
>
>
> On 10 July 2014 18:46, Suparno Datta <su...@gmail.com> wrote:
>
> @Stephan, Worked like a charm. How stupid of me not to change the local
> directory.
>
> @Harsha. Didnt solve the original problem :( .
>
> Now getting this ones
>
> #.s.m.n.Client [INFO] Reconnect started for
> Netty-Client-cluster1-fos-ThinkPad-T520/10.42.0.21:6700... [11]
>
> and after 30 retries finally the worker crashes
>
> 14-07-10 18:44:43 b.s.m.n.Client [INFO] Closing Netty Client
> Netty-Client-cluster1-fos-ThinkPad-T520/10.42.0.21:6700
> 2014-07-10 18:44:43 b.s.m.n.Client [INFO] Waiting for pending batchs to be
> sent with Netty-Client-cluster1-fos-ThinkPad-T520/10.42.0.21:6700...,
> timeout: 600000ms, pendings: 0
> 2014-07-10 18:44:43 b.s.util [ERROR] Async loop died!
> java.lang.RuntimeException: java.lang.RuntimeException: Client is being
> closed, and does not take requests any more
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$consume_loop_STAR_$fn__758.invoke(disruptor.clj:94)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_55]
>  Caused by: java.lang.RuntimeException: Client is being closed, and does
> not take requests any more
> at backtype.storm.messaging.netty.Client.send(Client.java:194)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.utils.TransferDrainer.send(TransferDrainer.java:54)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__5927$fn__5928.invoke(worker.clj:322)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__5927.invoke(worker.clj:320)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> ... 6 common frames omitted
> 2014-07-10 18:44:43 b.s.util [INFO] Halting process: ("Async loop died!")
>
> Seems 0.8.1 it is.
>
>
> On 10 July 2014 18:21, Kemper, Stephan <st...@viasat.com> wrote:
>
> We ran into this same problem this week.  The problem isn't with
> ZooKeeper, but the local state files in your ${storm.local.dir}.  If you
> delete the ./localstate directory there and restart the node, you should be
> OK again.
>
> More info on the problem was in last month's "v0.9.2-incubating and .ser
> files" thread from this mailing list.
>
>
> Stephan Kemper
> ViaSat
>
> *From: *Harsha <st...@harsha.io>
> *Reply-To: *"user@storm.incubator.apache.org" <
> user@storm.incubator.apache.org>
> *Date: *Thursday, July 10, 2014 at 9:15 AM
> *To: *"user@storm.incubator.apache.org" <us...@storm.incubator.apache.org>
> *Subject: *Re: b.s.m.n.Client [INFO] Reconnect
>
> Suparno,
>           Old storm data in zookeeper might conflict with newer versions
> of storm. I would suggest you to bring down the topologies and clean
> zookeeper /storm dir.
> -Harsha
>
>
>
> On Thu, Jul 10, 2014, at 09:06 AM, Suparno Datta wrote:
>
> okay that got worse. I just downloaded the 0.9.2. and failed to launch the
> supervisors (nimbus is running though). You don't have to don any clean up
> before you launch the new version right ?
>
> Anyways the stack trace of the error
>
> 014-07-10 18:01:27 b.s.event [ERROR] Error when processing event
> java.lang.RuntimeException: java.io.InvalidClassException:
> clojure.lang.APersistentMap; local class incompatible: stream classdesc
> serialVersionUID = 270281984708184947, local class serialVersionUID =
> 8648225932767613808
> at backtype.storm.utils.Utils.deserialize(Utils.java:93)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.utils.LocalState.snapshot(LocalState.java:45)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.utils.LocalState.get(LocalState.java:56)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:207)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at clojure.lang.AFn.applyToHelper(AFn.java:161) [clojure-1.5.1.jar:na]
> at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na]
> at clojure.core$apply.invoke(core.clj:619) ~[clojure-1.5.1.jar:na]
> at clojure.core$partial$fn__4190.doInvoke(core.clj:2396)
> ~[clojure-1.5.1.jar:na]
> at clojure.lang.RestFn.invoke(RestFn.java:397) ~[clojure-1.5.1.jar:na]
> at backtype.storm.event$event_manager$fn__2378.invoke(event.clj:39)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_55]
> Caused by: java.io.InvalidClassException: clojure.lang.APersistentMap;
> local class incompatible: stream classdesc serialVersionUID =
> 270281984708184947, local class serialVersionUID = 8648225932767613808
> at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
> ~[na:1.7.0_55]
> at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
> ~[na:1.7.0_55]
> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
> ~[na:1.7.0_55]
> at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
> ~[na:1.7.0_55]
> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
> ~[na:1.7.0_55]
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> ~[na:1.7.0_55]
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> ~[na:1.7.0_55]
> at java.util.HashMap.readObject(HashMap.java:1184) ~[na:1.7.0_55]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.7.0_55]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> ~[na:1.7.0_55]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.7.0_55]
> at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_55]
> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
> ~[na:1.7.0_55]
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> ~[na:1.7.0_55]
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> ~[na:1.7.0_55]
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> ~[na:1.7.0_55]
> at backtype.storm.utils.Utils.deserialize(Utils.java:89)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> ... 11 common frames omitted
>
>
> On 10 July 2014 17:20, Harsha <st...@harsha.io> wrote:
>
>
> Yes. As per the change log
> https://github.com/apache/incubator-storm/blob/v0.9.2-incubating/CHANGELOG.md STORM-187
> did make the 0.9.2 release.
>
>
> On Thu, Jul 10, 2014, at 08:11 AM, Suparno Datta wrote:
>
> You think it's fixed in 0.9.2 ?
>
>
>
> On 10 July 2014 17:08, Suparno Datta <su...@gmail.com> wrote:
>
> I just found that too. Seems it's becuse 0.9.1 usses netty by default
> instead of zeromq ( guess thats why it was working with 0.8.1). Presently
> looking for the configuration parameter by which i can tell it to use zmq
> instead of netty. Let me know if you have any clue. Otherwise i just have
> to chuck the 0.9.1 and get back to 0.8.1
>
>
> On 10 July 2014 17:02, Harsha <st...@harsha.io> wrote:
>
>
> Hi Suparno,
>        It might be because of
> https://issues.apache.org/jira/browse/STORM-187. Can you try using
> 0.9.2-incubating release.
> -Harsha
>
>
>
> On Thu, Jul 10, 2014, at 07:38 AM, Suparno Datta wrote:
>
> Hi,
>
>  I am using storm 0.9.1-incubating on a single machine cluster to run a
> simple twitter hashtag extractor.  I am using the Storm-twitter-workshop
> which i found to be extremely useful.
>
> https://github.com/kantega/storm-twitter-workshop
>
> I have used this program before with storm 0.8.1 and it ran like a charm.
> I might mention that was on a server machine with quite 2 quad xeon
> processors.
>
> This time i am trying it on my laptop( i5 , 8GB). But i am constantly
> getting this error in the worker log files
>
> 2014-07-10 13:01:47 b.s.m.n.Client [INFO] Reconnect ... [24]
> 2014-07-10 13:01:58 b.s.m.n.Client [INFO] Reconnect ... [25]
> 2014-07-10 13:02:09 b.s.m.n.Client [INFO] Reconnect ... [26]
> 2014-07-10 13:02:19 STDIO [ERROR] Jul 10, 2014 1:02:19 PM
> org.jboss.netty.channel.DefaultChannelPipeline
> WARNING: An exception was thrown by a user handler while handling an
> exception event ([id: 0x563f7062] EXCEPTION: java.net.ConnectException:
> connection timed out)
> java.lang.IllegalArgumentException: timeout value is negative
> at java.lang.Thread.sleep(Native Method)
> at backtype.storm.messaging.netty.Client.reconnect(Client.java:94)
> at
> backtype.storm.messaging.netty.StormClientHandler.exceptionCaught(StormClientHandler.java:118)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
> at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:140)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:82)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
> at
> org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
>
>
> Now here comes the stranges part. If i declare just one instance of the
> hashtag extractor Bolt it fails to get anything but for more that 1 it does
> manage to get me a few hashtags though with quite high latency. Another
> strange part related to this machine is if i declare more than 2
> supervisor.slots.ports the program doesnt even launch any more showing some
> initialitzation error.
>
> Sorry if i blabbered a lot about the hardware and stuff. But somehow to me
> it seemed quite related to the problem. Any sort of help will be really
> useful.
>
> Thanks,
>
> Suparno
>
>
>
>
>
>
>
>  --
> Suparno Datta
>
>
>
>
> --
> Suparno Datta
>
>
>
>
>
>
> --
> Suparno Datta
>
>
>
>
>
>
>  --
> Suparno Datta
>
>
>
>
> --
> Suparno Datta
>
>
>