You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by "Brian C. Huffman" <bh...@etinternational.com> on 2014/01/03 19:56:21 UTC
Segfault in worker when submitting topology
All,
I'm having trouble getting storm set up and submitting a topology. I've
tried two different installs of storm and associated dependencies: One
using the older storm (0.8.1), zeromq (2.1.7) and jzmq (frozen for
storm). And then another using the newest of each: storm(0.9.0.1),
zeromq (4.0.3), and latest jzmq master.
Then I'm using the storm-starter package with the WordCountTopology
Storm *appears* to be running fine. I have one nimbus node and 3
supervisors. And the UI looks good.
The topology submits fine:
[storm@thor4 bin]$ ./storm jar
/home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar
storm.starter.WordCountTopology wc5
Running: java -client -Dstorm.options=
-Dstorm.home=/share/storm/storm-0.9.0.1
-Djava.library.path=/share/storm/zeromq-4.0.3/lib:/share/storm/jzmq/lib:/usr/lib
-Dstorm.conf.file= -cp
/share/storm/storm-0.9.0.1/storm-netty-0.9.0.1.jar:/share/storm/storm-0.9.0.1/storm-console-logging-0.9.0.1.jar:/share/storm/storm-0.9.0.1/storm-core-0.9.0.1.jar:/share/storm/storm-0.9.0.1/lib/disruptor-2.10.1.jar:/share/storm/storm-0.9.0.1/lib/commons-exec-1.1.jar:/share/storm/storm-0.9.0.1/lib/jetty-6.1.26.jar:/share/storm/storm-0.9.0.1/lib/math.numeric-tower-0.0.1.jar:/share/storm/storm-0.9.0.1/lib/logback-core-1.0.6.jar:/share/storm/storm-0.9.0.1/lib/curator-framework-1.0.1.jar:/share/storm/storm-0.9.0.1/lib/joda-time-2.0.jar:/share/storm/storm-0.9.0.1/lib/ring-jetty-adapter-0.3.11.jar:/share/storm/storm-0.9.0.1/lib/commons-io-1.4.jar:/share/storm/storm-0.9.0.1/lib/logback-classic-1.0.6.jar:/share/storm/storm-0.9.0.1/lib/asm-4.0.jar:/share/storm/storm-0.9.0.1/lib/tools.cli-0.2.2.jar:/share/storm/storm-0.9.0.1/lib/reflectasm-1.07-shaded.jar:/share/storm/storm-0.9.0.1/lib/core.incubator-0.1.0.jar:/share/storm/storm-0.9.0.1/lib/clj-time-0.4.1.jar:/share/storm/storm-0.9.0.1/lib/minlog-1.2.jar:/share/storm/storm-0.9.0.1/lib/kryo-2.17.jar:/share/storm/storm-0.9.0.1/lib/log4j-over-slf4j-1.6.6.jar:/share/storm/storm-0.9.0.1/lib/netty-3.6.3.Final.jar:/share/storm/storm-0.9.0.1/lib/commons-codec-1.4.jar:/share/storm/storm-0.9.0.1/lib/clout-1.0.1.jar:/share/storm/storm-0.9.0.1/lib/objenesis-1.2.jar:/share/storm/storm-0.9.0.1/lib/curator-client-1.0.1.jar:/share/storm/storm-0.9.0.1/lib/mockito-all-1.9.5.jar:/share/storm/storm-0.9.0.1/lib/servlet-api-2.5.jar:/share/storm/storm-0.9.0.1/lib/jzmq-2.1.0.jar:/share/storm/storm-0.9.0.1/lib/json-simple-1.1.jar:/share/storm/storm-0.9.0.1/lib/guava-13.0.jar:/share/storm/storm-0.9.0.1/lib/libthrift7-0.7.0-2.jar:/share/storm/storm-0.9.0.1/lib/slf4j-api-1.6.5.jar:/share/storm/storm-0.9.0.1/lib/hiccup-0.3.6.jar:/share/storm/storm-0.9.0.1/lib/snakeyaml-1.11.jar:/share/storm/storm-0.9.0.1/lib/jetty-util-6.1.26.jar:/share/storm/storm-0.9.0.1/lib/ring-core-1.1.5.jar:/share/storm/storm-0.9.0.1/lib/ring-devel-0.3.11.jar:/share/storm/storm-0.9.0.1/lib/servlet-api-2.5-20081211.jar:/share/storm/storm-0.9.0.1/lib/jline-0.9.94.jar:/share/storm/storm-0.9.0.1/lib/commons-fileupload-1.2.1.jar:/share/storm/storm-0.9.0.1/lib/clojure-1.4.0.jar:/share/storm/storm-0.9.0.1/lib/tools.macro-0.1.0.jar:/share/storm/storm-0.9.0.1/lib/clj-stacktrace-0.2.2.jar:/share/storm/storm-0.9.0.1/lib/jgrapht-0.8.3.jar:/share/storm/storm-0.9.0.1/lib/ring-servlet-0.3.11.jar:/share/storm/storm-0.9.0.1/lib/compojure-1.1.3.jar:/share/storm/storm-0.9.0.1/lib/clojure-complete-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/tools.logging-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/commons-lang-2.5.jar:/share/storm/storm-0.9.0.1/lib/tools.nrepl-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/httpclient-4.1.1.jar:/share/storm/storm-0.9.0.1/lib/httpcore-4.1.jar:/share/storm/storm-0.9.0.1/lib/zookeeper-3.3.3.jar:/share/storm/storm-0.9.0.1/lib/carbonite-1.5.0.jar:/share/storm/storm-0.9.0.1/lib/commons-logging-1.1.1.jar:/share/storm/storm-0.9.0.1/lib/junit-3.8.1.jar:/home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar:/share/storm/storm-0.9.0.1/conf:/share/storm/storm-0.9.0.1/bin
-Dstorm.jar=/home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar
storm.starter.WordCountTopology wc5
375 [main] INFO backtype.storm.StormSubmitter - Jar not uploaded to
master yet. Submitting jar...
389 [main] INFO backtype.storm.StormSubmitter - Uploading topology jar
/home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar to
assigned location:
/usr/local/var/storm/nimbus/inbox/stormjar-e10a559a-59e8-4411-8cd3-c3b6e6c16bc9.jar
410 [main] INFO backtype.storm.StormSubmitter - Successfully uploaded
topology jar to assigned location:
/usr/local/var/storm/nimbus/inbox/stormjar-e10a559a-59e8-4411-8cd3-c3b6e6c16bc9.jar
411 [main] INFO backtype.storm.StormSubmitter - Submitting topology
wc5 in distributed mode with conf
{"topology.workers":3,"topology.debug":true}
846 [main] INFO backtype.storm.StormSubmitter - Finished submitting
topology: wc5
However, it's segfaulting. I see many hs_err_pid files with the
following content:
# Problematic frame:
# C [libzmq.so.3+0x28510] zmq::signaler_t::signaler_t()+0x30
<snip>
Stack: [0x00007f945ac1c000,0x00007f945ad1d000], sp=0x00007f945ad1aed8,
free space=1019k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
C=native code)
C [libzmq.so.3+0x28510] zmq::signaler_t::signaler_t()+0x30
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j org.zeromq.ZMQ$Socket.setLongSockopt(IJ)V+0
j org.zeromq.ZMQ$Socket.setLinger(J)V+17
I've spent a few hours troubleshooting but just can't find a solution.
Can any one make suggestions on what might be wrong?
Thanks,
Brian
Re: Segfault in worker when submitting topology
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
Nathan,
Yep - works with netty.
I did compile both zeromq and jzmq myself. So I don't know what the
issue was. But I'm happy to have it working.
Thanks!
Brian
On 01/03/2014 04:39 PM, Nathan Leung wrote:
> Presuming that's where you keep your libraries, and that you don't
> have the newer libjzmq installed in /share/storm/jzmq/lib it looks
> good. I would double check zmq.. did you build it yourself? Also,
> you can try netty as the transport (see:
> http://storm-project.net/2013/12/08/storm090-released.html) since that
> runs in the jvm. If this works then it would isolate zmq as the
> likely source of the problem (and I would suspect something like the
> installed version being built for a different platform).
>
>
> On Fri, Jan 3, 2014 at 4:25 PM, Brian C. Huffman
> <bhuffman@etinternational.com <ma...@etinternational.com>>
> wrote:
>
> Still breaks with storm 0.9.0.1 and zmq 2.1.7 and frozen jzmq.
>
> Just to be sure this isn't an issue - I've specified a library
> path in storm's config so that it can find the libraries for zmq
> and jzmq. Is this ok?
> java.library.path:
> "/share/storm/zeromq-2.1.7/lib:/share/storm/jzmq/lib:/usr/lib"
>
> Thanks,
> -b
>
>
>
> On 01/03/2014 03:06 PM, Nathan Leung wrote:
>> can you try storm 0.9.0.1 with zmq 2.1.7 and frozen jzmq?
>> assuming that it doesn't work, can you try with netty as your
>> transport? It would seem to me that something is wrong with your
>> zmq library,
>>
>>
>> On Fri, Jan 3, 2014 at 2:58 PM, Brian C. Huffman
>> <bhuffman@etinternational.com
>> <ma...@etinternational.com>> wrote:
>>
>> Yes - that log was from the zmq 4 run.
>>
>> Here's the hs_err_pid file from the 2.1.7 run:
>> # Problematic frame:
>> # C [libzmq.so.1+0x19100] zmq::i_inout::~i_inout()+0x10
>>
>> <snip>
>>
>> Stack: [0x00007f3a337d2000,0x00007f3a338d3000],
>> sp=0x00007f3a338d0f08, free space=1019k
>>
>> Native frames: (J=compiled Java code, j=interpreted, Vv=VM
>> code, C=native code)
>> C [libzmq.so.1+0x19100] zmq::i_inout::~i_inout()+0x10
>>
>>
>> Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
>> j org.zeromq.ZMQ$Socket.setLongSockopt(IJ)V+0
>> j org.zeromq.ZMQ$Socket.setLinger(J)V+17
>>
>> Thanks,
>> Brian
>>
>>
>> On 01/03/2014 02:46 PM, Nathan Leung wrote:
>>> You shouldn't use a newer version of zmq, the API changed.
>>> If you try a newer version of storm you should try it with
>>> zmq 2.1.7 and the frozen jzmq. Is the log you posted from
>>> the zmq 4 run? What does it look like with zmq 2.1.7?
>>>
>>>
>>> On Fri, Jan 3, 2014 at 1:56 PM, Brian C. Huffman
>>> <bhuffman@etinternational.com
>>> <ma...@etinternational.com>> wrote:
>>>
>>> All,
>>>
>>> I'm having trouble getting storm set up and submitting a
>>> topology. I've tried two different installs of storm and
>>> associated dependencies: One using the older storm
>>> (0.8.1), zeromq (2.1.7) and jzmq (frozen for storm).
>>> And then another using the newest of each:
>>> storm(0.9.0.1), zeromq (4.0.3), and latest jzmq master.
>>>
>>> Then I'm using the storm-starter package with the
>>> WordCountTopology
>>>
>>> Storm *appears* to be running fine. I have one nimbus
>>> node and 3 supervisors. And the UI looks good.
>>>
>>> The topology submits fine:
>>> [storm@thor4 bin]$ ./storm jar
>>> /home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar
>>> storm.starter.WordCountTopology wc5
>>> Running: java -client -Dstorm.options=
>>> -Dstorm.home=/share/storm/storm-0.9.0.1
>>> -Djava.library.path=/share/storm/zeromq-4.0.3/lib:/share/storm/jzmq/lib:/usr/lib
>>> -Dstorm.conf.file= -cp
>>> /share/storm/storm-0.9.0.1/storm-netty-0.9.0.1.jar:/share/storm/storm-0.9.0.1/storm-console-logging-0.9.0.1.jar:/share/storm/storm-0.9.0.1/storm-core-0.9.0.1.jar:/share/storm/storm-0.9.0.1/lib/disruptor-2.10.1.jar:/share/storm/storm-0.9.0.1/lib/commons-exec-1.1.jar:/share/storm/storm-0.9.0.1/lib/jetty-6.1.26.jar:/share/storm/storm-0.9.0.1/lib/math.numeric-
>>> <http://0.9.0.1/lib/math.numeric->tower-0.0.1.jar:/share/storm/storm-0.9.0.1/lib/logback-core-1.0.6.jar:/share/storm/storm-0.9.0.1/lib/curator-framework-1.0.1.jar:/share/storm/storm-0.9.0.1/lib/joda-time-2.0.jar:/share/storm/storm-0.9.0.1/lib/ring-jetty-adapter-0.3.11.jar:/share/storm/storm-0.9.0.1/lib/commons-io-1.4.jar:/share/storm/storm-0.9.0.1/lib/logback-classic-1.0.6.jar:/share/storm/storm-0.9.0.1/lib/asm-4.0.jar:/share/storm/storm-0.9.0.1/lib/tools.cli-0.2.2.jar:/share/storm/storm-0.9.0.1/lib/reflectasm-1.07-shaded.jar:/share/storm/storm-0.9.0.1/lib/core.incubator-0
>>> <http://0.9.0.1/lib/core.incubator-0>.1.0.jar:/share/storm/storm-0.9.0.1/lib/clj-time-0.4.1.jar:/share/storm/storm-0.9.0.1/lib/minlog-1.2.jar:/share/storm/storm-0.9.0.1/lib/kryo-2.17.jar:/share/storm/storm-0.9.0.1/lib/log4j-over-slf4j-1.6.6.jar:/share/storm/storm-0.9.0.1/lib/netty-3.6.3.Final.jar:/share/storm/storm-0.9.0.1/lib/commons-codec-1.4.jar:/share/storm/storm-0.9.0.1/lib/clout-1.0.1.jar:/share/storm/storm-0.9.0.1/lib/objenesis-1.2.jar
>>> <http://0.9.0.1/lib/objenesis-1.2.jar>:/share/storm/storm-0.9.0.1/lib/curator-client-1.0.1.jar:/share/storm/storm-0.9.0.1/lib/mockito-all-1.9.5.jar:/share/storm/storm-0.9.0.1/lib/servlet-api-2.5.jar:/share/storm/storm-0.9.0.1/lib/jzmq-2.1.0.jar:/share/storm/storm-0.9.0.1/lib/json-simple-1.1
>>> <http://0.9.0.1/lib/json-simple-1.1>.jar:/share/storm/storm-0.9.0.1/lib/guava-13.0.jar:/share/storm/storm-0.9.0.1/lib/libthrift7-0.7.0-2.jar:/share/storm/storm-0.9.0.1/lib/slf4j-api-1.6.5.jar:/share/storm/storm-0.9.0.1/lib/hiccup-0.3.6.jar:/share/storm/storm-0.9.0.1/lib/snakeyaml-1.11.jar:/share/storm/storm-0.9.0.1/lib/jetty-util-6.1.26.jar:/share/storm/storm-0.9.0.1/lib/ring-core-1.1.5.jar:/share/storm/storm-0.9.0.1/lib/ring-devel-0.3.11.jar:/share/storm/storm-0.9.0.1/lib/servlet-api-2.5-
>>> <http://0.9.0.1/lib/servlet-api-2.5->20081211.jar:/share/storm/storm-0.9.0.1/lib/jline-0.9.94.jar:/share/storm/storm-0.9.0.1/lib/commons-fileupload-1.2.1.jar:/share/storm/storm-0.9.0.1/lib/clojure-1.4.0.jar:/share/storm/storm-0.9.0.1/lib/tools.macro-0.1.0.jar:/share/storm/storm-0.9.0.1/lib/clj-stacktrace-0.2.2.jar:/share/storm/storm-0.9.0.1/lib/jgrapht-0.8.3.jar:/share/storm/storm-0.9.0.1/lib/ring-servlet-0.3.11.jar:/share/storm/storm-0.9.0.1/lib/compojure-1.1.3.jar:/share/storm/storm-0.9.0.1/lib/clojure-complete-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/tools.logging-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/commons-lang-2.5.jar:/share/storm/storm-0.9.0.1/lib/tools.nrepl-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/httpclient-4.1.1.jar:/share/storm/storm-0.9.0.1/lib/httpcore-4.1.jar:/
>>> <http://0.9.0.1/lib/httpcore-4.1.jar:/>share/storm/storm-0.9.0.1/lib/zookeeper-3.3.3.jar:/share/storm/storm-0.9.0.1/lib/carbonite-1.5.0.jar:/share/storm/storm-0.9.0.1/lib/commons-logging-1.1.1.jar:/share/storm/storm-0.9.0.1/lib/junit-3.8.1.jar:/home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar:/share/storm/storm-0.9.0.1/conf:/share/storm/storm-0.9.0.1/bin
>>> -Dstorm.jar=/home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar
>>> storm.starter.WordCountTopology wc5
>>> 375 [main] INFO backtype.storm.StormSubmitter - Jar
>>> not uploaded to master yet. Submitting jar...
>>> 389 [main] INFO backtype.storm.StormSubmitter -
>>> Uploading topology jar
>>> /home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar
>>> to assigned location:
>>> /usr/local/var/storm/nimbus/inbox/stormjar-e10a559a-59e8-4411-8cd3-c3b6e6c16bc9.jar
>>> 410 [main] INFO backtype.storm.StormSubmitter -
>>> Successfully uploaded topology jar to assigned location:
>>> /usr/local/var/storm/nimbus/inbox/stormjar-e10a559a-59e8-4411-8cd3-c3b6e6c16bc9.jar
>>> 411 [main] INFO backtype.storm.StormSubmitter -
>>> Submitting topology wc5 in distributed mode with conf
>>> {"topology.workers":3,"topology.debug":true}
>>> 846 [main] INFO backtype.storm.StormSubmitter -
>>> Finished submitting topology: wc5
>>>
>>>
>>> However, it's segfaulting. I see many hs_err_pid files
>>> with the following content:
>>> # Problematic frame:
>>> # C [libzmq.so.3+0x28510]
>>> zmq::signaler_t::signaler_t()+0x30
>>>
>>> <snip>
>>>
>>> Stack: [0x00007f945ac1c000,0x00007f945ad1d000],
>>> sp=0x00007f945ad1aed8, free space=1019k
>>> Native frames: (J=compiled Java code, j=interpreted,
>>> Vv=VM code, C=native code)
>>> C [libzmq.so.3+0x28510] zmq::signaler_t::signaler_t()+0x30
>>>
>>> Java frames: (J=compiled Java code, j=interpreted, Vv=VM
>>> code)
>>> j org.zeromq.ZMQ$Socket.setLongSockopt(IJ)V+0
>>> j org.zeromq.ZMQ$Socket.setLinger(J)V+17
>>>
>>> I've spent a few hours troubleshooting but just can't
>>> find a solution. Can any one make suggestions on what
>>> might be wrong?
>>>
>>> Thanks,
>>> Brian
>>>
>>>
>>>
>>>
>>
>>
>
>
Re: Segfault in worker when submitting topology
Posted by Nathan Leung <nc...@gmail.com>.
Presuming that's where you keep your libraries, and that you don't have the
newer libjzmq installed in /share/storm/jzmq/lib it looks good. I would
double check zmq.. did you build it yourself? Also, you can try netty as
the transport (see:
http://storm-project.net/2013/12/08/storm090-released.html) since that runs
in the jvm. If this works then it would isolate zmq as the likely source
of the problem (and I would suspect something like the installed version
being built for a different platform).
On Fri, Jan 3, 2014 at 4:25 PM, Brian C. Huffman <
bhuffman@etinternational.com> wrote:
> Still breaks with storm 0.9.0.1 and zmq 2.1.7 and frozen jzmq.
>
> Just to be sure this isn't an issue - I've specified a library path in
> storm's config so that it can find the libraries for zmq and jzmq. Is this
> ok?
> java.library.path:
> "/share/storm/zeromq-2.1.7/lib:/share/storm/jzmq/lib:/usr/lib"
>
> Thanks,
> -b
>
>
>
> On 01/03/2014 03:06 PM, Nathan Leung wrote:
>
> can you try storm 0.9.0.1 with zmq 2.1.7 and frozen jzmq? assuming that
> it doesn't work, can you try with netty as your transport? It would seem
> to me that something is wrong with your zmq library,
>
>
> On Fri, Jan 3, 2014 at 2:58 PM, Brian C. Huffman <
> bhuffman@etinternational.com> wrote:
>
>> Yes - that log was from the zmq 4 run.
>>
>> Here's the hs_err_pid file from the 2.1.7 run:
>> # Problematic frame:
>> # C [libzmq.so.1+0x19100] zmq::i_inout::~i_inout()+0x10
>>
>> <snip>
>>
>> Stack: [0x00007f3a337d2000,0x00007f3a338d3000], sp=0x00007f3a338d0f08,
>> free space=1019k
>>
>> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
>> code)
>> C [libzmq.so.1+0x19100] zmq::i_inout::~i_inout()+0x10
>>
>>
>> Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
>> j org.zeromq.ZMQ$Socket.setLongSockopt(IJ)V+0
>> j org.zeromq.ZMQ$Socket.setLinger(J)V+17
>>
>> Thanks,
>> Brian
>>
>>
>> On 01/03/2014 02:46 PM, Nathan Leung wrote:
>>
>> You shouldn't use a newer version of zmq, the API changed. If you try a
>> newer version of storm you should try it with zmq 2.1.7 and the frozen
>> jzmq. Is the log you posted from the zmq 4 run? What does it look like
>> with zmq 2.1.7?
>>
>>
>> On Fri, Jan 3, 2014 at 1:56 PM, Brian C. Huffman <
>> bhuffman@etinternational.com> wrote:
>>
>>> All,
>>>
>>> I'm having trouble getting storm set up and submitting a topology. I've
>>> tried two different installs of storm and associated dependencies: One
>>> using the older storm (0.8.1), zeromq (2.1.7) and jzmq (frozen for storm).
>>> And then another using the newest of each: storm(0.9.0.1), zeromq (4.0.3),
>>> and latest jzmq master.
>>>
>>> Then I'm using the storm-starter package with the WordCountTopology
>>>
>>> Storm *appears* to be running fine. I have one nimbus node and 3
>>> supervisors. And the UI looks good.
>>>
>>> The topology submits fine:
>>> [storm@thor4 bin]$ ./storm jar
>>> /home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar
>>> storm.starter.WordCountTopology wc5
>>> Running: java -client -Dstorm.options=
>>> -Dstorm.home=/share/storm/storm-0.9.0.1
>>> -Djava.library.path=/share/storm/zeromq-4.0.3/lib:/share/storm/jzmq/lib:/usr/lib
>>> -Dstorm.conf.file= -cp
>>> /share/storm/storm-0.9.0.1/storm-netty-0.9.0.1.jar:/share/storm/storm-0.9.0.1/storm-console-logging-0.9.0.1.jar:/share/storm/storm-0.9.0.1/storm-core-0.9.0.1.jar:/share/storm/storm-0.9.0.1/lib/disruptor-2.10.1.jar:/share/storm/storm-0.9.0.1/lib/commons-exec-1.1.jar:/share/storm/storm-0.9.0.1/lib/jetty-6.1.26.jar:/share/storm/storm-
>>> 0.9.0.1/lib/math.numeric-
>>> tower-0.0.1.jar:/share/storm/storm-0.9.0.1/lib/logback-core-1.0.6.jar:/share/storm/storm-0.9.0.1/lib/curator-framework-1.0.1.jar:/share/storm/storm-0.9.0.1/lib/joda-time-2.0.jar:/share/storm/storm-0.9.0.1/lib/ring-jetty-adapter-0.3.11.jar:/share/storm/storm-0.9.0.1/lib/commons-io-1.4.jar:/share/storm/storm-0.9.0.1/lib/logback-classic-1.0.6.jar:/share/storm/storm-0.9.0.1/lib/asm-4.0.jar:/share/storm/storm-0.9.0.1/lib/tools.cli-0.2.2.jar:/share/storm/storm-0.9.0.1/lib/reflectasm-1.07-shaded.jar:/share/storm/storm-
>>> 0.9.0.1/lib/core.incubator-0
>>> .1.0.jar:/share/storm/storm-0.9.0.1/lib/clj-time-0.4.1.jar:/share/storm/storm-0.9.0.1/lib/minlog-1.2.jar:/share/storm/storm-0.9.0.1/lib/kryo-2.17.jar:/share/storm/storm-0.9.0.1/lib/log4j-over-slf4j-1.6.6.jar:/share/storm/storm-0.9.0.1/lib/netty-3.6.3.Final.jar:/share/storm/storm-0.9.0.1/lib/commons-codec-1.4.jar:/share/storm/storm-0.9.0.1/lib/clout-1.0.1.jar:/share/storm/storm-
>>> 0.9.0.1/lib/objenesis-1.2.jar
>>> :/share/storm/storm-0.9.0.1/lib/curator-client-1.0.1.jar:/share/storm/storm-0.9.0.1/lib/mockito-all-1.9.5.jar:/share/storm/storm-0.9.0.1/lib/servlet-api-2.5.jar:/share/storm/storm-0.9.0.1/lib/jzmq-2.1.0.jar:/share/storm/storm-
>>> 0.9.0.1/lib/json-simple-1.1
>>> .jar:/share/storm/storm-0.9.0.1/lib/guava-13.0.jar:/share/storm/storm-0.9.0.1/lib/libthrift7-0.7.0-2.jar:/share/storm/storm-0.9.0.1/lib/slf4j-api-1.6.5.jar:/share/storm/storm-0.9.0.1/lib/hiccup-0.3.6.jar:/share/storm/storm-0.9.0.1/lib/snakeyaml-1.11.jar:/share/storm/storm-0.9.0.1/lib/jetty-util-6.1.26.jar:/share/storm/storm-0.9.0.1/lib/ring-core-1.1.5.jar:/share/storm/storm-0.9.0.1/lib/ring-devel-0.3.11.jar:/share/storm/storm-
>>> 0.9.0.1/lib/servlet-api-2.5-
>>> 20081211.jar:/share/storm/storm-0.9.0.1/lib/jline-0.9.94.jar:/share/storm/storm-0.9.0.1/lib/commons-fileupload-1.2.1.jar:/share/storm/storm-0.9.0.1/lib/clojure-1.4.0.jar:/share/storm/storm-0.9.0.1/lib/tools.macro-0.1.0.jar:/share/storm/storm-0.9.0.1/lib/clj-stacktrace-0.2.2.jar:/share/storm/storm-0.9.0.1/lib/jgrapht-0.8.3.jar:/share/storm/storm-0.9.0.1/lib/ring-servlet-0.3.11.jar:/share/storm/storm-0.9.0.1/lib/compojure-1.1.3.jar:/share/storm/storm-0.9.0.1/lib/clojure-complete-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/tools.logging-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/commons-lang-2.5.jar:/share/storm/storm-0.9.0.1/lib/tools.nrepl-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/httpclient-4.1.1.jar:/share/storm/storm-
>>> 0.9.0.1/lib/httpcore-4.1.jar:/share/storm/storm-0.9.0.1/lib/zookeeper-3.3.3.jar:/share/storm/storm-0.9.0.1/lib/carbonite-1.5.0.jar:/share/storm/storm-0.9.0.1/lib/commons-logging-1.1.1.jar:/share/storm/storm-0.9.0.1/lib/junit-3.8.1.jar:/home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar:/share/storm/storm-0.9.0.1/conf:/share/storm/storm-0.9.0.1/bin
>>> -Dstorm.jar=/home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar
>>> storm.starter.WordCountTopology wc5
>>> 375 [main] INFO backtype.storm.StormSubmitter - Jar not uploaded to
>>> master yet. Submitting jar...
>>> 389 [main] INFO backtype.storm.StormSubmitter - Uploading topology jar
>>> /home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar to
>>> assigned location:
>>> /usr/local/var/storm/nimbus/inbox/stormjar-e10a559a-59e8-4411-8cd3-c3b6e6c16bc9.jar
>>> 410 [main] INFO backtype.storm.StormSubmitter - Successfully uploaded
>>> topology jar to assigned location:
>>> /usr/local/var/storm/nimbus/inbox/stormjar-e10a559a-59e8-4411-8cd3-c3b6e6c16bc9.jar
>>> 411 [main] INFO backtype.storm.StormSubmitter - Submitting topology
>>> wc5 in distributed mode with conf
>>> {"topology.workers":3,"topology.debug":true}
>>> 846 [main] INFO backtype.storm.StormSubmitter - Finished submitting
>>> topology: wc5
>>>
>>>
>>> However, it's segfaulting. I see many hs_err_pid files with the
>>> following content:
>>> # Problematic frame:
>>> # C [libzmq.so.3+0x28510] zmq::signaler_t::signaler_t()+0x30
>>>
>>> <snip>
>>>
>>> Stack: [0x00007f945ac1c000,0x00007f945ad1d000], sp=0x00007f945ad1aed8,
>>> free space=1019k
>>> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
>>> C=native code)
>>> C [libzmq.so.3+0x28510] zmq::signaler_t::signaler_t()+0x30
>>>
>>> Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
>>> j org.zeromq.ZMQ$Socket.setLongSockopt(IJ)V+0
>>> j org.zeromq.ZMQ$Socket.setLinger(J)V+17
>>>
>>> I've spent a few hours troubleshooting but just can't find a solution.
>>> Can any one make suggestions on what might be wrong?
>>>
>>> Thanks,
>>> Brian
>>>
>>>
>>>
>>>
>>
>>
>
>
Re: Segfault in worker when submitting topology
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
Still breaks with storm 0.9.0.1 and zmq 2.1.7 and frozen jzmq.
Just to be sure this isn't an issue - I've specified a library path in
storm's config so that it can find the libraries for zmq and jzmq. Is
this ok?
java.library.path:
"/share/storm/zeromq-2.1.7/lib:/share/storm/jzmq/lib:/usr/lib"
Thanks,
-b
On 01/03/2014 03:06 PM, Nathan Leung wrote:
> can you try storm 0.9.0.1 with zmq 2.1.7 and frozen jzmq? assuming
> that it doesn't work, can you try with netty as your transport? It
> would seem to me that something is wrong with your zmq library,
>
>
> On Fri, Jan 3, 2014 at 2:58 PM, Brian C. Huffman
> <bhuffman@etinternational.com <ma...@etinternational.com>>
> wrote:
>
> Yes - that log was from the zmq 4 run.
>
> Here's the hs_err_pid file from the 2.1.7 run:
> # Problematic frame:
> # C [libzmq.so.1+0x19100] zmq::i_inout::~i_inout()+0x10
>
> <snip>
>
> Stack: [0x00007f3a337d2000,0x00007f3a338d3000],
> sp=0x00007f3a338d0f08, free space=1019k
>
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
> C=native code)
> C [libzmq.so.1+0x19100] zmq::i_inout::~i_inout()+0x10
>
>
> Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
> j org.zeromq.ZMQ$Socket.setLongSockopt(IJ)V+0
> j org.zeromq.ZMQ$Socket.setLinger(J)V+17
>
> Thanks,
> Brian
>
>
> On 01/03/2014 02:46 PM, Nathan Leung wrote:
>> You shouldn't use a newer version of zmq, the API changed. If
>> you try a newer version of storm you should try it with zmq 2.1.7
>> and the frozen jzmq. Is the log you posted from the zmq 4 run?
>> What does it look like with zmq 2.1.7?
>>
>>
>> On Fri, Jan 3, 2014 at 1:56 PM, Brian C. Huffman
>> <bhuffman@etinternational.com
>> <ma...@etinternational.com>> wrote:
>>
>> All,
>>
>> I'm having trouble getting storm set up and submitting a
>> topology. I've tried two different installs of storm and
>> associated dependencies: One using the older storm (0.8.1),
>> zeromq (2.1.7) and jzmq (frozen for storm). And then another
>> using the newest of each: storm(0.9.0.1), zeromq (4.0.3), and
>> latest jzmq master.
>>
>> Then I'm using the storm-starter package with the
>> WordCountTopology
>>
>> Storm *appears* to be running fine. I have one nimbus node
>> and 3 supervisors. And the UI looks good.
>>
>> The topology submits fine:
>> [storm@thor4 bin]$ ./storm jar
>> /home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar
>> storm.starter.WordCountTopology wc5
>> Running: java -client -Dstorm.options=
>> -Dstorm.home=/share/storm/storm-0.9.0.1
>> -Djava.library.path=/share/storm/zeromq-4.0.3/lib:/share/storm/jzmq/lib:/usr/lib
>> -Dstorm.conf.file= -cp
>> /share/storm/storm-0.9.0.1/storm-netty-0.9.0.1.jar:/share/storm/storm-0.9.0.1/storm-console-logging-0.9.0.1.jar:/share/storm/storm-0.9.0.1/storm-core-0.9.0.1.jar:/share/storm/storm-0.9.0.1/lib/disruptor-2.10.1.jar:/share/storm/storm-0.9.0.1/lib/commons-exec-1.1.jar:/share/storm/storm-0.9.0.1/lib/jetty-6.1.26.jar:/share/storm/storm-0.9.0.1/lib/math.numeric-
>> <http://0.9.0.1/lib/math.numeric->tower-0.0.1.jar:/share/storm/storm-0.9.0.1/lib/logback-core-1.0.6.jar:/share/storm/storm-0.9.0.1/lib/curator-framework-1.0.1.jar:/share/storm/storm-0.9.0.1/lib/joda-time-2.0.jar:/share/storm/storm-0.9.0.1/lib/ring-jetty-adapter-0.3.11.jar:/share/storm/storm-0.9.0.1/lib/commons-io-1.4.jar:/share/storm/storm-0.9.0.1/lib/logback-classic-1.0.6.jar:/share/storm/storm-0.9.0.1/lib/asm-4.0.jar:/share/storm/storm-0.9.0.1/lib/tools.cli-0.2.2.jar:/share/storm/storm-0.9.0.1/lib/reflectasm-1.07-shaded.jar:/share/storm/storm-0.9.0.1/lib/core.incubator-0
>> <http://0.9.0.1/lib/core.incubator-0>.1.0.jar:/share/storm/storm-0.9.0.1/lib/clj-time-0.4.1.jar:/share/storm/storm-0.9.0.1/lib/minlog-1.2.jar:/share/storm/storm-0.9.0.1/lib/kryo-2.17.jar:/share/storm/storm-0.9.0.1/lib/log4j-over-slf4j-1.6.6.jar:/share/storm/storm-0.9.0.1/lib/netty-3.6.3.Final.jar:/share/storm/storm-0.9.0.1/lib/commons-codec-1.4.jar:/share/storm/storm-0.9.0.1/lib/clout-1.0.1.jar:/share/storm/storm-0.9.0.1/lib/objenesis-1.2.jar
>> <http://0.9.0.1/lib/objenesis-1.2.jar>:/share/storm/storm-0.9.0.1/lib/curator-client-1.0.1.jar:/share/storm/storm-0.9.0.1/lib/mockito-all-1.9.5.jar:/share/storm/storm-0.9.0.1/lib/servlet-api-2.5.jar:/share/storm/storm-0.9.0.1/lib/jzmq-2.1.0.jar:/share/storm/storm-0.9.0.1/lib/json-simple-1.1
>> <http://0.9.0.1/lib/json-simple-1.1>.jar:/share/storm/storm-0.9.0.1/lib/guava-13.0.jar:/share/storm/storm-0.9.0.1/lib/libthrift7-0.7.0-2.jar:/share/storm/storm-0.9.0.1/lib/slf4j-api-1.6.5.jar:/share/storm/storm-0.9.0.1/lib/hiccup-0.3.6.jar:/share/storm/storm-0.9.0.1/lib/snakeyaml-1.11.jar:/share/storm/storm-0.9.0.1/lib/jetty-util-6.1.26.jar:/share/storm/storm-0.9.0.1/lib/ring-core-1.1.5.jar:/share/storm/storm-0.9.0.1/lib/ring-devel-0.3.11.jar:/share/storm/storm-0.9.0.1/lib/servlet-api-2.5-
>> <http://0.9.0.1/lib/servlet-api-2.5->20081211.jar:/share/storm/storm-0.9.0.1/lib/jline-0.9.94.jar:/share/storm/storm-0.9.0.1/lib/commons-fileupload-1.2.1.jar:/share/storm/storm-0.9.0.1/lib/clojure-1.4.0.jar:/share/storm/storm-0.9.0.1/lib/tools.macro-0.1.0.jar:/share/storm/storm-0.9.0.1/lib/clj-stacktrace-0.2.2.jar:/share/storm/storm-0.9.0.1/lib/jgrapht-0.8.3.jar:/share/storm/storm-0.9.0.1/lib/ring-servlet-0.3.11.jar:/share/storm/storm-0.9.0.1/lib/compojure-1.1.3.jar:/share/storm/storm-0.9.0.1/lib/clojure-complete-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/tools.logging-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/commons-lang-2.5.jar:/share/storm/storm-0.9.0.1/lib/tools.nrepl-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/httpclient-4.1.1.jar:/share/storm/storm-0.9.0.1/lib/httpcore-4.1.jar:/
>> <http://0.9.0.1/lib/httpcore-4.1.jar:/>share/storm/storm-0.9.0.1/lib/zookeeper-3.3.3.jar:/share/storm/storm-0.9.0.1/lib/carbonite-1.5.0.jar:/share/storm/storm-0.9.0.1/lib/commons-logging-1.1.1.jar:/share/storm/storm-0.9.0.1/lib/junit-3.8.1.jar:/home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar:/share/storm/storm-0.9.0.1/conf:/share/storm/storm-0.9.0.1/bin
>> -Dstorm.jar=/home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar
>> storm.starter.WordCountTopology wc5
>> 375 [main] INFO backtype.storm.StormSubmitter - Jar not
>> uploaded to master yet. Submitting jar...
>> 389 [main] INFO backtype.storm.StormSubmitter - Uploading
>> topology jar
>> /home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar
>> to assigned location:
>> /usr/local/var/storm/nimbus/inbox/stormjar-e10a559a-59e8-4411-8cd3-c3b6e6c16bc9.jar
>> 410 [main] INFO backtype.storm.StormSubmitter -
>> Successfully uploaded topology jar to assigned location:
>> /usr/local/var/storm/nimbus/inbox/stormjar-e10a559a-59e8-4411-8cd3-c3b6e6c16bc9.jar
>> 411 [main] INFO backtype.storm.StormSubmitter - Submitting
>> topology wc5 in distributed mode with conf
>> {"topology.workers":3,"topology.debug":true}
>> 846 [main] INFO backtype.storm.StormSubmitter - Finished
>> submitting topology: wc5
>>
>>
>> However, it's segfaulting. I see many hs_err_pid files with
>> the following content:
>> # Problematic frame:
>> # C [libzmq.so.3+0x28510] zmq::signaler_t::signaler_t()+0x30
>>
>> <snip>
>>
>> Stack: [0x00007f945ac1c000,0x00007f945ad1d000],
>> sp=0x00007f945ad1aed8, free space=1019k
>> Native frames: (J=compiled Java code, j=interpreted, Vv=VM
>> code, C=native code)
>> C [libzmq.so.3+0x28510] zmq::signaler_t::signaler_t()+0x30
>>
>> Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
>> j org.zeromq.ZMQ$Socket.setLongSockopt(IJ)V+0
>> j org.zeromq.ZMQ$Socket.setLinger(J)V+17
>>
>> I've spent a few hours troubleshooting but just can't find a
>> solution. Can any one make suggestions on what might be wrong?
>>
>> Thanks,
>> Brian
>>
>>
>>
>>
>
>
Re: Segfault in worker when submitting topology
Posted by Nathan Leung <nc...@gmail.com>.
can you try storm 0.9.0.1 with zmq 2.1.7 and frozen jzmq? assuming that it
doesn't work, can you try with netty as your transport? It would seem to
me that something is wrong with your zmq library,
On Fri, Jan 3, 2014 at 2:58 PM, Brian C. Huffman <
bhuffman@etinternational.com> wrote:
> Yes - that log was from the zmq 4 run.
>
> Here's the hs_err_pid file from the 2.1.7 run:
> # Problematic frame:
> # C [libzmq.so.1+0x19100] zmq::i_inout::~i_inout()+0x10
>
> <snip>
>
> Stack: [0x00007f3a337d2000,0x00007f3a338d3000], sp=0x00007f3a338d0f08,
> free space=1019k
>
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> C [libzmq.so.1+0x19100] zmq::i_inout::~i_inout()+0x10
>
>
> Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
> j org.zeromq.ZMQ$Socket.setLongSockopt(IJ)V+0
> j org.zeromq.ZMQ$Socket.setLinger(J)V+17
>
> Thanks,
> Brian
>
>
> On 01/03/2014 02:46 PM, Nathan Leung wrote:
>
> You shouldn't use a newer version of zmq, the API changed. If you try a
> newer version of storm you should try it with zmq 2.1.7 and the frozen
> jzmq. Is the log you posted from the zmq 4 run? What does it look like
> with zmq 2.1.7?
>
>
> On Fri, Jan 3, 2014 at 1:56 PM, Brian C. Huffman <
> bhuffman@etinternational.com> wrote:
>
>> All,
>>
>> I'm having trouble getting storm set up and submitting a topology. I've
>> tried two different installs of storm and associated dependencies: One
>> using the older storm (0.8.1), zeromq (2.1.7) and jzmq (frozen for storm).
>> And then another using the newest of each: storm(0.9.0.1), zeromq (4.0.3),
>> and latest jzmq master.
>>
>> Then I'm using the storm-starter package with the WordCountTopology
>>
>> Storm *appears* to be running fine. I have one nimbus node and 3
>> supervisors. And the UI looks good.
>>
>> The topology submits fine:
>> [storm@thor4 bin]$ ./storm jar
>> /home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar
>> storm.starter.WordCountTopology wc5
>> Running: java -client -Dstorm.options=
>> -Dstorm.home=/share/storm/storm-0.9.0.1
>> -Djava.library.path=/share/storm/zeromq-4.0.3/lib:/share/storm/jzmq/lib:/usr/lib
>> -Dstorm.conf.file= -cp
>> /share/storm/storm-0.9.0.1/storm-netty-0.9.0.1.jar:/share/storm/storm-0.9.0.1/storm-console-logging-0.9.0.1.jar:/share/storm/storm-0.9.0.1/storm-core-0.9.0.1.jar:/share/storm/storm-0.9.0.1/lib/disruptor-2.10.1.jar:/share/storm/storm-0.9.0.1/lib/commons-exec-1.1.jar:/share/storm/storm-0.9.0.1/lib/jetty-6.1.26.jar:/share/storm/storm-
>> 0.9.0.1/lib/math.numeric-
>> tower-0.0.1.jar:/share/storm/storm-0.9.0.1/lib/logback-core-1.0.6.jar:/share/storm/storm-0.9.0.1/lib/curator-framework-1.0.1.jar:/share/storm/storm-0.9.0.1/lib/joda-time-2.0.jar:/share/storm/storm-0.9.0.1/lib/ring-jetty-adapter-0.3.11.jar:/share/storm/storm-0.9.0.1/lib/commons-io-1.4.jar:/share/storm/storm-0.9.0.1/lib/logback-classic-1.0.6.jar:/share/storm/storm-0.9.0.1/lib/asm-4.0.jar:/share/storm/storm-0.9.0.1/lib/tools.cli-0.2.2.jar:/share/storm/storm-0.9.0.1/lib/reflectasm-1.07-shaded.jar:/share/storm/storm-
>> 0.9.0.1/lib/core.incubator-0
>> .1.0.jar:/share/storm/storm-0.9.0.1/lib/clj-time-0.4.1.jar:/share/storm/storm-0.9.0.1/lib/minlog-1.2.jar:/share/storm/storm-0.9.0.1/lib/kryo-2.17.jar:/share/storm/storm-0.9.0.1/lib/log4j-over-slf4j-1.6.6.jar:/share/storm/storm-0.9.0.1/lib/netty-3.6.3.Final.jar:/share/storm/storm-0.9.0.1/lib/commons-codec-1.4.jar:/share/storm/storm-0.9.0.1/lib/clout-1.0.1.jar:/share/storm/storm-
>> 0.9.0.1/lib/objenesis-1.2.jar
>> :/share/storm/storm-0.9.0.1/lib/curator-client-1.0.1.jar:/share/storm/storm-0.9.0.1/lib/mockito-all-1.9.5.jar:/share/storm/storm-0.9.0.1/lib/servlet-api-2.5.jar:/share/storm/storm-0.9.0.1/lib/jzmq-2.1.0.jar:/share/storm/storm-
>> 0.9.0.1/lib/json-simple-1.1
>> .jar:/share/storm/storm-0.9.0.1/lib/guava-13.0.jar:/share/storm/storm-0.9.0.1/lib/libthrift7-0.7.0-2.jar:/share/storm/storm-0.9.0.1/lib/slf4j-api-1.6.5.jar:/share/storm/storm-0.9.0.1/lib/hiccup-0.3.6.jar:/share/storm/storm-0.9.0.1/lib/snakeyaml-1.11.jar:/share/storm/storm-0.9.0.1/lib/jetty-util-6.1.26.jar:/share/storm/storm-0.9.0.1/lib/ring-core-1.1.5.jar:/share/storm/storm-0.9.0.1/lib/ring-devel-0.3.11.jar:/share/storm/storm-
>> 0.9.0.1/lib/servlet-api-2.5-
>> 20081211.jar:/share/storm/storm-0.9.0.1/lib/jline-0.9.94.jar:/share/storm/storm-0.9.0.1/lib/commons-fileupload-1.2.1.jar:/share/storm/storm-0.9.0.1/lib/clojure-1.4.0.jar:/share/storm/storm-0.9.0.1/lib/tools.macro-0.1.0.jar:/share/storm/storm-0.9.0.1/lib/clj-stacktrace-0.2.2.jar:/share/storm/storm-0.9.0.1/lib/jgrapht-0.8.3.jar:/share/storm/storm-0.9.0.1/lib/ring-servlet-0.3.11.jar:/share/storm/storm-0.9.0.1/lib/compojure-1.1.3.jar:/share/storm/storm-0.9.0.1/lib/clojure-complete-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/tools.logging-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/commons-lang-2.5.jar:/share/storm/storm-0.9.0.1/lib/tools.nrepl-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/httpclient-4.1.1.jar:/share/storm/storm-
>> 0.9.0.1/lib/httpcore-4.1.jar:/share/storm/storm-0.9.0.1/lib/zookeeper-3.3.3.jar:/share/storm/storm-0.9.0.1/lib/carbonite-1.5.0.jar:/share/storm/storm-0.9.0.1/lib/commons-logging-1.1.1.jar:/share/storm/storm-0.9.0.1/lib/junit-3.8.1.jar:/home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar:/share/storm/storm-0.9.0.1/conf:/share/storm/storm-0.9.0.1/bin
>> -Dstorm.jar=/home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar
>> storm.starter.WordCountTopology wc5
>> 375 [main] INFO backtype.storm.StormSubmitter - Jar not uploaded to
>> master yet. Submitting jar...
>> 389 [main] INFO backtype.storm.StormSubmitter - Uploading topology jar
>> /home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar to
>> assigned location:
>> /usr/local/var/storm/nimbus/inbox/stormjar-e10a559a-59e8-4411-8cd3-c3b6e6c16bc9.jar
>> 410 [main] INFO backtype.storm.StormSubmitter - Successfully uploaded
>> topology jar to assigned location:
>> /usr/local/var/storm/nimbus/inbox/stormjar-e10a559a-59e8-4411-8cd3-c3b6e6c16bc9.jar
>> 411 [main] INFO backtype.storm.StormSubmitter - Submitting topology wc5
>> in distributed mode with conf {"topology.workers":3,"topology.debug":true}
>> 846 [main] INFO backtype.storm.StormSubmitter - Finished submitting
>> topology: wc5
>>
>>
>> However, it's segfaulting. I see many hs_err_pid files with the
>> following content:
>> # Problematic frame:
>> # C [libzmq.so.3+0x28510] zmq::signaler_t::signaler_t()+0x30
>>
>> <snip>
>>
>> Stack: [0x00007f945ac1c000,0x00007f945ad1d000], sp=0x00007f945ad1aed8,
>> free space=1019k
>> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
>> code)
>> C [libzmq.so.3+0x28510] zmq::signaler_t::signaler_t()+0x30
>>
>> Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
>> j org.zeromq.ZMQ$Socket.setLongSockopt(IJ)V+0
>> j org.zeromq.ZMQ$Socket.setLinger(J)V+17
>>
>> I've spent a few hours troubleshooting but just can't find a solution.
>> Can any one make suggestions on what might be wrong?
>>
>> Thanks,
>> Brian
>>
>>
>>
>>
>
>
Re: Segfault in worker when submitting topology
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
Yes - that log was from the zmq 4 run.
Here's the hs_err_pid file from the 2.1.7 run:
# Problematic frame:
# C [libzmq.so.1+0x19100] zmq::i_inout::~i_inout()+0x10
<snip>
Stack: [0x00007f3a337d2000,0x00007f3a338d3000], sp=0x00007f3a338d0f08,
free space=1019k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
C=native code)
C [libzmq.so.1+0x19100] zmq::i_inout::~i_inout()+0x10
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j org.zeromq.ZMQ$Socket.setLongSockopt(IJ)V+0
j org.zeromq.ZMQ$Socket.setLinger(J)V+17
Thanks,
Brian
On 01/03/2014 02:46 PM, Nathan Leung wrote:
> You shouldn't use a newer version of zmq, the API changed. If you try
> a newer version of storm you should try it with zmq 2.1.7 and the
> frozen jzmq. Is the log you posted from the zmq 4 run? What does it
> look like with zmq 2.1.7?
>
>
> On Fri, Jan 3, 2014 at 1:56 PM, Brian C. Huffman
> <bhuffman@etinternational.com <ma...@etinternational.com>>
> wrote:
>
> All,
>
> I'm having trouble getting storm set up and submitting a topology.
> I've tried two different installs of storm and associated
> dependencies: One using the older storm (0.8.1), zeromq (2.1.7)
> and jzmq (frozen for storm). And then another using the newest of
> each: storm(0.9.0.1), zeromq (4.0.3), and latest jzmq master.
>
> Then I'm using the storm-starter package with the WordCountTopology
>
> Storm *appears* to be running fine. I have one nimbus node and 3
> supervisors. And the UI looks good.
>
> The topology submits fine:
> [storm@thor4 bin]$ ./storm jar
> /home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar
> storm.starter.WordCountTopology wc5
> Running: java -client -Dstorm.options=
> -Dstorm.home=/share/storm/storm-0.9.0.1
> -Djava.library.path=/share/storm/zeromq-4.0.3/lib:/share/storm/jzmq/lib:/usr/lib
> -Dstorm.conf.file= -cp
> /share/storm/storm-0.9.0.1/storm-netty-0.9.0.1.jar:/share/storm/storm-0.9.0.1/storm-console-logging-0.9.0.1.jar:/share/storm/storm-0.9.0.1/storm-core-0.9.0.1.jar:/share/storm/storm-0.9.0.1/lib/disruptor-2.10.1.jar:/share/storm/storm-0.9.0.1/lib/commons-exec-1.1.jar:/share/storm/storm-0.9.0.1/lib/jetty-6.1.26.jar:/share/storm/storm-0.9.0.1/lib/math.numeric-
> <http://0.9.0.1/lib/math.numeric->tower-0.0.1.jar:/share/storm/storm-0.9.0.1/lib/logback-core-1.0.6.jar:/share/storm/storm-0.9.0.1/lib/curator-framework-1.0.1.jar:/share/storm/storm-0.9.0.1/lib/joda-time-2.0.jar:/share/storm/storm-0.9.0.1/lib/ring-jetty-adapter-0.3.11.jar:/share/storm/storm-0.9.0.1/lib/commons-io-1.4.jar:/share/storm/storm-0.9.0.1/lib/logback-classic-1.0.6.jar:/share/storm/storm-0.9.0.1/lib/asm-4.0.jar:/share/storm/storm-0.9.0.1/lib/tools.cli-0.2.2.jar:/share/storm/storm-0.9.0.1/lib/reflectasm-1.07-shaded.jar:/share/storm/storm-0.9.0.1/lib/core.incubator-0
> <http://0.9.0.1/lib/core.incubator-0>.1.0.jar:/share/storm/storm-0.9.0.1/lib/clj-time-0.4.1.jar:/share/storm/storm-0.9.0.1/lib/minlog-1.2.jar:/share/storm/storm-0.9.0.1/lib/kryo-2.17.jar:/share/storm/storm-0.9.0.1/lib/log4j-over-slf4j-1.6.6.jar:/share/storm/storm-0.9.0.1/lib/netty-3.6.3.Final.jar:/share/storm/storm-0.9.0.1/lib/commons-codec-1.4.jar:/share/storm/storm-0.9.0.1/lib/clout-1.0.1.jar:/share/storm/storm-0.9.0.1/lib/objenesis-1.2.jar
> <http://0.9.0.1/lib/objenesis-1.2.jar>:/share/storm/storm-0.9.0.1/lib/curator-client-1.0.1.jar:/share/storm/storm-0.9.0.1/lib/mockito-all-1.9.5.jar:/share/storm/storm-0.9.0.1/lib/servlet-api-2.5.jar:/share/storm/storm-0.9.0.1/lib/jzmq-2.1.0.jar:/share/storm/storm-0.9.0.1/lib/json-simple-1.1
> <http://0.9.0.1/lib/json-simple-1.1>.jar:/share/storm/storm-0.9.0.1/lib/guava-13.0.jar:/share/storm/storm-0.9.0.1/lib/libthrift7-0.7.0-2.jar:/share/storm/storm-0.9.0.1/lib/slf4j-api-1.6.5.jar:/share/storm/storm-0.9.0.1/lib/hiccup-0.3.6.jar:/share/storm/storm-0.9.0.1/lib/snakeyaml-1.11.jar:/share/storm/storm-0.9.0.1/lib/jetty-util-6.1.26.jar:/share/storm/storm-0.9.0.1/lib/ring-core-1.1.5.jar:/share/storm/storm-0.9.0.1/lib/ring-devel-0.3.11.jar:/share/storm/storm-0.9.0.1/lib/servlet-api-2.5-
> <http://0.9.0.1/lib/servlet-api-2.5->20081211.jar:/share/storm/storm-0.9.0.1/lib/jline-0.9.94.jar:/share/storm/storm-0.9.0.1/lib/commons-fileupload-1.2.1.jar:/share/storm/storm-0.9.0.1/lib/clojure-1.4.0.jar:/share/storm/storm-0.9.0.1/lib/tools.macro-0.1.0.jar:/share/storm/storm-0.9.0.1/lib/clj-stacktrace-0.2.2.jar:/share/storm/storm-0.9.0.1/lib/jgrapht-0.8.3.jar:/share/storm/storm-0.9.0.1/lib/ring-servlet-0.3.11.jar:/share/storm/storm-0.9.0.1/lib/compojure-1.1.3.jar:/share/storm/storm-0.9.0.1/lib/clojure-complete-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/tools.logging-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/commons-lang-2.5.jar:/share/storm/storm-0.9.0.1/lib/tools.nrepl-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/httpclient-4.1.1.jar:/share/storm/storm-0.9.0.1/lib/httpcore-4.1.jar:/
> <http://0.9.0.1/lib/httpcore-4.1.jar:/>share/storm/storm-0.9.0.1/lib/zookeeper-3.3.3.jar:/share/storm/storm-0.9.0.1/lib/carbonite-1.5.0.jar:/share/storm/storm-0.9.0.1/lib/commons-logging-1.1.1.jar:/share/storm/storm-0.9.0.1/lib/junit-3.8.1.jar:/home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar:/share/storm/storm-0.9.0.1/conf:/share/storm/storm-0.9.0.1/bin
> -Dstorm.jar=/home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar
> storm.starter.WordCountTopology wc5
> 375 [main] INFO backtype.storm.StormSubmitter - Jar not uploaded
> to master yet. Submitting jar...
> 389 [main] INFO backtype.storm.StormSubmitter - Uploading
> topology jar
> /home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar
> to assigned location:
> /usr/local/var/storm/nimbus/inbox/stormjar-e10a559a-59e8-4411-8cd3-c3b6e6c16bc9.jar
> 410 [main] INFO backtype.storm.StormSubmitter - Successfully
> uploaded topology jar to assigned location:
> /usr/local/var/storm/nimbus/inbox/stormjar-e10a559a-59e8-4411-8cd3-c3b6e6c16bc9.jar
> 411 [main] INFO backtype.storm.StormSubmitter - Submitting
> topology wc5 in distributed mode with conf
> {"topology.workers":3,"topology.debug":true}
> 846 [main] INFO backtype.storm.StormSubmitter - Finished
> submitting topology: wc5
>
>
> However, it's segfaulting. I see many hs_err_pid files with the
> following content:
> # Problematic frame:
> # C [libzmq.so.3+0x28510] zmq::signaler_t::signaler_t()+0x30
>
> <snip>
>
> Stack: [0x00007f945ac1c000,0x00007f945ad1d000],
> sp=0x00007f945ad1aed8, free space=1019k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
> C=native code)
> C [libzmq.so.3+0x28510] zmq::signaler_t::signaler_t()+0x30
>
> Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
> j org.zeromq.ZMQ$Socket.setLongSockopt(IJ)V+0
> j org.zeromq.ZMQ$Socket.setLinger(J)V+17
>
> I've spent a few hours troubleshooting but just can't find a
> solution. Can any one make suggestions on what might be wrong?
>
> Thanks,
> Brian
>
>
>
>
Re: Segfault in worker when submitting topology
Posted by Nathan Leung <nc...@gmail.com>.
You shouldn't use a newer version of zmq, the API changed. If you try a
newer version of storm you should try it with zmq 2.1.7 and the frozen
jzmq. Is the log you posted from the zmq 4 run? What does it look like
with zmq 2.1.7?
On Fri, Jan 3, 2014 at 1:56 PM, Brian C. Huffman <
bhuffman@etinternational.com> wrote:
> All,
>
> I'm having trouble getting storm set up and submitting a topology. I've
> tried two different installs of storm and associated dependencies: One
> using the older storm (0.8.1), zeromq (2.1.7) and jzmq (frozen for storm).
> And then another using the newest of each: storm(0.9.0.1), zeromq (4.0.3),
> and latest jzmq master.
>
> Then I'm using the storm-starter package with the WordCountTopology
>
> Storm *appears* to be running fine. I have one nimbus node and 3
> supervisors. And the UI looks good.
>
> The topology submits fine:
> [storm@thor4 bin]$ ./storm jar /home/storm/storm-starter/
> target/storm-starter-0.0.1-SNAPSHOT.jar storm.starter.WordCountTopology
> wc5
> Running: java -client -Dstorm.options= -Dstorm.home=/share/storm/storm-0.9.0.1
> -Djava.library.path=/share/storm/zeromq-4.0.3/lib:/share/storm/jzmq/lib:/usr/lib
> -Dstorm.conf.file= -cp /share/storm/storm-0.9.0.1/
> storm-netty-0.9.0.1.jar:/share/storm/storm-0.9.0.1/
> storm-console-logging-0.9.0.1.jar:/share/storm/storm-0.9.0.
> 1/storm-core-0.9.0.1.jar:/share/storm/storm-0.9.0.1/lib/
> disruptor-2.10.1.jar:/share/storm/storm-0.9.0.1/lib/
> commons-exec-1.1.jar:/share/storm/storm-0.9.0.1/lib/jetty-
> 6.1.26.jar:/share/storm/storm-0.9.0.1/lib/math.numeric-
> tower-0.0.1.jar:/share/storm/storm-0.9.0.1/lib/logback-
> core-1.0.6.jar:/share/storm/storm-0.9.0.1/lib/curator-
> framework-1.0.1.jar:/share/storm/storm-0.9.0.1/lib/joda-
> time-2.0.jar:/share/storm/storm-0.9.0.1/lib/ring-jetty-
> adapter-0.3.11.jar:/share/storm/storm-0.9.0.1/lib/
> commons-io-1.4.jar:/share/storm/storm-0.9.0.1/lib/
> logback-classic-1.0.6.jar:/share/storm/storm-0.9.0.1/lib/
> asm-4.0.jar:/share/storm/storm-0.9.0.1/lib/tools.cli-0.
> 2.2.jar:/share/storm/storm-0.9.0.1/lib/reflectasm-1.07-
> shaded.jar:/share/storm/storm-0.9.0.1/lib/core.incubator-0.
> 1.0.jar:/share/storm/storm-0.9.0.1/lib/clj-time-0.4.1.jar:/
> share/storm/storm-0.9.0.1/lib/minlog-1.2.jar:/share/storm/
> storm-0.9.0.1/lib/kryo-2.17.jar:/share/storm/storm-0.9.0.
> 1/lib/log4j-over-slf4j-1.6.6.jar:/share/storm/storm-0.9.0.
> 1/lib/netty-3.6.3.Final.jar:/share/storm/storm-0.9.0.1/lib/
> commons-codec-1.4.jar:/share/storm/storm-0.9.0.1/lib/clout-
> 1.0.1.jar:/share/storm/storm-0.9.0.1/lib/objenesis-1.2.jar:
> /share/storm/storm-0.9.0.1/lib/curator-client-1.0.1.jar:/
> share/storm/storm-0.9.0.1/lib/mockito-all-1.9.5.jar:/share/
> storm/storm-0.9.0.1/lib/servlet-api-2.5.jar:/share/
> storm/storm-0.9.0.1/lib/jzmq-2.1.0.jar:/share/storm/storm-
> 0.9.0.1/lib/json-simple-1.1.jar:/share/storm/storm-0.9.0.
> 1/lib/guava-13.0.jar:/share/storm/storm-0.9.0.1/lib/
> libthrift7-0.7.0-2.jar:/share/storm/storm-0.9.0.1/lib/slf4j-
> api-1.6.5.jar:/share/storm/storm-0.9.0.1/lib/hiccup-0.3.
> 6.jar:/share/storm/storm-0.9.0.1/lib/snakeyaml-1.11.jar:/
> share/storm/storm-0.9.0.1/lib/jetty-util-6.1.26.jar:/share/
> storm/storm-0.9.0.1/lib/ring-core-1.1.5.jar:/share/storm/
> storm-0.9.0.1/lib/ring-devel-0.3.11.jar:/share/storm/storm-
> 0.9.0.1/lib/servlet-api-2.5-20081211.jar:/share/storm/
> storm-0.9.0.1/lib/jline-0.9.94.jar:/share/storm/storm-0.9.
> 0.1/lib/commons-fileupload-1.2.1.jar:/share/storm/storm-0.
> 9.0.1/lib/clojure-1.4.0.jar:/share/storm/storm-0.9.0.1/lib/
> tools.macro-0.1.0.jar:/share/storm/storm-0.9.0.1/lib/clj-
> stacktrace-0.2.2.jar:/share/storm/storm-0.9.0.1/lib/
> jgrapht-0.8.3.jar:/share/storm/storm-0.9.0.1/lib/ring-
> servlet-0.3.11.jar:/share/storm/storm-0.9.0.1/lib/
> compojure-1.1.3.jar:/share/storm/storm-0.9.0.1/lib/
> clojure-complete-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/
> tools.logging-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/
> commons-lang-2.5.jar:/share/storm/storm-0.9.0.1/lib/tools.
> nrepl-0.2.3.jar:/share/storm/storm-0.9.0.1/lib/httpclient-
> 4.1.1.jar:/share/storm/storm-0.9.0.1/lib/httpcore-4.1.jar:/
> share/storm/storm-0.9.0.1/lib/zookeeper-3.3.3.jar:/share/
> storm/storm-0.9.0.1/lib/carbonite-1.5.0.jar:/share/
> storm/storm-0.9.0.1/lib/commons-logging-1.1.1.jar:/
> share/storm/storm-0.9.0.1/lib/junit-3.8.1.jar:/home/storm/
> storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar:/
> share/storm/storm-0.9.0.1/conf:/share/storm/storm-0.9.0.1/bin
> -Dstorm.jar=/home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar
> storm.starter.WordCountTopology wc5
> 375 [main] INFO backtype.storm.StormSubmitter - Jar not uploaded to
> master yet. Submitting jar...
> 389 [main] INFO backtype.storm.StormSubmitter - Uploading topology jar
> /home/storm/storm-starter/target/storm-starter-0.0.1-SNAPSHOT.jar to
> assigned location: /usr/local/var/storm/nimbus/
> inbox/stormjar-e10a559a-59e8-4411-8cd3-c3b6e6c16bc9.jar
> 410 [main] INFO backtype.storm.StormSubmitter - Successfully uploaded
> topology jar to assigned location: /usr/local/var/storm/nimbus/
> inbox/stormjar-e10a559a-59e8-4411-8cd3-c3b6e6c16bc9.jar
> 411 [main] INFO backtype.storm.StormSubmitter - Submitting topology wc5
> in distributed mode with conf {"topology.workers":3,"topology.debug":true}
> 846 [main] INFO backtype.storm.StormSubmitter - Finished submitting
> topology: wc5
>
>
> However, it's segfaulting. I see many hs_err_pid files with the following
> content:
> # Problematic frame:
> # C [libzmq.so.3+0x28510] zmq::signaler_t::signaler_t()+0x30
>
> <snip>
>
> Stack: [0x00007f945ac1c000,0x00007f945ad1d000], sp=0x00007f945ad1aed8,
> free space=1019k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> C [libzmq.so.3+0x28510] zmq::signaler_t::signaler_t()+0x30
>
> Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
> j org.zeromq.ZMQ$Socket.setLongSockopt(IJ)V+0
> j org.zeromq.ZMQ$Socket.setLinger(J)V+17
>
> I've spent a few hours troubleshooting but just can't find a solution.
> Can any one make suggestions on what might be wrong?
>
> Thanks,
> Brian
>
>
>
>