You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Navin Ipe <na...@searchlighthealth.com> on 2016/05/23 06:55:49 UTC
Topology gets submitted to Storm and then nothing happens
I installed zookeeper on my local system. Started it. Extracted Storm 1.0.1
on my local system and *started it via this script:*
*#!/bin/bash./storm nimbus &./storm supervisor &./storm ui &*
*storm.yaml has:*
* storm.zookeeper.servers: - "localhost" nimbus.seeds: ["localhost"]*
*and zoo.conf has:*
*tickTime=2000initLimit=10syncLimit=5dataDir=/Users/nav/programming/zookeeper-3.4.8/dataclientPort=2181*
*Using this version of Storm via Gradle.*
*dependencies { compile 'org.apache.storm:storm-core:1.0.1'*
*Firewall on my system*
Switched off
*and submitting topology like this:*
*System.setProperty("storm.jar",
"/Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar");
StormSubmitter.submitTopology(c.TOPO_NAME, config,
builder.createTopology());*
But on running it in my program's build/lib folder with java -jar
myProgram.jar, the topology gets submitted (and I can see the submitted
topology on the storm UI via http://localhost:8080),* but after submitting,
no emits happen. The program seems to freeze.*
*BUILD SUCCESSFULTotal time: 12.78 secsThis build could be faster, please
consider using the Gradle Daemon:
https://docs.gradle.org/2.10/userguide/gradle_daemon.html
<https://docs.gradle.org/2.10/userguide/gradle_daemon.html>569 [main]
INFO c.s.c.s.Queries_ReaderWriter - Waiting few seconds until table is
dropped...589 [main] INFO c.s.c.s.Queries_ReaderWriter - New table
created.1007 [main] INFO o.a.s.StormSubmitter - Generated ZooKeeper secret
payload for MD5-digest: -5943870921993217077:-90358300078063322081074
[main] INFO o.a.s.s.a.AuthUtils - Got AutoCreds []1124 [main] INFO
o.a.s.StormSubmitter - Uploading topology jar
/Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar to
assigned location:
/Users/nav/programming/apache-storm-1.0.1/storm-local/nimbus/inbox/stormjar-036472f9-15ce-4f09-8663-b1500942e696.jar1271
[main] INFO o.a.s.StormSubmitter - Successfully uploaded topology jar to
assigned location:
/Users/nav/programming/apache-storm-1.0.1/storm-local/nimbus/inbox/stormjar-036472f9-15ce-4f09-8663-b1500942e696.jar1271
[main] INFO o.a.s.StormSubmitter - Submitting topology MyTopology in
distributed mode with conf
{"storm.zookeeper.topology.auth.scheme":"digest","topology.workers":2,"topology.debug":false,"topology.max.spout.pending":10,"storm.zookeeper.topology.auth.payload":"-5943870921993217077:-9035830007806332208","topology.transfer.buffer.size":32,"topology.executor.send.buffer.size":16384,"topology.executor.receive.buffer.size":16384}1427
[main] INFO o.a.s.StormSubmitter - Finished submitting topology:
MyTopology1427 [main] INFO c.s.c.CrystalStorm - Topology submitted*
*Two questions:*
1. What usually causes this kind of problem? I've seen the answers on
StackOverflow, but they didn't help.
2. Which log files would provide the most useful info in such situations?
--
Regards,
Navin
Re: Topology gets submitted to Storm and then nothing happens
Posted by Navin Ipe <na...@searchlighthealth.com>.
And a sample of what I tried, for which I'm still getting the error of:
Caused by: java.lang.RuntimeException: java.io.IOException: Found multiple
defaults.yaml resources. You're probably bundling the Storm jars with your
topology jar.
apply plugin: 'java'
apply plugin: 'eclipse'
defaultTasks 'fatJar'
//create a single Jar with all dependencies
task fatJar(type: Jar) {
version = '0.1.0'
baseName = 'myprogram'
manifest {
attributes 'Implementation-Title': baseName,
'Implementation-Version': version,
'Main-Class' : 'com.slh.myprogram.myprogram'
}
from { (configurations.compile - configurations.provided ).collect {
it.isDirectory() ? it : zipTree(it) } }
with jar
}
sourceCompatibility = '1.8'
[compileJava, compileTestJava]*.options*.encoding = 'UTF-8'
if (!hasProperty('mainClass')) {
ext.mainClass = 'com.slh.myprogram.myprogram'
}
repositories {
mavenCentral()
maven{
url 'http://clojars.org/repo'
}
}
configurations {
provided
compile.extendsFrom provided
}
dependencies {
//---apache storm
compile 'org.apache.storm:storm-core:1.0.1', {
ext {
fatJarExclude = true
}
}
testCompile group: 'junit', name: 'junit', version: '4.10'
}
On Mon, May 23, 2016 at 4:05 PM, Navin Ipe <na...@searchlighthealth.com>
wrote:
> Anyone with Gradle experience, kindly help. I've tried for the past 40
> minutes with help from posts like this
> <http://stackoverflow.com/questions/17421726/making-the-storm-jars-compile-time-only-in-a-gradle-project>,
> this
> <http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/#build-a-correct-standalone--fat-jar-file-of-my-storm-code>,
> this
> <http://kennethjorgensen.com/blog/2014/fat-jars-with-excluded-dependencies-in-gradle>
> and this
> <https://discuss.gradle.org/t/removing-dependencies-from-a-jar-file-during-jar-task/5521>,
> but it isn't working.
> This is all my original gradle script contains:
>
> apply plugin: 'java'
> apply plugin: 'eclipse'
>
> defaultTasks 'jar'
>
> jar {
> from {
> (configurations.runtime).collect {
> it.isDirectory() ? it : zipTree(it)
> }
> }
> manifest {
> attributes 'Main-Class': 'com.slh.myprogram.MyProgram'
> }
> }
>
> sourceCompatibility = '1.8'
> [compileJava, compileTestJava]*.options*.encoding = 'UTF-8'
>
> if (!hasProperty('mainClass')) {
> ext.mainClass = 'com.slh.myprogram.MyProgram'
> }
>
> repositories {
> mavenCentral()
> }
>
> dependencies {
> compile 'org.apache.storm:storm-core:1.0.1'
> testCompile group: 'junit', name: 'junit', version: '4.10'
> }
>
>
> On Mon, May 23, 2016 at 3:18 PM, Matthias J. Sax <mj...@apache.org> wrote:
>
>> You are doing it basically correct. But you use the wrong jar file...
>>
>> The jar you submit must contain your spouts and bolt classes, but not
>> the classes of Storm itself.
>>
>> I am not too familiar with Gradle, but you need to fix your build script
>> to assemble your topology jar file correctly.
>>
>> It would help, if you post your gradle file here, for other to help you
>> out with this.
>>
>> -Matthias
>>
>> On 05/23/2016 11:45 AM, Navin Ipe wrote:
>> > Bingo. Supervisor.log showed:
>> > /Worker Process 17d197c0-80c8-408d-ac70-11970a93d2b9 exited with code: 1
>> > Worker Process 204c20f5-a6d2-4739-b010-c90e651cb2cf exited with code: 1
>> > 204c20f5-a6d2-4739-b010-c90e651cb2cf still hasn't started
>> > /
>> > And the worker logs were located deep inside here
>> >
>> /Users/nav/programming/apache-storm-1.0.1/*logs/workers-artifacts*/MyTopologyName-6-1463986744/6701/worker.log.err
>> >
>> > *The error:*
>> > /Caused by: java.io.IOException: Found multiple defaults.yaml resources.
>> > You're probably bundling the Storm jars with your topology jar.
>> >
>> [jar:file:/Users/nav/programming/apache-storm-1.0.1/storm-local/supervisor/stormdist/MyTopologyName-6-1463986744/stormjar.jar!/defaults.yaml,
>> >
>> jar:file:/Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar!/defaults.yaml]
>> > at
>> > org.apache.storm.utils.Utils.getConfigFileInputStream(Utils.java:333)
>> > at
>> > org.apache.storm.utils.Utils.findAndReadConfigFile(Utils.java:287)
>> > ... 8 more
>> > Exception in thread "main" java.lang.ExceptionInInitializerError/
>> >
>> > But I don't understand. If I don't include the storm jars in the Gradle
>> > dependency, then I won't be able to compile the program itself.
>> >
>> > *What I'm doing:*
>> > *Using this version of Storm via Gradle in Netbeans.*
>> > dependencies { compile 'org.apache.storm:storm-core:1.0.1'
>> >
>> > *and submitting topology like this:*
>> > System.setProperty("storm.jar",
>> > "/Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar");
>> >
>> > StormSubmitter.submitTopology(c.TOPO_NAME, config,
>> > builder.createTopology());
>> >
>> > Is there some other way to do this?
>> >
>> >
>> >
>> > On Mon, May 23, 2016 at 1:36 PM, Matthias J. Sax <mjsax@apache.org
>> > <ma...@apache.org>> wrote:
>> >
>> > How many workers did you configure (ie, storm.yaml parameter
>> > "supervisor.slots.ports" -- the number of available slots is also
>> shown
>> > in the web UI)?
>> >
>> > If there are free workers available, look into logs/worker*.log --
>> if
>> > worker*.log is not available, no worker was started. You might also
>> have
>> > a look into logs/supervisor.log and maybe logs/nimbus.log
>> >
>> >
>> > -Matthias
>> >
>> > On 05/23/2016 08:55 AM, Navin Ipe wrote:
>> > > I installed zookeeper on my local system. Started it. Extracted
>> Storm
>> > > 1.0.1 on my local system and *started it via this script:*
>> > > /#!/bin/bash
>> > > ./storm nimbus &
>> > > ./storm supervisor &
>> > > ./storm ui &/
>> > >
>> > > *storm.yaml has:*
>> > > / storm.zookeeper.servers:
>> > > - "localhost"
>> > > nimbus.seeds: ["localhost"]/
>> > >
>> > > *and zoo.conf has:*
>> > > /tickTime=2000
>> > > initLimit=10
>> > > syncLimit=5
>> > > dataDir=/Users/nav/programming/zookeeper-3.4.8/data
>> > > clientPort=2181/
>> > > *
>> > > *
>> > > *Using this version of Storm via Gradle.*
>> > > /dependencies { compile 'org.apache.storm:storm-core:1.0.1'
>> > >
>> > > /
>> > > *Firewall on my system
>> > > *
>> > > Switched off*
>> > > *
>> > > //
>> > >
>> > > *and submitting topology like this:*
>> > > /System.setProperty("storm.jar",
>> > >
>> "/Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar");
>> > >
>> > > StormSubmitter.submitTopology(c.TOPO_NAME, config,
>> > > builder.createTopology());
>> > > /
>> > >
>> > >
>> > > But on running it in my program's build/lib folder with java -jar
>> > > myProgram.jar, the topology gets submitted (and I can see the
>> submitted
>> > > topology on the storm UI via http://localhost:8080),*but after
>> > > submitting, no emits happen. The program seems to freeze.*
>> > > /BUILD SUCCESSFUL
>> > >
>> > > Total time: 12.78 secs
>> > >
>> > > This build could be faster, please consider using the Gradle
>> Daemon:
>> > > https://docs.gradle.org/2.10/userguide/gradle_daemon.html
>> > > 569 [main] INFO c.s.c.s.Queries_ReaderWriter - Waiting few
>> seconds
>> > > until table is dropped...
>> > > 589 [main] INFO c.s.c.s.Queries_ReaderWriter - New table
>> created.
>> > > 1007 [main] INFO o.a.s.StormSubmitter - Generated ZooKeeper
>> secret
>> > > payload for MD5-digest: -5943870921993217077:-9035830007806332208
>> > > 1074 [main] INFO o.a.s.s.a.AuthUtils - Got AutoCreds []
>> > > 1124 [main] INFO o.a.s.StormSubmitter - Uploading topology jar
>> > >
>> /Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar to
>> > > assigned location:
>> > >
>> /Users/nav/programming/apache-storm-1.0.1/storm-local/nimbus/inbox/stormjar-036472f9-15ce-4f09-8663-b1500942e696.jar
>> > > 1271 [main] INFO o.a.s.StormSubmitter - Successfully uploaded
>> topology
>> > > jar to assigned location:
>> > >
>> /Users/nav/programming/apache-storm-1.0.1/storm-local/nimbus/inbox/stormjar-036472f9-15ce-4f09-8663-b1500942e696.jar
>> > > 1271 [main] INFO o.a.s.StormSubmitter - Submitting topology
>> MyTopology
>> > > in distributed mode with conf
>> > >
>> {"storm.zookeeper.topology.auth.scheme":"digest","topology.workers":2,"topology.debug":false,"topology.max.spout.pending":10,"storm.zookeeper.topology.auth.payload":"-5943870921993217077:-9035830007806332208","topology.transfer.buffer.size":32,"topology.executor.send.buffer.size":16384,"topology.executor.receive.buffer.size":16384}
>> > > 1427 [main] INFO o.a.s.StormSubmitter - Finished submitting
>> topology:
>> > > MyTopology
>> > > 1427 [main] INFO c.s.c.CrystalStorm - Topology submitted/
>> > >
>> > > *Two questions:*
>> > > 1. What usually causes this kind of problem? I've seen the
>> answers on
>> > > StackOverflow, but they didn't help.
>> > > 2. Which log files would provide the most useful info in such
>> > situations?
>> > >
>> > >
>> > > --
>> > > Regards,
>> > > Navin
>> >
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Navin
>>
>>
>
>
> --
> Regards,
> Navin
>
--
Regards,
Navin
Re: Topology gets submitted to Storm and then nothing happens
Posted by Navin Ipe <na...@searchlighthealth.com>.
Anyone with Gradle experience, kindly help. I've tried for the past 40
minutes with help from posts like this
<http://stackoverflow.com/questions/17421726/making-the-storm-jars-compile-time-only-in-a-gradle-project>,
this
<http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/#build-a-correct-standalone--fat-jar-file-of-my-storm-code>,
this
<http://kennethjorgensen.com/blog/2014/fat-jars-with-excluded-dependencies-in-gradle>
and this
<https://discuss.gradle.org/t/removing-dependencies-from-a-jar-file-during-jar-task/5521>,
but it isn't working.
This is all my original gradle script contains:
apply plugin: 'java'
apply plugin: 'eclipse'
defaultTasks 'jar'
jar {
from {
(configurations.runtime).collect {
it.isDirectory() ? it : zipTree(it)
}
}
manifest {
attributes 'Main-Class': 'com.slh.myprogram.MyProgram'
}
}
sourceCompatibility = '1.8'
[compileJava, compileTestJava]*.options*.encoding = 'UTF-8'
if (!hasProperty('mainClass')) {
ext.mainClass = 'com.slh.myprogram.MyProgram'
}
repositories {
mavenCentral()
}
dependencies {
compile 'org.apache.storm:storm-core:1.0.1'
testCompile group: 'junit', name: 'junit', version: '4.10'
}
On Mon, May 23, 2016 at 3:18 PM, Matthias J. Sax <mj...@apache.org> wrote:
> You are doing it basically correct. But you use the wrong jar file...
>
> The jar you submit must contain your spouts and bolt classes, but not
> the classes of Storm itself.
>
> I am not too familiar with Gradle, but you need to fix your build script
> to assemble your topology jar file correctly.
>
> It would help, if you post your gradle file here, for other to help you
> out with this.
>
> -Matthias
>
> On 05/23/2016 11:45 AM, Navin Ipe wrote:
> > Bingo. Supervisor.log showed:
> > /Worker Process 17d197c0-80c8-408d-ac70-11970a93d2b9 exited with code: 1
> > Worker Process 204c20f5-a6d2-4739-b010-c90e651cb2cf exited with code: 1
> > 204c20f5-a6d2-4739-b010-c90e651cb2cf still hasn't started
> > /
> > And the worker logs were located deep inside here
> >
> /Users/nav/programming/apache-storm-1.0.1/*logs/workers-artifacts*/MyTopologyName-6-1463986744/6701/worker.log.err
> >
> > *The error:*
> > /Caused by: java.io.IOException: Found multiple defaults.yaml resources.
> > You're probably bundling the Storm jars with your topology jar.
> >
> [jar:file:/Users/nav/programming/apache-storm-1.0.1/storm-local/supervisor/stormdist/MyTopologyName-6-1463986744/stormjar.jar!/defaults.yaml,
> >
> jar:file:/Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar!/defaults.yaml]
> > at
> > org.apache.storm.utils.Utils.getConfigFileInputStream(Utils.java:333)
> > at
> > org.apache.storm.utils.Utils.findAndReadConfigFile(Utils.java:287)
> > ... 8 more
> > Exception in thread "main" java.lang.ExceptionInInitializerError/
> >
> > But I don't understand. If I don't include the storm jars in the Gradle
> > dependency, then I won't be able to compile the program itself.
> >
> > *What I'm doing:*
> > *Using this version of Storm via Gradle in Netbeans.*
> > dependencies { compile 'org.apache.storm:storm-core:1.0.1'
> >
> > *and submitting topology like this:*
> > System.setProperty("storm.jar",
> > "/Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar");
> >
> > StormSubmitter.submitTopology(c.TOPO_NAME, config,
> > builder.createTopology());
> >
> > Is there some other way to do this?
> >
> >
> >
> > On Mon, May 23, 2016 at 1:36 PM, Matthias J. Sax <mjsax@apache.org
> > <ma...@apache.org>> wrote:
> >
> > How many workers did you configure (ie, storm.yaml parameter
> > "supervisor.slots.ports" -- the number of available slots is also
> shown
> > in the web UI)?
> >
> > If there are free workers available, look into logs/worker*.log -- if
> > worker*.log is not available, no worker was started. You might also
> have
> > a look into logs/supervisor.log and maybe logs/nimbus.log
> >
> >
> > -Matthias
> >
> > On 05/23/2016 08:55 AM, Navin Ipe wrote:
> > > I installed zookeeper on my local system. Started it. Extracted
> Storm
> > > 1.0.1 on my local system and *started it via this script:*
> > > /#!/bin/bash
> > > ./storm nimbus &
> > > ./storm supervisor &
> > > ./storm ui &/
> > >
> > > *storm.yaml has:*
> > > / storm.zookeeper.servers:
> > > - "localhost"
> > > nimbus.seeds: ["localhost"]/
> > >
> > > *and zoo.conf has:*
> > > /tickTime=2000
> > > initLimit=10
> > > syncLimit=5
> > > dataDir=/Users/nav/programming/zookeeper-3.4.8/data
> > > clientPort=2181/
> > > *
> > > *
> > > *Using this version of Storm via Gradle.*
> > > /dependencies { compile 'org.apache.storm:storm-core:1.0.1'
> > >
> > > /
> > > *Firewall on my system
> > > *
> > > Switched off*
> > > *
> > > //
> > >
> > > *and submitting topology like this:*
> > > /System.setProperty("storm.jar",
> > >
> "/Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar");
> > >
> > > StormSubmitter.submitTopology(c.TOPO_NAME, config,
> > > builder.createTopology());
> > > /
> > >
> > >
> > > But on running it in my program's build/lib folder with java -jar
> > > myProgram.jar, the topology gets submitted (and I can see the
> submitted
> > > topology on the storm UI via http://localhost:8080),*but after
> > > submitting, no emits happen. The program seems to freeze.*
> > > /BUILD SUCCESSFUL
> > >
> > > Total time: 12.78 secs
> > >
> > > This build could be faster, please consider using the Gradle
> Daemon:
> > > https://docs.gradle.org/2.10/userguide/gradle_daemon.html
> > > 569 [main] INFO c.s.c.s.Queries_ReaderWriter - Waiting few
> seconds
> > > until table is dropped...
> > > 589 [main] INFO c.s.c.s.Queries_ReaderWriter - New table created.
> > > 1007 [main] INFO o.a.s.StormSubmitter - Generated ZooKeeper secret
> > > payload for MD5-digest: -5943870921993217077:-9035830007806332208
> > > 1074 [main] INFO o.a.s.s.a.AuthUtils - Got AutoCreds []
> > > 1124 [main] INFO o.a.s.StormSubmitter - Uploading topology jar
> > > /Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar
> to
> > > assigned location:
> > >
> /Users/nav/programming/apache-storm-1.0.1/storm-local/nimbus/inbox/stormjar-036472f9-15ce-4f09-8663-b1500942e696.jar
> > > 1271 [main] INFO o.a.s.StormSubmitter - Successfully uploaded
> topology
> > > jar to assigned location:
> > >
> /Users/nav/programming/apache-storm-1.0.1/storm-local/nimbus/inbox/stormjar-036472f9-15ce-4f09-8663-b1500942e696.jar
> > > 1271 [main] INFO o.a.s.StormSubmitter - Submitting topology
> MyTopology
> > > in distributed mode with conf
> > >
> {"storm.zookeeper.topology.auth.scheme":"digest","topology.workers":2,"topology.debug":false,"topology.max.spout.pending":10,"storm.zookeeper.topology.auth.payload":"-5943870921993217077:-9035830007806332208","topology.transfer.buffer.size":32,"topology.executor.send.buffer.size":16384,"topology.executor.receive.buffer.size":16384}
> > > 1427 [main] INFO o.a.s.StormSubmitter - Finished submitting
> topology:
> > > MyTopology
> > > 1427 [main] INFO c.s.c.CrystalStorm - Topology submitted/
> > >
> > > *Two questions:*
> > > 1. What usually causes this kind of problem? I've seen the answers
> on
> > > StackOverflow, but they didn't help.
> > > 2. Which log files would provide the most useful info in such
> > situations?
> > >
> > >
> > > --
> > > Regards,
> > > Navin
> >
> >
> >
> >
> > --
> > Regards,
> > Navin
>
>
--
Regards,
Navin
Re: Topology gets submitted to Storm and then nothing happens
Posted by "Matthias J. Sax" <mj...@apache.org>.
You are doing it basically correct. But you use the wrong jar file...
The jar you submit must contain your spouts and bolt classes, but not
the classes of Storm itself.
I am not too familiar with Gradle, but you need to fix your build script
to assemble your topology jar file correctly.
It would help, if you post your gradle file here, for other to help you
out with this.
-Matthias
On 05/23/2016 11:45 AM, Navin Ipe wrote:
> Bingo. Supervisor.log showed:
> /Worker Process 17d197c0-80c8-408d-ac70-11970a93d2b9 exited with code: 1
> Worker Process 204c20f5-a6d2-4739-b010-c90e651cb2cf exited with code: 1
> 204c20f5-a6d2-4739-b010-c90e651cb2cf still hasn't started
> /
> And the worker logs were located deep inside here
> /Users/nav/programming/apache-storm-1.0.1/*logs/workers-artifacts*/MyTopologyName-6-1463986744/6701/worker.log.err
>
> *The error:*
> /Caused by: java.io.IOException: Found multiple defaults.yaml resources.
> You're probably bundling the Storm jars with your topology jar.
> [jar:file:/Users/nav/programming/apache-storm-1.0.1/storm-local/supervisor/stormdist/MyTopologyName-6-1463986744/stormjar.jar!/defaults.yaml,
> jar:file:/Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar!/defaults.yaml]
> at
> org.apache.storm.utils.Utils.getConfigFileInputStream(Utils.java:333)
> at
> org.apache.storm.utils.Utils.findAndReadConfigFile(Utils.java:287)
> ... 8 more
> Exception in thread "main" java.lang.ExceptionInInitializerError/
>
> But I don't understand. If I don't include the storm jars in the Gradle
> dependency, then I won't be able to compile the program itself.
>
> *What I'm doing:*
> *Using this version of Storm via Gradle in Netbeans.*
> dependencies { compile 'org.apache.storm:storm-core:1.0.1'
>
> *and submitting topology like this:*
> System.setProperty("storm.jar",
> "/Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar");
>
> StormSubmitter.submitTopology(c.TOPO_NAME, config,
> builder.createTopology());
>
> Is there some other way to do this?
>
>
>
> On Mon, May 23, 2016 at 1:36 PM, Matthias J. Sax <mjsax@apache.org
> <ma...@apache.org>> wrote:
>
> How many workers did you configure (ie, storm.yaml parameter
> "supervisor.slots.ports" -- the number of available slots is also shown
> in the web UI)?
>
> If there are free workers available, look into logs/worker*.log -- if
> worker*.log is not available, no worker was started. You might also have
> a look into logs/supervisor.log and maybe logs/nimbus.log
>
>
> -Matthias
>
> On 05/23/2016 08:55 AM, Navin Ipe wrote:
> > I installed zookeeper on my local system. Started it. Extracted Storm
> > 1.0.1 on my local system and *started it via this script:*
> > /#!/bin/bash
> > ./storm nimbus &
> > ./storm supervisor &
> > ./storm ui &/
> >
> > *storm.yaml has:*
> > / storm.zookeeper.servers:
> > - "localhost"
> > nimbus.seeds: ["localhost"]/
> >
> > *and zoo.conf has:*
> > /tickTime=2000
> > initLimit=10
> > syncLimit=5
> > dataDir=/Users/nav/programming/zookeeper-3.4.8/data
> > clientPort=2181/
> > *
> > *
> > *Using this version of Storm via Gradle.*
> > /dependencies { compile 'org.apache.storm:storm-core:1.0.1'
> >
> > /
> > *Firewall on my system
> > *
> > Switched off*
> > *
> > //
> >
> > *and submitting topology like this:*
> > /System.setProperty("storm.jar",
> > "/Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar");
> >
> > StormSubmitter.submitTopology(c.TOPO_NAME, config,
> > builder.createTopology());
> > /
> >
> >
> > But on running it in my program's build/lib folder with java -jar
> > myProgram.jar, the topology gets submitted (and I can see the submitted
> > topology on the storm UI via http://localhost:8080),*but after
> > submitting, no emits happen. The program seems to freeze.*
> > /BUILD SUCCESSFUL
> >
> > Total time: 12.78 secs
> >
> > This build could be faster, please consider using the Gradle Daemon:
> > https://docs.gradle.org/2.10/userguide/gradle_daemon.html
> > 569 [main] INFO c.s.c.s.Queries_ReaderWriter - Waiting few seconds
> > until table is dropped...
> > 589 [main] INFO c.s.c.s.Queries_ReaderWriter - New table created.
> > 1007 [main] INFO o.a.s.StormSubmitter - Generated ZooKeeper secret
> > payload for MD5-digest: -5943870921993217077:-9035830007806332208
> > 1074 [main] INFO o.a.s.s.a.AuthUtils - Got AutoCreds []
> > 1124 [main] INFO o.a.s.StormSubmitter - Uploading topology jar
> > /Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar to
> > assigned location:
> > /Users/nav/programming/apache-storm-1.0.1/storm-local/nimbus/inbox/stormjar-036472f9-15ce-4f09-8663-b1500942e696.jar
> > 1271 [main] INFO o.a.s.StormSubmitter - Successfully uploaded topology
> > jar to assigned location:
> > /Users/nav/programming/apache-storm-1.0.1/storm-local/nimbus/inbox/stormjar-036472f9-15ce-4f09-8663-b1500942e696.jar
> > 1271 [main] INFO o.a.s.StormSubmitter - Submitting topology MyTopology
> > in distributed mode with conf
> > {"storm.zookeeper.topology.auth.scheme":"digest","topology.workers":2,"topology.debug":false,"topology.max.spout.pending":10,"storm.zookeeper.topology.auth.payload":"-5943870921993217077:-9035830007806332208","topology.transfer.buffer.size":32,"topology.executor.send.buffer.size":16384,"topology.executor.receive.buffer.size":16384}
> > 1427 [main] INFO o.a.s.StormSubmitter - Finished submitting topology:
> > MyTopology
> > 1427 [main] INFO c.s.c.CrystalStorm - Topology submitted/
> >
> > *Two questions:*
> > 1. What usually causes this kind of problem? I've seen the answers on
> > StackOverflow, but they didn't help.
> > 2. Which log files would provide the most useful info in such
> situations?
> >
> >
> > --
> > Regards,
> > Navin
>
>
>
>
> --
> Regards,
> Navin
Re: Topology gets submitted to Storm and then nothing happens
Posted by Navin Ipe <na...@searchlighthealth.com>.
Bingo. Supervisor.log showed:
*Worker Process 17d197c0-80c8-408d-ac70-11970a93d2b9 exited with code:
1Worker Process 204c20f5-a6d2-4739-b010-c90e651cb2cf exited with code:
1204c20f5-a6d2-4739-b010-c90e651cb2cf still hasn't started*
And the worker logs were located deep inside here
/Users/nav/programming/apache-storm-1.0.1/*logs/workers-artifacts*
/MyTopologyName-6-1463986744/6701/worker.log.err
*The error:*
*Caused by: java.io.IOException: Found multiple defaults.yaml resources.
You're probably bundling the Storm jars with your topology jar.
[jar:file:/Users/nav/programming/apache-storm-1.0.1/storm-local/supervisor/stormdist/MyTopologyName-6-1463986744/stormjar.jar!/defaults.yaml,
jar:file:/Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar!/defaults.yaml]
at
org.apache.storm.utils.Utils.getConfigFileInputStream(Utils.java:333)
at
org.apache.storm.utils.Utils.findAndReadConfigFile(Utils.java:287)
... 8 moreException in thread "main" java.lang.ExceptionInInitializerError*
But I don't understand. If I don't include the storm jars in the Gradle
dependency, then I won't be able to compile the program itself.
*What I'm doing:*
*Using this version of Storm via Gradle in Netbeans.*
dependencies { compile 'org.apache.storm:storm-core:1.0.1'
*and submitting topology like this:*
System.setProperty("storm.jar",
"/Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar");
StormSubmitter.submitTopology(c.TOPO_NAME, config,
builder.createTopology());
Is there some other way to do this?
On Mon, May 23, 2016 at 1:36 PM, Matthias J. Sax <mj...@apache.org> wrote:
> How many workers did you configure (ie, storm.yaml parameter
> "supervisor.slots.ports" -- the number of available slots is also shown
> in the web UI)?
>
> If there are free workers available, look into logs/worker*.log -- if
> worker*.log is not available, no worker was started. You might also have
> a look into logs/supervisor.log and maybe logs/nimbus.log
>
>
> -Matthias
>
> On 05/23/2016 08:55 AM, Navin Ipe wrote:
> > I installed zookeeper on my local system. Started it. Extracted Storm
> > 1.0.1 on my local system and *started it via this script:*
> > /#!/bin/bash
> > ./storm nimbus &
> > ./storm supervisor &
> > ./storm ui &/
> >
> > *storm.yaml has:*
> > / storm.zookeeper.servers:
> > - "localhost"
> > nimbus.seeds: ["localhost"]/
> >
> > *and zoo.conf has:*
> > /tickTime=2000
> > initLimit=10
> > syncLimit=5
> > dataDir=/Users/nav/programming/zookeeper-3.4.8/data
> > clientPort=2181/
> > *
> > *
> > *Using this version of Storm via Gradle.*
> > /dependencies { compile 'org.apache.storm:storm-core:1.0.1'
> >
> > /
> > *Firewall on my system
> > *
> > Switched off*
> > *
> > //
> >
> > *and submitting topology like this:*
> > /System.setProperty("storm.jar",
> > "/Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar");
> >
> > StormSubmitter.submitTopology(c.TOPO_NAME, config,
> > builder.createTopology());
> > /
> >
> >
> > But on running it in my program's build/lib folder with java -jar
> > myProgram.jar, the topology gets submitted (and I can see the submitted
> > topology on the storm UI via http://localhost:8080),*but after
> > submitting, no emits happen. The program seems to freeze.*
> > /BUILD SUCCESSFUL
> >
> > Total time: 12.78 secs
> >
> > This build could be faster, please consider using the Gradle Daemon:
> > https://docs.gradle.org/2.10/userguide/gradle_daemon.html
> > 569 [main] INFO c.s.c.s.Queries_ReaderWriter - Waiting few seconds
> > until table is dropped...
> > 589 [main] INFO c.s.c.s.Queries_ReaderWriter - New table created.
> > 1007 [main] INFO o.a.s.StormSubmitter - Generated ZooKeeper secret
> > payload for MD5-digest: -5943870921993217077:-9035830007806332208
> > 1074 [main] INFO o.a.s.s.a.AuthUtils - Got AutoCreds []
> > 1124 [main] INFO o.a.s.StormSubmitter - Uploading topology jar
> > /Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar to
> > assigned location:
> >
> /Users/nav/programming/apache-storm-1.0.1/storm-local/nimbus/inbox/stormjar-036472f9-15ce-4f09-8663-b1500942e696.jar
> > 1271 [main] INFO o.a.s.StormSubmitter - Successfully uploaded topology
> > jar to assigned location:
> >
> /Users/nav/programming/apache-storm-1.0.1/storm-local/nimbus/inbox/stormjar-036472f9-15ce-4f09-8663-b1500942e696.jar
> > 1271 [main] INFO o.a.s.StormSubmitter - Submitting topology MyTopology
> > in distributed mode with conf
> >
> {"storm.zookeeper.topology.auth.scheme":"digest","topology.workers":2,"topology.debug":false,"topology.max.spout.pending":10,"storm.zookeeper.topology.auth.payload":"-5943870921993217077:-9035830007806332208","topology.transfer.buffer.size":32,"topology.executor.send.buffer.size":16384,"topology.executor.receive.buffer.size":16384}
> > 1427 [main] INFO o.a.s.StormSubmitter - Finished submitting topology:
> > MyTopology
> > 1427 [main] INFO c.s.c.CrystalStorm - Topology submitted/
> >
> > *Two questions:*
> > 1. What usually causes this kind of problem? I've seen the answers on
> > StackOverflow, but they didn't help.
> > 2. Which log files would provide the most useful info in such situations?
> >
> >
> > --
> > Regards,
> > Navin
>
>
--
Regards,
Navin
Re: Topology gets submitted to Storm and then nothing happens
Posted by "Matthias J. Sax" <mj...@apache.org>.
How many workers did you configure (ie, storm.yaml parameter
"supervisor.slots.ports" -- the number of available slots is also shown
in the web UI)?
If there are free workers available, look into logs/worker*.log -- if
worker*.log is not available, no worker was started. You might also have
a look into logs/supervisor.log and maybe logs/nimbus.log
-Matthias
On 05/23/2016 08:55 AM, Navin Ipe wrote:
> I installed zookeeper on my local system. Started it. Extracted Storm
> 1.0.1 on my local system and *started it via this script:*
> /#!/bin/bash
> ./storm nimbus &
> ./storm supervisor &
> ./storm ui &/
>
> *storm.yaml has:*
> / storm.zookeeper.servers:
> - "localhost"
> nimbus.seeds: ["localhost"]/
>
> *and zoo.conf has:*
> /tickTime=2000
> initLimit=10
> syncLimit=5
> dataDir=/Users/nav/programming/zookeeper-3.4.8/data
> clientPort=2181/
> *
> *
> *Using this version of Storm via Gradle.*
> /dependencies { compile 'org.apache.storm:storm-core:1.0.1'
>
> /
> *Firewall on my system
> *
> Switched off*
> *
> //
>
> *and submitting topology like this:*
> /System.setProperty("storm.jar",
> "/Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar");
>
> StormSubmitter.submitTopology(c.TOPO_NAME, config,
> builder.createTopology());
> /
>
>
> But on running it in my program's build/lib folder with java -jar
> myProgram.jar, the topology gets submitted (and I can see the submitted
> topology on the storm UI via http://localhost:8080),*but after
> submitting, no emits happen. The program seems to freeze.*
> /BUILD SUCCESSFUL
>
> Total time: 12.78 secs
>
> This build could be faster, please consider using the Gradle Daemon:
> https://docs.gradle.org/2.10/userguide/gradle_daemon.html
> 569 [main] INFO c.s.c.s.Queries_ReaderWriter - Waiting few seconds
> until table is dropped...
> 589 [main] INFO c.s.c.s.Queries_ReaderWriter - New table created.
> 1007 [main] INFO o.a.s.StormSubmitter - Generated ZooKeeper secret
> payload for MD5-digest: -5943870921993217077:-9035830007806332208
> 1074 [main] INFO o.a.s.s.a.AuthUtils - Got AutoCreds []
> 1124 [main] INFO o.a.s.StormSubmitter - Uploading topology jar
> /Users/nav/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar to
> assigned location:
> /Users/nav/programming/apache-storm-1.0.1/storm-local/nimbus/inbox/stormjar-036472f9-15ce-4f09-8663-b1500942e696.jar
> 1271 [main] INFO o.a.s.StormSubmitter - Successfully uploaded topology
> jar to assigned location:
> /Users/nav/programming/apache-storm-1.0.1/storm-local/nimbus/inbox/stormjar-036472f9-15ce-4f09-8663-b1500942e696.jar
> 1271 [main] INFO o.a.s.StormSubmitter - Submitting topology MyTopology
> in distributed mode with conf
> {"storm.zookeeper.topology.auth.scheme":"digest","topology.workers":2,"topology.debug":false,"topology.max.spout.pending":10,"storm.zookeeper.topology.auth.payload":"-5943870921993217077:-9035830007806332208","topology.transfer.buffer.size":32,"topology.executor.send.buffer.size":16384,"topology.executor.receive.buffer.size":16384}
> 1427 [main] INFO o.a.s.StormSubmitter - Finished submitting topology:
> MyTopology
> 1427 [main] INFO c.s.c.CrystalStorm - Topology submitted/
>
> *Two questions:*
> 1. What usually causes this kind of problem? I've seen the answers on
> StackOverflow, but they didn't help.
> 2. Which log files would provide the most useful info in such situations?
>
>
> --
> Regards,
> Navin