You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Sridhar G <Sr...@microland.com> on 2014/09/09 05:43:18 UTC
Running Topology on Remote Machine
Hi All,
We are stuck with running our topology on Remote Machine. We have be
able to run our topology on local machine which emits tuples. But when
we try to run it on Remote Machine it does not emit tuples. Whereas the
Formation of Topology is shown on STORM UI under TOPOLOGY SUMMARY.
TOPOLOGY STATS , SPOUTS and BOLTS are visible with ZERO emits.
The Code which we have used for Running Topology on REMOTE Machine is:
LIBRARIES IMPORTS ARE -
import backtype.storm.Config;
import backtype.storm.LocalCluster;
import backtype.storm.topology.TopologyBuilder;
import backtype.storm.utils.Utils;
import org.json.simple.JSONValue;
import backtype.storm.StormSubmitter;
import backtype.storm.generated.AlreadyAliveException;
import backtype.storm.generated.Nimbus.Client;
import backtype.storm.topology.TopologyBuilder;
import backtype.storm.utils.NimbusClient;
import java.util.Map;
------------------------------------------------------------------------
-------------------------------------------------
TOPOLOGY PROGRAM -
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("POCspout", new POCSpout(),3);
builder.setBolt("POCbolt", new
POCBolt(),3).shuffleGrouping("POCspout");
Config conf = new Config();
conf.put(Config.NIMBUS_HOST, "192.xxx.xx.x");
conf.setDebug(true);
conf.setNumWorkers(1);
conf.setMaxSpoutPending(5000);
conf.setNumAckers(1);
conf.setMaxTaskParallelism(10);
Map storm_conf = Utils.readStormConfig();
storm_conf.put("nimbus.host", "192.xxx.xx.x");
Client client =
NimbusClient.getConfiguredClient(storm_conf).getClient();
String inputJar =
"/root/SOIM_PaVan/target/SOIMTopology-1.0-SNAPSHOT.jar";
NimbusClient nimbus = new NimbusClient(storm_conf,
"192.xxx.xx.x",6627);
String uploadedJarLocation =
StormSubmitter.submitJar(storm_conf,inputJar);
try {
String jsonConf = JSONValue.toJSONString(storm_conf);
nimbus.getClient().submitTopology("TOPOLOGYPOC",uploadedJarLocation,json
Conf,builder.createTopology());
}
catch (AlreadyAliveException ae)
{
ae.printStackTrace();
}
For any Correction in code let us know.
Regards,
Sridhar
Believe in the impossible and remove the improbable
The information contained in this transmission may contain privileged and confidential information of Microland Limited, including information protected by privacy laws. It is intended only for the use of Microland Limited. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
Re: Running Topology on Remote Machine
Posted by Vikas Agarwal <vi...@infoobjects.com>.
Check the command just before the "still hasn't started" log in
supervisor.log file.
Run this command with storm user sudo su storm -c "<worker command>"
On Tue, Sep 9, 2014 at 1:11 PM, Spico Florin <sp...@gmail.com> wrote:
> Hello, Vikas!
> I'm interested in this problem too. How can you run the worker java
> command on console? Can you please give an example?
> Thanks.
> Regards,
> Florin
>
> On Tue, Sep 9, 2014 at 10:27 AM, Vikas Agarwal <vi...@infoobjects.com>
> wrote:
>
>> I stuck there for 2 weeks. :)
>>
>> Basically, I tried to debug every command that storm processes are trying
>> to run. You can manually run the worker java command on console to check
>> what error it is throwing. In our case there were some problems, few of
>> them are:
>>
>> 1) slf4j jar conflicts
>> 2) somehow wrong value of hostname in storm worker's configuration
>> 3) something wrong in bolt's prepare method or spout's open method (may
>> be waiting for some connection outside of storm)
>> 4) ports being blocked by firewall (i added few ports in Amazon EC2
>> security groups to make it runnable)
>>
>>
>> On Tue, Sep 9, 2014 at 12:48 PM, Pavan Jakati G <Pa...@microland.com>
>> wrote:
>>
>>> How did you fix it Vikas . Can you help us here . We are stuck here
>>> since a week . L
>>>
>>>
>>>
>>> *From:* Vikas Agarwal [mailto:vikas@infoobjects.com]
>>> *Sent:* 09 September 2014 12:39
>>> *To:* user@storm.incubator.apache.org
>>>
>>> *Subject:* Re: Running Topology on Remote Machine
>>>
>>>
>>>
>>> We faced similar errors.
>>>
>>>
>>>
>>> May be it can help
>>> http://mail-archives.apache.org/mod_mbox/incubator-storm-user/201408.mbox/browser
>>>
>>>
>>>
>>> On Tue, Sep 9, 2014 at 11:02 AM, Pavan Jakati G <Pa...@microland.com>
>>> wrote:
>>>
>>> Worker log
>>>
>>>
>>>
>>> 2014-09-08 07:19:09 b.s.d.executor [INFO] Prepared bolt __system:(-1)
>>>
>>> 2014-09-08 07:19:09 b.s.d.executor [INFO] Loading executor SOIMbolt:[1 1]
>>>
>>> 2014-09-08 07:19:09 b.s.d.executor [INFO] Loaded executor tasks
>>> SOIMbolt:[1 1]
>>>
>>> 2014-09-08 07:19:09 b.s.d.executor [INFO] Finished loading executor
>>> SOIMbolt:[1 1]
>>>
>>> 2014-09-08 07:19:09 b.s.d.executor [INFO] Preparing bolt SOIMbolt:(1)
>>>
>>> 2014-09-08 07:19:09 b.s.d.worker [INFO] Launching receive-thread for
>>> 51fb6b55-0522-437c-a18e-72212192db0f:6703
>>>
>>> 2014-09-08 07:19:09 b.s.d.executor [INFO] Prepared bolt SOIMbolt:(1)
>>>
>>> 2014-09-08 07:19:09 b.s.m.n.Server [INFO] Create Netty Server
>>> Netty-server-localhost-6703, buffer_size: 5242880, maxWorkers: 1
>>>
>>> 2014-09-08 07:19:09 b.s.m.loader [INFO] Starting receive-thread:
>>> [stormId: PaVan-1-1410160741, port: 6703, thread-id: 0 ]
>>>
>>> 2014-09-08 07:19:09 b.s.d.worker [INFO] Worker has topology config {"
>>> storm.id" "PaVan-1-1410160741", "dev.zookeeper.path"
>>> "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil,
>>> "topology.builtin.metrics.bucket.size.secs" 60,
>>> "topology.fall.back.on.java.serialization" true,
>>> "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000,
>>> "topology.skip.missing.kryo.registrations" false,
>>> "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m",
>>> "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true,
>>> "topology.trident.batch.emit.interval.millis" 500, "drpc.servers"
>>> ["192.168.10.7"], "storm.messaging.netty.flush.check.interval.ms" 10,
>>> "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m",
>>> "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib",
>>> "topology.executor.send.buffer.size" 1024, "storm.local.dir" "storm-local",
>>> "storm.messaging.netty.buffer_size" 5242880,
>>> "supervisor.worker.start.timeout.secs" 120,
>>> "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs"
>>> 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64,
>>> "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "192.168.10.7",
>>> "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181,
>>> "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size"
>>> 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root"
>>> "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000,
>>> "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1,
>>> "storm.zookeeper.servers" ["192.168.10.7"], "transactional.zookeeper.root"
>>> "/transactional", "topology.acker.executors" nil,
>>> "topology.kryo.decorators" (), "topology.name" "PaVan",
>>> "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil,
>>> "drpc.queue.size" 128, "worker.childopts" "-Xmx768m",
>>> "supervisor.heartbeat.frequency.secs" 5,
>>> "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772,
>>> "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m",
>>> "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3,
>>> "topology.tasks" nil, "storm.messaging.netty.max_retries" 30,
>>> "topology.spout.wait.strategy"
>>> "backtype.storm.spout.SleepSpoutWaitStrategy",
>>> "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil,
>>> "storm.zookeeper.retry.interval" 1000, "
>>> topology.sleep.spout.wait.strategy.time.ms" 1,
>>> "nimbus.topology.validator"
>>> "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports"
>>> [6700 6701 6702 6703], "topology.debug" false, "nimbus.task.launch.secs"
>>> 120, "nimbus.supervisor.timeout.secs" 60, "topology.kryo.register" nil,
>>> "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10,
>>> "topology.workers" 1, "supervisor.childopts" "-Xmx256m",
>>> "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05,
>>> "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer"
>>> "backtype.storm.serialization.types.ListDelegateSerializer",
>>> "topology.disruptor.wait.strategy"
>>> "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer"
>>> "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30,
>>> "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory"
>>> "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port"
>>> 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times"
>>> 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport"
>>> "backtype.storm.security.auth.SimpleTransportPlugin",
>>> "topology.state.synchronization.timeout.secs" 60,
>>> "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs"
>>> 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "
>>> logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms"
>>> 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false,
>>> "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode"
>>> "distributed", "topology.max.task.parallelism" nil,
>>> "storm.messaging.netty.transfer.batch.size" 262144}
>>>
>>> 2014-09-08 07:19:09 b.s.d.worker [INFO] Worker
>>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 for storm PaVan-1-1410160741 on
>>> 51fb6b55-0522-437c-a18e-72212192db0f:6703 has finished loading
>>>
>>>
>>>
>>> Supervisor Log:
>>>
>>>
>>>
>>> b/json-simple-1.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/core.incubator-0.1.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/asm-4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-io-2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/httpclient-4.3.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jline-2.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.logging-0.2.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/logback-classic-1.0.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-logging-1.1.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-core-1.1.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-devel-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/curator-client-2.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clj-stacktrace-0.2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clj-time-0.4.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-lang-2.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/compojure-1.1.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/joda-time-2.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/chill-java-0.3.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clout-1.0.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/kryo-2.21.jar:/usr/local/apache-storm-0.9.2-incubating/lib/snakeyaml-1.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/minlog-1.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubating.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jgrapht-core-0.9.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/slf4j-api-1.6.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/hiccup-0.3.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/netty-3.6.3.Final.jar:/usr/local/apache-storm-0.9.2-incubating/lib/curator-framework-2.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/guava-13.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-fileupload-1.2.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/servlet-api-2.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/reflectasm-1.07-shaded.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jetty-util-6.1.26.jar:/usr/local/apache-storm-0.9.2-incubating/lib/objenesis-1.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.cli-0.2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-codec-1.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/zookeeper-3.4.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clojure-1.5.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/netty-3.2.2.Final.jar:/usr/local/apache-storm-0.9.2-incubating/lib/math.numeric-tower-0.0.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/carbonite-1.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/disruptor-2.10.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-exec-1.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.macro-0.1.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jetty-6.1.26.jar:/usr/local/apache-storm-0.9.2-incubating/lib/httpcore-4.3.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/servlet-api-2.5-20081211.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-servlet-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/logback-core-1.0.6.jar:/usr/local/apache-storm-0.9.2-incubating/conf:storm-local/supervisor/stormdist/PaVan-1-1410160741/stormjar.jar'
>>> 'backtype.storm.daemon.worker' 'PaVan-1-1410160741'
>>> '51fb6b55-0522-437c-a18e-72212192db0f' '6703'
>>> '2c12526f-d588-4dcc-99b9-b26e8c1d1d70'
>>>
>>> 2014-09-08 07:19:02 b.s.d.supervisor [INFO]
>>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>>
>>> 2014-09-08 07:19:02 b.s.d.supervisor [INFO]
>>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>>
>>> 2014-09-08 07:19:03 b.s.d.supervisor [INFO]
>>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>>
>>> 2014-09-08 07:19:03 b.s.d.supervisor [INFO]
>>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>>
>>> 2014-09-08 07:19:04 b.s.d.supervisor [INFO]
>>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>>
>>> 2014-09-08 07:19:04 b.s.d.supervisor [INFO]
>>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>>
>>> 2014-09-08 07:19:05 b.s.d.supervisor [INFO]
>>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>>
>>> 2014-09-08 07:19:05 b.s.d.supervisor [INFO]
>>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>>
>>> 2014-09-08 07:19:06 b.s.d.supervisor [INFO]
>>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>>
>>> 2014-09-08 07:19:06 b.s.d.supervisor [INFO]
>>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>>
>>> 2014-09-08 07:19:07 b.s.d.supervisor [INFO]
>>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>>
>>> 2014-09-08 07:19:07 b.s.d.supervisor [INFO]
>>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>>
>>> 2014-09-08 07:19:08 b.s.d.supervisor [INFO]
>>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>>
>>> 2014-09-08 07:19:08 b.s.d.supervisor [INFO]
>>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>>
>>>
>>>
>>> *From:* Raghavendra Nandagopal [mailto:askaboutstorm@gmail.com]
>>> *Sent:* 09 September 2014 10:59
>>> *To:* user@storm.incubator.apache.org
>>> *Subject:* RE: Running Topology on Remote Machine
>>>
>>>
>>>
>>> Hi Sridhar,
>>> Can you check the worker logs and see if there are any exceptions?
>>>
>>> Thanks,
>>> Raghav
>>>
>>> On Sep 8, 2014 10:25 PM, "Sridhar G" <Sr...@microland.com> wrote:
>>>
>>> Hi Dhaval,
>>>
>>> We have tried with the below config but no luck . Anything else to add
>>> to it ?
>>>
>>> Regards
>>> Sridhar Gadi
>>>
>>>
>>> -----Original Message-----
>>> From: Dhaval Modi [mailto:dhavalmodi24@gmail.com
>>> <dh...@gmail.com>]
>>> Sent: Tue 9/9/2014 9:45 AM
>>> To: user
>>> Cc: Pavan Jakati G
>>> Subject: Re: Running Topology on Remote Machine
>>>
>>> Hi Sridhar,
>>>
>>> Did you try commenting " conf.setMaxTaskParallelism(10);" ?
>>>
>>> Regards,
>>> Dhaval Modi
>>> dhavalmodi24@gmail.com
>>>
>>> On 9 September 2014 09:13, Sridhar G <Sr...@microland.com> wrote:
>>>
>>> >
>>> >
>>> >
>>> >
>>> > Hi All,
>>> >
>>> >
>>> >
>>> > We are stuck with running our topology on Remote Machine. We have be
>>> able
>>> > to run our topology on local machine which emits tuples. But when we
>>> try to
>>> > run it on Remote Machine it does not emit tuples. Whereas the
>>> Formation of
>>> > Topology is shown on STORM UI under TOPOLOGY SUMMARY. TOPOLOGY STATS ,
>>> > SPOUTS and BOLTS are visible with ZERO emits.
>>> >
>>> >
>>> >
>>> > The Code which we have used for Running Topology on REMOTE Machine is:
>>> >
>>> >
>>> >
>>> > LIBRARIES IMPORTS ARE -
>>> >
>>> >
>>> >
>>> > import backtype.storm.Config;
>>> >
>>> > import backtype.storm.LocalCluster;
>>> >
>>> > import backtype.storm.topology.TopologyBuilder;
>>> >
>>> > import backtype.storm.utils.Utils;
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > import org.json.simple.JSONValue;
>>> >
>>> > import backtype.storm.StormSubmitter;
>>> >
>>> > import backtype.storm.generated.AlreadyAliveException;
>>> >
>>> > import backtype.storm.generated.Nimbus.Client;
>>> >
>>> > import backtype.storm.topology.TopologyBuilder;
>>> >
>>> > import backtype.storm.utils.NimbusClient;
>>> >
>>> > import java.util.Map;
>>> >
>>> >
>>> >
>>> -------------------------------------------------------------------------------------------------------------------------
>>> >
>>> > *TOPOLOGY PROGRAM -*
>>> >
>>> >
>>> >
>>> > TopologyBuilder builder = new TopologyBuilder();
>>> >
>>> > builder.setSpout("POCspout", new POCSpout(),3);
>>> >
>>> > builder.setBolt("POCbolt", new
>>> > POCBolt(),3).shuffleGrouping("POCspout");
>>> >
>>> >
>>> >
>>> > Config conf = new Config();
>>> >
>>> > conf.put(Config.NIMBUS_HOST, "192.xxx.xx.x");
>>> >
>>> > conf.setDebug(true);
>>> >
>>> > conf.setNumWorkers(1);
>>> >
>>> > conf.setMaxSpoutPending(5000);
>>> >
>>> > conf.setNumAckers(1);
>>> >
>>> > conf.setMaxTaskParallelism(10);
>>> >
>>> > Map storm_conf = Utils.readStormConfig();
>>> >
>>> >
>>> >
>>> > storm_conf.put("nimbus.host", "192.xxx.xx.x");
>>> >
>>> > Client client =
>>> > NimbusClient.getConfiguredClient(storm_conf).getClient();
>>> >
>>> >
>>> >
>>> > String inputJar =
>>> > "/root/SOIM_PaVan/target/SOIMTopology-1.0-SNAPSHOT.jar";
>>> >
>>> > NimbusClient nimbus = new NimbusClient(storm_conf,
>>> > "192.xxx.xx.x",6627);
>>> >
>>> > String uploadedJarLocation =
>>> > StormSubmitter.submitJar(storm_conf,inputJar);
>>> >
>>> >
>>> >
>>> > try {
>>> >
>>> > String jsonConf = JSONValue.toJSONString(storm_conf);
>>> >
>>> >
>>> >
>>> nimbus.getClient().submitTopology("TOPOLOGYPOC",uploadedJarLocation,jsonConf,builder.createTopology());
>>> >
>>> > }
>>> >
>>> > catch (AlreadyAliveException ae)
>>> >
>>> > {
>>> >
>>> > ae.printStackTrace();
>>> >
>>> > }
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > *For any Correction in code let us know.*
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > Regards,
>>> >
>>> > Sridhar
>>> >
>>> > *Believe in the impossible and remove the improbable*
>>> >
>>> > The information contained in this transmission may contain privileged
>>> and
>>> > confidential information of Microland Limited, including information
>>> > protected by privacy laws. It is intended only for the use of Microland
>>> > Limited. If you are not the intended recipient, you are hereby notified
>>> > that any review, dissemination, distribution, or duplication of this
>>> > communication is strictly prohibited. If you are not the intended
>>> > recipient, please contact the sender by reply email and destroy all
>>> copies
>>> > of the original message.
>>> >
>>>
>>> The information contained in this transmission may contain privileged
>>> and confidential information of Microland Limited, including information
>>> protected by privacy laws. It is intended only for the use of Microland
>>> Limited. If you are not the intended recipient, you are hereby notified
>>> that any review, dissemination, distribution, or duplication of this
>>> communication is strictly prohibited. If you are not the intended
>>> recipient, please contact the sender by reply email and destroy all copies
>>> of the original message.
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Vikas Agarwal
>>> 91 – 9928301411
>>>
>>> InfoObjects, Inc.
>>> Execution Matters
>>> http://www.infoobjects.com
>>> 2041 Mission College Boulevard, #280
>>> Santa Clara, CA 95054
>>> +1 (408) 988-2000 Work
>>> +1 (408) 716-2726 Fax
>>>
>>
>>
>>
>> --
>> Regards,
>> Vikas Agarwal
>> 91 – 9928301411
>>
>> InfoObjects, Inc.
>> Execution Matters
>> http://www.infoobjects.com
>> 2041 Mission College Boulevard, #280
>> Santa Clara, CA 95054
>> +1 (408) 988-2000 Work
>> +1 (408) 716-2726 Fax
>>
>>
>
--
Regards,
Vikas Agarwal
91 – 9928301411
InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
Re: Running Topology on Remote Machine
Posted by Spico Florin <sp...@gmail.com>.
Hello, Vikas!
I'm interested in this problem too. How can you run the worker java
command on console? Can you please give an example?
Thanks.
Regards,
Florin
On Tue, Sep 9, 2014 at 10:27 AM, Vikas Agarwal <vi...@infoobjects.com>
wrote:
> I stuck there for 2 weeks. :)
>
> Basically, I tried to debug every command that storm processes are trying
> to run. You can manually run the worker java command on console to check
> what error it is throwing. In our case there were some problems, few of
> them are:
>
> 1) slf4j jar conflicts
> 2) somehow wrong value of hostname in storm worker's configuration
> 3) something wrong in bolt's prepare method or spout's open method (may be
> waiting for some connection outside of storm)
> 4) ports being blocked by firewall (i added few ports in Amazon EC2
> security groups to make it runnable)
>
>
> On Tue, Sep 9, 2014 at 12:48 PM, Pavan Jakati G <Pa...@microland.com>
> wrote:
>
>> How did you fix it Vikas . Can you help us here . We are stuck here since
>> a week . L
>>
>>
>>
>> *From:* Vikas Agarwal [mailto:vikas@infoobjects.com]
>> *Sent:* 09 September 2014 12:39
>> *To:* user@storm.incubator.apache.org
>>
>> *Subject:* Re: Running Topology on Remote Machine
>>
>>
>>
>> We faced similar errors.
>>
>>
>>
>> May be it can help
>> http://mail-archives.apache.org/mod_mbox/incubator-storm-user/201408.mbox/browser
>>
>>
>>
>> On Tue, Sep 9, 2014 at 11:02 AM, Pavan Jakati G <Pa...@microland.com>
>> wrote:
>>
>> Worker log
>>
>>
>>
>> 2014-09-08 07:19:09 b.s.d.executor [INFO] Prepared bolt __system:(-1)
>>
>> 2014-09-08 07:19:09 b.s.d.executor [INFO] Loading executor SOIMbolt:[1 1]
>>
>> 2014-09-08 07:19:09 b.s.d.executor [INFO] Loaded executor tasks
>> SOIMbolt:[1 1]
>>
>> 2014-09-08 07:19:09 b.s.d.executor [INFO] Finished loading executor
>> SOIMbolt:[1 1]
>>
>> 2014-09-08 07:19:09 b.s.d.executor [INFO] Preparing bolt SOIMbolt:(1)
>>
>> 2014-09-08 07:19:09 b.s.d.worker [INFO] Launching receive-thread for
>> 51fb6b55-0522-437c-a18e-72212192db0f:6703
>>
>> 2014-09-08 07:19:09 b.s.d.executor [INFO] Prepared bolt SOIMbolt:(1)
>>
>> 2014-09-08 07:19:09 b.s.m.n.Server [INFO] Create Netty Server
>> Netty-server-localhost-6703, buffer_size: 5242880, maxWorkers: 1
>>
>> 2014-09-08 07:19:09 b.s.m.loader [INFO] Starting receive-thread:
>> [stormId: PaVan-1-1410160741, port: 6703, thread-id: 0 ]
>>
>> 2014-09-08 07:19:09 b.s.d.worker [INFO] Worker has topology config {"
>> storm.id" "PaVan-1-1410160741", "dev.zookeeper.path"
>> "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil,
>> "topology.builtin.metrics.bucket.size.secs" 60,
>> "topology.fall.back.on.java.serialization" true,
>> "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000,
>> "topology.skip.missing.kryo.registrations" false,
>> "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m",
>> "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true,
>> "topology.trident.batch.emit.interval.millis" 500, "drpc.servers"
>> ["192.168.10.7"], "storm.messaging.netty.flush.check.interval.ms" 10,
>> "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m",
>> "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib",
>> "topology.executor.send.buffer.size" 1024, "storm.local.dir" "storm-local",
>> "storm.messaging.netty.buffer_size" 5242880,
>> "supervisor.worker.start.timeout.secs" 120,
>> "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs"
>> 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64,
>> "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "192.168.10.7",
>> "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181,
>> "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size"
>> 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root"
>> "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000,
>> "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1,
>> "storm.zookeeper.servers" ["192.168.10.7"], "transactional.zookeeper.root"
>> "/transactional", "topology.acker.executors" nil,
>> "topology.kryo.decorators" (), "topology.name" "PaVan",
>> "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil,
>> "drpc.queue.size" 128, "worker.childopts" "-Xmx768m",
>> "supervisor.heartbeat.frequency.secs" 5,
>> "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772,
>> "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m",
>> "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3,
>> "topology.tasks" nil, "storm.messaging.netty.max_retries" 30,
>> "topology.spout.wait.strategy"
>> "backtype.storm.spout.SleepSpoutWaitStrategy",
>> "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil,
>> "storm.zookeeper.retry.interval" 1000, "
>> topology.sleep.spout.wait.strategy.time.ms" 1,
>> "nimbus.topology.validator"
>> "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports"
>> [6700 6701 6702 6703], "topology.debug" false, "nimbus.task.launch.secs"
>> 120, "nimbus.supervisor.timeout.secs" 60, "topology.kryo.register" nil,
>> "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10,
>> "topology.workers" 1, "supervisor.childopts" "-Xmx256m",
>> "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05,
>> "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer"
>> "backtype.storm.serialization.types.ListDelegateSerializer",
>> "topology.disruptor.wait.strategy"
>> "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer"
>> "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30,
>> "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory"
>> "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port"
>> 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times"
>> 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport"
>> "backtype.storm.security.auth.SimpleTransportPlugin",
>> "topology.state.synchronization.timeout.secs" 60,
>> "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs"
>> 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "
>> logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000,
>> "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port"
>> 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "distributed",
>> "topology.max.task.parallelism" nil,
>> "storm.messaging.netty.transfer.batch.size" 262144}
>>
>> 2014-09-08 07:19:09 b.s.d.worker [INFO] Worker
>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 for storm PaVan-1-1410160741 on
>> 51fb6b55-0522-437c-a18e-72212192db0f:6703 has finished loading
>>
>>
>>
>> Supervisor Log:
>>
>>
>>
>> b/json-simple-1.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/core.incubator-0.1.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/asm-4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-io-2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/httpclient-4.3.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jline-2.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.logging-0.2.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/logback-classic-1.0.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-logging-1.1.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-core-1.1.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-devel-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/curator-client-2.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clj-stacktrace-0.2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clj-time-0.4.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-lang-2.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/compojure-1.1.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/joda-time-2.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/chill-java-0.3.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clout-1.0.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/kryo-2.21.jar:/usr/local/apache-storm-0.9.2-incubating/lib/snakeyaml-1.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/minlog-1.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubating.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jgrapht-core-0.9.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/slf4j-api-1.6.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/hiccup-0.3.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/netty-3.6.3.Final.jar:/usr/local/apache-storm-0.9.2-incubating/lib/curator-framework-2.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/guava-13.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-fileupload-1.2.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/servlet-api-2.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/reflectasm-1.07-shaded.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jetty-util-6.1.26.jar:/usr/local/apache-storm-0.9.2-incubating/lib/objenesis-1.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.cli-0.2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-codec-1.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/zookeeper-3.4.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clojure-1.5.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/netty-3.2.2.Final.jar:/usr/local/apache-storm-0.9.2-incubating/lib/math.numeric-tower-0.0.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/carbonite-1.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/disruptor-2.10.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-exec-1.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.macro-0.1.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jetty-6.1.26.jar:/usr/local/apache-storm-0.9.2-incubating/lib/httpcore-4.3.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/servlet-api-2.5-20081211.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-servlet-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/logback-core-1.0.6.jar:/usr/local/apache-storm-0.9.2-incubating/conf:storm-local/supervisor/stormdist/PaVan-1-1410160741/stormjar.jar'
>> 'backtype.storm.daemon.worker' 'PaVan-1-1410160741'
>> '51fb6b55-0522-437c-a18e-72212192db0f' '6703'
>> '2c12526f-d588-4dcc-99b9-b26e8c1d1d70'
>>
>> 2014-09-08 07:19:02 b.s.d.supervisor [INFO]
>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>
>> 2014-09-08 07:19:02 b.s.d.supervisor [INFO]
>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>
>> 2014-09-08 07:19:03 b.s.d.supervisor [INFO]
>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>
>> 2014-09-08 07:19:03 b.s.d.supervisor [INFO]
>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>
>> 2014-09-08 07:19:04 b.s.d.supervisor [INFO]
>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>
>> 2014-09-08 07:19:04 b.s.d.supervisor [INFO]
>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>
>> 2014-09-08 07:19:05 b.s.d.supervisor [INFO]
>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>
>> 2014-09-08 07:19:05 b.s.d.supervisor [INFO]
>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>
>> 2014-09-08 07:19:06 b.s.d.supervisor [INFO]
>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>
>> 2014-09-08 07:19:06 b.s.d.supervisor [INFO]
>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>
>> 2014-09-08 07:19:07 b.s.d.supervisor [INFO]
>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>
>> 2014-09-08 07:19:07 b.s.d.supervisor [INFO]
>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>
>> 2014-09-08 07:19:08 b.s.d.supervisor [INFO]
>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>
>> 2014-09-08 07:19:08 b.s.d.supervisor [INFO]
>> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>>
>>
>>
>> *From:* Raghavendra Nandagopal [mailto:askaboutstorm@gmail.com]
>> *Sent:* 09 September 2014 10:59
>> *To:* user@storm.incubator.apache.org
>> *Subject:* RE: Running Topology on Remote Machine
>>
>>
>>
>> Hi Sridhar,
>> Can you check the worker logs and see if there are any exceptions?
>>
>> Thanks,
>> Raghav
>>
>> On Sep 8, 2014 10:25 PM, "Sridhar G" <Sr...@microland.com> wrote:
>>
>> Hi Dhaval,
>>
>> We have tried with the below config but no luck . Anything else to add to
>> it ?
>>
>> Regards
>> Sridhar Gadi
>>
>>
>> -----Original Message-----
>> From: Dhaval Modi [mailto:dhavalmodi24@gmail.com <dh...@gmail.com>
>> ]
>> Sent: Tue 9/9/2014 9:45 AM
>> To: user
>> Cc: Pavan Jakati G
>> Subject: Re: Running Topology on Remote Machine
>>
>> Hi Sridhar,
>>
>> Did you try commenting " conf.setMaxTaskParallelism(10);" ?
>>
>> Regards,
>> Dhaval Modi
>> dhavalmodi24@gmail.com
>>
>> On 9 September 2014 09:13, Sridhar G <Sr...@microland.com> wrote:
>>
>> >
>> >
>> >
>> >
>> > Hi All,
>> >
>> >
>> >
>> > We are stuck with running our topology on Remote Machine. We have be
>> able
>> > to run our topology on local machine which emits tuples. But when we
>> try to
>> > run it on Remote Machine it does not emit tuples. Whereas the Formation
>> of
>> > Topology is shown on STORM UI under TOPOLOGY SUMMARY. TOPOLOGY STATS ,
>> > SPOUTS and BOLTS are visible with ZERO emits.
>> >
>> >
>> >
>> > The Code which we have used for Running Topology on REMOTE Machine is:
>> >
>> >
>> >
>> > LIBRARIES IMPORTS ARE -
>> >
>> >
>> >
>> > import backtype.storm.Config;
>> >
>> > import backtype.storm.LocalCluster;
>> >
>> > import backtype.storm.topology.TopologyBuilder;
>> >
>> > import backtype.storm.utils.Utils;
>> >
>> >
>> >
>> >
>> >
>> > import org.json.simple.JSONValue;
>> >
>> > import backtype.storm.StormSubmitter;
>> >
>> > import backtype.storm.generated.AlreadyAliveException;
>> >
>> > import backtype.storm.generated.Nimbus.Client;
>> >
>> > import backtype.storm.topology.TopologyBuilder;
>> >
>> > import backtype.storm.utils.NimbusClient;
>> >
>> > import java.util.Map;
>> >
>> >
>> >
>> -------------------------------------------------------------------------------------------------------------------------
>> >
>> > *TOPOLOGY PROGRAM -*
>> >
>> >
>> >
>> > TopologyBuilder builder = new TopologyBuilder();
>> >
>> > builder.setSpout("POCspout", new POCSpout(),3);
>> >
>> > builder.setBolt("POCbolt", new
>> > POCBolt(),3).shuffleGrouping("POCspout");
>> >
>> >
>> >
>> > Config conf = new Config();
>> >
>> > conf.put(Config.NIMBUS_HOST, "192.xxx.xx.x");
>> >
>> > conf.setDebug(true);
>> >
>> > conf.setNumWorkers(1);
>> >
>> > conf.setMaxSpoutPending(5000);
>> >
>> > conf.setNumAckers(1);
>> >
>> > conf.setMaxTaskParallelism(10);
>> >
>> > Map storm_conf = Utils.readStormConfig();
>> >
>> >
>> >
>> > storm_conf.put("nimbus.host", "192.xxx.xx.x");
>> >
>> > Client client =
>> > NimbusClient.getConfiguredClient(storm_conf).getClient();
>> >
>> >
>> >
>> > String inputJar =
>> > "/root/SOIM_PaVan/target/SOIMTopology-1.0-SNAPSHOT.jar";
>> >
>> > NimbusClient nimbus = new NimbusClient(storm_conf,
>> > "192.xxx.xx.x",6627);
>> >
>> > String uploadedJarLocation =
>> > StormSubmitter.submitJar(storm_conf,inputJar);
>> >
>> >
>> >
>> > try {
>> >
>> > String jsonConf = JSONValue.toJSONString(storm_conf);
>> >
>> >
>> >
>> nimbus.getClient().submitTopology("TOPOLOGYPOC",uploadedJarLocation,jsonConf,builder.createTopology());
>> >
>> > }
>> >
>> > catch (AlreadyAliveException ae)
>> >
>> > {
>> >
>> > ae.printStackTrace();
>> >
>> > }
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > *For any Correction in code let us know.*
>> >
>> >
>> >
>> >
>> >
>> > Regards,
>> >
>> > Sridhar
>> >
>> > *Believe in the impossible and remove the improbable*
>> >
>> > The information contained in this transmission may contain privileged
>> and
>> > confidential information of Microland Limited, including information
>> > protected by privacy laws. It is intended only for the use of Microland
>> > Limited. If you are not the intended recipient, you are hereby notified
>> > that any review, dissemination, distribution, or duplication of this
>> > communication is strictly prohibited. If you are not the intended
>> > recipient, please contact the sender by reply email and destroy all
>> copies
>> > of the original message.
>> >
>>
>> The information contained in this transmission may contain privileged and
>> confidential information of Microland Limited, including information
>> protected by privacy laws. It is intended only for the use of Microland
>> Limited. If you are not the intended recipient, you are hereby notified
>> that any review, dissemination, distribution, or duplication of this
>> communication is strictly prohibited. If you are not the intended
>> recipient, please contact the sender by reply email and destroy all copies
>> of the original message.
>>
>>
>>
>>
>>
>> --
>> Regards,
>> Vikas Agarwal
>> 91 – 9928301411
>>
>> InfoObjects, Inc.
>> Execution Matters
>> http://www.infoobjects.com
>> 2041 Mission College Boulevard, #280
>> Santa Clara, CA 95054
>> +1 (408) 988-2000 Work
>> +1 (408) 716-2726 Fax
>>
>
>
>
> --
> Regards,
> Vikas Agarwal
> 91 – 9928301411
>
> InfoObjects, Inc.
> Execution Matters
> http://www.infoobjects.com
> 2041 Mission College Boulevard, #280
> Santa Clara, CA 95054
> +1 (408) 988-2000 Work
> +1 (408) 716-2726 Fax
>
>
Re: Running Topology on Remote Machine
Posted by Vikas Agarwal <vi...@infoobjects.com>.
No, we too tested things on single machine. I need to open ports for
external services like databases etc. We had external mongo, solr and some
other processes which needed to be opened in firewall otherwise topologies
would fail to start.
On Tue, Sep 9, 2014 at 1:06 PM, Pavan Jakati G <Pa...@microland.com>
wrote:
> We are running in single server . We have everything in same server .
> Would remote work only if we have multiple machines in place ?
>
>
>
> *From:* Vikas Agarwal [mailto:vikas@infoobjects.com]
> *Sent:* 09 September 2014 12:57
>
> *To:* user@storm.incubator.apache.org
> *Subject:* Re: Running Topology on Remote Machine
>
>
>
> I stuck there for 2 weeks. :)
>
>
>
> Basically, I tried to debug every command that storm processes are trying
> to run. You can manually run the worker java command on console to check
> what error it is throwing. In our case there were some problems, few of
> them are:
>
>
>
> 1) slf4j jar conflicts
>
> 2) somehow wrong value of hostname in storm worker's configuration
>
> 3) something wrong in bolt's prepare method or spout's open method (may be
> waiting for some connection outside of storm)
>
> 4) ports being blocked by firewall (i added few ports in Amazon EC2
> security groups to make it runnable)
>
>
>
> On Tue, Sep 9, 2014 at 12:48 PM, Pavan Jakati G <Pa...@microland.com>
> wrote:
>
> How did you fix it Vikas . Can you help us here . We are stuck here since
> a week . L
>
>
>
> *From:* Vikas Agarwal [mailto:vikas@infoobjects.com]
> *Sent:* 09 September 2014 12:39
> *To:* user@storm.incubator.apache.org
>
>
> *Subject:* Re: Running Topology on Remote Machine
>
>
>
> We faced similar errors.
>
>
>
> May be it can help
> http://mail-archives.apache.org/mod_mbox/incubator-storm-user/201408.mbox/browser
>
>
>
> On Tue, Sep 9, 2014 at 11:02 AM, Pavan Jakati G <Pa...@microland.com>
> wrote:
>
> Worker log
>
>
>
> 2014-09-08 07:19:09 b.s.d.executor [INFO] Prepared bolt __system:(-1)
>
> 2014-09-08 07:19:09 b.s.d.executor [INFO] Loading executor SOIMbolt:[1 1]
>
> 2014-09-08 07:19:09 b.s.d.executor [INFO] Loaded executor tasks
> SOIMbolt:[1 1]
>
> 2014-09-08 07:19:09 b.s.d.executor [INFO] Finished loading executor
> SOIMbolt:[1 1]
>
> 2014-09-08 07:19:09 b.s.d.executor [INFO] Preparing bolt SOIMbolt:(1)
>
> 2014-09-08 07:19:09 b.s.d.worker [INFO] Launching receive-thread for
> 51fb6b55-0522-437c-a18e-72212192db0f:6703
>
> 2014-09-08 07:19:09 b.s.d.executor [INFO] Prepared bolt SOIMbolt:(1)
>
> 2014-09-08 07:19:09 b.s.m.n.Server [INFO] Create Netty Server
> Netty-server-localhost-6703, buffer_size: 5242880, maxWorkers: 1
>
> 2014-09-08 07:19:09 b.s.m.loader [INFO] Starting receive-thread: [stormId:
> PaVan-1-1410160741, port: 6703, thread-id: 0 ]
>
> 2014-09-08 07:19:09 b.s.d.worker [INFO] Worker has topology config {"
> storm.id" "PaVan-1-1410160741", "dev.zookeeper.path"
> "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil,
> "topology.builtin.metrics.bucket.size.secs" 60,
> "topology.fall.back.on.java.serialization" true,
> "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000,
> "topology.skip.missing.kryo.registrations" false,
> "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m",
> "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true,
> "topology.trident.batch.emit.interval.millis" 500, "drpc.servers"
> ["192.168.10.7"], "storm.messaging.netty.flush.check.interval.ms" 10,
> "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m",
> "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib",
> "topology.executor.send.buffer.size" 1024, "storm.local.dir" "storm-local",
> "storm.messaging.netty.buffer_size" 5242880,
> "supervisor.worker.start.timeout.secs" 120,
> "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs"
> 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64,
> "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "192.168.10.7",
> "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181,
> "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size"
> 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root"
> "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000,
> "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1,
> "storm.zookeeper.servers" ["192.168.10.7"], "transactional.zookeeper.root"
> "/transactional", "topology.acker.executors" nil,
> "topology.kryo.decorators" (), "topology.name" "PaVan",
> "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil,
> "drpc.queue.size" 128, "worker.childopts" "-Xmx768m",
> "supervisor.heartbeat.frequency.secs" 5,
> "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772,
> "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m",
> "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3,
> "topology.tasks" nil, "storm.messaging.netty.max_retries" 30,
> "topology.spout.wait.strategy"
> "backtype.storm.spout.SleepSpoutWaitStrategy",
> "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil,
> "storm.zookeeper.retry.interval" 1000, "
> topology.sleep.spout.wait.strategy.time.ms" 1,
> "nimbus.topology.validator"
> "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports"
> [6700 6701 6702 6703], "topology.debug" false, "nimbus.task.launch.secs"
> 120, "nimbus.supervisor.timeout.secs" 60, "topology.kryo.register" nil,
> "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10,
> "topology.workers" 1, "supervisor.childopts" "-Xmx256m",
> "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05,
> "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer"
> "backtype.storm.serialization.types.ListDelegateSerializer",
> "topology.disruptor.wait.strategy"
> "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer"
> "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30,
> "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory"
> "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port"
> 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times"
> 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport"
> "backtype.storm.security.auth.SimpleTransportPlugin",
> "topology.state.synchronization.timeout.secs" 60,
> "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs"
> 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "
> logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000,
> "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port"
> 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "distributed",
> "topology.max.task.parallelism" nil,
> "storm.messaging.netty.transfer.batch.size" 262144}
>
> 2014-09-08 07:19:09 b.s.d.worker [INFO] Worker
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 for storm PaVan-1-1410160741 on
> 51fb6b55-0522-437c-a18e-72212192db0f:6703 has finished loading
>
>
>
> Supervisor Log:
>
>
>
> b/json-simple-1.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/core.incubator-0.1.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/asm-4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-io-2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/httpclient-4.3.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jline-2.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.logging-0.2.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/logback-classic-1.0.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-logging-1.1.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-core-1.1.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-devel-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/curator-client-2.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clj-stacktrace-0.2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clj-time-0.4.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-lang-2.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/compojure-1.1.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/joda-time-2.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/chill-java-0.3.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clout-1.0.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/kryo-2.21.jar:/usr/local/apache-storm-0.9.2-incubating/lib/snakeyaml-1.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/minlog-1.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubating.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jgrapht-core-0.9.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/slf4j-api-1.6.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/hiccup-0.3.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/netty-3.6.3.Final.jar:/usr/local/apache-storm-0.9.2-incubating/lib/curator-framework-2.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/guava-13.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-fileupload-1.2.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/servlet-api-2.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/reflectasm-1.07-shaded.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jetty-util-6.1.26.jar:/usr/local/apache-storm-0.9.2-incubating/lib/objenesis-1.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.cli-0.2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-codec-1.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/zookeeper-3.4.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clojure-1.5.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/netty-3.2.2.Final.jar:/usr/local/apache-storm-0.9.2-incubating/lib/math.numeric-tower-0.0.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/carbonite-1.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/disruptor-2.10.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-exec-1.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.macro-0.1.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jetty-6.1.26.jar:/usr/local/apache-storm-0.9.2-incubating/lib/httpcore-4.3.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/servlet-api-2.5-20081211.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-servlet-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/logback-core-1.0.6.jar:/usr/local/apache-storm-0.9.2-incubating/conf:storm-local/supervisor/stormdist/PaVan-1-1410160741/stormjar.jar'
> 'backtype.storm.daemon.worker' 'PaVan-1-1410160741'
> '51fb6b55-0522-437c-a18e-72212192db0f' '6703'
> '2c12526f-d588-4dcc-99b9-b26e8c1d1d70'
>
> 2014-09-08 07:19:02 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:02 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:03 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:03 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:04 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:04 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:05 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:05 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:06 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:06 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:07 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:07 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:08 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:08 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
>
>
> *From:* Raghavendra Nandagopal [mailto:askaboutstorm@gmail.com]
> *Sent:* 09 September 2014 10:59
> *To:* user@storm.incubator.apache.org
> *Subject:* RE: Running Topology on Remote Machine
>
>
>
> Hi Sridhar,
> Can you check the worker logs and see if there are any exceptions?
>
> Thanks,
> Raghav
>
> On Sep 8, 2014 10:25 PM, "Sridhar G" <Sr...@microland.com> wrote:
>
> Hi Dhaval,
>
> We have tried with the below config but no luck . Anything else to add to
> it ?
>
> Regards
> Sridhar Gadi
>
>
> -----Original Message-----
> From: Dhaval Modi [mailto:dhavalmodi24@gmail.com <dh...@gmail.com>]
> Sent: Tue 9/9/2014 9:45 AM
> To: user
> Cc: Pavan Jakati G
> Subject: Re: Running Topology on Remote Machine
>
> Hi Sridhar,
>
> Did you try commenting " conf.setMaxTaskParallelism(10);" ?
>
> Regards,
> Dhaval Modi
> dhavalmodi24@gmail.com
>
> On 9 September 2014 09:13, Sridhar G <Sr...@microland.com> wrote:
>
> >
> >
> >
> >
> > Hi All,
> >
> >
> >
> > We are stuck with running our topology on Remote Machine. We have be able
> > to run our topology on local machine which emits tuples. But when we try
> to
> > run it on Remote Machine it does not emit tuples. Whereas the Formation
> of
> > Topology is shown on STORM UI under TOPOLOGY SUMMARY. TOPOLOGY STATS ,
> > SPOUTS and BOLTS are visible with ZERO emits.
> >
> >
> >
> > The Code which we have used for Running Topology on REMOTE Machine is:
> >
> >
> >
> > LIBRARIES IMPORTS ARE -
> >
> >
> >
> > import backtype.storm.Config;
> >
> > import backtype.storm.LocalCluster;
> >
> > import backtype.storm.topology.TopologyBuilder;
> >
> > import backtype.storm.utils.Utils;
> >
> >
> >
> >
> >
> > import org.json.simple.JSONValue;
> >
> > import backtype.storm.StormSubmitter;
> >
> > import backtype.storm.generated.AlreadyAliveException;
> >
> > import backtype.storm.generated.Nimbus.Client;
> >
> > import backtype.storm.topology.TopologyBuilder;
> >
> > import backtype.storm.utils.NimbusClient;
> >
> > import java.util.Map;
> >
> >
> >
> -------------------------------------------------------------------------------------------------------------------------
> >
> > *TOPOLOGY PROGRAM -*
> >
> >
> >
> > TopologyBuilder builder = new TopologyBuilder();
> >
> > builder.setSpout("POCspout", new POCSpout(),3);
> >
> > builder.setBolt("POCbolt", new
> > POCBolt(),3).shuffleGrouping("POCspout");
> >
> >
> >
> > Config conf = new Config();
> >
> > conf.put(Config.NIMBUS_HOST, "192.xxx.xx.x");
> >
> > conf.setDebug(true);
> >
> > conf.setNumWorkers(1);
> >
> > conf.setMaxSpoutPending(5000);
> >
> > conf.setNumAckers(1);
> >
> > conf.setMaxTaskParallelism(10);
> >
> > Map storm_conf = Utils.readStormConfig();
> >
> >
> >
> > storm_conf.put("nimbus.host", "192.xxx.xx.x");
> >
> > Client client =
> > NimbusClient.getConfiguredClient(storm_conf).getClient();
> >
> >
> >
> > String inputJar =
> > "/root/SOIM_PaVan/target/SOIMTopology-1.0-SNAPSHOT.jar";
> >
> > NimbusClient nimbus = new NimbusClient(storm_conf,
> > "192.xxx.xx.x",6627);
> >
> > String uploadedJarLocation =
> > StormSubmitter.submitJar(storm_conf,inputJar);
> >
> >
> >
> > try {
> >
> > String jsonConf = JSONValue.toJSONString(storm_conf);
> >
> >
> >
> nimbus.getClient().submitTopology("TOPOLOGYPOC",uploadedJarLocation,jsonConf,builder.createTopology());
> >
> > }
> >
> > catch (AlreadyAliveException ae)
> >
> > {
> >
> > ae.printStackTrace();
> >
> > }
> >
> >
> >
> >
> >
> >
> >
> > *For any Correction in code let us know.*
> >
> >
> >
> >
> >
> > Regards,
> >
> > Sridhar
> >
> > *Believe in the impossible and remove the improbable*
> >
> > The information contained in this transmission may contain privileged and
> > confidential information of Microland Limited, including information
> > protected by privacy laws. It is intended only for the use of Microland
> > Limited. If you are not the intended recipient, you are hereby notified
> > that any review, dissemination, distribution, or duplication of this
> > communication is strictly prohibited. If you are not the intended
> > recipient, please contact the sender by reply email and destroy all
> copies
> > of the original message.
> >
>
> The information contained in this transmission may contain privileged and
> confidential information of Microland Limited, including information
> protected by privacy laws. It is intended only for the use of Microland
> Limited. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution, or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
>
>
>
>
>
> --
> Regards,
> Vikas Agarwal
> 91 – 9928301411
>
> InfoObjects, Inc.
> Execution Matters
> http://www.infoobjects.com
> 2041 Mission College Boulevard, #280
> Santa Clara, CA 95054
> +1 (408) 988-2000 Work
> +1 (408) 716-2726 Fax
>
>
>
>
>
> --
> Regards,
> Vikas Agarwal
> 91 – 9928301411
>
> InfoObjects, Inc.
> Execution Matters
> http://www.infoobjects.com
> 2041 Mission College Boulevard, #280
> Santa Clara, CA 95054
> +1 (408) 988-2000 Work
> +1 (408) 716-2726 Fax
>
--
Regards,
Vikas Agarwal
91 – 9928301411
InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
RE: Running Topology on Remote Machine
Posted by Pavan Jakati G <Pa...@microland.com>.
We are running in single server . We have everything in same server . Would remote work only if we have multiple machines in place ?
From: Vikas Agarwal [mailto:vikas@infoobjects.com]
Sent: 09 September 2014 12:57
To: user@storm.incubator.apache.org
Subject: Re: Running Topology on Remote Machine
I stuck there for 2 weeks. :)
Basically, I tried to debug every command that storm processes are trying to run. You can manually run the worker java command on console to check what error it is throwing. In our case there were some problems, few of them are:
1) slf4j jar conflicts
2) somehow wrong value of hostname in storm worker's configuration
3) something wrong in bolt's prepare method or spout's open method (may be waiting for some connection outside of storm)
4) ports being blocked by firewall (i added few ports in Amazon EC2 security groups to make it runnable)
On Tue, Sep 9, 2014 at 12:48 PM, Pavan Jakati G <Pa...@microland.com> wrote:
How did you fix it Vikas . Can you help us here . We are stuck here since a week . L
From: Vikas Agarwal [mailto:vikas@infoobjects.com]
Sent: 09 September 2014 12:39
To: user@storm.incubator.apache.org
Subject: Re: Running Topology on Remote Machine
We faced similar errors.
May be it can help http://mail-archives.apache.org/mod_mbox/incubator-storm-user/201408.mbox/browser
On Tue, Sep 9, 2014 at 11:02 AM, Pavan Jakati G <Pa...@microland.com> wrote:
Worker log
2014-09-08 07:19:09 b.s.d.executor [INFO] Prepared bolt __system:(-1)
2014-09-08 07:19:09 b.s.d.executor [INFO] Loading executor SOIMbolt:[1 1]
2014-09-08 07:19:09 b.s.d.executor [INFO] Loaded executor tasks SOIMbolt:[1 1]
2014-09-08 07:19:09 b.s.d.executor [INFO] Finished loading executor SOIMbolt:[1 1]
2014-09-08 07:19:09 b.s.d.executor [INFO] Preparing bolt SOIMbolt:(1)
2014-09-08 07:19:09 b.s.d.worker [INFO] Launching receive-thread for 51fb6b55-0522-437c-a18e-72212192db0f:6703
2014-09-08 07:19:09 b.s.d.executor [INFO] Prepared bolt SOIMbolt:(1)
2014-09-08 07:19:09 b.s.m.n.Server [INFO] Create Netty Server Netty-server-localhost-6703, buffer_size: 5242880, maxWorkers: 1
2014-09-08 07:19:09 b.s.m.loader [INFO] Starting receive-thread: [stormId: PaVan-1-1410160741, port: 6703, thread-id: 0 ]
2014-09-08 07:19:09 b.s.d.worker [INFO] Worker has topology config {"storm.id" "PaVan-1-1410160741", "dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 500, "drpc.servers" ["192.168.10.7"], "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "storm-local", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "192.168.10.7", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["192.168.10.7"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.kryo.decorators" (), "topology.name" "PaVan", "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701 6702 6703], "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.kryo.register" nil, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "distributed", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144}
2014-09-08 07:19:09 b.s.d.worker [INFO] Worker 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 for storm PaVan-1-1410160741 on 51fb6b55-0522-437c-a18e-72212192db0f:6703 has finished loading
Supervisor Log:
b/json-simple-1.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/core.incubator-0.1.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/asm-4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-io-2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/httpclient-4.3.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jline-2.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.logging-0.2.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/logback-classic-1.0.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-logging-1.1.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-core-1.1.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-devel-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/curator-client-2.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clj-stacktrace-0.2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clj-time-0.4.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-lang-2.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/compojure-1.1.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/joda-time-2.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/chill-java-0.3.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clout-1.0.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/kryo-2.21.jar:/usr/local/apache-storm-0.9.2-incubating/lib/snakeyaml-1.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/minlog-1.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubating.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jgrapht-core-0.9.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/slf4j-api-1.6.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/hiccup-0.3.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/netty-3.6.3.Final.jar:/usr/local/apache-storm-0.9.2-incubating/lib/curator-framework-2.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/guava-13.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-fileupload-1.2.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/servlet-api-2.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/reflectasm-1.07-shaded.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jetty-util-6.1.26.jar:/usr/local/apache-storm-0.9.2-incubating/lib/objenesis-1.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.cli-0.2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-codec-1.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/zookeeper-3.4.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clojure-1.5.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/netty-3.2.2.Final.jar:/usr/local/apache-storm-0.9.2-incubating/lib/math.numeric-tower-0.0.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/carbonite-1.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/disruptor-2.10.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-exec-1.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.macro-0.1.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jetty-6.1.26.jar:/usr/local/apache-storm-0.9.2-incubating/lib/httpcore-4.3.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/servlet-api-2.5-20081211.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-servlet-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/logback-core-1.0.6.jar:/usr/local/apache-storm-0.9.2-incubating/conf:storm-local/supervisor/stormdist/PaVan-1-1410160741/stormjar.jar' 'backtype.storm.daemon.worker' 'PaVan-1-1410160741' '51fb6b55-0522-437c-a18e-72212192db0f' '6703' '2c12526f-d588-4dcc-99b9-b26e8c1d1d70'
2014-09-08 07:19:02 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:02 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:03 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:03 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:04 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:04 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:05 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:05 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:06 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:06 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:07 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:07 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:08 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:08 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
From: Raghavendra Nandagopal [mailto:askaboutstorm@gmail.com]
Sent: 09 September 2014 10:59
To: user@storm.incubator.apache.org
Subject: RE: Running Topology on Remote Machine
Hi Sridhar,
Can you check the worker logs and see if there are any exceptions?
Thanks,
Raghav
On Sep 8, 2014 10:25 PM, "Sridhar G" <Sr...@microland.com> wrote:
Hi Dhaval,
We have tried with the below config but no luck . Anything else to add to it ?
Regards
Sridhar Gadi
-----Original Message-----
From: Dhaval Modi [mailto:dhavalmodi24@gmail.com]
Sent: Tue 9/9/2014 9:45 AM
To: user
Cc: Pavan Jakati G
Subject: Re: Running Topology on Remote Machine
Hi Sridhar,
Did you try commenting " conf.setMaxTaskParallelism(10);" ?
Regards,
Dhaval Modi
dhavalmodi24@gmail.com
On 9 September 2014 09:13, Sridhar G <Sr...@microland.com> wrote:
>
>
>
>
> Hi All,
>
>
>
> We are stuck with running our topology on Remote Machine. We have be able
> to run our topology on local machine which emits tuples. But when we try to
> run it on Remote Machine it does not emit tuples. Whereas the Formation of
> Topology is shown on STORM UI under TOPOLOGY SUMMARY. TOPOLOGY STATS ,
> SPOUTS and BOLTS are visible with ZERO emits.
>
>
>
> The Code which we have used for Running Topology on REMOTE Machine is:
>
>
>
> LIBRARIES IMPORTS ARE -
>
>
>
> import backtype.storm.Config;
>
> import backtype.storm.LocalCluster;
>
> import backtype.storm.topology.TopologyBuilder;
>
> import backtype.storm.utils.Utils;
>
>
>
>
>
> import org.json.simple.JSONValue;
>
> import backtype.storm.StormSubmitter;
>
> import backtype.storm.generated.AlreadyAliveException;
>
> import backtype.storm.generated.Nimbus.Client;
>
> import backtype.storm.topology.TopologyBuilder;
>
> import backtype.storm.utils.NimbusClient;
>
> import java.util.Map;
>
>
> -------------------------------------------------------------------------------------------------------------------------
>
> *TOPOLOGY PROGRAM -*
>
>
>
> TopologyBuilder builder = new TopologyBuilder();
>
> builder.setSpout("POCspout", new POCSpout(),3);
>
> builder.setBolt("POCbolt", new
> POCBolt(),3).shuffleGrouping("POCspout");
>
>
>
> Config conf = new Config();
>
> conf.put(Config.NIMBUS_HOST, "192.xxx.xx.x");
>
> conf.setDebug(true);
>
> conf.setNumWorkers(1);
>
> conf.setMaxSpoutPending(5000);
>
> conf.setNumAckers(1);
>
> conf.setMaxTaskParallelism(10);
>
> Map storm_conf = Utils.readStormConfig();
>
>
>
> storm_conf.put("nimbus.host", "192.xxx.xx.x");
>
> Client client =
> NimbusClient.getConfiguredClient(storm_conf).getClient();
>
>
>
> String inputJar =
> "/root/SOIM_PaVan/target/SOIMTopology-1.0-SNAPSHOT.jar";
>
> NimbusClient nimbus = new NimbusClient(storm_conf,
> "192.xxx.xx.x",6627);
>
> String uploadedJarLocation =
> StormSubmitter.submitJar(storm_conf,inputJar);
>
>
>
> try {
>
> String jsonConf = JSONValue.toJSONString(storm_conf);
>
>
> nimbus.getClient().submitTopology("TOPOLOGYPOC",uploadedJarLocation,jsonConf,builder.createTopology());
>
> }
>
> catch (AlreadyAliveException ae)
>
> {
>
> ae.printStackTrace();
>
> }
>
>
>
>
>
>
>
> *For any Correction in code let us know.*
>
>
>
>
>
> Regards,
>
> Sridhar
>
> *Believe in the impossible and remove the improbable*
>
> The information contained in this transmission may contain privileged and
> confidential information of Microland Limited, including information
> protected by privacy laws. It is intended only for the use of Microland
> Limited. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution, or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
>
The information contained in this transmission may contain privileged and confidential information of Microland Limited, including information protected by privacy laws. It is intended only for the use of Microland Limited. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
--
Regards,
Vikas Agarwal
91 – 9928301411
InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 <tel:%2B1%20%28408%29%20988-2000> Work
+1 (408) 716-2726 <tel:%2B1%20%28408%29%20716-2726> Fax
--
Regards,
Vikas Agarwal
91 – 9928301411
InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
Re: Running Topology on Remote Machine
Posted by Vikas Agarwal <vi...@infoobjects.com>.
I stuck there for 2 weeks. :)
Basically, I tried to debug every command that storm processes are trying
to run. You can manually run the worker java command on console to check
what error it is throwing. In our case there were some problems, few of
them are:
1) slf4j jar conflicts
2) somehow wrong value of hostname in storm worker's configuration
3) something wrong in bolt's prepare method or spout's open method (may be
waiting for some connection outside of storm)
4) ports being blocked by firewall (i added few ports in Amazon EC2
security groups to make it runnable)
On Tue, Sep 9, 2014 at 12:48 PM, Pavan Jakati G <Pa...@microland.com>
wrote:
> How did you fix it Vikas . Can you help us here . We are stuck here since
> a week . L
>
>
>
> *From:* Vikas Agarwal [mailto:vikas@infoobjects.com]
> *Sent:* 09 September 2014 12:39
> *To:* user@storm.incubator.apache.org
>
> *Subject:* Re: Running Topology on Remote Machine
>
>
>
> We faced similar errors.
>
>
>
> May be it can help
> http://mail-archives.apache.org/mod_mbox/incubator-storm-user/201408.mbox/browser
>
>
>
> On Tue, Sep 9, 2014 at 11:02 AM, Pavan Jakati G <Pa...@microland.com>
> wrote:
>
> Worker log
>
>
>
> 2014-09-08 07:19:09 b.s.d.executor [INFO] Prepared bolt __system:(-1)
>
> 2014-09-08 07:19:09 b.s.d.executor [INFO] Loading executor SOIMbolt:[1 1]
>
> 2014-09-08 07:19:09 b.s.d.executor [INFO] Loaded executor tasks
> SOIMbolt:[1 1]
>
> 2014-09-08 07:19:09 b.s.d.executor [INFO] Finished loading executor
> SOIMbolt:[1 1]
>
> 2014-09-08 07:19:09 b.s.d.executor [INFO] Preparing bolt SOIMbolt:(1)
>
> 2014-09-08 07:19:09 b.s.d.worker [INFO] Launching receive-thread for
> 51fb6b55-0522-437c-a18e-72212192db0f:6703
>
> 2014-09-08 07:19:09 b.s.d.executor [INFO] Prepared bolt SOIMbolt:(1)
>
> 2014-09-08 07:19:09 b.s.m.n.Server [INFO] Create Netty Server
> Netty-server-localhost-6703, buffer_size: 5242880, maxWorkers: 1
>
> 2014-09-08 07:19:09 b.s.m.loader [INFO] Starting receive-thread: [stormId:
> PaVan-1-1410160741, port: 6703, thread-id: 0 ]
>
> 2014-09-08 07:19:09 b.s.d.worker [INFO] Worker has topology config {"
> storm.id" "PaVan-1-1410160741", "dev.zookeeper.path"
> "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil,
> "topology.builtin.metrics.bucket.size.secs" 60,
> "topology.fall.back.on.java.serialization" true,
> "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000,
> "topology.skip.missing.kryo.registrations" false,
> "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m",
> "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true,
> "topology.trident.batch.emit.interval.millis" 500, "drpc.servers"
> ["192.168.10.7"], "storm.messaging.netty.flush.check.interval.ms" 10,
> "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m",
> "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib",
> "topology.executor.send.buffer.size" 1024, "storm.local.dir" "storm-local",
> "storm.messaging.netty.buffer_size" 5242880,
> "supervisor.worker.start.timeout.secs" 120,
> "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs"
> 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64,
> "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "192.168.10.7",
> "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181,
> "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size"
> 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root"
> "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000,
> "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1,
> "storm.zookeeper.servers" ["192.168.10.7"], "transactional.zookeeper.root"
> "/transactional", "topology.acker.executors" nil,
> "topology.kryo.decorators" (), "topology.name" "PaVan",
> "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil,
> "drpc.queue.size" 128, "worker.childopts" "-Xmx768m",
> "supervisor.heartbeat.frequency.secs" 5,
> "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772,
> "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m",
> "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3,
> "topology.tasks" nil, "storm.messaging.netty.max_retries" 30,
> "topology.spout.wait.strategy"
> "backtype.storm.spout.SleepSpoutWaitStrategy",
> "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil,
> "storm.zookeeper.retry.interval" 1000, "
> topology.sleep.spout.wait.strategy.time.ms" 1,
> "nimbus.topology.validator"
> "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports"
> [6700 6701 6702 6703], "topology.debug" false, "nimbus.task.launch.secs"
> 120, "nimbus.supervisor.timeout.secs" 60, "topology.kryo.register" nil,
> "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10,
> "topology.workers" 1, "supervisor.childopts" "-Xmx256m",
> "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05,
> "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer"
> "backtype.storm.serialization.types.ListDelegateSerializer",
> "topology.disruptor.wait.strategy"
> "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer"
> "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30,
> "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory"
> "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port"
> 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times"
> 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport"
> "backtype.storm.security.auth.SimpleTransportPlugin",
> "topology.state.synchronization.timeout.secs" 60,
> "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs"
> 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "
> logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000,
> "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port"
> 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "distributed",
> "topology.max.task.parallelism" nil,
> "storm.messaging.netty.transfer.batch.size" 262144}
>
> 2014-09-08 07:19:09 b.s.d.worker [INFO] Worker
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 for storm PaVan-1-1410160741 on
> 51fb6b55-0522-437c-a18e-72212192db0f:6703 has finished loading
>
>
>
> Supervisor Log:
>
>
>
> b/json-simple-1.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/core.incubator-0.1.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/asm-4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-io-2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/httpclient-4.3.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jline-2.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.logging-0.2.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/logback-classic-1.0.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-logging-1.1.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-core-1.1.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-devel-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/curator-client-2.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clj-stacktrace-0.2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clj-time-0.4.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-lang-2.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/compojure-1.1.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/joda-time-2.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/chill-java-0.3.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clout-1.0.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/kryo-2.21.jar:/usr/local/apache-storm-0.9.2-incubating/lib/snakeyaml-1.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/minlog-1.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubating.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jgrapht-core-0.9.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/slf4j-api-1.6.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/hiccup-0.3.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/netty-3.6.3.Final.jar:/usr/local/apache-storm-0.9.2-incubating/lib/curator-framework-2.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/guava-13.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-fileupload-1.2.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/servlet-api-2.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/reflectasm-1.07-shaded.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jetty-util-6.1.26.jar:/usr/local/apache-storm-0.9.2-incubating/lib/objenesis-1.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.cli-0.2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-codec-1.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/zookeeper-3.4.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clojure-1.5.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/netty-3.2.2.Final.jar:/usr/local/apache-storm-0.9.2-incubating/lib/math.numeric-tower-0.0.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/carbonite-1.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/disruptor-2.10.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-exec-1.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.macro-0.1.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jetty-6.1.26.jar:/usr/local/apache-storm-0.9.2-incubating/lib/httpcore-4.3.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/servlet-api-2.5-20081211.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-servlet-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/logback-core-1.0.6.jar:/usr/local/apache-storm-0.9.2-incubating/conf:storm-local/supervisor/stormdist/PaVan-1-1410160741/stormjar.jar'
> 'backtype.storm.daemon.worker' 'PaVan-1-1410160741'
> '51fb6b55-0522-437c-a18e-72212192db0f' '6703'
> '2c12526f-d588-4dcc-99b9-b26e8c1d1d70'
>
> 2014-09-08 07:19:02 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:02 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:03 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:03 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:04 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:04 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:05 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:05 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:06 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:06 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:07 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:07 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:08 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:08 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
>
>
> *From:* Raghavendra Nandagopal [mailto:askaboutstorm@gmail.com]
> *Sent:* 09 September 2014 10:59
> *To:* user@storm.incubator.apache.org
> *Subject:* RE: Running Topology on Remote Machine
>
>
>
> Hi Sridhar,
> Can you check the worker logs and see if there are any exceptions?
>
> Thanks,
> Raghav
>
> On Sep 8, 2014 10:25 PM, "Sridhar G" <Sr...@microland.com> wrote:
>
> Hi Dhaval,
>
> We have tried with the below config but no luck . Anything else to add to
> it ?
>
> Regards
> Sridhar Gadi
>
>
> -----Original Message-----
> From: Dhaval Modi [mailto:dhavalmodi24@gmail.com <dh...@gmail.com>]
> Sent: Tue 9/9/2014 9:45 AM
> To: user
> Cc: Pavan Jakati G
> Subject: Re: Running Topology on Remote Machine
>
> Hi Sridhar,
>
> Did you try commenting " conf.setMaxTaskParallelism(10);" ?
>
> Regards,
> Dhaval Modi
> dhavalmodi24@gmail.com
>
> On 9 September 2014 09:13, Sridhar G <Sr...@microland.com> wrote:
>
> >
> >
> >
> >
> > Hi All,
> >
> >
> >
> > We are stuck with running our topology on Remote Machine. We have be able
> > to run our topology on local machine which emits tuples. But when we try
> to
> > run it on Remote Machine it does not emit tuples. Whereas the Formation
> of
> > Topology is shown on STORM UI under TOPOLOGY SUMMARY. TOPOLOGY STATS ,
> > SPOUTS and BOLTS are visible with ZERO emits.
> >
> >
> >
> > The Code which we have used for Running Topology on REMOTE Machine is:
> >
> >
> >
> > LIBRARIES IMPORTS ARE -
> >
> >
> >
> > import backtype.storm.Config;
> >
> > import backtype.storm.LocalCluster;
> >
> > import backtype.storm.topology.TopologyBuilder;
> >
> > import backtype.storm.utils.Utils;
> >
> >
> >
> >
> >
> > import org.json.simple.JSONValue;
> >
> > import backtype.storm.StormSubmitter;
> >
> > import backtype.storm.generated.AlreadyAliveException;
> >
> > import backtype.storm.generated.Nimbus.Client;
> >
> > import backtype.storm.topology.TopologyBuilder;
> >
> > import backtype.storm.utils.NimbusClient;
> >
> > import java.util.Map;
> >
> >
> >
> -------------------------------------------------------------------------------------------------------------------------
> >
> > *TOPOLOGY PROGRAM -*
> >
> >
> >
> > TopologyBuilder builder = new TopologyBuilder();
> >
> > builder.setSpout("POCspout", new POCSpout(),3);
> >
> > builder.setBolt("POCbolt", new
> > POCBolt(),3).shuffleGrouping("POCspout");
> >
> >
> >
> > Config conf = new Config();
> >
> > conf.put(Config.NIMBUS_HOST, "192.xxx.xx.x");
> >
> > conf.setDebug(true);
> >
> > conf.setNumWorkers(1);
> >
> > conf.setMaxSpoutPending(5000);
> >
> > conf.setNumAckers(1);
> >
> > conf.setMaxTaskParallelism(10);
> >
> > Map storm_conf = Utils.readStormConfig();
> >
> >
> >
> > storm_conf.put("nimbus.host", "192.xxx.xx.x");
> >
> > Client client =
> > NimbusClient.getConfiguredClient(storm_conf).getClient();
> >
> >
> >
> > String inputJar =
> > "/root/SOIM_PaVan/target/SOIMTopology-1.0-SNAPSHOT.jar";
> >
> > NimbusClient nimbus = new NimbusClient(storm_conf,
> > "192.xxx.xx.x",6627);
> >
> > String uploadedJarLocation =
> > StormSubmitter.submitJar(storm_conf,inputJar);
> >
> >
> >
> > try {
> >
> > String jsonConf = JSONValue.toJSONString(storm_conf);
> >
> >
> >
> nimbus.getClient().submitTopology("TOPOLOGYPOC",uploadedJarLocation,jsonConf,builder.createTopology());
> >
> > }
> >
> > catch (AlreadyAliveException ae)
> >
> > {
> >
> > ae.printStackTrace();
> >
> > }
> >
> >
> >
> >
> >
> >
> >
> > *For any Correction in code let us know.*
> >
> >
> >
> >
> >
> > Regards,
> >
> > Sridhar
> >
> > *Believe in the impossible and remove the improbable*
> >
> > The information contained in this transmission may contain privileged and
> > confidential information of Microland Limited, including information
> > protected by privacy laws. It is intended only for the use of Microland
> > Limited. If you are not the intended recipient, you are hereby notified
> > that any review, dissemination, distribution, or duplication of this
> > communication is strictly prohibited. If you are not the intended
> > recipient, please contact the sender by reply email and destroy all
> copies
> > of the original message.
> >
>
> The information contained in this transmission may contain privileged and
> confidential information of Microland Limited, including information
> protected by privacy laws. It is intended only for the use of Microland
> Limited. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution, or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
>
>
>
>
>
> --
> Regards,
> Vikas Agarwal
> 91 – 9928301411
>
> InfoObjects, Inc.
> Execution Matters
> http://www.infoobjects.com
> 2041 Mission College Boulevard, #280
> Santa Clara, CA 95054
> +1 (408) 988-2000 Work
> +1 (408) 716-2726 Fax
>
--
Regards,
Vikas Agarwal
91 – 9928301411
InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
RE: Running Topology on Remote Machine
Posted by Pavan Jakati G <Pa...@microland.com>.
How did you fix it Vikas . Can you help us here . We are stuck here since a week . L
From: Vikas Agarwal [mailto:vikas@infoobjects.com]
Sent: 09 September 2014 12:39
To: user@storm.incubator.apache.org
Subject: Re: Running Topology on Remote Machine
We faced similar errors.
May be it can help http://mail-archives.apache.org/mod_mbox/incubator-storm-user/201408.mbox/browser
On Tue, Sep 9, 2014 at 11:02 AM, Pavan Jakati G <Pa...@microland.com> wrote:
Worker log
2014-09-08 07:19:09 b.s.d.executor [INFO] Prepared bolt __system:(-1)
2014-09-08 07:19:09 b.s.d.executor [INFO] Loading executor SOIMbolt:[1 1]
2014-09-08 07:19:09 b.s.d.executor [INFO] Loaded executor tasks SOIMbolt:[1 1]
2014-09-08 07:19:09 b.s.d.executor [INFO] Finished loading executor SOIMbolt:[1 1]
2014-09-08 07:19:09 b.s.d.executor [INFO] Preparing bolt SOIMbolt:(1)
2014-09-08 07:19:09 b.s.d.worker [INFO] Launching receive-thread for 51fb6b55-0522-437c-a18e-72212192db0f:6703
2014-09-08 07:19:09 b.s.d.executor [INFO] Prepared bolt SOIMbolt:(1)
2014-09-08 07:19:09 b.s.m.n.Server [INFO] Create Netty Server Netty-server-localhost-6703, buffer_size: 5242880, maxWorkers: 1
2014-09-08 07:19:09 b.s.m.loader [INFO] Starting receive-thread: [stormId: PaVan-1-1410160741, port: 6703, thread-id: 0 ]
2014-09-08 07:19:09 b.s.d.worker [INFO] Worker has topology config {"storm.id" "PaVan-1-1410160741", "dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 500, "drpc.servers" ["192.168.10.7"], "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "storm-local", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "192.168.10.7", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["192.168.10.7"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.kryo.decorators" (), "topology.name" "PaVan", "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701 6702 6703], "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.kryo.register" nil, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "distributed", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144}
2014-09-08 07:19:09 b.s.d.worker [INFO] Worker 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 for storm PaVan-1-1410160741 on 51fb6b55-0522-437c-a18e-72212192db0f:6703 has finished loading
Supervisor Log:
b/json-simple-1.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/core.incubator-0.1.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/asm-4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-io-2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/httpclient-4.3.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jline-2.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.logging-0.2.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/logback-classic-1.0.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-logging-1.1.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-core-1.1.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-devel-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/curator-client-2.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clj-stacktrace-0.2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clj-time-0.4.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-lang-2.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/compojure-1.1.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/joda-time-2.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/chill-java-0.3.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clout-1.0.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/kryo-2.21.jar:/usr/local/apache-storm-0.9.2-incubating/lib/snakeyaml-1.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/minlog-1.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubating.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jgrapht-core-0.9.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/slf4j-api-1.6.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/hiccup-0.3.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/netty-3.6.3.Final.jar:/usr/local/apache-storm-0.9.2-incubating/lib/curator-framework-2.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/guava-13.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-fileupload-1.2.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/servlet-api-2.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/reflectasm-1.07-shaded.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jetty-util-6.1.26.jar:/usr/local/apache-storm-0.9.2-incubating/lib/objenesis-1.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.cli-0.2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-codec-1.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/zookeeper-3.4.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clojure-1.5.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/netty-3.2.2.Final.jar:/usr/local/apache-storm-0.9.2-incubating/lib/math.numeric-tower-0.0.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/carbonite-1.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/disruptor-2.10.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-exec-1.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.macro-0.1.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jetty-6.1.26.jar:/usr/local/apache-storm-0.9.2-incubating/lib/httpcore-4.3.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/servlet-api-2.5-20081211.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-servlet-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/logback-core-1.0.6.jar:/usr/local/apache-storm-0.9.2-incubating/conf:storm-local/supervisor/stormdist/PaVan-1-1410160741/stormjar.jar' 'backtype.storm.daemon.worker' 'PaVan-1-1410160741' '51fb6b55-0522-437c-a18e-72212192db0f' '6703' '2c12526f-d588-4dcc-99b9-b26e8c1d1d70'
2014-09-08 07:19:02 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:02 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:03 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:03 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:04 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:04 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:05 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:05 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:06 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:06 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:07 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:07 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:08 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:08 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
From: Raghavendra Nandagopal [mailto:askaboutstorm@gmail.com]
Sent: 09 September 2014 10:59
To: user@storm.incubator.apache.org
Subject: RE: Running Topology on Remote Machine
Hi Sridhar,
Can you check the worker logs and see if there are any exceptions?
Thanks,
Raghav
On Sep 8, 2014 10:25 PM, "Sridhar G" <Sr...@microland.com> wrote:
Hi Dhaval,
We have tried with the below config but no luck . Anything else to add to it ?
Regards
Sridhar Gadi
-----Original Message-----
From: Dhaval Modi [mailto:dhavalmodi24@gmail.com]
Sent: Tue 9/9/2014 9:45 AM
To: user
Cc: Pavan Jakati G
Subject: Re: Running Topology on Remote Machine
Hi Sridhar,
Did you try commenting " conf.setMaxTaskParallelism(10);" ?
Regards,
Dhaval Modi
dhavalmodi24@gmail.com
On 9 September 2014 09:13, Sridhar G <Sr...@microland.com> wrote:
>
>
>
>
> Hi All,
>
>
>
> We are stuck with running our topology on Remote Machine. We have be able
> to run our topology on local machine which emits tuples. But when we try to
> run it on Remote Machine it does not emit tuples. Whereas the Formation of
> Topology is shown on STORM UI under TOPOLOGY SUMMARY. TOPOLOGY STATS ,
> SPOUTS and BOLTS are visible with ZERO emits.
>
>
>
> The Code which we have used for Running Topology on REMOTE Machine is:
>
>
>
> LIBRARIES IMPORTS ARE -
>
>
>
> import backtype.storm.Config;
>
> import backtype.storm.LocalCluster;
>
> import backtype.storm.topology.TopologyBuilder;
>
> import backtype.storm.utils.Utils;
>
>
>
>
>
> import org.json.simple.JSONValue;
>
> import backtype.storm.StormSubmitter;
>
> import backtype.storm.generated.AlreadyAliveException;
>
> import backtype.storm.generated.Nimbus.Client;
>
> import backtype.storm.topology.TopologyBuilder;
>
> import backtype.storm.utils.NimbusClient;
>
> import java.util.Map;
>
>
> -------------------------------------------------------------------------------------------------------------------------
>
> *TOPOLOGY PROGRAM -*
>
>
>
> TopologyBuilder builder = new TopologyBuilder();
>
> builder.setSpout("POCspout", new POCSpout(),3);
>
> builder.setBolt("POCbolt", new
> POCBolt(),3).shuffleGrouping("POCspout");
>
>
>
> Config conf = new Config();
>
> conf.put(Config.NIMBUS_HOST, "192.xxx.xx.x");
>
> conf.setDebug(true);
>
> conf.setNumWorkers(1);
>
> conf.setMaxSpoutPending(5000);
>
> conf.setNumAckers(1);
>
> conf.setMaxTaskParallelism(10);
>
> Map storm_conf = Utils.readStormConfig();
>
>
>
> storm_conf.put("nimbus.host", "192.xxx.xx.x");
>
> Client client =
> NimbusClient.getConfiguredClient(storm_conf).getClient();
>
>
>
> String inputJar =
> "/root/SOIM_PaVan/target/SOIMTopology-1.0-SNAPSHOT.jar";
>
> NimbusClient nimbus = new NimbusClient(storm_conf,
> "192.xxx.xx.x",6627);
>
> String uploadedJarLocation =
> StormSubmitter.submitJar(storm_conf,inputJar);
>
>
>
> try {
>
> String jsonConf = JSONValue.toJSONString(storm_conf);
>
>
> nimbus.getClient().submitTopology("TOPOLOGYPOC",uploadedJarLocation,jsonConf,builder.createTopology());
>
> }
>
> catch (AlreadyAliveException ae)
>
> {
>
> ae.printStackTrace();
>
> }
>
>
>
>
>
>
>
> *For any Correction in code let us know.*
>
>
>
>
>
> Regards,
>
> Sridhar
>
> *Believe in the impossible and remove the improbable*
>
> The information contained in this transmission may contain privileged and
> confidential information of Microland Limited, including information
> protected by privacy laws. It is intended only for the use of Microland
> Limited. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution, or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
>
The information contained in this transmission may contain privileged and confidential information of Microland Limited, including information protected by privacy laws. It is intended only for the use of Microland Limited. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
--
Regards,
Vikas Agarwal
91 – 9928301411
InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
Re: Running Topology on Remote Machine
Posted by Vikas Agarwal <vi...@infoobjects.com>.
We faced similar errors.
May be it can help
http://mail-archives.apache.org/mod_mbox/incubator-storm-user/201408.mbox/browser
On Tue, Sep 9, 2014 at 11:02 AM, Pavan Jakati G <Pa...@microland.com>
wrote:
> Worker log
>
>
>
> 2014-09-08 07:19:09 b.s.d.executor [INFO] Prepared bolt __system:(-1)
>
> 2014-09-08 07:19:09 b.s.d.executor [INFO] Loading executor SOIMbolt:[1 1]
>
> 2014-09-08 07:19:09 b.s.d.executor [INFO] Loaded executor tasks
> SOIMbolt:[1 1]
>
> 2014-09-08 07:19:09 b.s.d.executor [INFO] Finished loading executor
> SOIMbolt:[1 1]
>
> 2014-09-08 07:19:09 b.s.d.executor [INFO] Preparing bolt SOIMbolt:(1)
>
> 2014-09-08 07:19:09 b.s.d.worker [INFO] Launching receive-thread for
> 51fb6b55-0522-437c-a18e-72212192db0f:6703
>
> 2014-09-08 07:19:09 b.s.d.executor [INFO] Prepared bolt SOIMbolt:(1)
>
> 2014-09-08 07:19:09 b.s.m.n.Server [INFO] Create Netty Server
> Netty-server-localhost-6703, buffer_size: 5242880, maxWorkers: 1
>
> 2014-09-08 07:19:09 b.s.m.loader [INFO] Starting receive-thread: [stormId:
> PaVan-1-1410160741, port: 6703, thread-id: 0 ]
>
> 2014-09-08 07:19:09 b.s.d.worker [INFO] Worker has topology config {"
> storm.id" "PaVan-1-1410160741", "dev.zookeeper.path"
> "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil,
> "topology.builtin.metrics.bucket.size.secs" 60,
> "topology.fall.back.on.java.serialization" true,
> "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000,
> "topology.skip.missing.kryo.registrations" false,
> "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m",
> "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true,
> "topology.trident.batch.emit.interval.millis" 500, "drpc.servers"
> ["192.168.10.7"], "storm.messaging.netty.flush.check.interval.ms" 10,
> "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m",
> "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib",
> "topology.executor.send.buffer.size" 1024, "storm.local.dir" "storm-local",
> "storm.messaging.netty.buffer_size" 5242880,
> "supervisor.worker.start.timeout.secs" 120,
> "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs"
> 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64,
> "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "192.168.10.7",
> "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181,
> "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size"
> 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root"
> "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000,
> "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1,
> "storm.zookeeper.servers" ["192.168.10.7"], "transactional.zookeeper.root"
> "/transactional", "topology.acker.executors" nil,
> "topology.kryo.decorators" (), "topology.name" "PaVan",
> "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil,
> "drpc.queue.size" 128, "worker.childopts" "-Xmx768m",
> "supervisor.heartbeat.frequency.secs" 5,
> "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772,
> "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m",
> "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3,
> "topology.tasks" nil, "storm.messaging.netty.max_retries" 30,
> "topology.spout.wait.strategy"
> "backtype.storm.spout.SleepSpoutWaitStrategy",
> "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil,
> "storm.zookeeper.retry.interval" 1000, "
> topology.sleep.spout.wait.strategy.time.ms" 1,
> "nimbus.topology.validator"
> "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports"
> [6700 6701 6702 6703], "topology.debug" false, "nimbus.task.launch.secs"
> 120, "nimbus.supervisor.timeout.secs" 60, "topology.kryo.register" nil,
> "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10,
> "topology.workers" 1, "supervisor.childopts" "-Xmx256m",
> "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05,
> "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer"
> "backtype.storm.serialization.types.ListDelegateSerializer",
> "topology.disruptor.wait.strategy"
> "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer"
> "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30,
> "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory"
> "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port"
> 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times"
> 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport"
> "backtype.storm.security.auth.SimpleTransportPlugin",
> "topology.state.synchronization.timeout.secs" 60,
> "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs"
> 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "
> logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000,
> "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port"
> 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "distributed",
> "topology.max.task.parallelism" nil,
> "storm.messaging.netty.transfer.batch.size" 262144}
>
> 2014-09-08 07:19:09 b.s.d.worker [INFO] Worker
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 for storm PaVan-1-1410160741 on
> 51fb6b55-0522-437c-a18e-72212192db0f:6703 has finished loading
>
>
>
> Supervisor Log:
>
>
>
> b/json-simple-1.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/core.incubator-0.1.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/asm-4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-io-2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/httpclient-4.3.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jline-2.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.logging-0.2.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/logback-classic-1.0.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-logging-1.1.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-core-1.1.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-devel-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/curator-client-2.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clj-stacktrace-0.2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clj-time-0.4.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-lang-2.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/compojure-1.1.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/joda-time-2.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/chill-java-0.3.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clout-1.0.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/kryo-2.21.jar:/usr/local/apache-storm-0.9.2-incubating/lib/snakeyaml-1.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/minlog-1.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubating.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jgrapht-core-0.9.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/slf4j-api-1.6.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/hiccup-0.3.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/netty-3.6.3.Final.jar:/usr/local/apache-storm-0.9.2-incubating/lib/curator-framework-2.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/guava-13.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-fileupload-1.2.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/servlet-api-2.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/reflectasm-1.07-shaded.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jetty-util-6.1.26.jar:/usr/local/apache-storm-0.9.2-incubating/lib/objenesis-1.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.cli-0.2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-codec-1.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/zookeeper-3.4.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clojure-1.5.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/netty-3.2.2.Final.jar:/usr/local/apache-storm-0.9.2-incubating/lib/math.numeric-tower-0.0.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/carbonite-1.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/disruptor-2.10.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-exec-1.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.macro-0.1.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jetty-6.1.26.jar:/usr/local/apache-storm-0.9.2-incubating/lib/httpcore-4.3.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/servlet-api-2.5-20081211.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-servlet-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/logback-core-1.0.6.jar:/usr/local/apache-storm-0.9.2-incubating/conf:storm-local/supervisor/stormdist/PaVan-1-1410160741/stormjar.jar'
> 'backtype.storm.daemon.worker' 'PaVan-1-1410160741'
> '51fb6b55-0522-437c-a18e-72212192db0f' '6703'
> '2c12526f-d588-4dcc-99b9-b26e8c1d1d70'
>
> 2014-09-08 07:19:02 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:02 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:03 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:03 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:04 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:04 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:05 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:05 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:06 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:06 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:07 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:07 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:08 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
> 2014-09-08 07:19:08 b.s.d.supervisor [INFO]
> 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
>
>
>
> *From:* Raghavendra Nandagopal [mailto:askaboutstorm@gmail.com]
> *Sent:* 09 September 2014 10:59
> *To:* user@storm.incubator.apache.org
> *Subject:* RE: Running Topology on Remote Machine
>
>
>
> Hi Sridhar,
> Can you check the worker logs and see if there are any exceptions?
>
> Thanks,
> Raghav
>
> On Sep 8, 2014 10:25 PM, "Sridhar G" <Sr...@microland.com> wrote:
>
> Hi Dhaval,
>
> We have tried with the below config but no luck . Anything else to add to
> it ?
>
> Regards
> Sridhar Gadi
>
>
> -----Original Message-----
> From: Dhaval Modi [mailto:dhavalmodi24@gmail.com <dh...@gmail.com>]
> Sent: Tue 9/9/2014 9:45 AM
> To: user
> Cc: Pavan Jakati G
> Subject: Re: Running Topology on Remote Machine
>
> Hi Sridhar,
>
> Did you try commenting " conf.setMaxTaskParallelism(10);" ?
>
> Regards,
> Dhaval Modi
> dhavalmodi24@gmail.com
>
> On 9 September 2014 09:13, Sridhar G <Sr...@microland.com> wrote:
>
> >
> >
> >
> >
> > Hi All,
> >
> >
> >
> > We are stuck with running our topology on Remote Machine. We have be able
> > to run our topology on local machine which emits tuples. But when we try
> to
> > run it on Remote Machine it does not emit tuples. Whereas the Formation
> of
> > Topology is shown on STORM UI under TOPOLOGY SUMMARY. TOPOLOGY STATS ,
> > SPOUTS and BOLTS are visible with ZERO emits.
> >
> >
> >
> > The Code which we have used for Running Topology on REMOTE Machine is:
> >
> >
> >
> > LIBRARIES IMPORTS ARE -
> >
> >
> >
> > import backtype.storm.Config;
> >
> > import backtype.storm.LocalCluster;
> >
> > import backtype.storm.topology.TopologyBuilder;
> >
> > import backtype.storm.utils.Utils;
> >
> >
> >
> >
> >
> > import org.json.simple.JSONValue;
> >
> > import backtype.storm.StormSubmitter;
> >
> > import backtype.storm.generated.AlreadyAliveException;
> >
> > import backtype.storm.generated.Nimbus.Client;
> >
> > import backtype.storm.topology.TopologyBuilder;
> >
> > import backtype.storm.utils.NimbusClient;
> >
> > import java.util.Map;
> >
> >
> >
> -------------------------------------------------------------------------------------------------------------------------
> >
> > *TOPOLOGY PROGRAM -*
> >
> >
> >
> > TopologyBuilder builder = new TopologyBuilder();
> >
> > builder.setSpout("POCspout", new POCSpout(),3);
> >
> > builder.setBolt("POCbolt", new
> > POCBolt(),3).shuffleGrouping("POCspout");
> >
> >
> >
> > Config conf = new Config();
> >
> > conf.put(Config.NIMBUS_HOST, "192.xxx.xx.x");
> >
> > conf.setDebug(true);
> >
> > conf.setNumWorkers(1);
> >
> > conf.setMaxSpoutPending(5000);
> >
> > conf.setNumAckers(1);
> >
> > conf.setMaxTaskParallelism(10);
> >
> > Map storm_conf = Utils.readStormConfig();
> >
> >
> >
> > storm_conf.put("nimbus.host", "192.xxx.xx.x");
> >
> > Client client =
> > NimbusClient.getConfiguredClient(storm_conf).getClient();
> >
> >
> >
> > String inputJar =
> > "/root/SOIM_PaVan/target/SOIMTopology-1.0-SNAPSHOT.jar";
> >
> > NimbusClient nimbus = new NimbusClient(storm_conf,
> > "192.xxx.xx.x",6627);
> >
> > String uploadedJarLocation =
> > StormSubmitter.submitJar(storm_conf,inputJar);
> >
> >
> >
> > try {
> >
> > String jsonConf = JSONValue.toJSONString(storm_conf);
> >
> >
> >
> nimbus.getClient().submitTopology("TOPOLOGYPOC",uploadedJarLocation,jsonConf,builder.createTopology());
> >
> > }
> >
> > catch (AlreadyAliveException ae)
> >
> > {
> >
> > ae.printStackTrace();
> >
> > }
> >
> >
> >
> >
> >
> >
> >
> > *For any Correction in code let us know.*
> >
> >
> >
> >
> >
> > Regards,
> >
> > Sridhar
> >
> > *Believe in the impossible and remove the improbable*
> >
> > The information contained in this transmission may contain privileged and
> > confidential information of Microland Limited, including information
> > protected by privacy laws. It is intended only for the use of Microland
> > Limited. If you are not the intended recipient, you are hereby notified
> > that any review, dissemination, distribution, or duplication of this
> > communication is strictly prohibited. If you are not the intended
> > recipient, please contact the sender by reply email and destroy all
> copies
> > of the original message.
> >
>
> The information contained in this transmission may contain privileged and
> confidential information of Microland Limited, including information
> protected by privacy laws. It is intended only for the use of Microland
> Limited. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution, or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
>
--
Regards,
Vikas Agarwal
91 – 9928301411
InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
RE: Running Topology on Remote Machine
Posted by Pavan Jakati G <Pa...@microland.com>.
Worker log
2014-09-08 07:19:09 b.s.d.executor [INFO] Prepared bolt __system:(-1)
2014-09-08 07:19:09 b.s.d.executor [INFO] Loading executor SOIMbolt:[1 1]
2014-09-08 07:19:09 b.s.d.executor [INFO] Loaded executor tasks SOIMbolt:[1 1]
2014-09-08 07:19:09 b.s.d.executor [INFO] Finished loading executor SOIMbolt:[1 1]
2014-09-08 07:19:09 b.s.d.executor [INFO] Preparing bolt SOIMbolt:(1)
2014-09-08 07:19:09 b.s.d.worker [INFO] Launching receive-thread for 51fb6b55-0522-437c-a18e-72212192db0f:6703
2014-09-08 07:19:09 b.s.d.executor [INFO] Prepared bolt SOIMbolt:(1)
2014-09-08 07:19:09 b.s.m.n.Server [INFO] Create Netty Server Netty-server-localhost-6703, buffer_size: 5242880, maxWorkers: 1
2014-09-08 07:19:09 b.s.m.loader [INFO] Starting receive-thread: [stormId: PaVan-1-1410160741, port: 6703, thread-id: 0 ]
2014-09-08 07:19:09 b.s.d.worker [INFO] Worker has topology config {"storm.id" "PaVan-1-1410160741", "dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 500, "drpc.servers" ["192.168.10.7"], "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "storm-local", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "192.168.10.7", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["192.168.10.7"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.kryo.decorators" (), "topology.name" "PaVan", "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701 6702 6703], "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.kryo.register" nil, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "distributed", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144}
2014-09-08 07:19:09 b.s.d.worker [INFO] Worker 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 for storm PaVan-1-1410160741 on 51fb6b55-0522-437c-a18e-72212192db0f:6703 has finished loading
Supervisor Log:
b/json-simple-1.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/core.incubator-0.1.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/asm-4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-io-2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/httpclient-4.3.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jline-2.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.logging-0.2.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/logback-classic-1.0.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-logging-1.1.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-core-1.1.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-devel-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/curator-client-2.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clj-stacktrace-0.2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clj-time-0.4.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-lang-2.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/compojure-1.1.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/joda-time-2.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/chill-java-0.3.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clout-1.0.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/kryo-2.21.jar:/usr/local/apache-storm-0.9.2-incubating/lib/snakeyaml-1.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/minlog-1.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubating.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jgrapht-core-0.9.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/slf4j-api-1.6.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/hiccup-0.3.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/netty-3.6.3.Final.jar:/usr/local/apache-storm-0.9.2-incubating/lib/curator-framework-2.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/guava-13.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-fileupload-1.2.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/servlet-api-2.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/reflectasm-1.07-shaded.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jetty-util-6.1.26.jar:/usr/local/apache-storm-0.9.2-incubating/lib/objenesis-1.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.cli-0.2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-codec-1.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/zookeeper-3.4.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clojure-1.5.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/netty-3.2.2.Final.jar:/usr/local/apache-storm-0.9.2-incubating/lib/math.numeric-tower-0.0.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/carbonite-1.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/disruptor-2.10.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-exec-1.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.macro-0.1.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jetty-6.1.26.jar:/usr/local/apache-storm-0.9.2-incubating/lib/httpcore-4.3.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/servlet-api-2.5-20081211.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-servlet-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/logback-core-1.0.6.jar:/usr/local/apache-storm-0.9.2-incubating/conf:storm-local/supervisor/stormdist/PaVan-1-1410160741/stormjar.jar' 'backtype.storm.daemon.worker' 'PaVan-1-1410160741' '51fb6b55-0522-437c-a18e-72212192db0f' '6703' '2c12526f-d588-4dcc-99b9-b26e8c1d1d70'
2014-09-08 07:19:02 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:02 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:03 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:03 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:04 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:04 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:05 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:05 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:06 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:06 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:07 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:07 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:08 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
2014-09-08 07:19:08 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started
From: Raghavendra Nandagopal [mailto:askaboutstorm@gmail.com]
Sent: 09 September 2014 10:59
To: user@storm.incubator.apache.org
Subject: RE: Running Topology on Remote Machine
Hi Sridhar,
Can you check the worker logs and see if there are any exceptions?
Thanks,
Raghav
On Sep 8, 2014 10:25 PM, "Sridhar G" <Sr...@microland.com> wrote:
Hi Dhaval,
We have tried with the below config but no luck . Anything else to add to it ?
Regards
Sridhar Gadi
-----Original Message-----
From: Dhaval Modi [mailto:dhavalmodi24@gmail.com]
Sent: Tue 9/9/2014 9:45 AM
To: user
Cc: Pavan Jakati G
Subject: Re: Running Topology on Remote Machine
Hi Sridhar,
Did you try commenting " conf.setMaxTaskParallelism(10);" ?
Regards,
Dhaval Modi
dhavalmodi24@gmail.com
On 9 September 2014 09:13, Sridhar G <Sr...@microland.com> wrote:
>
>
>
>
> Hi All,
>
>
>
> We are stuck with running our topology on Remote Machine. We have be able
> to run our topology on local machine which emits tuples. But when we try to
> run it on Remote Machine it does not emit tuples. Whereas the Formation of
> Topology is shown on STORM UI under TOPOLOGY SUMMARY. TOPOLOGY STATS ,
> SPOUTS and BOLTS are visible with ZERO emits.
>
>
>
> The Code which we have used for Running Topology on REMOTE Machine is:
>
>
>
> LIBRARIES IMPORTS ARE -
>
>
>
> import backtype.storm.Config;
>
> import backtype.storm.LocalCluster;
>
> import backtype.storm.topology.TopologyBuilder;
>
> import backtype.storm.utils.Utils;
>
>
>
>
>
> import org.json.simple.JSONValue;
>
> import backtype.storm.StormSubmitter;
>
> import backtype.storm.generated.AlreadyAliveException;
>
> import backtype.storm.generated.Nimbus.Client;
>
> import backtype.storm.topology.TopologyBuilder;
>
> import backtype.storm.utils.NimbusClient;
>
> import java.util.Map;
>
>
> -------------------------------------------------------------------------------------------------------------------------
>
> *TOPOLOGY PROGRAM -*
>
>
>
> TopologyBuilder builder = new TopologyBuilder();
>
> builder.setSpout("POCspout", new POCSpout(),3);
>
> builder.setBolt("POCbolt", new
> POCBolt(),3).shuffleGrouping("POCspout");
>
>
>
> Config conf = new Config();
>
> conf.put(Config.NIMBUS_HOST, "192.xxx.xx.x");
>
> conf.setDebug(true);
>
> conf.setNumWorkers(1);
>
> conf.setMaxSpoutPending(5000);
>
> conf.setNumAckers(1);
>
> conf.setMaxTaskParallelism(10);
>
> Map storm_conf = Utils.readStormConfig();
>
>
>
> storm_conf.put("nimbus.host", "192.xxx.xx.x");
>
> Client client =
> NimbusClient.getConfiguredClient(storm_conf).getClient();
>
>
>
> String inputJar =
> "/root/SOIM_PaVan/target/SOIMTopology-1.0-SNAPSHOT.jar";
>
> NimbusClient nimbus = new NimbusClient(storm_conf,
> "192.xxx.xx.x",6627);
>
> String uploadedJarLocation =
> StormSubmitter.submitJar(storm_conf,inputJar);
>
>
>
> try {
>
> String jsonConf = JSONValue.toJSONString(storm_conf);
>
>
> nimbus.getClient().submitTopology("TOPOLOGYPOC",uploadedJarLocation,jsonConf,builder.createTopology());
>
> }
>
> catch (AlreadyAliveException ae)
>
> {
>
> ae.printStackTrace();
>
> }
>
>
>
>
>
>
>
> *For any Correction in code let us know.*
>
>
>
>
>
> Regards,
>
> Sridhar
>
> *Believe in the impossible and remove the improbable*
>
> The information contained in this transmission may contain privileged and
> confidential information of Microland Limited, including information
> protected by privacy laws. It is intended only for the use of Microland
> Limited. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution, or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
>
The information contained in this transmission may contain privileged and confidential information of Microland Limited, including information protected by privacy laws. It is intended only for the use of Microland Limited. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
RE: Running Topology on Remote Machine
Posted by Raghavendra Nandagopal <as...@gmail.com>.
Hi Sridhar,
Can you check the worker logs and see if there are any exceptions?
Thanks,
Raghav
On Sep 8, 2014 10:25 PM, "Sridhar G" <Sr...@microland.com> wrote:
> Hi Dhaval,
>
> We have tried with the below config but no luck . Anything else to add to
> it ?
>
> Regards
> Sridhar Gadi
>
>
> -----Original Message-----
> From: Dhaval Modi [mailto:dhavalmodi24@gmail.com <dh...@gmail.com>]
> Sent: Tue 9/9/2014 9:45 AM
> To: user
> Cc: Pavan Jakati G
> Subject: Re: Running Topology on Remote Machine
>
> Hi Sridhar,
>
> Did you try commenting " conf.setMaxTaskParallelism(10);" ?
>
> Regards,
> Dhaval Modi
> dhavalmodi24@gmail.com
>
> On 9 September 2014 09:13, Sridhar G <Sr...@microland.com> wrote:
>
> >
> >
> >
> >
> > Hi All,
> >
> >
> >
> > We are stuck with running our topology on Remote Machine. We have be able
> > to run our topology on local machine which emits tuples. But when we try
> to
> > run it on Remote Machine it does not emit tuples. Whereas the Formation
> of
> > Topology is shown on STORM UI under TOPOLOGY SUMMARY. TOPOLOGY STATS ,
> > SPOUTS and BOLTS are visible with ZERO emits.
> >
> >
> >
> > The Code which we have used for Running Topology on REMOTE Machine is:
> >
> >
> >
> > LIBRARIES IMPORTS ARE -
> >
> >
> >
> > import backtype.storm.Config;
> >
> > import backtype.storm.LocalCluster;
> >
> > import backtype.storm.topology.TopologyBuilder;
> >
> > import backtype.storm.utils.Utils;
> >
> >
> >
> >
> >
> > import org.json.simple.JSONValue;
> >
> > import backtype.storm.StormSubmitter;
> >
> > import backtype.storm.generated.AlreadyAliveException;
> >
> > import backtype.storm.generated.Nimbus.Client;
> >
> > import backtype.storm.topology.TopologyBuilder;
> >
> > import backtype.storm.utils.NimbusClient;
> >
> > import java.util.Map;
> >
> >
> >
> -------------------------------------------------------------------------------------------------------------------------
> >
> > *TOPOLOGY PROGRAM -*
> >
> >
> >
> > TopologyBuilder builder = new TopologyBuilder();
> >
> > builder.setSpout("POCspout", new POCSpout(),3);
> >
> > builder.setBolt("POCbolt", new
> > POCBolt(),3).shuffleGrouping("POCspout");
> >
> >
> >
> > Config conf = new Config();
> >
> > conf.put(Config.NIMBUS_HOST, "192.xxx.xx.x");
> >
> > conf.setDebug(true);
> >
> > conf.setNumWorkers(1);
> >
> > conf.setMaxSpoutPending(5000);
> >
> > conf.setNumAckers(1);
> >
> > conf.setMaxTaskParallelism(10);
> >
> > Map storm_conf = Utils.readStormConfig();
> >
> >
> >
> > storm_conf.put("nimbus.host", "192.xxx.xx.x");
> >
> > Client client =
> > NimbusClient.getConfiguredClient(storm_conf).getClient();
> >
> >
> >
> > String inputJar =
> > "/root/SOIM_PaVan/target/SOIMTopology-1.0-SNAPSHOT.jar";
> >
> > NimbusClient nimbus = new NimbusClient(storm_conf,
> > "192.xxx.xx.x",6627);
> >
> > String uploadedJarLocation =
> > StormSubmitter.submitJar(storm_conf,inputJar);
> >
> >
> >
> > try {
> >
> > String jsonConf = JSONValue.toJSONString(storm_conf);
> >
> >
> >
> nimbus.getClient().submitTopology("TOPOLOGYPOC",uploadedJarLocation,jsonConf,builder.createTopology());
> >
> > }
> >
> > catch (AlreadyAliveException ae)
> >
> > {
> >
> > ae.printStackTrace();
> >
> > }
> >
> >
> >
> >
> >
> >
> >
> > *For any Correction in code let us know.*
> >
> >
> >
> >
> >
> > Regards,
> >
> > Sridhar
> >
> > *Believe in the impossible and remove the improbable*
> >
> > The information contained in this transmission may contain privileged and
> > confidential information of Microland Limited, including information
> > protected by privacy laws. It is intended only for the use of Microland
> > Limited. If you are not the intended recipient, you are hereby notified
> > that any review, dissemination, distribution, or duplication of this
> > communication is strictly prohibited. If you are not the intended
> > recipient, please contact the sender by reply email and destroy all
> copies
> > of the original message.
> >
>
> The information contained in this transmission may contain privileged
> and confidential information of Microland Limited, including information
> protected by privacy laws. It is intended only for the use of Microland
> Limited. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution, or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
>
RE: Running Topology on Remote Machine
Posted by Sridhar G <Sr...@microland.com>.
Hi Dhaval,
We have tried with the below config but no luck . Anything else to add to it ?
Regards
Sridhar Gadi
-----Original Message-----
From: Dhaval Modi [mailto:dhavalmodi24@gmail.com]
Sent: Tue 9/9/2014 9:45 AM
To: user
Cc: Pavan Jakati G
Subject: Re: Running Topology on Remote Machine
Hi Sridhar,
Did you try commenting " conf.setMaxTaskParallelism(10);" ?
Regards,
Dhaval Modi
dhavalmodi24@gmail.com
On 9 September 2014 09:13, Sridhar G <Sr...@microland.com> wrote:
>
>
>
>
> Hi All,
>
>
>
> We are stuck with running our topology on Remote Machine. We have be able
> to run our topology on local machine which emits tuples. But when we try to
> run it on Remote Machine it does not emit tuples. Whereas the Formation of
> Topology is shown on STORM UI under TOPOLOGY SUMMARY. TOPOLOGY STATS ,
> SPOUTS and BOLTS are visible with ZERO emits.
>
>
>
> The Code which we have used for Running Topology on REMOTE Machine is:
>
>
>
> LIBRARIES IMPORTS ARE -
>
>
>
> import backtype.storm.Config;
>
> import backtype.storm.LocalCluster;
>
> import backtype.storm.topology.TopologyBuilder;
>
> import backtype.storm.utils.Utils;
>
>
>
>
>
> import org.json.simple.JSONValue;
>
> import backtype.storm.StormSubmitter;
>
> import backtype.storm.generated.AlreadyAliveException;
>
> import backtype.storm.generated.Nimbus.Client;
>
> import backtype.storm.topology.TopologyBuilder;
>
> import backtype.storm.utils.NimbusClient;
>
> import java.util.Map;
>
>
> -------------------------------------------------------------------------------------------------------------------------
>
> *TOPOLOGY PROGRAM -*
>
>
>
> TopologyBuilder builder = new TopologyBuilder();
>
> builder.setSpout("POCspout", new POCSpout(),3);
>
> builder.setBolt("POCbolt", new
> POCBolt(),3).shuffleGrouping("POCspout");
>
>
>
> Config conf = new Config();
>
> conf.put(Config.NIMBUS_HOST, "192.xxx.xx.x");
>
> conf.setDebug(true);
>
> conf.setNumWorkers(1);
>
> conf.setMaxSpoutPending(5000);
>
> conf.setNumAckers(1);
>
> conf.setMaxTaskParallelism(10);
>
> Map storm_conf = Utils.readStormConfig();
>
>
>
> storm_conf.put("nimbus.host", "192.xxx.xx.x");
>
> Client client =
> NimbusClient.getConfiguredClient(storm_conf).getClient();
>
>
>
> String inputJar =
> "/root/SOIM_PaVan/target/SOIMTopology-1.0-SNAPSHOT.jar";
>
> NimbusClient nimbus = new NimbusClient(storm_conf,
> "192.xxx.xx.x",6627);
>
> String uploadedJarLocation =
> StormSubmitter.submitJar(storm_conf,inputJar);
>
>
>
> try {
>
> String jsonConf = JSONValue.toJSONString(storm_conf);
>
>
> nimbus.getClient().submitTopology("TOPOLOGYPOC",uploadedJarLocation,jsonConf,builder.createTopology());
>
> }
>
> catch (AlreadyAliveException ae)
>
> {
>
> ae.printStackTrace();
>
> }
>
>
>
>
>
>
>
> *For any Correction in code let us know.*
>
>
>
>
>
> Regards,
>
> Sridhar
>
> *Believe in the impossible and remove the improbable*
>
> The information contained in this transmission may contain privileged and
> confidential information of Microland Limited, including information
> protected by privacy laws. It is intended only for the use of Microland
> Limited. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution, or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
>
The information contained in this transmission may contain privileged and confidential information of Microland Limited, including information protected by privacy laws. It is intended only for the use of Microland Limited. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
Re: Running Topology on Remote Machine
Posted by Dhaval Modi <dh...@gmail.com>.
Hi Sridhar,
Did you try commenting " conf.setMaxTaskParallelism(10);" ?
Regards,
Dhaval Modi
dhavalmodi24@gmail.com
On 9 September 2014 09:13, Sridhar G <Sr...@microland.com> wrote:
>
>
>
>
> Hi All,
>
>
>
> We are stuck with running our topology on Remote Machine. We have be able
> to run our topology on local machine which emits tuples. But when we try to
> run it on Remote Machine it does not emit tuples. Whereas the Formation of
> Topology is shown on STORM UI under TOPOLOGY SUMMARY. TOPOLOGY STATS ,
> SPOUTS and BOLTS are visible with ZERO emits.
>
>
>
> The Code which we have used for Running Topology on REMOTE Machine is:
>
>
>
> LIBRARIES IMPORTS ARE –
>
>
>
> import backtype.storm.Config;
>
> import backtype.storm.LocalCluster;
>
> import backtype.storm.topology.TopologyBuilder;
>
> import backtype.storm.utils.Utils;
>
>
>
>
>
> import org.json.simple.JSONValue;
>
> import backtype.storm.StormSubmitter;
>
> import backtype.storm.generated.AlreadyAliveException;
>
> import backtype.storm.generated.Nimbus.Client;
>
> import backtype.storm.topology.TopologyBuilder;
>
> import backtype.storm.utils.NimbusClient;
>
> import java.util.Map;
>
>
> -------------------------------------------------------------------------------------------------------------------------
>
> *TOPOLOGY PROGRAM –*
>
>
>
> TopologyBuilder builder = new TopologyBuilder();
>
> builder.setSpout("POCspout", new POCSpout(),3);
>
> builder.setBolt("POCbolt", new
> POCBolt(),3).shuffleGrouping("POCspout");
>
>
>
> Config conf = new Config();
>
> conf.put(Config.NIMBUS_HOST, "192.xxx.xx.x");
>
> conf.setDebug(true);
>
> conf.setNumWorkers(1);
>
> conf.setMaxSpoutPending(5000);
>
> conf.setNumAckers(1);
>
> conf.setMaxTaskParallelism(10);
>
> Map storm_conf = Utils.readStormConfig();
>
>
>
> storm_conf.put("nimbus.host", "192.xxx.xx.x");
>
> Client client =
> NimbusClient.getConfiguredClient(storm_conf).getClient();
>
>
>
> String inputJar =
> "/root/SOIM_PaVan/target/SOIMTopology-1.0-SNAPSHOT.jar";
>
> NimbusClient nimbus = new NimbusClient(storm_conf,
> "192.xxx.xx.x",6627);
>
> String uploadedJarLocation =
> StormSubmitter.submitJar(storm_conf,inputJar);
>
>
>
> try {
>
> String jsonConf = JSONValue.toJSONString(storm_conf);
>
>
> nimbus.getClient().submitTopology("TOPOLOGYPOC",uploadedJarLocation,jsonConf,builder.createTopology());
>
> }
>
> catch (AlreadyAliveException ae)
>
> {
>
> ae.printStackTrace();
>
> }
>
>
>
>
>
>
>
> *For any Correction in code let us know.*
>
>
>
>
>
> Regards,
>
> Sridhar
>
> *Believe in the impossible and remove the improbable*
>
> The information contained in this transmission may contain privileged and
> confidential information of Microland Limited, including information
> protected by privacy laws. It is intended only for the use of Microland
> Limited. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution, or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
>