You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by ch huang <ju...@gmail.com> on 2014/01/13 03:12:12 UTC
how can i connect remote storm cluster and submit topology?
hi,maillist:
i create a new storm project in eclipse and now question is how
can i connect to remote storm cluster ,submit the topology?
Re: how can i connect remote storm cluster and submit topology?
Posted by Richards Peter <hb...@gmail.com>.
Hi,
The following is a query I posted in storm-user group:
https://groups.google.com/forum/#!searchin/storm-user/from$3Ame$20program$20topology/storm-user/5JjZQyviRqk/TGRCgciHKWwJ
The approach mentioned in this post still works well for me. You can try it
out. However with this approach you will have to package the code to a jar
file before submitting it. We create jar using maven.
Regards,
Richards Peter.
Re: 答复: how can i connect remote storm cluster and submit topology?
Posted by ch huang <ju...@gmail.com>.
hi, the code try to run on local ,not remote storm cluster,because i just
download the storm to local, not install any zookeeper in local ,so when i
offer an argument to ExclamationTopology
and modify the config code ,let it point to my storm nimbus ,all work fine
,thank you reply me very quickly and thank dingdongchao offer the solution
On Mon, Jan 13, 2014 at 2:37 PM, Nathan Leung <nc...@gmail.com> wrote:
> Zookeeper is trying to remove some log file and is unable to. What user
> is zookeeper running as? Does the folder it's trying to delete from exist?
> I don't run ZK on windows but I know in linux it usually is given a
> permanent folder to store its logs.
>
>
> On Mon, Jan 13, 2014 at 12:46 AM, ch huang <ju...@gmail.com> wrote:
>
>> i try run ExclamationTopology which no python involved ,also get error
>>
>>
>> 12656 [main] INFO backtype.storm.daemon.executor - Shutting down executor __system:[-1 -1]
>> 12656 [Thread-46-__system] INFO backtype.storm.util - Async loop interrupted!
>> 12656 [Thread-45] INFO backtype.storm.util - Async loop interrupted!
>> 12656 [main] INFO backtype.storm.daemon.executor - Shut down executor __system:[-1 -1]
>> 12656 [main] INFO backtype.storm.daemon.executor - Shutting down executor __acker:[1 1]
>> 12656 [Thread-48-__acker] INFO backtype.storm.util - Async loop interrupted!
>> 12656 [Thread-47] INFO backtype.storm.util - Async loop interrupted!
>> 12656 [main] INFO backtype.storm.daemon.executor - Shut down executor __acker:[1 1]
>> 12656 [main] INFO backtype.storm.daemon.worker - Shut down executors
>> 12656 [main] INFO backtype.storm.daemon.worker - Shutting down transfer thread
>> 12656 [Thread-50] INFO backtype.storm.util - Async loop interrupted!
>> 12656 [main] INFO backtype.storm.daemon.worker - Shut down transfer thread
>> 12656 [main] INFO backtype.storm.daemon.worker - Shutting down default resources
>> 12656 [main] INFO backtype.storm.daemon.worker - Shut down default resources
>> 12688 [main] INFO backtype.storm.daemon.worker - Disconnecting from storm cluster state context
>> 12703 [main] INFO backtype.storm.daemon.worker - Shut down worker test-1-1389591370 6f6a4263-93ce-48c8-bace-f5a0a28e84d4 1
>>
>> 12703 [main] INFO backtype.storm.daemon.supervisor - Shut down 6f6a4263-93ce-48c8-bace-f5a0a28e84d4:8857888b-0a27-4b0b-ad51-c7200d4960b8
>>
>> 12703 [main] INFO backtype.storm.daemon.supervisor - Shutting down supervisor 6f6a4263-93ce-48c8-bace-f5a0a28e84d4
>> 12703 [Thread-5] INFO backtype.storm.event - Event manager interrupted
>> 12703 [Thread-6] INFO backtype.storm.event - Event manager interrupted
>> 12719 [main] INFO backtype.storm.daemon.supervisor - Shutting down supervisor 0da74f54-468b-405a-9d30-3b0a05298641
>> 12719 [Thread-8] INFO backtype.storm.event - Event manager interrupted
>> 12719 [Thread-9] INFO backtype.storm.event - Event manager interrupted
>> 12750 [main] INFO backtype.storm.testing - Shutting down in process zookeeper
>> 12750 [main] INFO backtype.storm.testing - Done shutting down in process zookeeper
>> 12750 [main] INFO backtype.storm.testing - Deleting temporary path C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\/384fcb8d-55c5-4822-9109-2d63acdc70d1
>>
>> 12750 [main] INFO backtype.storm.testing - Deleting temporary path C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\/b39b1754-278e-490e-bcee-bd6c84bf5c1b
>> 12750 [main] ERROR org.apache.zookeeper.server.NIOServerCnxn - Thread Thread[main,5,main] died
>>
>> java.io.IOException: Unable to delete file: C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\b39b1754-278e-490e-bcee-bd6c84bf5c1b\version-2\log.1
>>
>> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1390) ~[commons-io-1.4.jar:1.4]
>> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1044) ~[commons-io-1.4.jar:1.4]
>> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:977) ~[commons-io-1.4.jar:1.4]
>> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1381) ~[commons-io-1.4.jar:1.4]
>> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1044) ~[commons-io-1.4.jar:1.4]
>> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:977) ~[commons-io-1.4.jar:1.4]
>> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1381) ~[commons-io-1.4.jar:1.4]
>> at backtype.storm.util$rmr.invoke(util.clj:442) ~[storm-core-0.9.0.1.jar:na]
>> at backtype.storm.testing$kill_local_storm_cluster.invoke(testing.clj:164) ~[storm-core-0.9.0.1.jar:na]
>> at backtype.storm.LocalCluster$_shutdown.invoke(LocalCluster.clj:32) ~[storm-core-0.9.0.1.jar:na]
>>
>> at backtype.storm.LocalCluster.shutdown(Unknown Source) ~[storm-core-0.9.0.1.jar:na]
>> at storm.starter.ExclamationTopology.main(ExclamationTopology.java:67) ~[classes/:na]
>>
>>
>>
>>
>> On Mon, Jan 13, 2014 at 1:40 PM, Nathan Leung <nc...@gmail.com> wrote:
>>
>>> multilang is trying to run "python" but your binary is under "python.exe"
>>>
>>>
>>> On Mon, Jan 13, 2014 at 12:15 AM, ch huang <ju...@gmail.com> wrote:
>>>
>>>> i get following error when i run the wordcounttopology,i already
>>>> installed python
>>>>
>>>> C:\Documents and Settings\Administrator>python.exe -V
>>>> Python 2.6.6
>>>>
>>>> 3781 [Thread-24-split] INFO backtype.storm.daemon.executor - Preparing
>>>> bolt split:(6)
>>>> 3781 [Thread-24-split] ERROR backtype.storm.util - Async loop died!
>>>> java.lang.RuntimeException: Error when launching multilang subprocess
>>>> at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:88)
>>>> ~[storm-core-0.9.0.1.jar:na]
>>>> at
>>>> backtype.storm.daemon.executor$fn__3498$fn__3510.invoke(executor.clj:674)
>>>> ~[storm-core-0.9.0.1.jar:na]
>>>> at backtype.storm.util$async_loop$fn__444.invoke(util.clj:401)
>>>> ~[storm-core-0.9.0.1.jar:na]
>>>> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
>>>> at java.lang.Thread.run(Thread.java:619) [na:1.6.0_14]
>>>> Caused by: java.io.IOException: Cannot run program "python" (in
>>>> directory
>>>> "C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\40b2df00-5a53-4034-8c20-3f0a084bdd56\supervisor\stormdist\word-count-1-1389585368\resources"):
>>>> CreateProcess error=2, ϵͳÕҲ»µ½ָ¶
>>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
>>>> ~[na:1.6.0_14]
>>>> at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:33)
>>>> ~[storm-core-0.9.0.1.jar:na]
>>>> at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:85)
>>>> ~[storm-core-0.9.0.1.jar:na]
>>>> ... 4 common frames omitted
>>>> Caused by: java.io.IOException: CreateProcess error=2, ϵͳÕҲ»µ½ָ¶
>>>> at java.lang.ProcessImpl.create(Native Method) ~[na:1.6.0_14]
>>>> at java.lang.ProcessImpl.<init>(ProcessImpl.java:81) ~[na:1.6.0_14]
>>>> at java.lang.ProcessImpl.start(ProcessImpl.java:30) ~[na:1.6.0_14]
>>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
>>>> ~[na:1.6.0_14]
>>>> ... 6 common frames omitted
>>>> 3781 [Thread-24-split] ERROR backtype.storm.daemon.executor -
>>>>
>>>> On Mon, Jan 13, 2014 at 11:50 AM, Nathan Leung <nc...@gmail.com>wrote:
>>>>
>>>>> You can run maven on eclipse and package on windows. If you have
>>>>> python you can probably submit from a windows box to a linux nimbus (if you
>>>>> look at the storm file it's a python script) but I've never tried that
>>>>> before so I can't say for sure. If you decide to try this make sure you
>>>>> have the same version of jre/jdk installed on both windows and linux
>>>>> environments.
>>>>>
>>>>>
>>>>> On Sun, Jan 12, 2014 at 10:25 PM, dingdongchao <
>>>>> dingdongchaoxiao@gmail.com> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> “package my code” is must , because running your topology
>>>>>> on storm cluster needs this jar .
>>>>>>
>>>>>> ===========================================================================>
>>>>>>
>>>>>>
>>>>>> System.*setProperty*("storm.jar",
>>>>>> "E:/workspace/cdc-dp-data-storm-project/target/cdc-cdp.jar");
>>>>>>
>>>>>> Config config=*new* Config();
>>>>>>
>>>>>> config.setDebug(*false*);
>>>>>>
>>>>>> config.put(Config.*NIMBUS_HOST*,"cq01-crm-lin2rd52.vm");
>>>>>>
>>>>>> config.put(Config.*NIMBUS_THRIFT_PORT*,8889);
>>>>>>
>>>>>> config.setNumWorkers(2);
>>>>>>
>>>>>> StormSubmitter.*submitTopology*("cdp-reader-online", config,
>>>>>> builder.createTopology());
>>>>>>
>>>>>>
>>>>>>
>>>>>> Every time , I package my code into one jar location "E:/workspace/cdc-dp-data-storm-project/target/cdc-cdp.jar"
>>>>>> again , then run this codes to *submit the topology *
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thx
>>>>>>
>>>>>>
>>>>>>
>>>>>> *发件人:* ch huang [mailto:justlooks@gmail.com]
>>>>>> *发送时间:* 2014年1月13日 11:02
>>>>>> *收件人:* user@storm.incubator.apache.org
>>>>>> *主题:* Re: how can i connect remote storm cluster and submit topology?
>>>>>>
>>>>>>
>>>>>>
>>>>>> now ,problem is i the storm cluster is ready , i get a linux box for
>>>>>> test ,change /root/.storm/storm.yaml to point to the cluster nimbus , and
>>>>>> submit the ExclamationTopology through the following command
>>>>>>
>>>>>>
>>>>>>
>>>>>> # storm-0.9.0-wip21/bin/storm jar
>>>>>> storm-starter-master/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar
>>>>>> storm.starter.ExclamationTopology exclamation-topology
>>>>>>
>>>>>> it's work fine ,but i do storm topology write on a windowsXP OS box
>>>>>> using eclipse as IDE,i do not know how to get my code runing and submit the
>>>>>> topology to my remote cluster,i do not want to package my code ,and upload
>>>>>> it to my linux box ,then run it
>>>>>>
>>>>>> On Mon, Jan 13, 2014 at 10:32 AM, Nathan Leung <nc...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Assuming everything else is setup, configure your storm.yaml and then
>>>>>> just run the "storm jar" command. Assuming it's not, these two links are
>>>>>> good references:
>>>>>>
>>>>>>
>>>>>>
>>>>>> https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster
>>>>>>
>>>>>>
>>>>>> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Jan 12, 2014 at 9:12 PM, ch huang <ju...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> hi,maillist:
>>>>>>
>>>>>> i create a new storm project in eclipse and now question is
>>>>>> how can i connect to remote storm cluster ,submit the topology?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: 答复: how can i connect remote storm cluster and submit topology?
Posted by Nathan Leung <nc...@gmail.com>.
Zookeeper is trying to remove some log file and is unable to. What user is
zookeeper running as? Does the folder it's trying to delete from exist? I
don't run ZK on windows but I know in linux it usually is given a permanent
folder to store its logs.
On Mon, Jan 13, 2014 at 12:46 AM, ch huang <ju...@gmail.com> wrote:
> i try run ExclamationTopology which no python involved ,also get error
>
>
> 12656 [main] INFO backtype.storm.daemon.executor - Shutting down executor __system:[-1 -1]
> 12656 [Thread-46-__system] INFO backtype.storm.util - Async loop interrupted!
> 12656 [Thread-45] INFO backtype.storm.util - Async loop interrupted!
> 12656 [main] INFO backtype.storm.daemon.executor - Shut down executor __system:[-1 -1]
> 12656 [main] INFO backtype.storm.daemon.executor - Shutting down executor __acker:[1 1]
> 12656 [Thread-48-__acker] INFO backtype.storm.util - Async loop interrupted!
> 12656 [Thread-47] INFO backtype.storm.util - Async loop interrupted!
> 12656 [main] INFO backtype.storm.daemon.executor - Shut down executor __acker:[1 1]
> 12656 [main] INFO backtype.storm.daemon.worker - Shut down executors
> 12656 [main] INFO backtype.storm.daemon.worker - Shutting down transfer thread
> 12656 [Thread-50] INFO backtype.storm.util - Async loop interrupted!
> 12656 [main] INFO backtype.storm.daemon.worker - Shut down transfer thread
> 12656 [main] INFO backtype.storm.daemon.worker - Shutting down default resources
> 12656 [main] INFO backtype.storm.daemon.worker - Shut down default resources
> 12688 [main] INFO backtype.storm.daemon.worker - Disconnecting from storm cluster state context
> 12703 [main] INFO backtype.storm.daemon.worker - Shut down worker test-1-1389591370 6f6a4263-93ce-48c8-bace-f5a0a28e84d4 1
> 12703 [main] INFO backtype.storm.daemon.supervisor - Shut down 6f6a4263-93ce-48c8-bace-f5a0a28e84d4:8857888b-0a27-4b0b-ad51-c7200d4960b8
>
> 12703 [main] INFO backtype.storm.daemon.supervisor - Shutting down supervisor 6f6a4263-93ce-48c8-bace-f5a0a28e84d4
> 12703 [Thread-5] INFO backtype.storm.event - Event manager interrupted
> 12703 [Thread-6] INFO backtype.storm.event - Event manager interrupted
> 12719 [main] INFO backtype.storm.daemon.supervisor - Shutting down supervisor 0da74f54-468b-405a-9d30-3b0a05298641
> 12719 [Thread-8] INFO backtype.storm.event - Event manager interrupted
> 12719 [Thread-9] INFO backtype.storm.event - Event manager interrupted
> 12750 [main] INFO backtype.storm.testing - Shutting down in process zookeeper
> 12750 [main] INFO backtype.storm.testing - Done shutting down in process zookeeper
> 12750 [main] INFO backtype.storm.testing - Deleting temporary path C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\/384fcb8d-55c5-4822-9109-2d63acdc70d1
>
> 12750 [main] INFO backtype.storm.testing - Deleting temporary path C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\/b39b1754-278e-490e-bcee-bd6c84bf5c1b
> 12750 [main] ERROR org.apache.zookeeper.server.NIOServerCnxn - Thread Thread[main,5,main] died
> java.io.IOException: Unable to delete file: C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\b39b1754-278e-490e-bcee-bd6c84bf5c1b\version-2\log.1
>
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1390) ~[commons-io-1.4.jar:1.4]
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1044) ~[commons-io-1.4.jar:1.4]
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:977) ~[commons-io-1.4.jar:1.4]
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1381) ~[commons-io-1.4.jar:1.4]
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1044) ~[commons-io-1.4.jar:1.4]
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:977) ~[commons-io-1.4.jar:1.4]
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1381) ~[commons-io-1.4.jar:1.4]
> at backtype.storm.util$rmr.invoke(util.clj:442) ~[storm-core-0.9.0.1.jar:na]
> at backtype.storm.testing$kill_local_storm_cluster.invoke(testing.clj:164) ~[storm-core-0.9.0.1.jar:na]
> at backtype.storm.LocalCluster$_shutdown.invoke(LocalCluster.clj:32) ~[storm-core-0.9.0.1.jar:na]
> at backtype.storm.LocalCluster.shutdown(Unknown Source) ~[storm-core-0.9.0.1.jar:na]
> at storm.starter.ExclamationTopology.main(ExclamationTopology.java:67) ~[classes/:na]
>
>
>
>
> On Mon, Jan 13, 2014 at 1:40 PM, Nathan Leung <nc...@gmail.com> wrote:
>
>> multilang is trying to run "python" but your binary is under "python.exe"
>>
>>
>> On Mon, Jan 13, 2014 at 12:15 AM, ch huang <ju...@gmail.com> wrote:
>>
>>> i get following error when i run the wordcounttopology,i already
>>> installed python
>>>
>>> C:\Documents and Settings\Administrator>python.exe -V
>>> Python 2.6.6
>>>
>>> 3781 [Thread-24-split] INFO backtype.storm.daemon.executor - Preparing
>>> bolt split:(6)
>>> 3781 [Thread-24-split] ERROR backtype.storm.util - Async loop died!
>>> java.lang.RuntimeException: Error when launching multilang subprocess
>>> at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:88)
>>> ~[storm-core-0.9.0.1.jar:na]
>>> at
>>> backtype.storm.daemon.executor$fn__3498$fn__3510.invoke(executor.clj:674)
>>> ~[storm-core-0.9.0.1.jar:na]
>>> at backtype.storm.util$async_loop$fn__444.invoke(util.clj:401)
>>> ~[storm-core-0.9.0.1.jar:na]
>>> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
>>> at java.lang.Thread.run(Thread.java:619) [na:1.6.0_14]
>>> Caused by: java.io.IOException: Cannot run program "python" (in
>>> directory
>>> "C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\40b2df00-5a53-4034-8c20-3f0a084bdd56\supervisor\stormdist\word-count-1-1389585368\resources"):
>>> CreateProcess error=2, ϵͳÕҲ»µ½ָ¶
>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
>>> ~[na:1.6.0_14]
>>> at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:33)
>>> ~[storm-core-0.9.0.1.jar:na]
>>> at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:85)
>>> ~[storm-core-0.9.0.1.jar:na]
>>> ... 4 common frames omitted
>>> Caused by: java.io.IOException: CreateProcess error=2, ϵͳÕҲ»µ½ָ¶
>>> at java.lang.ProcessImpl.create(Native Method) ~[na:1.6.0_14]
>>> at java.lang.ProcessImpl.<init>(ProcessImpl.java:81) ~[na:1.6.0_14]
>>> at java.lang.ProcessImpl.start(ProcessImpl.java:30) ~[na:1.6.0_14]
>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
>>> ~[na:1.6.0_14]
>>> ... 6 common frames omitted
>>> 3781 [Thread-24-split] ERROR backtype.storm.daemon.executor -
>>>
>>> On Mon, Jan 13, 2014 at 11:50 AM, Nathan Leung <nc...@gmail.com>wrote:
>>>
>>>> You can run maven on eclipse and package on windows. If you have
>>>> python you can probably submit from a windows box to a linux nimbus (if you
>>>> look at the storm file it's a python script) but I've never tried that
>>>> before so I can't say for sure. If you decide to try this make sure you
>>>> have the same version of jre/jdk installed on both windows and linux
>>>> environments.
>>>>
>>>>
>>>> On Sun, Jan 12, 2014 at 10:25 PM, dingdongchao <
>>>> dingdongchaoxiao@gmail.com> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> “package my code” is must , because running your topology on
>>>>> storm cluster needs this jar .
>>>>>
>>>>> ===========================================================================>
>>>>>
>>>>>
>>>>> System.*setProperty*("storm.jar",
>>>>> "E:/workspace/cdc-dp-data-storm-project/target/cdc-cdp.jar");
>>>>>
>>>>> Config config=*new* Config();
>>>>>
>>>>> config.setDebug(*false*);
>>>>>
>>>>> config.put(Config.*NIMBUS_HOST*,"cq01-crm-lin2rd52.vm");
>>>>>
>>>>> config.put(Config.*NIMBUS_THRIFT_PORT*,8889);
>>>>>
>>>>> config.setNumWorkers(2);
>>>>>
>>>>> StormSubmitter.*submitTopology*("cdp-reader-online", config,
>>>>> builder.createTopology());
>>>>>
>>>>>
>>>>>
>>>>> Every time , I package my code into one jar location "E:/workspace/cdc-dp-data-storm-project/target/cdc-cdp.jar"
>>>>> again , then run this codes to *submit the topology *
>>>>>
>>>>>
>>>>>
>>>>> Thx
>>>>>
>>>>>
>>>>>
>>>>> *发件人:* ch huang [mailto:justlooks@gmail.com]
>>>>> *发送时间:* 2014年1月13日 11:02
>>>>> *收件人:* user@storm.incubator.apache.org
>>>>> *主题:* Re: how can i connect remote storm cluster and submit topology?
>>>>>
>>>>>
>>>>>
>>>>> now ,problem is i the storm cluster is ready , i get a linux box for
>>>>> test ,change /root/.storm/storm.yaml to point to the cluster nimbus , and
>>>>> submit the ExclamationTopology through the following command
>>>>>
>>>>>
>>>>>
>>>>> # storm-0.9.0-wip21/bin/storm jar
>>>>> storm-starter-master/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar
>>>>> storm.starter.ExclamationTopology exclamation-topology
>>>>>
>>>>> it's work fine ,but i do storm topology write on a windowsXP OS box
>>>>> using eclipse as IDE,i do not know how to get my code runing and submit the
>>>>> topology to my remote cluster,i do not want to package my code ,and upload
>>>>> it to my linux box ,then run it
>>>>>
>>>>> On Mon, Jan 13, 2014 at 10:32 AM, Nathan Leung <nc...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Assuming everything else is setup, configure your storm.yaml and then
>>>>> just run the "storm jar" command. Assuming it's not, these two links are
>>>>> good references:
>>>>>
>>>>>
>>>>>
>>>>> https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster
>>>>>
>>>>> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Jan 12, 2014 at 9:12 PM, ch huang <ju...@gmail.com> wrote:
>>>>>
>>>>> hi,maillist:
>>>>>
>>>>> i create a new storm project in eclipse and now question is
>>>>> how can i connect to remote storm cluster ,submit the topology?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
Re: 答复: how can i connect remote storm cluster and submit topology?
Posted by ch huang <ju...@gmail.com>.
i try run ExclamationTopology which no python involved ,also get error
12656 [main] INFO backtype.storm.daemon.executor - Shutting down
executor __system:[-1 -1]
12656 [Thread-46-__system] INFO backtype.storm.util - Async loop interrupted!
12656 [Thread-45] INFO backtype.storm.util - Async loop interrupted!
12656 [main] INFO backtype.storm.daemon.executor - Shut down executor
__system:[-1 -1]
12656 [main] INFO backtype.storm.daemon.executor - Shutting down
executor __acker:[1 1]
12656 [Thread-48-__acker] INFO backtype.storm.util - Async loop interrupted!
12656 [Thread-47] INFO backtype.storm.util - Async loop interrupted!
12656 [main] INFO backtype.storm.daemon.executor - Shut down executor
__acker:[1 1]
12656 [main] INFO backtype.storm.daemon.worker - Shut down executors
12656 [main] INFO backtype.storm.daemon.worker - Shutting down transfer thread
12656 [Thread-50] INFO backtype.storm.util - Async loop interrupted!
12656 [main] INFO backtype.storm.daemon.worker - Shut down transfer thread
12656 [main] INFO backtype.storm.daemon.worker - Shutting down
default resources
12656 [main] INFO backtype.storm.daemon.worker - Shut down default resources
12688 [main] INFO backtype.storm.daemon.worker - Disconnecting from
storm cluster state context
12703 [main] INFO backtype.storm.daemon.worker - Shut down worker
test-1-1389591370 6f6a4263-93ce-48c8-bace-f5a0a28e84d4 1
12703 [main] INFO backtype.storm.daemon.supervisor - Shut down
6f6a4263-93ce-48c8-bace-f5a0a28e84d4:8857888b-0a27-4b0b-ad51-c7200d4960b8
12703 [main] INFO backtype.storm.daemon.supervisor - Shutting down
supervisor 6f6a4263-93ce-48c8-bace-f5a0a28e84d4
12703 [Thread-5] INFO backtype.storm.event - Event manager interrupted
12703 [Thread-6] INFO backtype.storm.event - Event manager interrupted
12719 [main] INFO backtype.storm.daemon.supervisor - Shutting down
supervisor 0da74f54-468b-405a-9d30-3b0a05298641
12719 [Thread-8] INFO backtype.storm.event - Event manager interrupted
12719 [Thread-9] INFO backtype.storm.event - Event manager interrupted
12750 [main] INFO backtype.storm.testing - Shutting down in process zookeeper
12750 [main] INFO backtype.storm.testing - Done shutting down in
process zookeeper
12750 [main] INFO backtype.storm.testing - Deleting temporary path
C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\/384fcb8d-55c5-4822-9109-2d63acdc70d1
12750 [main] INFO backtype.storm.testing - Deleting temporary path
C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\/b39b1754-278e-490e-bcee-bd6c84bf5c1b
12750 [main] ERROR org.apache.zookeeper.server.NIOServerCnxn - Thread
Thread[main,5,main] died
java.io.IOException: Unable to delete file:
C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\b39b1754-278e-490e-bcee-bd6c84bf5c1b\version-2\log.1
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1390)
~[commons-io-1.4.jar:1.4]
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1044)
~[commons-io-1.4.jar:1.4]
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:977)
~[commons-io-1.4.jar:1.4]
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1381)
~[commons-io-1.4.jar:1.4]
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1044)
~[commons-io-1.4.jar:1.4]
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:977)
~[commons-io-1.4.jar:1.4]
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1381)
~[commons-io-1.4.jar:1.4]
at backtype.storm.util$rmr.invoke(util.clj:442) ~[storm-core-0.9.0.1.jar:na]
at backtype.storm.testing$kill_local_storm_cluster.invoke(testing.clj:164)
~[storm-core-0.9.0.1.jar:na]
at backtype.storm.LocalCluster$_shutdown.invoke(LocalCluster.clj:32)
~[storm-core-0.9.0.1.jar:na]
at backtype.storm.LocalCluster.shutdown(Unknown Source)
~[storm-core-0.9.0.1.jar:na]
at storm.starter.ExclamationTopology.main(ExclamationTopology.java:67)
~[classes/:na]
On Mon, Jan 13, 2014 at 1:40 PM, Nathan Leung <nc...@gmail.com> wrote:
> multilang is trying to run "python" but your binary is under "python.exe"
>
>
> On Mon, Jan 13, 2014 at 12:15 AM, ch huang <ju...@gmail.com> wrote:
>
>> i get following error when i run the wordcounttopology,i already
>> installed python
>>
>> C:\Documents and Settings\Administrator>python.exe -V
>> Python 2.6.6
>>
>> 3781 [Thread-24-split] INFO backtype.storm.daemon.executor - Preparing
>> bolt split:(6)
>> 3781 [Thread-24-split] ERROR backtype.storm.util - Async loop died!
>> java.lang.RuntimeException: Error when launching multilang subprocess
>> at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:88)
>> ~[storm-core-0.9.0.1.jar:na]
>> at
>> backtype.storm.daemon.executor$fn__3498$fn__3510.invoke(executor.clj:674)
>> ~[storm-core-0.9.0.1.jar:na]
>> at backtype.storm.util$async_loop$fn__444.invoke(util.clj:401)
>> ~[storm-core-0.9.0.1.jar:na]
>> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
>> at java.lang.Thread.run(Thread.java:619) [na:1.6.0_14]
>> Caused by: java.io.IOException: Cannot run program "python" (in directory
>> "C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\40b2df00-5a53-4034-8c20-3f0a084bdd56\supervisor\stormdist\word-count-1-1389585368\resources"):
>> CreateProcess error=2, ϵͳÕҲ»µ½ָ¶
>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:459) ~[na:1.6.0_14]
>> at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:33)
>> ~[storm-core-0.9.0.1.jar:na]
>> at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:85)
>> ~[storm-core-0.9.0.1.jar:na]
>> ... 4 common frames omitted
>> Caused by: java.io.IOException: CreateProcess error=2, ϵͳÕҲ»µ½ָ¶
>> at java.lang.ProcessImpl.create(Native Method) ~[na:1.6.0_14]
>> at java.lang.ProcessImpl.<init>(ProcessImpl.java:81) ~[na:1.6.0_14]
>> at java.lang.ProcessImpl.start(ProcessImpl.java:30) ~[na:1.6.0_14]
>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:452) ~[na:1.6.0_14]
>> ... 6 common frames omitted
>> 3781 [Thread-24-split] ERROR backtype.storm.daemon.executor -
>>
>> On Mon, Jan 13, 2014 at 11:50 AM, Nathan Leung <nc...@gmail.com>wrote:
>>
>>> You can run maven on eclipse and package on windows. If you have python
>>> you can probably submit from a windows box to a linux nimbus (if you look
>>> at the storm file it's a python script) but I've never tried that before so
>>> I can't say for sure. If you decide to try this make sure you have the
>>> same version of jre/jdk installed on both windows and linux environments.
>>>
>>>
>>> On Sun, Jan 12, 2014 at 10:25 PM, dingdongchao <
>>> dingdongchaoxiao@gmail.com> wrote:
>>>
>>>> Hi
>>>>
>>>> “package my code” is must , because running your topology on
>>>> storm cluster needs this jar .
>>>>
>>>> ===========================================================================>
>>>>
>>>>
>>>> System.*setProperty*("storm.jar",
>>>> "E:/workspace/cdc-dp-data-storm-project/target/cdc-cdp.jar");
>>>>
>>>> Config config=*new* Config();
>>>>
>>>> config.setDebug(*false*);
>>>>
>>>> config.put(Config.*NIMBUS_HOST*,"cq01-crm-lin2rd52.vm");
>>>>
>>>> config.put(Config.*NIMBUS_THRIFT_PORT*,8889);
>>>>
>>>> config.setNumWorkers(2);
>>>>
>>>> StormSubmitter.*submitTopology*("cdp-reader-online", config,
>>>> builder.createTopology());
>>>>
>>>>
>>>>
>>>> Every time , I package my code into one jar location "E:/workspace/cdc-dp-data-storm-project/target/cdc-cdp.jar"
>>>> again , then run this codes to *submit the topology *
>>>>
>>>>
>>>>
>>>> Thx
>>>>
>>>>
>>>>
>>>> *发件人:* ch huang [mailto:justlooks@gmail.com]
>>>> *发送时间:* 2014年1月13日 11:02
>>>> *收件人:* user@storm.incubator.apache.org
>>>> *主题:* Re: how can i connect remote storm cluster and submit topology?
>>>>
>>>>
>>>>
>>>> now ,problem is i the storm cluster is ready , i get a linux box for
>>>> test ,change /root/.storm/storm.yaml to point to the cluster nimbus , and
>>>> submit the ExclamationTopology through the following command
>>>>
>>>>
>>>>
>>>> # storm-0.9.0-wip21/bin/storm jar
>>>> storm-starter-master/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar
>>>> storm.starter.ExclamationTopology exclamation-topology
>>>>
>>>> it's work fine ,but i do storm topology write on a windowsXP OS box
>>>> using eclipse as IDE,i do not know how to get my code runing and submit the
>>>> topology to my remote cluster,i do not want to package my code ,and upload
>>>> it to my linux box ,then run it
>>>>
>>>> On Mon, Jan 13, 2014 at 10:32 AM, Nathan Leung <nc...@gmail.com>
>>>> wrote:
>>>>
>>>> Assuming everything else is setup, configure your storm.yaml and then
>>>> just run the "storm jar" command. Assuming it's not, these two links are
>>>> good references:
>>>>
>>>>
>>>>
>>>> https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster
>>>>
>>>> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
>>>>
>>>>
>>>>
>>>> On Sun, Jan 12, 2014 at 9:12 PM, ch huang <ju...@gmail.com> wrote:
>>>>
>>>> hi,maillist:
>>>>
>>>> i create a new storm project in eclipse and now question is
>>>> how can i connect to remote storm cluster ,submit the topology?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>
Re: 答复: how can i connect remote storm cluster and submit topology?
Posted by Nathan Leung <nc...@gmail.com>.
multilang is trying to run "python" but your binary is under "python.exe"
On Mon, Jan 13, 2014 at 12:15 AM, ch huang <ju...@gmail.com> wrote:
> i get following error when i run the wordcounttopology,i already installed
> python
>
> C:\Documents and Settings\Administrator>python.exe -V
> Python 2.6.6
>
> 3781 [Thread-24-split] INFO backtype.storm.daemon.executor - Preparing
> bolt split:(6)
> 3781 [Thread-24-split] ERROR backtype.storm.util - Async loop died!
> java.lang.RuntimeException: Error when launching multilang subprocess
> at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:88)
> ~[storm-core-0.9.0.1.jar:na]
> at
> backtype.storm.daemon.executor$fn__3498$fn__3510.invoke(executor.clj:674)
> ~[storm-core-0.9.0.1.jar:na]
> at backtype.storm.util$async_loop$fn__444.invoke(util.clj:401)
> ~[storm-core-0.9.0.1.jar:na]
> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
> at java.lang.Thread.run(Thread.java:619) [na:1.6.0_14]
> Caused by: java.io.IOException: Cannot run program "python" (in directory
> "C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\40b2df00-5a53-4034-8c20-3f0a084bdd56\supervisor\stormdist\word-count-1-1389585368\resources"):
> CreateProcess error=2, ϵͳÕҲ»µ½ָ¶
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:459) ~[na:1.6.0_14]
> at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:33)
> ~[storm-core-0.9.0.1.jar:na]
> at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:85)
> ~[storm-core-0.9.0.1.jar:na]
> ... 4 common frames omitted
> Caused by: java.io.IOException: CreateProcess error=2, ϵͳÕҲ»µ½ָ¶
> at java.lang.ProcessImpl.create(Native Method) ~[na:1.6.0_14]
> at java.lang.ProcessImpl.<init>(ProcessImpl.java:81) ~[na:1.6.0_14]
> at java.lang.ProcessImpl.start(ProcessImpl.java:30) ~[na:1.6.0_14]
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:452) ~[na:1.6.0_14]
> ... 6 common frames omitted
> 3781 [Thread-24-split] ERROR backtype.storm.daemon.executor -
>
> On Mon, Jan 13, 2014 at 11:50 AM, Nathan Leung <nc...@gmail.com> wrote:
>
>> You can run maven on eclipse and package on windows. If you have python
>> you can probably submit from a windows box to a linux nimbus (if you look
>> at the storm file it's a python script) but I've never tried that before so
>> I can't say for sure. If you decide to try this make sure you have the
>> same version of jre/jdk installed on both windows and linux environments.
>>
>>
>> On Sun, Jan 12, 2014 at 10:25 PM, dingdongchao <
>> dingdongchaoxiao@gmail.com> wrote:
>>
>>> Hi
>>>
>>> “package my code” is must , because running your topology on
>>> storm cluster needs this jar .
>>>
>>> ===========================================================================>
>>>
>>>
>>> System.*setProperty*("storm.jar",
>>> "E:/workspace/cdc-dp-data-storm-project/target/cdc-cdp.jar");
>>>
>>> Config config=*new* Config();
>>>
>>> config.setDebug(*false*);
>>>
>>> config.put(Config.*NIMBUS_HOST*,"cq01-crm-lin2rd52.vm");
>>>
>>> config.put(Config.*NIMBUS_THRIFT_PORT*,8889);
>>>
>>> config.setNumWorkers(2);
>>>
>>> StormSubmitter.*submitTopology*("cdp-reader-online", config,
>>> builder.createTopology());
>>>
>>>
>>>
>>> Every time , I package my code into one jar location "E:/workspace/cdc-dp-data-storm-project/target/cdc-cdp.jar"
>>> again , then run this codes to *submit the topology *
>>>
>>>
>>>
>>> Thx
>>>
>>>
>>>
>>> *发件人:* ch huang [mailto:justlooks@gmail.com]
>>> *发送时间:* 2014年1月13日 11:02
>>> *收件人:* user@storm.incubator.apache.org
>>> *主题:* Re: how can i connect remote storm cluster and submit topology?
>>>
>>>
>>>
>>> now ,problem is i the storm cluster is ready , i get a linux box for
>>> test ,change /root/.storm/storm.yaml to point to the cluster nimbus , and
>>> submit the ExclamationTopology through the following command
>>>
>>>
>>>
>>> # storm-0.9.0-wip21/bin/storm jar
>>> storm-starter-master/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar
>>> storm.starter.ExclamationTopology exclamation-topology
>>>
>>> it's work fine ,but i do storm topology write on a windowsXP OS box
>>> using eclipse as IDE,i do not know how to get my code runing and submit the
>>> topology to my remote cluster,i do not want to package my code ,and upload
>>> it to my linux box ,then run it
>>>
>>> On Mon, Jan 13, 2014 at 10:32 AM, Nathan Leung <nc...@gmail.com>
>>> wrote:
>>>
>>> Assuming everything else is setup, configure your storm.yaml and then
>>> just run the "storm jar" command. Assuming it's not, these two links are
>>> good references:
>>>
>>>
>>>
>>> https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster
>>>
>>> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
>>>
>>>
>>>
>>> On Sun, Jan 12, 2014 at 9:12 PM, ch huang <ju...@gmail.com> wrote:
>>>
>>> hi,maillist:
>>>
>>> i create a new storm project in eclipse and now question is how
>>> can i connect to remote storm cluster ,submit the topology?
>>>
>>>
>>>
>>>
>>>
>>
>>
>
Re: 答复: how can i connect remote storm cluster and submit topology?
Posted by ch huang <ju...@gmail.com>.
i get following error when i run the wordcounttopology,i already installed
python
C:\Documents and Settings\Administrator>python.exe -V
Python 2.6.6
3781 [Thread-24-split] INFO backtype.storm.daemon.executor - Preparing
bolt split:(6)
3781 [Thread-24-split] ERROR backtype.storm.util - Async loop died!
java.lang.RuntimeException: Error when launching multilang subprocess
at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:88)
~[storm-core-0.9.0.1.jar:na]
at
backtype.storm.daemon.executor$fn__3498$fn__3510.invoke(executor.clj:674)
~[storm-core-0.9.0.1.jar:na]
at backtype.storm.util$async_loop$fn__444.invoke(util.clj:401)
~[storm-core-0.9.0.1.jar:na]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
at java.lang.Thread.run(Thread.java:619) [na:1.6.0_14]
Caused by: java.io.IOException: Cannot run program "python" (in directory
"C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\40b2df00-5a53-4034-8c20-3f0a084bdd56\supervisor\stormdist\word-count-1-1389585368\resources"):
CreateProcess error=2, ϵͳÕҲ»µ½ָ¶
at java.lang.ProcessBuilder.start(ProcessBuilder.java:459) ~[na:1.6.0_14]
at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:33)
~[storm-core-0.9.0.1.jar:na]
at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:85)
~[storm-core-0.9.0.1.jar:na]
... 4 common frames omitted
Caused by: java.io.IOException: CreateProcess error=2, ϵͳÕҲ»µ½ָ¶
at java.lang.ProcessImpl.create(Native Method) ~[na:1.6.0_14]
at java.lang.ProcessImpl.<init>(ProcessImpl.java:81) ~[na:1.6.0_14]
at java.lang.ProcessImpl.start(ProcessImpl.java:30) ~[na:1.6.0_14]
at java.lang.ProcessBuilder.start(ProcessBuilder.java:452) ~[na:1.6.0_14]
... 6 common frames omitted
3781 [Thread-24-split] ERROR backtype.storm.daemon.executor -
On Mon, Jan 13, 2014 at 11:50 AM, Nathan Leung <nc...@gmail.com> wrote:
> You can run maven on eclipse and package on windows. If you have python
> you can probably submit from a windows box to a linux nimbus (if you look
> at the storm file it's a python script) but I've never tried that before so
> I can't say for sure. If you decide to try this make sure you have the
> same version of jre/jdk installed on both windows and linux environments.
>
>
> On Sun, Jan 12, 2014 at 10:25 PM, dingdongchao <dingdongchaoxiao@gmail.com
> > wrote:
>
>> Hi
>>
>> “package my code” is must , because running your topology on
>> storm cluster needs this jar .
>>
>> ===========================================================================>
>>
>>
>> System.*setProperty*("storm.jar",
>> "E:/workspace/cdc-dp-data-storm-project/target/cdc-cdp.jar");
>>
>> Config config=*new* Config();
>>
>> config.setDebug(*false*);
>>
>> config.put(Config.*NIMBUS_HOST*,"cq01-crm-lin2rd52.vm");
>>
>> config.put(Config.*NIMBUS_THRIFT_PORT*,8889);
>>
>> config.setNumWorkers(2);
>>
>> StormSubmitter.*submitTopology*("cdp-reader-online", config,
>> builder.createTopology());
>>
>>
>>
>> Every time , I package my code into one jar location "E:/workspace/cdc-dp-data-storm-project/target/cdc-cdp.jar"
>> again , then run this codes to *submit the topology *
>>
>>
>>
>> Thx
>>
>>
>>
>> *发件人:* ch huang [mailto:justlooks@gmail.com]
>> *发送时间:* 2014年1月13日 11:02
>> *收件人:* user@storm.incubator.apache.org
>> *主题:* Re: how can i connect remote storm cluster and submit topology?
>>
>>
>>
>> now ,problem is i the storm cluster is ready , i get a linux box for test
>> ,change /root/.storm/storm.yaml to point to the cluster nimbus , and submit
>> the ExclamationTopology through the following command
>>
>>
>>
>> # storm-0.9.0-wip21/bin/storm jar
>> storm-starter-master/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar
>> storm.starter.ExclamationTopology exclamation-topology
>>
>> it's work fine ,but i do storm topology write on a windowsXP OS box using
>> eclipse as IDE,i do not know how to get my code runing and submit the
>> topology to my remote cluster,i do not want to package my code ,and upload
>> it to my linux box ,then run it
>>
>> On Mon, Jan 13, 2014 at 10:32 AM, Nathan Leung <nc...@gmail.com> wrote:
>>
>> Assuming everything else is setup, configure your storm.yaml and then
>> just run the "storm jar" command. Assuming it's not, these two links are
>> good references:
>>
>>
>>
>> https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster
>>
>> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
>>
>>
>>
>> On Sun, Jan 12, 2014 at 9:12 PM, ch huang <ju...@gmail.com> wrote:
>>
>> hi,maillist:
>>
>> i create a new storm project in eclipse and now question is how
>> can i connect to remote storm cluster ,submit the topology?
>>
>>
>>
>>
>>
>
>
Re: 答复: how can i connect remote storm cluster and submit topology?
Posted by Nathan Leung <nc...@gmail.com>.
You can run maven on eclipse and package on windows. If you have python
you can probably submit from a windows box to a linux nimbus (if you look
at the storm file it's a python script) but I've never tried that before so
I can't say for sure. If you decide to try this make sure you have the
same version of jre/jdk installed on both windows and linux environments.
On Sun, Jan 12, 2014 at 10:25 PM, dingdongchao
<di...@gmail.com>wrote:
> Hi
>
> “package my code” is must , because running your topology on
> storm cluster needs this jar .
>
> ===========================================================================>
>
>
> System.*setProperty*("storm.jar",
> "E:/workspace/cdc-dp-data-storm-project/target/cdc-cdp.jar");
>
> Config config=*new* Config();
>
> config.setDebug(*false*);
>
> config.put(Config.*NIMBUS_HOST*,"cq01-crm-lin2rd52.vm");
>
> config.put(Config.*NIMBUS_THRIFT_PORT*,8889);
>
> config.setNumWorkers(2);
>
> StormSubmitter.*submitTopology*("cdp-reader-online", config,
> builder.createTopology());
>
>
>
> Every time , I package my code into one jar location "E:/workspace/cdc-dp-data-storm-project/target/cdc-cdp.jar"
> again , then run this codes to *submit the topology *
>
>
>
> Thx
>
>
>
> *发件人:* ch huang [mailto:justlooks@gmail.com]
> *发送时间:* 2014年1月13日 11:02
> *收件人:* user@storm.incubator.apache.org
> *主题:* Re: how can i connect remote storm cluster and submit topology?
>
>
>
> now ,problem is i the storm cluster is ready , i get a linux box for test
> ,change /root/.storm/storm.yaml to point to the cluster nimbus , and submit
> the ExclamationTopology through the following command
>
>
>
> # storm-0.9.0-wip21/bin/storm jar
> storm-starter-master/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar
> storm.starter.ExclamationTopology exclamation-topology
>
> it's work fine ,but i do storm topology write on a windowsXP OS box using
> eclipse as IDE,i do not know how to get my code runing and submit the
> topology to my remote cluster,i do not want to package my code ,and upload
> it to my linux box ,then run it
>
> On Mon, Jan 13, 2014 at 10:32 AM, Nathan Leung <nc...@gmail.com> wrote:
>
> Assuming everything else is setup, configure your storm.yaml and then just
> run the "storm jar" command. Assuming it's not, these two links are good
> references:
>
>
>
> https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster
>
> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
>
>
>
> On Sun, Jan 12, 2014 at 9:12 PM, ch huang <ju...@gmail.com> wrote:
>
> hi,maillist:
>
> i create a new storm project in eclipse and now question is how
> can i connect to remote storm cluster ,submit the topology?
>
>
>
>
>
Re: 答复: how can i connect remote storm cluster and submit topology?
Posted by ch huang <ju...@gmail.com>.
yes,you are right
On Mon, Jan 13, 2014 at 11:25 AM, dingdongchao
<di...@gmail.com>wrote:
> Hi
>
> “package my code” is must , because running your topology on
> storm cluster needs this jar .
>
> ===========================================================================>
>
>
> System.*setProperty*("storm.jar",
> "E:/workspace/cdc-dp-data-storm-project/target/cdc-cdp.jar");
>
> Config config=*new* Config();
>
> config.setDebug(*false*);
>
> config.put(Config.*NIMBUS_HOST*,"cq01-crm-lin2rd52.vm");
>
> config.put(Config.*NIMBUS_THRIFT_PORT*,8889);
>
> config.setNumWorkers(2);
>
> StormSubmitter.*submitTopology*("cdp-reader-online", config,
> builder.createTopology());
>
>
>
> Every time , I package my code into one jar location "E:/workspace/cdc-dp-data-storm-project/target/cdc-cdp.jar"
> again , then run this codes to *submit the topology *
>
>
>
> Thx
>
>
>
> *发件人:* ch huang [mailto:justlooks@gmail.com]
> *发送时间:* 2014年1月13日 11:02
> *收件人:* user@storm.incubator.apache.org
> *主题:* Re: how can i connect remote storm cluster and submit topology?
>
>
>
> now ,problem is i the storm cluster is ready , i get a linux box for test
> ,change /root/.storm/storm.yaml to point to the cluster nimbus , and submit
> the ExclamationTopology through the following command
>
>
>
> # storm-0.9.0-wip21/bin/storm jar
> storm-starter-master/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar
> storm.starter.ExclamationTopology exclamation-topology
>
> it's work fine ,but i do storm topology write on a windowsXP OS box using
> eclipse as IDE,i do not know how to get my code runing and submit the
> topology to my remote cluster,i do not want to package my code ,and upload
> it to my linux box ,then run it
>
> On Mon, Jan 13, 2014 at 10:32 AM, Nathan Leung <nc...@gmail.com> wrote:
>
> Assuming everything else is setup, configure your storm.yaml and then just
> run the "storm jar" command. Assuming it's not, these two links are good
> references:
>
>
>
> https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster
>
> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
>
>
>
> On Sun, Jan 12, 2014 at 9:12 PM, ch huang <ju...@gmail.com> wrote:
>
> hi,maillist:
>
> i create a new storm project in eclipse and now question is how
> can i connect to remote storm cluster ,submit the topology?
>
>
>
>
>
答复: how can i connect remote storm cluster and submit topology?
Posted by dingdongchao <di...@gmail.com>.
Hi
“package my code” is must , because running your topology on
storm cluster needs this jar .
===========================================================================>
System.setProperty("storm.jar",
"E:/workspace/cdc-dp-data-storm-project/target/cdc-cdp.jar");
Config config=new Config();
config.setDebug(false);
config.put(Config.NIMBUS_HOST,"cq01-crm-lin2rd52.vm");
config.put(Config.NIMBUS_THRIFT_PORT,8889);
config.setNumWorkers(2);
StormSubmitter.submitTopology("cdp-reader-online", config,
builder.createTopology());
Every time , I package my code into one jar location
"E:/workspace/cdc-dp-data-storm-project/target/cdc-cdp.jar" again , then
run this codes to submit the topology
Thx
发件人: ch huang [mailto:justlooks@gmail.com]
发送时间: 2014年1月13日 11:02
收件人: user@storm.incubator.apache.org
主题: Re: how can i connect remote storm cluster and submit topology?
now ,problem is i the storm cluster is ready , i get a linux box for test
,change /root/.storm/storm.yaml to point to the cluster nimbus , and submit
the ExclamationTopology through the following command
# storm-0.9.0-wip21/bin/storm jar
storm-starter-master/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependenci
es.jar storm.starter.ExclamationTopology exclamation-topology
it's work fine ,but i do storm topology write on a windowsXP OS box using
eclipse as IDE,i do not know how to get my code runing and submit the
topology to my remote cluster,i do not want to package my code ,and upload
it to my linux box ,then run it
On Mon, Jan 13, 2014 at 10:32 AM, Nathan Leung <nc...@gmail.com> wrote:
Assuming everything else is setup, configure your storm.yaml and then just
run the "storm jar" command. Assuming it's not, these two links are good
references:
https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster
http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
On Sun, Jan 12, 2014 at 9:12 PM, ch huang <ju...@gmail.com> wrote:
hi,maillist:
i create a new storm project in eclipse and now question is how can
i connect to remote storm cluster ,submit the topology?
Re: how can i connect remote storm cluster and submit topology?
Posted by ch huang <ju...@gmail.com>.
now ,problem is i the storm cluster is ready , i get a linux box for test
,change /root/.storm/storm.yaml to point to the cluster nimbus , and submit
the ExclamationTopology through the following command
# storm-0.9.0-wip21/bin/storm jar
storm-starter-master/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar
storm.starter.ExclamationTopology exclamation-topology
it's work fine ,but i do storm topology write on a windowsXP OS box using
eclipse as IDE,i do not know how to get my code runing and submit the
topology to my remote cluster,i do not want to package my code ,and upload
it to my linux box ,then run it
On Mon, Jan 13, 2014 at 10:32 AM, Nathan Leung <nc...@gmail.com> wrote:
> Assuming everything else is setup, configure your storm.yaml and then just
> run the "storm jar" command. Assuming it's not, these two links are good
> references:
>
> https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster
> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
>
>
> On Sun, Jan 12, 2014 at 9:12 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi,maillist:
>> i create a new storm project in eclipse and now question is how
>> can i connect to remote storm cluster ,submit the topology?
>>
>
>
Re: how can i connect remote storm cluster and submit topology?
Posted by Nathan Leung <nc...@gmail.com>.
Assuming everything else is setup, configure your storm.yaml and then just
run the "storm jar" command. Assuming it's not, these two links are good
references:
https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster
http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
On Sun, Jan 12, 2014 at 9:12 PM, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i create a new storm project in eclipse and now question is how
> can i connect to remote storm cluster ,submit the topology?
>