You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Yaron Rosenbaum <ya...@whatson-social.com> on 2014/08/15 18:18:28 UTC
Storm on top of Mesos running on Docker
Hi
I've set up a Docker based Mesos setup, including a mesos-storm framework installed (plus chronos, marathon, ..).
You can find it here: https://github.com/yaronr/docker-mesos
It works fine, storm UI is accessible.
I am able to deploy topologies, and storm-ui reflects these deployments correctly.
However, from the storm logs, it appears that the executors are not able to communicate with nimbus. Or they don't exist (mesos is responsible for spawning 'supervisors' or executors).
2014-08-15 16:03:11 b.s.d.nimbus [INFO] Executor wordcount-1-1408118461:[3 3] not alive
I'm suspecting this has something to do with some ports / links that I have to define, but I have nothing to go on but this wild guess.
Any help would be appreciated!
(Y)
Re: Storm on top of Mesos running on Docker
Posted by Yaron Rosenbaum <ya...@gmail.com>.
I am trying to do the same.
I started this project:
https://github.com/yaronr/docker-mesos
Docker is much more efficient compared to VMs, which allows you to run everything on a single computer with few resources.
There is still one bug with relation to storm, if you manage to solve it - we all win :)
(Y)
On Aug 22, 2014, at 2:41 PM, M Tarkeshwar Rao <m....@ericsson.com> wrote:
> Hi,
>
> Sorry but I need some information about this?
>
> We have a application in which we are using storm as execution engine and hdfs,hbase,redis as distributed db.
> I want to test complete application and debug on my local laptop.
>
> Can i do it? can i launch Storm UI locally?
>
> is it possible to launch nimbus,supervisonrs and all the workers on my laptop itself?
>
> Regards
> <image001.gif>
> Tarkeshwar
>
>
> From: Ankit Toshniwal [mailto:ankitoshniwal@gmail.com]
> Sent: 20 August 2014 11:14
> To: user@storm.incubator.apache.org
> Subject: Re: Storm on top of Mesos running on Docker
>
> On the storm UI are you seeing your topology getting the required number of workers and is the topology emitting any tuples?
>
> Also Storm UI should tell you if any of the supervisors were started and if they were on which machine and you can ssh to that machine to look at supervisor logs.
>
> But the general behavior is that once workers start up they will report their heartbeats into ZK, and Nimbus will pull this data from ZK to confirm which worker is up and which is down.
>
> Hope this helps!
>
> Ankit
>
>
> On Fri, Aug 15, 2014 at 10:15 PM, Yaron Rosenbaum <ya...@gmail.com> wrote:
> That's the thing - I don't know.
> Mesos is supposed to start the workers, so either they don't start, or they start - but they can't communicate with Nimbus.
> Or do they need to communicate with Zookeeper directly?
>
> This is a slightly more complicated setup than usual.
>
> (Y)
>
> On Aug 15, 2014, at 8:07 PM, Ankit Toshniwal <an...@gmail.com> wrote:
>
>
> Are the workers starting fine? I have seen this in the past when the workers were not reporting their heartbeats leading to this message in nimbus.log
>
> Ankit
>
>
> On Fri, Aug 15, 2014 at 9:18 AM, Yaron Rosenbaum <ya...@whatson-social.com> wrote:
> Hi
>
> I've set up a Docker based Mesos setup, including a mesos-storm framework installed (plus chronos, marathon, ..).
> You can find it here: https://github.com/yaronr/docker-mesos
>
> It works fine, storm UI is accessible.
> I am able to deploy topologies, and storm-ui reflects these deployments correctly.
> However, from the storm logs, it appears that the executors are not able to communicate with nimbus. Or they don't exist (mesos is responsible for spawning 'supervisors' or executors).
> 2014-08-15 16:03:11 b.s.d.nimbus [INFO] Executor wordcount-1-1408118461:[3 3] not alive
>
> I'm suspecting this has something to do with some ports / links that I have to define, but I have nothing to go on but this wild guess.
>
> Any help would be appreciated!
>
> (Y)
RE: Storm on top of Mesos running on Docker
Posted by M Tarkeshwar Rao <m....@ericsson.com>.
Hi,
Sorry but I need some information about this?
We have a application in which we are using storm as execution engine and hdfs,hbase,redis as distributed db.
I want to test complete application and debug on my local laptop.
Can i do it? can i launch Storm UI locally?
is it possible to launch nimbus,supervisonrs and all the workers on my laptop itself?
Regards
Tarkeshwar
From: Ankit Toshniwal [mailto:ankitoshniwal@gmail.com]
Sent: 20 August 2014 11:14
To: user@storm.incubator.apache.org
Subject: Re: Storm on top of Mesos running on Docker
On the storm UI are you seeing your topology getting the required number of workers and is the topology emitting any tuples?
Also Storm UI should tell you if any of the supervisors were started and if they were on which machine and you can ssh to that machine to look at supervisor logs.
But the general behavior is that once workers start up they will report their heartbeats into ZK, and Nimbus will pull this data from ZK to confirm which worker is up and which is down.
Hope this helps!
Ankit
On Fri, Aug 15, 2014 at 10:15 PM, Yaron Rosenbaum <ya...@gmail.com>> wrote:
That's the thing - I don't know.
Mesos is supposed to start the workers, so either they don't start, or they start - but they can't communicate with Nimbus.
Or do they need to communicate with Zookeeper directly?
This is a slightly more complicated setup than usual.
(Y)
On Aug 15, 2014, at 8:07 PM, Ankit Toshniwal <an...@gmail.com>> wrote:
Are the workers starting fine? I have seen this in the past when the workers were not reporting their heartbeats leading to this message in nimbus.log
Ankit
On Fri, Aug 15, 2014 at 9:18 AM, Yaron Rosenbaum <ya...@whatson-social.com>> wrote:
Hi
I've set up a Docker based Mesos setup, including a mesos-storm framework installed (plus chronos, marathon, ..).
You can find it here: https://github.com/yaronr/docker-mesos
It works fine, storm UI is accessible.
I am able to deploy topologies, and storm-ui reflects these deployments correctly.
However, from the storm logs, it appears that the executors are not able to communicate with nimbus. Or they don't exist (mesos is responsible for spawning 'supervisors' or executors).
2014-08-15 16:03:11 b.s.d.nimbus [INFO] Executor wordcount-1-1408118461:[3 3] not alive
I'm suspecting this has something to do with some ports / links that I have to define, but I have nothing to go on but this wild guess.
Any help would be appreciated!
(Y)
Re: Storm on top of Mesos running on Docker
Posted by Yaron Rosenbaum <ya...@gmail.com>.
stderr on the slave:
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0818 19:06:55.033699 22 fetcher.cpp:73] Fetching URI 'http://downloads.mesosphere.io/storm/storm-mesos-0.9.tgz'
I0818 19:06:55.033994 22 fetcher.cpp:123] Downloading 'http://downloads.mesosphere.io/storm/storm-mesos-0.9.tgz' to '/tmp/mesos/slaves/20140818-190538-2466255276-5050-11-0/frameworks/20140818-190538-2466255276-5050-11-0002/executors/wordcount-1-1408388814/runs/69496890-fc18-43f3-be87-198bceba7226/storm-mesos-0.9.tgz'
I0818 19:07:11.567514 22 fetcher.cpp:61] Extracted resource '/tmp/mesos/slaves/20140818-190538-2466255276-5050-11-0/frameworks/20140818-190538-2466255276-5050-11-0002/executors/wordcount-1-1408388814/runs/69496890-fc18-43f3-be87-198bceba7226/storm-mesos-0.9.tgz' into '/tmp/mesos/slaves/20140818-190538-2466255276-5050-11-0/frameworks/20140818-190538-2466255276-5050-11-0002/executors/wordcount-1-1408388814/runs/69496890-fc18-43f3-be87-198bceba7226'
--2014-08-18 19:07:12-- http://master:35468/conf/storm.yaml
Resolving master (master)... 172.17.0.147
Connecting to master (master)|172.17.0.147|:35468... connected.
HTTP request sent, awaiting response... 404 Not Found
2014-08-18 19:07:12 ERROR 404: Not Found.
root@master:/# cat /var/log/supervisor/mesos-master-stderr.log
...
I0818 19:11:10.456274 19 master.cpp:2704] Executor wordcount-1-1408388814 of framework 20140818-190538-2466255276-5050-11-0002 on slave 20140818-190538-2466255276-5050-11-0 at slave(1)@172.17.0.149:5051 (slave) has exited with status 8
I0818 19:11:10.457824 19 master.cpp:2628] Status update TASK_LOST (UUID: ddd2a5c6-39d6-4450-824b-2ddc5b39869b) for task slave-31000 of framework 20140818-190538-2466255276-5050-11-0002 from slave 20140818-190538-2466255276-5050-11-0 at slave(1)@172.17.0.149:5051 (slave)
I0818 19:11:10.457898 19 master.hpp:673] Removing task slave-31000 with resources cpus(*):1; mem(*):1000; ports(*):[31000-31000] on slave 20140818-190538-2466255276-5050
root@master:/# cat /var/log/supervisor/nimbus-stderr.log
I0818 19:06:23.683955 190 sched.cpp:126] Version: 0.19.1
2014-08-18 19:06:23,684:26(0x7f3575014700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5
2014-08-18 19:06:23,684:26(0x7f3575014700):ZOO_INFO@log_env@716: Client environment:host.name=master
2014-08-18 19:06:23,684:26(0x7f3575014700):ZOO_INFO@log_env@723: Client environment:os.name=Linux
2014-08-18 19:06:23,684:26(0x7f3575014700):ZOO_INFO@log_env@724: Client environment:os.arch=3.15.3-tinycore64
2014-08-18 19:06:23,684:26(0x7f3575014700):ZOO_INFO@log_env@725: Client environment:os.version=#1 SMP Fri Aug 15 09:11:44 UTC 2014
2014-08-18 19:06:23,684:26(0x7f3575014700):ZOO_INFO@log_env@733: Client environment:user.name=(null)
2014-08-18 19:06:23,684:26(0x7f3575014700):ZOO_INFO@log_env@741: Client environment:user.home=/root
2014-08-18 19:06:23,685:26(0x7f3575014700):ZOO_INFO@log_env@753: Client environment:user.dir=/
2014-08-18 19:06:23,685:26(0x7f3575014700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=zookeeper:2181 sessionTimeout=10000 watcher=0x7f3576f9cf80 sessionId=0 sessionPasswd=<null> context=0x7f3554000e00 flags=0
2014-08-18 19:06:23,712:26(0x7f3573010700):ZOO_INFO@check_events@1703: initiated connection to server [172.17.0.145:2181]
2014-08-18 19:06:23,724:26(0x7f3573010700):ZOO_INFO@check_events@1750: session establishment complete on server [172.17.0.145:2181], sessionId=0x147ea82a658000c, negotiated timeout=10000
I0818 19:06:23.729141 242 group.cpp:310] Group process ((3)@172.17.0.147:49673) connected to ZooKeeper
I0818 19:06:23.729308 242 group.cpp:784] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I0818 19:06:23.729367 242 group.cpp:382] Trying to create path '/mesos' in ZooKeeper
I0818 19:06:23.745023 242 detector.cpp:135] Detected a new leader: (id='1')
I0818 19:06:23.745312 242 group.cpp:655] Trying to get '/mesos/info_0000000001' in ZooKeeper
I0818 19:06:23.752063 242 detector.cpp:377] A new leading master (UPID=master@172.17.0.147:5050) is detected
I0818 19:06:23.752250 242 sched.cpp:222] New master detected at master@172.17.0.147:5050
I0818 19:06:23.752893 242 sched.cpp:230] No credentials provided. Attempting to register without authentication
I0818 19:06:23.755734 242 sched.cpp:397] Framework registered with 20140818-190538-2466255276-5050-11-0002
W0818 19:06:54.991662 245 sched.cpp:901] Attempting to launch task slave-31001 with an unknown offer 20140818-190538-2466255276-5050-11-18
2014-08-18 19:09:10,656:26(0x7f3573010700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 28ms
W0818 19:10:58.976002 248 sched.cpp:901] Attempting to launch task slave-31001 with an unknown offer 20140818-190538-2466255276-5050-11-57
2014-08-18 19:11:40,927:26(0x7f3573010700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 107ms
2014-08-18 19:12:07,700:26(0x7f3573010700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 72ms
2014-08-18 19:15:54,659:26(0x7f3573010700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 20ms
W0818 19:16:41.581099 241 sched.cpp:901] Attempting to launch task slave-31001 with an unknown offer 20140818-190538-2466255276-5050-11-259
W0818 19:19:52.968051 242 sched.cpp:901] Attempting to launch task slave-31001 with an unknown offer 20140818-190538-2466255276-5050-11-367
2014-08-18 19:20:14,970:26(0x7f3573010700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 24ms
2014-08-18 19:20:31,676:26(0x7f3573010700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 13ms
2014-08-18 19:20:48,375:26(0x7f3573010700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 12ms
W0818 19:22:33.935534 244 sched.cpp:901] Attempting to launch task slave-31001 with an unknown offer 20140818-190538-2466255276-5050-11-395
(Y)
On Aug 20, 2014, at 8:43 AM, Ankit Toshniwal <an...@gmail.com> wrote:
> On the storm UI are you seeing your topology getting the required number of workers and is the topology emitting any tuples?
>
> Also Storm UI should tell you if any of the supervisors were started and if they were on which machine and you can ssh to that machine to look at supervisor logs.
>
> But the general behavior is that once workers start up they will report their heartbeats into ZK, and Nimbus will pull this data from ZK to confirm which worker is up and which is down.
>
> Hope this helps!
>
> Ankit
>
>
> On Fri, Aug 15, 2014 at 10:15 PM, Yaron Rosenbaum <ya...@gmail.com> wrote:
> That's the thing - I don't know.
> Mesos is supposed to start the workers, so either they don't start, or they start - but they can't communicate with Nimbus.
> Or do they need to communicate with Zookeeper directly?
>
> This is a slightly more complicated setup than usual.
>
> (Y)
>
> On Aug 15, 2014, at 8:07 PM, Ankit Toshniwal <an...@gmail.com> wrote:
>
>> Are the workers starting fine? I have seen this in the past when the workers were not reporting their heartbeats leading to this message in nimbus.log
>>
>> Ankit
>>
>>
>> On Fri, Aug 15, 2014 at 9:18 AM, Yaron Rosenbaum <ya...@whatson-social.com> wrote:
>> Hi
>>
>> I've set up a Docker based Mesos setup, including a mesos-storm framework installed (plus chronos, marathon, ..).
>> You can find it here: https://github.com/yaronr/docker-mesos
>>
>> It works fine, storm UI is accessible.
>> I am able to deploy topologies, and storm-ui reflects these deployments correctly.
>> However, from the storm logs, it appears that the executors are not able to communicate with nimbus. Or they don't exist (mesos is responsible for spawning 'supervisors' or executors).
>> 2014-08-15 16:03:11 b.s.d.nimbus [INFO] Executor wordcount-1-1408118461:[3 3] not alive
>>
>> I'm suspecting this has something to do with some ports / links that I have to define, but I have nothing to go on but this wild guess.
>>
>> Any help would be appreciated!
>>
>> (Y)
>>
>>
>
>
Re: Storm on top of Mesos running on Docker
Posted by Ankit Toshniwal <an...@gmail.com>.
On the storm UI are you seeing your topology getting the required number of
workers and is the topology emitting any tuples?
Also Storm UI should tell you if any of the supervisors were started and if
they were on which machine and you can ssh to that machine to look at
supervisor logs.
But the general behavior is that once workers start up they will report
their heartbeats into ZK, and Nimbus will pull this data from ZK to confirm
which worker is up and which is down.
Hope this helps!
Ankit
On Fri, Aug 15, 2014 at 10:15 PM, Yaron Rosenbaum <yaron.rosenbaum@gmail.com
> wrote:
> That's the thing - I don't know.
> Mesos is supposed to start the workers, so either they don't start, or
> they start - but they can't communicate with Nimbus.
> Or do they need to communicate with Zookeeper directly?
>
> This is a slightly more complicated setup than usual.
>
> (Y)
>
> On Aug 15, 2014, at 8:07 PM, Ankit Toshniwal <an...@gmail.com>
> wrote:
>
> Are the workers starting fine? I have seen this in the past when the
> workers were not reporting their heartbeats leading to this message in
> nimbus.log
>
> Ankit
>
>
> On Fri, Aug 15, 2014 at 9:18 AM, Yaron Rosenbaum <yaron@whatson-social.com
> > wrote:
>
>> Hi
>>
>> I've set up a Docker based Mesos setup, including a mesos-storm framework
>> installed (plus chronos, marathon, ..).
>> You can find it here: https://github.com/yaronr/docker-mesos
>>
>> It works fine, storm UI is accessible.
>> I am able to deploy topologies, and storm-ui reflects these deployments
>> correctly.
>> However, from the storm logs, it appears that the executors are not able
>> to communicate with nimbus. Or they don't exist (mesos is responsible for
>> spawning 'supervisors' or executors).
>> 2014-08-15 16:03:11 b.s.d.nimbus [INFO] Executor
>> wordcount-1-1408118461:[3 3] not alive
>>
>> I'm suspecting this has something to do with some ports / links that I
>> have to define, but I have nothing to go on but this wild guess.
>>
>> Any help would be appreciated!
>>
>> (Y)
>>
>>
>
>
Re: Storm on top of Mesos running on Docker
Posted by Yaron Rosenbaum <ya...@gmail.com>.
That's the thing - I don't know.
Mesos is supposed to start the workers, so either they don't start, or they start - but they can't communicate with Nimbus.
Or do they need to communicate with Zookeeper directly?
This is a slightly more complicated setup than usual.
(Y)
On Aug 15, 2014, at 8:07 PM, Ankit Toshniwal <an...@gmail.com> wrote:
> Are the workers starting fine? I have seen this in the past when the workers were not reporting their heartbeats leading to this message in nimbus.log
>
> Ankit
>
>
> On Fri, Aug 15, 2014 at 9:18 AM, Yaron Rosenbaum <ya...@whatson-social.com> wrote:
> Hi
>
> I've set up a Docker based Mesos setup, including a mesos-storm framework installed (plus chronos, marathon, ..).
> You can find it here: https://github.com/yaronr/docker-mesos
>
> It works fine, storm UI is accessible.
> I am able to deploy topologies, and storm-ui reflects these deployments correctly.
> However, from the storm logs, it appears that the executors are not able to communicate with nimbus. Or they don't exist (mesos is responsible for spawning 'supervisors' or executors).
> 2014-08-15 16:03:11 b.s.d.nimbus [INFO] Executor wordcount-1-1408118461:[3 3] not alive
>
> I'm suspecting this has something to do with some ports / links that I have to define, but I have nothing to go on but this wild guess.
>
> Any help would be appreciated!
>
> (Y)
>
>
Re: Storm on top of Mesos running on Docker
Posted by Ankit Toshniwal <an...@gmail.com>.
Are the workers starting fine? I have seen this in the past when the
workers were not reporting their heartbeats leading to this message in
nimbus.log
Ankit
On Fri, Aug 15, 2014 at 9:18 AM, Yaron Rosenbaum <ya...@whatson-social.com>
wrote:
> Hi
>
> I've set up a Docker based Mesos setup, including a mesos-storm framework
> installed (plus chronos, marathon, ..).
> You can find it here: https://github.com/yaronr/docker-mesos
>
> It works fine, storm UI is accessible.
> I am able to deploy topologies, and storm-ui reflects these deployments
> correctly.
> However, from the storm logs, it appears that the executors are not able
> to communicate with nimbus. Or they don't exist (mesos is responsible for
> spawning 'supervisors' or executors).
> 2014-08-15 16:03:11 b.s.d.nimbus [INFO] Executor wordcount-1-1408118461:[3
> 3] not alive
>
> I'm suspecting this has something to do with some ports / links that I
> have to define, but I have nothing to go on but this wild guess.
>
> Any help would be appreciated!
>
> (Y)
>
>