You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Luyi Wang <wa...@gmail.com> on 2014/09/17 19:43:54 UTC
TASK_LOST on storm task
Have anyone experience TASK_LOST status for storm tasks on mesos.
I checked the stderr. Everything seems normal.
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0917 00:21:36.164840 4831 fetcher.cpp:76] Fetching URI 'hdfs://
192.168.123.27/storm-mesos-0.9.2-incubating.tgz'
I0917 00:21:36.165225 4831 fetcher.cpp:105] Downloading resource from
'hdfs://192.168.123.27/storm-mesos-0.9.2-incubating.tgz' to
'/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-3/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-1410913050/runs/53d06991-cb84-49d3-a530-f83efcf339e9/storm-mesos-0.9.2-incubating.tgz'
I0917 00:21:52.202791 4831 fetcher.cpp:64] Extracted resource
'/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-3/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-1410913050/runs/53d06991-cb84-49d3-a530-f83efcf339e9/storm-mesos-0.9.2-incubating.tgz'
into
'/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-3/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-1410913050/runs/53d06991-cb84-49d3-a530-f83efcf339e9'
I0917 00:21:52.206581 4831 fetcher.cpp:76] Fetching URI '
http://mesos:45579/conf/storm.yaml'
I0917 00:21:52.206626 4831 fetcher.cpp:126] Downloading '
http://mesos:45579/conf/storm.yaml' to
'/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-3/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-/runs/53d06991-cb84-49d3-a530-f83efcf339e9/storm.yaml'
I0917 00:22:01.829298 4984 exec.cpp:132] Version: 0.21.0
I0917 00:22:01.832185 5006 exec.cpp:206] Executor registered on slave
20140915-185627-326871232-5050-8074-3
I also checked the mesos info log. Here is what it logged.
I0917 00:19:59.020755 5145 master.cpp:3261] Executor
production-topology-1-1410913050 of framework
20140915-230424-326871232-5050-13574-0000 on slave
20140915-185627-326871232-5050-8074-2 at slave(1)@192.168.123.29:5051 (
dev10-cdh5-03.int.dev10.smcl.pure-breeze.com) exited with status 0
Any idea on this?
-Luyi.
Re: TASK_LOST on storm task
Posted by Luyi Wang <wa...@gmail.com>.
Nice to see this update. I think this serves the purpose. Deployment time
update should be enough. In my environment now, I tweak the conf and put
them on hdfs, everytime, executor pulls down and spins with my customized
conf.
I think the issue is solved now. Thanks a lot!
-Luyi.
On Sat, Sep 20, 2014 at 12:22 PM, James DeFelice <ja...@gmail.com>
wrote:
> correction: in my setup I changed the executor URI to point to
> http://MY_NIMBUS_SERVER_ADDR:MY_FILESERVER_PORT/conf/MY_
> CUSTOM_MESOS_STORM_SPIN.
>
> On Sat, Sep 20, 2014 at 3:21 PM, James DeFelice <ja...@gmail.com>
> wrote:
>
>> Right, so in my setup I changed the executor URI to point to
>> http://localhost:MY_FILESERVER_PORT/conf/MY_CUSTOM_MESOS_STORM_SPIN.
>> It was possible to do that because MY_FILESERVER_PORT is predictable when
>> you set it via the "nimbus.fileserver.port" property.
>>
>> So in my environment I customize the logback configuration at framework
>> deploy time, repack the storm distro with the updated config and store the
>> distro it in conf/, the YAML is configured as mentioned above. This
>> facilitates the storm supervisors pulling down the storm spin with the
>> customized logback config.
>>
>> If you don't need something so dynamic then you could just tweak the
>> mesos/storm tarball once (with your updated configs) and push it to a local
>> web server somewhere, then update your executor URI to point to that.
>>
>>
>> On Fri, Sep 19, 2014 at 6:23 PM, Luyi Wang <wa...@gmail.com>
>> wrote:
>>
>>> This is a little bit different from what I asked but might on the same
>>> page.
>>>
>>> I am asking if it should use the CONF_EXECUTOR_URI to setup the
>>> executor environment.
>>>
>>>
>>> https://github.com/jdef/storm/blob/5e18be19574ca3e09b0068033558f6e429099e65/src/storm/mesos/MesosNimbus.java#L441
>>>
>>> But still thanks for giving me insight of the implementation.
>>>
>>> Thanks.
>>>
>>>
>>>
>>>
>>> -Luyi.
>>>
>>>
>>>
>>>
>>> On Fri, Sep 19, 2014 at 2:12 PM, James DeFelice <
>>> james.defelice@gmail.com> wrote:
>>>
>>>> https://github.com/mesos/storm/pull/11
>>>>
>>>> Looks like some cleanup has been requested... but should work as-is.
>>>>
>>>> On Fri, Sep 19, 2014 at 1:57 AM, Luyi Wang <wa...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks, james. I will pull the latest change and see what you
>>>>> committed. Thanks.
>>>>> On Sep 18, 2014 8:21 PM, "James DeFelice" <ja...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I submitted a pull request that facilitates what you're asking for.
>>>>>> It lets you specify a port number for the built in file server on nimbus.
>>>>>> Once you have a predictable uri for that built in server you can rebuild
>>>>>> the storm tarball with whatever config you want your executors to have and
>>>>>> throw it in conf/ so nimbus can serve it up. I've done exactly this and
>>>>>> it's been working great for us.
>>>>>>
>>>>>> --sent from my phone
>>>>>> On Sep 18, 2014 6:06 PM, "Luyi Wang" <wa...@gmail.com> wrote:
>>>>>>
>>>>>>> Well. After investigating the problem. It turns out to be a setting
>>>>>>> problem.
>>>>>>>
>>>>>>> My old storm task never ran correctly as it kept trying to connect a
>>>>>>> wrong zookeeper server.
>>>>>>>
>>>>>>> The way I started the mesos is following. I use zookeeper to store
>>>>>>> configuration but this zookeeper is embedded and running standalone on
>>>>>>> master node(192.168.1.11).
>>>>>>>
>>>>>>> nohup sudo /home/ubuntu/mesos/build/bin/mesos-master.sh
>>>>>>> --work_dir=/var/lib/mesos --zk=zk://0.0.0.0:2181/mesos --quorum=1
>>>>>>> --log_dir=/var/log/mesos </dev/null >/dev/null 2>&1 &
>>>>>>>
>>>>>>>
>>>>>>> And I started the slave using following command.
>>>>>>>
>>>>>>> nohup sudo /home/ubuntu/mesos/build/bin/mesos-slave.sh --master=zk://
>>>>>>> 192.168.123.19:2181/mesos --log_dir=/var/log/mesos </dev/null
>>>>>>> >/dev/null 2>&1 &
>>>>>>>
>>>>>>>
>>>>>>> Under this situation, everything looks fine.
>>>>>>>
>>>>>>> To set up the storm framework, I change the storm.yaml in the conf
>>>>>>> folder.
>>>>>>>
>>>>>>> mesos.master.url: "zk://192.168.123.19:2181/mesos"
>>>>>>> storm.zookeeper.servers:
>>>>>>> - "localhost"
>>>>>>> nimbus.host: "localhost"
>>>>>>>
>>>>>>> and running "storm-mesos nimbus" and "storm ui".
>>>>>>>
>>>>>>> The problem raised here in this configuration.For every task,
>>>>>>> storm-mesos created an executor with storm-mesos environment by
>>>>>>> downloading the full tar ball from either http from memosphere or hdfs
>>>>>>> which includes configuration file may or may not as same as the one using
>>>>>>> in the master node. In my case, all executor using the above
>>>>>>> configuration. The new created executor would fetch from zookeeper server
>>>>>>> to but here what it tried to talked with is still "localhost". The
>>>>>>> zookeeper server exists on the slave but never used for mesos, so the task
>>>>>>> were marked as LOST. To make it work,In my case, the set up should be like
>>>>>>> this.( I assume I also need to change the nimbus host to the master node").
>>>>>>>
>>>>>>> mesos.master.url: "zk://192.168.123.19:2181/mesos"
>>>>>>> storm.zookeeper.servers:
>>>>>>> - "192.168.123.19"
>>>>>>> nimbus.host: "192.168.123.19"
>>>>>>>
>>>>>>>
>>>>>>> After making this change, everything works fine now.
>>>>>>> I hope this would help people having the same issues.
>>>>>>>
>>>>>>> Meanwhile. In my opinion, they way downloading whole tarball with
>>>>>>> fixed configuration from somewhere should be avoided or improved.
>>>>>>>
>>>>>>> Probably worth to discuss.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -Luyi.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Sep 18, 2014 at 11:36 AM, Luyi Wang <wa...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I attached nimbus.log and supervisor.log for your reference
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Sep 17, 2014 at 5:30 PM, Benjamin Mahler <
>>>>>>>> benjamin.mahler@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> logs
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -Luyi.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>
>>>>
>>>> --
>>>> James DeFelice
>>>> 585.241.9488 (voice)
>>>> 650.649.6071 (fax)
>>>>
>>>
>>>
>>
>>
>> --
>> James DeFelice
>> 585.241.9488 (voice)
>> 650.649.6071 (fax)
>>
>
>
>
> --
> James DeFelice
> 585.241.9488 (voice)
> 650.649.6071 (fax)
>
Re: TASK_LOST on storm task
Posted by James DeFelice <ja...@gmail.com>.
correction: in my setup I changed the executor URI to point to
http://MY_NIMBUS_SERVER_ADDR:MY_FILESERVER_PORT/conf/MY_
CUSTOM_MESOS_STORM_SPIN.
On Sat, Sep 20, 2014 at 3:21 PM, James DeFelice <ja...@gmail.com>
wrote:
> Right, so in my setup I changed the executor URI to point to
> http://localhost:MY_FILESERVER_PORT/conf/MY_CUSTOM_MESOS_STORM_SPIN.
> It was possible to do that because MY_FILESERVER_PORT is predictable when
> you set it via the "nimbus.fileserver.port" property.
>
> So in my environment I customize the logback configuration at framework
> deploy time, repack the storm distro with the updated config and store the
> distro it in conf/, the YAML is configured as mentioned above. This
> facilitates the storm supervisors pulling down the storm spin with the
> customized logback config.
>
> If you don't need something so dynamic then you could just tweak the
> mesos/storm tarball once (with your updated configs) and push it to a local
> web server somewhere, then update your executor URI to point to that.
>
>
> On Fri, Sep 19, 2014 at 6:23 PM, Luyi Wang <wa...@gmail.com> wrote:
>
>> This is a little bit different from what I asked but might on the same
>> page.
>>
>> I am asking if it should use the CONF_EXECUTOR_URI to setup the executor
>> environment.
>>
>>
>> https://github.com/jdef/storm/blob/5e18be19574ca3e09b0068033558f6e429099e65/src/storm/mesos/MesosNimbus.java#L441
>>
>> But still thanks for giving me insight of the implementation.
>>
>> Thanks.
>>
>>
>>
>>
>> -Luyi.
>>
>>
>>
>>
>> On Fri, Sep 19, 2014 at 2:12 PM, James DeFelice <james.defelice@gmail.com
>> > wrote:
>>
>>> https://github.com/mesos/storm/pull/11
>>>
>>> Looks like some cleanup has been requested... but should work as-is.
>>>
>>> On Fri, Sep 19, 2014 at 1:57 AM, Luyi Wang <wa...@gmail.com>
>>> wrote:
>>>
>>>> Thanks, james. I will pull the latest change and see what you
>>>> committed. Thanks.
>>>> On Sep 18, 2014 8:21 PM, "James DeFelice" <ja...@gmail.com>
>>>> wrote:
>>>>
>>>>> I submitted a pull request that facilitates what you're asking for. It
>>>>> lets you specify a port number for the built in file server on nimbus. Once
>>>>> you have a predictable uri for that built in server you can rebuild the
>>>>> storm tarball with whatever config you want your executors to have and
>>>>> throw it in conf/ so nimbus can serve it up. I've done exactly this and
>>>>> it's been working great for us.
>>>>>
>>>>> --sent from my phone
>>>>> On Sep 18, 2014 6:06 PM, "Luyi Wang" <wa...@gmail.com> wrote:
>>>>>
>>>>>> Well. After investigating the problem. It turns out to be a setting
>>>>>> problem.
>>>>>>
>>>>>> My old storm task never ran correctly as it kept trying to connect a
>>>>>> wrong zookeeper server.
>>>>>>
>>>>>> The way I started the mesos is following. I use zookeeper to store
>>>>>> configuration but this zookeeper is embedded and running standalone on
>>>>>> master node(192.168.1.11).
>>>>>>
>>>>>> nohup sudo /home/ubuntu/mesos/build/bin/mesos-master.sh
>>>>>> --work_dir=/var/lib/mesos --zk=zk://0.0.0.0:2181/mesos --quorum=1
>>>>>> --log_dir=/var/log/mesos </dev/null >/dev/null 2>&1 &
>>>>>>
>>>>>>
>>>>>> And I started the slave using following command.
>>>>>>
>>>>>> nohup sudo /home/ubuntu/mesos/build/bin/mesos-slave.sh --master=zk://
>>>>>> 192.168.123.19:2181/mesos --log_dir=/var/log/mesos </dev/null
>>>>>> >/dev/null 2>&1 &
>>>>>>
>>>>>>
>>>>>> Under this situation, everything looks fine.
>>>>>>
>>>>>> To set up the storm framework, I change the storm.yaml in the conf
>>>>>> folder.
>>>>>>
>>>>>> mesos.master.url: "zk://192.168.123.19:2181/mesos"
>>>>>> storm.zookeeper.servers:
>>>>>> - "localhost"
>>>>>> nimbus.host: "localhost"
>>>>>>
>>>>>> and running "storm-mesos nimbus" and "storm ui".
>>>>>>
>>>>>> The problem raised here in this configuration.For every task,
>>>>>> storm-mesos created an executor with storm-mesos environment by
>>>>>> downloading the full tar ball from either http from memosphere or hdfs
>>>>>> which includes configuration file may or may not as same as the one using
>>>>>> in the master node. In my case, all executor using the above
>>>>>> configuration. The new created executor would fetch from zookeeper server
>>>>>> to but here what it tried to talked with is still "localhost". The
>>>>>> zookeeper server exists on the slave but never used for mesos, so the task
>>>>>> were marked as LOST. To make it work,In my case, the set up should be like
>>>>>> this.( I assume I also need to change the nimbus host to the master node").
>>>>>>
>>>>>> mesos.master.url: "zk://192.168.123.19:2181/mesos"
>>>>>> storm.zookeeper.servers:
>>>>>> - "192.168.123.19"
>>>>>> nimbus.host: "192.168.123.19"
>>>>>>
>>>>>>
>>>>>> After making this change, everything works fine now.
>>>>>> I hope this would help people having the same issues.
>>>>>>
>>>>>> Meanwhile. In my opinion, they way downloading whole tarball with
>>>>>> fixed configuration from somewhere should be avoided or improved.
>>>>>>
>>>>>> Probably worth to discuss.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -Luyi.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Sep 18, 2014 at 11:36 AM, Luyi Wang <wa...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I attached nimbus.log and supervisor.log for your reference
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Sep 17, 2014 at 5:30 PM, Benjamin Mahler <
>>>>>>> benjamin.mahler@gmail.com> wrote:
>>>>>>>
>>>>>>>> logs
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -Luyi.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>
>>>
>>> --
>>> James DeFelice
>>> 585.241.9488 (voice)
>>> 650.649.6071 (fax)
>>>
>>
>>
>
>
> --
> James DeFelice
> 585.241.9488 (voice)
> 650.649.6071 (fax)
>
--
James DeFelice
585.241.9488 (voice)
650.649.6071 (fax)
Re: TASK_LOST on storm task
Posted by James DeFelice <ja...@gmail.com>.
Right, so in my setup I changed the executor URI to point to
http://localhost:MY_FILESERVER_PORT/conf/MY_CUSTOM_MESOS_STORM_SPIN.
It was possible to do that because MY_FILESERVER_PORT is predictable when
you set it via the "nimbus.fileserver.port" property.
So in my environment I customize the logback configuration at framework
deploy time, repack the storm distro with the updated config and store the
distro it in conf/, the YAML is configured as mentioned above. This
facilitates the storm supervisors pulling down the storm spin with the
customized logback config.
If you don't need something so dynamic then you could just tweak the
mesos/storm tarball once (with your updated configs) and push it to a local
web server somewhere, then update your executor URI to point to that.
On Fri, Sep 19, 2014 at 6:23 PM, Luyi Wang <wa...@gmail.com> wrote:
> This is a little bit different from what I asked but might on the same
> page.
>
> I am asking if it should use the CONF_EXECUTOR_URI to setup the executor
> environment.
>
>
> https://github.com/jdef/storm/blob/5e18be19574ca3e09b0068033558f6e429099e65/src/storm/mesos/MesosNimbus.java#L441
>
> But still thanks for giving me insight of the implementation.
>
> Thanks.
>
>
>
>
> -Luyi.
>
>
>
>
> On Fri, Sep 19, 2014 at 2:12 PM, James DeFelice <ja...@gmail.com>
> wrote:
>
>> https://github.com/mesos/storm/pull/11
>>
>> Looks like some cleanup has been requested... but should work as-is.
>>
>> On Fri, Sep 19, 2014 at 1:57 AM, Luyi Wang <wa...@gmail.com>
>> wrote:
>>
>>> Thanks, james. I will pull the latest change and see what you
>>> committed. Thanks.
>>> On Sep 18, 2014 8:21 PM, "James DeFelice" <ja...@gmail.com>
>>> wrote:
>>>
>>>> I submitted a pull request that facilitates what you're asking for. It
>>>> lets you specify a port number for the built in file server on nimbus. Once
>>>> you have a predictable uri for that built in server you can rebuild the
>>>> storm tarball with whatever config you want your executors to have and
>>>> throw it in conf/ so nimbus can serve it up. I've done exactly this and
>>>> it's been working great for us.
>>>>
>>>> --sent from my phone
>>>> On Sep 18, 2014 6:06 PM, "Luyi Wang" <wa...@gmail.com> wrote:
>>>>
>>>>> Well. After investigating the problem. It turns out to be a setting
>>>>> problem.
>>>>>
>>>>> My old storm task never ran correctly as it kept trying to connect a
>>>>> wrong zookeeper server.
>>>>>
>>>>> The way I started the mesos is following. I use zookeeper to store
>>>>> configuration but this zookeeper is embedded and running standalone on
>>>>> master node(192.168.1.11).
>>>>>
>>>>> nohup sudo /home/ubuntu/mesos/build/bin/mesos-master.sh
>>>>> --work_dir=/var/lib/mesos --zk=zk://0.0.0.0:2181/mesos --quorum=1
>>>>> --log_dir=/var/log/mesos </dev/null >/dev/null 2>&1 &
>>>>>
>>>>>
>>>>> And I started the slave using following command.
>>>>>
>>>>> nohup sudo /home/ubuntu/mesos/build/bin/mesos-slave.sh --master=zk://
>>>>> 192.168.123.19:2181/mesos --log_dir=/var/log/mesos </dev/null
>>>>> >/dev/null 2>&1 &
>>>>>
>>>>>
>>>>> Under this situation, everything looks fine.
>>>>>
>>>>> To set up the storm framework, I change the storm.yaml in the conf
>>>>> folder.
>>>>>
>>>>> mesos.master.url: "zk://192.168.123.19:2181/mesos"
>>>>> storm.zookeeper.servers:
>>>>> - "localhost"
>>>>> nimbus.host: "localhost"
>>>>>
>>>>> and running "storm-mesos nimbus" and "storm ui".
>>>>>
>>>>> The problem raised here in this configuration.For every task,
>>>>> storm-mesos created an executor with storm-mesos environment by
>>>>> downloading the full tar ball from either http from memosphere or hdfs
>>>>> which includes configuration file may or may not as same as the one using
>>>>> in the master node. In my case, all executor using the above
>>>>> configuration. The new created executor would fetch from zookeeper server
>>>>> to but here what it tried to talked with is still "localhost". The
>>>>> zookeeper server exists on the slave but never used for mesos, so the task
>>>>> were marked as LOST. To make it work,In my case, the set up should be like
>>>>> this.( I assume I also need to change the nimbus host to the master node").
>>>>>
>>>>> mesos.master.url: "zk://192.168.123.19:2181/mesos"
>>>>> storm.zookeeper.servers:
>>>>> - "192.168.123.19"
>>>>> nimbus.host: "192.168.123.19"
>>>>>
>>>>>
>>>>> After making this change, everything works fine now.
>>>>> I hope this would help people having the same issues.
>>>>>
>>>>> Meanwhile. In my opinion, they way downloading whole tarball with
>>>>> fixed configuration from somewhere should be avoided or improved.
>>>>>
>>>>> Probably worth to discuss.
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -Luyi.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Sep 18, 2014 at 11:36 AM, Luyi Wang <wa...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I attached nimbus.log and supervisor.log for your reference
>>>>>>
>>>>>>
>>>>>> On Wed, Sep 17, 2014 at 5:30 PM, Benjamin Mahler <
>>>>>> benjamin.mahler@gmail.com> wrote:
>>>>>>
>>>>>>> logs
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -Luyi.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>
>>
>> --
>> James DeFelice
>> 585.241.9488 (voice)
>> 650.649.6071 (fax)
>>
>
>
--
James DeFelice
585.241.9488 (voice)
650.649.6071 (fax)
Re: TASK_LOST on storm task
Posted by Luyi Wang <wa...@gmail.com>.
This is a little bit different from what I asked but might on the same
page.
I am asking if it should use the CONF_EXECUTOR_URI to setup the executor
environment.
https://github.com/jdef/storm/blob/5e18be19574ca3e09b0068033558f6e429099e65/src/storm/mesos/MesosNimbus.java#L441
But still thanks for giving me insight of the implementation.
Thanks.
-Luyi.
On Fri, Sep 19, 2014 at 2:12 PM, James DeFelice <ja...@gmail.com>
wrote:
> https://github.com/mesos/storm/pull/11
>
> Looks like some cleanup has been requested... but should work as-is.
>
> On Fri, Sep 19, 2014 at 1:57 AM, Luyi Wang <wa...@gmail.com> wrote:
>
>> Thanks, james. I will pull the latest change and see what you committed.
>> Thanks.
>> On Sep 18, 2014 8:21 PM, "James DeFelice" <ja...@gmail.com>
>> wrote:
>>
>>> I submitted a pull request that facilitates what you're asking for. It
>>> lets you specify a port number for the built in file server on nimbus. Once
>>> you have a predictable uri for that built in server you can rebuild the
>>> storm tarball with whatever config you want your executors to have and
>>> throw it in conf/ so nimbus can serve it up. I've done exactly this and
>>> it's been working great for us.
>>>
>>> --sent from my phone
>>> On Sep 18, 2014 6:06 PM, "Luyi Wang" <wa...@gmail.com> wrote:
>>>
>>>> Well. After investigating the problem. It turns out to be a setting
>>>> problem.
>>>>
>>>> My old storm task never ran correctly as it kept trying to connect a
>>>> wrong zookeeper server.
>>>>
>>>> The way I started the mesos is following. I use zookeeper to store
>>>> configuration but this zookeeper is embedded and running standalone on
>>>> master node(192.168.1.11).
>>>>
>>>> nohup sudo /home/ubuntu/mesos/build/bin/mesos-master.sh
>>>> --work_dir=/var/lib/mesos --zk=zk://0.0.0.0:2181/mesos --quorum=1
>>>> --log_dir=/var/log/mesos </dev/null >/dev/null 2>&1 &
>>>>
>>>>
>>>> And I started the slave using following command.
>>>>
>>>> nohup sudo /home/ubuntu/mesos/build/bin/mesos-slave.sh --master=zk://
>>>> 192.168.123.19:2181/mesos --log_dir=/var/log/mesos </dev/null
>>>> >/dev/null 2>&1 &
>>>>
>>>>
>>>> Under this situation, everything looks fine.
>>>>
>>>> To set up the storm framework, I change the storm.yaml in the conf
>>>> folder.
>>>>
>>>> mesos.master.url: "zk://192.168.123.19:2181/mesos"
>>>> storm.zookeeper.servers:
>>>> - "localhost"
>>>> nimbus.host: "localhost"
>>>>
>>>> and running "storm-mesos nimbus" and "storm ui".
>>>>
>>>> The problem raised here in this configuration.For every task,
>>>> storm-mesos created an executor with storm-mesos environment by
>>>> downloading the full tar ball from either http from memosphere or hdfs
>>>> which includes configuration file may or may not as same as the one using
>>>> in the master node. In my case, all executor using the above
>>>> configuration. The new created executor would fetch from zookeeper server
>>>> to but here what it tried to talked with is still "localhost". The
>>>> zookeeper server exists on the slave but never used for mesos, so the task
>>>> were marked as LOST. To make it work,In my case, the set up should be like
>>>> this.( I assume I also need to change the nimbus host to the master node").
>>>>
>>>> mesos.master.url: "zk://192.168.123.19:2181/mesos"
>>>> storm.zookeeper.servers:
>>>> - "192.168.123.19"
>>>> nimbus.host: "192.168.123.19"
>>>>
>>>>
>>>> After making this change, everything works fine now.
>>>> I hope this would help people having the same issues.
>>>>
>>>> Meanwhile. In my opinion, they way downloading whole tarball with fixed
>>>> configuration from somewhere should be avoided or improved.
>>>>
>>>> Probably worth to discuss.
>>>>
>>>> Thanks.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> -Luyi.
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Sep 18, 2014 at 11:36 AM, Luyi Wang <wa...@gmail.com>
>>>> wrote:
>>>>
>>>>> I attached nimbus.log and supervisor.log for your reference
>>>>>
>>>>>
>>>>> On Wed, Sep 17, 2014 at 5:30 PM, Benjamin Mahler <
>>>>> benjamin.mahler@gmail.com> wrote:
>>>>>
>>>>>> logs
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -Luyi.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>
>
> --
> James DeFelice
> 585.241.9488 (voice)
> 650.649.6071 (fax)
>
Re: TASK_LOST on storm task
Posted by James DeFelice <ja...@gmail.com>.
https://github.com/mesos/storm/pull/11
Looks like some cleanup has been requested... but should work as-is.
On Fri, Sep 19, 2014 at 1:57 AM, Luyi Wang <wa...@gmail.com> wrote:
> Thanks, james. I will pull the latest change and see what you committed.
> Thanks.
> On Sep 18, 2014 8:21 PM, "James DeFelice" <ja...@gmail.com>
> wrote:
>
>> I submitted a pull request that facilitates what you're asking for. It
>> lets you specify a port number for the built in file server on nimbus. Once
>> you have a predictable uri for that built in server you can rebuild the
>> storm tarball with whatever config you want your executors to have and
>> throw it in conf/ so nimbus can serve it up. I've done exactly this and
>> it's been working great for us.
>>
>> --sent from my phone
>> On Sep 18, 2014 6:06 PM, "Luyi Wang" <wa...@gmail.com> wrote:
>>
>>> Well. After investigating the problem. It turns out to be a setting
>>> problem.
>>>
>>> My old storm task never ran correctly as it kept trying to connect a
>>> wrong zookeeper server.
>>>
>>> The way I started the mesos is following. I use zookeeper to store
>>> configuration but this zookeeper is embedded and running standalone on
>>> master node(192.168.1.11).
>>>
>>> nohup sudo /home/ubuntu/mesos/build/bin/mesos-master.sh
>>> --work_dir=/var/lib/mesos --zk=zk://0.0.0.0:2181/mesos --quorum=1
>>> --log_dir=/var/log/mesos </dev/null >/dev/null 2>&1 &
>>>
>>>
>>> And I started the slave using following command.
>>>
>>> nohup sudo /home/ubuntu/mesos/build/bin/mesos-slave.sh --master=zk://
>>> 192.168.123.19:2181/mesos --log_dir=/var/log/mesos </dev/null
>>> >/dev/null 2>&1 &
>>>
>>>
>>> Under this situation, everything looks fine.
>>>
>>> To set up the storm framework, I change the storm.yaml in the conf
>>> folder.
>>>
>>> mesos.master.url: "zk://192.168.123.19:2181/mesos"
>>> storm.zookeeper.servers:
>>> - "localhost"
>>> nimbus.host: "localhost"
>>>
>>> and running "storm-mesos nimbus" and "storm ui".
>>>
>>> The problem raised here in this configuration.For every task,
>>> storm-mesos created an executor with storm-mesos environment by
>>> downloading the full tar ball from either http from memosphere or hdfs
>>> which includes configuration file may or may not as same as the one using
>>> in the master node. In my case, all executor using the above
>>> configuration. The new created executor would fetch from zookeeper server
>>> to but here what it tried to talked with is still "localhost". The
>>> zookeeper server exists on the slave but never used for mesos, so the task
>>> were marked as LOST. To make it work,In my case, the set up should be like
>>> this.( I assume I also need to change the nimbus host to the master node").
>>>
>>> mesos.master.url: "zk://192.168.123.19:2181/mesos"
>>> storm.zookeeper.servers:
>>> - "192.168.123.19"
>>> nimbus.host: "192.168.123.19"
>>>
>>>
>>> After making this change, everything works fine now.
>>> I hope this would help people having the same issues.
>>>
>>> Meanwhile. In my opinion, they way downloading whole tarball with fixed
>>> configuration from somewhere should be avoided or improved.
>>>
>>> Probably worth to discuss.
>>>
>>> Thanks.
>>>
>>>
>>>
>>>
>>>
>>>
>>> -Luyi.
>>>
>>>
>>>
>>>
>>> On Thu, Sep 18, 2014 at 11:36 AM, Luyi Wang <wa...@gmail.com>
>>> wrote:
>>>
>>>> I attached nimbus.log and supervisor.log for your reference
>>>>
>>>>
>>>> On Wed, Sep 17, 2014 at 5:30 PM, Benjamin Mahler <
>>>> benjamin.mahler@gmail.com> wrote:
>>>>
>>>>> logs
>>>>
>>>>
>>>>
>>>>
>>>> -Luyi.
>>>>
>>>>
>>>>
>>>>
>>>
--
James DeFelice
585.241.9488 (voice)
650.649.6071 (fax)
Re: TASK_LOST on storm task
Posted by Luyi Wang <wa...@gmail.com>.
Thanks, james. I will pull the latest change and see what you committed.
Thanks.
On Sep 18, 2014 8:21 PM, "James DeFelice" <ja...@gmail.com> wrote:
> I submitted a pull request that facilitates what you're asking for. It
> lets you specify a port number for the built in file server on nimbus. Once
> you have a predictable uri for that built in server you can rebuild the
> storm tarball with whatever config you want your executors to have and
> throw it in conf/ so nimbus can serve it up. I've done exactly this and
> it's been working great for us.
>
> --sent from my phone
> On Sep 18, 2014 6:06 PM, "Luyi Wang" <wa...@gmail.com> wrote:
>
>> Well. After investigating the problem. It turns out to be a setting
>> problem.
>>
>> My old storm task never ran correctly as it kept trying to connect a
>> wrong zookeeper server.
>>
>> The way I started the mesos is following. I use zookeeper to store
>> configuration but this zookeeper is embedded and running standalone on
>> master node(192.168.1.11).
>>
>> nohup sudo /home/ubuntu/mesos/build/bin/mesos-master.sh
>> --work_dir=/var/lib/mesos --zk=zk://0.0.0.0:2181/mesos --quorum=1
>> --log_dir=/var/log/mesos </dev/null >/dev/null 2>&1 &
>>
>>
>> And I started the slave using following command.
>>
>> nohup sudo /home/ubuntu/mesos/build/bin/mesos-slave.sh --master=zk://
>> 192.168.123.19:2181/mesos --log_dir=/var/log/mesos </dev/null >/dev/null
>> 2>&1 &
>>
>>
>> Under this situation, everything looks fine.
>>
>> To set up the storm framework, I change the storm.yaml in the conf
>> folder.
>>
>> mesos.master.url: "zk://192.168.123.19:2181/mesos"
>> storm.zookeeper.servers:
>> - "localhost"
>> nimbus.host: "localhost"
>>
>> and running "storm-mesos nimbus" and "storm ui".
>>
>> The problem raised here in this configuration.For every task, storm-mesos
>> created an executor with storm-mesos environment by downloading the full
>> tar ball from either http from memosphere or hdfs which includes
>> configuration file may or may not as same as the one using in the master
>> node. In my case, all executor using the above configuration. The new
>> created executor would fetch from zookeeper server to but here what it
>> tried to talked with is still "localhost". The zookeeper server exists on
>> the slave but never used for mesos, so the task were marked as LOST. To
>> make it work,In my case, the set up should be like this.( I assume I also
>> need to change the nimbus host to the master node").
>>
>> mesos.master.url: "zk://192.168.123.19:2181/mesos"
>> storm.zookeeper.servers:
>> - "192.168.123.19"
>> nimbus.host: "192.168.123.19"
>>
>>
>> After making this change, everything works fine now.
>> I hope this would help people having the same issues.
>>
>> Meanwhile. In my opinion, they way downloading whole tarball with fixed
>> configuration from somewhere should be avoided or improved.
>>
>> Probably worth to discuss.
>>
>> Thanks.
>>
>>
>>
>>
>>
>>
>> -Luyi.
>>
>>
>>
>>
>> On Thu, Sep 18, 2014 at 11:36 AM, Luyi Wang <wa...@gmail.com>
>> wrote:
>>
>>> I attached nimbus.log and supervisor.log for your reference
>>>
>>>
>>> On Wed, Sep 17, 2014 at 5:30 PM, Benjamin Mahler <
>>> benjamin.mahler@gmail.com> wrote:
>>>
>>>> logs
>>>
>>>
>>>
>>>
>>> -Luyi.
>>>
>>>
>>>
>>>
>>
Re: TASK_LOST on storm task
Posted by James DeFelice <ja...@gmail.com>.
I submitted a pull request that facilitates what you're asking for. It lets
you specify a port number for the built in file server on nimbus. Once you
have a predictable uri for that built in server you can rebuild the storm
tarball with whatever config you want your executors to have and throw it
in conf/ so nimbus can serve it up. I've done exactly this and it's been
working great for us.
--sent from my phone
On Sep 18, 2014 6:06 PM, "Luyi Wang" <wa...@gmail.com> wrote:
> Well. After investigating the problem. It turns out to be a setting
> problem.
>
> My old storm task never ran correctly as it kept trying to connect a wrong
> zookeeper server.
>
> The way I started the mesos is following. I use zookeeper to store
> configuration but this zookeeper is embedded and running standalone on
> master node(192.168.1.11).
>
> nohup sudo /home/ubuntu/mesos/build/bin/mesos-master.sh
> --work_dir=/var/lib/mesos --zk=zk://0.0.0.0:2181/mesos --quorum=1
> --log_dir=/var/log/mesos </dev/null >/dev/null 2>&1 &
>
>
> And I started the slave using following command.
>
> nohup sudo /home/ubuntu/mesos/build/bin/mesos-slave.sh --master=zk://
> 192.168.123.19:2181/mesos --log_dir=/var/log/mesos </dev/null >/dev/null
> 2>&1 &
>
>
> Under this situation, everything looks fine.
>
> To set up the storm framework, I change the storm.yaml in the conf folder.
>
> mesos.master.url: "zk://192.168.123.19:2181/mesos"
> storm.zookeeper.servers:
> - "localhost"
> nimbus.host: "localhost"
>
> and running "storm-mesos nimbus" and "storm ui".
>
> The problem raised here in this configuration.For every task, storm-mesos
> created an executor with storm-mesos environment by downloading the full
> tar ball from either http from memosphere or hdfs which includes
> configuration file may or may not as same as the one using in the master
> node. In my case, all executor using the above configuration. The new
> created executor would fetch from zookeeper server to but here what it
> tried to talked with is still "localhost". The zookeeper server exists on
> the slave but never used for mesos, so the task were marked as LOST. To
> make it work,In my case, the set up should be like this.( I assume I also
> need to change the nimbus host to the master node").
>
> mesos.master.url: "zk://192.168.123.19:2181/mesos"
> storm.zookeeper.servers:
> - "192.168.123.19"
> nimbus.host: "192.168.123.19"
>
>
> After making this change, everything works fine now.
> I hope this would help people having the same issues.
>
> Meanwhile. In my opinion, they way downloading whole tarball with fixed
> configuration from somewhere should be avoided or improved.
>
> Probably worth to discuss.
>
> Thanks.
>
>
>
>
>
>
> -Luyi.
>
>
>
>
> On Thu, Sep 18, 2014 at 11:36 AM, Luyi Wang <wa...@gmail.com>
> wrote:
>
>> I attached nimbus.log and supervisor.log for your reference
>>
>>
>> On Wed, Sep 17, 2014 at 5:30 PM, Benjamin Mahler <
>> benjamin.mahler@gmail.com> wrote:
>>
>>> logs
>>
>>
>>
>>
>> -Luyi.
>>
>>
>>
>>
>
Re: TASK_LOST on storm task
Posted by Luyi Wang <wa...@gmail.com>.
Well. After investigating the problem. It turns out to be a setting
problem.
My old storm task never ran correctly as it kept trying to connect a wrong
zookeeper server.
The way I started the mesos is following. I use zookeeper to store
configuration but this zookeeper is embedded and running standalone on
master node(192.168.1.11).
nohup sudo /home/ubuntu/mesos/build/bin/mesos-master.sh
--work_dir=/var/lib/mesos --zk=zk://0.0.0.0:2181/mesos --quorum=1
--log_dir=/var/log/mesos </dev/null >/dev/null 2>&1 &
And I started the slave using following command.
nohup sudo /home/ubuntu/mesos/build/bin/mesos-slave.sh --master=zk://
192.168.123.19:2181/mesos --log_dir=/var/log/mesos </dev/null >/dev/null
2>&1 &
Under this situation, everything looks fine.
To set up the storm framework, I change the storm.yaml in the conf folder.
mesos.master.url: "zk://192.168.123.19:2181/mesos"
storm.zookeeper.servers:
- "localhost"
nimbus.host: "localhost"
and running "storm-mesos nimbus" and "storm ui".
The problem raised here in this configuration.For every task, storm-mesos
created an executor with storm-mesos environment by downloading the full
tar ball from either http from memosphere or hdfs which includes
configuration file may or may not as same as the one using in the master
node. In my case, all executor using the above configuration. The new
created executor would fetch from zookeeper server to but here what it
tried to talked with is still "localhost". The zookeeper server exists on
the slave but never used for mesos, so the task were marked as LOST. To
make it work,In my case, the set up should be like this.( I assume I also
need to change the nimbus host to the master node").
mesos.master.url: "zk://192.168.123.19:2181/mesos"
storm.zookeeper.servers:
- "192.168.123.19"
nimbus.host: "192.168.123.19"
After making this change, everything works fine now.
I hope this would help people having the same issues.
Meanwhile. In my opinion, they way downloading whole tarball with fixed
configuration from somewhere should be avoided or improved.
Probably worth to discuss.
Thanks.
-Luyi.
On Thu, Sep 18, 2014 at 11:36 AM, Luyi Wang <wa...@gmail.com> wrote:
> I attached nimbus.log and supervisor.log for your reference
>
>
> On Wed, Sep 17, 2014 at 5:30 PM, Benjamin Mahler <
> benjamin.mahler@gmail.com> wrote:
>
>> logs
>
>
>
>
> -Luyi.
>
>
>
>
Re: TASK_LOST on storm task
Posted by Luyi Wang <wa...@gmail.com>.
I attached nimbus.log and supervisor.log for your reference
On Wed, Sep 17, 2014 at 5:30 PM, Benjamin Mahler <be...@gmail.com>
wrote:
> logs
-Luyi.
Re: TASK_LOST on storm task
Posted by Benjamin Mahler <be...@gmail.com>.
It looks like the executor exited while holding a RUNNING task. Do you have
the executor logs handy? You can find them in the mesos webui or in the
sandbox location of this executor.
On Wed, Sep 17, 2014 at 2:32 PM, Luyi Wang <wa...@gmail.com> wrote:
> Here is the slave log.
>
> I0917 00:16:45.759867 10209 slave.cpp:3057] Current usage 43.77%. Max
> allowed age: 3.235877672809236days
> I0917 00:17:31.145267 10212 slave.cpp:1011] Got assigned task
> dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 for framework
> 20140915-230424-326871232-5050-13574-0000
> I0917 00:17:31.145606 10212 gc.cpp:84] Unscheduling
> '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-2/frameworks/20140915-230424-326871232-5050-13574-0000'
> from gc
> I0917 00:17:31.145678 10212 slave.cpp:1121] Launching task
> dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 for framework
> 20140915-230424-326871232-5050-13574-0000
> I0917 00:17:31.146811 10212 slave.cpp:1231] Queuing task
> 'dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000' for executor
> production-topology-1-1410913050 of framework
> '20140915-230424-326871232-5050-13574-0000
> I0917 00:17:31.146898 10212 containerizer.cpp:394] Starting container
> 'dfbf47af-c8b7-481f-b8fc-6a33a8f115d9' for executor
> 'production-topology-1-1410913050' of framework
> '20140915-230424-326871232-5050-13574-0000'
> I0917 00:17:31.147886 10212 launcher.cpp:137] Forked child with pid
> '20604' for container 'dfbf47af-c8b7-481f-b8fc-6a33a8f115d9'
> I0917 00:17:31.148232 10212 containerizer.cpp:510] Fetching URIs for
> container 'dfbf47af-c8b7-481f-b8fc-6a33a8f115d9' using command
> '/home/ubuntu/mesos/build/src/mesos-fetcher'
> I0917 00:17:45.761070 10210 slave.cpp:3057] Current usage 43.77%. Max
> allowed age: 3.235862073509317days
> I0917 00:17:48.932791 10214 slave.cpp:2542] Monitoring executor
> 'production-topology-1-1410913050' of framework
> '20140915-230424-326871232-5050-13574-0000' in container
> 'dfbf47af-c8b7-481f-b8fc-6a33a8f115d9'
> I0917 00:17:58.360540 10215 slave.cpp:1741] Got registration for executor
> 'production-topology-1-1410913050' of framework
> 20140915-230424-326871232-5050-13574-0000
> I0917 00:17:58.360841 10215 slave.cpp:1859] Flushing queued task
> dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 for executor
> 'production-topology-1-1410913050' of framework
> 20140915-230424-326871232-5050-13574-0000
> I0917 00:17:58.461596 10215 slave.cpp:2093] Handling status update
> TASK_RUNNING (UUID: d05dbbc9-a4a6-4f83-b5a5-4d60079a26c0) for task
> dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework
> 20140915-230424-326871232-5050-13574-0000 from executor(1)@
> 192.168.123.29:41892
> I0917 00:17:58.461767 10215 status_update_manager.cpp:320] Received status
> update TASK_RUNNING (UUID: d05dbbc9-a4a6-4f83-b5a5-4d60079a26c0) for task
> dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework
> 20140915-230424-326871232-5050-13574-0000
> I0917 00:17:58.461879 10215 status_update_manager.cpp:373] Forwarding
> status update TASK_RUNNING (UUID: d05dbbc9-a4a6-4f83-b5a5-4d60079a26c0) for
> task dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework
> 20140915-230424-326871232-5050-13574-0000 to master@192.168.123.19:5050
> I0917 00:17:58.462002 10215 slave.cpp:2256] Sending acknowledgement for
> status update TASK_RUNNING (UUID: d05dbbc9-a4a6-4f83-b5a5-4d60079a26c0) for
> task dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework
> 20140915-230424-326871232-5050-13574-0000 to executor(1)@
> 192.168.123.29:41892
> I0917 00:17:58.470211 10213 status_update_manager.cpp:398] Received status
> update acknowledgement (UUID: d05dbbc9-a4a6-4f83-b5a5-4d60079a26c0) for
> task dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework
> 20140915-230424-326871232-5050-13574-0000
> I0917 00:18:11.776726 10209 http.cpp:330] HTTP request for
> '/slave(1)/state.json'
> I0917 00:18:45.616175 10211 http.cpp:330] HTTP request for
> '/slave(1)/state.json'
> I0917 00:18:45.762120 10213 slave.cpp:3057] Current usage 43.92%. Max
> allowed age: 3.225739484327836days
> I0917 00:19:43.761559 10210 http.cpp:330] HTTP request for
> '/slave(1)/state.json'
> I0917 00:19:45.762634 10208 slave.cpp:3057] Current usage 43.92%. Max
> allowed age: 3.225736771406111days
> I0917 00:19:59.021102 10208 containerizer.cpp:997] Executor for container
> 'dfbf47af-c8b7-481f-b8fc-6a33a8f115d9' has exited
> I0917 00:19:59.021178 10208 containerizer.cpp:882] Destroying container
> 'dfbf47af-c8b7-481f-b8fc-6a33a8f115d9'
> I0917 00:19:59.026793 10208 slave.cpp:2600] Executor
> 'production-topology-1-1410913050' of framework
> 20140915-230424-326871232-5050-13574-0000 exited with status 0
> I0917 00:19:59.027725 10208 slave.cpp:2093] Handling status update
> TASK_LOST (UUID: 8e3543de-161d-4a5a-bcd7-b7be6f053b24) for task
> dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework
> 20140915-230424-326871232-5050-13574-0000 from @0.0.0.0:0
> W0917 00:19:59.027933 10213 containerizer.cpp:788] Ignoring update for
> unknown container: dfbf47af-c8b7-481f-b8fc-6a33a8f115d9
> I0917 00:19:59.028105 10213 status_update_manager.cpp:320] Received status
> update TASK_LOST (UUID: 8e3543de-161d-4a5a-bcd7-b7be6f053b24) for task
> dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework
> 20140915-230424-326871232-5050-13574-0000
> I0917 00:19:59.028143 10213 status_update_manager.cpp:373] Forwarding
> status update TASK_LOST (UUID: 8e3543de-161d-4a5a-bcd7-b7be6f053b24) for
> task dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework
> 20140915-230424-326871232-5050-13574-0000 to master@192.168.123.19:5050
> I0917 00:19:59.033900 10213 slave.cpp:2736] Cleaning up executor
> 'production-topology-1-1410913050' of framework
> 20140915-230424-326871232-5050-13574-0000
> I0917 00:19:59.034037 10213 slave.cpp:2811] Cleaning up framework
> 20140915-230424-326871232-5050-13574-0000
> I0917 00:19:59.034162 10211 status_update_manager.cpp:282] Closing status
> update streams for framework 20140915-230424-326871232-5050-13574-0000
> I0917 00:19:59.034098 10213 gc.cpp:56] Scheduling
> '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-2/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-1410913050/runs/dfbf47af-c8b7-481f-b8fc-6a33a8f115d9'
> for gc 6.99999960653926days in the future
> I0917 00:19:59.034209 10213 gc.cpp:56] Scheduling
> '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-2/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-1410913050'
> for gc 6.99999960616889days in the future
> I0917 00:19:59.034255 10213 gc.cpp:56] Scheduling
> '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-2/frameworks/20140915-230424-326871232-5050-13574-0000'
> for gc 6.99999960553185days in the future
>
>
>
>
>
>
> -Luyi.
>
>
>
>
> On Wed, Sep 17, 2014 at 12:56 PM, Benjamin Mahler <
> benjamin.mahler@gmail.com> wrote:
>
>> Can you show us the the slave log and more of the master log?
>>
>> There should be a TASK_LOST somewhere within them.
>>
>> On Wed, Sep 17, 2014 at 10:43 AM, Luyi Wang <wa...@gmail.com>
>> wrote:
>>
>>> Have anyone experience TASK_LOST status for storm tasks on mesos.
>>>
>>>
>>> I checked the stderr. Everything seems normal.
>>> WARNING: Logging before InitGoogleLogging() is written to STDERR
>>> I0917 00:21:36.164840 4831 fetcher.cpp:76] Fetching URI 'hdfs://
>>> 192.168.123.27/storm-mesos-0.9.2-incubating.tgz'
>>> I0917 00:21:36.165225 4831 fetcher.cpp:105] Downloading resource from
>>> 'hdfs://192.168.123.27/storm-mesos-0.9.2-incubating.tgz' to
>>> '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-3/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-1410913050/runs/53d06991-cb84-49d3-a530-f83efcf339e9/storm-mesos-0.9.2-incubating.tgz'
>>> I0917 00:21:52.202791 4831 fetcher.cpp:64] Extracted resource
>>> '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-3/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-1410913050/runs/53d06991-cb84-49d3-a530-f83efcf339e9/storm-mesos-0.9.2-incubating.tgz'
>>> into
>>> '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-3/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-1410913050/runs/53d06991-cb84-49d3-a530-f83efcf339e9'
>>> I0917 00:21:52.206581 4831 fetcher.cpp:76] Fetching URI '
>>> http://mesos:45579/conf/storm.yaml'
>>> I0917 00:21:52.206626 4831 fetcher.cpp:126] Downloading '
>>> http://mesos:45579/conf/storm.yaml' to
>>> '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-3/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-/runs/53d06991-cb84-49d3-a530-f83efcf339e9/storm.yaml'
>>> I0917 00:22:01.829298 4984 exec.cpp:132] Version: 0.21.0
>>> I0917 00:22:01.832185 5006 exec.cpp:206] Executor registered on slave
>>> 20140915-185627-326871232-5050-8074-3
>>>
>>>
>>>
>>>
>>> I also checked the mesos info log. Here is what it logged.
>>>
>>> I0917 00:19:59.020755 5145 master.cpp:3261] Executor
>>> production-topology-1-1410913050 of framework
>>> 20140915-230424-326871232-5050-13574-0000 on slave
>>> 20140915-185627-326871232-5050-8074-2 at slave(1)@192.168.123.29:5051 (
>>> dev10-cdh5-03.int.dev10.smcl.pure-breeze.com) exited with status 0
>>>
>>>
>>> Any idea on this?
>>>
>>>
>>>
>>>
>>> -Luyi.
>>>
>>>
>>>
>>>
>>
>
Re: TASK_LOST on storm task
Posted by Luyi Wang <wa...@gmail.com>.
Here is the slave log.
I0917 00:16:45.759867 10209 slave.cpp:3057] Current usage 43.77%. Max
allowed age: 3.235877672809236days
I0917 00:17:31.145267 10212 slave.cpp:1011] Got assigned task
dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 for framework
20140915-230424-326871232-5050-13574-0000
I0917 00:17:31.145606 10212 gc.cpp:84] Unscheduling
'/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-2/frameworks/20140915-230424-326871232-5050-13574-0000'
from gc
I0917 00:17:31.145678 10212 slave.cpp:1121] Launching task
dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 for framework
20140915-230424-326871232-5050-13574-0000
I0917 00:17:31.146811 10212 slave.cpp:1231] Queuing task
'dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000' for executor
production-topology-1-1410913050 of framework
'20140915-230424-326871232-5050-13574-0000
I0917 00:17:31.146898 10212 containerizer.cpp:394] Starting container
'dfbf47af-c8b7-481f-b8fc-6a33a8f115d9' for executor
'production-topology-1-1410913050' of framework
'20140915-230424-326871232-5050-13574-0000'
I0917 00:17:31.147886 10212 launcher.cpp:137] Forked child with pid '20604'
for container 'dfbf47af-c8b7-481f-b8fc-6a33a8f115d9'
I0917 00:17:31.148232 10212 containerizer.cpp:510] Fetching URIs for
container 'dfbf47af-c8b7-481f-b8fc-6a33a8f115d9' using command
'/home/ubuntu/mesos/build/src/mesos-fetcher'
I0917 00:17:45.761070 10210 slave.cpp:3057] Current usage 43.77%. Max
allowed age: 3.235862073509317days
I0917 00:17:48.932791 10214 slave.cpp:2542] Monitoring executor
'production-topology-1-1410913050' of framework
'20140915-230424-326871232-5050-13574-0000' in container
'dfbf47af-c8b7-481f-b8fc-6a33a8f115d9'
I0917 00:17:58.360540 10215 slave.cpp:1741] Got registration for executor
'production-topology-1-1410913050' of framework
20140915-230424-326871232-5050-13574-0000
I0917 00:17:58.360841 10215 slave.cpp:1859] Flushing queued task
dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 for executor
'production-topology-1-1410913050' of framework
20140915-230424-326871232-5050-13574-0000
I0917 00:17:58.461596 10215 slave.cpp:2093] Handling status update
TASK_RUNNING (UUID: d05dbbc9-a4a6-4f83-b5a5-4d60079a26c0) for task
dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework
20140915-230424-326871232-5050-13574-0000 from executor(1)@
192.168.123.29:41892
I0917 00:17:58.461767 10215 status_update_manager.cpp:320] Received status
update TASK_RUNNING (UUID: d05dbbc9-a4a6-4f83-b5a5-4d60079a26c0) for task
dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework
20140915-230424-326871232-5050-13574-0000
I0917 00:17:58.461879 10215 status_update_manager.cpp:373] Forwarding
status update TASK_RUNNING (UUID: d05dbbc9-a4a6-4f83-b5a5-4d60079a26c0) for
task dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework
20140915-230424-326871232-5050-13574-0000 to master@192.168.123.19:5050
I0917 00:17:58.462002 10215 slave.cpp:2256] Sending acknowledgement for
status update TASK_RUNNING (UUID: d05dbbc9-a4a6-4f83-b5a5-4d60079a26c0) for
task dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework
20140915-230424-326871232-5050-13574-0000 to executor(1)@
192.168.123.29:41892
I0917 00:17:58.470211 10213 status_update_manager.cpp:398] Received status
update acknowledgement (UUID: d05dbbc9-a4a6-4f83-b5a5-4d60079a26c0) for
task dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework
20140915-230424-326871232-5050-13574-0000
I0917 00:18:11.776726 10209 http.cpp:330] HTTP request for
'/slave(1)/state.json'
I0917 00:18:45.616175 10211 http.cpp:330] HTTP request for
'/slave(1)/state.json'
I0917 00:18:45.762120 10213 slave.cpp:3057] Current usage 43.92%. Max
allowed age: 3.225739484327836days
I0917 00:19:43.761559 10210 http.cpp:330] HTTP request for
'/slave(1)/state.json'
I0917 00:19:45.762634 10208 slave.cpp:3057] Current usage 43.92%. Max
allowed age: 3.225736771406111days
I0917 00:19:59.021102 10208 containerizer.cpp:997] Executor for container
'dfbf47af-c8b7-481f-b8fc-6a33a8f115d9' has exited
I0917 00:19:59.021178 10208 containerizer.cpp:882] Destroying container
'dfbf47af-c8b7-481f-b8fc-6a33a8f115d9'
I0917 00:19:59.026793 10208 slave.cpp:2600] Executor
'production-topology-1-1410913050' of framework
20140915-230424-326871232-5050-13574-0000 exited with status 0
I0917 00:19:59.027725 10208 slave.cpp:2093] Handling status update
TASK_LOST (UUID: 8e3543de-161d-4a5a-bcd7-b7be6f053b24) for task
dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework
20140915-230424-326871232-5050-13574-0000 from @0.0.0.0:0
W0917 00:19:59.027933 10213 containerizer.cpp:788] Ignoring update for
unknown container: dfbf47af-c8b7-481f-b8fc-6a33a8f115d9
I0917 00:19:59.028105 10213 status_update_manager.cpp:320] Received status
update TASK_LOST (UUID: 8e3543de-161d-4a5a-bcd7-b7be6f053b24) for task
dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework
20140915-230424-326871232-5050-13574-0000
I0917 00:19:59.028143 10213 status_update_manager.cpp:373] Forwarding
status update TASK_LOST (UUID: 8e3543de-161d-4a5a-bcd7-b7be6f053b24) for
task dev10-cdh5-03.int.dev10.smcl.pure-breeze.com-31000 of framework
20140915-230424-326871232-5050-13574-0000 to master@192.168.123.19:5050
I0917 00:19:59.033900 10213 slave.cpp:2736] Cleaning up executor
'production-topology-1-1410913050' of framework
20140915-230424-326871232-5050-13574-0000
I0917 00:19:59.034037 10213 slave.cpp:2811] Cleaning up framework
20140915-230424-326871232-5050-13574-0000
I0917 00:19:59.034162 10211 status_update_manager.cpp:282] Closing status
update streams for framework 20140915-230424-326871232-5050-13574-0000
I0917 00:19:59.034098 10213 gc.cpp:56] Scheduling
'/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-2/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-1410913050/runs/dfbf47af-c8b7-481f-b8fc-6a33a8f115d9'
for gc 6.99999960653926days in the future
I0917 00:19:59.034209 10213 gc.cpp:56] Scheduling
'/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-2/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-1410913050'
for gc 6.99999960616889days in the future
I0917 00:19:59.034255 10213 gc.cpp:56] Scheduling
'/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-2/frameworks/20140915-230424-326871232-5050-13574-0000'
for gc 6.99999960553185days in the future
-Luyi.
On Wed, Sep 17, 2014 at 12:56 PM, Benjamin Mahler <benjamin.mahler@gmail.com
> wrote:
> Can you show us the the slave log and more of the master log?
>
> There should be a TASK_LOST somewhere within them.
>
> On Wed, Sep 17, 2014 at 10:43 AM, Luyi Wang <wa...@gmail.com>
> wrote:
>
>> Have anyone experience TASK_LOST status for storm tasks on mesos.
>>
>>
>> I checked the stderr. Everything seems normal.
>> WARNING: Logging before InitGoogleLogging() is written to STDERR
>> I0917 00:21:36.164840 4831 fetcher.cpp:76] Fetching URI 'hdfs://
>> 192.168.123.27/storm-mesos-0.9.2-incubating.tgz'
>> I0917 00:21:36.165225 4831 fetcher.cpp:105] Downloading resource from
>> 'hdfs://192.168.123.27/storm-mesos-0.9.2-incubating.tgz' to
>> '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-3/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-1410913050/runs/53d06991-cb84-49d3-a530-f83efcf339e9/storm-mesos-0.9.2-incubating.tgz'
>> I0917 00:21:52.202791 4831 fetcher.cpp:64] Extracted resource
>> '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-3/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-1410913050/runs/53d06991-cb84-49d3-a530-f83efcf339e9/storm-mesos-0.9.2-incubating.tgz'
>> into
>> '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-3/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-1410913050/runs/53d06991-cb84-49d3-a530-f83efcf339e9'
>> I0917 00:21:52.206581 4831 fetcher.cpp:76] Fetching URI '
>> http://mesos:45579/conf/storm.yaml'
>> I0917 00:21:52.206626 4831 fetcher.cpp:126] Downloading '
>> http://mesos:45579/conf/storm.yaml' to
>> '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-3/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-/runs/53d06991-cb84-49d3-a530-f83efcf339e9/storm.yaml'
>> I0917 00:22:01.829298 4984 exec.cpp:132] Version: 0.21.0
>> I0917 00:22:01.832185 5006 exec.cpp:206] Executor registered on slave
>> 20140915-185627-326871232-5050-8074-3
>>
>>
>>
>>
>> I also checked the mesos info log. Here is what it logged.
>>
>> I0917 00:19:59.020755 5145 master.cpp:3261] Executor
>> production-topology-1-1410913050 of framework
>> 20140915-230424-326871232-5050-13574-0000 on slave
>> 20140915-185627-326871232-5050-8074-2 at slave(1)@192.168.123.29:5051 (
>> dev10-cdh5-03.int.dev10.smcl.pure-breeze.com) exited with status 0
>>
>>
>> Any idea on this?
>>
>>
>>
>>
>> -Luyi.
>>
>>
>>
>>
>
Re: TASK_LOST on storm task
Posted by Benjamin Mahler <be...@gmail.com>.
Can you show us the the slave log and more of the master log?
There should be a TASK_LOST somewhere within them.
On Wed, Sep 17, 2014 at 10:43 AM, Luyi Wang <wa...@gmail.com> wrote:
> Have anyone experience TASK_LOST status for storm tasks on mesos.
>
>
> I checked the stderr. Everything seems normal.
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I0917 00:21:36.164840 4831 fetcher.cpp:76] Fetching URI 'hdfs://
> 192.168.123.27/storm-mesos-0.9.2-incubating.tgz'
> I0917 00:21:36.165225 4831 fetcher.cpp:105] Downloading resource from
> 'hdfs://192.168.123.27/storm-mesos-0.9.2-incubating.tgz' to
> '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-3/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-1410913050/runs/53d06991-cb84-49d3-a530-f83efcf339e9/storm-mesos-0.9.2-incubating.tgz'
> I0917 00:21:52.202791 4831 fetcher.cpp:64] Extracted resource
> '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-3/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-1410913050/runs/53d06991-cb84-49d3-a530-f83efcf339e9/storm-mesos-0.9.2-incubating.tgz'
> into
> '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-3/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-1410913050/runs/53d06991-cb84-49d3-a530-f83efcf339e9'
> I0917 00:21:52.206581 4831 fetcher.cpp:76] Fetching URI '
> http://mesos:45579/conf/storm.yaml'
> I0917 00:21:52.206626 4831 fetcher.cpp:126] Downloading '
> http://mesos:45579/conf/storm.yaml' to
> '/tmp/mesos/slaves/20140915-185627-326871232-5050-8074-3/frameworks/20140915-230424-326871232-5050-13574-0000/executors/production-topology-1-/runs/53d06991-cb84-49d3-a530-f83efcf339e9/storm.yaml'
> I0917 00:22:01.829298 4984 exec.cpp:132] Version: 0.21.0
> I0917 00:22:01.832185 5006 exec.cpp:206] Executor registered on slave
> 20140915-185627-326871232-5050-8074-3
>
>
>
>
> I also checked the mesos info log. Here is what it logged.
>
> I0917 00:19:59.020755 5145 master.cpp:3261] Executor
> production-topology-1-1410913050 of framework
> 20140915-230424-326871232-5050-13574-0000 on slave
> 20140915-185627-326871232-5050-8074-2 at slave(1)@192.168.123.29:5051 (
> dev10-cdh5-03.int.dev10.smcl.pure-breeze.com) exited with status 0
>
>
> Any idea on this?
>
>
>
>
> -Luyi.
>
>
>
>