You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by 木内満歳 <m-...@creationline.com> on 2015/11/23 15:30:01 UTC
Spark task sometimes won't start
Hi,
I'm experiencing that some spark task on Mesos 0.25 occasionally won't
start.
Please tell some advice how to see more detail against it.
Here is the slave log about bad task
Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.677291 18516
slave.cpp:2379] Got registration for executor
'235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework
235498ca-6603-4cfe-bfc7-94005bb235fb-1442 from executor(1)@
10.130.91.16:60295
Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.679875 18516
slave.cpp:1760] Sending queued task '0' to executor
'235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework
235498ca-6603-4cfe-bfc7-94005bb235fb-1442
(no more log about this task)
When task succeed to run, slave log shows like that.
Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.637285 8658
slave.cpp:2379] Got registration for executor
'235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework
235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from executor(1)@
10.130.98.65:52273
Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.639233 8658
slave.cpp:1760] Sending queued task '6' to executor
'235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework
235498ca-6603-4cfe-bfc7-94005bb235fb-1437
Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.608182 8658
slave.cpp:2717] Handling status update TASK_RUNNING (UUID:
ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework
235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from executor(1)@
10.130.98.65:52273
Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.612318 8658
status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID:
ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework
235498ca-6603-4cfe-bfc7-94005bb235fb-1437
Any advice is welcome.
Best Regards,
Mitsutoshi Kiuchi
Re: Spark task sometimes won't start
Posted by tommy xiao <xi...@gmail.com>.
maybe send to spark user mailist is better.
2015-11-23 22:30 GMT+08:00 木内満歳 <m-...@creationline.com>:
> Hi,
>
> I'm experiencing that some spark task on Mesos 0.25 occasionally won't
> start.
> Please tell some advice how to see more detail against it.
>
> Here is the slave log about bad task
>
> Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.677291 18516
> slave.cpp:2379] Got registration for executor
> '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1442 from executor(1)@
> 10.130.91.16:60295
> Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.679875 18516
> slave.cpp:1760] Sending queued task '0' to executor
> '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1442
> (no more log about this task)
>
> When task succeed to run, slave log shows like that.
>
> Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.637285 8658
> slave.cpp:2379] Got registration for executor
> '235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from executor(1)@
> 10.130.98.65:52273
> Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.639233 8658
> slave.cpp:1760] Sending queued task '6' to executor
> '235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437
> Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.608182 8658
> slave.cpp:2717] Handling status update TASK_RUNNING (UUID:
> ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from executor(1)@
> 10.130.98.65:52273
> Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.612318 8658
> status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID:
> ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437
>
> Any advice is welcome.
>
> Best Regards,
> Mitsutoshi Kiuchi
>
>
--
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com
Re: Spark task sometimes won't start
Posted by tommy xiao <xi...@gmail.com>.
I only use oarse grain mode in production, the fine-grained
mode can't flexible in any time.
2015-11-24 18:34 GMT+08:00 Rad Gruchalski <ra...@gruchalski.com>:
> Indeed, also, keep in mind, there’s a talk here that fine-grained mode
> might be removed in Spark 2.
>
> Kind regards,
> Radek Gruchalski
> radek@gruchalski.com <ra...@gruchalski.com>
> de.linkedin.com/in/radgruchalski/
>
>
> *Confidentiality:*This communication is intended for the above-named
> person and may be confidential and/or legally privileged.
> If it has come to you in error you must take no action based on it, nor
> must you copy or show it to anyone; please delete/destroy and inform the
> sender immediately.
>
> On Tuesday, 24 November 2015 at 11:32, 木内満歳 wrote:
>
> I agree. "coarse grain mode" is "take me to safe side" mode, I think.
> Unfortunately, it looks more hard to resolve...
>
> Anyway, I appreciate your advice. Thanks much !
>
> Mitsutoshi
>
>
> 2015-11-24 19:23 GMT+09:00 Rad Gruchalski <ra...@gruchalski.com>:
>
> Ah, I see, I am experiencing a similar thing with fine-grained where one
> of the tasks would stay in staging and fail the whole job but never in
> coarse mode.
>
> Kind regards,
> Radek Gruchalski
> radek@gruchalski.com <ra...@gruchalski.com>
> de.linkedin.com/in/radgruchalski/
>
>
> *Confidentiality:*This communication is intended for the above-named
> person and may be confidential and/or legally privileged.
> If it has come to you in error you must take no action based on it, nor
> must you copy or show it to anyone; please delete/destroy and inform the
> sender immediately.
>
> On Tuesday, 24 November 2015 at 10:07, 木内満歳 wrote:
>
> Hi Rad,
>
> I've tried both. I've experienced same symptom on both case.
>
> Thanks,
> Mitsutoshi Kiuchi
>
>
> 2015-11-24 17:57 GMT+09:00 Rad Gruchalski <ra...@gruchalski.com>:
>
> Mitsutoshi,
>
> Is this in a fine-grained mode?
>
> Kind regards,
> Radek Gruchalski
> radek@gruchalski.com <ra...@gruchalski.com>
> de.linkedin.com/in/radgruchalski/
>
>
> *Confidentiality:*This communication is intended for the above-named
> person and may be confidential and/or legally privileged.
> If it has come to you in error you must take no action based on it, nor
> must you copy or show it to anyone; please delete/destroy and inform the
> sender immediately.
>
> On Tuesday, 24 November 2015 at 07:18, 木内満歳 wrote:
>
> Hi, Tim
>
> I've reproduced and taken debug logs(attached).
> I cannot understand what is going on, but it seems that the slave is
> repeatedly sending ACCEPT message to master.
>
> Please have your comment.
>
> Best Regards,
> Mitsutoshi Kiuchi
>
>
> 2015-11-24 5:28 GMT+09:00 Tim Chen <ti...@mesosphere.io>:
>
> Hi Mitsutoshi,
>
> Can you enable TRACING log on Spark (modify your log4j.properties file)?
>
> It should have more information on why offers are being rejected, but most
> of the time it's due to not enough resources in your cluster to satifsy
> launching your Spark job. You can either increase your slave(s) resources
> or lower your cpu/memory requirement for your job through configuration.
>
> Tim
>
> On Mon, Nov 23, 2015 at 6:30 AM, 木内満歳 <m-...@creationline.com> wrote:
>
> Hi,
>
> I'm experiencing that some spark task on Mesos 0.25 occasionally won't
> start.
> Please tell some advice how to see more detail against it.
>
> Here is the slave log about bad task
>
> Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.677291 18516
> slave.cpp:2379] Got registration for executor
> '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1442 from executor(1)@
> 10.130.91.16:60295
> Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.679875 18516
> slave.cpp:1760] Sending queued task '0' to executor
> '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1442
> (no more log about this task)
>
> When task succeed to run, slave log shows like that.
>
> Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.637285 8658
> slave.cpp:2379] Got registration for executor
> '235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from executor(1)@
> 10.130.98.65:52273
> Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.639233 8658
> slave.cpp:1760] Sending queued task '6' to executor
> '235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437
> Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.608182 8658
> slave.cpp:2717] Handling status update TASK_RUNNING (UUID:
> ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from executor(1)@
> 10.130.98.65:52273
> Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.612318 8658
> status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID:
> ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437
>
> Any advice is welcome.
>
> Best Regards,
> Mitsutoshi Kiuchi
>
>
>
>
> Attachments:
> - log.driverStdErr.gz
> - log.mesosMaster.gz
> - log.mesosSlave.gz
>
>
>
>
>
>
>
--
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com
Re: Spark task sometimes won't start
Posted by Rad Gruchalski <ra...@gruchalski.com>.
Indeed, also, keep in mind, there’s a talk here that fine-grained mode might be removed in Spark 2.
Kind regards,
Radek Gruchalski
radek@gruchalski.com (mailto:radek@gruchalski.com)
(mailto:radek@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
Confidentiality:
This communication is intended for the above-named person and may be confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately.
On Tuesday, 24 November 2015 at 11:32, 木内満歳 wrote:
> I agree. "coarse grain mode" is "take me to safe side" mode, I think.
> Unfortunately, it looks more hard to resolve...
>
> Anyway, I appreciate your advice. Thanks much !
>
> Mitsutoshi
>
>
> 2015-11-24 19:23 GMT+09:00 Rad Gruchalski <radek@gruchalski.com (mailto:radek@gruchalski.com)>:
> > Ah, I see, I am experiencing a similar thing with fine-grained where one of the tasks would stay in staging and fail the whole job but never in coarse mode.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Kind regards,
> > Radek Gruchalski
> >
radek@gruchalski.com (mailto:radek@gruchalski.com)
(mailto:radek@gruchalski.com)
> > de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
> >
> > Confidentiality:
> > This communication is intended for the above-named person and may be confidential and/or legally privileged.
> > If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately.
> >
> >
> >
> > On Tuesday, 24 November 2015 at 10:07, 木内満歳 wrote:
> >
> > > Hi Rad,
> > >
> > > I've tried both. I've experienced same symptom on both case.
> > >
> > > Thanks,
> > > Mitsutoshi Kiuchi
> > >
> > >
> > > 2015-11-24 17:57 GMT+09:00 Rad Gruchalski <radek@gruchalski.com (mailto:radek@gruchalski.com)>:
> > > > Mitsutoshi,
> > > >
> > > > Is this in a fine-grained mode?
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Kind regards,
> > > > Radek Gruchalski
> > > >
radek@gruchalski.com (mailto:radek@gruchalski.com)
(mailto:radek@gruchalski.com)
> > > > de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
> > > >
> > > > Confidentiality:
> > > > This communication is intended for the above-named person and may be confidential and/or legally privileged.
> > > > If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately.
> > > >
> > > >
> > > >
> > > > On Tuesday, 24 November 2015 at 07:18, 木内満歳 wrote:
> > > >
> > > > > Hi, Tim
> > > > >
> > > > > I've reproduced and taken debug logs(attached).
> > > > > I cannot understand what is going on, but it seems that the slave is repeatedly sending ACCEPT message to master.
> > > > >
> > > > > Please have your comment.
> > > > >
> > > > > Best Regards,
> > > > > Mitsutoshi Kiuchi
> > > > >
> > > > >
> > > > > 2015-11-24 5:28 GMT+09:00 Tim Chen <tim@mesosphere.io (mailto:tim@mesosphere.io)>:
> > > > > > Hi Mitsutoshi,
> > > > > >
> > > > > > Can you enable TRACING log on Spark (modify your log4j.properties file)?
> > > > > >
> > > > > > It should have more information on why offers are being rejected, but most of the time it's due to not enough resources in your cluster to satifsy launching your Spark job. You can either increase your slave(s) resources or lower your cpu/memory requirement for your job through configuration.
> > > > > >
> > > > > > Tim
> > > > > >
> > > > > > On Mon, Nov 23, 2015 at 6:30 AM, 木内満歳 <m-kiuchi@creationline.com (mailto:m-kiuchi@creationline.com)> wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > I'm experiencing that some spark task on Mesos 0.25 occasionally won't start.
> > > > > > > Please tell some advice how to see more detail against it.
> > > > > > >
> > > > > > > Here is the slave log about bad task
> > > > > > >
> > > > > > > Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.677291 18516 slave.cpp:2379] Got registration for executor '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework 235498ca-6603-4cfe-bfc7-94005bb235fb-1442 from executor(1)@10.130.91.16:60295 (http://10.130.91.16:60295)
> > > > > > > Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.679875 18516 slave.cpp:1760] Sending queued task '0' to executor '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework 235498ca-6603-4cfe-bfc7-94005bb235fb-1442
> > > > > > > (no more log about this task)
> > > > > > >
> > > > > > > When task succeed to run, slave log shows like that.
> > > > > > >
> > > > > > > Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.637285 8658 slave.cpp:2379] Got registration for executor '235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework 235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from executor(1)@10.130.98.65:52273 (http://10.130.98.65:52273)
> > > > > > > Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.639233 8658 slave.cpp:1760] Sending queued task '6' to executor '235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework 235498ca-6603-4cfe-bfc7-94005bb235fb-1437
> > > > > > > Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.608182 8658 slave.cpp:2717] Handling status update TASK_RUNNING (UUID: ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework 235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from executor(1)@10.130.98.65:52273 (http://10.130.98.65:52273)
> > > > > > > Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.612318 8658 status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID: ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework 235498ca-6603-4cfe-bfc7-94005bb235fb-1437
> > > > > > >
> > > > > > > Any advice is welcome.
> > > > > > >
> > > > > > > Best Regards,
> > > > > > > Mitsutoshi Kiuchi
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > Attachments:
> > > > > - log.driverStdErr.gz
> > > > >
> > > > > - log.mesosMaster.gz
> > > > >
> > > > > - log.mesosSlave.gz
> > > > >
> > > > >
> > > >
> > > >
> > >
> >
>
Re: Spark task sometimes won't start
Posted by 木内満歳 <m-...@creationline.com>.
I agree. "coarse grain mode" is "take me to safe side" mode, I think.
Unfortunately, it looks more hard to resolve...
Anyway, I appreciate your advice. Thanks much !
Mitsutoshi
2015-11-24 19:23 GMT+09:00 Rad Gruchalski <ra...@gruchalski.com>:
> Ah, I see, I am experiencing a similar thing with fine-grained where one
> of the tasks would stay in staging and fail the whole job but never in
> coarse mode.
>
> Kind regards,
> Radek Gruchalski
> radek@gruchalski.com <ra...@gruchalski.com>
> de.linkedin.com/in/radgruchalski/
>
>
> *Confidentiality:*This communication is intended for the above-named
> person and may be confidential and/or legally privileged.
> If it has come to you in error you must take no action based on it, nor
> must you copy or show it to anyone; please delete/destroy and inform the
> sender immediately.
>
> On Tuesday, 24 November 2015 at 10:07, 木内満歳 wrote:
>
> Hi Rad,
>
> I've tried both. I've experienced same symptom on both case.
>
> Thanks,
> Mitsutoshi Kiuchi
>
>
> 2015-11-24 17:57 GMT+09:00 Rad Gruchalski <ra...@gruchalski.com>:
>
> Mitsutoshi,
>
> Is this in a fine-grained mode?
>
> Kind regards,
> Radek Gruchalski
> radek@gruchalski.com <ra...@gruchalski.com>
> de.linkedin.com/in/radgruchalski/
>
>
> *Confidentiality:*This communication is intended for the above-named
> person and may be confidential and/or legally privileged.
> If it has come to you in error you must take no action based on it, nor
> must you copy or show it to anyone; please delete/destroy and inform the
> sender immediately.
>
> On Tuesday, 24 November 2015 at 07:18, 木内満歳 wrote:
>
> Hi, Tim
>
> I've reproduced and taken debug logs(attached).
> I cannot understand what is going on, but it seems that the slave is
> repeatedly sending ACCEPT message to master.
>
> Please have your comment.
>
> Best Regards,
> Mitsutoshi Kiuchi
>
>
> 2015-11-24 5:28 GMT+09:00 Tim Chen <ti...@mesosphere.io>:
>
> Hi Mitsutoshi,
>
> Can you enable TRACING log on Spark (modify your log4j.properties file)?
>
> It should have more information on why offers are being rejected, but most
> of the time it's due to not enough resources in your cluster to satifsy
> launching your Spark job. You can either increase your slave(s) resources
> or lower your cpu/memory requirement for your job through configuration.
>
> Tim
>
> On Mon, Nov 23, 2015 at 6:30 AM, 木内満歳 <m-...@creationline.com> wrote:
>
> Hi,
>
> I'm experiencing that some spark task on Mesos 0.25 occasionally won't
> start.
> Please tell some advice how to see more detail against it.
>
> Here is the slave log about bad task
>
> Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.677291 18516
> slave.cpp:2379] Got registration for executor
> '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1442 from executor(1)@
> 10.130.91.16:60295
> Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.679875 18516
> slave.cpp:1760] Sending queued task '0' to executor
> '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1442
> (no more log about this task)
>
> When task succeed to run, slave log shows like that.
>
> Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.637285 8658
> slave.cpp:2379] Got registration for executor
> '235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from executor(1)@
> 10.130.98.65:52273
> Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.639233 8658
> slave.cpp:1760] Sending queued task '6' to executor
> '235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437
> Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.608182 8658
> slave.cpp:2717] Handling status update TASK_RUNNING (UUID:
> ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from executor(1)@
> 10.130.98.65:52273
> Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.612318 8658
> status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID:
> ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437
>
> Any advice is welcome.
>
> Best Regards,
> Mitsutoshi Kiuchi
>
>
>
>
> Attachments:
> - log.driverStdErr.gz
> - log.mesosMaster.gz
> - log.mesosSlave.gz
>
>
>
>
>
Re: Spark task sometimes won't start
Posted by Rad Gruchalski <ra...@gruchalski.com>.
Ah, I see, I am experiencing a similar thing with fine-grained where one of the tasks would stay in staging and fail the whole job but never in coarse mode.
Kind regards,
Radek Gruchalski
radek@gruchalski.com (mailto:radek@gruchalski.com)
(mailto:radek@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
Confidentiality:
This communication is intended for the above-named person and may be confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately.
On Tuesday, 24 November 2015 at 10:07, 木内満歳 wrote:
> Hi Rad,
>
> I've tried both. I've experienced same symptom on both case.
>
> Thanks,
> Mitsutoshi Kiuchi
>
>
> 2015-11-24 17:57 GMT+09:00 Rad Gruchalski <radek@gruchalski.com (mailto:radek@gruchalski.com)>:
> > Mitsutoshi,
> >
> > Is this in a fine-grained mode?
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Kind regards,
> > Radek Gruchalski
> >
radek@gruchalski.com (mailto:radek@gruchalski.com)
(mailto:radek@gruchalski.com)
> > de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
> >
> > Confidentiality:
> > This communication is intended for the above-named person and may be confidential and/or legally privileged.
> > If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately.
> >
> >
> >
> > On Tuesday, 24 November 2015 at 07:18, 木内満歳 wrote:
> >
> > > Hi, Tim
> > >
> > > I've reproduced and taken debug logs(attached).
> > > I cannot understand what is going on, but it seems that the slave is repeatedly sending ACCEPT message to master.
> > >
> > > Please have your comment.
> > >
> > > Best Regards,
> > > Mitsutoshi Kiuchi
> > >
> > >
> > > 2015-11-24 5:28 GMT+09:00 Tim Chen <tim@mesosphere.io (mailto:tim@mesosphere.io)>:
> > > > Hi Mitsutoshi,
> > > >
> > > > Can you enable TRACING log on Spark (modify your log4j.properties file)?
> > > >
> > > > It should have more information on why offers are being rejected, but most of the time it's due to not enough resources in your cluster to satifsy launching your Spark job. You can either increase your slave(s) resources or lower your cpu/memory requirement for your job through configuration.
> > > >
> > > > Tim
> > > >
> > > > On Mon, Nov 23, 2015 at 6:30 AM, 木内満歳 <m-kiuchi@creationline.com (mailto:m-kiuchi@creationline.com)> wrote:
> > > > > Hi,
> > > > >
> > > > > I'm experiencing that some spark task on Mesos 0.25 occasionally won't start.
> > > > > Please tell some advice how to see more detail against it.
> > > > >
> > > > > Here is the slave log about bad task
> > > > >
> > > > > Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.677291 18516 slave.cpp:2379] Got registration for executor '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework 235498ca-6603-4cfe-bfc7-94005bb235fb-1442 from executor(1)@10.130.91.16:60295 (http://10.130.91.16:60295)
> > > > > Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.679875 18516 slave.cpp:1760] Sending queued task '0' to executor '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework 235498ca-6603-4cfe-bfc7-94005bb235fb-1442
> > > > > (no more log about this task)
> > > > >
> > > > > When task succeed to run, slave log shows like that.
> > > > >
> > > > > Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.637285 8658 slave.cpp:2379] Got registration for executor '235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework 235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from executor(1)@10.130.98.65:52273 (http://10.130.98.65:52273)
> > > > > Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.639233 8658 slave.cpp:1760] Sending queued task '6' to executor '235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework 235498ca-6603-4cfe-bfc7-94005bb235fb-1437
> > > > > Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.608182 8658 slave.cpp:2717] Handling status update TASK_RUNNING (UUID: ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework 235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from executor(1)@10.130.98.65:52273 (http://10.130.98.65:52273)
> > > > > Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.612318 8658 status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID: ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework 235498ca-6603-4cfe-bfc7-94005bb235fb-1437
> > > > >
> > > > > Any advice is welcome.
> > > > >
> > > > > Best Regards,
> > > > > Mitsutoshi Kiuchi
> > > > >
> > > >
> > >
> > >
> > > Attachments:
> > > - log.driverStdErr.gz
> > >
> > > - log.mesosMaster.gz
> > >
> > > - log.mesosSlave.gz
> > >
> > >
> >
> >
>
Re: Spark task sometimes won't start
Posted by 木内満歳 <m-...@creationline.com>.
Hi Rad,
I've tried both. I've experienced same symptom on both case.
Thanks,
Mitsutoshi Kiuchi
2015-11-24 17:57 GMT+09:00 Rad Gruchalski <ra...@gruchalski.com>:
> Mitsutoshi,
>
> Is this in a fine-grained mode?
>
> Kind regards,
> Radek Gruchalski
> radek@gruchalski.com <ra...@gruchalski.com>
> de.linkedin.com/in/radgruchalski/
>
>
> *Confidentiality:*This communication is intended for the above-named
> person and may be confidential and/or legally privileged.
> If it has come to you in error you must take no action based on it, nor
> must you copy or show it to anyone; please delete/destroy and inform the
> sender immediately.
>
> On Tuesday, 24 November 2015 at 07:18, 木内満歳 wrote:
>
> Hi, Tim
>
> I've reproduced and taken debug logs(attached).
> I cannot understand what is going on, but it seems that the slave is
> repeatedly sending ACCEPT message to master.
>
> Please have your comment.
>
> Best Regards,
> Mitsutoshi Kiuchi
>
>
> 2015-11-24 5:28 GMT+09:00 Tim Chen <ti...@mesosphere.io>:
>
> Hi Mitsutoshi,
>
> Can you enable TRACING log on Spark (modify your log4j.properties file)?
>
> It should have more information on why offers are being rejected, but most
> of the time it's due to not enough resources in your cluster to satifsy
> launching your Spark job. You can either increase your slave(s) resources
> or lower your cpu/memory requirement for your job through configuration.
>
> Tim
>
> On Mon, Nov 23, 2015 at 6:30 AM, 木内満歳 <m-...@creationline.com> wrote:
>
> Hi,
>
> I'm experiencing that some spark task on Mesos 0.25 occasionally won't
> start.
> Please tell some advice how to see more detail against it.
>
> Here is the slave log about bad task
>
> Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.677291 18516
> slave.cpp:2379] Got registration for executor
> '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1442 from executor(1)@
> 10.130.91.16:60295
> Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.679875 18516
> slave.cpp:1760] Sending queued task '0' to executor
> '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1442
> (no more log about this task)
>
> When task succeed to run, slave log shows like that.
>
> Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.637285 8658
> slave.cpp:2379] Got registration for executor
> '235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from executor(1)@
> 10.130.98.65:52273
> Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.639233 8658
> slave.cpp:1760] Sending queued task '6' to executor
> '235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437
> Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.608182 8658
> slave.cpp:2717] Handling status update TASK_RUNNING (UUID:
> ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from executor(1)@
> 10.130.98.65:52273
> Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.612318 8658
> status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID:
> ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437
>
> Any advice is welcome.
>
> Best Regards,
> Mitsutoshi Kiuchi
>
>
>
>
> Attachments:
> - log.driverStdErr.gz
> - log.mesosMaster.gz
> - log.mesosSlave.gz
>
>
>
Re: Spark task sometimes won't start
Posted by Rad Gruchalski <ra...@gruchalski.com>.
Mitsutoshi,
Is this in a fine-grained mode?
Kind regards,
Radek Gruchalski
radek@gruchalski.com (mailto:radek@gruchalski.com)
(mailto:radek@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
Confidentiality:
This communication is intended for the above-named person and may be confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately.
On Tuesday, 24 November 2015 at 07:18, 木内満歳 wrote:
> Hi, Tim
>
> I've reproduced and taken debug logs(attached).
> I cannot understand what is going on, but it seems that the slave is repeatedly sending ACCEPT message to master.
>
> Please have your comment.
>
> Best Regards,
> Mitsutoshi Kiuchi
>
>
> 2015-11-24 5:28 GMT+09:00 Tim Chen <tim@mesosphere.io (mailto:tim@mesosphere.io)>:
> > Hi Mitsutoshi,
> >
> > Can you enable TRACING log on Spark (modify your log4j.properties file)?
> >
> > It should have more information on why offers are being rejected, but most of the time it's due to not enough resources in your cluster to satifsy launching your Spark job. You can either increase your slave(s) resources or lower your cpu/memory requirement for your job through configuration.
> >
> > Tim
> >
> > On Mon, Nov 23, 2015 at 6:30 AM, 木内満歳 <m-kiuchi@creationline.com (mailto:m-kiuchi@creationline.com)> wrote:
> > > Hi,
> > >
> > > I'm experiencing that some spark task on Mesos 0.25 occasionally won't start.
> > > Please tell some advice how to see more detail against it.
> > >
> > > Here is the slave log about bad task
> > >
> > > Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.677291 18516 slave.cpp:2379] Got registration for executor '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework 235498ca-6603-4cfe-bfc7-94005bb235fb-1442 from executor(1)@10.130.91.16:60295 (http://10.130.91.16:60295)
> > > Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.679875 18516 slave.cpp:1760] Sending queued task '0' to executor '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework 235498ca-6603-4cfe-bfc7-94005bb235fb-1442
> > > (no more log about this task)
> > >
> > > When task succeed to run, slave log shows like that.
> > >
> > > Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.637285 8658 slave.cpp:2379] Got registration for executor '235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework 235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from executor(1)@10.130.98.65:52273 (http://10.130.98.65:52273)
> > > Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.639233 8658 slave.cpp:1760] Sending queued task '6' to executor '235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework 235498ca-6603-4cfe-bfc7-94005bb235fb-1437
> > > Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.608182 8658 slave.cpp:2717] Handling status update TASK_RUNNING (UUID: ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework 235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from executor(1)@10.130.98.65:52273 (http://10.130.98.65:52273)
> > > Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.612318 8658 status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID: ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework 235498ca-6603-4cfe-bfc7-94005bb235fb-1437
> > >
> > > Any advice is welcome.
> > >
> > > Best Regards,
> > > Mitsutoshi Kiuchi
> > >
> >
>
>
> Attachments:
> - log.driverStdErr.gz
>
> - log.mesosMaster.gz
>
> - log.mesosSlave.gz
>
Re: Spark task sometimes won't start
Posted by 木内満歳 <m-...@creationline.com>.
I've acquired the debug log on slave side(attached).
It seems that MesosExecutorBackend lanched with null backend.
15/11/24 22:31:26 ERROR MesosExecutorBackend: Received launchTask but
executor was null
Anyway, I'll increase memory size of each slave.
Best Regards,
Mitsutoshi Kiuchi
2015-11-24 15:18 GMT+09:00 木内満歳 <m-...@creationline.com>:
> Hi, Tim
>
> I've reproduced and taken debug logs(attached).
> I cannot understand what is going on, but it seems that the slave is
> repeatedly sending ACCEPT message to master.
>
> Please have your comment.
>
> Best Regards,
> Mitsutoshi Kiuchi
>
>
> 2015-11-24 5:28 GMT+09:00 Tim Chen <ti...@mesosphere.io>:
>
>> Hi Mitsutoshi,
>>
>> Can you enable TRACING log on Spark (modify your log4j.properties file)?
>>
>> It should have more information on why offers are being rejected, but
>> most of the time it's due to not enough resources in your cluster to
>> satifsy launching your Spark job. You can either increase your slave(s)
>> resources or lower your cpu/memory requirement for your job through
>> configuration.
>>
>> Tim
>>
>> On Mon, Nov 23, 2015 at 6:30 AM, 木内満歳 <m-...@creationline.com> wrote:
>>
>>> Hi,
>>>
>>> I'm experiencing that some spark task on Mesos 0.25 occasionally won't
>>> start.
>>> Please tell some advice how to see more detail against it.
>>>
>>> Here is the slave log about bad task
>>>
>>> Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.677291 18516
>>> slave.cpp:2379] Got registration for executor
>>> '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework
>>> 235498ca-6603-4cfe-bfc7-94005bb235fb-1442 from executor(1)@
>>> 10.130.91.16:60295
>>> Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.679875 18516
>>> slave.cpp:1760] Sending queued task '0' to executor
>>> '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework
>>> 235498ca-6603-4cfe-bfc7-94005bb235fb-1442
>>> (no more log about this task)
>>>
>>> When task succeed to run, slave log shows like that.
>>>
>>> Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.637285
>>> 8658 slave.cpp:2379] Got registration for executor
>>> '235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework
>>> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from executor(1)@
>>> 10.130.98.65:52273
>>> Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.639233
>>> 8658 slave.cpp:1760] Sending queued task '6' to executor
>>> '235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework
>>> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437
>>> Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.608182
>>> 8658 slave.cpp:2717] Handling status update TASK_RUNNING (UUID:
>>> ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework
>>> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from executor(1)@
>>> 10.130.98.65:52273
>>> Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.612318
>>> 8658 status_update_manager.cpp:322] Received status update TASK_RUNNING
>>> (UUID: ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework
>>> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437
>>>
>>> Any advice is welcome.
>>>
>>> Best Regards,
>>> Mitsutoshi Kiuchi
>>>
>>>
>>
>
Re: Spark task sometimes won't start
Posted by 木内満歳 <m-...@creationline.com>.
Hi, Tim
I've reproduced and taken debug logs(attached).
I cannot understand what is going on, but it seems that the slave is
repeatedly sending ACCEPT message to master.
Please have your comment.
Best Regards,
Mitsutoshi Kiuchi
2015-11-24 5:28 GMT+09:00 Tim Chen <ti...@mesosphere.io>:
> Hi Mitsutoshi,
>
> Can you enable TRACING log on Spark (modify your log4j.properties file)?
>
> It should have more information on why offers are being rejected, but most
> of the time it's due to not enough resources in your cluster to satifsy
> launching your Spark job. You can either increase your slave(s) resources
> or lower your cpu/memory requirement for your job through configuration.
>
> Tim
>
> On Mon, Nov 23, 2015 at 6:30 AM, 木内満歳 <m-...@creationline.com> wrote:
>
>> Hi,
>>
>> I'm experiencing that some spark task on Mesos 0.25 occasionally won't
>> start.
>> Please tell some advice how to see more detail against it.
>>
>> Here is the slave log about bad task
>>
>> Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.677291 18516
>> slave.cpp:2379] Got registration for executor
>> '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework
>> 235498ca-6603-4cfe-bfc7-94005bb235fb-1442 from executor(1)@
>> 10.130.91.16:60295
>> Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.679875 18516
>> slave.cpp:1760] Sending queued task '0' to executor
>> '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework
>> 235498ca-6603-4cfe-bfc7-94005bb235fb-1442
>> (no more log about this task)
>>
>> When task succeed to run, slave log shows like that.
>>
>> Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.637285
>> 8658 slave.cpp:2379] Got registration for executor
>> '235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework
>> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from executor(1)@
>> 10.130.98.65:52273
>> Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.639233
>> 8658 slave.cpp:1760] Sending queued task '6' to executor
>> '235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework
>> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437
>> Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.608182
>> 8658 slave.cpp:2717] Handling status update TASK_RUNNING (UUID:
>> ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework
>> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from executor(1)@
>> 10.130.98.65:52273
>> Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.612318
>> 8658 status_update_manager.cpp:322] Received status update TASK_RUNNING
>> (UUID: ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework
>> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437
>>
>> Any advice is welcome.
>>
>> Best Regards,
>> Mitsutoshi Kiuchi
>>
>>
>
Re: Spark task sometimes won't start
Posted by Tim Chen <ti...@mesosphere.io>.
Hi Mitsutoshi,
Can you enable TRACING log on Spark (modify your log4j.properties file)?
It should have more information on why offers are being rejected, but most
of the time it's due to not enough resources in your cluster to satifsy
launching your Spark job. You can either increase your slave(s) resources
or lower your cpu/memory requirement for your job through configuration.
Tim
On Mon, Nov 23, 2015 at 6:30 AM, 木内満歳 <m-...@creationline.com> wrote:
> Hi,
>
> I'm experiencing that some spark task on Mesos 0.25 occasionally won't
> start.
> Please tell some advice how to see more detail against it.
>
> Here is the slave log about bad task
>
> Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.677291 18516
> slave.cpp:2379] Got registration for executor
> '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1442 from executor(1)@
> 10.130.91.16:60295
> Nov 23 08:54:26 mesos-s2 mesos-slave[18499]: I1123 08:54:26.679875 18516
> slave.cpp:1760] Sending queued task '0' to executor
> '235498ca-6603-4cfe-bfc7-94005bb235fb-S5' of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1442
> (no more log about this task)
>
> When task succeed to run, slave log shows like that.
>
> Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.637285 8658
> slave.cpp:2379] Got registration for executor
> '235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from executor(1)@
> 10.130.98.65:52273
> Nov 23 08:44:39 al-mesos-s3 mesos-slave[8644]: I1123 08:44:39.639233 8658
> slave.cpp:1760] Sending queued task '6' to executor
> '235498ca-6603-4cfe-bfc7-94005bb235fb-S6' of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437
> Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.608182 8658
> slave.cpp:2717] Handling status update TASK_RUNNING (UUID:
> ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437 from executor(1)@
> 10.130.98.65:52273
> Nov 23 08:44:42 al-mesos-s3 mesos-slave[8644]: I1123 08:44:42.612318 8658
> status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID:
> ff5a2278-0753-4541-bd33-a55f3a09fb69) for task 6 of framework
> 235498ca-6603-4cfe-bfc7-94005bb235fb-1437
>
> Any advice is welcome.
>
> Best Regards,
> Mitsutoshi Kiuchi
>
>