You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Franco Maria Nardini <fr...@isti.cnr.it> on 2013/08/16 17:16:19 UTC
problem with mesos slaves and spark
Hi all,
when I run a simple example on my new mesos/spark cluster I get this log on
the slave nodes.
I0816 16:49:02.749058 6429 slave.cpp:436] Got assigned task 0 for
framework 201308161531542257298-5050-13853-0005
I0816 16:49:02.749480 6429 slave.cpp:1484] Generating a unique work
directory for executor '201308161531542257298-5050-13853-9' of framework
201308161531542257298-5050-13853-0005
I0816 16:49:02.750033 6429 slave.cpp:501] Using
'/tmp/mesos/slaves/201308161531542257298-5050-13853-9/frameworks/201308161531542257298-5050-13853-0005/executors/201308161531542257298-5050-13853-9/runs/0'
as work directory for executor '201308161531542257298-5050-13853-9' of
framework 201308161531542257298-5050-13853-0005
I0816 16:49:02.753378 6429 slave.cpp:436] Got assigned task 1 for
framework 201308161531542257298-5050-13853-0005
I0816 16:49:02.753449 6429 slave.cpp:472] Queuing task '1' for executor
201308161531542257298-5050-13853-9 of framework
'201308161531542257298-5050-13853-0005
I0816 16:49:02.753518 6426 process_based_isolation_module.cpp:91]
Launching 201308161531542257298-5050-13853-9 (/home/spark/spark-executor)
in
/tmp/mesos/slaves/201308161531542257298-5050-13853-9/frameworks/201308161531542257298-5050-13853-0005/executors/201308161531542257298-5050-13853-9/runs/0
with resources mem=512' for framework 201308161531542257298-5050-13853-0005
I0816 16:49:02.755309 6426 process_based_isolation_module.cpp:114] Forked
executor at 6459
I0816 16:49:03.395040 6434 process_based_isolation_module.cpp:207] Telling
slave of lost executor 201308161531542257298-5050-13853-9 of framework
201308161531542257298-5050-13853-0005
I0816 16:49:03.395249 6426 slave.cpp:1307] Executor
'201308161531542257298-5050-13853-9' of framework
201308161531542257298-5050-13853-0005 has exited with status 127
/home/spark/BINs/mesos-0.9.0/libexec/mesos/killtree.sh: 94:
/home/spark/BINs/mesos-0.9.0/libexec/mesos/killtree.sh: Syntax error: Bad
fd number
I0816 16:49:03.434700 6443 slave.cpp:436] Got assigned task 2 for
framework 201308161531542257298-5050-13853-0005
I0816 16:49:03.435022 6443 slave.cpp:1484] Generating a unique work
directory for executor '201308161531542257298-5050-13853-9' of framework
201308161531542257298-5050-13853-0005
I0816 16:49:03.435271 6443 slave.cpp:501] Using
'/tmp/mesos/slaves/201308161531542257298-5050-13853-9/frameworks/201308161531542257298-5050-13853-0005/executors/201308161531542257298-5050-13853-9/runs/1'
as work directory for executor '201308161531542257298-5050-13853-9' of
framework 201308161531542257298-5050-13853-0005
I0816 16:49:03.438539 6443 slave.cpp:436] Got assigned task 3 for
framework 201308161531542257298-5050-13853-0005
I0816 16:49:03.438614 6443 slave.cpp:472] Queuing task '3' for executor
201308161531542257298-5050-13853-9 of framework
'201308161531542257298-5050-13853-0005
I0816 16:49:03.438657 6445 process_based_isolation_module.cpp:91]
Launching 201308161531542257298-5050-13853-9 (/home/spark/spark-executor)
in
/tmp/mesos/slaves/201308161531542257298-5050-13853-9/frameworks/201308161531542257298-5050-13853-0005/executors/201308161531542257298-5050-13853-9/runs/1
with resources mem=512' for framework 201308161531542257298-5050-13853-0005
I0816 16:49:03.440618 6445 process_based_isolation_module.cpp:114] Forked
executor at 6464
I0816 16:49:04.395736 6450 process_based_isolation_module.cpp:207] Telling
slave of lost executor 201308161531542257298-5050-13853-9 of framework
201308161531542257298-5050-13853-0005
I0816 16:49:04.396492 6455 slave.cpp:1307] Executor
'201308161531542257298-5050-13853-9' of framework
201308161531542257298-5050-13853-0005 has exited with status 127
/home/spark/BINs/mesos-0.9.0/libexec/mesos/killtree.sh: 94:
/home/spark/BINs/mesos-0.9.0/libexec/mesos/killtree.sh: Syntax error: Bad
fd number
and the stderr file in
"/tmp/mesos/slaves/201308161531542257298-5050-13853-9/frameworks/201308161531542257298-5050-13853-0005/executors/201308161531542257298-5050-13853-9/runs/1"
contains this error:
"sh: 1: /home/spark/spark-executor: not found"
How can I tell mesos-slave the right path to the spark-executor? Secondly,
do you have any idea of how to solve the second error:
"/home/spark/BINs/mesos-0.9.0/libexec/mesos/killtree.sh: 94:
/home/spark/BINs/mesos-0.9.0/libexec/mesos/killtree.sh: Syntax error: Bad
fd number"
Thanks a lot for your help.
Best,
Franco Maria Nardini
Re: problem with mesos slaves and spark
Posted by Vinod Kone <vi...@gmail.com>.
Hey Franco,
Mesos-0.9.0 is really old and no longer supported. The latest stable
version is 0.12.1. You should give it a try!
For the spark executor question, its probably best to ping spark's mailing
list.
Cheers,
On Fri, Aug 16, 2013 at 8:16 AM, Franco Maria Nardini <
francomaria.nardini@isti.cnr.it> wrote:
> Hi all,
>
> when I run a simple example on my new mesos/spark cluster I get this log
> on the slave nodes.
>
> I0816 16:49:02.749058 6429 slave.cpp:436] Got assigned task 0 for
> framework 201308161531542257298-5050-13853-0005
> I0816 16:49:02.749480 6429 slave.cpp:1484] Generating a unique work
> directory for executor '201308161531542257298-5050-13853-9' of framework
> 201308161531542257298-5050-13853-0005
> I0816 16:49:02.750033 6429 slave.cpp:501] Using
> '/tmp/mesos/slaves/201308161531542257298-5050-13853-9/frameworks/201308161531542257298-5050-13853-0005/executors/201308161531542257298-5050-13853-9/runs/0'
> as work directory for executor '201308161531542257298-5050-13853-9' of
> framework 201308161531542257298-5050-13853-0005
> I0816 16:49:02.753378 6429 slave.cpp:436] Got assigned task 1 for
> framework 201308161531542257298-5050-13853-0005
> I0816 16:49:02.753449 6429 slave.cpp:472] Queuing task '1' for executor
> 201308161531542257298-5050-13853-9 of framework
> '201308161531542257298-5050-13853-0005
> I0816 16:49:02.753518 6426 process_based_isolation_module.cpp:91]
> Launching 201308161531542257298-5050-13853-9 (/home/spark/spark-executor)
> in
> /tmp/mesos/slaves/201308161531542257298-5050-13853-9/frameworks/201308161531542257298-5050-13853-0005/executors/201308161531542257298-5050-13853-9/runs/0
> with resources mem=512' for framework 201308161531542257298-5050-13853-0005
> I0816 16:49:02.755309 6426 process_based_isolation_module.cpp:114] Forked
> executor at 6459
> I0816 16:49:03.395040 6434 process_based_isolation_module.cpp:207]
> Telling slave of lost executor 201308161531542257298-5050-13853-9 of
> framework 201308161531542257298-5050-13853-0005
> I0816 16:49:03.395249 6426 slave.cpp:1307] Executor
> '201308161531542257298-5050-13853-9' of framework
> 201308161531542257298-5050-13853-0005 has exited with status 127
> /home/spark/BINs/mesos-0.9.0/libexec/mesos/killtree.sh: 94:
> /home/spark/BINs/mesos-0.9.0/libexec/mesos/killtree.sh: Syntax error: Bad
> fd number
> I0816 16:49:03.434700 6443 slave.cpp:436] Got assigned task 2 for
> framework 201308161531542257298-5050-13853-0005
> I0816 16:49:03.435022 6443 slave.cpp:1484] Generating a unique work
> directory for executor '201308161531542257298-5050-13853-9' of framework
> 201308161531542257298-5050-13853-0005
> I0816 16:49:03.435271 6443 slave.cpp:501] Using
> '/tmp/mesos/slaves/201308161531542257298-5050-13853-9/frameworks/201308161531542257298-5050-13853-0005/executors/201308161531542257298-5050-13853-9/runs/1'
> as work directory for executor '201308161531542257298-5050-13853-9' of
> framework 201308161531542257298-5050-13853-0005
> I0816 16:49:03.438539 6443 slave.cpp:436] Got assigned task 3 for
> framework 201308161531542257298-5050-13853-0005
> I0816 16:49:03.438614 6443 slave.cpp:472] Queuing task '3' for executor
> 201308161531542257298-5050-13853-9 of framework
> '201308161531542257298-5050-13853-0005
> I0816 16:49:03.438657 6445 process_based_isolation_module.cpp:91]
> Launching 201308161531542257298-5050-13853-9 (/home/spark/spark-executor)
> in
> /tmp/mesos/slaves/201308161531542257298-5050-13853-9/frameworks/201308161531542257298-5050-13853-0005/executors/201308161531542257298-5050-13853-9/runs/1
> with resources mem=512' for framework 201308161531542257298-5050-13853-0005
> I0816 16:49:03.440618 6445 process_based_isolation_module.cpp:114] Forked
> executor at 6464
> I0816 16:49:04.395736 6450 process_based_isolation_module.cpp:207]
> Telling slave of lost executor 201308161531542257298-5050-13853-9 of
> framework 201308161531542257298-5050-13853-0005
> I0816 16:49:04.396492 6455 slave.cpp:1307] Executor
> '201308161531542257298-5050-13853-9' of framework
> 201308161531542257298-5050-13853-0005 has exited with status 127
> /home/spark/BINs/mesos-0.9.0/libexec/mesos/killtree.sh: 94:
> /home/spark/BINs/mesos-0.9.0/libexec/mesos/killtree.sh: Syntax error: Bad
> fd number
>
> and the stderr file in
> "/tmp/mesos/slaves/201308161531542257298-5050-13853-9/frameworks/201308161531542257298-5050-13853-0005/executors/201308161531542257298-5050-13853-9/runs/1"
> contains this error:
>
> "sh: 1: /home/spark/spark-executor: not found"
>
> How can I tell mesos-slave the right path to the spark-executor? Secondly,
> do you have any idea of how to solve the second error:
> "/home/spark/BINs/mesos-0.9.0/libexec/mesos/killtree.sh: 94:
> /home/spark/BINs/mesos-0.9.0/libexec/mesos/killtree.sh: Syntax error: Bad
> fd number"
>
> Thanks a lot for your help.
>
> Best,
>
> Franco Maria Nardini
>