You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Dick Davies <di...@hellooperator.net> on 2015/08/23 21:20:06 UTC
is there a 'knack' to docker and mesos?
Really excited to try out the new Docker executor support on 1.4.1, I'm
making progress but feel like I'm missing something.
(versions:
spark-1.4.1-hadoop2.6 - not using hadoop yet
mac os x yosemite java 8 spark-shell
mesos 0.22.1 : 2 slaves, 1 master + zk , all on centos 6.x
docker 1.8.x
)
I wanted to use a generic docker image with e.g. Java 8 in it, and then
deploy the 1.4.1 distro into it. The docs seem to indicate that's supported.
Mesos and Docker pull the requested docker images ( 'docker pull java'
essentially)
and extract the spark.executor.uri distro correctly. It seems to fall
over at cd'ing of
all places:
"cd: can't cd to spark-1*"
this causes the task to fail and eventually have spark blacklist the slave.
Is this just because 'spark-1*' matches both the tarball and the directory?
full mesos stderr looks like:
--------------------------------------------
I0823 19:13:25.608206 3069 fetcher.cpp:214] Fetching URI
'http://d3kbcqa49mib13.cloudfront.net/spark-1.4.1-bin-hadoop2.6.tgz'
I0823 19:13:25.608582 3069 fetcher.cpp:125] Fetching URI
'http://d3kbcqa49mib13.cloudfront.net/spark-1.4.1-bin-hadoop2.6.tgz'
with os::net
I0823 19:13:25.608620 3069 fetcher.cpp:135] Downloading
'http://d3kbcqa49mib13.cloudfront.net/spark-1.4.1-bin-hadoop2.6.tgz'
to '/var/mesos/slaves/20150823-110659-1862270986-5050-3230-S1/frameworks/20150823-191138-1862270986-5050-3768-0000/executors/0/runs/a5a13cd4-013a-4ebc-8ef7-eb9c33358429/spark-1.4.1-bin-hadoop2.6.tgz'
I0823 19:14:37.765060 3069 fetcher.cpp:78] Extracted resource
'/var/mesos/slaves/20150823-110659-1862270986-5050-3230-S1/frameworks/20150823-191138-1862270986-5050-3768-0000/executors/0/runs/a5a13cd4-013a-4ebc-8ef7-eb9c33358429/spark-1.4.1-bin-hadoop2.6.tgz'
into '/var/mesos/slaves/20150823-110659-1862270986-5050-3230-S1/frameworks/20150823-191138-1862270986-5050-3768-0000/executors/0/runs/a5a13cd4-013a-4ebc-8ef7-eb9c33358429'
/bin/sh: 1: cd: can't cd to spark-1*
/bin/sh: 1: ./bin/spark-class: not found
I0823 19:14:38.365190 3138 exec.cpp:132] Version: 0.22.1
I0823 19:14:38.369495 3156 exec.cpp:206] Executor registered on slave
20150823-110659-1862270986-5050-3230-S1
-----------------------------------------------------
and the spark-shell says:
15/08/23 20:15:51 INFO CoarseMesosSchedulerBackend: Mesos task 3 is
now TASK_RUNNING
15/08/23 20:15:51 INFO CoarseMesosSchedulerBackend: Mesos task 3 is
now TASK_FAILED
15/08/23 20:15:51 INFO CoarseMesosSchedulerBackend: Blacklisting Mesos
slave value: "20150823-110659-1862270986-5050-3230-S0"
due to too many failures; is Spark installed on it?
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org