You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "g.eynard.bontemps@gmail.com" <g....@gmail.com> on 2016/02/17 23:00:46 UTC
Problem mixing MESOS Cluster Mode and Docker task execution
Hi everybody,
I am testing the use of Docker for executing Spark algorithms on MESOS. I
managed to execute Spark in client mode with executors inside Docker, but I
wanted to go further and have also my Driver running into a Docker
Container. Here I ran into a behavior that I'm not sure is normal, let me
try to explain.
I submit my spark application through MesosClusterDispatcher using a command
like:
$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master
mesos://spark-master-1:7077 --deploy-mode cluster --conf
spark.mesos.executor.docker.image=myuser/myimage:0.0.2
https://storage.googleapis.com/some-bucket/spark-examples-1.5.2-hadoop2.6.0.jar
10
My driver is running fine, inside its docker container, but the executors
fail:
"sh: /some/spark/home/bin/spark-class: No such file or directory"
Looking on MESOS slaves log, I think that the executors do not run inside
docker: "docker.cpp:775] No container info found, skipping launch". As my
Mesos slaves do not have spark installed, it fails.
*It seems that the spark conf that I gave in the first spark-submit is not
transmitted to the Driver submitted conf*, when launched in the docker
container. The only workaround I found is to modify my Docker image in order
to define inside its spark conf the spark.mesos.executor.docker.image
property. This way, my executors get the conf well and are launched inside
docker on Mesos. This seems a little complicated to me, and I feel the
configuration passed to the early spark-submit should be transmitted to the
Driver submit...
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-mixing-MESOS-Cluster-Mode-and-Docker-task-execution-tp26258.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Problem mixing MESOS Cluster Mode and Docker task execution
Posted by Tim Chen <ti...@mesosphere.io>.
Here is an example dockerfile, although it's a bit dated now if you build
it today it should still work:
https://github.com/tnachen/spark/tree/dockerfile/mesos_docker
Tim
On Thu, Mar 10, 2016 at 8:06 AM, Ashish Soni <as...@gmail.com> wrote:
> Hi Tim ,
>
> Can you please share your dockerfiles and configuration as it will help a
> lot , I am planing to publish a blog post on the same .
>
> Ashish
>
> On Thu, Mar 10, 2016 at 10:34 AM, Timothy Chen <ti...@mesosphere.io> wrote:
>
>> No you don't need to install spark on each slave, we have been running
>> this setup in Mesosphere without any problem at this point, I think most
>> likely configuration problem and perhaps a chance something is missing in
>> the code to handle some cases.
>>
>> What version of spark are you guys running? And also did you set the
>> working dir in your image to be spark home?
>>
>> Tim
>>
>>
>> On Mar 10, 2016, at 3:11 AM, Ashish Soni <as...@gmail.com> wrote:
>>
>> You need to install spark on each mesos slave and then while starting
>> container make a workdir to your spark home so that it can find the spark
>> class.
>>
>> Ashish
>>
>> On Mar 10, 2016, at 5:22 AM, Guillaume Eynard Bontemps <
>> g.eynard.bontemps@gmail.com> wrote:
>>
>> For an answer to my question see this:
>> http://stackoverflow.com/a/35660466?noredirect=1.
>>
>> But for your problem did you define the Spark.mesos.docker. home or
>> something like that property?
>>
>> Le jeu. 10 mars 2016 04:26, Eran Chinthaka Withana <
>> eran.chinthaka@gmail.com> a écrit :
>>
>>> Hi
>>>
>>> I'm also having this issue and can not get the tasks to work inside
>>> mesos.
>>>
>>> In my case, the spark-submit command is the following.
>>>
>>> $SPARK_HOME/bin/spark-submit \
>>> --class com.mycompany.SparkStarter \
>>> --master mesos://mesos-dispatcher:7077 \ --name SparkStarterJob \
>>> --driver-memory 1G \
>>> --executor-memory 4G \
>>> --deploy-mode cluster \
>>> --total-executor-cores 1 \
>>> --conf spark.mesos.executor.docker.image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6 \
>>> http://abc.com/spark-starter.jar
>>>
>>>
>>> And the error I'm getting is the following.
>>>
>>> I0310 03:13:11.417009 131594 exec.cpp:132] Version: 0.23.1
>>> I0310 03:13:11.419452 131601 exec.cpp:206] Executor registered on slave 20160223-000314-3439362570-5050-631-S0
>>> sh: 1: /usr/spark-1.6.0-bin-hadoop2.6/bin/spark-class: not found
>>>
>>>
>>> (Looked into Spark JIRA and I found that
>>> https://issues.apache.org/jira/browse/SPARK-11759 is marked as closed
>>> since https://issues.apache.org/jira/browse/SPARK-12345 is marked as
>>> resolved)
>>>
>>> Really appreciate if I can get some help here.
>>>
>>> Thanks,
>>> Eran Chinthaka Withana
>>>
>>> On Wed, Feb 17, 2016 at 2:00 PM, g.eynard.bontemps@gmail.com <
>>> g.eynard.bontemps@gmail.com> wrote:
>>>
>>>> Hi everybody,
>>>>
>>>> I am testing the use of Docker for executing Spark algorithms on MESOS.
>>>> I
>>>> managed to execute Spark in client mode with executors inside Docker,
>>>> but I
>>>> wanted to go further and have also my Driver running into a Docker
>>>> Container. Here I ran into a behavior that I'm not sure is normal, let
>>>> me
>>>> try to explain.
>>>>
>>>> I submit my spark application through MesosClusterDispatcher using a
>>>> command
>>>> like:
>>>> $ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master
>>>> mesos://spark-master-1:7077 --deploy-mode cluster --conf
>>>> spark.mesos.executor.docker.image=myuser/myimage:0.0.2
>>>>
>>>> https://storage.googleapis.com/some-bucket/spark-examples-1.5.2-hadoop2.6.0.jar
>>>> 10
>>>>
>>>> My driver is running fine, inside its docker container, but the
>>>> executors
>>>> fail:
>>>> "sh: /some/spark/home/bin/spark-class: No such file or directory"
>>>>
>>>> Looking on MESOS slaves log, I think that the executors do not run
>>>> inside
>>>> docker: "docker.cpp:775] No container info found, skipping launch". As
>>>> my
>>>> Mesos slaves do not have spark installed, it fails.
>>>>
>>>> *It seems that the spark conf that I gave in the first spark-submit is
>>>> not
>>>> transmitted to the Driver submitted conf*, when launched in the docker
>>>> container. The only workaround I found is to modify my Docker image in
>>>> order
>>>> to define inside its spark conf the spark.mesos.executor.docker.image
>>>> property. This way, my executors get the conf well and are launched
>>>> inside
>>>> docker on Mesos. This seems a little complicated to me, and I feel the
>>>> configuration passed to the early spark-submit should be transmitted to
>>>> the
>>>> Driver submit...
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Problem-mixing-MESOS-Cluster-Mode-and-Docker-task-execution-tp26258.html
>>>> Sent from the Apache Spark User List mailing list archive at Nabble.com
>>>> <http://nabble.com>.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>>
>>>
>
Re: Problem mixing MESOS Cluster Mode and Docker task execution
Posted by Ashish Soni <as...@gmail.com>.
Hi Tim ,
Can you please share your dockerfiles and configuration as it will help a
lot , I am planing to publish a blog post on the same .
Ashish
On Thu, Mar 10, 2016 at 10:34 AM, Timothy Chen <ti...@mesosphere.io> wrote:
> No you don't need to install spark on each slave, we have been running
> this setup in Mesosphere without any problem at this point, I think most
> likely configuration problem and perhaps a chance something is missing in
> the code to handle some cases.
>
> What version of spark are you guys running? And also did you set the
> working dir in your image to be spark home?
>
> Tim
>
>
> On Mar 10, 2016, at 3:11 AM, Ashish Soni <as...@gmail.com> wrote:
>
> You need to install spark on each mesos slave and then while starting
> container make a workdir to your spark home so that it can find the spark
> class.
>
> Ashish
>
> On Mar 10, 2016, at 5:22 AM, Guillaume Eynard Bontemps <
> g.eynard.bontemps@gmail.com> wrote:
>
> For an answer to my question see this:
> http://stackoverflow.com/a/35660466?noredirect=1.
>
> But for your problem did you define the Spark.mesos.docker. home or
> something like that property?
>
> Le jeu. 10 mars 2016 04:26, Eran Chinthaka Withana <
> eran.chinthaka@gmail.com> a écrit :
>
>> Hi
>>
>> I'm also having this issue and can not get the tasks to work inside mesos.
>>
>> In my case, the spark-submit command is the following.
>>
>> $SPARK_HOME/bin/spark-submit \
>> --class com.mycompany.SparkStarter \
>> --master mesos://mesos-dispatcher:7077 \ --name SparkStarterJob \
>> --driver-memory 1G \
>> --executor-memory 4G \
>> --deploy-mode cluster \
>> --total-executor-cores 1 \
>> --conf spark.mesos.executor.docker.image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6 \
>> http://abc.com/spark-starter.jar
>>
>>
>> And the error I'm getting is the following.
>>
>> I0310 03:13:11.417009 131594 exec.cpp:132] Version: 0.23.1
>> I0310 03:13:11.419452 131601 exec.cpp:206] Executor registered on slave 20160223-000314-3439362570-5050-631-S0
>> sh: 1: /usr/spark-1.6.0-bin-hadoop2.6/bin/spark-class: not found
>>
>>
>> (Looked into Spark JIRA and I found that
>> https://issues.apache.org/jira/browse/SPARK-11759 is marked as closed
>> since https://issues.apache.org/jira/browse/SPARK-12345 is marked as
>> resolved)
>>
>> Really appreciate if I can get some help here.
>>
>> Thanks,
>> Eran Chinthaka Withana
>>
>> On Wed, Feb 17, 2016 at 2:00 PM, g.eynard.bontemps@gmail.com <
>> g.eynard.bontemps@gmail.com> wrote:
>>
>>> Hi everybody,
>>>
>>> I am testing the use of Docker for executing Spark algorithms on MESOS. I
>>> managed to execute Spark in client mode with executors inside Docker,
>>> but I
>>> wanted to go further and have also my Driver running into a Docker
>>> Container. Here I ran into a behavior that I'm not sure is normal, let me
>>> try to explain.
>>>
>>> I submit my spark application through MesosClusterDispatcher using a
>>> command
>>> like:
>>> $ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master
>>> mesos://spark-master-1:7077 --deploy-mode cluster --conf
>>> spark.mesos.executor.docker.image=myuser/myimage:0.0.2
>>>
>>> https://storage.googleapis.com/some-bucket/spark-examples-1.5.2-hadoop2.6.0.jar
>>> 10
>>>
>>> My driver is running fine, inside its docker container, but the executors
>>> fail:
>>> "sh: /some/spark/home/bin/spark-class: No such file or directory"
>>>
>>> Looking on MESOS slaves log, I think that the executors do not run inside
>>> docker: "docker.cpp:775] No container info found, skipping launch". As my
>>> Mesos slaves do not have spark installed, it fails.
>>>
>>> *It seems that the spark conf that I gave in the first spark-submit is
>>> not
>>> transmitted to the Driver submitted conf*, when launched in the docker
>>> container. The only workaround I found is to modify my Docker image in
>>> order
>>> to define inside its spark conf the spark.mesos.executor.docker.image
>>> property. This way, my executors get the conf well and are launched
>>> inside
>>> docker on Mesos. This seems a little complicated to me, and I feel the
>>> configuration passed to the early spark-submit should be transmitted to
>>> the
>>> Driver submit...
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/Problem-mixing-MESOS-Cluster-Mode-and-Docker-task-execution-tp26258.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com
>>> <http://nabble.com>.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
Re: Problem mixing MESOS Cluster Mode and Docker task execution
Posted by Ashish Soni <as...@gmail.com>.
When you say driver running on mesos can you explain how are you doing that...??
> On Mar 10, 2016, at 4:44 PM, Eran Chinthaka Withana <er...@gmail.com> wrote:
>
> Yanling I'm already running the driver on mesos (through docker). FYI, I'm running this on cluster mode with MesosClusterDispatcher.
>
> Mac (client) ----> MesosClusterDispatcher ----> Driver running on Mesos --> Workers running on Mesos
>
> My next step is to run MesosClusterDispatcher in mesos through marathon.
>
> Thanks,
> Eran Chinthaka Withana
>
>> On Thu, Mar 10, 2016 at 11:11 AM, yanlin wang <ya...@me.com> wrote:
>> How you guys make driver docker within container to be reachable from spark worker ?
>>
>> Would you share your driver docker? i am trying to put only driver in docker and spark running with yarn outside of container and i don’t want to use —net=host
>>
>> Thx
>> Yanlin
>>
>>> On Mar 10, 2016, at 11:06 AM, Guillaume Eynard Bontemps <g....@gmail.com> wrote:
>>>
>>> Glad to hear it. Thanks all for sharing your solutions.
>>>
>>>
>>> Le jeu. 10 mars 2016 19:19, Eran Chinthaka Withana <er...@gmail.com> a écrit :
>>>> Phew, it worked. All I had to do was to add export SPARK_JAVA_OPTS="-Dspark.mesos.executor.docker.image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6" before calling spark-submit. Guillaume, thanks for the pointer.
>>>>
>>>> Timothy, thanks for looking into this. Looking forward to see a fix soon.
>>>>
>>>> Thanks,
>>>> Eran Chinthaka Withana
>>>>
>>>>> On Thu, Mar 10, 2016 at 10:10 AM, Tim Chen <ti...@mesosphere.io> wrote:
>>>>> Hi Eran,
>>>>>
>>>>> I need to investigate but perhaps that's true, we're using SPARK_JAVA_OPTS to pass all the options and not --conf.
>>>>>
>>>>> I'll take a look at the bug, but if you can try the workaround and see if that fixes your problem.
>>>>>
>>>>> Tim
>>>>>
>>>>>> On Thu, Mar 10, 2016 at 10:08 AM, Eran Chinthaka Withana <er...@gmail.com> wrote:
>>>>>> Hi Timothy
>>>>>>
>>>>>>> What version of spark are you guys running?
>>>>>>
>>>>>> I'm using Spark 1.6.0. You can see the Dockerfile I used here: https://github.com/echinthaka/spark-mesos-docker/blob/master/docker/mesos-spark/Dockerfile
>>>>>>
>>>>>>> And also did you set the working dir in your image to be spark home?
>>>>>>
>>>>>> Yes I did. You can see it here: https://goo.gl/8PxtV8
>>>>>>
>>>>>> Can it be because of this: https://issues.apache.org/jira/browse/SPARK-13258 as Guillaume pointed out above? As you can see, I'm passing in the docker image URI through spark-submit (--conf spark.mesos.executor.docker.image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6)
>>>>>>
>>>>>> Thanks,
>>>>>> Eran
>
Re: Problem mixing MESOS Cluster Mode and Docker task execution
Posted by Eran Chinthaka Withana <er...@gmail.com>.
Yanling I'm already running the driver on mesos (through docker). FYI, I'm
running this on cluster mode with MesosClusterDispatcher.
Mac (client) ----> MesosClusterDispatcher ----> Driver running on Mesos -->
Workers running on Mesos
My next step is to run MesosClusterDispatcher in mesos through marathon.
Thanks,
Eran Chinthaka Withana
On Thu, Mar 10, 2016 at 11:11 AM, yanlin wang <ya...@me.com> wrote:
> How you guys make driver docker within container to be reachable from
> spark worker ?
>
> Would you share your driver docker? i am trying to put only driver in
> docker and spark running with yarn outside of container and i don’t want to
> use —net=host
>
> Thx
> Yanlin
>
> On Mar 10, 2016, at 11:06 AM, Guillaume Eynard Bontemps <
> g.eynard.bontemps@gmail.com> wrote:
>
> Glad to hear it. Thanks all for sharing your solutions.
>
> Le jeu. 10 mars 2016 19:19, Eran Chinthaka Withana <
> eran.chinthaka@gmail.com> a écrit :
>
>> Phew, it worked. All I had to do was to add *export
>> SPARK_JAVA_OPTS="-Dspark.mesos.executor.docker.image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6"
>> *before calling spark-submit. Guillaume, thanks for the pointer.
>>
>> Timothy, thanks for looking into this. Looking forward to see a fix soon.
>>
>> Thanks,
>> Eran Chinthaka Withana
>>
>> On Thu, Mar 10, 2016 at 10:10 AM, Tim Chen <ti...@mesosphere.io> wrote:
>>
>>> Hi Eran,
>>>
>>> I need to investigate but perhaps that's true, we're using
>>> SPARK_JAVA_OPTS to pass all the options and not --conf.
>>>
>>> I'll take a look at the bug, but if you can try the workaround and see
>>> if that fixes your problem.
>>>
>>> Tim
>>>
>>> On Thu, Mar 10, 2016 at 10:08 AM, Eran Chinthaka Withana <
>>> eran.chinthaka@gmail.com> wrote:
>>>
>>>> Hi Timothy
>>>>
>>>> What version of spark are you guys running?
>>>>>
>>>>
>>>> I'm using Spark 1.6.0. You can see the Dockerfile I used here:
>>>> https://github.com/echinthaka/spark-mesos-docker/blob/master/docker/mesos-spark/Dockerfile
>>>>
>>>>
>>>>
>>>>> And also did you set the working dir in your image to be spark home?
>>>>>
>>>>
>>>> Yes I did. You can see it here: https://goo.gl/8PxtV8
>>>>
>>>> Can it be because of this:
>>>> https://issues.apache.org/jira/browse/SPARK-13258 as Guillaume pointed
>>>> out above? As you can see, I'm passing in the docker image URI through
>>>> spark-submit (--conf spark.mesos.executor.docker.
>>>> image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6)
>>>>
>>>> Thanks,
>>>> Eran
>>>>
>>>>
>>>>
>>>
>>
>
Re: Problem mixing MESOS Cluster Mode and Docker task execution
Posted by Ajay Chander <it...@gmail.com>.
Hi Everyone, a quick question with in this context. What is the underneath
persistent storage that you guys are using? With regards to this
containerized environment? Thanks
On Thursday, March 10, 2016, yanlin wang <ya...@me.com> wrote:
> How you guys make driver docker within container to be reachable from
> spark worker ?
>
> Would you share your driver docker? i am trying to put only driver in
> docker and spark running with yarn outside of container and i don’t want to
> use —net=host
>
> Thx
> Yanlin
>
> On Mar 10, 2016, at 11:06 AM, Guillaume Eynard Bontemps <
> g.eynard.bontemps@gmail.com
> <javascript:_e(%7B%7D,'cvml','g.eynard.bontemps@gmail.com');>> wrote:
>
> Glad to hear it. Thanks all for sharing your solutions.
>
> Le jeu. 10 mars 2016 19:19, Eran Chinthaka Withana <
> eran.chinthaka@gmail.com
> <javascript:_e(%7B%7D,'cvml','eran.chinthaka@gmail.com');>> a écrit :
>
>> Phew, it worked. All I had to do was to add *export
>> SPARK_JAVA_OPTS="-Dspark.mesos.executor.docker.image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6"
>> *before calling spark-submit. Guillaume, thanks for the pointer.
>>
>> Timothy, thanks for looking into this. Looking forward to see a fix soon.
>>
>> Thanks,
>> Eran Chinthaka Withana
>>
>> On Thu, Mar 10, 2016 at 10:10 AM, Tim Chen <tim@mesosphere.io
>> <javascript:_e(%7B%7D,'cvml','tim@mesosphere.io');>> wrote:
>>
>>> Hi Eran,
>>>
>>> I need to investigate but perhaps that's true, we're using
>>> SPARK_JAVA_OPTS to pass all the options and not --conf.
>>>
>>> I'll take a look at the bug, but if you can try the workaround and see
>>> if that fixes your problem.
>>>
>>> Tim
>>>
>>> On Thu, Mar 10, 2016 at 10:08 AM, Eran Chinthaka Withana <
>>> eran.chinthaka@gmail.com
>>> <javascript:_e(%7B%7D,'cvml','eran.chinthaka@gmail.com');>> wrote:
>>>
>>>> Hi Timothy
>>>>
>>>> What version of spark are you guys running?
>>>>>
>>>>
>>>> I'm using Spark 1.6.0. You can see the Dockerfile I used here:
>>>> https://github.com/echinthaka/spark-mesos-docker/blob/master/docker/mesos-spark/Dockerfile
>>>>
>>>>
>>>>
>>>>> And also did you set the working dir in your image to be spark home?
>>>>>
>>>>
>>>> Yes I did. You can see it here: https://goo.gl/8PxtV8
>>>>
>>>> Can it be because of this:
>>>> https://issues.apache.org/jira/browse/SPARK-13258 as Guillaume pointed
>>>> out above? As you can see, I'm passing in the docker image URI through
>>>> spark-submit (--conf spark.mesos.executor.docker.
>>>> image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6)
>>>>
>>>> Thanks,
>>>> Eran
>>>>
>>>>
>>>>
>>>
>>
>
Re: Problem mixing MESOS Cluster Mode and Docker task execution
Posted by yanlin wang <ya...@me.com>.
How you guys make driver docker within container to be reachable from spark worker ?
Would you share your driver docker? i am trying to put only driver in docker and spark running with yarn outside of container and i don’t want to use —net=host
Thx
Yanlin
> On Mar 10, 2016, at 11:06 AM, Guillaume Eynard Bontemps <g....@gmail.com> wrote:
>
> Glad to hear it. Thanks all for sharing your solutions.
>
>
> Le jeu. 10 mars 2016 19:19, Eran Chinthaka Withana <eran.chinthaka@gmail.com <ma...@gmail.com>> a écrit :
> Phew, it worked. All I had to do was to add export SPARK_JAVA_OPTS="-Dspark.mesos.executor.docker.image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6" before calling spark-submit. Guillaume, thanks for the pointer.
>
> Timothy, thanks for looking into this. Looking forward to see a fix soon.
>
> Thanks,
> Eran Chinthaka Withana
>
> On Thu, Mar 10, 2016 at 10:10 AM, Tim Chen <tim@mesosphere.io <ma...@mesosphere.io>> wrote:
> Hi Eran,
>
> I need to investigate but perhaps that's true, we're using SPARK_JAVA_OPTS to pass all the options and not --conf.
>
> I'll take a look at the bug, but if you can try the workaround and see if that fixes your problem.
>
> Tim
>
> On Thu, Mar 10, 2016 at 10:08 AM, Eran Chinthaka Withana <eran.chinthaka@gmail.com <ma...@gmail.com>> wrote:
> Hi Timothy
>
> What version of spark are you guys running?
>
> I'm using Spark 1.6.0. You can see the Dockerfile I used here: https://github.com/echinthaka/spark-mesos-docker/blob/master/docker/mesos-spark/Dockerfile <https://github.com/echinthaka/spark-mesos-docker/blob/master/docker/mesos-spark/Dockerfile>
>
> And also did you set the working dir in your image to be spark home?
>
> Yes I did. You can see it here: https://goo.gl/8PxtV8 <https://goo.gl/8PxtV8>
>
> Can it be because of this: https://issues.apache.org/jira/browse/SPARK-13258 <https://issues.apache.org/jira/browse/SPARK-13258> as Guillaume pointed out above? As you can see, I'm passing in the docker image URI through spark-submit (--conf spark.mesos.executor.docker.image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6)
>
> Thanks,
> Eran
>
>
>
>
Re: Problem mixing MESOS Cluster Mode and Docker task execution
Posted by Guillaume Eynard Bontemps <g....@gmail.com>.
Glad to hear it. Thanks all for sharing your solutions.
Le jeu. 10 mars 2016 19:19, Eran Chinthaka Withana <er...@gmail.com>
a écrit :
> Phew, it worked. All I had to do was to add *export
> SPARK_JAVA_OPTS="-Dspark.mesos.executor.docker.image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6"
> *before calling spark-submit. Guillaume, thanks for the pointer.
>
> Timothy, thanks for looking into this. Looking forward to see a fix soon.
>
> Thanks,
> Eran Chinthaka Withana
>
> On Thu, Mar 10, 2016 at 10:10 AM, Tim Chen <ti...@mesosphere.io> wrote:
>
>> Hi Eran,
>>
>> I need to investigate but perhaps that's true, we're using
>> SPARK_JAVA_OPTS to pass all the options and not --conf.
>>
>> I'll take a look at the bug, but if you can try the workaround and see if
>> that fixes your problem.
>>
>> Tim
>>
>> On Thu, Mar 10, 2016 at 10:08 AM, Eran Chinthaka Withana <
>> eran.chinthaka@gmail.com> wrote:
>>
>>> Hi Timothy
>>>
>>> What version of spark are you guys running?
>>>>
>>>
>>> I'm using Spark 1.6.0. You can see the Dockerfile I used here:
>>> https://github.com/echinthaka/spark-mesos-docker/blob/master/docker/mesos-spark/Dockerfile
>>>
>>>
>>>
>>>> And also did you set the working dir in your image to be spark home?
>>>>
>>>
>>> Yes I did. You can see it here: https://goo.gl/8PxtV8
>>>
>>> Can it be because of this:
>>> https://issues.apache.org/jira/browse/SPARK-13258 as Guillaume pointed
>>> out above? As you can see, I'm passing in the docker image URI through
>>> spark-submit (--conf spark.mesos.executor.docker.
>>> image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6)
>>>
>>> Thanks,
>>> Eran
>>>
>>>
>>>
>>
>
Re: Problem mixing MESOS Cluster Mode and Docker task execution
Posted by Eran Chinthaka Withana <er...@gmail.com>.
Phew, it worked. All I had to do was to add *export
SPARK_JAVA_OPTS="-Dspark.mesos.executor.docker.image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6"
*before calling spark-submit. Guillaume, thanks for the pointer.
Timothy, thanks for looking into this. Looking forward to see a fix soon.
Thanks,
Eran Chinthaka Withana
On Thu, Mar 10, 2016 at 10:10 AM, Tim Chen <ti...@mesosphere.io> wrote:
> Hi Eran,
>
> I need to investigate but perhaps that's true, we're using SPARK_JAVA_OPTS
> to pass all the options and not --conf.
>
> I'll take a look at the bug, but if you can try the workaround and see if
> that fixes your problem.
>
> Tim
>
> On Thu, Mar 10, 2016 at 10:08 AM, Eran Chinthaka Withana <
> eran.chinthaka@gmail.com> wrote:
>
>> Hi Timothy
>>
>> What version of spark are you guys running?
>>>
>>
>> I'm using Spark 1.6.0. You can see the Dockerfile I used here:
>> https://github.com/echinthaka/spark-mesos-docker/blob/master/docker/mesos-spark/Dockerfile
>>
>>
>>
>>> And also did you set the working dir in your image to be spark home?
>>>
>>
>> Yes I did. You can see it here: https://goo.gl/8PxtV8
>>
>> Can it be because of this:
>> https://issues.apache.org/jira/browse/SPARK-13258 as Guillaume pointed
>> out above? As you can see, I'm passing in the docker image URI through
>> spark-submit (--conf spark.mesos.executor.docker.
>> image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6)
>>
>> Thanks,
>> Eran
>>
>>
>>
>
Re: Problem mixing MESOS Cluster Mode and Docker task execution
Posted by Tim Chen <ti...@mesosphere.io>.
Hi Eran,
I need to investigate but perhaps that's true, we're using SPARK_JAVA_OPTS
to pass all the options and not --conf.
I'll take a look at the bug, but if you can try the workaround and see if
that fixes your problem.
Tim
On Thu, Mar 10, 2016 at 10:08 AM, Eran Chinthaka Withana <
eran.chinthaka@gmail.com> wrote:
> Hi Timothy
>
> What version of spark are you guys running?
>>
>
> I'm using Spark 1.6.0. You can see the Dockerfile I used here:
> https://github.com/echinthaka/spark-mesos-docker/blob/master/docker/mesos-spark/Dockerfile
>
>
>
>> And also did you set the working dir in your image to be spark home?
>>
>
> Yes I did. You can see it here: https://goo.gl/8PxtV8
>
> Can it be because of this:
> https://issues.apache.org/jira/browse/SPARK-13258 as Guillaume pointed
> out above? As you can see, I'm passing in the docker image URI through
> spark-submit (--conf spark.mesos.executor.docker.
> image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6)
>
> Thanks,
> Eran
>
>
>
Re: Problem mixing MESOS Cluster Mode and Docker task execution
Posted by Eran Chinthaka Withana <er...@gmail.com>.
Hi Timothy
What version of spark are you guys running?
>
I'm using Spark 1.6.0. You can see the Dockerfile I used here:
https://github.com/echinthaka/spark-mesos-docker/blob/master/docker/mesos-spark/Dockerfile
> And also did you set the working dir in your image to be spark home?
>
Yes I did. You can see it here: https://goo.gl/8PxtV8
Can it be because of this: https://issues.apache.org/jira/browse/SPARK-13258
as Guillaume pointed out above? As you can see, I'm passing in the docker
image URI through spark-submit (--conf spark.mesos.executor.docker.
image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6)
Thanks,
Eran
Re: Problem mixing MESOS Cluster Mode and Docker task execution
Posted by Timothy Chen <ti...@mesosphere.io>.
No you don't need to install spark on each slave, we have been running this setup in Mesosphere without any problem at this point, I think most likely configuration problem and perhaps a chance something is missing in the code to handle some cases.
What version of spark are you guys running? And also did you set the working dir in your image to be spark home?
Tim
> On Mar 10, 2016, at 3:11 AM, Ashish Soni <as...@gmail.com> wrote:
>
> You need to install spark on each mesos slave and then while starting container make a workdir to your spark home so that it can find the spark class.
>
> Ashish
>
>> On Mar 10, 2016, at 5:22 AM, Guillaume Eynard Bontemps <g....@gmail.com> wrote:
>>
>> For an answer to my question see this: http://stackoverflow.com/a/35660466?noredirect=1.
>>
>> But for your problem did you define the Spark.mesos.docker. home or something like that property?
>>
>>
>> Le jeu. 10 mars 2016 04:26, Eran Chinthaka Withana <er...@gmail.com> a écrit :
>>> Hi
>>>
>>> I'm also having this issue and can not get the tasks to work inside mesos.
>>>
>>> In my case, the spark-submit command is the following.
>>>
>>> $SPARK_HOME/bin/spark-submit \
>>> --class com.mycompany.SparkStarter \
>>> --master mesos://mesos-dispatcher:7077 \
>>> --name SparkStarterJob \
>>> --driver-memory 1G \
>>> --executor-memory 4G \
>>> --deploy-mode cluster \
>>> --total-executor-cores 1 \
>>> --conf spark.mesos.executor.docker.image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6 \
>>> http://abc.com/spark-starter.jar
>>>
>>> And the error I'm getting is the following.
>>>
>>> I0310 03:13:11.417009 131594 exec.cpp:132] Version: 0.23.1
>>> I0310 03:13:11.419452 131601 exec.cpp:206] Executor registered on slave 20160223-000314-3439362570-5050-631-S0
>>> sh: 1: /usr/spark-1.6.0-bin-hadoop2.6/bin/spark-class: not found
>>>
>>> (Looked into Spark JIRA and I found that https://issues.apache.org/jira/browse/SPARK-11759 is marked as closed since https://issues.apache.org/jira/browse/SPARK-12345 is marked as resolved)
>>>
>>> Really appreciate if I can get some help here.
>>>
>>> Thanks,
>>> Eran Chinthaka Withana
>>>
>>>> On Wed, Feb 17, 2016 at 2:00 PM, g.eynard.bontemps@gmail.com <g....@gmail.com> wrote:
>>>> Hi everybody,
>>>>
>>>> I am testing the use of Docker for executing Spark algorithms on MESOS. I
>>>> managed to execute Spark in client mode with executors inside Docker, but I
>>>> wanted to go further and have also my Driver running into a Docker
>>>> Container. Here I ran into a behavior that I'm not sure is normal, let me
>>>> try to explain.
>>>>
>>>> I submit my spark application through MesosClusterDispatcher using a command
>>>> like:
>>>> $ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master
>>>> mesos://spark-master-1:7077 --deploy-mode cluster --conf
>>>> spark.mesos.executor.docker.image=myuser/myimage:0.0.2
>>>> https://storage.googleapis.com/some-bucket/spark-examples-1.5.2-hadoop2.6.0.jar
>>>> 10
>>>>
>>>> My driver is running fine, inside its docker container, but the executors
>>>> fail:
>>>> "sh: /some/spark/home/bin/spark-class: No such file or directory"
>>>>
>>>> Looking on MESOS slaves log, I think that the executors do not run inside
>>>> docker: "docker.cpp:775] No container info found, skipping launch". As my
>>>> Mesos slaves do not have spark installed, it fails.
>>>>
>>>> *It seems that the spark conf that I gave in the first spark-submit is not
>>>> transmitted to the Driver submitted conf*, when launched in the docker
>>>> container. The only workaround I found is to modify my Docker image in order
>>>> to define inside its spark conf the spark.mesos.executor.docker.image
>>>> property. This way, my executors get the conf well and are launched inside
>>>> docker on Mesos. This seems a little complicated to me, and I feel the
>>>> configuration passed to the early spark-submit should be transmitted to the
>>>> Driver submit...
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-mixing-MESOS-Cluster-Mode-and-Docker-task-execution-tp26258.html
>>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
Re: Problem mixing MESOS Cluster Mode and Docker task execution
Posted by Ashish Soni <as...@gmail.com>.
You need to install spark on each mesos slave and then while starting container make a workdir to your spark home so that it can find the spark class.
Ashish
> On Mar 10, 2016, at 5:22 AM, Guillaume Eynard Bontemps <g....@gmail.com> wrote:
>
> For an answer to my question see this: http://stackoverflow.com/a/35660466?noredirect=1.
>
> But for your problem did you define the Spark.mesos.docker. home or something like that property?
>
>
> Le jeu. 10 mars 2016 04:26, Eran Chinthaka Withana <er...@gmail.com> a écrit :
>> Hi
>>
>> I'm also having this issue and can not get the tasks to work inside mesos.
>>
>> In my case, the spark-submit command is the following.
>>
>> $SPARK_HOME/bin/spark-submit \
>> --class com.mycompany.SparkStarter \
>> --master mesos://mesos-dispatcher:7077 \
>> --name SparkStarterJob \
>> --driver-memory 1G \
>> --executor-memory 4G \
>> --deploy-mode cluster \
>> --total-executor-cores 1 \
>> --conf spark.mesos.executor.docker.image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6 \
>> http://abc.com/spark-starter.jar
>>
>> And the error I'm getting is the following.
>>
>> I0310 03:13:11.417009 131594 exec.cpp:132] Version: 0.23.1
>> I0310 03:13:11.419452 131601 exec.cpp:206] Executor registered on slave 20160223-000314-3439362570-5050-631-S0
>> sh: 1: /usr/spark-1.6.0-bin-hadoop2.6/bin/spark-class: not found
>>
>> (Looked into Spark JIRA and I found that https://issues.apache.org/jira/browse/SPARK-11759 is marked as closed since https://issues.apache.org/jira/browse/SPARK-12345 is marked as resolved)
>>
>> Really appreciate if I can get some help here.
>>
>> Thanks,
>> Eran Chinthaka Withana
>>
>>> On Wed, Feb 17, 2016 at 2:00 PM, g.eynard.bontemps@gmail.com <g....@gmail.com> wrote:
>>> Hi everybody,
>>>
>>> I am testing the use of Docker for executing Spark algorithms on MESOS. I
>>> managed to execute Spark in client mode with executors inside Docker, but I
>>> wanted to go further and have also my Driver running into a Docker
>>> Container. Here I ran into a behavior that I'm not sure is normal, let me
>>> try to explain.
>>>
>>> I submit my spark application through MesosClusterDispatcher using a command
>>> like:
>>> $ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master
>>> mesos://spark-master-1:7077 --deploy-mode cluster --conf
>>> spark.mesos.executor.docker.image=myuser/myimage:0.0.2
>>> https://storage.googleapis.com/some-bucket/spark-examples-1.5.2-hadoop2.6.0.jar
>>> 10
>>>
>>> My driver is running fine, inside its docker container, but the executors
>>> fail:
>>> "sh: /some/spark/home/bin/spark-class: No such file or directory"
>>>
>>> Looking on MESOS slaves log, I think that the executors do not run inside
>>> docker: "docker.cpp:775] No container info found, skipping launch". As my
>>> Mesos slaves do not have spark installed, it fails.
>>>
>>> *It seems that the spark conf that I gave in the first spark-submit is not
>>> transmitted to the Driver submitted conf*, when launched in the docker
>>> container. The only workaround I found is to modify my Docker image in order
>>> to define inside its spark conf the spark.mesos.executor.docker.image
>>> property. This way, my executors get the conf well and are launched inside
>>> docker on Mesos. This seems a little complicated to me, and I feel the
>>> configuration passed to the early spark-submit should be transmitted to the
>>> Driver submit...
>>>
>>>
>>>
>>> --
>>> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-mixing-MESOS-Cluster-Mode-and-Docker-task-execution-tp26258.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
Re: Problem mixing MESOS Cluster Mode and Docker task execution
Posted by Guillaume Eynard Bontemps <g....@gmail.com>.
For an answer to my question see this:
http://stackoverflow.com/a/35660466?noredirect=1.
But for your problem did you define the Spark.mesos.docker. home or
something like that property?
Le jeu. 10 mars 2016 04:26, Eran Chinthaka Withana <er...@gmail.com>
a écrit :
> Hi
>
> I'm also having this issue and can not get the tasks to work inside mesos.
>
> In my case, the spark-submit command is the following.
>
> $SPARK_HOME/bin/spark-submit \
> --class com.mycompany.SparkStarter \
> --master mesos://mesos-dispatcher:7077 \ --name SparkStarterJob \
> --driver-memory 1G \
> --executor-memory 4G \
> --deploy-mode cluster \
> --total-executor-cores 1 \
> --conf spark.mesos.executor.docker.image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6 \
> http://abc.com/spark-starter.jar
>
>
> And the error I'm getting is the following.
>
> I0310 03:13:11.417009 131594 exec.cpp:132] Version: 0.23.1
> I0310 03:13:11.419452 131601 exec.cpp:206] Executor registered on slave 20160223-000314-3439362570-5050-631-S0
> sh: 1: /usr/spark-1.6.0-bin-hadoop2.6/bin/spark-class: not found
>
>
> (Looked into Spark JIRA and I found that
> https://issues.apache.org/jira/browse/SPARK-11759 is marked as closed
> since https://issues.apache.org/jira/browse/SPARK-12345 is marked as
> resolved)
>
> Really appreciate if I can get some help here.
>
> Thanks,
> Eran Chinthaka Withana
>
> On Wed, Feb 17, 2016 at 2:00 PM, g.eynard.bontemps@gmail.com <
> g.eynard.bontemps@gmail.com> wrote:
>
>> Hi everybody,
>>
>> I am testing the use of Docker for executing Spark algorithms on MESOS. I
>> managed to execute Spark in client mode with executors inside Docker, but
>> I
>> wanted to go further and have also my Driver running into a Docker
>> Container. Here I ran into a behavior that I'm not sure is normal, let me
>> try to explain.
>>
>> I submit my spark application through MesosClusterDispatcher using a
>> command
>> like:
>> $ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master
>> mesos://spark-master-1:7077 --deploy-mode cluster --conf
>> spark.mesos.executor.docker.image=myuser/myimage:0.0.2
>>
>> https://storage.googleapis.com/some-bucket/spark-examples-1.5.2-hadoop2.6.0.jar
>> 10
>>
>> My driver is running fine, inside its docker container, but the executors
>> fail:
>> "sh: /some/spark/home/bin/spark-class: No such file or directory"
>>
>> Looking on MESOS slaves log, I think that the executors do not run inside
>> docker: "docker.cpp:775] No container info found, skipping launch". As my
>> Mesos slaves do not have spark installed, it fails.
>>
>> *It seems that the spark conf that I gave in the first spark-submit is not
>> transmitted to the Driver submitted conf*, when launched in the docker
>> container. The only workaround I found is to modify my Docker image in
>> order
>> to define inside its spark conf the spark.mesos.executor.docker.image
>> property. This way, my executors get the conf well and are launched inside
>> docker on Mesos. This seems a little complicated to me, and I feel the
>> configuration passed to the early spark-submit should be transmitted to
>> the
>> Driver submit...
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Problem-mixing-MESOS-Cluster-Mode-and-Docker-task-execution-tp26258.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: Problem mixing MESOS Cluster Mode and Docker task execution
Posted by Eran Chinthaka Withana <er...@gmail.com>.
Hi
I'm also having this issue and can not get the tasks to work inside mesos.
In my case, the spark-submit command is the following.
$SPARK_HOME/bin/spark-submit \
--class com.mycompany.SparkStarter \
--master mesos://mesos-dispatcher:7077 \ --name SparkStarterJob \
--driver-memory 1G \
--executor-memory 4G \
--deploy-mode cluster \
--total-executor-cores 1 \
--conf spark.mesos.executor.docker.image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6
\
http://abc.com/spark-starter.jar
And the error I'm getting is the following.
I0310 03:13:11.417009 131594 exec.cpp:132] Version: 0.23.1
I0310 03:13:11.419452 131601 exec.cpp:206] Executor registered on
slave 20160223-000314-3439362570-5050-631-S0
sh: 1: /usr/spark-1.6.0-bin-hadoop2.6/bin/spark-class: not found
(Looked into Spark JIRA and I found that
https://issues.apache.org/jira/browse/SPARK-11759 is marked as closed since
https://issues.apache.org/jira/browse/SPARK-12345 is marked as resolved)
Really appreciate if I can get some help here.
Thanks,
Eran Chinthaka Withana
On Wed, Feb 17, 2016 at 2:00 PM, g.eynard.bontemps@gmail.com <
g.eynard.bontemps@gmail.com> wrote:
> Hi everybody,
>
> I am testing the use of Docker for executing Spark algorithms on MESOS. I
> managed to execute Spark in client mode with executors inside Docker, but I
> wanted to go further and have also my Driver running into a Docker
> Container. Here I ran into a behavior that I'm not sure is normal, let me
> try to explain.
>
> I submit my spark application through MesosClusterDispatcher using a
> command
> like:
> $ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master
> mesos://spark-master-1:7077 --deploy-mode cluster --conf
> spark.mesos.executor.docker.image=myuser/myimage:0.0.2
>
> https://storage.googleapis.com/some-bucket/spark-examples-1.5.2-hadoop2.6.0.jar
> 10
>
> My driver is running fine, inside its docker container, but the executors
> fail:
> "sh: /some/spark/home/bin/spark-class: No such file or directory"
>
> Looking on MESOS slaves log, I think that the executors do not run inside
> docker: "docker.cpp:775] No container info found, skipping launch". As my
> Mesos slaves do not have spark installed, it fails.
>
> *It seems that the spark conf that I gave in the first spark-submit is not
> transmitted to the Driver submitted conf*, when launched in the docker
> container. The only workaround I found is to modify my Docker image in
> order
> to define inside its spark conf the spark.mesos.executor.docker.image
> property. This way, my executors get the conf well and are launched inside
> docker on Mesos. This seems a little complicated to me, and I feel the
> configuration passed to the early spark-submit should be transmitted to the
> Driver submit...
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Problem-mixing-MESOS-Cluster-Mode-and-Docker-task-execution-tp26258.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>