You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by yanlin wang <ya...@me.com> on 2016/03/04 22:37:19 UTC
spark driver in docker
We would like to run multiple spark driver in docker container. Any suggestion for the port expose and network settings for docker so driver is reachable by the worker nodes? —net=“hosts” is the last thing we want to do.
Thx
Yanlin
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark driver in docker
Posted by Timothy Chen <ti...@mesosphere.io>.
Will need more information to help you, what's the commands you used to launch slave/master, and what error message did you see in the driver logs?
Tim
> On Mar 5, 2016, at 4:34 AM, Mailing List <as...@gmail.com> wrote:
>
> I am trying to do the same but till now no luck...
> I have everything running inside docker container including mesos master, mesos slave , marathon , spark mesos cluster dispatcher.
>
> But when I try to submit the job using spark submit as a docker container it fails ...
>
> Between this setup is on single centos 7 machine.
>
> Let me know if you have any insights as how to make the submit work.
>
> Ashish
>
> Sent from my iPad
>
>> On Mar 5, 2016, at 3:33 AM, Tamas Szuromi <ta...@odigeo.com.INVALID> wrote:
>>
>> Hi, Have a look at on http://spark.apache.org/docs/latest/configuration.html what ports need to be exposed. With mesos we had a lot of problems with container networking but yes the --net=host is a shortcut.
>>
>> Tamas
>>
>>
>>
>>> On 4 March 2016 at 22:37, yanlin wang <ya...@me.com> wrote:
>>> We would like to run multiple spark driver in docker container. Any suggestion for the port expose and network settings for docker so driver is reachable by the worker nodes? —net=“hosts” is the last thing we want to do.
>>>
>>> Thx
>>> Yanlin
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>
Re: spark driver in docker
Posted by Mailing List <as...@gmail.com>.
I am trying to do the same but till now no luck...
I have everything running inside docker container including mesos master, mesos slave , marathon , spark mesos cluster dispatcher.
But when I try to submit the job using spark submit as a docker container it fails ...
Between this setup is on single centos 7 machine.
Let me know if you have any insights as how to make the submit work.
Ashish
Sent from my iPad
> On Mar 5, 2016, at 3:33 AM, Tamas Szuromi <ta...@odigeo.com.INVALID> wrote:
>
> Hi, Have a look at on http://spark.apache.org/docs/latest/configuration.html what ports need to be exposed. With mesos we had a lot of problems with container networking but yes the --net=host is a shortcut.
>
> Tamas
>
>
>
>> On 4 March 2016 at 22:37, yanlin wang <ya...@me.com> wrote:
>> We would like to run multiple spark driver in docker container. Any suggestion for the port expose and network settings for docker so driver is reachable by the worker nodes? —net=“hosts” is the last thing we want to do.
>>
>> Thx
>> Yanlin
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>
Re: spark driver in docker
Posted by Tamas Szuromi <ta...@odigeo.com.INVALID>.
Hi, Have a look at on
http://spark.apache.org/docs/latest/configuration.html what
ports need to be exposed. With mesos we had a lot of problems with
container networking but yes the --net=host is a shortcut.
Tamas
On 4 March 2016 at 22:37, yanlin wang <ya...@me.com> wrote:
> We would like to run multiple spark driver in docker container. Any
> suggestion for the port expose and network settings for docker so driver is
> reachable by the worker nodes? —net=“hosts” is the last thing we want to do.
>
> Thx
> Yanlin
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>