You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Dick Davies <di...@hellooperator.net> on 2014/12/03 18:46:48 UTC
dockerized spark executor on mesos?
Just wondered if anyone had managed to start spark
jobs on mesos wrapped in a docker container?
At present (i.e. very early testing) I'm able to submit executors
to mesos via spark-submit easily enough, but they fall over
as we don't have a JVM on our slaves out of the box.
I can push one out via our CM system if push comes to shove,
but it'd be nice to have that as part of the job (I'm thinking it might
be a way to get some of the dependencies deployed too).
bear in mind I'm a total clueless newbie at this so please be gentle
if I'm doing this completely wrong.
Thanks!
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: dockerized spark executor on mesos?
Posted by Dick Davies <di...@hellooperator.net>.
Oh great, thanks Tim - I'll give that a whirl then.
A Spark rollout isn't in our immediate future, I'm just looking for a good
framework for compute alongside our marathon deployment. So a good
time to experiment!
On 4 December 2014 at 02:36, Tim Chen <ti...@mesosphere.io> wrote:
> Hi Dick,
>
> There is a PR on Spark to enable launching Spark executors with Docker
> containers, and pending a committer to review and merge
> (https://github.com/apache/spark/pull/3074)
>
> If you like to try it out please take the patch and try it!
>
> Tim
>
> On Wed, Dec 3, 2014 at 6:31 PM, Kyle Ellrott <ke...@soe.ucsc.edu> wrote:
>>
>> I'd like to tag a question onto this; has anybody attempted to deploy
>> spark under Kubernetes
>> https://github.com/googlecloudplatform/kubernetes or Kubernetes mesos (
>> https://github.com/mesosphere/kubernetes-mesos ) .
>> On Wednesday, December 3, 2014, Matei Zaharia <ma...@gmail.com>
>> wrote:
>>>
>>> I'd suggest asking about this on the Mesos list (CCed). As far as I know,
>>> there was actually some ongoing work for this.
>>>
>>> Matei
>>>
>>> > On Dec 3, 2014, at 9:46 AM, Dick Davies <di...@hellooperator.net> wrote:
>>> >
>>> > Just wondered if anyone had managed to start spark
>>> > jobs on mesos wrapped in a docker container?
>>> >
>>> > At present (i.e. very early testing) I'm able to submit executors
>>> > to mesos via spark-submit easily enough, but they fall over
>>> > as we don't have a JVM on our slaves out of the box.
>>> >
>>> > I can push one out via our CM system if push comes to shove,
>>> > but it'd be nice to have that as part of the job (I'm thinking it might
>>> > be a way to get some of the dependencies deployed too).
>>> >
>>> > bear in mind I'm a total clueless newbie at this so please be gentle
>>> > if I'm doing this completely wrong.
>>> >
>>> > Thanks!
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> > For additional commands, e-mail: user-help@spark.apache.org
>>> >
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: dockerized spark executor on mesos?
Posted by Dick Davies <di...@hellooperator.net>.
Oh great, thanks Tim - I'll give that a whirl then.
A Spark rollout isn't in our immediate future, I'm just looking for a good
framework for compute alongside our marathon deployment. So a good
time to experiment!
On 4 December 2014 at 02:36, Tim Chen <ti...@mesosphere.io> wrote:
> Hi Dick,
>
> There is a PR on Spark to enable launching Spark executors with Docker
> containers, and pending a committer to review and merge
> (https://github.com/apache/spark/pull/3074)
>
> If you like to try it out please take the patch and try it!
>
> Tim
>
> On Wed, Dec 3, 2014 at 6:31 PM, Kyle Ellrott <ke...@soe.ucsc.edu> wrote:
>>
>> I'd like to tag a question onto this; has anybody attempted to deploy
>> spark under Kubernetes
>> https://github.com/googlecloudplatform/kubernetes or Kubernetes mesos (
>> https://github.com/mesosphere/kubernetes-mesos ) .
>> On Wednesday, December 3, 2014, Matei Zaharia <ma...@gmail.com>
>> wrote:
>>>
>>> I'd suggest asking about this on the Mesos list (CCed). As far as I know,
>>> there was actually some ongoing work for this.
>>>
>>> Matei
>>>
>>> > On Dec 3, 2014, at 9:46 AM, Dick Davies <di...@hellooperator.net> wrote:
>>> >
>>> > Just wondered if anyone had managed to start spark
>>> > jobs on mesos wrapped in a docker container?
>>> >
>>> > At present (i.e. very early testing) I'm able to submit executors
>>> > to mesos via spark-submit easily enough, but they fall over
>>> > as we don't have a JVM on our slaves out of the box.
>>> >
>>> > I can push one out via our CM system if push comes to shove,
>>> > but it'd be nice to have that as part of the job (I'm thinking it might
>>> > be a way to get some of the dependencies deployed too).
>>> >
>>> > bear in mind I'm a total clueless newbie at this so please be gentle
>>> > if I'm doing this completely wrong.
>>> >
>>> > Thanks!
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> > For additional commands, e-mail: user-help@spark.apache.org
>>> >
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>
Re: dockerized spark executor on mesos?
Posted by Tim Chen <ti...@mesosphere.io>.
Hi Dick,
There is a PR on Spark to enable launching Spark executors with Docker
containers, and pending a committer to review and merge (
https://github.com/apache/spark/pull/3074)
If you like to try it out please take the patch and try it!
Tim
On Wed, Dec 3, 2014 at 6:31 PM, Kyle Ellrott <ke...@soe.ucsc.edu> wrote:
> I'd like to tag a question onto this; has anybody attempted to deploy
> spark under Kubernetes
> https://github.com/googlecloudplatform/kubernetes or Kubernetes mesos (
> https://github.com/mesosphere/kubernetes-mesos ) .
> On Wednesday, December 3, 2014, Matei Zaharia <ma...@gmail.com>
> wrote:
>
>> I'd suggest asking about this on the Mesos list (CCed). As far as I know,
>> there was actually some ongoing work for this.
>>
>> Matei
>>
>> > On Dec 3, 2014, at 9:46 AM, Dick Davies <di...@hellooperator.net> wrote:
>> >
>> > Just wondered if anyone had managed to start spark
>> > jobs on mesos wrapped in a docker container?
>> >
>> > At present (i.e. very early testing) I'm able to submit executors
>> > to mesos via spark-submit easily enough, but they fall over
>> > as we don't have a JVM on our slaves out of the box.
>> >
>> > I can push one out via our CM system if push comes to shove,
>> > but it'd be nice to have that as part of the job (I'm thinking it might
>> > be a way to get some of the dependencies deployed too).
>> >
>> > bear in mind I'm a total clueless newbie at this so please be gentle
>> > if I'm doing this completely wrong.
>> >
>> > Thanks!
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> > For additional commands, e-mail: user-help@spark.apache.org
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
Re: dockerized spark executor on mesos?
Posted by Kyle Ellrott <ke...@soe.ucsc.edu>.
I'd like to tag a question onto this; has anybody attempted to deploy spark
under Kubernetes
https://github.com/googlecloudplatform/kubernetes or Kubernetes mesos (
https://github.com/mesosphere/kubernetes-mesos ) .
On Wednesday, December 3, 2014, Matei Zaharia <ma...@gmail.com>
wrote:
> I'd suggest asking about this on the Mesos list (CCed). As far as I know,
> there was actually some ongoing work for this.
>
> Matei
>
> > On Dec 3, 2014, at 9:46 AM, Dick Davies <dick@hellooperator.net
> <javascript:;>> wrote:
> >
> > Just wondered if anyone had managed to start spark
> > jobs on mesos wrapped in a docker container?
> >
> > At present (i.e. very early testing) I'm able to submit executors
> > to mesos via spark-submit easily enough, but they fall over
> > as we don't have a JVM on our slaves out of the box.
> >
> > I can push one out via our CM system if push comes to shove,
> > but it'd be nice to have that as part of the job (I'm thinking it might
> > be a way to get some of the dependencies deployed too).
> >
> > bear in mind I'm a total clueless newbie at this so please be gentle
> > if I'm doing this completely wrong.
> >
> > Thanks!
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org <javascript:;>
> > For additional commands, e-mail: user-help@spark.apache.org
> <javascript:;>
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org <javascript:;>
> For additional commands, e-mail: user-help@spark.apache.org <javascript:;>
>
>
Re: dockerized spark executor on mesos?
Posted by Kyle Ellrott <ke...@soe.ucsc.edu>.
I'd like to tag a question onto this; has anybody attempted to deploy spark
under Kubernetes
https://github.com/googlecloudplatform/kubernetes or Kubernetes mesos (
https://github.com/mesosphere/kubernetes-mesos ) .
On Wednesday, December 3, 2014, Matei Zaharia <ma...@gmail.com>
wrote:
> I'd suggest asking about this on the Mesos list (CCed). As far as I know,
> there was actually some ongoing work for this.
>
> Matei
>
> > On Dec 3, 2014, at 9:46 AM, Dick Davies <dick@hellooperator.net
> <javascript:;>> wrote:
> >
> > Just wondered if anyone had managed to start spark
> > jobs on mesos wrapped in a docker container?
> >
> > At present (i.e. very early testing) I'm able to submit executors
> > to mesos via spark-submit easily enough, but they fall over
> > as we don't have a JVM on our slaves out of the box.
> >
> > I can push one out via our CM system if push comes to shove,
> > but it'd be nice to have that as part of the job (I'm thinking it might
> > be a way to get some of the dependencies deployed too).
> >
> > bear in mind I'm a total clueless newbie at this so please be gentle
> > if I'm doing this completely wrong.
> >
> > Thanks!
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org <javascript:;>
> > For additional commands, e-mail: user-help@spark.apache.org
> <javascript:;>
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org <javascript:;>
> For additional commands, e-mail: user-help@spark.apache.org <javascript:;>
>
>
Re: dockerized spark executor on mesos?
Posted by Matei Zaharia <ma...@gmail.com>.
I'd suggest asking about this on the Mesos list (CCed). As far as I know, there was actually some ongoing work for this.
Matei
> On Dec 3, 2014, at 9:46 AM, Dick Davies <di...@hellooperator.net> wrote:
>
> Just wondered if anyone had managed to start spark
> jobs on mesos wrapped in a docker container?
>
> At present (i.e. very early testing) I'm able to submit executors
> to mesos via spark-submit easily enough, but they fall over
> as we don't have a JVM on our slaves out of the box.
>
> I can push one out via our CM system if push comes to shove,
> but it'd be nice to have that as part of the job (I'm thinking it might
> be a way to get some of the dependencies deployed too).
>
> bear in mind I'm a total clueless newbie at this so please be gentle
> if I'm doing this completely wrong.
>
> Thanks!
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: dockerized spark executor on mesos?
Posted by Tim Chen <ti...@mesosphere.io>.
Just throwing this out here, there is existing PR to add docker support for
spark framework to launch executors with docker image.
https://github.com/apache/spark/pull/3074
Hopefully this will be merged sometime.
Tim
On Thu, Jan 15, 2015 at 9:18 AM, Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:
> The AMPLab maintains a bunch of Docker files for Spark here:
> https://github.com/amplab/docker-scripts
>
> Hasn't been updated since 1.0.0, but might be a good starting point.
>
> On Wed Jan 14 2015 at 12:14:13 PM Josh J <jo...@gmail.com> wrote:
>
>> We have dockerized Spark Master and worker(s) separately and are using it
>>> in
>>> our dev environment.
>>
>>
>> Is this setup available on github or dockerhub?
>>
>> On Tue, Dec 9, 2014 at 3:50 PM, Venkat Subramanian <vs...@gmail.com>
>> wrote:
>>
>>> We have dockerized Spark Master and worker(s) separately and are using
>>> it in
>>> our dev environment. We don't use Mesos though, running it in Standalone
>>> mode, but adding Mesos should not be that difficult I think.
>>>
>>> Regards
>>>
>>> Venkat
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/dockerized-spark-executor-on-mesos-tp20276p20603.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
Re: dockerized spark executor on mesos?
Posted by Nicholas Chammas <ni...@gmail.com>.
The AMPLab maintains a bunch of Docker files for Spark here:
https://github.com/amplab/docker-scripts
Hasn't been updated since 1.0.0, but might be a good starting point.
On Wed Jan 14 2015 at 12:14:13 PM Josh J <jo...@gmail.com> wrote:
> We have dockerized Spark Master and worker(s) separately and are using it
>> in
>> our dev environment.
>
>
> Is this setup available on github or dockerhub?
>
> On Tue, Dec 9, 2014 at 3:50 PM, Venkat Subramanian <vs...@gmail.com>
> wrote:
>
>> We have dockerized Spark Master and worker(s) separately and are using it
>> in
>> our dev environment. We don't use Mesos though, running it in Standalone
>> mode, but adding Mesos should not be that difficult I think.
>>
>> Regards
>>
>> Venkat
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/dockerized-spark-executor-on-mesos-tp20276p20603.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: dockerized spark executor on mesos?
Posted by Josh J <jo...@gmail.com>.
>
> We have dockerized Spark Master and worker(s) separately and are using it
> in
> our dev environment.
Is this setup available on github or dockerhub?
On Tue, Dec 9, 2014 at 3:50 PM, Venkat Subramanian <vs...@gmail.com>
wrote:
> We have dockerized Spark Master and worker(s) separately and are using it
> in
> our dev environment. We don't use Mesos though, running it in Standalone
> mode, but adding Mesos should not be that difficult I think.
>
> Regards
>
> Venkat
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/dockerized-spark-executor-on-mesos-tp20276p20603.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: dockerized spark executor on mesos?
Posted by Venkat Subramanian <vs...@gmail.com>.
We have dockerized Spark Master and worker(s) separately and are using it in
our dev environment. We don't use Mesos though, running it in Standalone
mode, but adding Mesos should not be that difficult I think.
Regards
Venkat
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/dockerized-spark-executor-on-mesos-tp20276p20603.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org