You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Pradeep Chhetri <pr...@gmail.com> on 2016/03/15 16:41:26 UTC
Apache Spark Over Mesos
Hello,
I am able to run Apache Spark over Mesos. Its quite simple to run Spark
Dispatcher over marathon and ask it to run Spark Executor (I guess also can
be called as Spark Driver) as docker container.
I have a query regarding this:
All spark tasks are spawned directly by first downloading the spark
artifacts. I was thinking if there is some way I can start them too as
docker containers. This will save the time for downloading the spark
artifacts. I am running spark in fine-grained mode.
I have attached a screenshot of a sample job
Thanks,
--
Pradeep Chhetri
Re: Apache Spark Over Mesos
Posted by Rad Gruchalski <ra...@gruchalski.com>.
That’s a great alternative!
Kind regards,
Radek Gruchalski
radek@gruchalski.com (mailto:radek@gruchalski.com)
(mailto:radek@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
Confidentiality:
This communication is intended for the above-named person and may be confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately.
On Tuesday, 15 March 2016 at 17:19, Timothy Chen wrote:
> You can launch the driver and executor in docker containers as well by setting spark.mesos.executor.docker.image to the image you want to use to launch them.
>
> Tim
>
> On Mar 15, 2016, at 8:49 AM, Radoslaw Gruchalski <radek@gruchalski.com (mailto:radek@gruchalski.com)> wrote:
>
> > Pradeep,
> >
> > You can mount a spark directory as a volume. This means you have to have spark deployed on every agent.
> >
> > Another thing you can do, place spark in hdfs, assuming that you have hdfs available but that too will download a copy to the sandbox.
> >
> > I'd prefer the former.
> >
> > Sent from Outlook Mobile (https://aka.ms/qtex0l)
> > _____________________________
> > From: Pradeep Chhetri <pradeep.chhetri89@gmail.com (mailto:pradeep.chhetri89@gmail.com)>
> > Sent: Tuesday, March 15, 2016 4:41 pm
> > Subject: Apache Spark Over Mesos
> > To: <user@mesos.apache.org (mailto:user@mesos.apache.org)>
> >
> >
> > Hello,
> >
> > I am able to run Apache Spark over Mesos. Its quite simple to run Spark Dispatcher over marathon and ask it to run Spark Executor (I guess also can be called as Spark Driver) as docker container.
> >
> > I have a query regarding this:
> >
> > All spark tasks are spawned directly by first downloading the spark artifacts. I was thinking if there is some way I can start them too as docker containers. This will save the time for downloading the spark artifacts. I am running spark in fine-grained mode.
> >
> > I have attached a screenshot of a sample job
> >
> > <Screen Shot 2016-03-15 at 15.15.06.png>
> >
> > Thanks,
> >
> > --
> > Pradeep Chhetri
> >
Re: Apache Spark Over Mesos
Posted by Tim Chen <ti...@mesosphere.io>.
Hi Pradeep,
You'll need to specify a s3:// url instead of specifying a relative path
like you did, you can also also use http or hdfs if you want.
You also need to make sure s3 credentials are available in your hadoop
configuration that's also embedded in your docker image, you should be able
to find help information about that easily (We're still working around
these user experience problems around configuration, for now it's easier if
the docker image has all the right configuration).
Tim
On Tue, Mar 15, 2016 at 11:18 AM, Pradeep Chhetri <
pradeep.chhetri89@gmail.com> wrote:
> Hello TIm,
>
> I was able to start the spark tasks also as docker containers.
>
> I have one question:
>
> Currently, when i am submit a sample job like follows:
>
> $ bin/spark-submit --deploy-mode cluster --master
> mesos://spark-dispatcher.service.consul:7077 --class
> org.apache.spark.examples.SparkPi lib/spark-examples*.jar 10
>
> It tries to copy the spark-examples*.jar on host from absolute path to the
> sandbox. Is there any way i can make it to pull from S3 instead of looking
> on host for the jar ? So that my build pipeline will push the jar to S3 and
> running spark-submit as a deployment job will pull that jar from S3.
>
> Thanks.
>
>
>
> On Tue, Mar 15, 2016 at 5:17 PM, Pradeep Chhetri <
> pradeep.chhetri89@gmail.com> wrote:
>
>> TIm, sorry I am wrong above.
>>
>> The above config is from spark dispatcher container and those
>> configuration is not being propagated to the driver.
>>
>> I will try the workaround you told and let you know how it goes.
>>
>> Thanks
>>
>>
>>
>> On Tue, Mar 15, 2016 at 4:42 PM, Pradeep Chhetri <
>> pradeep.chhetri89@gmail.com> wrote:
>>
>>> Hello Tim,
>>>
>>> Here is my conf/spark-defaults.conf which is inside the docker image:
>>>
>>> $ cat conf/spark-defaults.conf
>>>
>>> spark.mesos.coarse: false
>>> spark.mesos.executor.docker.image: docker-registry/mesos-spark:master-12
>>> spark.mesos.mesosExecutor.cores: 0.25
>>> spark.mesos.executor.home: /opt/spark
>>> spark.mesos.uris: file:///etc/docker.tar.gz
>>>
>>> I am already setting it inside the docker image.
>>>
>>> Am I missing something ?
>>>
>>> Regards,
>>>
>>> On Tue, Mar 15, 2016 at 4:37 PM, Tim Chen <ti...@mesosphere.io> wrote:
>>>
>>>> Hi Pradeep,
>>>>
>>>> Yes we still have a pending PR that will start propagating these
>>>> settings down to the executors, right now it's only applied on the driver.
>>>> As a work around you can download or set spark.mesos.executor.docker.image
>>>> in the spark-default.conf file in the docker image you use to launch the
>>>> driver and it should automatically get this setting when the driver is
>>>> launched.
>>>>
>>>> Tim
>>>>
>>>> On Tue, Mar 15, 2016 at 9:26 AM, Pradeep Chhetri <
>>>> pradeep.chhetri89@gmail.com> wrote:
>>>>
>>>>> Hello Timothy,
>>>>>
>>>>> I am setting spark.mesos.executor.docker.image. In my case, the
>>>>> driver is actually started as a docker container (SparkPi in screenshot)
>>>>> but the tasks which are spawned by driver are not starting as containers
>>>>> but plain java processes. Is this expected ?
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Tue, Mar 15, 2016 at 4:19 PM, Timothy Chen <ti...@mesosphere.io>
>>>>> wrote:
>>>>>
>>>>>> You can launch the driver and executor in docker containers as well
>>>>>> by setting spark.mesos.executor.docker.image to the image you want to use
>>>>>> to launch them.
>>>>>>
>>>>>> Tim
>>>>>>
>>>>>> On Mar 15, 2016, at 8:49 AM, Radoslaw Gruchalski <
>>>>>> radek@gruchalski.com> wrote:
>>>>>>
>>>>>> Pradeep,
>>>>>>
>>>>>> You can mount a spark directory as a volume. This means you have to
>>>>>> have spark deployed on every agent.
>>>>>>
>>>>>> Another thing you can do, place spark in hdfs, assuming that you have
>>>>>> hdfs available but that too will download a copy to the sandbox.
>>>>>>
>>>>>> I'd prefer the former.
>>>>>>
>>>>>> Sent from Outlook Mobile <https://aka.ms/qtex0l>
>>>>>>
>>>>>> _____________________________
>>>>>> From: Pradeep Chhetri <pr...@gmail.com>
>>>>>> Sent: Tuesday, March 15, 2016 4:41 pm
>>>>>> Subject: Apache Spark Over Mesos
>>>>>> To: <us...@mesos.apache.org>
>>>>>>
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I am able to run Apache Spark over Mesos. Its quite simple to run
>>>>>> Spark Dispatcher over marathon and ask it to run Spark Executor (I guess
>>>>>> also can be called as Spark Driver) as docker container.
>>>>>>
>>>>>> I have a query regarding this:
>>>>>>
>>>>>> All spark tasks are spawned directly by first downloading the spark
>>>>>> artifacts. I was thinking if there is some way I can start them too as
>>>>>> docker containers. This will save the time for downloading the spark
>>>>>> artifacts. I am running spark in fine-grained mode.
>>>>>>
>>>>>> I have attached a screenshot of a sample job
>>>>>>
>>>>>> <Screen Shot 2016-03-15 at 15.15.06.png>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> --
>>>>>> Pradeep Chhetri
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Pradeep Chhetri
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Pradeep Chhetri
>>>
>>
>>
>>
>> --
>> Pradeep Chhetri
>>
>
>
>
> --
> Pradeep Chhetri
>
Re: Apache Spark Over Mesos
Posted by Pradeep Chhetri <pr...@gmail.com>.
Hello TIm,
I was able to start the spark tasks also as docker containers.
I have one question:
Currently, when i am submit a sample job like follows:
$ bin/spark-submit --deploy-mode cluster --master
mesos://spark-dispatcher.service.consul:7077 --class
org.apache.spark.examples.SparkPi lib/spark-examples*.jar 10
It tries to copy the spark-examples*.jar on host from absolute path to the
sandbox. Is there any way i can make it to pull from S3 instead of looking
on host for the jar ? So that my build pipeline will push the jar to S3 and
running spark-submit as a deployment job will pull that jar from S3.
Thanks.
On Tue, Mar 15, 2016 at 5:17 PM, Pradeep Chhetri <
pradeep.chhetri89@gmail.com> wrote:
> TIm, sorry I am wrong above.
>
> The above config is from spark dispatcher container and those
> configuration is not being propagated to the driver.
>
> I will try the workaround you told and let you know how it goes.
>
> Thanks
>
>
>
> On Tue, Mar 15, 2016 at 4:42 PM, Pradeep Chhetri <
> pradeep.chhetri89@gmail.com> wrote:
>
>> Hello Tim,
>>
>> Here is my conf/spark-defaults.conf which is inside the docker image:
>>
>> $ cat conf/spark-defaults.conf
>>
>> spark.mesos.coarse: false
>> spark.mesos.executor.docker.image: docker-registry/mesos-spark:master-12
>> spark.mesos.mesosExecutor.cores: 0.25
>> spark.mesos.executor.home: /opt/spark
>> spark.mesos.uris: file:///etc/docker.tar.gz
>>
>> I am already setting it inside the docker image.
>>
>> Am I missing something ?
>>
>> Regards,
>>
>> On Tue, Mar 15, 2016 at 4:37 PM, Tim Chen <ti...@mesosphere.io> wrote:
>>
>>> Hi Pradeep,
>>>
>>> Yes we still have a pending PR that will start propagating these
>>> settings down to the executors, right now it's only applied on the driver.
>>> As a work around you can download or set spark.mesos.executor.docker.image
>>> in the spark-default.conf file in the docker image you use to launch the
>>> driver and it should automatically get this setting when the driver is
>>> launched.
>>>
>>> Tim
>>>
>>> On Tue, Mar 15, 2016 at 9:26 AM, Pradeep Chhetri <
>>> pradeep.chhetri89@gmail.com> wrote:
>>>
>>>> Hello Timothy,
>>>>
>>>> I am setting spark.mesos.executor.docker.image. In my case, the driver
>>>> is actually started as a docker container (SparkPi in screenshot) but the
>>>> tasks which are spawned by driver are not starting as containers but plain
>>>> java processes. Is this expected ?
>>>>
>>>> Thanks
>>>>
>>>> On Tue, Mar 15, 2016 at 4:19 PM, Timothy Chen <ti...@mesosphere.io>
>>>> wrote:
>>>>
>>>>> You can launch the driver and executor in docker containers as well by
>>>>> setting spark.mesos.executor.docker.image to the image you want to use to
>>>>> launch them.
>>>>>
>>>>> Tim
>>>>>
>>>>> On Mar 15, 2016, at 8:49 AM, Radoslaw Gruchalski <ra...@gruchalski.com>
>>>>> wrote:
>>>>>
>>>>> Pradeep,
>>>>>
>>>>> You can mount a spark directory as a volume. This means you have to
>>>>> have spark deployed on every agent.
>>>>>
>>>>> Another thing you can do, place spark in hdfs, assuming that you have
>>>>> hdfs available but that too will download a copy to the sandbox.
>>>>>
>>>>> I'd prefer the former.
>>>>>
>>>>> Sent from Outlook Mobile <https://aka.ms/qtex0l>
>>>>>
>>>>> _____________________________
>>>>> From: Pradeep Chhetri <pr...@gmail.com>
>>>>> Sent: Tuesday, March 15, 2016 4:41 pm
>>>>> Subject: Apache Spark Over Mesos
>>>>> To: <us...@mesos.apache.org>
>>>>>
>>>>>
>>>>> Hello,
>>>>>
>>>>> I am able to run Apache Spark over Mesos. Its quite simple to run
>>>>> Spark Dispatcher over marathon and ask it to run Spark Executor (I guess
>>>>> also can be called as Spark Driver) as docker container.
>>>>>
>>>>> I have a query regarding this:
>>>>>
>>>>> All spark tasks are spawned directly by first downloading the spark
>>>>> artifacts. I was thinking if there is some way I can start them too as
>>>>> docker containers. This will save the time for downloading the spark
>>>>> artifacts. I am running spark in fine-grained mode.
>>>>>
>>>>> I have attached a screenshot of a sample job
>>>>>
>>>>> <Screen Shot 2016-03-15 at 15.15.06.png>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> --
>>>>> Pradeep Chhetri
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Pradeep Chhetri
>>>>
>>>
>>>
>>
>>
>> --
>> Pradeep Chhetri
>>
>
>
>
> --
> Pradeep Chhetri
>
--
Pradeep Chhetri
Re: Apache Spark Over Mesos
Posted by Pradeep Chhetri <pr...@gmail.com>.
TIm, sorry I am wrong above.
The above config is from spark dispatcher container and those configuration
is not being propagated to the driver.
I will try the workaround you told and let you know how it goes.
Thanks
On Tue, Mar 15, 2016 at 4:42 PM, Pradeep Chhetri <
pradeep.chhetri89@gmail.com> wrote:
> Hello Tim,
>
> Here is my conf/spark-defaults.conf which is inside the docker image:
>
> $ cat conf/spark-defaults.conf
>
> spark.mesos.coarse: false
> spark.mesos.executor.docker.image: docker-registry/mesos-spark:master-12
> spark.mesos.mesosExecutor.cores: 0.25
> spark.mesos.executor.home: /opt/spark
> spark.mesos.uris: file:///etc/docker.tar.gz
>
> I am already setting it inside the docker image.
>
> Am I missing something ?
>
> Regards,
>
> On Tue, Mar 15, 2016 at 4:37 PM, Tim Chen <ti...@mesosphere.io> wrote:
>
>> Hi Pradeep,
>>
>> Yes we still have a pending PR that will start propagating these settings
>> down to the executors, right now it's only applied on the driver. As a work
>> around you can download or set spark.mesos.executor.docker.image in the
>> spark-default.conf file in the docker image you use to launch the driver
>> and it should automatically get this setting when the driver is launched.
>>
>> Tim
>>
>> On Tue, Mar 15, 2016 at 9:26 AM, Pradeep Chhetri <
>> pradeep.chhetri89@gmail.com> wrote:
>>
>>> Hello Timothy,
>>>
>>> I am setting spark.mesos.executor.docker.image. In my case, the driver
>>> is actually started as a docker container (SparkPi in screenshot) but the
>>> tasks which are spawned by driver are not starting as containers but plain
>>> java processes. Is this expected ?
>>>
>>> Thanks
>>>
>>> On Tue, Mar 15, 2016 at 4:19 PM, Timothy Chen <ti...@mesosphere.io> wrote:
>>>
>>>> You can launch the driver and executor in docker containers as well by
>>>> setting spark.mesos.executor.docker.image to the image you want to use to
>>>> launch them.
>>>>
>>>> Tim
>>>>
>>>> On Mar 15, 2016, at 8:49 AM, Radoslaw Gruchalski <ra...@gruchalski.com>
>>>> wrote:
>>>>
>>>> Pradeep,
>>>>
>>>> You can mount a spark directory as a volume. This means you have to
>>>> have spark deployed on every agent.
>>>>
>>>> Another thing you can do, place spark in hdfs, assuming that you have
>>>> hdfs available but that too will download a copy to the sandbox.
>>>>
>>>> I'd prefer the former.
>>>>
>>>> Sent from Outlook Mobile <https://aka.ms/qtex0l>
>>>>
>>>> _____________________________
>>>> From: Pradeep Chhetri <pr...@gmail.com>
>>>> Sent: Tuesday, March 15, 2016 4:41 pm
>>>> Subject: Apache Spark Over Mesos
>>>> To: <us...@mesos.apache.org>
>>>>
>>>>
>>>> Hello,
>>>>
>>>> I am able to run Apache Spark over Mesos. Its quite simple to run Spark
>>>> Dispatcher over marathon and ask it to run Spark Executor (I guess also can
>>>> be called as Spark Driver) as docker container.
>>>>
>>>> I have a query regarding this:
>>>>
>>>> All spark tasks are spawned directly by first downloading the spark
>>>> artifacts. I was thinking if there is some way I can start them too as
>>>> docker containers. This will save the time for downloading the spark
>>>> artifacts. I am running spark in fine-grained mode.
>>>>
>>>> I have attached a screenshot of a sample job
>>>>
>>>> <Screen Shot 2016-03-15 at 15.15.06.png>
>>>>
>>>> Thanks,
>>>>
>>>> --
>>>> Pradeep Chhetri
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Pradeep Chhetri
>>>
>>
>>
>
>
> --
> Pradeep Chhetri
>
--
Pradeep Chhetri
Re: Apache Spark Over Mesos
Posted by Pradeep Chhetri <pr...@gmail.com>.
Hello Tim,
Here is my conf/spark-defaults.conf which is inside the docker image:
$ cat conf/spark-defaults.conf
spark.mesos.coarse: false
spark.mesos.executor.docker.image: docker-registry/mesos-spark:master-12
spark.mesos.mesosExecutor.cores: 0.25
spark.mesos.executor.home: /opt/spark
spark.mesos.uris: file:///etc/docker.tar.gz
I am already setting it inside the docker image.
Am I missing something ?
Regards,
On Tue, Mar 15, 2016 at 4:37 PM, Tim Chen <ti...@mesosphere.io> wrote:
> Hi Pradeep,
>
> Yes we still have a pending PR that will start propagating these settings
> down to the executors, right now it's only applied on the driver. As a work
> around you can download or set spark.mesos.executor.docker.image in the
> spark-default.conf file in the docker image you use to launch the driver
> and it should automatically get this setting when the driver is launched.
>
> Tim
>
> On Tue, Mar 15, 2016 at 9:26 AM, Pradeep Chhetri <
> pradeep.chhetri89@gmail.com> wrote:
>
>> Hello Timothy,
>>
>> I am setting spark.mesos.executor.docker.image. In my case, the driver
>> is actually started as a docker container (SparkPi in screenshot) but the
>> tasks which are spawned by driver are not starting as containers but plain
>> java processes. Is this expected ?
>>
>> Thanks
>>
>> On Tue, Mar 15, 2016 at 4:19 PM, Timothy Chen <ti...@mesosphere.io> wrote:
>>
>>> You can launch the driver and executor in docker containers as well by
>>> setting spark.mesos.executor.docker.image to the image you want to use to
>>> launch them.
>>>
>>> Tim
>>>
>>> On Mar 15, 2016, at 8:49 AM, Radoslaw Gruchalski <ra...@gruchalski.com>
>>> wrote:
>>>
>>> Pradeep,
>>>
>>> You can mount a spark directory as a volume. This means you have to have
>>> spark deployed on every agent.
>>>
>>> Another thing you can do, place spark in hdfs, assuming that you have
>>> hdfs available but that too will download a copy to the sandbox.
>>>
>>> I'd prefer the former.
>>>
>>> Sent from Outlook Mobile <https://aka.ms/qtex0l>
>>>
>>> _____________________________
>>> From: Pradeep Chhetri <pr...@gmail.com>
>>> Sent: Tuesday, March 15, 2016 4:41 pm
>>> Subject: Apache Spark Over Mesos
>>> To: <us...@mesos.apache.org>
>>>
>>>
>>> Hello,
>>>
>>> I am able to run Apache Spark over Mesos. Its quite simple to run Spark
>>> Dispatcher over marathon and ask it to run Spark Executor (I guess also can
>>> be called as Spark Driver) as docker container.
>>>
>>> I have a query regarding this:
>>>
>>> All spark tasks are spawned directly by first downloading the spark
>>> artifacts. I was thinking if there is some way I can start them too as
>>> docker containers. This will save the time for downloading the spark
>>> artifacts. I am running spark in fine-grained mode.
>>>
>>> I have attached a screenshot of a sample job
>>>
>>> <Screen Shot 2016-03-15 at 15.15.06.png>
>>>
>>> Thanks,
>>>
>>> --
>>> Pradeep Chhetri
>>>
>>>
>>>
>>
>>
>> --
>> Pradeep Chhetri
>>
>
>
--
Pradeep Chhetri
Re: Apache Spark Over Mesos
Posted by Tim Chen <ti...@mesosphere.io>.
Hi Pradeep,
Yes we still have a pending PR that will start propagating these settings
down to the executors, right now it's only applied on the driver. As a work
around you can download or set spark.mesos.executor.docker.image in the
spark-default.conf file in the docker image you use to launch the driver
and it should automatically get this setting when the driver is launched.
Tim
On Tue, Mar 15, 2016 at 9:26 AM, Pradeep Chhetri <
pradeep.chhetri89@gmail.com> wrote:
> Hello Timothy,
>
> I am setting spark.mesos.executor.docker.image. In my case, the driver is
> actually started as a docker container (SparkPi in screenshot) but the
> tasks which are spawned by driver are not starting as containers but plain
> java processes. Is this expected ?
>
> Thanks
>
> On Tue, Mar 15, 2016 at 4:19 PM, Timothy Chen <ti...@mesosphere.io> wrote:
>
>> You can launch the driver and executor in docker containers as well by
>> setting spark.mesos.executor.docker.image to the image you want to use to
>> launch them.
>>
>> Tim
>>
>> On Mar 15, 2016, at 8:49 AM, Radoslaw Gruchalski <ra...@gruchalski.com>
>> wrote:
>>
>> Pradeep,
>>
>> You can mount a spark directory as a volume. This means you have to have
>> spark deployed on every agent.
>>
>> Another thing you can do, place spark in hdfs, assuming that you have
>> hdfs available but that too will download a copy to the sandbox.
>>
>> I'd prefer the former.
>>
>> Sent from Outlook Mobile <https://aka.ms/qtex0l>
>>
>> _____________________________
>> From: Pradeep Chhetri <pr...@gmail.com>
>> Sent: Tuesday, March 15, 2016 4:41 pm
>> Subject: Apache Spark Over Mesos
>> To: <us...@mesos.apache.org>
>>
>>
>> Hello,
>>
>> I am able to run Apache Spark over Mesos. Its quite simple to run Spark
>> Dispatcher over marathon and ask it to run Spark Executor (I guess also can
>> be called as Spark Driver) as docker container.
>>
>> I have a query regarding this:
>>
>> All spark tasks are spawned directly by first downloading the spark
>> artifacts. I was thinking if there is some way I can start them too as
>> docker containers. This will save the time for downloading the spark
>> artifacts. I am running spark in fine-grained mode.
>>
>> I have attached a screenshot of a sample job
>>
>> <Screen Shot 2016-03-15 at 15.15.06.png>
>>
>> Thanks,
>>
>> --
>> Pradeep Chhetri
>>
>>
>>
>
>
> --
> Pradeep Chhetri
>
Re: Apache Spark Over Mesos
Posted by Pradeep Chhetri <pr...@gmail.com>.
Hello Timothy,
I am setting spark.mesos.executor.docker.image. In my case, the driver is
actually started as a docker container (SparkPi in screenshot) but the
tasks which are spawned by driver are not starting as containers but plain
java processes. Is this expected ?
Thanks
On Tue, Mar 15, 2016 at 4:19 PM, Timothy Chen <ti...@mesosphere.io> wrote:
> You can launch the driver and executor in docker containers as well by
> setting spark.mesos.executor.docker.image to the image you want to use to
> launch them.
>
> Tim
>
> On Mar 15, 2016, at 8:49 AM, Radoslaw Gruchalski <ra...@gruchalski.com>
> wrote:
>
> Pradeep,
>
> You can mount a spark directory as a volume. This means you have to have
> spark deployed on every agent.
>
> Another thing you can do, place spark in hdfs, assuming that you have hdfs
> available but that too will download a copy to the sandbox.
>
> I'd prefer the former.
>
> Sent from Outlook Mobile <https://aka.ms/qtex0l>
>
> _____________________________
> From: Pradeep Chhetri <pr...@gmail.com>
> Sent: Tuesday, March 15, 2016 4:41 pm
> Subject: Apache Spark Over Mesos
> To: <us...@mesos.apache.org>
>
>
> Hello,
>
> I am able to run Apache Spark over Mesos. Its quite simple to run Spark
> Dispatcher over marathon and ask it to run Spark Executor (I guess also can
> be called as Spark Driver) as docker container.
>
> I have a query regarding this:
>
> All spark tasks are spawned directly by first downloading the spark
> artifacts. I was thinking if there is some way I can start them too as
> docker containers. This will save the time for downloading the spark
> artifacts. I am running spark in fine-grained mode.
>
> I have attached a screenshot of a sample job
>
> <Screen Shot 2016-03-15 at 15.15.06.png>
>
> Thanks,
>
> --
> Pradeep Chhetri
>
>
>
--
Pradeep Chhetri
Re: Apache Spark Over Mesos
Posted by Timothy Chen <ti...@mesosphere.io>.
You can launch the driver and executor in docker containers as well by setting spark.mesos.executor.docker.image to the image you want to use to launch them.
Tim
> On Mar 15, 2016, at 8:49 AM, Radoslaw Gruchalski <ra...@gruchalski.com> wrote:
>
> Pradeep,
>
> You can mount a spark directory as a volume. This means you have to have spark deployed on every agent.
>
> Another thing you can do, place spark in hdfs, assuming that you have hdfs available but that too will download a copy to the sandbox.
>
> I'd prefer the former.
>
> Sent from Outlook Mobile
>
> _____________________________
> From: Pradeep Chhetri <pr...@gmail.com>
> Sent: Tuesday, March 15, 2016 4:41 pm
> Subject: Apache Spark Over Mesos
> To: <us...@mesos.apache.org>
>
>
> Hello,
>
> I am able to run Apache Spark over Mesos. Its quite simple to run Spark Dispatcher over marathon and ask it to run Spark Executor (I guess also can be called as Spark Driver) as docker container.
>
> I have a query regarding this:
>
> All spark tasks are spawned directly by first downloading the spark artifacts. I was thinking if there is some way I can start them too as docker containers. This will save the time for downloading the spark artifacts. I am running spark in fine-grained mode.
>
> I have attached a screenshot of a sample job
>
> <Screen Shot 2016-03-15 at 15.15.06.png>
>
> Thanks,
>
> --
> Pradeep Chhetri
>
>
Re: Apache Spark Over Mesos
Posted by Pradeep Chhetri <pr...@gmail.com>.
Just to be clean:
I am already setting spark.mesos.executor.docker.image to a docker image. I
am starting the Spark Dispatcher over marathon using that image. While
submitting jobs using spark-submit, mesos is able to start the driver using
the same docker image but the tasks are starting as plain java process.
$ cat conf/spark-defaults.conf
spark.mesos.coarse: false
spark.mesos.executor.docker.image: docker-registry/mesos-spark:master-12
spark.mesos.mesosExecutor.cores: 0.25
spark.mesos.executor.home: /opt/spark
spark.mesos.uris: file:///etc/docker.tar.gz
Thanks
On Tue, Mar 15, 2016 at 4:24 PM, Rad Gruchalski <ra...@gruchalski.com>
wrote:
> As Tim suggested: spark.mesos.executor.docker.image is your friend.
>
> Kind regards,
> Radek Gruchalski
> radek@gruchalski.com <ra...@gruchalski.com>
> de.linkedin.com/in/radgruchalski/
>
>
> *Confidentiality:*This communication is intended for the above-named
> person and may be confidential and/or legally privileged.
> If it has come to you in error you must take no action based on it, nor
> must you copy or show it to anyone; please delete/destroy and inform the
> sender immediately.
>
> On Tuesday, 15 March 2016 at 17:23, Pradeep Chhetri wrote:
>
> Hello Radoslaw,
>
> Thank you for the quick reply. Few questions:
>
> 1) Do you mean mounting spark artifacts as a volume on each mesos agent
> node? This means number of volumes = number of mesos agents.
>
> 2) Since I am not using HDFS at all, that is definitely not an option for
> me.
>
> Isn't there a way to just launch the spark tasks also as docker containers
> which are self contained with spark artifacts ?
>
> Thanks.
>
> On Tue, Mar 15, 2016 at 3:49 PM, Radoslaw Gruchalski <radek@gruchalski.com
> > wrote:
>
> Pradeep,
>
> You can mount a spark directory as a volume. This means you have to have
> spark deployed on every agent.
>
> Another thing you can do, place spark in hdfs, assuming that you have hdfs
> available but that too will download a copy to the sandbox.
>
> I'd prefer the former.
>
> Sent from Outlook Mobile <https://aka.ms/qtex0l>
>
> _____________________________
> From: Pradeep Chhetri <pr...@gmail.com>
> Sent: Tuesday, March 15, 2016 4:41 pm
> Subject: Apache Spark Over Mesos
> To: <us...@mesos.apache.org>
>
>
>
> Hello,
>
> I am able to run Apache Spark over Mesos. Its quite simple to run Spark
> Dispatcher over marathon and ask it to run Spark Executor (I guess also can
> be called as Spark Driver) as docker container.
>
> I have a query regarding this:
>
> All spark tasks are spawned directly by first downloading the spark
> artifacts. I was thinking if there is some way I can start them too as
> docker containers. This will save the time for downloading the spark
> artifacts. I am running spark in fine-grained mode.
>
> I have attached a screenshot of a sample job
>
>
>
> Thanks,
>
> --
> Pradeep Chhetri
>
>
>
>
>
> --
> Pradeep Chhetri
>
>
>
--
Pradeep Chhetri
Re: Apache Spark Over Mesos
Posted by Rad Gruchalski <ra...@gruchalski.com>.
As Tim suggested: spark.mesos.executor.docker.image is your friend.
Kind regards,
Radek Gruchalski
radek@gruchalski.com (mailto:radek@gruchalski.com)
(mailto:radek@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
Confidentiality:
This communication is intended for the above-named person and may be confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately.
On Tuesday, 15 March 2016 at 17:23, Pradeep Chhetri wrote:
> Hello Radoslaw,
>
> Thank you for the quick reply. Few questions:
>
> 1) Do you mean mounting spark artifacts as a volume on each mesos agent node? This means number of volumes = number of mesos agents.
>
> 2) Since I am not using HDFS at all, that is definitely not an option for me.
>
> Isn't there a way to just launch the spark tasks also as docker containers which are self contained with spark artifacts ?
>
> Thanks.
>
> On Tue, Mar 15, 2016 at 3:49 PM, Radoslaw Gruchalski <radek@gruchalski.com (mailto:radek@gruchalski.com)> wrote:
> > Pradeep,
> >
> > You can mount a spark directory as a volume. This means you have to have spark deployed on every agent.
> >
> > Another thing you can do, place spark in hdfs, assuming that you have hdfs available but that too will download a copy to the sandbox.
> >
> > I'd prefer the former.
> >
> > Sent from Outlook Mobile (https://aka.ms/qtex0l)
> > _____________________________
> > From: Pradeep Chhetri <pradeep.chhetri89@gmail.com (mailto:pradeep.chhetri89@gmail.com)>
> > Sent: Tuesday, March 15, 2016 4:41 pm
> > Subject: Apache Spark Over Mesos
> > To: <user@mesos.apache.org (mailto:user@mesos.apache.org)>
> >
> >
> >
> > Hello,
> >
> > I am able to run Apache Spark over Mesos. Its quite simple to run Spark Dispatcher over marathon and ask it to run Spark Executor (I guess also can be called as Spark Driver) as docker container.
> >
> > I have a query regarding this:
> >
> > All spark tasks are spawned directly by first downloading the spark artifacts. I was thinking if there is some way I can start them too as docker containers. This will save the time for downloading the spark artifacts. I am running spark in fine-grained mode.
> >
> > I have attached a screenshot of a sample job
> >
> >
> >
> > Thanks,
> >
> > --
> > Pradeep Chhetri
> >
>
>
>
> --
> Pradeep Chhetri
Re: Apache Spark Over Mesos
Posted by Pradeep Chhetri <pr...@gmail.com>.
Hello Radoslaw,
Thank you for the quick reply. Few questions:
1) Do you mean mounting spark artifacts as a volume on each mesos agent
node? This means number of volumes = number of mesos agents.
2) Since I am not using HDFS at all, that is definitely not an option for
me.
Isn't there a way to just launch the spark tasks also as docker containers
which are self contained with spark artifacts ?
Thanks.
On Tue, Mar 15, 2016 at 3:49 PM, Radoslaw Gruchalski <ra...@gruchalski.com>
wrote:
> Pradeep,
>
> You can mount a spark directory as a volume. This means you have to have
> spark deployed on every agent.
>
> Another thing you can do, place spark in hdfs, assuming that you have hdfs
> available but that too will download a copy to the sandbox.
>
> I'd prefer the former.
>
> Sent from Outlook Mobile <https://aka.ms/qtex0l>
>
> _____________________________
> From: Pradeep Chhetri <pr...@gmail.com>
> Sent: Tuesday, March 15, 2016 4:41 pm
> Subject: Apache Spark Over Mesos
> To: <us...@mesos.apache.org>
>
>
>
> Hello,
>
> I am able to run Apache Spark over Mesos. Its quite simple to run Spark
> Dispatcher over marathon and ask it to run Spark Executor (I guess also can
> be called as Spark Driver) as docker container.
>
> I have a query regarding this:
>
> All spark tasks are spawned directly by first downloading the spark
> artifacts. I was thinking if there is some way I can start them too as
> docker containers. This will save the time for downloading the spark
> artifacts. I am running spark in fine-grained mode.
>
> I have attached a screenshot of a sample job
>
>
>
> Thanks,
>
> --
> Pradeep Chhetri
>
>
>
--
Pradeep Chhetri
Re: Apache Spark Over Mesos
Posted by Radoslaw Gruchalski <ra...@gruchalski.com>.
Pradeep,
You can mount a spark directory as a volume. This means you have to have spark deployed on every agent.
Another thing you can do, place spark in hdfs, assuming that you have hdfs available but that too will download a copy to the sandbox.
I'd prefer the former.
Sent from Outlook Mobile
_____________________________
From: Pradeep Chhetri <pr...@gmail.com>
Sent: Tuesday, March 15, 2016 4:41 pm
Subject: Apache Spark Over Mesos
To: <us...@mesos.apache.org>
Hello,
I am able to run Apache Spark over Mesos. Its quite simple to run Spark Dispatcher over marathon and ask it to run Spark Executor (I guess also can be called as Spark Driver) as docker container.
I have a query regarding this:
All spark tasks are spawned directly by first downloading the spark artifacts. I was thinking if there is some way I can start them too as docker containers. This will save the time for downloading the spark artifacts. I am running spark in fine-grained mode.
I have attached a screenshot of a sample job
Thanks,
--
Pradeep Chhetri