You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Yu Wei <yu...@hotmail.com> on 2016/11/14 11:10:34 UTC
Two questions about running spark on mesos
Hi Guys,
Two questions about running spark on mesos.
1, Does spark configuration of conf/slaves still work when running spark on mesos?
According to my observations, it seemed that conf/slaves still took effect when running spark-shell.
However, it doesn't take effect when deploying in cluster mode.
Is this expected behavior?
Or did I miss anything?
2, Could I kill submitted jobs when running spark on mesos in cluster mode?
I launched spark on mesos in cluster mode. Then submitted a long running job succeeded.
Then I want to kill the job.
How could I do that? Is there any similar commands as launching spark on yarn?
Thanks,
Jared, (??)
Software developer
Interested in open source software, big data, Linux
Re: Two questions about running spark on mesos
Posted by Michael Gummelt <mg...@mesosphere.io>.
1. I had never even heard of conf/slaves until this email, and I only see
it referenced in the docs next to Spark Standalone, so I doubt that works.
2. Yes. See the --kill option in spark-submit.
Also, we're considering dropping the Spark dispatcher in DC/OS in favor of
Metronome, which will be our consolidated method of running any one-off
jobs. The dispatcher is really just a lesser maintained and more
feature-sparse metronome. If I were you, I would look into running
Metronome rather than the dispatcher (or just run DC/OS).
On Mon, Nov 14, 2016 at 3:10 AM, Yu Wei <yu...@hotmail.com> wrote:
> Hi Guys,
>
>
> Two questions about running spark on mesos.
>
> 1, Does spark configuration of conf/slaves still work when running spark
> on mesos?
>
> According to my observations, it seemed that conf/slaves still took
> effect when running spark-shell.
>
> However, it doesn't take effect when deploying in cluster mode.
>
> Is this expected behavior?
>
> Or did I miss anything?
>
>
> 2, Could I kill submitted jobs when running spark on mesos in cluster mode?
>
> I launched spark on mesos in cluster mode. Then submitted a long
> running job succeeded.
>
> Then I want to kill the job.
> How could I do that? Is there any similar commands as launching spark
> on yarn?
>
>
> Thanks,
>
> Jared, (韦煜）
> Software developer
> Interested in open source software, big data, Linux
>
--
Michael Gummelt
Software Engineer
Mesosphere
Re: Two questions about running spark on mesos
Posted by Michael Gummelt <mg...@mesosphere.io>.
1. I had never even heard of conf/slaves until this email, and I only see
it referenced in the docs next to Spark Standalone, so I doubt that works.
2. Yes. See the --kill option in spark-submit.
Also, we're considering dropping the Spark dispatcher in DC/OS in favor of
Metronome, which will be our consolidated method of running any one-off
jobs. The dispatcher is really just a lesser maintained and more
feature-sparse metronome. If I were you, I would look into running
Metronome rather than the dispatcher (or just run DC/OS).
On Mon, Nov 14, 2016 at 3:10 AM, Yu Wei <yu...@hotmail.com> wrote:
> Hi Guys,
>
>
> Two questions about running spark on mesos.
>
> 1, Does spark configuration of conf/slaves still work when running spark
> on mesos?
>
> According to my observations, it seemed that conf/slaves still took
> effect when running spark-shell.
>
> However, it doesn't take effect when deploying in cluster mode.
>
> Is this expected behavior?
>
> Or did I miss anything?
>
>
> 2, Could I kill submitted jobs when running spark on mesos in cluster mode?
>
> I launched spark on mesos in cluster mode. Then submitted a long
> running job succeeded.
>
> Then I want to kill the job.
> How could I do that? Is there any similar commands as launching spark
> on yarn?
>
>
> Thanks,
>
> Jared, (韦煜）
> Software developer
> Interested in open source software, big data, Linux
>
--
Michael Gummelt
Software Engineer
Mesosphere