You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Yu Wei <yu...@hotmail.com> on 2016/11/14 10:58:38 UTC
Two questions about running spark on mesos
Hi Guys,
Two questions about running spark on mesos.
1, Does spark configuration of conf/slaves still work when running spark on mesos?
According to my observations, it seemed that conf/slaves still took effect when running spark-shell.
However, it doesn't take effect when deploying in cluster mode.
Is this expected behavior?
Or did I miss anything?
2, Could I kill submitted jobs when running spark on mesos in cluster mode?
I launched spark on mesos in cluster mode. Then submitted a long running job succeeded.
Then I want to kill the job.
How could I do that? Is there any similar commands as launching spark on yarn?
Thanks,
Jared, (??)
Software developer
Interested in open source software, big data, Linux
Re: Two questions about running spark on mesos
Posted by Yu Wei <yu...@hotmail.com>.
Hi Joseph,
I read the document you pointed. However, I didn't find anything about that, so I turn to community for help and guidance.
For the 2nd question, I already know how to do that.
Thanks,
Jared, (??)
Software developer
Interested in open source software, big data, Linux
________________________________
From: Joseph Wu <jo...@mesosphere.io>
Sent: Tuesday, November 15, 2016 3:40 AM
To: dev
Cc: dev
Subject: Re: Two questions about running spark on mesos
1) You should read through this page:
https://spark.apache.org/docs/latest/running-on-mesos.html
I (Mesos person) can't answer any questions that aren't already answered on
that page :)
2) Your normal spark commands (whatever they are) should still work
regardless of the backend.
On Mon, Nov 14, 2016 at 2:58 AM, Yu Wei <yu...@hotmail.com> wrote:
> Hi Guys,
>
>
> Two questions about running spark on mesos.
>
> 1, Does spark configuration of conf/slaves still work when running spark
> on mesos?
>
> According to my observations, it seemed that conf/slaves still took
> effect when running spark-shell.
>
> However, it doesn't take effect when deploying in cluster mode.
>
> Is this expected behavior?
>
> Or did I miss anything?
>
>
> 2, Could I kill submitted jobs when running spark on mesos in cluster mode?
>
> I launched spark on mesos in cluster mode. Then submitted a long
> running job succeeded.
>
> Then I want to kill the job.
>
> How could I do that? Is there any similar commands as launching spark
> on yarn?
>
>
>
>
> Thanks,
>
> Jared, (??)
> Software developer
> Interested in open source software, big data, Linux
>
Re: Two questions about running spark on mesos
Posted by Joseph Wu <jo...@mesosphere.io>.
1) You should read through this page:
https://spark.apache.org/docs/latest/running-on-mesos.html
I (Mesos person) can't answer any questions that aren't already answered on
that page :)
2) Your normal spark commands (whatever they are) should still work
regardless of the backend.
On Mon, Nov 14, 2016 at 2:58 AM, Yu Wei <yu...@hotmail.com> wrote:
> Hi Guys,
>
>
> Two questions about running spark on mesos.
>
> 1, Does spark configuration of conf/slaves still work when running spark
> on mesos?
>
> According to my observations, it seemed that conf/slaves still took
> effect when running spark-shell.
>
> However, it doesn't take effect when deploying in cluster mode.
>
> Is this expected behavior?
>
> Or did I miss anything?
>
>
> 2, Could I kill submitted jobs when running spark on mesos in cluster mode?
>
> I launched spark on mesos in cluster mode. Then submitted a long
> running job succeeded.
>
> Then I want to kill the job.
>
> How could I do that? Is there any similar commands as launching spark
> on yarn?
>
>
>
>
> Thanks,
>
> Jared, (??)
> Software developer
> Interested in open source software, big data, Linux
>
Re: Two questions about running spark on mesos
Posted by Joseph Wu <jo...@mesosphere.io>.
1) You should read through this page:
https://spark.apache.org/docs/latest/running-on-mesos.html
I (Mesos person) can't answer any questions that aren't already answered on
that page :)
2) Your normal spark commands (whatever they are) should still work
regardless of the backend.
On Mon, Nov 14, 2016 at 2:58 AM, Yu Wei <yu...@hotmail.com> wrote:
> Hi Guys,
>
>
> Two questions about running spark on mesos.
>
> 1, Does spark configuration of conf/slaves still work when running spark
> on mesos?
>
> According to my observations, it seemed that conf/slaves still took
> effect when running spark-shell.
>
> However, it doesn't take effect when deploying in cluster mode.
>
> Is this expected behavior?
>
> Or did I miss anything?
>
>
> 2, Could I kill submitted jobs when running spark on mesos in cluster mode?
>
> I launched spark on mesos in cluster mode. Then submitted a long
> running job succeeded.
>
> Then I want to kill the job.
>
> How could I do that? Is there any similar commands as launching spark
> on yarn?
>
>
>
>
> Thanks,
>
> Jared, (??)
> Software developer
> Interested in open source software, big data, Linux
>