You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by satish saley <sa...@gmail.com> on 2016/05/06 18:50:39 UTC
killing spark job which is submitted using SparkSubmit
Hello,
I am submitting a spark job using SparkSubmit. When I kill my application,
it does not kill the corresponding spark job. How would I kill the
corresponding spark job? I know, one way is to use SparkSubmit again with
appropriate options. Is there any way though which I can tell SparkSubmit
at the time of job submission itself. Here is my code:
-
import org.apache.spark.deploy.SparkSubmit;
- class MyClass{
-
- public static void main(String args[]){
- //preparing args
- SparkSubmit.main(args);
- }
-
- }
Re: killing spark job which is submitted using SparkSubmit
Posted by satish saley <sa...@gmail.com>.
Thank you Anthony. I am clearer on yarn-cluster and yarn-client now.
On Fri, May 6, 2016 at 1:05 PM, Anthony May <an...@gmail.com> wrote:
> Making the master yarn-cluster means that the driver is then running on
> YARN not just the executor nodes. It's then independent of your application
> and can only be killed via YARN commands, or if it's batch and completes.
> The simplest way to tie the driver to your app is to pass in yarn-client as
> master instead.
>
> On Fri, May 6, 2016 at 2:00 PM satish saley <sa...@gmail.com>
> wrote:
>
>> Hi Anthony,
>>
>> I am passing
>>
>> --master
>> yarn-cluster
>> --name
>> pysparkexample
>> --executor-memory
>> 1G
>> --driver-memory
>> 1G
>> --conf
>> spark.yarn.historyServer.address=http://localhost:18080
>> --conf
>> spark.eventLog.enabled=true
>>
>> --verbose
>>
>> pi.py
>>
>>
>> I am able to run the job successfully. I just want to get it killed automatically whenever I kill my application.
>>
>>
>> On Fri, May 6, 2016 at 11:58 AM, Anthony May <an...@gmail.com>
>> wrote:
>>
>>> Greetings Satish,
>>>
>>> What are the arguments you're passing in?
>>>
>>> On Fri, 6 May 2016 at 12:50 satish saley <sa...@gmail.com>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> I am submitting a spark job using SparkSubmit. When I kill my
>>>> application, it does not kill the corresponding spark job. How would I kill
>>>> the corresponding spark job? I know, one way is to use SparkSubmit again
>>>> with appropriate options. Is there any way though which I can tell
>>>> SparkSubmit at the time of job submission itself. Here is my code:
>>>>
>>>>
>>>> -
>>>> import org.apache.spark.deploy.SparkSubmit;
>>>> - class MyClass{
>>>> -
>>>> - public static void main(String args[]){
>>>> - //preparing args
>>>> - SparkSubmit.main(args);
>>>> - }
>>>> -
>>>> - }
>>>>
>>>>
>>
Re: killing spark job which is submitted using SparkSubmit
Posted by Anthony May <an...@gmail.com>.
Making the master yarn-cluster means that the driver is then running on
YARN not just the executor nodes. It's then independent of your application
and can only be killed via YARN commands, or if it's batch and completes.
The simplest way to tie the driver to your app is to pass in yarn-client as
master instead.
On Fri, May 6, 2016 at 2:00 PM satish saley <sa...@gmail.com> wrote:
> Hi Anthony,
>
> I am passing
>
> --master
> yarn-cluster
> --name
> pysparkexample
> --executor-memory
> 1G
> --driver-memory
> 1G
> --conf
> spark.yarn.historyServer.address=http://localhost:18080
> --conf
> spark.eventLog.enabled=true
>
> --verbose
>
> pi.py
>
>
> I am able to run the job successfully. I just want to get it killed automatically whenever I kill my application.
>
>
> On Fri, May 6, 2016 at 11:58 AM, Anthony May <an...@gmail.com> wrote:
>
>> Greetings Satish,
>>
>> What are the arguments you're passing in?
>>
>> On Fri, 6 May 2016 at 12:50 satish saley <sa...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I am submitting a spark job using SparkSubmit. When I kill my
>>> application, it does not kill the corresponding spark job. How would I kill
>>> the corresponding spark job? I know, one way is to use SparkSubmit again
>>> with appropriate options. Is there any way though which I can tell
>>> SparkSubmit at the time of job submission itself. Here is my code:
>>>
>>>
>>> -
>>> import org.apache.spark.deploy.SparkSubmit;
>>> - class MyClass{
>>> -
>>> - public static void main(String args[]){
>>> - //preparing args
>>> - SparkSubmit.main(args);
>>> - }
>>> -
>>> - }
>>>
>>>
>
Re: killing spark job which is submitted using SparkSubmit
Posted by satish saley <sa...@gmail.com>.
Hi Anthony,
I am passing
--master
yarn-cluster
--name
pysparkexample
--executor-memory
1G
--driver-memory
1G
--conf
spark.yarn.historyServer.address=http://localhost:18080
--conf
spark.eventLog.enabled=true
--verbose
pi.py
I am able to run the job successfully. I just want to get it killed
automatically whenever I kill my application.
On Fri, May 6, 2016 at 11:58 AM, Anthony May <an...@gmail.com> wrote:
> Greetings Satish,
>
> What are the arguments you're passing in?
>
> On Fri, 6 May 2016 at 12:50 satish saley <sa...@gmail.com> wrote:
>
>> Hello,
>>
>> I am submitting a spark job using SparkSubmit. When I kill my
>> application, it does not kill the corresponding spark job. How would I kill
>> the corresponding spark job? I know, one way is to use SparkSubmit again
>> with appropriate options. Is there any way though which I can tell
>> SparkSubmit at the time of job submission itself. Here is my code:
>>
>>
>> -
>> import org.apache.spark.deploy.SparkSubmit;
>> - class MyClass{
>> -
>> - public static void main(String args[]){
>> - //preparing args
>> - SparkSubmit.main(args);
>> - }
>> -
>> - }
>>
>>
Re: killing spark job which is submitted using SparkSubmit
Posted by Anthony May <an...@gmail.com>.
Greetings Satish,
What are the arguments you're passing in?
On Fri, 6 May 2016 at 12:50 satish saley <sa...@gmail.com> wrote:
> Hello,
>
> I am submitting a spark job using SparkSubmit. When I kill my application,
> it does not kill the corresponding spark job. How would I kill the
> corresponding spark job? I know, one way is to use SparkSubmit again with
> appropriate options. Is there any way though which I can tell SparkSubmit
> at the time of job submission itself. Here is my code:
>
>
> -
> import org.apache.spark.deploy.SparkSubmit;
> - class MyClass{
> -
> - public static void main(String args[]){
> - //preparing args
> - SparkSubmit.main(args);
> - }
> -
> - }
>
>