You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "hsy541@gmail.com" <hs...@gmail.com> on 2014/07/14 23:08:35 UTC
How to kill running spark yarn application
Hi all,
A newbie question, I start a spark yarn application through spark-submit
....
How do I kill this app. I can kill the yarn app by "yarn application -kill
appid" but the application master is still running. What's the proper way
to shutdown the entire app?
Best,
Siyuan
Re: How to kill running spark yarn application
Posted by "hsy541@gmail.com" <hs...@gmail.com>.
Interesting, I run on my local one node cluster using apache hadoop
On Tue, Jul 15, 2014 at 7:55 AM, Jerry Lam <ch...@gmail.com> wrote:
> For your information, the SparkSubmit runs at the host you executed the
> spark-submit shell script (which in turns invoke the SparkSubmit program).
> Since you are running in yarn-cluster mode, the SparkSubmit program just
> reported the status of the job submitted to Yarn. So when you killed the
> ApplicationMaster, the SparkSubmit will return "yarnAppState: KILLED" and
> then terminated itself. This is what happens to me using cdh 5.0.2
>
> Which distribution of hadoop you are using?
>
>
> On Tue, Jul 15, 2014 at 10:42 AM, Jerry Lam <ch...@gmail.com> wrote:
>
>> when I use yarn application -kill, both SparkSubmit and ApplicationMaster
>> are killed. I also checked jps at the machine that has SparkSubmit running,
>> it is terminated as well. Sorry, I cannot reproduce it.
>>
>>
>> On Mon, Jul 14, 2014 at 7:36 PM, hsy541@gmail.com <hs...@gmail.com>
>> wrote:
>>
>>> Before "yarn application -kill" If you do jps You'll have a list
>>> of SparkSubmit and ApplicationMaster
>>>
>>> After you use yarn applicaton -kill you only kill the SparkSubmit
>>>
>>>
>>>
>>> On Mon, Jul 14, 2014 at 4:29 PM, Jerry Lam <ch...@gmail.com> wrote:
>>>
>>>> Then yarn application -kill appid should work. This is what I did 2
>>>> hours ago.
>>>>
>>>> Sorry I cannot provide more help.
>>>>
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On 14 Jul, 2014, at 6:05 pm, "hsy541@gmail.com" <hs...@gmail.com>
>>>> wrote:
>>>>
>>>> yarn-cluster
>>>>
>>>>
>>>> On Mon, Jul 14, 2014 at 2:44 PM, Jerry Lam <ch...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Siyuan,
>>>>>
>>>>> I wonder if you --master yarn-cluster or yarn-client?
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Jerry
>>>>>
>>>>>
>>>>> On Mon, Jul 14, 2014 at 5:08 PM, hsy541@gmail.com <hs...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> A newbie question, I start a spark yarn application through
>>>>>> spark-submit ....
>>>>>> How do I kill this app. I can kill the yarn app by "yarn application
>>>>>> -kill appid" but the application master is still running. What's the proper
>>>>>> way to shutdown the entire app?
>>>>>>
>>>>>> Best,
>>>>>> Siyuan
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: How to kill running spark yarn application
Posted by Jerry Lam <ch...@gmail.com>.
For your information, the SparkSubmit runs at the host you executed the
spark-submit shell script (which in turns invoke the SparkSubmit program).
Since you are running in yarn-cluster mode, the SparkSubmit program just
reported the status of the job submitted to Yarn. So when you killed the
ApplicationMaster, the SparkSubmit will return "yarnAppState: KILLED" and
then terminated itself. This is what happens to me using cdh 5.0.2
Which distribution of hadoop you are using?
On Tue, Jul 15, 2014 at 10:42 AM, Jerry Lam <ch...@gmail.com> wrote:
> when I use yarn application -kill, both SparkSubmit and ApplicationMaster
> are killed. I also checked jps at the machine that has SparkSubmit running,
> it is terminated as well. Sorry, I cannot reproduce it.
>
>
> On Mon, Jul 14, 2014 at 7:36 PM, hsy541@gmail.com <hs...@gmail.com>
> wrote:
>
>> Before "yarn application -kill" If you do jps You'll have a list
>> of SparkSubmit and ApplicationMaster
>>
>> After you use yarn applicaton -kill you only kill the SparkSubmit
>>
>>
>>
>> On Mon, Jul 14, 2014 at 4:29 PM, Jerry Lam <ch...@gmail.com> wrote:
>>
>>> Then yarn application -kill appid should work. This is what I did 2
>>> hours ago.
>>>
>>> Sorry I cannot provide more help.
>>>
>>>
>>> Sent from my iPhone
>>>
>>> On 14 Jul, 2014, at 6:05 pm, "hsy541@gmail.com" <hs...@gmail.com>
>>> wrote:
>>>
>>> yarn-cluster
>>>
>>>
>>> On Mon, Jul 14, 2014 at 2:44 PM, Jerry Lam <ch...@gmail.com> wrote:
>>>
>>>> Hi Siyuan,
>>>>
>>>> I wonder if you --master yarn-cluster or yarn-client?
>>>>
>>>> Best Regards,
>>>>
>>>> Jerry
>>>>
>>>>
>>>> On Mon, Jul 14, 2014 at 5:08 PM, hsy541@gmail.com <hs...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> A newbie question, I start a spark yarn application through
>>>>> spark-submit ....
>>>>> How do I kill this app. I can kill the yarn app by "yarn application
>>>>> -kill appid" but the application master is still running. What's the proper
>>>>> way to shutdown the entire app?
>>>>>
>>>>> Best,
>>>>> Siyuan
>>>>>
>>>>
>>>>
>>>
>>
>
Re: How to kill running spark yarn application
Posted by Jerry Lam <ch...@gmail.com>.
when I use yarn application -kill, both SparkSubmit and ApplicationMaster
are killed. I also checked jps at the machine that has SparkSubmit running,
it is terminated as well. Sorry, I cannot reproduce it.
On Mon, Jul 14, 2014 at 7:36 PM, hsy541@gmail.com <hs...@gmail.com> wrote:
> Before "yarn application -kill" If you do jps You'll have a list
> of SparkSubmit and ApplicationMaster
>
> After you use yarn applicaton -kill you only kill the SparkSubmit
>
>
>
> On Mon, Jul 14, 2014 at 4:29 PM, Jerry Lam <ch...@gmail.com> wrote:
>
>> Then yarn application -kill appid should work. This is what I did 2 hours
>> ago.
>>
>> Sorry I cannot provide more help.
>>
>>
>> Sent from my iPhone
>>
>> On 14 Jul, 2014, at 6:05 pm, "hsy541@gmail.com" <hs...@gmail.com> wrote:
>>
>> yarn-cluster
>>
>>
>> On Mon, Jul 14, 2014 at 2:44 PM, Jerry Lam <ch...@gmail.com> wrote:
>>
>>> Hi Siyuan,
>>>
>>> I wonder if you --master yarn-cluster or yarn-client?
>>>
>>> Best Regards,
>>>
>>> Jerry
>>>
>>>
>>> On Mon, Jul 14, 2014 at 5:08 PM, hsy541@gmail.com <hs...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> A newbie question, I start a spark yarn application through
>>>> spark-submit ....
>>>> How do I kill this app. I can kill the yarn app by "yarn application
>>>> -kill appid" but the application master is still running. What's the proper
>>>> way to shutdown the entire app?
>>>>
>>>> Best,
>>>> Siyuan
>>>>
>>>
>>>
>>
>
Re: How to kill running spark yarn application
Posted by "hsy541@gmail.com" <hs...@gmail.com>.
Before "yarn application -kill" If you do jps You'll have a list
of SparkSubmit and ApplicationMaster
After you use yarn applicaton -kill you only kill the SparkSubmit
On Mon, Jul 14, 2014 at 4:29 PM, Jerry Lam <ch...@gmail.com> wrote:
> Then yarn application -kill appid should work. This is what I did 2 hours
> ago.
>
> Sorry I cannot provide more help.
>
>
> Sent from my iPhone
>
> On 14 Jul, 2014, at 6:05 pm, "hsy541@gmail.com" <hs...@gmail.com> wrote:
>
> yarn-cluster
>
>
> On Mon, Jul 14, 2014 at 2:44 PM, Jerry Lam <ch...@gmail.com> wrote:
>
>> Hi Siyuan,
>>
>> I wonder if you --master yarn-cluster or yarn-client?
>>
>> Best Regards,
>>
>> Jerry
>>
>>
>> On Mon, Jul 14, 2014 at 5:08 PM, hsy541@gmail.com <hs...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> A newbie question, I start a spark yarn application through spark-submit
>>> ....
>>> How do I kill this app. I can kill the yarn app by "yarn application
>>> -kill appid" but the application master is still running. What's the proper
>>> way to shutdown the entire app?
>>>
>>> Best,
>>> Siyuan
>>>
>>
>>
>
Re: How to kill running spark yarn application
Posted by Jerry Lam <ch...@gmail.com>.
Then yarn application -kill appid should work. This is what I did 2 hours ago.
Sorry I cannot provide more help.
Sent from my iPhone
> On 14 Jul, 2014, at 6:05 pm, "hsy541@gmail.com" <hs...@gmail.com> wrote:
>
> yarn-cluster
>
>
>> On Mon, Jul 14, 2014 at 2:44 PM, Jerry Lam <ch...@gmail.com> wrote:
>> Hi Siyuan,
>>
>> I wonder if you --master yarn-cluster or yarn-client?
>>
>> Best Regards,
>>
>> Jerry
>>
>>
>>> On Mon, Jul 14, 2014 at 5:08 PM, hsy541@gmail.com <hs...@gmail.com> wrote:
>>> Hi all,
>>>
>>> A newbie question, I start a spark yarn application through spark-submit ....
>>> How do I kill this app. I can kill the yarn app by "yarn application -kill appid" but the application master is still running. What's the proper way to shutdown the entire app?
>>>
>>> Best,
>>> Siyuan
>
Re: How to kill running spark yarn application
Posted by "hsy541@gmail.com" <hs...@gmail.com>.
yarn-cluster
On Mon, Jul 14, 2014 at 2:44 PM, Jerry Lam <ch...@gmail.com> wrote:
> Hi Siyuan,
>
> I wonder if you --master yarn-cluster or yarn-client?
>
> Best Regards,
>
> Jerry
>
>
> On Mon, Jul 14, 2014 at 5:08 PM, hsy541@gmail.com <hs...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> A newbie question, I start a spark yarn application through spark-submit
>> ....
>> How do I kill this app. I can kill the yarn app by "yarn application
>> -kill appid" but the application master is still running. What's the proper
>> way to shutdown the entire app?
>>
>> Best,
>> Siyuan
>>
>
>
Re: How to kill running spark yarn application
Posted by Jerry Lam <ch...@gmail.com>.
Hi Siyuan,
I wonder if you --master yarn-cluster or yarn-client?
Best Regards,
Jerry
On Mon, Jul 14, 2014 at 5:08 PM, hsy541@gmail.com <hs...@gmail.com> wrote:
> Hi all,
>
> A newbie question, I start a spark yarn application through spark-submit
> ....
> How do I kill this app. I can kill the yarn app by "yarn application -kill
> appid" but the application master is still running. What's the proper way
> to shutdown the entire app?
>
> Best,
> Siyuan
>