You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by David Yu <da...@optimizely.com> on 2016/05/18 06:54:42 UTC
Samza job killed by left orphaned on YARN
Samza version = 0.10.0
YARN version = Hadoop 2.6.0-cdh5.4.9
We are experience issues when killing a Samza job:
$ yarn application -kill application_1463512986427_0007
Killing application application_1463512986427_0007
16/05/18 06:29:05 INFO impl.YarnClientImpl: Killed application
application_1463512986427_0007
RM shows that the job is killed. However, the samza containers are still
left running.
Any idea why this is happening?
Thanks,
David
Re: Samza job killed by left orphaned on YARN
Posted by Yi Pan <ni...@gmail.com>.
Hi, David and all,
The "ultimate" solution is probably to implement SAMZA-871
<https://issues.apache.org/jira/browse/SAMZA-871>, which allows Samza
JobCoordinator directly identifies whether a container is alive or not w/o
dependency on the cluster management systems. This is also considered
together w/ SAMZA-881 where we are refactoring the JobCoordinator code to
allow standalone deployment of Samza jobs. Hope that set the direction
right. If you have other concerns on this, please speak up in those JIRAs.
Thanks!
-Yi
On Thu, May 19, 2016 at 2:33 PM, David Yu <da...@optimizely.com> wrote:
> Just stumbled upon this post and sees to be the same issue:
>
> https://issues.apache.org/jira/browse/SAMZA-498
>
>
> We followed the fix to create a wrapper kill script and everything works.
>
> Do we have a plan to fix this in the next version of Samza?
>
> Thanks,
> David
>
> On Wed, May 18, 2016 at 11:53 AM, Jacob Maes <ja...@gmail.com> wrote:
>
> > Hmm, could there be something in your job holding up the container
> shutdown
> > process? Perhaps something ignoring SIGTERM/Thread.interrupt, by chance?
> >
> > Also, I think there's a YARN property specifying the amount of time the
> NM
> > waits between sending a SIGTERM and a SIGKILL, though I can't find it at
> > the moment.
> >
> > -Jake
> >
> > On Wed, May 18, 2016 at 10:32 AM, David Yu <da...@optimizely.com>
> > wrote:
> >
> > > From the NM log, I'm seeing:
> > >
> > > 2016-05-18 06:29:06,248 INFO
> > >
> > >
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
> > > Cleaning up container
> > container_e01_1463512986427_0007_01_0000022016-05-18
> > > 06:29:06,265 INFO
> > >
> > >
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> > > Application *application_1463512986427_0007* transitioned from RUNNING
> to
> > > FINISHING_CONTAINERS_WAIT
> > >
> > > (*Highlighted* is the particular samza application.)
> > >
> > > The status never transitioned from FINISHING_CONTAINERS_WAIT :(
> > >
> > >
> > >
> > > On Wed, May 18, 2016 at 10:21 AM, David Yu <da...@optimizely.com>
> > > wrote:
> > >
> > > > Jacob,
> > > >
> > > > I have checked and made sure that NM is running on the node:
> > > >
> > > > $ ps aux | grep java
> > > > ...
> > > > yarn 25623 0.5 0.8 2366536 275488 ? Sl May17 7:04
> > > > /usr/java/jdk1.8.0_51/bin/java -Dproc_nodemanager
> > > > ... org.apache.hadoop.yarn.server.nodemanager.NodeManager
> > > >
> > > >
> > > >
> > > > Thanks,
> > > > David
> > > >
> > > > On Wed, May 18, 2016 at 7:08 AM, Jacob Maes <ja...@gmail.com>
> > > wrote:
> > > >
> > > >> Hey David,
> > > >>
> > > >> The only time I've seen orphaned containers is when the NM dies. If
> > the
> > > NM
> > > >> isn't running, the RM has no means to kill the containers on a node.
> > Can
> > > >> you verify that the NM was healthy at the time of the shut down?
> > > >>
> > > >> If it wasn't healthy and/or it was restarted, one option that may
> help
> > > is
> > > >> NM Recovery:
> > > >>
> > > >>
> > >
> >
> https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/NodeManagerRestart.html
> > > >>
> > > >> With NM Recovery, the NM will resume control over containers that
> were
> > > >> running when the NM shut down. This option has virtually eliminated
> > > >> orphaned containers in our clusters.
> > > >>
> > > >> -Jake
> > > >>
> > > >> On Tue, May 17, 2016 at 11:54 PM, David Yu <david.yu@optimizely.com
> >
> > > >> wrote:
> > > >>
> > > >> > Samza version = 0.10.0
> > > >> > YARN version = Hadoop 2.6.0-cdh5.4.9
> > > >> >
> > > >> > We are experience issues when killing a Samza job:
> > > >> >
> > > >> > $ yarn application -kill application_1463512986427_0007
> > > >> >
> > > >> > Killing application application_1463512986427_0007
> > > >> >
> > > >> > 16/05/18 06:29:05 INFO impl.YarnClientImpl: Killed application
> > > >> > application_1463512986427_0007
> > > >> >
> > > >> > RM shows that the job is killed. However, the samza containers are
> > > still
> > > >> > left running.
> > > >> >
> > > >> > Any idea why this is happening?
> > > >> >
> > > >> > Thanks,
> > > >> > David
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
Re: Samza job killed by left orphaned on YARN
Posted by David Yu <da...@optimizely.com>.
Just stumbled upon this post and sees to be the same issue:
https://issues.apache.org/jira/browse/SAMZA-498
We followed the fix to create a wrapper kill script and everything works.
Do we have a plan to fix this in the next version of Samza?
Thanks,
David
On Wed, May 18, 2016 at 11:53 AM, Jacob Maes <ja...@gmail.com> wrote:
> Hmm, could there be something in your job holding up the container shutdown
> process? Perhaps something ignoring SIGTERM/Thread.interrupt, by chance?
>
> Also, I think there's a YARN property specifying the amount of time the NM
> waits between sending a SIGTERM and a SIGKILL, though I can't find it at
> the moment.
>
> -Jake
>
> On Wed, May 18, 2016 at 10:32 AM, David Yu <da...@optimizely.com>
> wrote:
>
> > From the NM log, I'm seeing:
> >
> > 2016-05-18 06:29:06,248 INFO
> >
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
> > Cleaning up container
> container_e01_1463512986427_0007_01_0000022016-05-18
> > 06:29:06,265 INFO
> >
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> > Application *application_1463512986427_0007* transitioned from RUNNING to
> > FINISHING_CONTAINERS_WAIT
> >
> > (*Highlighted* is the particular samza application.)
> >
> > The status never transitioned from FINISHING_CONTAINERS_WAIT :(
> >
> >
> >
> > On Wed, May 18, 2016 at 10:21 AM, David Yu <da...@optimizely.com>
> > wrote:
> >
> > > Jacob,
> > >
> > > I have checked and made sure that NM is running on the node:
> > >
> > > $ ps aux | grep java
> > > ...
> > > yarn 25623 0.5 0.8 2366536 275488 ? Sl May17 7:04
> > > /usr/java/jdk1.8.0_51/bin/java -Dproc_nodemanager
> > > ... org.apache.hadoop.yarn.server.nodemanager.NodeManager
> > >
> > >
> > >
> > > Thanks,
> > > David
> > >
> > > On Wed, May 18, 2016 at 7:08 AM, Jacob Maes <ja...@gmail.com>
> > wrote:
> > >
> > >> Hey David,
> > >>
> > >> The only time I've seen orphaned containers is when the NM dies. If
> the
> > NM
> > >> isn't running, the RM has no means to kill the containers on a node.
> Can
> > >> you verify that the NM was healthy at the time of the shut down?
> > >>
> > >> If it wasn't healthy and/or it was restarted, one option that may help
> > is
> > >> NM Recovery:
> > >>
> > >>
> >
> https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/NodeManagerRestart.html
> > >>
> > >> With NM Recovery, the NM will resume control over containers that were
> > >> running when the NM shut down. This option has virtually eliminated
> > >> orphaned containers in our clusters.
> > >>
> > >> -Jake
> > >>
> > >> On Tue, May 17, 2016 at 11:54 PM, David Yu <da...@optimizely.com>
> > >> wrote:
> > >>
> > >> > Samza version = 0.10.0
> > >> > YARN version = Hadoop 2.6.0-cdh5.4.9
> > >> >
> > >> > We are experience issues when killing a Samza job:
> > >> >
> > >> > $ yarn application -kill application_1463512986427_0007
> > >> >
> > >> > Killing application application_1463512986427_0007
> > >> >
> > >> > 16/05/18 06:29:05 INFO impl.YarnClientImpl: Killed application
> > >> > application_1463512986427_0007
> > >> >
> > >> > RM shows that the job is killed. However, the samza containers are
> > still
> > >> > left running.
> > >> >
> > >> > Any idea why this is happening?
> > >> >
> > >> > Thanks,
> > >> > David
> > >> >
> > >>
> > >
> > >
> >
>
Re: Samza job killed by left orphaned on YARN
Posted by Jacob Maes <ja...@gmail.com>.
Hmm, could there be something in your job holding up the container shutdown
process? Perhaps something ignoring SIGTERM/Thread.interrupt, by chance?
Also, I think there's a YARN property specifying the amount of time the NM
waits between sending a SIGTERM and a SIGKILL, though I can't find it at
the moment.
-Jake
On Wed, May 18, 2016 at 10:32 AM, David Yu <da...@optimizely.com> wrote:
> From the NM log, I'm seeing:
>
> 2016-05-18 06:29:06,248 INFO
>
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
> Cleaning up container container_e01_1463512986427_0007_01_0000022016-05-18
> 06:29:06,265 INFO
>
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Application *application_1463512986427_0007* transitioned from RUNNING to
> FINISHING_CONTAINERS_WAIT
>
> (*Highlighted* is the particular samza application.)
>
> The status never transitioned from FINISHING_CONTAINERS_WAIT :(
>
>
>
> On Wed, May 18, 2016 at 10:21 AM, David Yu <da...@optimizely.com>
> wrote:
>
> > Jacob,
> >
> > I have checked and made sure that NM is running on the node:
> >
> > $ ps aux | grep java
> > ...
> > yarn 25623 0.5 0.8 2366536 275488 ? Sl May17 7:04
> > /usr/java/jdk1.8.0_51/bin/java -Dproc_nodemanager
> > ... org.apache.hadoop.yarn.server.nodemanager.NodeManager
> >
> >
> >
> > Thanks,
> > David
> >
> > On Wed, May 18, 2016 at 7:08 AM, Jacob Maes <ja...@gmail.com>
> wrote:
> >
> >> Hey David,
> >>
> >> The only time I've seen orphaned containers is when the NM dies. If the
> NM
> >> isn't running, the RM has no means to kill the containers on a node. Can
> >> you verify that the NM was healthy at the time of the shut down?
> >>
> >> If it wasn't healthy and/or it was restarted, one option that may help
> is
> >> NM Recovery:
> >>
> >>
> https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/NodeManagerRestart.html
> >>
> >> With NM Recovery, the NM will resume control over containers that were
> >> running when the NM shut down. This option has virtually eliminated
> >> orphaned containers in our clusters.
> >>
> >> -Jake
> >>
> >> On Tue, May 17, 2016 at 11:54 PM, David Yu <da...@optimizely.com>
> >> wrote:
> >>
> >> > Samza version = 0.10.0
> >> > YARN version = Hadoop 2.6.0-cdh5.4.9
> >> >
> >> > We are experience issues when killing a Samza job:
> >> >
> >> > $ yarn application -kill application_1463512986427_0007
> >> >
> >> > Killing application application_1463512986427_0007
> >> >
> >> > 16/05/18 06:29:05 INFO impl.YarnClientImpl: Killed application
> >> > application_1463512986427_0007
> >> >
> >> > RM shows that the job is killed. However, the samza containers are
> still
> >> > left running.
> >> >
> >> > Any idea why this is happening?
> >> >
> >> > Thanks,
> >> > David
> >> >
> >>
> >
> >
>
Re: Samza job killed by left orphaned on YARN
Posted by David Yu <da...@optimizely.com>.
From the NM log, I'm seeing:
2016-05-18 06:29:06,248 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Cleaning up container container_e01_1463512986427_0007_01_0000022016-05-18
06:29:06,265 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Application *application_1463512986427_0007* transitioned from RUNNING to
FINISHING_CONTAINERS_WAIT
(*Highlighted* is the particular samza application.)
The status never transitioned from FINISHING_CONTAINERS_WAIT :(
On Wed, May 18, 2016 at 10:21 AM, David Yu <da...@optimizely.com> wrote:
> Jacob,
>
> I have checked and made sure that NM is running on the node:
>
> $ ps aux | grep java
> ...
> yarn 25623 0.5 0.8 2366536 275488 ? Sl May17 7:04
> /usr/java/jdk1.8.0_51/bin/java -Dproc_nodemanager
> ... org.apache.hadoop.yarn.server.nodemanager.NodeManager
>
>
>
> Thanks,
> David
>
> On Wed, May 18, 2016 at 7:08 AM, Jacob Maes <ja...@gmail.com> wrote:
>
>> Hey David,
>>
>> The only time I've seen orphaned containers is when the NM dies. If the NM
>> isn't running, the RM has no means to kill the containers on a node. Can
>> you verify that the NM was healthy at the time of the shut down?
>>
>> If it wasn't healthy and/or it was restarted, one option that may help is
>> NM Recovery:
>>
>> https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/NodeManagerRestart.html
>>
>> With NM Recovery, the NM will resume control over containers that were
>> running when the NM shut down. This option has virtually eliminated
>> orphaned containers in our clusters.
>>
>> -Jake
>>
>> On Tue, May 17, 2016 at 11:54 PM, David Yu <da...@optimizely.com>
>> wrote:
>>
>> > Samza version = 0.10.0
>> > YARN version = Hadoop 2.6.0-cdh5.4.9
>> >
>> > We are experience issues when killing a Samza job:
>> >
>> > $ yarn application -kill application_1463512986427_0007
>> >
>> > Killing application application_1463512986427_0007
>> >
>> > 16/05/18 06:29:05 INFO impl.YarnClientImpl: Killed application
>> > application_1463512986427_0007
>> >
>> > RM shows that the job is killed. However, the samza containers are still
>> > left running.
>> >
>> > Any idea why this is happening?
>> >
>> > Thanks,
>> > David
>> >
>>
>
>
Re: Samza job killed by left orphaned on YARN
Posted by David Yu <da...@optimizely.com>.
Jacob,
I have checked and made sure that NM is running on the node:
$ ps aux | grep java
...
yarn 25623 0.5 0.8 2366536 275488 ? Sl May17 7:04
/usr/java/jdk1.8.0_51/bin/java -Dproc_nodemanager
... org.apache.hadoop.yarn.server.nodemanager.NodeManager
Thanks,
David
On Wed, May 18, 2016 at 7:08 AM, Jacob Maes <ja...@gmail.com> wrote:
> Hey David,
>
> The only time I've seen orphaned containers is when the NM dies. If the NM
> isn't running, the RM has no means to kill the containers on a node. Can
> you verify that the NM was healthy at the time of the shut down?
>
> If it wasn't healthy and/or it was restarted, one option that may help is
> NM Recovery:
>
> https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/NodeManagerRestart.html
>
> With NM Recovery, the NM will resume control over containers that were
> running when the NM shut down. This option has virtually eliminated
> orphaned containers in our clusters.
>
> -Jake
>
> On Tue, May 17, 2016 at 11:54 PM, David Yu <da...@optimizely.com>
> wrote:
>
> > Samza version = 0.10.0
> > YARN version = Hadoop 2.6.0-cdh5.4.9
> >
> > We are experience issues when killing a Samza job:
> >
> > $ yarn application -kill application_1463512986427_0007
> >
> > Killing application application_1463512986427_0007
> >
> > 16/05/18 06:29:05 INFO impl.YarnClientImpl: Killed application
> > application_1463512986427_0007
> >
> > RM shows that the job is killed. However, the samza containers are still
> > left running.
> >
> > Any idea why this is happening?
> >
> > Thanks,
> > David
> >
>
Re: Samza job killed by left orphaned on YARN
Posted by Jacob Maes <ja...@gmail.com>.
Hey David,
The only time I've seen orphaned containers is when the NM dies. If the NM
isn't running, the RM has no means to kill the containers on a node. Can
you verify that the NM was healthy at the time of the shut down?
If it wasn't healthy and/or it was restarted, one option that may help is
NM Recovery:
https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/NodeManagerRestart.html
With NM Recovery, the NM will resume control over containers that were
running when the NM shut down. This option has virtually eliminated
orphaned containers in our clusters.
-Jake
On Tue, May 17, 2016 at 11:54 PM, David Yu <da...@optimizely.com> wrote:
> Samza version = 0.10.0
> YARN version = Hadoop 2.6.0-cdh5.4.9
>
> We are experience issues when killing a Samza job:
>
> $ yarn application -kill application_1463512986427_0007
>
> Killing application application_1463512986427_0007
>
> 16/05/18 06:29:05 INFO impl.YarnClientImpl: Killed application
> application_1463512986427_0007
>
> RM shows that the job is killed. However, the samza containers are still
> left running.
>
> Any idea why this is happening?
>
> Thanks,
> David
>