You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@samza.apache.org by Zach Cox <zc...@gmail.com> on 2015/01/13 20:21:46 UTC

Graceful shutdown

Hi - just wondering what other folks do when a Samza job needs to be
shutdown purposefully, such as during a deployment. Seems like in this case
you'd want all of the job's Samza containers to cleanly stop processing
messages, commit offsets/changelog, etc instead of just SIGKILL the jvms
which will result in out-of-date checkpoints, etc.

I noticed TaskCoordinator has a shutdown method, which the StreamTasks in a
container can use to signal a clean shutdown. Does everyone use this, with
some kind of shutdown message sent to the StreamTasks using e.g. jmx or a
kafka topic, as in this thread?
http://www.mail-archive.com/dev%40samza.incubator.apache.org/msg01248.html

We're actively developing Samza support for Mesos, so maybe this is
specific to that world, but would it make sense for a jvm shutdown hook in
the container to catch a SIGTERM and somehow signal to the container that
it should shutdown? I can sorta make this work, but not in a very clean way.

Just curious what others are doing about graceful shutdown.

Thanks,
Zach

Re: Graceful shutdown

Posted by Dotan Patrich <do...@fortscale.com>.

Hey,

Just adding my 2c here - we've been pretty ok with creating a custom job
factory that listens on JVM shutdown hooks and notify the ThreadJob to stop
executing. This will trigger the RunLoop to stop executing and call the
close method on all running task instances. You could see an example for it
here:
https://gist.github.com/anonymous/6ceb7ecbf80775380725

I haven't had a chance to investigate how the YARN based job factory
handles it, but if you have any experience on running it on top of
mesos/docker it would be very interesting to hear

Thanks,
Dotan


On Wed, Jan 14, 2015 at 12:15 AM, Chris Riccomini <
criccomini@linkedin.com.invalid> wrote:

> Hey Zach,
>
> Ah, I see what you're saying. Yes, to implement this as a bolt-on on top
> of the framework is somewhat cumbersome.
>
> The "right" thing to do is to put the addShutdownHook call into RunLoop,
> itself, and have it change the shutdownNow variable when it gets a
> SIGTERM. That requires a patch, though. :) In the meantime, what you have
> seems like it should work, but I agree that it's a bit hacky.
>
> Cheers,
> Chris
>
> On 1/13/15 1:57 PM, "Zach Cox" <zc...@gmail.com> wrote:
>
> >I created a jira ticket for this:
> >https://issues.apache.org/jira/browse/SAMZA-506
> >
> >On Tue, Jan 13, 2015 at 3:49 PM, Zach Cox <zc...@gmail.com> wrote:
> >
> >> Hi Chris,
> >>
> >> Here's the main idea of the SIGTERM shutdown hook approach that we're
> >> trying out: https://gist.github.com/zcox/6ec8910bd3f18e36c1a2
> >>
> >> We use our own main class to run either the job or the container
> >>depending
> >> on args, and this main is the ENTRYPOINT of our Docker container,
> >>instead
> >> of using the standard run-job.sh and run-container.sh scripts. So this
> >>is
> >> an easy place to add the shutdown hook. Not sure how reusable that
> >>would be
> >> for others...
> >>
> >> By "not in a very clean way" I meant that the shutdown hook needs to
> >>block
> >> jvm shutdown until SamzaContainer has gone through its entire list of
> >> shutdown calls, and the only way I could find for the shutdown hook to
> >>know
> >> this is finished is the shutdownIsComplete flag, set true after
> >> SamzaContainer.safeMain() returns, which feels a bit jank, but it seems
> >> to work.
> >>
> >> Thanks,
> >> Zach
> >>
> >>
> >> On Tue, Jan 13, 2015 at 2:34 PM, Chris Riccomini <
> >> criccomini@linkedin.com.invalid> wrote:
> >>
> >>> Hey Zach,
> >>>
> >>> Honestly, we just tell YARN to kill everything off. As you said, this
> >>> means there is no clean shutdown, but we've lived with thus far.
> >>>
> >>> Some folks have also used a control message, as you've said, and had
> >>>their
> >>> task call coordinator.shutdown(). Others have used a JMX switch to do
> >>>the
> >>> same.
> >>>
> >>> I agree that catching a SIGTERM is a little cleaner. Can you open a
> >>>JIRA
> >>> for this?
> >>>
> >>> Basically, all the solutions you've described are possible. Only the
> >>> SIGTERM trap hasn't been implemented.
> >>>
> >>> > I can sorta make this work, but not in a very clean way.
> >>>
> >>>
> >>> Curious: what's not clean?
> >>>
> >>> Cheers,
> >>> Chris
> >>>
> >>> On 1/13/15 11:21 AM, "Zach Cox" <zc...@gmail.com> wrote:
> >>>
> >>> >Hi - just wondering what other folks do when a Samza job needs to be
> >>> >shutdown purposefully, such as during a deployment. Seems like in this
> >>> >case
> >>> >you'd want all of the job's Samza containers to cleanly stop
> >>>processing
> >>> >messages, commit offsets/changelog, etc instead of just SIGKILL the
> >>>jvms
> >>> >which will result in out-of-date checkpoints, etc.
> >>> >
> >>> >I noticed TaskCoordinator has a shutdown method, which the
> >>>StreamTasks in
> >>> >a
> >>> >container can use to signal a clean shutdown. Does everyone use this,
> >>> with
> >>> >some kind of shutdown message sent to the StreamTasks using e.g. jmx
> >>>or a
> >>> >kafka topic, as in this thread?
> >>> >
> >>>
> >>>
> http://www.mail-archive.com/dev%40samza.incubator.apache.org/msg01248.ht
> >>>ml
> >>> >
> >>> >We're actively developing Samza support for Mesos, so maybe this is
> >>> >specific to that world, but would it make sense for a jvm shutdown
> >>>hook
> >>> in
> >>> >the container to catch a SIGTERM and somehow signal to the container
> >>>that
> >>> >it should shutdown? I can sorta make this work, but not in a very
> >>>clean
> >>> >way.
> >>> >
> >>> >Just curious what others are doing about graceful shutdown.
> >>> >
> >>> >Thanks,
> >>> >Zach
> >>>
> >>>
> >>
>
>

Re: Graceful shutdown

Posted by Chris Riccomini <cr...@linkedin.com.INVALID>.

Hey Zach,

Ah, I see what you're saying. Yes, to implement this as a bolt-on on top
of the framework is somewhat cumbersome.

The "right" thing to do is to put the addShutdownHook call into RunLoop,
itself, and have it change the shutdownNow variable when it gets a
SIGTERM. That requires a patch, though. :) In the meantime, what you have
seems like it should work, but I agree that it's a bit hacky.

Cheers,
Chris

On 1/13/15 1:57 PM, "Zach Cox" <zc...@gmail.com> wrote:

>I created a jira ticket for this:
>https://issues.apache.org/jira/browse/SAMZA-506
>
>On Tue, Jan 13, 2015 at 3:49 PM, Zach Cox <zc...@gmail.com> wrote:
>
>> Hi Chris,
>>
>> Here's the main idea of the SIGTERM shutdown hook approach that we're
>> trying out: https://gist.github.com/zcox/6ec8910bd3f18e36c1a2
>>
>> We use our own main class to run either the job or the container
>>depending
>> on args, and this main is the ENTRYPOINT of our Docker container,
>>instead
>> of using the standard run-job.sh and run-container.sh scripts. So this
>>is
>> an easy place to add the shutdown hook. Not sure how reusable that
>>would be
>> for others...
>>
>> By "not in a very clean way" I meant that the shutdown hook needs to
>>block
>> jvm shutdown until SamzaContainer has gone through its entire list of
>> shutdown calls, and the only way I could find for the shutdown hook to
>>know
>> this is finished is the shutdownIsComplete flag, set true after
>> SamzaContainer.safeMain() returns, which feels a bit jank, but it seems
>> to work.
>>
>> Thanks,
>> Zach
>>
>>
>> On Tue, Jan 13, 2015 at 2:34 PM, Chris Riccomini <
>> criccomini@linkedin.com.invalid> wrote:
>>
>>> Hey Zach,
>>>
>>> Honestly, we just tell YARN to kill everything off. As you said, this
>>> means there is no clean shutdown, but we've lived with thus far.
>>>
>>> Some folks have also used a control message, as you've said, and had
>>>their
>>> task call coordinator.shutdown(). Others have used a JMX switch to do
>>>the
>>> same.
>>>
>>> I agree that catching a SIGTERM is a little cleaner. Can you open a
>>>JIRA
>>> for this?
>>>
>>> Basically, all the solutions you've described are possible. Only the
>>> SIGTERM trap hasn't been implemented.
>>>
>>> > I can sorta make this work, but not in a very clean way.
>>>
>>>
>>> Curious: what's not clean?
>>>
>>> Cheers,
>>> Chris
>>>
>>> On 1/13/15 11:21 AM, "Zach Cox" <zc...@gmail.com> wrote:
>>>
>>> >Hi - just wondering what other folks do when a Samza job needs to be
>>> >shutdown purposefully, such as during a deployment. Seems like in this
>>> >case
>>> >you'd want all of the job's Samza containers to cleanly stop
>>>processing
>>> >messages, commit offsets/changelog, etc instead of just SIGKILL the
>>>jvms
>>> >which will result in out-of-date checkpoints, etc.
>>> >
>>> >I noticed TaskCoordinator has a shutdown method, which the
>>>StreamTasks in
>>> >a
>>> >container can use to signal a clean shutdown. Does everyone use this,
>>> with
>>> >some kind of shutdown message sent to the StreamTasks using e.g. jmx
>>>or a
>>> >kafka topic, as in this thread?
>>> >
>>> 
>>>http://www.mail-archive.com/dev%40samza.incubator.apache.org/msg01248.ht
>>>ml
>>> >
>>> >We're actively developing Samza support for Mesos, so maybe this is
>>> >specific to that world, but would it make sense for a jvm shutdown
>>>hook
>>> in
>>> >the container to catch a SIGTERM and somehow signal to the container
>>>that
>>> >it should shutdown? I can sorta make this work, but not in a very
>>>clean
>>> >way.
>>> >
>>> >Just curious what others are doing about graceful shutdown.
>>> >
>>> >Thanks,
>>> >Zach
>>>
>>>
>>

Re: Graceful shutdown

Posted by Zach Cox <zc...@gmail.com>.

I created a jira ticket for this:
https://issues.apache.org/jira/browse/SAMZA-506

On Tue, Jan 13, 2015 at 3:49 PM, Zach Cox <zc...@gmail.com> wrote:

> Hi Chris,
>
> Here's the main idea of the SIGTERM shutdown hook approach that we're
> trying out: https://gist.github.com/zcox/6ec8910bd3f18e36c1a2
>
> We use our own main class to run either the job or the container depending
> on args, and this main is the ENTRYPOINT of our Docker container, instead
> of using the standard run-job.sh and run-container.sh scripts. So this is
> an easy place to add the shutdown hook. Not sure how reusable that would be
> for others...
>
> By "not in a very clean way" I meant that the shutdown hook needs to block
> jvm shutdown until SamzaContainer has gone through its entire list of
> shutdown calls, and the only way I could find for the shutdown hook to know
> this is finished is the shutdownIsComplete flag, set true after
> SamzaContainer.safeMain() returns, which feels a bit jank, but it seems
> to work.
>
> Thanks,
> Zach
>
>
> On Tue, Jan 13, 2015 at 2:34 PM, Chris Riccomini <
> criccomini@linkedin.com.invalid> wrote:
>
>> Hey Zach,
>>
>> Honestly, we just tell YARN to kill everything off. As you said, this
>> means there is no clean shutdown, but we've lived with thus far.
>>
>> Some folks have also used a control message, as you've said, and had their
>> task call coordinator.shutdown(). Others have used a JMX switch to do the
>> same.
>>
>> I agree that catching a SIGTERM is a little cleaner. Can you open a JIRA
>> for this?
>>
>> Basically, all the solutions you've described are possible. Only the
>> SIGTERM trap hasn't been implemented.
>>
>> > I can sorta make this work, but not in a very clean way.
>>
>>
>> Curious: what's not clean?
>>
>> Cheers,
>> Chris
>>
>> On 1/13/15 11:21 AM, "Zach Cox" <zc...@gmail.com> wrote:
>>
>> >Hi - just wondering what other folks do when a Samza job needs to be
>> >shutdown purposefully, such as during a deployment. Seems like in this
>> >case
>> >you'd want all of the job's Samza containers to cleanly stop processing
>> >messages, commit offsets/changelog, etc instead of just SIGKILL the jvms
>> >which will result in out-of-date checkpoints, etc.
>> >
>> >I noticed TaskCoordinator has a shutdown method, which the StreamTasks in
>> >a
>> >container can use to signal a clean shutdown. Does everyone use this,
>> with
>> >some kind of shutdown message sent to the StreamTasks using e.g. jmx or a
>> >kafka topic, as in this thread?
>> >
>> http://www.mail-archive.com/dev%40samza.incubator.apache.org/msg01248.html
>> >
>> >We're actively developing Samza support for Mesos, so maybe this is
>> >specific to that world, but would it make sense for a jvm shutdown hook
>> in
>> >the container to catch a SIGTERM and somehow signal to the container that
>> >it should shutdown? I can sorta make this work, but not in a very clean
>> >way.
>> >
>> >Just curious what others are doing about graceful shutdown.
>> >
>> >Thanks,
>> >Zach
>>
>>
>

Re: Graceful shutdown

Posted by Zach Cox <zc...@gmail.com>.

Hi Chris,

Here's the main idea of the SIGTERM shutdown hook approach that we're
trying out: https://gist.github.com/zcox/6ec8910bd3f18e36c1a2

We use our own main class to run either the job or the container depending
on args, and this main is the ENTRYPOINT of our Docker container, instead
of using the standard run-job.sh and run-container.sh scripts. So this is
an easy place to add the shutdown hook. Not sure how reusable that would be
for others...

By "not in a very clean way" I meant that the shutdown hook needs to block
jvm shutdown until SamzaContainer has gone through its entire list of
shutdown calls, and the only way I could find for the shutdown hook to know
this is finished is the shutdownIsComplete flag, set true after
SamzaContainer.safeMain() returns, which feels a bit jank, but it seems to
work.

Thanks,
Zach


On Tue, Jan 13, 2015 at 2:34 PM, Chris Riccomini <
criccomini@linkedin.com.invalid> wrote:

> Hey Zach,
>
> Honestly, we just tell YARN to kill everything off. As you said, this
> means there is no clean shutdown, but we've lived with thus far.
>
> Some folks have also used a control message, as you've said, and had their
> task call coordinator.shutdown(). Others have used a JMX switch to do the
> same.
>
> I agree that catching a SIGTERM is a little cleaner. Can you open a JIRA
> for this?
>
> Basically, all the solutions you've described are possible. Only the
> SIGTERM trap hasn't been implemented.
>
> > I can sorta make this work, but not in a very clean way.
>
>
> Curious: what's not clean?
>
> Cheers,
> Chris
>
> On 1/13/15 11:21 AM, "Zach Cox" <zc...@gmail.com> wrote:
>
> >Hi - just wondering what other folks do when a Samza job needs to be
> >shutdown purposefully, such as during a deployment. Seems like in this
> >case
> >you'd want all of the job's Samza containers to cleanly stop processing
> >messages, commit offsets/changelog, etc instead of just SIGKILL the jvms
> >which will result in out-of-date checkpoints, etc.
> >
> >I noticed TaskCoordinator has a shutdown method, which the StreamTasks in
> >a
> >container can use to signal a clean shutdown. Does everyone use this, with
> >some kind of shutdown message sent to the StreamTasks using e.g. jmx or a
> >kafka topic, as in this thread?
> >
> http://www.mail-archive.com/dev%40samza.incubator.apache.org/msg01248.html
> >
> >We're actively developing Samza support for Mesos, so maybe this is
> >specific to that world, but would it make sense for a jvm shutdown hook in
> >the container to catch a SIGTERM and somehow signal to the container that
> >it should shutdown? I can sorta make this work, but not in a very clean
> >way.
> >
> >Just curious what others are doing about graceful shutdown.
> >
> >Thanks,
> >Zach
>
>

Re: Graceful shutdown

Posted by Chris Riccomini <cr...@linkedin.com.INVALID>.

Hey Zach,

Honestly, we just tell YARN to kill everything off. As you said, this
means there is no clean shutdown, but we've lived with thus far.

Some folks have also used a control message, as you've said, and had their
task call coordinator.shutdown(). Others have used a JMX switch to do the
same.

I agree that catching a SIGTERM is a little cleaner. Can you open a JIRA
for this?

Basically, all the solutions you've described are possible. Only the
SIGTERM trap hasn't been implemented.

> I can sorta make this work, but not in a very clean way.

Curious: what's not clean?

Cheers,
Chris

On 1/13/15 11:21 AM, "Zach Cox" <zc...@gmail.com> wrote:

>Hi - just wondering what other folks do when a Samza job needs to be
>shutdown purposefully, such as during a deployment. Seems like in this
>case
>you'd want all of the job's Samza containers to cleanly stop processing
>messages, commit offsets/changelog, etc instead of just SIGKILL the jvms
>which will result in out-of-date checkpoints, etc.
>
>I noticed TaskCoordinator has a shutdown method, which the StreamTasks in
>a
>container can use to signal a clean shutdown. Does everyone use this, with
>some kind of shutdown message sent to the StreamTasks using e.g. jmx or a
>kafka topic, as in this thread?
>http://www.mail-archive.com/dev%40samza.incubator.apache.org/msg01248.html
>
>We're actively developing Samza support for Mesos, so maybe this is
>specific to that world, but would it make sense for a jvm shutdown hook in
>the container to catch a SIGTERM and somehow signal to the container that
>it should shutdown? I can sorta make this work, but not in a very clean
>way.
>
>Just curious what others are doing about graceful shutdown.
>
>Thanks,
>Zach