Graceful shutdown of a Trident topology

[Vincent Rischmann <ze...@gmail.com>]

Hi,

I'm trying to find a reliable way to gracefully shutdown a Trident
topology, meaning when we want to stop it, it processes all remaining
tuples but the spout does not emit anymore tuples.

Right now I have done something like this:

- when we want to stop it, a flag is set

- keep track of pending tasks with a list of batchId

- each call to emitBatch in the spout will check if the topology is "paused"

- if paused, do nothing

- if not paused, emit a tuple by taking data from a queue

- each call to ack(batchId) will remove the corresponding batchId from the
pending tasks

- if no more pending tasks, and the stop flag is set, notify by sending a
event via Redis


First, is this a correct approach in your opinion ? Is it guaranteed that
ack() will be called for each batchId ?

Second, I'm curious to hear about how everyone does graceful stopping. It
seems to me it's something a lot of people would want yet I didn't find a
lot of information about it.

Vincent.

Re: Graceful shutdown of a Trident topology

[Nikhil Singh <ns...@yahoo.com>]

For the storm topology we did graceful shutdown by setting the wait time with -w flag. This causes the topology to be deactivated for the wait time before its actually shut down. This gives the topology a chance to finish processing what it's currently processing before shutting down the workers.
In case of Trident. . for some reason if the tuple was not processed then it will be reprocessed next time the topology comes up given the guaranteed exactly once processing of Trident (if implemented correctly).

Re: Graceful shutdown of a Trident topology

[Kavi Kumar <ka...@gmail.com>]

Your approach looks interesting and I personally never thought of graceful
shutdown of topology.
Here just one small point that sometimes you may get callback to fail
method also.
Callback may not be to ack method always.
Overall your approach looks little complicate but looks good and guarantees
the expected results.

On Mon, Dec 22, 2014 at 9:19 PM, Vincent Rischmann <ze...@gmail.com>
wrote:

> Hi,
>
> I'm trying to find a reliable way to gracefully shutdown a Trident
> topology, meaning when we want to stop it, it processes all remaining
> tuples but the spout does not emit anymore tuples.
>
> Right now I have done something like this:
>
> - when we want to stop it, a flag is set
>
> - keep track of pending tasks with a list of batchId
>
> - each call to emitBatch in the spout will check if the topology is
> "paused"
>
> - if paused, do nothing
>
> - if not paused, emit a tuple by taking data from a queue
>
> - each call to ack(batchId) will remove the corresponding batchId from the
> pending tasks
>
> - if no more pending tasks, and the stop flag is set, notify by sending a
> event via Redis
>
>
> First, is this a correct approach in your opinion ? Is it guaranteed that
> ack() will be called for each batchId ?
>
> Second, I'm curious to hear about how everyone does graceful stopping. It
> seems to me it's something a lot of people would want yet I didn't find a
> lot of information about it.
>
> Vincent.
>