streaming restored state after restart

[Adam Atrea <at...@gmail.com>]

Hi,

I'm new to Flink - but after reading the documentation -

What would be the best approach to stream data from a restored state
following a job restart ?
Say I have a MapState that gets populated during streaming with various
computed results within a stateful operator.

Upon job restart or on task failure recovery I am looking to stream the
data from that restored state downstream so that the operator will replay
it.
The data just to be directed to the downstream - at least once - so
multiple tasks may submit the same data.

Thanks,

Adam

Re: streaming restored state after restart

[Till Rohrmann <tr...@apache.org>]

Hi Adam,

you could have a local field in your operator which you initialize with
true on creation. Then in the map or process function you could check this
field and if true output all buffered state. At last you only need to set
this field to false so that any subsequent call to the map/process function
won't trigger the emission of the buffered elements in your state.

Cheers,
Till

On Thu, Jun 11, 2020 at 2:58 PM Adam Atrea <at...@gmail.com> wrote:

> Hi,
>
> I'm new to Flink - but after reading the documentation -
>
> What would be the best approach to stream data from a restored state
> following a job restart ?
> Say I have a MapState that gets populated during streaming with various
> computed results within a stateful operator.
>
> Upon job restart or on task failure recovery I am looking to stream the
> data from that restored state downstream so that the operator will replay
> it.
> The data just to be directed to the downstream - at least once - so
> multiple tasks may submit the same data.
>
> Thanks,
>
> Adam
>