replay kinesis events

[Guoqin Zheng <la...@gmail.com>]

Hi Flink experts,

Wondering if there is a built-in way to replay already-processed events in
an event queue. For example, if I have a flink app processing event stream
from Kinesis. Now if I find a bug in the flink app and make a fix. And I
would like to re-process events that are already processed in the kinesis
stream. Is there a simple mechanism to allow me to do this?

Thanks,
-Guoqin

Re: replay kinesis events

[Guoqin Zheng <la...@gmail.com>]

Hi Danny,

Thanks for getting back to me. This is very helpful and makes a lot of
sense to me.

Thanks,
-Guoqin


On Wed, Mar 9, 2022 at 1:32 AM Danny Cranmer <da...@apache.org>
wrote:

> Hey Guoqin,
>
> In order to achieve this you would need to either:
> - Restart the job and resume from an old savepoint (taken before the
> events you want to replay), assuming the state is still compatible with
> your bugfix, or
> - Restart the job without any state and seed the consumer with the start
> position [1]. You can use AT_TIMESTAMP and specify a suitable time in the
> past
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/datastream/kinesis/#configuring-starting-position
>
>
> Hope this helps,
> Thanks
>
>
> On Wed, Mar 9, 2022 at 12:15 AM Guoqin Zheng <la...@gmail.com>
> wrote:
>
>> Hi Flink experts,
>>
>> Wondering if there is a built-in way to replay already-processed events
>> in an event queue. For example, if I have a flink app processing event
>> stream from Kinesis. Now if I find a bug in the flink app and make a fix.
>> And I would like to re-process events that are already processed in the
>> kinesis stream. Is there a simple mechanism to allow me to do this?
>>
>> Thanks,
>> -Guoqin
>>
>

Re: replay kinesis events

[Danny Cranmer <da...@apache.org>]

Hey Guoqin,

In order to achieve this you would need to either:
- Restart the job and resume from an old savepoint (taken before the events
you want to replay), assuming the state is still compatible with your
bugfix, or
- Restart the job without any state and seed the consumer with the start
position [1]. You can use AT_TIMESTAMP and specify a suitable time in the
past

[1]
https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/datastream/kinesis/#configuring-starting-position


Hope this helps,
Thanks


On Wed, Mar 9, 2022 at 12:15 AM Guoqin Zheng <la...@gmail.com> wrote:

> Hi Flink experts,
>
> Wondering if there is a built-in way to replay already-processed events in
> an event queue. For example, if I have a flink app processing event stream
> from Kinesis. Now if I find a bug in the flink app and make a fix. And I
> would like to re-process events that are already processed in the kinesis
> stream. Is there a simple mechanism to allow me to do this?
>
> Thanks,
> -Guoqin
>