Checkpointing in StateFun

[Christopher Gustafson <ch...@kth.se>]

Hi,


I am doing some experiments with StateFun and different Flink state backends. I was looking through the StateFun source code and saw that it is doing some own logic when it comes to checkpointing. My initial understanding was that checkpointing was implicitly leveraged with Flink. I did not find a lot of documentation about this, so I am asking if you could provide a description of how checkpointing works in StateFun, and how it is different from regular Flink checkpointing?


Thanks in advance,

Christopher Gustafson

Re: Checkpointing in StateFun

[Seth Wiesman <sj...@gmail.com>]

There's a passing reference to it in the 1.2 docs but we removed it to
discourage use.

https://nightlies.apache.org/flink/flink-docs-release-1.2/dev/stream/state.html#raw-and-managed-state

On Sat, Mar 12, 2022 at 2:50 AM Christopher Gustafson <ch...@kth.se> wrote:

> Hi,
>
>
> Thanks for the explanation, I had not heard about the Raw State before,
> and it sounds like that is not surprising if it is rarely used, and doesn't
> seem to be included in the latest versions of the documentation. Is there
> perhaps some older version of the documentation you could point me to where
> I can see how Raw state can be used in Flink?
>
>
> Best Regards,
>
> Christopher Gustafson
> ------------------------------
> *Från:* Seth Wiesman <sj...@gmail.com>
> *Skickat:* den 11 mars 2022 17:57:21
> *Till:* Christopher Gustafson
> *Kopia:* user@flink.apache.org
> *Ämne:* Re: Checkpointing in StateFun
>
> I assume you are talking about the checkpointing in the feedback package?
>
> StateFun only relies on Flink checkpointing for fault tolerance. All state
> is stored in standard checkpoint / savepoints and can be used to restore
> from failure, upgrade a job, rescale, etc. Just like any other snapshot.
>
> StateFun internally relies on a form of streaming iteration. The feedback
> package contains the code for that iteration which sends records back
> upstream. To ensure exactly-once semantics, in-flight records on the
> feedback edge need to be stored in the snapshot. It does so through Flinks
> raw keyed state abstraction. This is the same thing state backends use
> under the hood to snapshot out your keyed state in any rich function.
>
> Raw keyed state has been a stable part of Flinks API forever, but is
> rarely seen in user code. I often describe the statefun runtime to people
> as the world's most complex datastream application; you will find advanced
> usage patterns most people have not seen, but nothing that isn't well
> supported by Flink.
>
> Seth
>
> On Fri, Mar 11, 2022 at 2:24 AM Christopher Gustafson <ch...@kth.se>
> wrote:
>
>> Hi,
>>
>>
>> I am doing some experiments with StateFun and different Flink state
>> backends. I was looking through the StateFun source code and saw that it is
>> doing some own logic when it comes to checkpointing. My initial
>> understanding was that checkpointing was implicitly leveraged with Flink. I
>> did not find a lot of documentation about this, so I am asking if you could
>> provide a description of how checkpointing works in StateFun, and how it is
>> different from regular Flink checkpointing?
>>
>>
>> Thanks in advance,
>>
>> Christopher Gustafson
>>
>

SV: Checkpointing in StateFun

[Christopher Gustafson <ch...@kth.se>]

Hi,


Thanks for the explanation, I had not heard about the Raw State before, and it sounds like that is not surprising if it is rarely used, and doesn't seem to be included in the latest versions of the documentation. Is there perhaps some older version of the documentation you could point me to where I can see how Raw state can be used in Flink?


Best Regards,

Christopher Gustafson

________________________________
Från: Seth Wiesman <sj...@gmail.com>
Skickat: den 11 mars 2022 17:57:21
Till: Christopher Gustafson
Kopia: user@flink.apache.org
Ämne: Re: Checkpointing in StateFun

I assume you are talking about the checkpointing in the feedback package?

StateFun only relies on Flink checkpointing for fault tolerance. All state is stored in standard checkpoint / savepoints and can be used to restore from failure, upgrade a job, rescale, etc. Just like any other snapshot.

StateFun internally relies on a form of streaming iteration. The feedback package contains the code for that iteration which sends records back upstream. To ensure exactly-once semantics, in-flight records on the feedback edge need to be stored in the snapshot. It does so through Flinks raw keyed state abstraction. This is the same thing state backends use under the hood to snapshot out your keyed state in any rich function.

Raw keyed state has been a stable part of Flinks API forever, but is rarely seen in user code. I often describe the statefun runtime to people as the world's most complex datastream application; you will find advanced usage patterns most people have not seen, but nothing that isn't well supported by Flink.

Seth

On Fri, Mar 11, 2022 at 2:24 AM Christopher Gustafson <ch...@kth.se>> wrote:

Hi,


I am doing some experiments with StateFun and different Flink state backends. I was looking through the StateFun source code and saw that it is doing some own logic when it comes to checkpointing. My initial understanding was that checkpointing was implicitly leveraged with Flink. I did not find a lot of documentation about this, so I am asking if you could provide a description of how checkpointing works in StateFun, and how it is different from regular Flink checkpointing?


Thanks in advance,

Christopher Gustafson

Re: Checkpointing in StateFun

[Seth Wiesman <sj...@gmail.com>]

I assume you are talking about the checkpointing in the feedback package?

StateFun only relies on Flink checkpointing for fault tolerance. All state
is stored in standard checkpoint / savepoints and can be used to restore
from failure, upgrade a job, rescale, etc. Just like any other snapshot.

StateFun internally relies on a form of streaming iteration. The feedback
package contains the code for that iteration which sends records back
upstream. To ensure exactly-once semantics, in-flight records on the
feedback edge need to be stored in the snapshot. It does so through Flinks
raw keyed state abstraction. This is the same thing state backends use
under the hood to snapshot out your keyed state in any rich function.

Raw keyed state has been a stable part of Flinks API forever, but is rarely
seen in user code. I often describe the statefun runtime to people as the
world's most complex datastream application; you will find advanced usage
patterns most people have not seen, but nothing that isn't well supported
by Flink.

Seth

On Fri, Mar 11, 2022 at 2:24 AM Christopher Gustafson <ch...@kth.se> wrote:

> Hi,
>
>
> I am doing some experiments with StateFun and different Flink state
> backends. I was looking through the StateFun source code and saw that it is
> doing some own logic when it comes to checkpointing. My initial
> understanding was that checkpointing was implicitly leveraged with Flink. I
> did not find a lot of documentation about this, so I am asking if you could
> provide a description of how checkpointing works in StateFun, and how it is
> different from regular Flink checkpointing?
>
>
> Thanks in advance,
>
> Christopher Gustafson
>