You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Timothy Bess <td...@gmail.com> on 2021/02/24 14:59:47 UTC
Flink Statefun TTL
Hey,
I noticed that the Flink Statefun 2.1.0 release notes had this snippet with
regards to TTL:
Note: The state expiration mode for remote functions is currently
> restricted to AFTER_READ_AND_WRITE, and the actual TTL being set is the
> longest duration across all registered state, not for each individual state
> entry. This is planned to be improved in upcoming releases (FLINK-17954).
>
I noticed that the Ticket and PR for this have been closed with a reference
to commit "289c30e8cdb54d2504ee47a57858a1d179f9a540". Does this mean that
if I upgrade to 2.2.2 and set an expiration in my modules.yaml it is now
"per function id" rather than across instances of said function?
Thanks,
Tim
Re: Flink Statefun TTL
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
On Thu, Feb 25, 2021 at 12:06 PM Timothy Bess <td...@gmail.com> wrote:
> Hi Gordon,
>
> Ah so when it said "all registered state" that means all state keys
> defined in the "module.yaml", not all state for all function instances. So
> the expiration has always been _per_ instance then and not across all
> instances of a function.
>
Exactly! Expiration happens individually for each function instance per
declared state.
>
> Thanks for the heads up, that sounds like a good change! I definitely like
> the idea of putting more configuration into the SDK so that there's not two
> sources that have to be kept up to date. Would be neat if eventually the
> SDK just hosts some "/spec" endpoint that serves a list of functions and
> all their configuration options to Statefun on boot.
>
> Btw, I ended up also making a Scala replica of my Haskell library to use
> at work (some of my examples in the microsite are a bit out of date, need
> to revisit that):
> https://github.com/BlueChipFinancial/flink-statefun4s
>
> I know it seems weird to not use an embedded function, but it keeps us
> from having to deal with mismatched Scala versions since Flink is still on
> 2.12, and generally reduces friction using stuff in the Scala Cats
> ecosystem.
>
Really cool to hear about your efforts on a Scala SDK!
I would not say it is weird to implement a Scala SDK for remote functions.
In fact, with the changes upcoming in 3.0, the community is doubling down
on remote as the primary deployment mode for functions, and would like to
have a wider array of supported language SDKs. There's actually a remote
Java SDK that was just merged to master and to be released in 3.0 [1].
Cheers,
Gordon
[1] https://github.com/apache/flink-statefun/tree/master/statefun-sdk-java
> Thanks,
>
> Tim
>
> On Wed, Feb 24, 2021 at 11:49 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
> wrote:
>
>> Hi Timothy,
>>
>> Starting from StateFun 2.2.x, in the module.yaml file, you can set for
>> each individual state of a function an "expireMode" field, which values can
>> be either "after-invoke" or "after-write". For example:
>>
>> ```
>> - function:
>> meta:
>> ...
>> spec:
>> states:
>> - name: state-1
>> expireMode: after-write
>> expireAfter: 1min
>> - name: state-2
>> expireMode: after-invoke
>> expireAfter: 5sec
>> ```
>>
>> In earlier versions, expireMode can not be individually set for each
>> state. This is more flexible with 2.2.x.
>>
>> As a side note which is somewhat related, all state related
>> configurations will be removed from the module.yaml, instead to be defined
>> by the language SDKs starting from StateFun 3.0.
>> This opens up even more flexibility, such as zero-downtime upgrades of
>> remote functions which allows adding / removing state declarations without
>> restarting the StateFun cluster.
>> We're planning to reach out to the language SDK developers we know of
>> (which includes you for the Haskell SDK ;) ) soon on a briefing of this
>> change, as there is a change in the remote invocation protocol and will
>> require existing SDKs to be updated in order to work with StateFun 3.0.
>>
>> Cheers,
>> Gordon
>>
>> On Wed, Feb 24, 2021 at 11:00 PM Timothy Bess <td...@gmail.com> wrote:
>>
>>> Hey,
>>>
>>> I noticed that the Flink Statefun 2.1.0 release notes had this snippet
>>> with regards to TTL:
>>>
>>> Note: The state expiration mode for remote functions is currently
>>>> restricted to AFTER_READ_AND_WRITE, and the actual TTL being set is the
>>>> longest duration across all registered state, not for each individual state
>>>> entry. This is planned to be improved in upcoming releases (FLINK-17954).
>>>>
>>>
>>> I noticed that the Ticket and PR for this have been closed with a
>>> reference to commit "289c30e8cdb54d2504ee47a57858a1d179f9a540". Does this
>>> mean that if I upgrade to 2.2.2 and set an expiration in my modules.yaml it
>>> is now "per function id" rather than across instances of said function?
>>>
>>> Thanks,
>>>
>>> Tim
>>>
>>
Re: Flink Statefun TTL
Posted by Timothy Bess <td...@gmail.com>.
Hi Gordon,
Ah so when it said "all registered state" that means all state keys defined
in the "module.yaml", not all state for all function instances. So the
expiration has always been _per_ instance then and not across all instances
of a function.
Thanks for the heads up, that sounds like a good change! I definitely like
the idea of putting more configuration into the SDK so that there's not two
sources that have to be kept up to date. Would be neat if eventually the
SDK just hosts some "/spec" endpoint that serves a list of functions and
all their configuration options to Statefun on boot.
Btw, I ended up also making a Scala replica of my Haskell library to use at
work (some of my examples in the microsite are a bit out of date, need to
revisit that):
https://github.com/BlueChipFinancial/flink-statefun4s
I know it seems weird to not use an embedded function, but it keeps us from
having to deal with mismatched Scala versions since Flink is still on 2.12,
and generally reduces friction using stuff in the Scala Cats ecosystem.
Thanks,
Tim
On Wed, Feb 24, 2021 at 11:49 AM Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:
> Hi Timothy,
>
> Starting from StateFun 2.2.x, in the module.yaml file, you can set for
> each individual state of a function an "expireMode" field, which values can
> be either "after-invoke" or "after-write". For example:
>
> ```
> - function:
> meta:
> ...
> spec:
> states:
> - name: state-1
> expireMode: after-write
> expireAfter: 1min
> - name: state-2
> expireMode: after-invoke
> expireAfter: 5sec
> ```
>
> In earlier versions, expireMode can not be individually set for each
> state. This is more flexible with 2.2.x.
>
> As a side note which is somewhat related, all state related configurations
> will be removed from the module.yaml, instead to be defined by the language
> SDKs starting from StateFun 3.0.
> This opens up even more flexibility, such as zero-downtime upgrades of
> remote functions which allows adding / removing state declarations without
> restarting the StateFun cluster.
> We're planning to reach out to the language SDK developers we know of
> (which includes you for the Haskell SDK ;) ) soon on a briefing of this
> change, as there is a change in the remote invocation protocol and will
> require existing SDKs to be updated in order to work with StateFun 3.0.
>
> Cheers,
> Gordon
>
> On Wed, Feb 24, 2021 at 11:00 PM Timothy Bess <td...@gmail.com> wrote:
>
>> Hey,
>>
>> I noticed that the Flink Statefun 2.1.0 release notes had this snippet
>> with regards to TTL:
>>
>> Note: The state expiration mode for remote functions is currently
>>> restricted to AFTER_READ_AND_WRITE, and the actual TTL being set is the
>>> longest duration across all registered state, not for each individual state
>>> entry. This is planned to be improved in upcoming releases (FLINK-17954).
>>>
>>
>> I noticed that the Ticket and PR for this have been closed with a
>> reference to commit "289c30e8cdb54d2504ee47a57858a1d179f9a540". Does this
>> mean that if I upgrade to 2.2.2 and set an expiration in my modules.yaml it
>> is now "per function id" rather than across instances of said function?
>>
>> Thanks,
>>
>> Tim
>>
>
Re: Flink Statefun TTL
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi Timothy,
Starting from StateFun 2.2.x, in the module.yaml file, you can set for each
individual state of a function an "expireMode" field, which values can be
either "after-invoke" or "after-write". For example:
```
- function:
meta:
...
spec:
states:
- name: state-1
expireMode: after-write
expireAfter: 1min
- name: state-2
expireMode: after-invoke
expireAfter: 5sec
```
In earlier versions, expireMode can not be individually set for each state.
This is more flexible with 2.2.x.
As a side note which is somewhat related, all state related configurations
will be removed from the module.yaml, instead to be defined by the language
SDKs starting from StateFun 3.0.
This opens up even more flexibility, such as zero-downtime upgrades of
remote functions which allows adding / removing state declarations without
restarting the StateFun cluster.
We're planning to reach out to the language SDK developers we know of
(which includes you for the Haskell SDK ;) ) soon on a briefing of this
change, as there is a change in the remote invocation protocol and will
require existing SDKs to be updated in order to work with StateFun 3.0.
Cheers,
Gordon
On Wed, Feb 24, 2021 at 11:00 PM Timothy Bess <td...@gmail.com> wrote:
> Hey,
>
> I noticed that the Flink Statefun 2.1.0 release notes had this snippet
> with regards to TTL:
>
> Note: The state expiration mode for remote functions is currently
>> restricted to AFTER_READ_AND_WRITE, and the actual TTL being set is the
>> longest duration across all registered state, not for each individual state
>> entry. This is planned to be improved in upcoming releases (FLINK-17954).
>>
>
> I noticed that the Ticket and PR for this have been closed with a
> reference to commit "289c30e8cdb54d2504ee47a57858a1d179f9a540". Does this
> mean that if I upgrade to 2.2.2 and set an expiration in my modules.yaml it
> is now "per function id" rather than across instances of said function?
>
> Thanks,
>
> Tim
>