You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/06/21 13:35:00 UTC
[jira] [Commented] (FLINK-9513) Wrap state binder with TTL logic
[ https://issues.apache.org/jira/browse/FLINK-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519369#comment-16519369 ]
ASF GitHub Bot commented on FLINK-9513:
---------------------------------------
GitHub user azagrebin opened a pull request:
https://github.com/apache/flink/pull/6196
[FLINK-9513] Implement TTL state wrappers factory and serializer for value with TTL
## What is the purpose of the change
This PR introduces a state factory for wrapping state objects with TTL logic and serialiser of user value with expiration timestamp.
NOTE: This PR is based on #6186 and only last commit makes difference with it and needs review.
## Brief change log
- abstract state creation in backends with `KeyedStateFactory` interface
- add `TtlStateFactory`
- add `CompositeSerializer`
## Verifying this change
This change is a trivial addition without any test coverage in this PR and should be covered together with TTL feature activation by final integration and e2e tests.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (yes)
- The runtime per-record code paths (performance sensitive): (not yet)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (not yet)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (not applicable at the moment)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/azagrebin/flink FLINK-9513
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6196.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6196
----
commit 62faa8ee220c21fa824fec690073c27a0a994be5
Author: Andrey Zagrebin <az...@...>
Date: 2018-06-04T15:28:40Z
[FLINK-9514,FLINK-9515,FLINK-9516] State ttl wrappers
commit 74c689e1660d40176b3c131fb0f3f9dcafa33889
Author: Andrey Zagrebin <az...@...>
Date: 2018-06-20T15:05:28Z
Check overflow in expiration timestamp, allow only non-negative TTL
commit 1164aa2a9c4298461eaa44322ef9cefa00b4f0fe
Author: Andrey Zagrebin <az...@...>
Date: 2018-06-21T12:24:04Z
small fixes
commit 1d19d4ac2b73ac83290b4b117b82895c99b51865
Author: Andrey Zagrebin <az...@...>
Date: 2018-06-21T13:13:42Z
Make AbstractTtlState.getSerializedValue() unsupported for now in case of queryable state
commit 99994dedb9a20244a2addd337617778b17fe8349
Author: Andrey Zagrebin <az...@...>
Date: 2018-06-11T17:34:47Z
[FLINK-9513] Implement TTL state wrappers factory and serializer for value with TTL
----
> Wrap state binder with TTL logic
> --------------------------------
>
> Key: FLINK-9513
> URL: https://issues.apache.org/jira/browse/FLINK-9513
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
> Affects Versions: 1.6.0
> Reporter: Andrey Zagrebin
> Assignee: Andrey Zagrebin
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.0
>
>
> The main idea is to wrap user state value with a class holding the value and the expiration timestamp (maybe meta data in future) and use the new object as a value in the existing implementations:
> {code:java}
> class TtlValue<V> {
> V value;
> long expirationTimestamp;
> }
> {code}
> The original state binder factory is wrapped with TtlStateBinder if TTL is enabled:
> {code:java}
> state = ttlConfig.updateType == DISABLED ?
> bind(binder) : bind(new TtlStateBinder(binder, timerService));
> {code}
> TtlStateBinder decorates the states produced by the original binder with TTL logic wrappers and adds TtlValue serialisation logic:
> {code:java}
> TtlStateBinder {
> StateBinder binder;
> ProcessingTimeProvier timeProvider; // System.currentTimeMillis()
> <V> TtlValueState<V> createValueState(valueDesc) {
> serializer = new TtlValueSerializer(valueDesc.getSerializer);
> ttlValueDesc = new ValueDesc(serializer, ...);
> // or implement custom TypeInfo
> originalStateWithTtl = binder.createValueState(valueDesc);
> return new TtlValueState(originalStateWithTtl, timeProvider);
> }
> // List, Map, ...
> }
> {code}
> TTL serializer should add expiration timestamp
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)