You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by tzulitai <gi...@git.apache.org> on 2017/03/27 12:43:48 UTC
[GitHub] flink issue #3001: [FLINK-4821] [kinesis] Implement rescalable non-partition...
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/3001
Hi @tony810430!
Sorry for the long pause on this PR. After some back and forth offline discussions with others on how exactly we want to proceed with this, we decided to stick with using union state to cope with the shard discovery on restore problem (at least for 1.3.0). Therefore, we can finally continue work here :-D
First of all, to use union state, instead of `ListCheckpointed`, we should use `CheckpointedFunction` instead. There is a PR for exposing union state to the public API (#3508), but in case that isn't merged yet within the next few days, I suggest that you don't need to be blocked when you continue your work on this PR. For now, you can cast the operator state store instance retrieved through the `FunctionInitializationContext` to `DefaultOperatorStateStore` to use broadcast state.
One thing to also note, which is missing in you previous work on this, is that we need a migration path from the old state access (i.e., via `CheckpointedAsynchronously`) to the new state (i.e. `CheckpointedFunction`).
The `FlinkKafkaConsumerBase` class in the Kafka connector provides a very good example of how to do this. Simply put, in the end, the `FlinkKinesisConsumer` should implement both `CheckpointedRestoring` and `CheckpointedFunction`, and bridge the old state read from the legacy `restoreState(...)` method to the new `initializeState(...)` method. The bridge would simply be a field variable in the consumer class.
The `FlinkKafkaConsumerBase` also serves as a good example of how to use the `CheckpointedFunction` if you have questions there.
Let me know if you have any questions with this, and feel free to ping me any time!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---