You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Shengnan YU <sy...@connect.ust.hk> on 2017/11/14 04:53:24 UTC
The right approach to store intermediate value
Dear Flink committers:
I am starting to learn implementing some data streaming algorithm based on Flink API. Now I am working on MisraGries algorithm. It keeps k counters to record the data summary by increment or decrement. I want to know what is the best approach to store those counters(a hashmap data structure maybe) when using DataStream API to implement the algorithm. Now I just declared a map variable in the operator, is it right or I need to use some other features like state. This may be a very basic question forgive me since I am just a beginnerT^T.
Best regards
------------------------------------------------------
Shengnan YU(於圣楠）
syuam@connect.ust.hk<ma...@connect.ust.hk>
+852 69963058
Re: The right approach to store intermediate value
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
You should use the State API [1] for this if you want your program to be fault-tolerant. Flink will checkpoint state and make sure that it is restored in case of failures.
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/state.html <https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/state.html>
Best,
Aljoscha
> On 14. Nov 2017, at 05:53, Shengnan YU <sy...@connect.ust.hk> wrote:
>
> Dear Flink committers:
> I am starting to learn implementing some data streaming algorithm based on Flink API. Now I am working on MisraGries algorithm. It keeps k counters to record the data summary by increment or decrement. I want to know what is the best approach to store those counters(a hashmap data structure maybe) when using DataStream API to implement the algorithm. Now I just declared a map variable in the operator, is it right or I need to use some other features like state. This may be a very basic question forgive me since I am just a beginnerT^T.
>
> Best regards
> ------------------------------------------------------
> Shengnan YU(於圣楠）
>
> syuam@connect.ust.hk<ma...@connect.ust.hk>
> +852 69963058
>