You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Annemarie Burger <an...@campus.tu-berlin.de> on 2020/05/12 18:46:35 UTC
Incremental state with purging
Hi,
I'm trying to implement the most efficient way to incrementally put incoming
DataStream elements in my (map)state, while removing old elements (older
that x) from that same state. I then want to output the state every y
seconds. I've looked into using the ProcessFunction with onTimer, or
building my own Trigger for a window function, but I struggle with putting
all this together in a logical and efficient way. Since the state is very
big I don't want to duplicate it over multiple (sliding)windows. Does
anybody know the best way to achieve this? Some pseudo code would be very
helpful.
Thanks!
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Incremental state with purging
Posted by Thomas Huang <ly...@hotmail.com>.
I’m wondering that why you use a beta feature for production. Why not push the latest state into down sink like redis or hbase with Apache phoenix .
________________________________
From: Annemarie Burger <an...@campus.tu-berlin.de>
Sent: Monday, May 18, 2020 11:19:23 PM
To: user@flink.apache.org <us...@flink.apache.org>
Subject: Re: Incremental state with purging
Hi,
Thanks for your suggestions!
However, as I'm reading the docs for queryable state, it says that it can
only be used for Processing time, and my windows are defined using event
time. So, I guess this means I should use the KeyedProcessFunction. Could
you maybe suggest a rough implementation for this? I can't seem to get the
implementation working right.
Best,
Annemarie
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Incremental state with purging
Posted by Annemarie Burger <an...@campus.tu-berlin.de>.
Hi,
Thanks for your suggestions!
However, as I'm reading the docs for queryable state, it says that it can
only be used for Processing time, and my windows are defined using event
time. So, I guess this means I should use the KeyedProcessFunction. Could
you maybe suggest a rough implementation for this? I can't seem to get the
implementation working right.
Best,
Annemarie
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Incremental state with purging
Posted by Congxian Qiu <qc...@gmail.com>.
Hi
From your description, you want to do two things:
1 update state and remote the state older than x
2 output the state every y second
From my side, the first can be done by using TTL state as Yun said,
the second can be done by using KeyedProcessFunction[1]
If you want to have complex logic to remove the older state in step 1,
maybe you can also use the KeyedProcessFunction and timer()
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/stream/operators/process_function.html#the-keyedprocessfunction
Best,
Congxian
Yun Tang <my...@live.com> 于2020年5月13日周三 下午7:42写道：
> Hi
>
> From your description: "output the state every y seconds and remove old
> elements", I think TTL [1] is the proper solution for your scenario. And
> you could define the ttl of your state as y seconds so that processfunction
> could only print elements in the last y seconds.
>
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#state-time-to-live-ttl
>
> Best
> Yun Tang
> ------------------------------
> *From:* Annemarie Burger <an...@campus.tu-berlin.de>
> *Sent:* Wednesday, May 13, 2020 2:46
> *To:* user@flink.apache.org <us...@flink.apache.org>
> *Subject:* Incremental state with purging
>
> Hi,
>
> I'm trying to implement the most efficient way to incrementally put
> incoming
> DataStream elements in my (map)state, while removing old elements (older
> that x) from that same state. I then want to output the state every y
> seconds. I've looked into using the ProcessFunction with onTimer, or
> building my own Trigger for a window function, but I struggle with putting
> all this together in a logical and efficient way. Since the state is very
> big I don't want to duplicate it over multiple (sliding)windows. Does
> anybody know the best way to achieve this? Some pseudo code would be very
> helpful.
>
> Thanks!
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>
Re: Incremental state with purging
Posted by Yun Tang <my...@live.com>.
Hi
From your description: "output the state every y seconds and remove old elements", I think TTL [1] is the proper solution for your scenario. And you could define the ttl of your state as y seconds so that processfunction could only print elements in the last y seconds.
[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#state-time-to-live-ttl
Best
Yun Tang
________________________________
From: Annemarie Burger <an...@campus.tu-berlin.de>
Sent: Wednesday, May 13, 2020 2:46
To: user@flink.apache.org <us...@flink.apache.org>
Subject: Incremental state with purging
Hi,
I'm trying to implement the most efficient way to incrementally put incoming
DataStream elements in my (map)state, while removing old elements (older
that x) from that same state. I then want to output the state every y
seconds. I've looked into using the ProcessFunction with onTimer, or
building my own Trigger for a window function, but I struggle with putting
all this together in a logical and efficient way. Since the state is very
big I don't want to duplicate it over multiple (sliding)windows. Does
anybody know the best way to achieve this? Some pseudo code would be very
helpful.
Thanks!
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/