You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-zh@flink.apache.org by "casel.chen" <ca...@126.com> on 2021/12/16 07:49:08 UTC
How to do windowing on flink retract stream?
业务需要按每分钟统计不同交易状态的交易数,接了业务mysql库binlog到flink计算。这是一个retract回撤流。是不能够直接使用Tumble window计算的。
1. 那么是不是只能用全量窗口的方式实现?即group by交易时间落到其时长为一分钟的窗口,再配合state TTL来过期不需要的状态?
2. 这样一来的话每来一笔交易都会更新状态,如果直接输出到下游mysql保存的话会对mysql造成很大写压力,那么是不是可以再接一个Tumble window获取每个指标的最新统计值输出?
3. 另外我在想是不是可以将retract流转变成append流,再在上面使用Tumble window计算?例如: 统计10秒种内各种color个数
[id, color, time]
+I (2, red, 00:05)
-U (2, red, 00:05))
+U (2, green, 00:07))
-D (2, green, 00:09))
select window_start, window_end, color, count(*)
from tbl, LATERAL TABLE(WINDOW_IN_SECONDS(ts, 10)) AS T(window_start, window_end)
group by color;
其中WINDOW_IN_SECONDS是一个UDTF
变成
[id, color, time, flag]
+I (2, red, 1, 00:05, 1)
+I (2, red, 1, 00:05, -1)
+I (2, green, 1, 00:07, 1)
+I (2, green, 1, 00:09, -1)
SELECT window_start, window_end, color, sum(flag)
FROM TABLE(TUMBLE(TABLE tbl, DESCRIPTOR(ts), INTERVAL '10' SECONDS))
GROUP BY window_start, window_end, color;