You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by forideal <fs...@163.com> on 2020/08/16 07:14:23 UTC

Re:回复：How to get the evaluation result of a time-based window aggregation in time after a new event falling into the window?

Hi Chengcheng Zhang,

You are welcome.
I also got the material from the community to answer your question.
There is also a hidden method here, as if the community has not been placed publicly in the document.
table.exec.emit.early-fire.enabled = true
table.exec.emit.early-fire.delay = 60 s
[1]http://apache-flink.147419.n8.nabble.com/FLINKSQL1-10-UV-td4003.html

Best, forideal

At 2020-08-16 13:21:25, "Chengcheng Zhang" <27...@qq.com> wrote:

Hi, forideal
Thank you so much, it does help a lot.
The approach you mentioned earlier, happened to be the same path we took two days ago, and it worked well as expected.
To be honest, after some effort-taking searches on the Internet, I' am a little convinced that, this maybe the best solution at the moment. However, the time-based window aggregation is a great feature in Flink, as we all know. Would it be perfect if we could use time-based windows and still get the latest result at the same time?
Best, Chengcheng Zhang

------------------ 原始邮件 ------------------
发件人: "forideal" <fs...@163.com>;
发送时间: 2020年8月16日(星期天) 中午12:24
收件人: "Chengcheng Zhang"<27...@qq.com>;
抄送: "user"<us...@flink.apache.org>;
主题: Re:How to get the evaluation result of a time-based window aggregation in time after a new event falling into the window?

Hi Chengcheng Zhang,

Is this your scene? For example, every day is divided into 12 hours, let’s take today as an example, 2020081600 2020081601,...2020081623
For example, if we count pv, we can count like this
INSERT INTO cumulative_pv
SELECT time_str, count(1)
FROM pv_per_hour
GROUP BY time_str;
In this sql, time_str is an hour in 2020081600, 2020081601,...2020081623.

[1]http://apache-flink.147419.n8.nabble.com/flink-sql-5-5-td2011.html
[2]http://wuchong.me/blog/2020/02/25/demo-building-real-time-application-with-flink-sql/

Hope this helps.

Best, forideal

At 2020-08-16 12:05:04, "Chengcheng Zhang" <27...@qq.com> wrote:

Hi,
I'm a new user of Flink, and have been puzzled a lot by the time-based window aggregation result.
For our business, hourly and daily reports have to been created best in a real time style. So, I used a event-time based window aggregation to consume the Kafka data stream, but found that, only after the current hour or day passed, the newest result could be seen on console or upserted to MySQL.
How can I get the latest window result immediately after a stream record falling into it? Is there a specific configuration option for this, hopefully? Please help and rescue me.
Best regards.

Re: 回复：How to get the evaluation result of a time-based window aggregation in time after a new event falling into the window?

Posted by Theo Diefenthal <th...@scoop-software.de>.

Hi Chengcheng Zhang,

I think your request is related to this feature request from two years ago here [1], with me asking about the status one year ago [2].
You might want to upvote this so we can hope that it gets some more attention in future.

Today, it is possible to write your own DataStream API where you customize the triggers to your wishes (CountTigger of 1), but with Flink SQL, you sadly lose most of that flexibility.

Thanks @ forideal for mentioning that there is something in the configuration. I wasn't aware of that.

[1] [ http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/SQL-Do-Not-Support-Custom-Trigger-td20932.html | http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/SQL-Do-Not-Support-Custom-Trigger-td20932.html ]
[2] [ http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Are-there-any-news-on-custom-trigger-support-for-SQL-Table-API-td29600.html | http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Are-there-any-news-on-custom-trigger-support-for-SQL-Table-API-td29600.html ]

Best regards
Theo

Von: "forideal" <fs...@163.com>
An: "Chengcheng Zhang" <27...@qq.com>
CC: "user" <us...@flink.apache.org>
Gesendet: Sonntag, 16. August 2020 09:14:23
Betreff: Re:回复：How to get the evaluation result of a time-based window aggregation in time after a new event falling into the window?

Hi Chengcheng Zhang,

You are welcome.
I also got the material from the community to answer your question.
There is also a hidden method here, as if the community has not been placed publicly in the document.
table.exec.emit.early-fire.enabled = true
table.exec.emit.early-fire.delay = 60 s
[1] [ http://apache-flink.147419.n8.nabble.com/FLINKSQL1-10-UV-td4003.html | http://apache-flink.147419.n8.nabble.com/FLINKSQL1-10-UV-td4003.html ]

Best, forideal

At 2020-08-16 13:21:25, "Chengcheng Zhang" <27...@qq.com> wrote:

Hi Chengcheng Zhang,
Is this your scene? For example, every day is divided into 12 hours, let’s take today as an example, 2020081600 2020081601,...2020081623
For example, if we count pv, we can count like this
INSERT INTO cumulative_pv
SELECT time_str, count(1)
FROM pv_per_hour
GROUP BY time_str;
In this sql, time_str is an hour in 2020081600, 2020081601,...2020081623.

[1] [ http://apache-flink.147419.n8.nabble.com/flink-sql-5-5-td2011.html | http://apache-flink.147419.n8.nabble.com/flink-sql-5-5-td2011.html ]
[2] [ http://wuchong.me/blog/2020/02/25/demo-building-real-time-application-with-flink-sql/ | http://wuchong.me/blog/2020/02/25/demo-building-real-time-application-with-flink-sql/ ]

Hope this helps.

Best, forideal

At 2020-08-16 12:05:04, "Chengcheng Zhang" <27...@qq.com> wrote:
BQ_BEGIN

BQ_END