[jira] [Commented] (FLINK-6649) Improve Non-window group aggregate with configurable `earlyFire`.

["Fabian Hueske (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16018621#comment-16018621 ] 

Fabian Hueske commented on FLINK-6649:
--------------------------------------

I agree, that we need to provide a mechanism to reduce the number of rows emitted by non-windowed aggregate.

However, I have two concerns about the proposed method:

1. IMO, the mechanism should not be called {{Early Fire}}. In my understanding, early firing refers to enabling the emission of incomplete results from a certain point in time. For example emitting early results for an hourly tumbling window after the first 15 minutes have passed. In this example, {{Early Fire = -45 Minutes}} would specify that the first results for the window may be emitted after 15 minutes (60 minutes - 45 minutes). By definition, a non-windowed aggregation is never complete. Whenever a new record for the group arrives it has to be added to the result. So non-windowed aggregates are always early firing, because the result never completes and we would otherwise never emit a result. Instead of {{Early Fire}}, I would call this mechanism {{Update Rate}} to specify how often a result is emitted.
2. The {{Update Rate}} may not be defined in terms of record count, but only in terms of time. Given an non-windowed grouped aggregate with {{Update Rate = 10 Rows}}, we would never emit a result for a group that only received 9 rows. By defining the update rate as a time interval, we can effectively reduce the number of outgoing records and ensure that updates are eventually propagated.

What do you think [~sunjincheng121]?

> Improve Non-window group aggregate with configurable `earlyFire`.
> -----------------------------------------------------------------
>
>                 Key: FLINK-6649
>                 URL: https://issues.apache.org/jira/browse/FLINK-6649
>             Project: Flink
>          Issue Type: Improvement
>          Components: Table API & SQL
>    Affects Versions: 1.4.0
>            Reporter: sunjincheng
>            Assignee: sunjincheng
>
> Currently,  Non-windowed group aggregate is earlyFiring at count(1), that is every row will emit a aggregate result. But some times user want config count number (`early firing with count[N]`) , to reduce the downstream pressure. This JIRA. will enable the config of e`earlyFiring` for  Non-windowed group aggregate.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)