You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Hannan Kan <ha...@foxmail.com> on 2022/08/01 15:35:24 UTC
[DISCUSS] FLIP-255 Introduce pre-aggregated merge to Table Store
Hi everyone,
I would like to open a discussion on FLIP-255 Introduce pre-aggregated merge to table store [1].
Pre-aggregation mechanism has been adopted by many big data systems (such as Apache Doris,
Apache Kylin
, Druid etc.) to save storage and accelerate the aggregate query.
FLIP-255 proposes to introduce pre-aggregated merge into Flink Table Store to acquire the same benefit.
Supported aggregate functions include sum, max/min, count, replace_if_not_null, replace, concatenate, or/and.
Looking forward to your feedback.
[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-255+Introduce+pre-aggregated+merge+to+table+store
Best,
Nathan Kan (Hongnan Gan)
Re: [Phishing Risk] [External] [DISCUSS] FLIP-255 Introduce pre-aggregated merge to Table Store
Posted by Jingsong Li <ji...@gmail.com>.
Thanks Nathan for starting this discussion.
This [1] is a very good requirement to build the materialized view on Flink
Table Store.
## Aggregate Functions
For 'aggregate-function' = '{sum_field1:sum,max_field2:max}'.
Do you refer to any other systems? For us at Flink, a viable approach is
something like the Datagen connector [2]. Something like
`'fields.sum_field1.function'='sum'`.
## Default function
>> Tips: Columns which do not have designated aggregate functions using
newest value to overwrite old value.
Do you mean `replace`?
Is there anything about default functions that other systems can refer to?
## Supported functions
I'm not quite sure that the names of these functions are standard enough:
`replace_if_not_null/replace/concatenate`. Can you look at other systems?
You can also specify whether they support retraction messages.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-255+Introduce+pre-aggregated+merge+to+Table+Store
[2]
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/table/datagen/#connector-options
Best,
Jingsong
On Tue, Aug 2, 2022 at 11:09 AM 李国君 <li...@bytedance.com> wrote:
> Hi Nathan,
>
> Seems a great proposal for table store aggregation.
> In the example, I think the 'max_field1' should be 1 instead of 2 after
> the max aggregation in the output result.
> And there may be a minor typo in the WITH clause, 'max_field2' ->
> 'max_field1'.
>
> Best,
> Guojun
>
> From: "Hannan Kan"<ha...@foxmail.com>
> Date: Mon, Aug 1, 2022, 11:35 PM
> Subject: [Phishing Risk] [External] [DISCUSS] FLIP-255 Introduce
> pre-aggregated merge to Table Store
> To: "dev"<de...@flink.apache.org>
> Cc: "lzljs3620320"<lz...@apache.org>
> Hi everyone, I would like to open a discussion on FLIP-255 Introduce
> pre-aggregated merge to table store [1]. Pre-aggregation mechanism has
> been adopted by many big data systems (such as Apache Doris,
> Apache Kylin , Druid etc.) to save storage and accelerate the
> aggregate query. FLIP-255 proposes to introduce pre-aggregated merge into
> Flink Table Store to acquire the same benefit. Supported aggregate
> functions include sum, max/min, count, replace_if_not_null,
> replace, concatenate, or/and. Looking forward to your feedback.
> [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-255+Introduce+pre-aggregated+merge+to+table+store
> Best, Nathan Kan (Hongnan Gan)
>
Re: [Phishing Risk] [External] [DISCUSS] FLIP-255 Introduce pre-aggregated merge to Table Store
Posted by 李国君 <li...@bytedance.com>.
Hi Nathan,
Seems a great proposal for table store aggregation.
In the example, I think the 'max_field1' should be 1 instead of 2 after the
max aggregation in the output result.
And there may be a minor typo in the WITH clause, 'max_field2' ->
'max_field1'.
Best,
Guojun
From: "Hannan Kan"<ha...@foxmail.com>
Date: Mon, Aug 1, 2022, 11:35 PM
Subject: [Phishing Risk] [External] [DISCUSS] FLIP-255 Introduce
pre-aggregated merge to Table Store
To: "dev"<de...@flink.apache.org>
Cc: "lzljs3620320"<lz...@apache.org>
Hi everyone, I would like to open a discussion on FLIP-255 Introduce
pre-aggregated merge to table store [1]. Pre-aggregation mechanism has
been adopted by many big data systems (such as Apache Doris,
Apache Kylin , Druid etc.) to save storage and accelerate the
aggregate query. FLIP-255 proposes to introduce pre-aggregated merge into
Flink Table Store to acquire the same benefit. Supported aggregate
functions include sum, max/min, count, replace_if_not_null,
replace, concatenate, or/and. Looking forward to your feedback.
[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-255+Introduce+pre-aggregated+merge+to+table+store
Best, Nathan Kan (Hongnan Gan)