You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Jark Wu (Jira)" <ji...@apache.org> on 2020/10/30 10:26:00 UTC
[jira] [Comment Edited] (FLINK-19896) Support first-n-rows
deduplication in the Deduplicate operator
[ https://issues.apache.org/jira/browse/FLINK-19896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17223557#comment-17223557 ]
Jark Wu edited comment on FLINK-19896 at 10/30/20, 10:25 AM:
-------------------------------------------------------------
I'm not sure about this. If we support first-n-rows for `DeduplicateKeepFirstRowFunction`, then it breaks the semantic of "Deduplicate".
Besides, if we want to support first-n-rows for `DeduplicateKeepFirstRowFunction`, the `DeduplicateKeepFirstRowFunction` will end up to be another top-n (Rank) operator implementation.
I think we should focus on how to improve the performance of Rank operator in this case.
was (Author: jark):
I'm not sure about this. If we support first-n-rows for `DeduplicateKeepFirstRowFunction`, then it breaks the semantic of "Deduplicate".
Besides, if we want to support first-n-rows for `DeduplicateKeepFirstRowFunction`, the `DeduplicateKeepFirstRowFunction` will end up to be another top-n (Rank) operator implementation.
I think we should focus how to improve the performance of Rank operator in this case.
> Support first-n-rows deduplication in the Deduplicate operator
> --------------------------------------------------------------
>
> Key: FLINK-19896
> URL: https://issues.apache.org/jira/browse/FLINK-19896
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner, Table SQL / Runtime
> Affects Versions: 1.12.0, 1.11.3
> Reporter: Jun Zhang
> Priority: Major
> Fix For: 1.11.2
>
>
> Currently Deduplicate operator only supports first-row deduplication (ordered by proc-time). In scenario of first-n-rows deduplication, the planner has to resort to Rank operator. However, Rank operator is less efficient than Deduplicate due to larger state and more state access.
> This issue proposes to extend DeduplicateKeepFirstRowFunction to support first-n-rows deduplication. And the original first-row deduplication would be a special case of first-n-rows deduplication.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)