You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Gustavo de Morais (Jira)" <ji...@apache.org> on 2026/10/02 13:14:00 UTC
[jira] [Updated] (FLINK-40896) FLIP-614: Add built-in DEDUPLICATE_KEEP_FIRST PTF
[ https://issues.apache.org/jira/browse/FLINK-40896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gustavo de Morais updated FLINK-40896:
--------------------------------------
Summary: FLIP-614: Add built-in DEDUPLICATE_KEEP_FIRST PTF (was: Add built-in DEDUPLICATE_KEEP_FIRST PTF)
> FLIP-614: Add built-in DEDUPLICATE_KEEP_FIRST PTF
> -------------------------------------------------
>
> Key: FLINK-40896
> URL: https://issues.apache.org/jira/browse/FLINK-40896
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API, Table SQL / Runtime
> Reporter: Gustavo de Morais
> Assignee: Vas Shabu
> Priority: Major
>
> Deduplication is a common Flink SQL need. Today users must express "keep first" dedup via verbose {{ROW_NUMBER()}} windowing, which must match an exact pattern to be planner-recognized and buries the intent in a windowed rank.
> This ticket adds {{{}DEDUPLICATE_KEEP_FIRST{}}}, a built-in PTF with two modes:
> * {*}Watermarkless{*}: first record per key wins, no watermark required.
> * {*}Event-time{*}: with {{{}on_time{}}}, the smallest-event-time row per key wins, finalized once the watermark passes it.
>
> {code:java}
> DEDUPLICATE_KEEP_FIRST(
> input => TABLE(t) [PARTITION BY <cols>],
> on_time => DESCRIPTOR(<rowtime>),
> state_ttl => <interval>,
> reset_ttl_on_duplicate => <boolean>,
> uid => <string>
> ) {code}
> Event-time mode requires insert-only input (updating input + {{on_time}} rejected at planning time). Watermarkless mode accepts updating input, absorbing later changes into the kept row.
> See [FLIP-614|https://cwiki.apache.org/confluence/spaces/FLINK/pages/451975182/FLIP-614+Add+built-in+DEDUPLICATE_KEEP_FIRST+PTF] for full motivation, rejected alternatives, and compatibility plan.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)