You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Dian Fu (Jira)" <ji...@apache.org> on 2026/08/21 14:15:00 UTC

[jira] [Assigned] (FLINK-40194) Add drop_duplicates to DataFrame API

     [ https://issues.apache.org/jira/browse/FLINK-40194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dian Fu reassigned FLINK-40194:
-------------------------------

    Assignee: Timo Theusner

> Add drop_duplicates to DataFrame API
> ------------------------------------
>
>                 Key: FLINK-40194
>                 URL: https://issues.apache.org/jira/browse/FLINK-40194
>             Project: Flink
>          Issue Type: Sub-task
>          Components: API / Python
>            Reporter: Dian Fu
>            Assignee: Timo Theusner
>            Priority: Major
>             Fix For: 2.4.0
>
>
> def drop_duplicates(
>     self,
>     subset: Union[str, List[str]] = None,
>     *,
>     keep: str = "first",
>     order_by: Union[str, Expression, List[Union[str, Expression]]] = None,
>     nulls_first: Union[bool, List[bool]] = None,
> ) -> "DataFrame"
> Remove duplicate rows. When subset is omitted, fully identical rows are dropped; when subset is given, rows are deduplicated by those key columns, keeping one row per key. distinct and unique are provided as aliases of this method, matching the common naming convention used by other DataFrame libraries.
> * subset — a column name or list of column names that define a duplicate. When omitted (default), all columns are considered (whole-row distinct).
> * keep — "first" keeps the earliest row, "last" keeps the latest row within each duplicate group.
> * order_by — a column name or Expression (or a list of them) that defines the order in which keep selects the surviving row. When omitted, processing time is used, so keep="first"/"last" keep the first/last row to arrive. Ignored when subset is omitted.
> * nulls_first — a bool, or a list of bool aligned with order_by, controlling where NULLs rank when keep selects the surviving row. When omitted, Flink's default null ordering applies. Ignored when order_by is omitted.
> Example:
> {code:python}
> # Whole-row deduplication: drop fully identical rows
> df.drop_duplicates()
> # Keep the earliest row per device by arrival (defaults to processing time)
> df.drop_duplicates(subset="device_id")
> # Keep the earliest event per order, ordered by event time
> df.drop_duplicates(subset=["order_id"], order_by="event_time")
> # Keep the latest record per user, ordered by event time
> df.drop_duplicates(subset="user_id", order_by="event_time", keep="last")
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)