You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "jiayuasu (via GitHub)" <gi...@apache.org> on 2026/06/15 05:53:23 UTC
[PR] feat(python/sedonadb): add DataFrame.distinct and distinct_on [sedona-db]
jiayuasu opened a new pull request, #961:
URL: https://github.com/apache/sedona-db/pull/961
Adds two relational de-duplication methods to the Python `DataFrame`, matching the Ibis/DuckDB relational surface the `DataFrame` already exposes ("derived primarily from Ibis and DuckDB's relational APIs").
## API
```python
df.distinct() # drop rows equal across all columns (SELECT DISTINCT)
df.distinct_on("k") # one row per distinct key, all columns kept (DISTINCT ON)
df.distinct_on("k1", "k2")
df.distinct_on(col("x") > 2) # computed key
```
- `distinct()` — no args; de-dupes on all columns.
- `distinct_on(*cols)` — `cols` are column-name strings or `Expr`s. Keeps all columns; with no ordering applied, which row survives per key is arbitrary (same as SQL `DISTINCT ON` without `ORDER BY`) — documented. A sort-to-pick option can follow later if wanted.
Naming note: `distinct` / `distinct_on` is the SQL / Ibis / DuckDB / Spark spelling (pandas uses `drop_duplicates`, Polars uses `unique`). That keeps it consistent with the engine-neutral relational framing of the core `DataFrame`; pandas/Polars spellings would be aliases in a separate compatibility layer rather than on the core type.
## Implementation
| File | Change |
|---|---|
| `python/sedonadb/src/dataframe.rs` | `InternalDataFrame::distinct()` and `distinct_on(on_exprs)`. Thin wrappers over DataFusion's `DataFrame::distinct` / `distinct_on`. For `distinct_on`, the "keep all columns" select list is built from the full qualified schema (mirrors `qualified_column_expr`); `sort_expr` is `None`. |
| `python/sedonadb/python/sedonadb/dataframe.py` | `DataFrame.distinct()` and `DataFrame.distinct_on(*cols)` with str/`Expr` coercion and validation. |
## Test plan
11 tests in `tests/expr/test_dataframe_distinct.py`:
- `distinct()`: all-column dedup; identity on already-unique input; lazy return.
- `distinct_on()`: single key (deterministic when companion column is constant per key), multi-key, `Expr` key, computed-`Expr` key, lazy return.
- Non-deterministic survivor case asserts only the distinct-key set (no `ORDER BY`).
- Errors: empty `distinct_on()`; bad argument type.
Local: 11 unit + 27 doctests + full expr suite + `ruff` + `cargo fmt --check` + `clippy -Dwarnings` all clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python/sedonadb): add DataFrame.distinct and distinct_on [sedona-db]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu merged PR #961:
URL: https://github.com/apache/sedona-db/pull/961
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python/sedonadb): add DataFrame.distinct and distinct_on [sedona-db]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #961:
URL: https://github.com/apache/sedona-db/pull/961#discussion_r3411376122
##########
python/sedonadb/python/sedonadb/dataframe.py:
##########
@@ -834,6 +834,75 @@ def cross_join(self, other: "DataFrame") -> "DataFrame":
)
return DataFrame(self._ctx, self._impl.cross_join(other._impl))
+ def distinct(self) -> "DataFrame":
+ """Remove duplicate rows, comparing all columns.
+
+ Returns a new lazy DataFrame keeping one row from each set of
+ rows that are equal across every column (SQL `SELECT DISTINCT`).
+
+ Examples:
+
+ >>> sd = sedona.db.connect()
+ >>> df = sd.sql(
+ ... "SELECT * FROM (VALUES (1), (1), (2)) AS t(x)"
+ ... )
+ >>> df.distinct().sort("x").show()
+ ┌───────┐
+ │ x │
+ │ int64 │
+ ╞═══════╡
+ │ 1 │
+ ├╌╌╌╌╌╌╌┤
+ │ 2 │
+ └───────┘
+ """
+ return DataFrame(self._ctx, self._impl.distinct())
+
+ def distinct_on(self, *cols: Union[str, Expr]) -> "DataFrame":
+ """Remove duplicate rows, comparing only the given key columns.
+
+ Keeps one row for each distinct combination of `cols` (SQL
+ `DISTINCT ON`) and returns all columns. With no ordering applied,
+ which row is kept for each key is arbitrary; sort beforehand (or
+ restrict the columns) if you need a deterministic choice.
Review Comment:
Good catch — fixed in 28a3d04. The docstring no longer implies a prior `.sort()` controls the survivor; it now states the surviving row is unspecified and not controllable today (no ordering passed to DataFusion, and an upstream sort isn't guaranteed to be respected), and that a deterministic result requires the non-key columns to be constant per key. If we want controllable selection later, the clean way is to expose the `sort_expr` argument of DataFusion's `distinct_on` rather than relying on input order.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python/sedonadb): add DataFrame.distinct and distinct_on [sedona-db]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #961:
URL: https://github.com/apache/sedona-db/pull/961#discussion_r3411325540
##########
python/sedonadb/python/sedonadb/dataframe.py:
##########
@@ -834,6 +834,75 @@ def cross_join(self, other: "DataFrame") -> "DataFrame":
)
return DataFrame(self._ctx, self._impl.cross_join(other._impl))
+ def distinct(self) -> "DataFrame":
+ """Remove duplicate rows, comparing all columns.
+
+ Returns a new lazy DataFrame keeping one row from each set of
+ rows that are equal across every column (SQL `SELECT DISTINCT`).
+
+ Examples:
+
+ >>> sd = sedona.db.connect()
+ >>> df = sd.sql(
+ ... "SELECT * FROM (VALUES (1), (1), (2)) AS t(x)"
+ ... )
+ >>> df.distinct().sort("x").show()
+ ┌───────┐
+ │ x │
+ │ int64 │
+ ╞═══════╡
+ │ 1 │
+ ├╌╌╌╌╌╌╌┤
+ │ 2 │
+ └───────┘
+ """
+ return DataFrame(self._ctx, self._impl.distinct())
+
+ def distinct_on(self, *cols: Union[str, Expr]) -> "DataFrame":
+ """Remove duplicate rows, comparing only the given key columns.
+
+ Keeps one row for each distinct combination of `cols` (SQL
+ `DISTINCT ON`) and returns all columns. With no ordering applied,
+ which row is kept for each key is arbitrary; sort beforehand (or
+ restrict the columns) if you need a deterministic choice.
Review Comment:
The docstring suggests sorting before `distinct_on()` to get a deterministic survivor row, but the current implementation passes `sort_expr=None` down to DataFusion, so the chosen row is explicitly arbitrary and not guaranteed to be affected by a prior `.sort(...)` (the optimizer/physical plan may reorder). This documentation should avoid implying deterministic selection is possible today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org