You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ur...@apache.org on 2026/09/01 10:54:28 UTC
(spark) branch branch-4.x updated: [SPARK-47397][SQL][DOC] Document that COUNT(DISTINCT) skips nulls
This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 7a147724aa0f [SPARK-47397][SQL][DOC] Document that COUNT(DISTINCT) skips nulls
7a147724aa0f is described below
commit 7a147724aa0f0d981186fa5c4a5c7afd1cf5990b
Author: deepinsight coder <nr...@gmail.com>
AuthorDate: Tue Sep 1 10:54:07 2026 +0000
[SPARK-47397][SQL][DOC] Document that COUNT(DISTINCT) skips nulls
### What changes were proposed in this pull request?
Spell out that `COUNT(DISTINCT)` / `count_distinct` skips nulls, in `docs/sql-ref-null-semantics.md`.
- Add a `count(DISTINCT age)` example next to the existing `count(*)` / `count(age)` block. On the `person` table the result is 3 (ages 30, 18, 50).
- Add a one-liner in the GROUP BY / DISTINCT section: `COUNT(DISTINCT expr)` still follows the aggregate rule and does not count null.
No code change. `Count.scala`, `count.sql.out`, and `DataFrameAggregateSuite` already treat distinct count as unique non-null values. GROUP BY / SELECT DISTINCT still bucket nulls. Those are different rules.
### Why are the changes needed?
The page already states both rules, and it is easy to mix them up for `COUNT(DISTINCT)`. The ticket reporter hit that. Philipp pointed at the SQL standard. Nicholas asked to document it on this page. The implementation is already correct.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Docs only. Mentally executed on the `person` table on that page: distinct non-null ages are 30, 18, 50, so the result is 3. I did not run the Jekyll build here because Bundler was not available.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor Grok 4.6
Made with [Cursor](https://cursor.com)
Closes #58038 from Vamsi-klu/spark-47397-count-distinct-nulls.
Lead-authored-by: deepinsight coder <nr...@gmail.com>
Co-authored-by: Cursor Agent <cu...@cursor.com>
Signed-off-by: Uros Bojanic <22...@users.noreply.github.com>
(cherry picked from commit f759028b8609d221ca6c52c2cf449dee31dae79c)
Signed-off-by: Uros Bojanic <22...@users.noreply.github.com>
---
docs/sql-ref-null-semantics.md | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/docs/sql-ref-null-semantics.md b/docs/sql-ref-null-semantics.md
index 36493bced767..6d475f6b2a86 100644
--- a/docs/sql-ref-null-semantics.md
+++ b/docs/sql-ref-null-semantics.md
@@ -273,6 +273,15 @@ SELECT count(age) FROM person;
| 5|
+----------+
+-- `count(DISTINCT age)` also skips `NULL`. This is different from GROUP BY / SELECT DISTINCT,
+-- which put all nulls in one bucket.
+SELECT count(DISTINCT age) FROM person;
++-------------------+
+|count(DISTINCT age)|
++-------------------+
+| 3|
++-------------------+
+
-- `count(*)` on an empty input set returns 0. This is unlike the other
-- aggregate functions, such as `max`, which return `NULL`.
SELECT count(*) FROM person where 1 = 0;
@@ -385,6 +394,8 @@ two `NULL` values are not equal. However, for the purpose of grouping and distin
values with `NULL data`are grouped together into the same bucket. This behaviour is conformant with SQL
standard and with other enterprise database management systems.
+`COUNT(DISTINCT expr)` ignores `NULL` values. Only distinct non-null values are counted.
+
**Examples**
```sql
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org