You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "peterxcli (via GitHub)" <gi...@apache.org> on 2026/09/12 17:45:14 UTC
[I] Parquet runtime pruning can drop NULLS FIRST TopK rows when null_count is missing [datafusion]
peterxcli opened a new issue, #25239:
URL: https://github.com/apache/datafusion/issues/25239
### Describe the bug
Parquet runtime row-group pruning treats an absent `null_count` as zero. This can incorrectly discard nulls needed by a TopK with `NULLS FIRST`.
The assumption is present in [DataFusion 55.1.0's `RowGroupPruner::should_prune`](https://github.com/apache/datafusion/blob/55.1.0/datafusion/datasource-parquet/src/push_decoder.rs#L212-L225), and remains on [current main](https://github.com/apache/datafusion/blob/main/datafusion/datasource-parquet/src/push_decoder.rs#L247-L265). It sets `missing_null_counts_as_zero: true`, which the [statistics adapter](https://github.com/apache/datafusion/blob/55.1.0/datafusion/datasource-parquet/src/row_group_filter.rs#L477-L509) passes to Parquet's `StatisticsConverter`.
The [Parquet specification](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L283-L291) requires an absent null count to remain distinct from a known zero.
### Concrete case to reproduce
Use one file and one scan partition, with a nullable INT column `k` and these row groups in scan order:
| Row group | Values | Column-chunk statistics |
| --- | --- | --- |
| 0 | `0, 1, 2` | Ordinary statistics |
| 1 | `NULL, 100, 101` | `min=100`, `max=101`, but `null_count` absent |
| 2 | `200, 201, 202` | Ordinary statistics |
Keep min/max statistics on group 1 and remove only its optional null-count field. Disabling all statistics is a different case.
Run:
```sql
SELECT k FROM t ORDER BY k ASC NULLS FIRST LIMIT 1;
```
Connect TopK's live predicate to the Parquet scan, use small batches, and disable page-index and row-filter pushdown to isolate runtime row-group pruning.
After the first group, TopK retains `0` and [publishes](https://github.com/apache/datafusion/blob/55.1.0/datafusion/physical-plan/src/topk/mod.rs#L605-L635):
```sql
k IS NULL OR k < 0
```
For group 1, treating the unknown null count as zero makes `IS NULL` appear impossible, while `min=100` rules out `k < 0`. The group can therefore be pruned even though its null must win. The expected result is `NULL`; source tracing predicts `0` if that group is skipped.
### Expected behavior and fix direction
Preserve unknown null counts during pruning. A group must remain readable when its statistics cannot rule out matching nulls. Review the other callers that enable the same assumption, including static statistics pruning.
Add a regression with min/max present and null count omitted, covering TopK filtering off/on and both ascending and descending `NULLS FIRST`. Keep a known-zero-null-count control to verify that valid pruning remains effective.
### Validation and downstream context
This is a source-derived report against DataFusion 55.1.0 and the current main implementation. An isolated native pruning probe compiled, but its process was killed before producing output. I have not completed an end-to-end reproduction, so the predicted wrong result above is not a measured test result.
Found while reviewing [apache/datafusion-comet#5785](https://github.com/apache/datafusion-comet/pull/5785), which connects Spark's local TopK to this reader path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Parquet runtime pruning can drop NULLS FIRST TopK rows when null_count is missing [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 closed issue #25239: Parquet runtime pruning can drop NULLS FIRST TopK rows when null_count is missing
URL: https://github.com/apache/datafusion/issues/25239
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Parquet runtime pruning can drop NULLS FIRST TopK rows when null_count is missing [datafusion]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on issue #25239:
URL: https://github.com/apache/datafusion/issues/25239#issuecomment-5647624630
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org