You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "peterxcli (via GitHub)" <gi...@apache.org> on 2026/09/12 19:43:20 UTC
[PR] fix: preserve missing Parquet null counts [datafusion]
peterxcli opened a new pull request, #25241:
URL: https://github.com/apache/datafusion/pull/25241
## Which issue does this PR close?
Closes #25239. Follows up on #21907 and [Arrow #6485](https://github.com/apache/arrow-rs/pull/6485).
## Rationale for this change
An absent Parquet `null_count` is unknown. Treating it as zero can discard NULL rows during `NULLS FIRST` TopK and `IS NULL` pruning, and let metadata-only `COUNT(column)` return an incorrect result.
## What changes are included in this PR?
Preserve missing null counts in the shared row-group statistics adapter and file statistics. Remove the per-caller compatibility flag so static pruning, runtime pruning, and fully matched proofs use the same interpretation.
## What is the testing strategy for this PR?
Regressions cover missing and explicit-zero counts, static pruning, file-statistics precision, `COUNT`, and TopK with ASC/DESC, NULLS FIRST/LAST, and dynamic pruning enabled/disabled. The affected cases fail without the fix. Extended workspace tests, strict all-target/all-feature Clippy, and the full lint suite pass.
## Are there any user-facing changes?
Correct results for Parquet files with missing null counts. These files, including older parquet-rs output, may lose optimizations that require known null counts; this also affects `IS NULL` and metadata-only `COUNT`, beyond `NULLS FIRST`. Explicit counts retain their existing behavior. No public API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve missing Parquet null counts [datafusion]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli closed pull request #25241: fix: preserve missing Parquet null counts
URL: https://github.com/apache/datafusion/pull/25241
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve missing Parquet null counts [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25241:
URL: https://github.com/apache/datafusion/pull/25241#issuecomment-5648372857
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/25241?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 81.95%. Comparing base ([`6079a1e`](https://app.codecov.io/gh/apache/datafusion/commit/6079a1e6ef811958b41593b559b572df5bd55690?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`ec3925a`](https://app.codecov.io/gh/apache/datafusion/commit/ec3925a6adc4cdfb4ec0ca1271e2ccaa3c6e183d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #25241 +/- ##
==========================================
- Coverage 81.95% 81.95% -0.01%
==========================================
Files 1133 1133
Lines 423799 423853 +54
Branches 423799 423853 +54
==========================================
+ Hits 347307 347349 +42
- Misses 55899 55906 +7
- Partials 20593 20598 +5
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/25241?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org