You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/14 15:48:02 UTC
(datafusion) branch asf-site updated: Publish built docs triggered by c14976481ea53d7e1806eb9dbc1cff803c6eba31
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 1c82bedef1 Publish built docs triggered by c14976481ea53d7e1806eb9dbc1cff803c6eba31
1c82bedef1 is described below
commit 1c82bedef19e69322e953da3010dcf2d23a06345
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Mon Sep 14 15:47:56 2026 +0000
Publish built docs triggered by c14976481ea53d7e1806eb9dbc1cff803c6eba31
---
.../library-user-guide/upgrading/56.0.0.md.txt | 37 ++++++++++++++++++++++
library-user-guide/upgrading/56.0.0.html | 34 ++++++++++++++++++++
searchindex.js | 2 +-
3 files changed, 72 insertions(+), 1 deletion(-)
diff --git a/_sources/library-user-guide/upgrading/56.0.0.md.txt b/_sources/library-user-guide/upgrading/56.0.0.md.txt
index 8de2abc5a0..7658ce60ed 100644
--- a/_sources/library-user-guide/upgrading/56.0.0.md.txt
+++ b/_sources/library-user-guide/upgrading/56.0.0.md.txt
@@ -25,6 +25,43 @@
in this section pertains to features and changes that have already been merged
to the main branch and are awaiting release in this version.
+### Missing Parquet null counts are treated as unknown
+
+DataFusion now preserves an omitted Parquet `null_count` statistic as unknown.
+Previously, it assumed zero nulls, which could discard matching NULL rows in
+`IS NULL` filters and `ORDER BY ... NULLS FIRST LIMIT` queries, or produce an
+incorrectly high `COUNT(column)` from metadata.
+
+Queries over affected files may read or sort more data because these optimizations
+require a known null count:
+
+- Pruning row groups for `IS NULL` filters.
+- Computing `COUNT(column)` using only file metadata.
+- Pruning row groups with a dynamic `NULLS FIRST` TopK filter.
+- Eliminating a sort over nullable columns in sorted, non-overlapping files.
+ Such queries may now retain a full `SortExec`, even when the data has no NULLs.
+
+parquet-rs versions before 53.1.0 omitted zero null counts. This includes files
+produced with the older parquet-rs dependency used by DataFusion releases before
+42.1.0. [Arrow #6490](https://github.com/apache/arrow-rs/pull/6490) changed the
+writer to record known zero counts. Files with explicit counts retain their
+existing behavior.
+
+To identify column chunks with bounds but no null count, run this query in the
+DataFusion CLI:
+
+```sql
+SELECT row_group_id, path_in_schema
+FROM parquet_metadata('data.parquet')
+WHERE (stats_min IS NOT NULL OR stats_max IS NOT NULL)
+ AND stats_null_count IS NULL;
+```
+
+Rewriting affected files with a current DataFusion version, for example using
+`COPY ... TO` with Parquet statistics enabled, records the missing counts and
+restores optimizations that depend on them. Preserve the required data ordering
+and ordering metadata when rewriting sorted files.
+
### `ForeignSession::create_physical_plan` is unsupported
`ForeignSession::create_physical_plan` no longer forwards to the library that
diff --git a/library-user-guide/upgrading/56.0.0.html b/library-user-guide/upgrading/56.0.0.html
index ec2fffe8a0..0fdd3b852f 100644
--- a/library-user-guide/upgrading/56.0.0.html
+++ b/library-user-guide/upgrading/56.0.0.html
@@ -456,6 +456,39 @@
<p><strong>Note:</strong> DataFusion <code class="docutils literal notranslate"><span class="pre">56.0.0</span></code> has not been released yet. The information provided
in this section pertains to features and changes that have already been merged
to the main branch and are awaiting release in this version.</p>
+<section id="missing-parquet-null-counts-are-treated-as-unknown">
+<h3>Missing Parquet null counts are treated as unknown<a class="headerlink" href="#missing-parquet-null-counts-are-treated-as-unknown" title="Link to this heading">#</a></h3>
+<p>DataFusion now preserves an omitted Parquet <code class="docutils literal notranslate"><span class="pre">null_count</span></code> statistic as unknown.
+Previously, it assumed zero nulls, which could discard matching NULL rows in
+<code class="docutils literal notranslate"><span class="pre">IS</span> <span class="pre">NULL</span></code> filters and <code class="docutils literal notranslate"><span class="pre">ORDER</span> <span class="pre">BY</span> <span class="pre">...</span> <span class="pre">NULLS</span> <span class="pre">FIRST</span> <span class="pre">LIMIT</span></code> queries, or produce an
+incorrectly high <code class="docutils literal notranslate"><span class="pre">COUNT(column)</span></code> from metadata.</p>
+<p>Queries over affected files may read or sort more data because these optimizations
+require a known null count:</p>
+<ul class="simple">
+<li><p>Pruning row groups for <code class="docutils literal notranslate"><span class="pre">IS</span> <span class="pre">NULL</span></code> filters.</p></li>
+<li><p>Computing <code class="docutils literal notranslate"><span class="pre">COUNT(column)</span></code> using only file metadata.</p></li>
+<li><p>Pruning row groups with a dynamic <code class="docutils literal notranslate"><span class="pre">NULLS</span> <span class="pre">FIRST</span></code> TopK filter.</p></li>
+<li><p>Eliminating a sort over nullable columns in sorted, non-overlapping files.
+Such queries may now retain a full <code class="docutils literal notranslate"><span class="pre">SortExec</span></code>, even when the data has no NULLs.</p></li>
+</ul>
+<p>parquet-rs versions before 53.1.0 omitted zero null counts. This includes files
+produced with the older parquet-rs dependency used by DataFusion releases before
+42.1.0. <a class="reference external" href="https://github.com/apache/arrow-rs/pull/6490">Arrow #6490</a> changed the
+writer to record known zero counts. Files with explicit counts retain their
+existing behavior.</p>
+<p>To identify column chunks with bounds but no null count, run this query in the
+DataFusion CLI:</p>
+<div class="highlight-sql notranslate"><div class="highlight"><pre><span></span><span class="k">SELECT</span><span class="w"> </span><span class="n">row_group_id</span><span class="p">,</span><span class="w"> </span><span class="n">path_in_schema</span>
+<span class="k">FROM</span><span class="w"> </span><span class="n">parquet_metadata</span><span class="p">(</span><span class="s1">'data.parquet'</span><span class="p">)</span>
+<span class="k">WHERE</span><span class="w"> </span><span class="p">(</span><span class="n">stats_min</span><span class="w"> </span><span class="k">IS</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="w"> </span><span class="k">OR</span><span class="w"> </span><span class="n">stats_max</span><span class="w"> </span><span class="k">IS</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span [...]
+<span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">stats_null_count</span><span class="w"> </span><span class="k">IS</span><span class="w"> </span><span class="k">NULL</span><span class="p">;</span>
+</pre></div>
+</div>
+<p>Rewriting affected files with a current DataFusion version, for example using
+<code class="docutils literal notranslate"><span class="pre">COPY</span> <span class="pre">...</span> <span class="pre">TO</span></code> with Parquet statistics enabled, records the missing counts and
+restores optimizations that depend on them. Preserve the required data ordering
+and ordering metadata when rewriting sorted files.</p>
+</section>
<section id="foreignsession-create-physical-plan-is-unsupported">
<h3><code class="docutils literal notranslate"><span class="pre">ForeignSession::create_physical_plan</span></code> is unsupported<a class="headerlink" href="#foreignsession-create-physical-plan-is-unsupported" title="Link to this heading">#</a></h3>
<p><code class="docutils literal notranslate"><span class="pre">ForeignSession::create_physical_plan</span></code> no longer forwards to the library that
@@ -735,6 +768,7 @@ to be updated to account for the new generic API.</p>
<nav id="pst-page-toc-nav" class="page-toc" aria-labelledby="pst-page-navigation-heading-2">
<ul class="pst-show_toc_level nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion-56-0-0">DataFusion 56.0.0</a><ul class="nav section-nav flex-column">
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#missing-parquet-null-counts-are-treated-as-unknown">Missing Parquet null counts are treated as unknown</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#foreignsession-create-physical-plan-is-unsupported"><code class="docutils literal notranslate"><span class="pre">ForeignSession::create_physical_plan</span></code> is unsupported</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#groupcolumn-now-requires-values-preserving"><code class="docutils literal notranslate"><span class="pre">GroupColumn</span></code> now requires <code class="docutils literal notranslate"><span class="pre">values_preserving</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion-proto-common-option-and-constraint-conversions-are-fallible"><code class="docutils literal notranslate"><span class="pre">datafusion-proto</span></code>: common option and constraint conversions are fallible</a></li>
diff --git a/searchindex.js b/searchindex.js
index 45e67e643a..a94bfb298f 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles":{"!=":[[77,"op-neq"]],"!~":[[77,"op-re-not-match"]],"!~*":[[77,"op-re-not-match-i"]],"!~~":[[77,"id19"]],"!~~*":[[77,"id20"]],"#":[[77,"op-bit-xor"]],"%":[[77,"op-modulo"]],"&":[[77,"op-bit-and"]],"(relation, name) tuples in logical fields and logical columns are unique":[[16,"relation-name-tuples-in-logical-fields-and-logical-columns-are-unique"]],"*":[[77,"op-multiply"]],"+":[[77,"op-plus"]],"-":[[77,"op-minus"]],"/":[[77,"op-divide"]],"1. Array Literal Con [...]
\ No newline at end of file
+Search.setIndex({"alltitles":{"!=":[[77,"op-neq"]],"!~":[[77,"op-re-not-match"]],"!~*":[[77,"op-re-not-match-i"]],"!~~":[[77,"id19"]],"!~~*":[[77,"id20"]],"#":[[77,"op-bit-xor"]],"%":[[77,"op-modulo"]],"&":[[77,"op-bit-and"]],"(relation, name) tuples in logical fields and logical columns are unique":[[16,"relation-name-tuples-in-logical-fields-and-logical-columns-are-unique"]],"*":[[77,"op-multiply"]],"+":[[77,"op-plus"]],"-":[[77,"op-minus"]],"/":[[77,"op-divide"]],"1. Array Literal Con [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org