You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/06/25 20:41:57 UTC

(datafusion-comet) branch asf-site updated: Publish built docs triggered by be9a9650bd2893f1923c2a68eb77374ec991e2e6

This is an automated email from the ASF dual-hosted git repository.

github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git


The following commit(s) were added to refs/heads/asf-site by this push:
     new deb8fc8c93 Publish built docs triggered by be9a9650bd2893f1923c2a68eb77374ec991e2e6
deb8fc8c93 is described below

commit deb8fc8c93507244e1b511c56ac4ae216cb14da3
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Thu Jun 25 20:41:53 2026 +0000

    Publish built docs triggered by be9a9650bd2893f1923c2a68eb77374ec991e2e6
---
 .../contributor-guide/expression-audits/window_funcs.md.txt  |  8 ++++++++
 _sources/user-guide/latest/configs.md.txt                    |  2 +-
 contributor-guide/expression-audits/window_funcs.html        | 12 ++++++++++++
 searchindex.js                                               |  2 +-
 user-guide/latest/configs.html                               |  4 ++--
 5 files changed, 24 insertions(+), 4 deletions(-)

diff --git a/_sources/contributor-guide/expression-audits/window_funcs.md.txt b/_sources/contributor-guide/expression-audits/window_funcs.md.txt
index 290337c15a..8a6b79e583 100644
--- a/_sources/contributor-guide/expression-audits/window_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/window_funcs.md.txt
@@ -21,6 +21,10 @@
 
 > Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide [Spark Expression Support] for current support status.
 
+## avg (window)
+
+- 4.1.1, audited 2026-06-25: `AVG(<decimal>)` over a window falls back to Spark on Spark 4.x. Spark wraps the result as `Cast(avg(UnscaledValue(v)) / pow10 as decimal)`, a shape `CometWindowExec.convert` does not unwrap (it recognizes only `Alias(WindowExpression)` and `Alias(MakeDecimal(WindowExpression))`), so the `AvgDecimal` native window branch in `process_agg_func` is effectively dead on Spark 4.x. Results stay correct via fallback; this is a coverage gap, not a correctness diverge [...]
+
 ## lag
 
 - Supported via `CometWindowExec` (operator level), not the expression serde.
@@ -29,4 +33,8 @@
 
 - Supported via `CometWindowExec` (operator level), not the expression serde.
 
+## sum (window)
+
+- 4.1.1, audited 2026-06-25: `SUM(<decimal>)` over a sliding window frame (lower bound not `UNBOUNDED PRECEDING`) falls back to Spark. The native sliding path would use DataFusion's built-in `sum`, which wraps on overflow rather than returning Spark's NULL, and overflow cannot be detected at plan time, so the whole sliding decimal `SUM` case falls back. Ever-expanding frames use Comet's overflow-aware `SumDecimal` UDAF and run natively, matching Spark including overflow-to-NULL. Bigint s [...]
+
 [Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/user-guide/latest/configs.md.txt b/_sources/user-guide/latest/configs.md.txt
index 05e52ff3f7..ebd6e90ea7 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -46,7 +46,7 @@ Comet provides the following configuration settings.
 | `spark.comet.parquet.read.io.mergeRanges.delta` | The delta in bytes between consecutive read ranges below which the parallel reader will try to merge the ranges. The default is 8MB. | 8388608 |
 | `spark.comet.parquet.read.parallel.io.enabled` | Whether to enable Comet's parallel reader for Parquet files. The parallel reader reads ranges of consecutive data in a  file in parallel. It is faster for large files and row groups but uses more resources. | true |
 | `spark.comet.parquet.read.parallel.io.thread-pool.size` | The maximum number of parallel threads the parallel reader will use in a single executor. For executors configured with a smaller number of cores, use a smaller number. | 16 |
-| `spark.comet.parquet.respectFilterPushdown` | Whether to respect Spark's PARQUET_FILTER_PUSHDOWN_ENABLED config. This needs to be respected when running the Spark SQL test suite but the default setting results in poor performance in Comet when using the new native scans, disabled by default | false |
+| `spark.comet.parquet.rowFilterPushdown.enabled` | When enabled, the native Parquet reader evaluates pushed filters during decode and lazily materializes projected columns for surviving rows (DataFusion's pushdown_filters / late-materialization). Format-level pruning (row-group statistics, page index, bloom filters) is independent of this flag and runs whenever Spark's spark.sql.parquet.filterPushdown is enabled. Disabling this flag still lets format-level pruning work; the per-row eval [...]
 <!-- prettier-ignore-end -->
 <!--END:CONFIG_TABLE-->
 
diff --git a/contributor-guide/expression-audits/window_funcs.html b/contributor-guide/expression-audits/window_funcs.html
index 9f17cdf77e..bfe80178b9 100644
--- a/contributor-guide/expression-audits/window_funcs.html
+++ b/contributor-guide/expression-audits/window_funcs.html
@@ -550,6 +550,12 @@ under the License.
 <blockquote>
 <div><p>Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide <a class="reference internal" href="../../user-guide/latest/expressions.html"><span class="std std-doc">Spark Expression Support</span></a> for current support status.</p>
 </div></blockquote>
+<section id="avg-window">
+<h2>avg (window)<a class="headerlink" href="#avg-window" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>4.1.1, audited 2026-06-25: <code class="docutils literal notranslate"><span class="pre">AVG(&lt;decimal&gt;)</span></code> over a window falls back to Spark on Spark 4.x. Spark wraps the result as <code class="docutils literal notranslate"><span class="pre">Cast(avg(UnscaledValue(v))</span> <span class="pre">/</span> <span class="pre">pow10</span> <span class="pre">as</span> <span class="pre">decimal)</span></code>, a shape <code class="docutils literal notranslate"><span class="p [...]
+</ul>
+</section>
 <section id="lag">
 <h2>lag<a class="headerlink" href="#lag" title="Link to this heading">#</a></h2>
 <ul class="simple">
@@ -562,6 +568,12 @@ under the License.
 <li><p>Supported via <code class="docutils literal notranslate"><span class="pre">CometWindowExec</span></code> (operator level), not the expression serde.</p></li>
 </ul>
 </section>
+<section id="sum-window">
+<h2>sum (window)<a class="headerlink" href="#sum-window" title="Link to this heading">#</a></h2>
+<ul class="simple">
+<li><p>4.1.1, audited 2026-06-25: <code class="docutils literal notranslate"><span class="pre">SUM(&lt;decimal&gt;)</span></code> over a sliding window frame (lower bound not <code class="docutils literal notranslate"><span class="pre">UNBOUNDED</span> <span class="pre">PRECEDING</span></code>) falls back to Spark. The native sliding path would use DataFusion’s built-in <code class="docutils literal notranslate"><span class="pre">sum</span></code>, which wraps on overflow rather than ret [...]
+</ul>
+</section>
 </section>
 
 
diff --git a/searchindex.js b/searchindex.js
index a1402e4b01..f0afd5fbfe 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[50, "id1"]], "%": [[48, "id1"]], "&": [[38, "id1"]], "*": [[48, "id2"]], "+": [[48, "id3"]], "-": [[48, "id4"]], "/": [[48, "id5"]], "1. Format Your Code": [[35, "format-your-code"]], "1. Install Comet": [[56, "install-comet"], [65, "install-comet"]], "1. Native Operators (nativeExecs map)": [[24, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[35, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[56, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[50, "id1"]], "%": [[48, "id1"]], "&": [[38, "id1"]], "*": [[48, "id2"]], "+": [[48, "id3"]], "-": [[48, "id4"]], "/": [[48, "id5"]], "1. Format Your Code": [[35, "format-your-code"]], "1. Install Comet": [[56, "install-comet"], [65, "install-comet"]], "1. Native Operators (nativeExecs map)": [[24, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[35, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[56, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index f6ffeea058..56171f78ed 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -639,8 +639,8 @@ under the License.
 <td><p>The maximum number of parallel threads the parallel reader will use in a single executor. For executors configured with a smaller number of cores, use a smaller number.</p></td>
 <td><p>16</p></td>
 </tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.parquet.respectFilterPushdown</span></code></p></td>
-<td><p>Whether to respect Spark’s PARQUET_FILTER_PUSHDOWN_ENABLED config. This needs to be respected when running the Spark SQL test suite but the default setting results in poor performance in Comet when using the new native scans, disabled by default</p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.parquet.rowFilterPushdown.enabled</span></code></p></td>
+<td><p>When enabled, the native Parquet reader evaluates pushed filters during decode and lazily materializes projected columns for surviving rows (DataFusion’s pushdown_filters / late-materialization). Format-level pruning (row-group statistics, page index, bloom filters) is independent of this flag and runs whenever Spark’s spark.sql.parquet.filterPushdown is enabled. Disabling this flag still lets format-level pruning work; the per-row eval falls back to the CometFilter operator above [...]
 <td><p>false</p></td>
 </tr>
 </tbody>


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org