You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/04/21 22:15:01 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by ee140c7b7d712f8e13465a12f121f034218113b6
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new c06e51edc Publish built docs triggered by ee140c7b7d712f8e13465a12f121f034218113b6
c06e51edc is described below
commit c06e51edcfc511b19b3893722c439603be3f821c
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Tue Apr 21 22:14:56 2026 +0000
Publish built docs triggered by ee140c7b7d712f8e13465a12f121f034218113b6
---
_sources/contributor-guide/parquet_scans.md.txt | 2 +-
_sources/contributor-guide/roadmap.md.txt | 10 ++++-----
_sources/user-guide/latest/configs.md.txt | 1 -
contributor-guide/parquet_scans.html | 2 +-
contributor-guide/roadmap.html | 9 ++++----
searchindex.js | 2 +-
user-guide/latest/configs.html | 30 +++++++++++--------------
7 files changed, 25 insertions(+), 31 deletions(-)
diff --git a/_sources/contributor-guide/parquet_scans.md.txt b/_sources/contributor-guide/parquet_scans.md.txt
index 134d4c711..cb94c2851 100644
--- a/_sources/contributor-guide/parquet_scans.md.txt
+++ b/_sources/contributor-guide/parquet_scans.md.txt
@@ -46,7 +46,7 @@ The following features are not supported by either scan implementation, and Come
- Spark's Datasource V2 API. When `spark.sql.sources.useV1SourceList` does not include `parquet`, Spark uses the
V2 API for Parquet scans. The DataFusion-based implementations only support the V1 API.
- Spark metadata columns (e.g., `_metadata.file_path`)
-- No support for Dynamic Partition Pruning (DPP)
+- No support for AQE Dynamic Partition Pruning (DPP). Non-AQE DPP is supported.
The following shared limitation may produce incorrect results without falling back to Spark:
diff --git a/_sources/contributor-guide/roadmap.md.txt b/_sources/contributor-guide/roadmap.md.txt
index c685e0fb8..3f9fc7eeb 100644
--- a/_sources/contributor-guide/roadmap.md.txt
+++ b/_sources/contributor-guide/roadmap.md.txt
@@ -42,14 +42,14 @@ more Spark SQL tests and fully implementing ANSI support ([#313]) for all suppor
### Dynamic Partition Pruning
-Iceberg table scans support Dynamic Partition Pruning (DPP) filters generated by Spark's `PlanDynamicPruningFilters`
-optimizer rule ([#3349]). However, we still need to bring this functionality to our Parquet reader. Furthermore,
-Spark's `PlanAdaptiveDynamicPruningFilters` optimizer rule runs after Comet's rules, so DPP with Adaptive Query
-Execution requires a redesign of Comet's plan translation. We are focused on implementing DPP to keep Comet competitive
-with benchmarks that benefit from this feature like TPC-DS. This effort can be tracked at [#3510].
+Both Iceberg table scans and Parquet V1 native scans (`CometNativeScanExec`) support non-AQE Dynamic Partition Pruning
+(DPP) filters generated by Spark's `PlanDynamicPruningFilters` optimizer rule ([#3349], [#3511]). However, Spark's
+`PlanAdaptiveDynamicPruningFilters` optimizer rule runs after Comet's rules, so DPP with Adaptive Query Execution
+requires a redesign of Comet's plan translation. This effort can be tracked at [#3510].
[#3349]: https://github.com/apache/datafusion-comet/pull/3349
[#3510]: https://github.com/apache/datafusion-comet/issues/3510
+[#3511]: https://github.com/apache/datafusion-comet/pull/3511
## Ongoing Improvements
diff --git a/_sources/user-guide/latest/configs.md.txt b/_sources/user-guide/latest/configs.md.txt
index facd0ad5b..1dd5bb90f 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -62,7 +62,6 @@ Comet provides the following configuration settings.
| `spark.comet.convert.json.enabled` | When enabled, data from Spark (non-native) JSON v1 and v2 scans will be converted to Arrow format. | false |
| `spark.comet.convert.parquet.enabled` | When enabled, data from Spark (non-native) Parquet v1 and v2 scans will be converted to Arrow format. | false |
| `spark.comet.debug.enabled` | Whether to enable debug mode for Comet. When enabled, Comet will do additional checks for debugging purpose. For example, validating array when importing arrays from JVM at native side. Note that these checks may be expensive in performance and should only be enabled for debugging purpose. | false |
-| `spark.comet.dppFallback.enabled` | Whether to fall back to Spark for queries that use DPP. | true |
| `spark.comet.enabled` | Whether to enable Comet extension for Spark. When this is turned on, Spark will use Comet to read Parquet data source. Note that to enable native vectorized execution, both this config and `spark.comet.exec.enabled` need to be enabled. It can be overridden by the environment variable `ENABLE_COMET`. | true |
| `spark.comet.exceptionOnDatetimeRebase` | Whether to throw exception when seeing dates/timestamps from the legacy hybrid (Julian + Gregorian) calendar. Since Spark 3, dates/timestamps were written according to the Proleptic Gregorian calendar. When this is true, Comet will throw exceptions when seeing these dates/timestamps that were written by Spark version before 3.0. If this is false, these dates/timestamps will be read as if they were written to the Proleptic Gregorian calendar and [...]
| `spark.comet.exec.columnarToRow.native.enabled` | Whether to enable native columnar to row conversion. When enabled, Comet will use native Rust code to convert Arrow columnar data to Spark UnsafeRow format instead of the JVM implementation. This can improve performance for queries that need to convert between columnar and row formats. | true |
diff --git a/contributor-guide/parquet_scans.html b/contributor-guide/parquet_scans.html
index 71cf69b12..d2b28c34f 100644
--- a/contributor-guide/parquet_scans.html
+++ b/contributor-guide/parquet_scans.html
@@ -495,7 +495,7 @@ columns are always safe because they can only come from signed <code class="docu
<li><p>Spark’s Datasource V2 API. When <code class="docutils literal notranslate"><span class="pre">spark.sql.sources.useV1SourceList</span></code> does not include <code class="docutils literal notranslate"><span class="pre">parquet</span></code>, Spark uses the
V2 API for Parquet scans. The DataFusion-based implementations only support the V1 API.</p></li>
<li><p>Spark metadata columns (e.g., <code class="docutils literal notranslate"><span class="pre">_metadata.file_path</span></code>)</p></li>
-<li><p>No support for Dynamic Partition Pruning (DPP)</p></li>
+<li><p>No support for AQE Dynamic Partition Pruning (DPP). Non-AQE DPP is supported.</p></li>
</ul>
<p>The following shared limitation may produce incorrect results without falling back to Spark:</p>
<ul class="simple">
diff --git a/contributor-guide/roadmap.html b/contributor-guide/roadmap.html
index bec9301d7..8549e791b 100644
--- a/contributor-guide/roadmap.html
+++ b/contributor-guide/roadmap.html
@@ -476,11 +476,10 @@ more Spark SQL tests and fully implementing ANSI support (<a class="reference ex
</section>
<section id="dynamic-partition-pruning">
<h3>Dynamic Partition Pruning<a class="headerlink" href="#dynamic-partition-pruning" title="Link to this heading">#</a></h3>
-<p>Iceberg table scans support Dynamic Partition Pruning (DPP) filters generated by Spark’s <code class="docutils literal notranslate"><span class="pre">PlanDynamicPruningFilters</span></code>
-optimizer rule (<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/3349">#3349</a>). However, we still need to bring this functionality to our Parquet reader. Furthermore,
-Spark’s <code class="docutils literal notranslate"><span class="pre">PlanAdaptiveDynamicPruningFilters</span></code> optimizer rule runs after Comet’s rules, so DPP with Adaptive Query
-Execution requires a redesign of Comet’s plan translation. We are focused on implementing DPP to keep Comet competitive
-with benchmarks that benefit from this feature like TPC-DS. This effort can be tracked at <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/3510">#3510</a>.</p>
+<p>Both Iceberg table scans and Parquet V1 native scans (<code class="docutils literal notranslate"><span class="pre">CometNativeScanExec</span></code>) support non-AQE Dynamic Partition Pruning
+(DPP) filters generated by Spark’s <code class="docutils literal notranslate"><span class="pre">PlanDynamicPruningFilters</span></code> optimizer rule (<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/3349">#3349</a>, <a class="reference external" href="https://github.com/apache/datafusion-comet/pull/3511">#3511</a>). However, Spark’s
+<code class="docutils literal notranslate"><span class="pre">PlanAdaptiveDynamicPruningFilters</span></code> optimizer rule runs after Comet’s rules, so DPP with Adaptive Query Execution
+requires a redesign of Comet’s plan translation. This effort can be tracked at <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/3510">#3510</a>.</p>
</section>
</section>
<section id="ongoing-improvements">
diff --git a/searchindex.js b/searchindex.js
index b05f8cd33..27e55693f 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Format Your Code": [[14, "format-your-code"]], "1. Install Comet": [[17, "install-comet"], [26, "install-comet"]], "1. Native Operators (nativeExecs map)": [[4, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[14, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[17, "clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[26, "clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[4, "sink-operators-sinks-m [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Format Your Code": [[14, "format-your-code"]], "1. Install Comet": [[17, "install-comet"], [26, "install-comet"]], "1. Native Operators (nativeExecs map)": [[4, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[14, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[17, "clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[26, "clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[4, "sink-operators-sinks-m [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index 954179cca..7a434fe25 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -577,59 +577,55 @@ under the License.
<td><p>Whether to enable debug mode for Comet. When enabled, Comet will do additional checks for debugging purpose. For example, validating array when importing arrays from JVM at native side. Note that these checks may be expensive in performance and should only be enabled for debugging purpose.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.dppFallback.enabled</span></code></p></td>
-<td><p>Whether to fall back to Spark for queries that use DPP.</p></td>
-<td><p>true</p></td>
-</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.enabled</span></code></p></td>
<td><p>Whether to enable Comet extension for Spark. When this is turned on, Spark will use Comet to read Parquet data source. Note that to enable native vectorized execution, both this config and <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.enabled</span></code> need to be enabled. It can be overridden by the environment variable <code class="docutils literal notranslate"><span class="pre">ENABLE_COMET</span></code>.</p></td>
<td><p>true</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exceptionOnDatetimeRebase</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exceptionOnDatetimeRebase</span></code></p></td>
<td><p>Whether to throw exception when seeing dates/timestamps from the legacy hybrid (Julian + Gregorian) calendar. Since Spark 3, dates/timestamps were written according to the Proleptic Gregorian calendar. When this is true, Comet will throw exceptions when seeing these dates/timestamps that were written by Spark version before 3.0. If this is false, these dates/timestamps will be read as if they were written to the Proleptic Gregorian calendar and will not be rebased.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.columnarToRow.native.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.columnarToRow.native.enabled</span></code></p></td>
<td><p>Whether to enable native columnar to row conversion. When enabled, Comet will use native Rust code to convert Arrow columnar data to Spark UnsafeRow format instead of the JVM implementation. This can improve performance for queries that need to convert between columnar and row formats.</p></td>
<td><p>true</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.enabled</span></code></p></td>
<td><p>Whether to enable Comet native vectorized execution for Spark. This controls whether Spark should convert operators into their Comet counterparts and execute them in native space. Note: each operator is associated with a separate config in the format of <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.<operator_name>.enabled</span></code> at the moment, and both the config and this need to be turned on, in order for the operator to be executed in [...]
<td><p>true</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.replaceSortMergeJoin</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.replaceSortMergeJoin</span></code></p></td>
<td><p>Experimental feature to force Spark to replace SortMergeJoin with ShuffledHashJoin for improved performance. This feature is not stable yet. For more information, refer to the <a class="reference external" href="https://datafusion.apache.org/comet/user-guide/tuning.html">Comet Tuning Guide</a>.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint</span></code></p></td>
<td><p>When enabled, fall back to Spark for floating-point operations that may differ from Spark, such as when comparing or sorting -0.0 and 0.0. For more information, refer to the <a class="reference external" href="https://datafusion.apache.org/comet/user-guide/compatibility.html">Comet Compatibility Guide</a>.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.maxTempDirectorySize</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.maxTempDirectorySize</span></code></p></td>
<td><p>The maximum amount of data (in bytes) stored inside the temporary directories.</p></td>
<td><p>107374182400b</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.metrics.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.metrics.enabled</span></code></p></td>
<td><p>Whether to enable Comet metrics reporting through Spark’s external monitoring system. When enabled, Comet exposes metrics such as native operators, Spark operators, queries planned, transitions, and acceleration ratio. These metrics can be visualized through tools like Grafana when a metrics sink (e.g., Prometheus) is configured. Disabled by default because Spark plan traversal adds overhead and metrics require a sink to be useful. This config must be set before the SparkSession i [...]
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.metrics.updateInterval</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.metrics.updateInterval</span></code></p></td>
<td><p>The interval in milliseconds to update metrics. If interval is negative, metrics will be updated upon task completion.</p></td>
<td><p>3000</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.nativeLoadRequired</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.nativeLoadRequired</span></code></p></td>
<td><p>Whether to require Comet native library to load successfully when Comet is enabled. If not, Comet will silently fallback to Spark when it fails to load the native lib. Otherwise, an error will be thrown and the Spark job will be aborted.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.operator.DataWritingCommandExec.allowIncompatible</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.operator.DataWritingCommandExec.allowIncompatible</span></code></p></td>
<td><p>Whether to allow incompatibility for operator: DataWritingCommandExec. False by default. Can be overridden with SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE env variable It can be overridden by the environment variable <code class="docutils literal notranslate"><span class="pre">SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE</span></code>.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled</span></code></p></td>
<td><p>Whether to enable Spark to Arrow columnar conversion. When this is turned on, Comet will convert operators in <code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code> into Arrow columnar format before processing.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code></p></td>
<td><p>A comma-separated list of operators that will be converted to Arrow columnar format when <code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled</span></code> is true.</p></td>
<td><p>Range,InMemoryTableScan,RDDScan</p></td>
</tr>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org