You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/06 06:24:04 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by ba7c89203813b848fb620f1078276b9f42614f96
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 4128c980b5 Publish built docs triggered by ba7c89203813b848fb620f1078276b9f42614f96
4128c980b5 is described below
commit 4128c980b534c7d4ced473599e53b956e5f5a248
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Tue Oct 6 06:23:59 2026 +0000
Publish built docs triggered by ba7c89203813b848fb620f1078276b9f42614f96
---
...id-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg | 2 +-
.../contributor-guide/memory_management.md.txt | 25 ++++++++--
_sources/user-guide/latest/configs.md.txt | 3 +-
_sources/user-guide/latest/datasources.md.txt | 2 +
_sources/user-guide/latest/migration-guide.md.txt | 18 +++----
contributor-guide/memory_management.html | 26 ++++++++--
searchindex.js | 2 +-
user-guide/latest/configs.html | 56 ++++++++++++----------
user-guide/latest/datasources.html | 2 +
user-guide/latest/migration-guide.html | 20 ++++----
10 files changed, 101 insertions(+), 55 deletions(-)
diff --git a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
index eaec63ba7e..fcfa1f29d9 100644
--- a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
+++ b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/memory_management.md.txt b/_sources/contributor-guide/memory_management.md.txt
index d899d02029..2a48baa3f1 100644
--- a/_sources/contributor-guide/memory_management.md.txt
+++ b/_sources/contributor-guide/memory_management.md.txt
@@ -253,14 +253,17 @@ ignored and the pool is always `UnboundedMemoryPool`.
the inside out, a Comet plan in the default configuration sees:
```text
-[LoggingMemoryPool] <- only when spark.comet.debug.memory=true
- [TaskSharedMemoryPool] <- RAII handle for the per-task registry
- [TrackConsumersPool] <- DataFusion; names the top 10 consumers in error messages
- [CometFairMemoryPool] <- delegates acquire/release to Spark over JNI
+[LoggingMemoryPool] <- only when spark.comet.debug.memory=true
+ [TaskSharedMemoryPool] <- RAII handle for the per-task registry
+ [SpillReplayPool] <- lets a spilled final aggregate read its spill files back
+ [TrackConsumersPool] <- DataFusion; names the top 10 consumers in error messages
+ [CometFairMemoryPool] <- delegates acquire/release to Spark over JNI
```
Each decorator forwards every `MemoryPool` method to its inner pool, so `reserved()` at any level
-reports the base pool's number.
+reports the base pool's number. `SpillReplayPool` also turns one kind of refused `try_grow` into a
+`grow`; see
+[Final aggregates reading their spill files back](#final-aggregates-reading-their-spill-files-back).
### The unified pools
@@ -346,6 +349,18 @@ hands every native plan in a task the same one and drops it when the task comple
covers the whole task, so `CometExecIterator.close()` warns about memory still in use only when
the task's last open native plan closes.
+### Final aggregates reading their spill files back
+
+`SpillReplayPool` (`spill_replay.rs`) wraps both Comet pools to work around
+[issue #6254](https://github.com/apache/datafusion-comet/issues/6254). Once one of DataFusion 55's
+final aggregates has spilled, it reads its spill files back through an aggregate that cannot spill,
+so a refused `try_grow` there fails the task. When the pool refuses such a request,
+`SpillReplayPool` records it with the pool's `grow` instead, which skips `CometFairMemoryPool`'s
+local checks and carries what Spark does not grant as overcommit. Every other refusal is passed on
+unchanged. `spill_replay.rs` describes how the wrapper recognizes these requests and what that relies
+on in DataFusion. [Issue #6583](https://github.com/apache/datafusion-comet/issues/6583) tracks
+removing it.
+
## How DataFusion consumes the pool
Native operators reserve through DataFusion's `MemoryConsumer` / `MemoryReservation` API:
diff --git a/_sources/user-guide/latest/configs.md.txt b/_sources/user-guide/latest/configs.md.txt
index 340b449c22..7eb912756e 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -61,6 +61,7 @@ Comet provides the following configuration settings.
| `spark.comet.convert.parquet.enabled` | When enabled, data from Spark (non-native) Parquet v1 and v2 scans will be converted to Arrow format. | false |
| `spark.comet.convert.range.enabled` | When enabled, the rows of `spark.range` and SQL `range()` that Comet does not generate natively (see `spark.comet.exec.range.enabled`) will be converted to Arrow format. | false |
| `spark.comet.convert.rdd.enabled` | When enabled, data from a DataFrame created from an RDD of rows, for example with `spark.createDataFrame(rdd, schema)`, will be converted to Arrow format. | false |
+| `spark.comet.convert.rowDataSource.enabled` | When enabled, data from Data Source V1 relations that are not file-based, such as JDBC tables, will be converted to Arrow format. Spark scans these relations with `RowDataSourceScanExec`. | false |
| `spark.comet.convert.typedDataset.enabled` | When enabled, the output of typed Dataset operations, such as `map`, `flatMap`, `mapPartitions` and `groupByKey(...).mapGroups`, will be converted to Arrow format so that the operators above them can run natively. The user function still runs in Spark. This pays off when the operators above do enough work, such as an aggregation over many groups, and can be slower when they are cheap, such as an aggregation over a few groups after a selectiv [...]
| `spark.comet.debug.enabled` | Whether to enable debug mode for Comet. When enabled, Comet will do additional checks for debugging purpose. For example, validating array when importing arrays from JVM at native side. Note that these checks may be expensive in performance and should only be enabled for debugging purpose. | false |
| `spark.comet.enabled` | Whether to enable Comet extension for Spark. When this is turned on, Spark will use Comet to read Parquet data source. Note that to enable native vectorized execution, both this config and `spark.comet.exec.enabled` need to be enabled. It can be overridden by the environment variable `ENABLE_COMET`. | true |
@@ -85,7 +86,7 @@ Comet provides the following configuration settings.
| `spark.comet.operator.DataWritingCommandExec.allowIncompatible` | Whether to allow incompatibility for operator: DataWritingCommandExec. False by default. Can be overridden with SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE env variable It can be overridden by the environment variable `SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE`. | false |
| `spark.comet.operator.WriteFilesExec.allowIncompatible` | Whether to allow incompatibility for operator: WriteFilesExec. False by default. Can be overridden with SPARK_COMET_OPERATOR_WRITEFILESEXEC_ALLOWINCOMPATIBLE env variable It can be overridden by the environment variable `SPARK_COMET_OPERATOR_WRITEFILESEXEC_ALLOWINCOMPATIBLE`. | false |
| `spark.comet.sparkToColumnar.enabled` | Whether to convert the output of the leaf operators named in `spark.comet.sparkToColumnar.supportedOperatorList` to Arrow format. Ranges, in-memory cached tables, RDDs and queries without a FROM clause have `spark.comet.convert` configs of their own. When the list is not set, this converts those four instead, which is deprecated and will be removed in a future major release. | false |
-| `spark.comet.sparkToColumnar.supportedOperatorList` | A comma-separated list of leaf operators to convert to Arrow format when `spark.comet.sparkToColumnar.enabled` is true, each named by its Spark class name without the `Exec` suffix. It is for operators that have no `spark.comet.convert` config of their own, such as `BatchScan` for a Data Source V2 connector, or `FileSourceScan` for file formats other than Parquet, JSON and CSV. Naming `Range`, `InMemoryTableScan`, `RDDScan` or `OneR [...]
+| `spark.comet.sparkToColumnar.supportedOperatorList` | A comma-separated list of leaf operators to convert to Arrow format when `spark.comet.sparkToColumnar.enabled` is true, each named by its Spark class name without the `Exec` suffix. It is for operators that have no `spark.comet.convert` config of their own, such as `BatchScan` for a Data Source V2 connector, or `FileSourceScan` for file formats other than Parquet, JSON and CSV. Naming `Range`, `InMemoryTableScan`, `RDDScan`, `OneRow [...]
<!-- prettier-ignore-end -->
<!--END:CONFIG_TABLE-->
diff --git a/_sources/user-guide/latest/datasources.md.txt b/_sources/user-guide/latest/datasources.md.txt
index 2ac92ec518..14d69d472a 100644
--- a/_sources/user-guide/latest/datasources.md.txt
+++ b/_sources/user-guide/latest/datasources.md.txt
@@ -62,6 +62,8 @@ above them can run in Comet. Each conversion is off by default.
`spark.createDataFrame(rdd, schema)`.
- `spark.comet.convert.oneRowRelation.enabled`: the single row that a query without a `FROM`
clause, such as `SELECT 1`, reads.
+- `spark.comet.convert.rowDataSource.enabled`: Data Source V1 relations that are not file-based,
+ such as JDBC tables, which Spark scans with `RowDataSourceScanExec`.
To convert any other leaf operator, such as the scan of a Data Source V2 connector or of a file
format other than Parquet, JSON and CSV, set `spark.comet.sparkToColumnar.enabled=true` and name the
diff --git a/_sources/user-guide/latest/migration-guide.md.txt b/_sources/user-guide/latest/migration-guide.md.txt
index 0a9731a2fa..b55cd94d97 100644
--- a/_sources/user-guide/latest/migration-guide.md.txt
+++ b/_sources/user-guide/latest/migration-guide.md.txt
@@ -62,9 +62,9 @@ key is removed.
### Deprecated and Removed Settings
Using `spark.comet.sparkToColumnar.enabled` and `spark.comet.sparkToColumnar.supportedOperatorList`
-to convert ranges, in-memory cached tables, RDDs and queries without a `FROM` clause to Arrow is
-deprecated, and will stop working in a future major release. Each of these now has a config of its
-own:
+to convert ranges, in-memory cached tables, RDDs, queries without a `FROM` clause and Data Source V1
+relations that are not file-based to Arrow is deprecated, and will stop working in a future major
+release. Each of these now has a config of its own:
| Operator in the list | Config |
| -------------------- | -------------------------------------------- |
@@ -72,13 +72,15 @@ own:
| `InMemoryTableScan` | `spark.comet.convert.inMemoryCache.enabled` |
| `RDDScan` | `spark.comet.convert.rdd.enabled` |
| `OneRowRelation` | `spark.comet.convert.oneRowRelation.enabled` |
+| `RowDataSourceScan` | `spark.comet.convert.rowDataSource.enabled` |
`spark.comet.sparkToColumnar.supportedOperatorList` now defaults to an empty list. When it is not
-set, `spark.comet.sparkToColumnar.enabled=true` still converts these four, and a list that names one
-of them still converts it, as before. In both cases the driver logs a warning that names the config
-to use instead. Before Spark 4.1, Spark plans a query without a `FROM` clause as an `RDDScan`, so on
-those versions `RDDScan` in the list also converted it. The list remains the way to convert other
-leaf operators, such as the scan of a Data Source V2 connector.
+set, `spark.comet.sparkToColumnar.enabled=true` still converts the first four, which the list named
+by default, and a list that names any of the five still converts it, as before. In both cases the
+driver logs a warning that names the config to use instead. Before Spark 4.1, Spark plans a query
+without a `FROM` clause as an `RDDScan`, so on those versions `RDDScan` in the list also converted
+it. The list remains the way to convert other leaf operators, such as the scan of a Data Source V2
+connector.
## Upgrading to Comet 1.1.0
diff --git a/contributor-guide/memory_management.html b/contributor-guide/memory_management.html
index 2d9ccf9655..80a46fbcf8 100644
--- a/contributor-guide/memory_management.html
+++ b/contributor-guide/memory_management.html
@@ -818,14 +818,17 @@ ignored and the pool is always <code class="docutils literal notranslate"><span
<h2>The pool stack<a class="headerlink" href="#the-pool-stack" title="Link to this heading">#</a></h2>
<p><code class="docutils literal notranslate"><span class="pre">create_memory_pool</span></code> builds a base pool and <code class="docutils literal notranslate"><span class="pre">createPlan</span></code> then wraps it in decorators. Reading from
the inside out, a Comet plan in the default configuration sees:</p>
-<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>[LoggingMemoryPool] <- only when spark.comet.debug.memory=true
- [TaskSharedMemoryPool] <- RAII handle for the per-task registry
- [TrackConsumersPool] <- DataFusion; names the top 10 consumers in error messages
- [CometFairMemoryPool] <- delegates acquire/release to Spark over JNI
+<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>[LoggingMemoryPool] <- only when spark.comet.debug.memory=true
+ [TaskSharedMemoryPool] <- RAII handle for the per-task registry
+ [SpillReplayPool] <- lets a spilled final aggregate read its spill files back
+ [TrackConsumersPool] <- DataFusion; names the top 10 consumers in error messages
+ [CometFairMemoryPool] <- delegates acquire/release to Spark over JNI
</pre></div>
</div>
<p>Each decorator forwards every <code class="docutils literal notranslate"><span class="pre">MemoryPool</span></code> method to its inner pool, so <code class="docutils literal notranslate"><span class="pre">reserved()</span></code> at any level
-reports the base pool’s number.</p>
+reports the base pool’s number. <code class="docutils literal notranslate"><span class="pre">SpillReplayPool</span></code> also turns one kind of refused <code class="docutils literal notranslate"><span class="pre">try_grow</span></code> into a
+<code class="docutils literal notranslate"><span class="pre">grow</span></code>; see
+<a class="reference internal" href="#final-aggregates-reading-their-spill-files-back">Final aggregates reading their spill files back</a>.</p>
<section id="the-unified-pools">
<h3>The unified pools<a class="headerlink" href="#the-unified-pools" title="Link to this heading">#</a></h3>
<p><code class="docutils literal notranslate"><span class="pre">CometUnifiedMemoryPool</span></code> and <code class="docutils literal notranslate"><span class="pre">CometFairMemoryPool</span></code> (<code class="docutils literal notranslate"><span class="pre">unified_pool.rs</span></code>, <code class="docutils literal notranslate"><span class="pre">fair_pool.rs</span></code>) are the
@@ -907,6 +910,18 @@ hands every native plan in a task the same one and drops it when the task comple
covers the whole task, so <code class="docutils literal notranslate"><span class="pre">CometExecIterator.close()</span></code> warns about memory still in use only when
the task’s last open native plan closes.</p>
</section>
+<section id="final-aggregates-reading-their-spill-files-back">
+<h3>Final aggregates reading their spill files back<a class="headerlink" href="#final-aggregates-reading-their-spill-files-back" title="Link to this heading">#</a></h3>
+<p><code class="docutils literal notranslate"><span class="pre">SpillReplayPool</span></code> (<code class="docutils literal notranslate"><span class="pre">spill_replay.rs</span></code>) wraps both Comet pools to work around
+<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/6254">issue #6254</a>. Once one of DataFusion 55’s
+final aggregates has spilled, it reads its spill files back through an aggregate that cannot spill,
+so a refused <code class="docutils literal notranslate"><span class="pre">try_grow</span></code> there fails the task. When the pool refuses such a request,
+<code class="docutils literal notranslate"><span class="pre">SpillReplayPool</span></code> records it with the pool’s <code class="docutils literal notranslate"><span class="pre">grow</span></code> instead, which skips <code class="docutils literal notranslate"><span class="pre">CometFairMemoryPool</span></code>’s
+local checks and carries what Spark does not grant as overcommit. Every other refusal is passed on
+unchanged. <code class="docutils literal notranslate"><span class="pre">spill_replay.rs</span></code> describes how the wrapper recognizes these requests and what that relies
+on in DataFusion. <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/6583">Issue #6583</a> tracks
+removing it.</p>
+</section>
</section>
<section id="how-datafusion-consumes-the-pool">
<h2>How DataFusion consumes the pool<a class="headerlink" href="#how-datafusion-consumes-the-pool" title="Link to this heading">#</a></h2>
@@ -1211,6 +1226,7 @@ is not.</p></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-pool-stack">The pool stack</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#the-unified-pools">The unified pools</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#task-shared-pools-and-their-lifetime">Task-shared pools and their lifetime</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#final-aggregates-reading-their-spill-files-back">Final aggregates reading their spill files back</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#how-datafusion-consumes-the-pool">How DataFusion consumes the pool</a></li>
diff --git a/searchindex.js b/searchindex.js
index fa1e43181b..6fbb02c061 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[57, "id1"]], "%": [[55, "id1"]], "&": [[45, "id1"]], "*": [[55, "id2"]], "+": [[55, "id3"]], "-": [[55, "id4"]], "/": [[55, "id5"]], "1. Format Your Code": [[42, "format-your-code"]], "1. Install Comet": [[63, "install-comet"], [75, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[42, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[63, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[57, "id1"]], "%": [[55, "id1"]], "&": [[45, "id1"]], "*": [[55, "id2"]], "+": [[55, "id3"]], "-": [[55, "id4"]], "/": [[55, "id5"]], "1. Format Your Code": [[42, "format-your-code"]], "1. Install Comet": [[63, "install-comet"], [75, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[42, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[63, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index de7133dae0..6c03af8ae2 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -914,104 +914,108 @@ under the License.
<td><p>When enabled, data from a DataFrame created from an RDD of rows, for example with <code class="docutils literal notranslate"><span class="pre">spark.createDataFrame(rdd,</span> <span class="pre">schema)</span></code>, will be converted to Arrow format.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.typedDataset.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.rowDataSource.enabled</span></code></p></td>
+<td><p>When enabled, data from Data Source V1 relations that are not file-based, such as JDBC tables, will be converted to Arrow format. Spark scans these relations with <code class="docutils literal notranslate"><span class="pre">RowDataSourceScanExec</span></code>.</p></td>
+<td><p>false</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.typedDataset.enabled</span></code></p></td>
<td><p>When enabled, the output of typed Dataset operations, such as <code class="docutils literal notranslate"><span class="pre">map</span></code>, <code class="docutils literal notranslate"><span class="pre">flatMap</span></code>, <code class="docutils literal notranslate"><span class="pre">mapPartitions</span></code> and <code class="docutils literal notranslate"><span class="pre">groupByKey(...).mapGroups</span></code>, will be converted to Arrow format so that the operators above th [...]
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.debug.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.debug.enabled</span></code></p></td>
<td><p>Whether to enable debug mode for Comet. When enabled, Comet will do additional checks for debugging purpose. For example, validating array when importing arrays from JVM at native side. Note that these checks may be expensive in performance and should only be enabled for debugging purpose.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.enabled</span></code></p></td>
<td><p>Whether to enable Comet extension for Spark. When this is turned on, Spark will use Comet to read Parquet data source. Note that to enable native vectorized execution, both this config and <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.enabled</span></code> need to be enabled. It can be overridden by the environment variable <code class="docutils literal notranslate"><span class="pre">ENABLE_COMET</span></code>.</p></td>
<td><p>true</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.aggregate.skipPartial.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.aggregate.skipPartial.enabled</span></code></p></td>
<td><p>Experimental opt-in: let a native partial aggregate stop aggregating once its input looks mostly distinct, and send the rest of the task’s rows to the shuffle unaggregated. Only applies to partial aggregates that feed Comet native shuffle and whose aggregate functions, if any, are all single-argument COUNT. The check starts after the first 100,000 input rows of a task, and once aggregation stops it does not resume, so a task whose keys repeat after a mostly distinct start can shuf [...]
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.columnarToRow.native.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.columnarToRow.native.enabled</span></code></p></td>
<td><p>Whether to enable native columnar to row conversion. When enabled, Comet will use native Rust code to convert Arrow columnar data to Spark UnsafeRow format instead of the JVM implementation. The native conversion carries a fixed JNI cost per batch and is slower than the JVM implementation for small batches.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.enabled</span></code></p></td>
<td><p>Whether to enable Comet native vectorized execution for Spark. This controls whether Spark should convert operators into their Comet counterparts and execute them in native space. Note: each operator is associated with a separate config in the format of <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.<operator_name>.enabled</span></code> at the moment, and both the config and this need to be turned on, in order for the operator to be executed in [...]
<td><p>true</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.forceShuffledHashJoin</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.forceShuffledHashJoin</span></code></p></td>
<td><p>Experimental feature to force Spark to replace SortMergeJoin with ShuffledHashJoin for improved performance. This feature is not stable yet. For more information, refer to the <a class="reference external" href="https://datafusion.apache.org/comet/user-guide/latest/tuning.html">Comet Tuning Guide</a>.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.inMemoryCache.compression.codec</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.inMemoryCache.compression.codec</span></code></p></td>
<td><p>The Arrow IPC compression codec used when Comet’s cache serializer writes cached data. Unlike spark.io.compression.codec, this compresses each Arrow buffer separately rather than the batch as a whole, which is what lets a projected scan decompress only the columns it selected. zstd is the default for footprint rather than speed: it stores cached data in a fraction of the memory, but makes materializing slower, and a read slower in proportion to how many columns it selects. Set to [...]
<td><p>zstd</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.inMemoryCache.compression.zstd.level</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.inMemoryCache.compression.zstd.level</span></code></p></td>
<td><p>The compression level to use when Comet’s cache serializer compresses cached data with zstd. Ignored for other codecs.</p></td>
<td><p>1</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.inMemoryCache.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.inMemoryCache.enabled</span></code></p></td>
<td><p>Whether to enable Comet native scans and fused Spark reads of in-memory cached tables. Requires spark.comet.enabled=true. At startup, this setting also decides whether CometDriverPlugin installs Comet’s cache serializer, which stores cached data in Arrow format. The plugin installs it only if spark.comet.enabled and spark.comet.exec.enabled are also enabled at startup, and only with one of Comet’s shuffle managers while Comet shuffle is enabled. Because spark.sql.cache.serializer [...]
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.join.dynamicFilter.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.join.dynamicFilter.enabled</span></code></p></td>
<td><p>Experimental opt-in: use a completed native hash join build’s key domain to prune eligible native Parquet row groups and filter probe batches before the hash probe. Supports inner joins with one direct signed integer key and one native partition per input, including both Spark build sides. The probe filter remains active when reader pruning is unavailable. Unsupported joins retain their existing execution path. Filters do not cross Spark exchanges or JVM/Arrow boundaries.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.pyarrowUDF.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.pyarrowUDF.enabled</span></code></p></td>
<td><p>Experimental: whether to enable optimized execution of PyArrow UDFs (mapInArrow/mapInPandas). When enabled, Comet passes Arrow columnar data directly to Python UDFs without the intermediate Arrow-to-Row-to-Arrow conversion that Spark normally performs. Disabled by default while the feature stabilizes.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.scalaUDF.codegen.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.scalaUDF.codegen.enabled</span></code></p></td>
<td><p>Whether to route Spark <code class="docutils literal notranslate"><span class="pre">ScalaUDF</span></code> expressions through Comet’s Arrow-direct codegen dispatcher. When enabled, a supported ScalaUDF is compiled into a per-batch kernel that reads and writes Arrow vectors directly from native execution. When disabled, plans containing a ScalaUDF fall back to Spark for the enclosing operator. The same dispatcher backs the regex family (<code class="docutils literal notranslate">< [...]
<td><p>true</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.strictFloatingPoint</span></code></p></td>
<td><p>When enabled, fall back to Spark for floating-point operations that may differ from Spark, such as comparing -0.0 and 0.0. <code class="docutils literal notranslate"><span class="pre">ORDER</span> <span class="pre">BY</span></code>, window ordering and range partitioning keys are unaffected, including floating-point values nested in arrays and structs, because Comet normalizes those comparison keys to match Spark. The exception is a nested key whose type can hold a null element or [...]
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.topK.dynamicFilter.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.topK.dynamicFilter.enabled</span></code></p></td>
<td><p>Experimental opt-in: use a local TopK heap’s improving threshold to prune eligible native Parquet input. Requires spark.comet.exec.topK.fusion.enabled. Supports one direct signed integer sort key. Thresholds are local to each native execution and do not cross Spark exchanges or JVM/Arrow boundaries. Unsupported readers retain ordinary TopK execution.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.topK.fusion.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.topK.fusion.enabled</span></code></p></td>
<td><p>Experimental opt-in: run an eligible local TopK in the same native execution as its Parquet scan. Supports one direct signed integer sort key. This changes the local execution pipeline and can reduce scan/TopK overlap, so it may be slower for some workloads.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.transitionRevert.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.transitionRevert.enabled</span></code></p></td>
<td><p>When enabled, Comet reverts a query stage to Spark row-based execution if the number of columnar-to-row (C2R) transitions in the stage exceeds the configured threshold. This avoids the overhead of repeated format conversions in stages where many operators fall back to row-based execution.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.transitionRevert.maxTransitions</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.transitionRevert.maxTransitions</span></code></p></td>
<td><p>The maximum number of columnar-to-row (C2R) transitions allowed in a single query stage before Comet reverts the entire stage to Spark row-based execution. When columnar shuffle is enabled, each such C2R typically implies a corresponding row-to-columnar conversion to feed back into the columnar shuffle, so each counted C2R is a useful proxy for the conversion overhead in the stage. Set to 0 to revert any stage with transitions. Only effective when spark.comet.exec.transitionRevert [...]
<td><p>2</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.metrics.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.metrics.enabled</span></code></p></td>
<td><p>Whether to enable Comet metrics reporting through Spark’s external monitoring system. When enabled, Comet exposes metrics such as native operators, Spark operators, queries planned, transitions, and acceleration ratio. These metrics can be visualized through tools like Grafana when a metrics sink (e.g., Prometheus) is configured. Disabled by default because Spark plan traversal adds overhead and metrics require a sink to be useful. This config must be set before the SparkSession i [...]
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.metrics.updateInterval</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.metrics.updateInterval</span></code></p></td>
<td><p>The interval in milliseconds to update metrics. If interval is negative, metrics will be updated upon task completion.</p></td>
<td><p>3000</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.nativeLoadRequired</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.nativeLoadRequired</span></code></p></td>
<td><p>Whether to require Comet native library to load successfully when Comet is enabled. If not, Comet will silently fallback to Spark when it fails to load the native lib. Otherwise, an error will be thrown and the Spark job will be aborted.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.operator.DataWritingCommandExec.allowIncompatible</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.operator.DataWritingCommandExec.allowIncompatible</span></code></p></td>
<td><p>Whether to allow incompatibility for operator: DataWritingCommandExec. False by default. Can be overridden with SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE env variable It can be overridden by the environment variable <code class="docutils literal notranslate"><span class="pre">SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE</span></code>.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.operator.WriteFilesExec.allowIncompatible</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.operator.WriteFilesExec.allowIncompatible</span></code></p></td>
<td><p>Whether to allow incompatibility for operator: WriteFilesExec. False by default. Can be overridden with SPARK_COMET_OPERATOR_WRITEFILESEXEC_ALLOWINCOMPATIBLE env variable It can be overridden by the environment variable <code class="docutils literal notranslate"><span class="pre">SPARK_COMET_OPERATOR_WRITEFILESEXEC_ALLOWINCOMPATIBLE</span></code>.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled</span></code></p></td>
<td><p>Whether to convert the output of the leaf operators named in <code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code> to Arrow format. Ranges, in-memory cached tables, RDDs and queries without a FROM clause have <code class="docutils literal notranslate"><span class="pre">spark.comet.convert</span></code> configs of their own. When the list is not set, this converts those four instead, which is deprecated and will [...]
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code></p></td>
-<td><p>A comma-separated list of leaf operators to convert to Arrow format when <code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled</span></code> is true, each named by its Spark class name without the <code class="docutils literal notranslate"><span class="pre">Exec</span></code> suffix. It is for operators that have no <code class="docutils literal notranslate"><span class="pre">spark.comet.convert</span></code> config of their own, such as [...]
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code></p></td>
+<td><p>A comma-separated list of leaf operators to convert to Arrow format when <code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled</span></code> is true, each named by its Spark class name without the <code class="docutils literal notranslate"><span class="pre">Exec</span></code> suffix. It is for operators that have no <code class="docutils literal notranslate"><span class="pre">spark.comet.convert</span></code> config of their own, such as [...]
<td><p></p></td>
</tr>
</tbody>
diff --git a/user-guide/latest/datasources.html b/user-guide/latest/datasources.html
index a53d8e86f1..1d93455a40 100644
--- a/user-guide/latest/datasources.html
+++ b/user-guide/latest/datasources.html
@@ -841,6 +841,8 @@ scan does not read, such as tables cached in Spark’s default format.</p></li>
<code class="docutils literal notranslate"><span class="pre">spark.createDataFrame(rdd,</span> <span class="pre">schema)</span></code>.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.oneRowRelation.enabled</span></code>: the single row that a query without a <code class="docutils literal notranslate"><span class="pre">FROM</span></code>
clause, such as <code class="docutils literal notranslate"><span class="pre">SELECT</span> <span class="pre">1</span></code>, reads.</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.rowDataSource.enabled</span></code>: Data Source V1 relations that are not file-based,
+such as JDBC tables, which Spark scans with <code class="docutils literal notranslate"><span class="pre">RowDataSourceScanExec</span></code>.</p></li>
</ul>
<p>To convert any other leaf operator, such as the scan of a Data Source V2 connector or of a file
format other than Parquet, JSON and CSV, set <code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled=true</span></code> and name the
diff --git a/user-guide/latest/migration-guide.html b/user-guide/latest/migration-guide.html
index 66e1e036bf..fa57723295 100644
--- a/user-guide/latest/migration-guide.html
+++ b/user-guide/latest/migration-guide.html
@@ -839,9 +839,9 @@ key is removed.</p>
<section id="deprecated-and-removed-settings">
<h3>Deprecated and Removed Settings<a class="headerlink" href="#deprecated-and-removed-settings" title="Link to this heading">#</a></h3>
<p>Using <code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled</span></code> and <code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code>
-to convert ranges, in-memory cached tables, RDDs and queries without a <code class="docutils literal notranslate"><span class="pre">FROM</span></code> clause to Arrow is
-deprecated, and will stop working in a future major release. Each of these now has a config of its
-own:</p>
+to convert ranges, in-memory cached tables, RDDs, queries without a <code class="docutils literal notranslate"><span class="pre">FROM</span></code> clause and Data Source V1
+relations that are not file-based to Arrow is deprecated, and will stop working in a future major
+release. Each of these now has a config of its own:</p>
<div class="pst-scrollable-table-container"><table class="table">
<thead>
<tr class="row-odd"><th class="head"><p>Operator in the list</p></th>
@@ -861,15 +861,19 @@ own:</p>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">OneRowRelation</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.oneRowRelation.enabled</span></code></p></td>
</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">RowDataSourceScan</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.rowDataSource.enabled</span></code></p></td>
+</tr>
</tbody>
</table>
</div>
<p><code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code> now defaults to an empty list. When it is not
-set, <code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled=true</span></code> still converts these four, and a list that names one
-of them still converts it, as before. In both cases the driver logs a warning that names the config
-to use instead. Before Spark 4.1, Spark plans a query without a <code class="docutils literal notranslate"><span class="pre">FROM</span></code> clause as an <code class="docutils literal notranslate"><span class="pre">RDDScan</span></code>, so on
-those versions <code class="docutils literal notranslate"><span class="pre">RDDScan</span></code> in the list also converted it. The list remains the way to convert other
-leaf operators, such as the scan of a Data Source V2 connector.</p>
+set, <code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled=true</span></code> still converts the first four, which the list named
+by default, and a list that names any of the five still converts it, as before. In both cases the
+driver logs a warning that names the config to use instead. Before Spark 4.1, Spark plans a query
+without a <code class="docutils literal notranslate"><span class="pre">FROM</span></code> clause as an <code class="docutils literal notranslate"><span class="pre">RDDScan</span></code>, so on those versions <code class="docutils literal notranslate"><span class="pre">RDDScan</span></code> in the list also converted
+it. The list remains the way to convert other leaf operators, such as the scan of a Data Source V2
+connector.</p>
</section>
</section>
<section id="upgrading-to-comet-1-1-0">
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org