You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/04 21:21:21 UTC

(datafusion-comet) branch asf-site updated: Publish built docs triggered by bdbc4605d6f5814d9f1c9809c6f04d7419d67cc3

This is an automated email from the ASF dual-hosted git repository.

github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git


The following commit(s) were added to refs/heads/asf-site by this push:
     new 66ba7972c9 Publish built docs triggered by bdbc4605d6f5814d9f1c9809c6f04d7419d67cc3
66ba7972c9 is described below

commit 66ba7972c93503509e7bd9f1a74f604291d0fe3a
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Sun Oct 4 21:21:16 2026 +0000

    Publish built docs triggered by bdbc4605d6f5814d9f1c9809c6f04d7419d67cc3
---
 ...id-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg |  2 +-
 _sources/user-guide/latest/configs.md.txt          |  8 +++-
 _sources/user-guide/latest/datasources.md.txt      | 25 +++++++++--
 _sources/user-guide/latest/migration-guide.md.txt  | 23 +++++++++++
 searchindex.js                                     |  2 +-
 user-guide/latest/configs.html                     | 24 +++++++++--
 user-guide/latest/datasources.html                 | 26 ++++++++++--
 user-guide/latest/migration-guide.html             | 48 ++++++++++++++++++++--
 8 files changed, 139 insertions(+), 19 deletions(-)

diff --git a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
index d9e21262f1..d798159cb4 100644
--- a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
+++ b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
diff --git a/_sources/user-guide/latest/configs.md.txt b/_sources/user-guide/latest/configs.md.txt
index 747d4109ff..bd253d9bf0 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -54,8 +54,12 @@ Comet provides the following configuration settings.
 |--------|-------------|---------------|
 | `spark.comet.caseConversion.enabled` | Java uses locale-specific rules when converting strings to upper or lower case and Rust does not, so we disable upper and lower by default. | false |
 | `spark.comet.convert.csv.enabled` | When enabled, data from Spark (non-native) CSV v1 and v2 scans will be converted to Arrow format. | false |
+| `spark.comet.convert.inMemoryCache.enabled` | When enabled, data from in-memory cached tables that Comet's native cache scan does not read (see `spark.comet.exec.inMemoryCache.enabled`), such as tables cached in Spark's default format, will be converted to Arrow format. | false |
 | `spark.comet.convert.json.enabled` | When enabled, data from Spark (non-native) JSON v1 and v2 scans will be converted to Arrow format. | false |
+| `spark.comet.convert.oneRowRelation.enabled` | When enabled, the single row that a query without a FROM clause, such as `SELECT 1`, reads will be converted to Arrow format. | false |
 | `spark.comet.convert.parquet.enabled` | When enabled, data from Spark (non-native) Parquet v1 and v2 scans will be converted to Arrow format. | false |
+| `spark.comet.convert.range.enabled` | When enabled, the rows of `spark.range` and SQL `range()` that Comet does not generate natively (see `spark.comet.exec.range.enabled`) will be converted to Arrow format. | false |
+| `spark.comet.convert.rdd.enabled` | When enabled, data from a DataFrame created from an RDD of rows, for example with `spark.createDataFrame(rdd, schema)`, will be converted to Arrow format. | false |
 | `spark.comet.debug.enabled` | Whether to enable debug mode for Comet. When enabled, Comet will do additional checks for debugging purpose. For example, validating array when importing arrays from JVM at native side. Note that these checks may be expensive in performance and should only be enabled for debugging purpose. | false |
 | `spark.comet.enabled` | Whether to enable Comet extension for Spark. When this is turned on, Spark will use Comet to read Parquet data source. Note that to enable native vectorized execution, both this config and `spark.comet.exec.enabled` need to be enabled. It can be overridden by the environment variable `ENABLE_COMET`. | true |
 | `spark.comet.exec.aggregate.skipPartial.enabled` | Experimental opt-in: let a native partial aggregate stop aggregating once its input looks mostly distinct, and send the rest of the task's rows to the shuffle unaggregated. Only applies to partial aggregates that feed Comet native shuffle and whose aggregate functions, if any, are all single-argument COUNT. The check starts after the first 100,000 input rows of a task, and once aggregation stops it does not resume, so a task whose keys [...]
@@ -78,8 +82,8 @@ Comet provides the following configuration settings.
 | `spark.comet.nativeLoadRequired` | Whether to require Comet native library to load successfully when Comet is enabled. If not, Comet will silently fallback to Spark when it fails to load the native lib. Otherwise, an error will be thrown and the Spark job will be aborted. | false |
 | `spark.comet.operator.DataWritingCommandExec.allowIncompatible` | Whether to allow incompatibility for operator: DataWritingCommandExec. False by default. Can be overridden with SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE env variable It can be overridden by the environment variable `SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE`. | false |
 | `spark.comet.operator.WriteFilesExec.allowIncompatible` | Whether to allow incompatibility for operator: WriteFilesExec. False by default. Can be overridden with SPARK_COMET_OPERATOR_WRITEFILESEXEC_ALLOWINCOMPATIBLE env variable It can be overridden by the environment variable `SPARK_COMET_OPERATOR_WRITEFILESEXEC_ALLOWINCOMPATIBLE`. | false |
-| `spark.comet.sparkToColumnar.enabled` | Whether to enable Spark to Arrow columnar conversion. When this is turned on, Comet will convert operators in `spark.comet.sparkToColumnar.supportedOperatorList` into Arrow columnar format before processing. | false |
-| `spark.comet.sparkToColumnar.supportedOperatorList` | A comma-separated list of operators that will be converted to Arrow columnar format when `spark.comet.sparkToColumnar.enabled` is true. | Range,InMemoryTableScan,RDDScan,OneRowRelation |
+| `spark.comet.sparkToColumnar.enabled` | Whether to convert the output of the leaf operators named in `spark.comet.sparkToColumnar.supportedOperatorList` to Arrow format. Ranges, in-memory cached tables, RDDs and queries without a FROM clause have `spark.comet.convert` configs of their own. When the list is not set, this converts those four instead, which is deprecated and will be removed in a future major release. | false |
+| `spark.comet.sparkToColumnar.supportedOperatorList` | A comma-separated list of leaf operators to convert to Arrow format when `spark.comet.sparkToColumnar.enabled` is true, each named by its Spark class name without the `Exec` suffix. It is for operators that have no `spark.comet.convert` config of their own, such as `BatchScan` for a Data Source V2 connector, or `FileSourceScan` for file formats other than Parquet, JSON and CSV. Naming `Range`, `InMemoryTableScan`, `RDDScan` or `OneR [...]
 <!-- prettier-ignore-end -->
 <!--END:CONFIG_TABLE-->
 
diff --git a/_sources/user-guide/latest/datasources.md.txt b/_sources/user-guide/latest/datasources.md.txt
index b03d96b434..2a940ab734 100644
--- a/_sources/user-guide/latest/datasources.md.txt
+++ b/_sources/user-guide/latest/datasources.md.txt
@@ -49,6 +49,25 @@ converted into Arrow format, allowing the Comet pipeline to take over after that
 Comet does not provide a Rust-based JSON scan, but when `spark.comet.convert.json.enabled` is enabled, data is immediately
 converted into Arrow format, allowing the Comet pipeline to take over after that.
 
+### Other Spark inputs
+
+Comet can also convert the output of these Spark inputs to Arrow format, so that the operators
+above them can run in Comet. Each conversion is off by default.
+
+- `spark.comet.convert.range.enabled`: `spark.range` and SQL `range()`, for ranges that Comet does
+  not generate natively with `spark.comet.exec.range.enabled`.
+- `spark.comet.convert.inMemoryCache.enabled`: in-memory cached tables that Comet's native cache
+  scan does not read, such as tables cached in Spark's default format.
+- `spark.comet.convert.rdd.enabled`: a DataFrame created from an RDD of rows, for example with
+  `spark.createDataFrame(rdd, schema)`.
+- `spark.comet.convert.oneRowRelation.enabled`: the single row that a query without a `FROM`
+  clause, such as `SELECT 1`, reads.
+
+To convert any other leaf operator, such as the scan of a Data Source V2 connector or of a file
+format other than Parquet, JSON and CSV, set `spark.comet.sparkToColumnar.enabled=true` and name the
+operator in `spark.comet.sparkToColumnar.supportedOperatorList` by its Spark class name without the
+`Exec` suffix, such as `BatchScan` or `FileSourceScan`.
+
 ### Spark-to-Comet conversion types
 
 Spark-to-Comet conversion supports `ARRAY<STRING>` and `MAP<STRING,STRING>` with binary
@@ -58,10 +77,8 @@ This applies to Spark row and columnar inputs when conversion is enabled for the
 Other array element types, other map key/value types, nested collections, and non-binary
 string collations remain unsupported at this conversion boundary. Source defaults are unchanged.
 
-This includes row-backed `ExistingRDD` inputs when
-`spark.comet.sparkToColumnar.enabled=true` and
-`spark.comet.sparkToColumnar.supportedOperatorList` includes `RDDScan`. Spark still produces
-the RDD rows; conversion lets eligible downstream operators execute in Comet.
+This includes row-backed `ExistingRDD` inputs when `spark.comet.convert.rdd.enabled=true`. Spark
+still produces the RDD rows; conversion lets eligible downstream operators execute in Comet.
 
 ## Data Catalogs
 
diff --git a/_sources/user-guide/latest/migration-guide.md.txt b/_sources/user-guide/latest/migration-guide.md.txt
index b44dbf9cd7..0a9731a2fa 100644
--- a/_sources/user-guide/latest/migration-guide.md.txt
+++ b/_sources/user-guide/latest/migration-guide.md.txt
@@ -57,6 +57,29 @@ Treat setting one of these keys as a temporary measure. If you find you cannot s
 legacy behavior, please open an issue describing your use case so it can be considered before the
 key is removed.
 
+## Upgrading to Comet 1.2.0
+
+### Deprecated and Removed Settings
+
+Using `spark.comet.sparkToColumnar.enabled` and `spark.comet.sparkToColumnar.supportedOperatorList`
+to convert ranges, in-memory cached tables, RDDs and queries without a `FROM` clause to Arrow is
+deprecated, and will stop working in a future major release. Each of these now has a config of its
+own:
+
+| Operator in the list | Config                                       |
+| -------------------- | -------------------------------------------- |
+| `Range`              | `spark.comet.convert.range.enabled`          |
+| `InMemoryTableScan`  | `spark.comet.convert.inMemoryCache.enabled`  |
+| `RDDScan`            | `spark.comet.convert.rdd.enabled`            |
+| `OneRowRelation`     | `spark.comet.convert.oneRowRelation.enabled` |
+
+`spark.comet.sparkToColumnar.supportedOperatorList` now defaults to an empty list. When it is not
+set, `spark.comet.sparkToColumnar.enabled=true` still converts these four, and a list that names one
+of them still converts it, as before. In both cases the driver logs a warning that names the config
+to use instead. Before Spark 4.1, Spark plans a query without a `FROM` clause as an `RDDScan`, so on
+those versions `RDDScan` in the list also converted it. The list remains the way to convert other
+leaf operators, such as the scan of a Data Source V2 connector.
+
 ## Upgrading to Comet 1.1.0
 
 Comet `1.1.0` makes no behavior changes that need a `spark.comet.legacy.*` key. The changes below
diff --git a/searchindex.js b/searchindex.js
index 7c3cf43cae..17be421250 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[57, "id1"]], "%": [[55, "id1"]], "&": [[45, "id1"]], "*": [[55, "id2"]], "+": [[55, "id3"]], "-": [[55, "id4"]], "/": [[55, "id5"]], "1. Format Your Code": [[42, "format-your-code"]], "1. Install Comet": [[63, "install-comet"], [75, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[42, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[63, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[57, "id1"]], "%": [[55, "id1"]], "&": [[45, "id1"]], "*": [[55, "id2"]], "+": [[55, "id3"]], "-": [[55, "id4"]], "/": [[55, "id5"]], "1. Format Your Code": [[42, "format-your-code"]], "1. Install Comet": [[63, "install-comet"], [75, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[42, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[63, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index 61a1d238c3..328eddc4db 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -885,14 +885,30 @@ under the License.
 <td><p>When enabled, data from Spark (non-native) CSV v1 and v2 scans will be converted to Arrow format.</p></td>
 <td><p>false</p></td>
 </tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.json.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.inMemoryCache.enabled</span></code></p></td>
+<td><p>When enabled, data from in-memory cached tables that Comet’s native cache scan does not read (see <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.inMemoryCache.enabled</span></code>), such as tables cached in Spark’s default format, will be converted to Arrow format.</p></td>
+<td><p>false</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.json.enabled</span></code></p></td>
 <td><p>When enabled, data from Spark (non-native) JSON v1 and v2 scans will be converted to Arrow format.</p></td>
 <td><p>false</p></td>
 </tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.oneRowRelation.enabled</span></code></p></td>
+<td><p>When enabled, the single row that a query without a FROM clause, such as <code class="docutils literal notranslate"><span class="pre">SELECT</span> <span class="pre">1</span></code>, reads will be converted to Arrow format.</p></td>
+<td><p>false</p></td>
+</tr>
 <tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.parquet.enabled</span></code></p></td>
 <td><p>When enabled, data from Spark (non-native) Parquet v1 and v2 scans will be converted to Arrow format.</p></td>
 <td><p>false</p></td>
 </tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.range.enabled</span></code></p></td>
+<td><p>When enabled, the rows of <code class="docutils literal notranslate"><span class="pre">spark.range</span></code> and SQL <code class="docutils literal notranslate"><span class="pre">range()</span></code> that Comet does not generate natively (see <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.range.enabled</span></code>) will be converted to Arrow format.</p></td>
+<td><p>false</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.rdd.enabled</span></code></p></td>
+<td><p>When enabled, data from a DataFrame created from an RDD of rows, for example with <code class="docutils literal notranslate"><span class="pre">spark.createDataFrame(rdd,</span> <span class="pre">schema)</span></code>, will be converted to Arrow format.</p></td>
+<td><p>false</p></td>
+</tr>
 <tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.debug.enabled</span></code></p></td>
 <td><p>Whether to enable debug mode for Comet. When enabled, Comet will do additional checks for debugging purpose. For example, validating array when importing arrays from JVM at native side. Note that these checks may be expensive in performance and should only be enabled for debugging purpose.</p></td>
 <td><p>false</p></td>
@@ -982,12 +998,12 @@ under the License.
 <td><p>false</p></td>
 </tr>
 <tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled</span></code></p></td>
-<td><p>Whether to enable Spark to Arrow columnar conversion. When this is turned on, Comet will convert operators in <code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code> into Arrow columnar format before processing.</p></td>
+<td><p>Whether to convert the output of the leaf operators named in <code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code> to Arrow format. Ranges, in-memory cached tables, RDDs and queries without a FROM clause have <code class="docutils literal notranslate"><span class="pre">spark.comet.convert</span></code> configs of their own. When the list is not set, this converts those four instead, which is deprecated and will [...]
 <td><p>false</p></td>
 </tr>
 <tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code></p></td>
-<td><p>A comma-separated list of operators that will be converted to Arrow columnar format when <code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled</span></code> is true.</p></td>
-<td><p>Range,InMemoryTableScan,RDDScan,OneRowRelation</p></td>
+<td><p>A comma-separated list of leaf operators to convert to Arrow format when <code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled</span></code> is true, each named by its Spark class name without the <code class="docutils literal notranslate"><span class="pre">Exec</span></code> suffix. It is for operators that have no <code class="docutils literal notranslate"><span class="pre">spark.comet.convert</span></code> config of their own, such as  [...]
+<td><p></p></td>
 </tr>
 </tbody>
 </table>
diff --git a/user-guide/latest/datasources.html b/user-guide/latest/datasources.html
index 8e3bf4ade1..f5adafb1e8 100644
--- a/user-guide/latest/datasources.html
+++ b/user-guide/latest/datasources.html
@@ -827,6 +827,25 @@ converted into Arrow format, allowing the Comet pipeline to take over after that
 <p>Comet does not provide a Rust-based JSON scan, but when <code class="docutils literal notranslate"><span class="pre">spark.comet.convert.json.enabled</span></code> is enabled, data is immediately
 converted into Arrow format, allowing the Comet pipeline to take over after that.</p>
 </section>
+<section id="other-spark-inputs">
+<h3>Other Spark inputs<a class="headerlink" href="#other-spark-inputs" title="Link to this heading">#</a></h3>
+<p>Comet can also convert the output of these Spark inputs to Arrow format, so that the operators
+above them can run in Comet. Each conversion is off by default.</p>
+<ul class="simple">
+<li><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.range.enabled</span></code>: <code class="docutils literal notranslate"><span class="pre">spark.range</span></code> and SQL <code class="docutils literal notranslate"><span class="pre">range()</span></code>, for ranges that Comet does
+not generate natively with <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.range.enabled</span></code>.</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.inMemoryCache.enabled</span></code>: in-memory cached tables that Comet’s native cache
+scan does not read, such as tables cached in Spark’s default format.</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.rdd.enabled</span></code>: a DataFrame created from an RDD of rows, for example with
+<code class="docutils literal notranslate"><span class="pre">spark.createDataFrame(rdd,</span> <span class="pre">schema)</span></code>.</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.oneRowRelation.enabled</span></code>: the single row that a query without a <code class="docutils literal notranslate"><span class="pre">FROM</span></code>
+clause, such as <code class="docutils literal notranslate"><span class="pre">SELECT</span> <span class="pre">1</span></code>, reads.</p></li>
+</ul>
+<p>To convert any other leaf operator, such as the scan of a Data Source V2 connector or of a file
+format other than Parquet, JSON and CSV, set <code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled=true</span></code> and name the
+operator in <code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code> by its Spark class name without the
+<code class="docutils literal notranslate"><span class="pre">Exec</span></code> suffix, such as <code class="docutils literal notranslate"><span class="pre">BatchScan</span></code> or <code class="docutils literal notranslate"><span class="pre">FileSourceScan</span></code>.</p>
+</section>
 <section id="spark-to-comet-conversion-types">
 <h3>Spark-to-Comet conversion types<a class="headerlink" href="#spark-to-comet-conversion-types" title="Link to this heading">#</a></h3>
 <p>Spark-to-Comet conversion supports <code class="docutils literal notranslate"><span class="pre">ARRAY&lt;STRING&gt;</span></code> and <code class="docutils literal notranslate"><span class="pre">MAP&lt;STRING,STRING&gt;</span></code> with binary
@@ -835,10 +854,8 @@ be null; array elements and map values may also be null. Map keys must be non-nu
 This applies to Spark row and columnar inputs when conversion is enabled for the source.
 Other array element types, other map key/value types, nested collections, and non-binary
 string collations remain unsupported at this conversion boundary. Source defaults are unchanged.</p>
-<p>This includes row-backed <code class="docutils literal notranslate"><span class="pre">ExistingRDD</span></code> inputs when
-<code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled=true</span></code> and
-<code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code> includes <code class="docutils literal notranslate"><span class="pre">RDDScan</span></code>. Spark still produces
-the RDD rows; conversion lets eligible downstream operators execute in Comet.</p>
+<p>This includes row-backed <code class="docutils literal notranslate"><span class="pre">ExistingRDD</span></code> inputs when <code class="docutils literal notranslate"><span class="pre">spark.comet.convert.rdd.enabled=true</span></code>. Spark
+still produces the RDD rows; conversion lets eligible downstream operators execute in Comet.</p>
 </section>
 </section>
 <section id="data-catalogs">
@@ -1378,6 +1395,7 @@ As with S3, standard Hadoop ABFS configurations (the <code class="docutils liter
 <li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#apache-iceberg">Apache Iceberg</a></li>
 <li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#csv">CSV</a></li>
 <li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#json">JSON</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#other-spark-inputs">Other Spark inputs</a></li>
 <li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#spark-to-comet-conversion-types">Spark-to-Comet conversion types</a></li>
 </ul>
 </li>
diff --git a/user-guide/latest/migration-guide.html b/user-guide/latest/migration-guide.html
index eb367e566c..8f5415a304 100644
--- a/user-guide/latest/migration-guide.html
+++ b/user-guide/latest/migration-guide.html
@@ -833,6 +833,44 @@ which point the newer behavior becomes unconditional.</p>
 legacy behavior, please open an issue describing your use case so it can be considered before the
 key is removed.</p>
 </section>
+<section id="upgrading-to-comet-1-2-0">
+<h2>Upgrading to Comet 1.2.0<a class="headerlink" href="#upgrading-to-comet-1-2-0" title="Link to this heading">#</a></h2>
+<section id="deprecated-and-removed-settings">
+<h3>Deprecated and Removed Settings<a class="headerlink" href="#deprecated-and-removed-settings" title="Link to this heading">#</a></h3>
+<p>Using <code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled</span></code> and <code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code>
+to convert ranges, in-memory cached tables, RDDs and queries without a <code class="docutils literal notranslate"><span class="pre">FROM</span></code> clause to Arrow is
+deprecated, and will stop working in a future major release. Each of these now has a config of its
+own:</p>
+<div class="pst-scrollable-table-container"><table class="table">
+<thead>
+<tr class="row-odd"><th class="head"><p>Operator in the list</p></th>
+<th class="head"><p>Config</p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">Range</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.range.enabled</span></code></p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">InMemoryTableScan</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.inMemoryCache.enabled</span></code></p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">RDDScan</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.rdd.enabled</span></code></p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">OneRowRelation</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.convert.oneRowRelation.enabled</span></code></p></td>
+</tr>
+</tbody>
+</table>
+</div>
+<p><code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code> now defaults to an empty list. When it is not
+set, <code class="docutils literal notranslate"><span class="pre">spark.comet.sparkToColumnar.enabled=true</span></code> still converts these four, and a list that names one
+of them still converts it, as before. In both cases the driver logs a warning that names the config
+to use instead. Before Spark 4.1, Spark plans a query without a <code class="docutils literal notranslate"><span class="pre">FROM</span></code> clause as an <code class="docutils literal notranslate"><span class="pre">RDDScan</span></code>, so on
+those versions <code class="docutils literal notranslate"><span class="pre">RDDScan</span></code> in the list also converted it. The list remains the way to convert other
+leaf operators, such as the scan of a Data Source V2 connector.</p>
+</section>
+</section>
 <section id="upgrading-to-comet-1-1-0">
 <h2>Upgrading to Comet 1.1.0<a class="headerlink" href="#upgrading-to-comet-1-1-0" title="Link to this heading">#</a></h2>
 <p>Comet <code class="docutils literal notranslate"><span class="pre">1.1.0</span></code> makes no behavior changes that need a <code class="docutils literal notranslate"><span class="pre">spark.comet.legacy.*</span></code> key. The changes below
@@ -910,8 +948,8 @@ credential chain for a catalog, set
 <code class="docutils literal notranslate"><span class="pre">spark.sql.catalog.&lt;catalog&gt;.s3.comet.credential.webIdentity.enabled=false</span></code>. A table loaded by path
 has no catalog to set that on, so for it set <code class="docutils literal notranslate"><span class="pre">spark.comet.scan.icebergNative.enabled=false</span></code>.</p>
 </section>
-<section id="deprecated-and-removed-settings">
-<h3>Deprecated and Removed Settings<a class="headerlink" href="#deprecated-and-removed-settings" title="Link to this heading">#</a></h3>
+<section id="id1">
+<h3>Deprecated and Removed Settings<a class="headerlink" href="#id1" title="Link to this heading">#</a></h3>
 <p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.memoryPool.fraction</span></code> is deprecated and will be removed in a future major release.
 It was documented as leaving room in <code class="docutils literal notranslate"><span class="pre">spark.memory.offHeap.size</span></code> for the native memory that Comet’s
 memory pools do not track, but it cannot: Spark hands out the whole off-heap pool whatever it is set
@@ -985,12 +1023,16 @@ release, review the release notes for the versions in between.</p>
   <nav class="bd-toc-nav page-toc" aria-labelledby="pst-page-navigation-heading-2">
     <ul class="visible nav section-nav flex-column">
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#legacy-configuration-keys">Legacy Configuration Keys</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#upgrading-to-comet-1-2-0">Upgrading to Comet 1.2.0</a><ul class="nav section-nav flex-column">
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#deprecated-and-removed-settings">Deprecated and Removed Settings</a></li>
+</ul>
+</li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#upgrading-to-comet-1-1-0">Upgrading to Comet 1.1.0</a><ul class="nav section-nav flex-column">
 <li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#settings-that-now-take-effect-as-documented">Settings That Now Take Effect as Documented</a></li>
 <li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#memory-pool-limits">Memory Pool Limits</a></li>
 <li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#conditions-for-enabling-comet">Conditions for Enabling Comet</a></li>
 <li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#native-iceberg-reads-on-eks-with-irsa">Native Iceberg Reads on EKS with IRSA</a></li>
-<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#deprecated-and-removed-settings">Deprecated and Removed Settings</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">Deprecated and Removed Settings</a></li>
 </ul>
 </li>
 <li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#upgrading-to-comet-1-0-0">Upgrading to Comet 1.0.0</a></li>


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org