You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/04 16:46:39 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by d98fd2494c9a6cc8552efe6aa0a73d5714486b23
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new b696f38e8d Publish built docs triggered by d98fd2494c9a6cc8552efe6aa0a73d5714486b23
b696f38e8d is described below
commit b696f38e8d26aeaa78f1a37d9bb382c325ca3cd2
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Sun Oct 4 16:46:35 2026 +0000
Publish built docs triggered by d98fd2494c9a6cc8552efe6aa0a73d5714486b23
---
...id-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg | 2 +-
_sources/contributor-guide/plugin_overview.md.txt | 2 ++
_sources/user-guide/latest/configs.md.txt | 2 +-
_sources/user-guide/latest/in-memory-cache.md.txt | 27 +++++++++++++++-------
_sources/user-guide/latest/installation.md.txt | 11 +++++----
contributor-guide/plugin_overview.html | 4 +++-
searchindex.js | 2 +-
user-guide/latest/configs.html | 2 +-
user-guide/latest/in-memory-cache.html | 26 ++++++++++++++-------
user-guide/latest/installation.html | 11 +++++----
10 files changed, 58 insertions(+), 31 deletions(-)
diff --git a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
index fbeb3b282c..4d6f389004 100644
--- a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
+++ b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/plugin_overview.md.txt b/_sources/contributor-guide/plugin_overview.md.txt
index 89586bf7b6..072f59e01a 100644
--- a/_sources/contributor-guide/plugin_overview.md.txt
+++ b/_sources/contributor-guide/plugin_overview.md.txt
@@ -48,6 +48,8 @@ and skips the remaining steps. Otherwise it:
- Appends `CometSparkSessionExtensions` to `spark.sql.extensions`, unless it is already listed.
- Sets `spark.sql.cache.serializer` to Comet's `ArrowCachedBatchSerializer` when
`spark.comet.exec.inMemoryCache.enabled=true`, unless the application has chosen a different serializer.
+ It leaves Spark's serializer in place where Comet could not scan its format natively or Kryo would
+ reject it; see [In-Memory Cache](../user-guide/latest/in-memory-cache.md).
- Registers `CometSource` with Spark's metrics system and adds `CometMetricsListener` to
`spark.sql.queryExecutionListeners` when `spark.comet.metrics.enabled=true`.
- Logs a warning for settings that are likely to cause problems, such as an unset `spark.executor.memoryOverhead`.
diff --git a/_sources/user-guide/latest/configs.md.txt b/_sources/user-guide/latest/configs.md.txt
index 4377dafc21..51255e80f1 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -64,7 +64,7 @@ Comet provides the following configuration settings.
| `spark.comet.exec.forceShuffledHashJoin` | Experimental feature to force Spark to replace SortMergeJoin with ShuffledHashJoin for improved performance. This feature is not stable yet. For more information, refer to the [Comet Tuning Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). | false |
| `spark.comet.exec.inMemoryCache.compression.codec` | The Arrow IPC compression codec used when Comet's cache serializer writes cached data. Unlike spark.io.compression.codec, this compresses each Arrow buffer separately rather than the batch as a whole, which is what lets a projected scan decompress only the columns it selected. zstd is the default for footprint rather than speed: it stores cached data in a fraction of the memory, but makes materializing slower, and a read slower in pr [...]
| `spark.comet.exec.inMemoryCache.compression.zstd.level` | The compression level to use when Comet's cache serializer compresses cached data with zstd. Ignored for other codecs. | 1 |
-| `spark.comet.exec.inMemoryCache.enabled` | Whether to enable Comet native execution for in-memory cached tables. Its value at startup also decides whether CometDriverPlugin installs Comet's cache serializer, which stores cached data in Arrow format. The plugin installs it only if spark.comet.enabled and spark.comet.exec.enabled are also enabled at startup. Because spark.sql.cache.serializer is a static config, the cached format is fixed for the application, and disabling this at runtim [...]
+| `spark.comet.exec.inMemoryCache.enabled` | Whether to enable Comet native execution for in-memory cached tables. Its value at startup also decides whether CometDriverPlugin installs Comet's cache serializer, which stores cached data in Arrow format. The plugin installs it only if spark.comet.enabled and spark.comet.exec.enabled are also enabled at startup, and only with one of Comet's shuffle managers while Comet shuffle is enabled. Because spark.sql.cache.serializer is a static config [...]
| `spark.comet.exec.join.dynamicFilter.enabled` | Experimental opt-in: use a completed native hash join build's key domain to prune eligible native Parquet row groups and filter probe batches before the hash probe. Supports inner joins with one direct signed integer key and one native partition per input, including both Spark build sides. The probe filter remains active when reader pruning is unavailable. Unsupported joins retain their existing execution path. Filters do not cross Spark [...]
| `spark.comet.exec.pyarrowUDF.enabled` | Experimental: whether to enable optimized execution of PyArrow UDFs (mapInArrow/mapInPandas). When enabled, Comet passes Arrow columnar data directly to Python UDFs without the intermediate Arrow-to-Row-to-Arrow conversion that Spark normally performs. Disabled by default while the feature stabilizes. | false |
| `spark.comet.exec.scalaUDF.codegen.enabled` | Whether to route Spark `ScalaUDF` expressions through Comet's Arrow-direct codegen dispatcher. When enabled, a supported ScalaUDF is compiled into a per-batch kernel that reads and writes Arrow vectors directly from native execution. When disabled, plans containing a ScalaUDF fall back to Spark for the enclosing operator. The same dispatcher backs the regex family (`rlike`, `regexp_replace`, `split`, `regexp_extract`, `regexp_extract_all`, [...]
diff --git a/_sources/user-guide/latest/in-memory-cache.md.txt b/_sources/user-guide/latest/in-memory-cache.md.txt
index 8de4759957..fb3ba9cc63 100644
--- a/_sources/user-guide/latest/in-memory-cache.md.txt
+++ b/_sources/user-guide/latest/in-memory-cache.md.txt
@@ -34,11 +34,13 @@ $SPARK_HOME/bin/spark-shell \
```
It has to be set before the `SparkContext` starts. Comet's driver plugin chooses
-`spark.sql.cache.serializer` once, while the context is initializing, so a session that started
-with the default goes on using Spark's cache format however the config is set afterwards. The
-plugin installs Comet's serializer only if `spark.comet.enabled` and `spark.comet.exec.enabled`
-are enabled at that point too, because an application that starts without native execution could
-not scan Comet's format natively.
+`spark.sql.cache.serializer` once, while the context is initializing, so an application keeps the
+cache format it started with however the config is set afterwards. The plugin installs Comet's
+serializer only if `spark.comet.enabled` and `spark.comet.exec.enabled` are enabled at that point
+too, because an application that starts without native execution could not scan Comet's format
+natively. It also keeps Spark's format when Comet shuffle is enabled but `spark.shuffle.manager` is
+not one of Comet's shuffle managers, since Comet then disables itself, and when Kryo would reject
+Comet's format; see [Kryo](#kryo).
## What changes when it is enabled
@@ -179,11 +181,20 @@ spark.kryo.registrator=org.apache.comet.CometKryoRegistrator
Comet cannot set `spark.kryo.registrator` for you the way it sets `spark.sql.cache.serializer`:
`KryoSerializer` reads it when `SparkEnv` builds the serializer, which happens before any plugin
-runs. Without it, caching fails with a "Class is not registered" error that does not name this
-feature. Comet's driver plugin warns at startup when it sees Kryo, `registrationRequired`, and no
-registrator. Native broadcast needs the same registrator even when the cache is disabled; see
+runs. Without it, Kryo would reject Comet's cached batch with a "Class is not registered" error
+that does not name this feature. So when Kryo requires registration and has not registered
+Comet's cached batch, Comet's driver plugin does not install Comet's serializer, and caches stay in
+Spark's format. Registrations made another way, through a registrator of the application's own or
+`spark.kryo.classesToRegister`, count as well. The plugin warns at startup when Kryo requires
+registration and has not registered every class `CometKryoRegistrator` registers. An application
+that sets `spark.sql.cache.serializer` to Comet's serializer itself gets the error instead. Native
+broadcast needs the same registrator even when the cache is disabled; see
[Kryo serialization](installation.md#kryo-serialization).
+Spark registers its own cached batch with Kryo only from Spark 4.1, so on earlier versions caching
+in either format under `registrationRequired` needs a registrator. `CometKryoRegistrator` registers
+Spark's cached batch too.
+
## Limitations
Reads that feed **Spark** operators rather than Comet ones are slower than Spark's own cache
diff --git a/_sources/user-guide/latest/installation.md.txt b/_sources/user-guide/latest/installation.md.txt
index 813cc74170..5be4e958d5 100644
--- a/_sources/user-guide/latest/installation.md.txt
+++ b/_sources/user-guide/latest/installation.md.txt
@@ -245,8 +245,9 @@ If the application uses Kryo (`spark.serializer=org.apache.spark.serializer.Kryo
Without it, any query that uses Comet's native broadcast exchange, which is enabled by default,
fails with Kryo's "Class is not registered" error, for example on the first broadcast hash join.
-The [in-memory cache](in-memory-cache.md#kryo) needs the same registrator. Set it before the
-`SparkContext` is created: `KryoSerializer` reads it before Comet's plugin runs, so Comet cannot
-add it for you. `spark.kryo.registrator` accepts a comma-separated list, so an application with
-its own registrator can list both. Comet logs a warning at startup when Kryo requires registration
-and this registrator is missing.
+Comet's [in-memory cache](in-memory-cache.md#kryo) format needs the same registrations, and while
+Kryo has not registered Comet's cached batch, Comet's plugin keeps caches in Spark's format. Set it
+before the `SparkContext` is created: `KryoSerializer` reads it before Comet's plugin runs, so
+Comet cannot add it for you. `spark.kryo.registrator` accepts a comma-separated list, so an
+application with its own registrator can list both. Comet logs a warning at startup when Kryo
+requires registration and has not registered the classes this registrator covers.
diff --git a/contributor-guide/plugin_overview.html b/contributor-guide/plugin_overview.html
index ac691dc8e0..9aae711412 100644
--- a/contributor-guide/plugin_overview.html
+++ b/contributor-guide/plugin_overview.html
@@ -576,7 +576,9 @@ and skips the remaining steps. Otherwise it:</p>
<ul class="simple">
<li><p>Appends <code class="docutils literal notranslate"><span class="pre">CometSparkSessionExtensions</span></code> to <code class="docutils literal notranslate"><span class="pre">spark.sql.extensions</span></code>, unless it is already listed.</p></li>
<li><p>Sets <code class="docutils literal notranslate"><span class="pre">spark.sql.cache.serializer</span></code> to Comet’s <code class="docutils literal notranslate"><span class="pre">ArrowCachedBatchSerializer</span></code> when
-<code class="docutils literal notranslate"><span class="pre">spark.comet.exec.inMemoryCache.enabled=true</span></code>, unless the application has chosen a different serializer.</p></li>
+<code class="docutils literal notranslate"><span class="pre">spark.comet.exec.inMemoryCache.enabled=true</span></code>, unless the application has chosen a different serializer.
+It leaves Spark’s serializer in place where Comet could not scan its format natively or Kryo would
+reject it; see <a class="reference internal" href="../user-guide/latest/in-memory-cache.html"><span class="std std-doc">In-Memory Cache</span></a>.</p></li>
<li><p>Registers <code class="docutils literal notranslate"><span class="pre">CometSource</span></code> with Spark’s metrics system and adds <code class="docutils literal notranslate"><span class="pre">CometMetricsListener</span></code> to
<code class="docutils literal notranslate"><span class="pre">spark.sql.queryExecutionListeners</span></code> when <code class="docutils literal notranslate"><span class="pre">spark.comet.metrics.enabled=true</span></code>.</p></li>
<li><p>Logs a warning for settings that are likely to cause problems, such as an unset <code class="docutils literal notranslate"><span class="pre">spark.executor.memoryOverhead</span></code>.</p></li>
diff --git a/searchindex.js b/searchindex.js
index e7c6f7df37..ddfdd39871 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[57, "id1"]], "%": [[55, "id1"]], "&": [[45, "id1"]], "*": [[55, "id2"]], "+": [[55, "id3"]], "-": [[55, "id4"]], "/": [[55, "id5"]], "1. Format Your Code": [[42, "format-your-code"]], "1. Install Comet": [[63, "install-comet"], [75, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[42, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[63, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[57, "id1"]], "%": [[55, "id1"]], "&": [[45, "id1"]], "*": [[55, "id2"]], "+": [[55, "id3"]], "-": [[55, "id4"]], "/": [[55, "id5"]], "1. Format Your Code": [[42, "format-your-code"]], "1. Install Comet": [[63, "install-comet"], [75, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[42, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[63, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index ee4b7acf11..410cd53c25 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -926,7 +926,7 @@ under the License.
<td><p>1</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.inMemoryCache.enabled</span></code></p></td>
-<td><p>Whether to enable Comet native execution for in-memory cached tables. Its value at startup also decides whether CometDriverPlugin installs Comet’s cache serializer, which stores cached data in Arrow format. The plugin installs it only if spark.comet.enabled and spark.comet.exec.enabled are also enabled at startup. Because spark.sql.cache.serializer is a static config, the cached format is fixed for the application, and disabling this at runtime only sends cached scans back to Spar [...]
+<td><p>Whether to enable Comet native execution for in-memory cached tables. Its value at startup also decides whether CometDriverPlugin installs Comet’s cache serializer, which stores cached data in Arrow format. The plugin installs it only if spark.comet.enabled and spark.comet.exec.enabled are also enabled at startup, and only with one of Comet’s shuffle managers while Comet shuffle is enabled. Because spark.sql.cache.serializer is a static config, the cached format is fixed for the a [...]
<td><p>false</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.comet.exec.join.dynamicFilter.enabled</span></code></p></td>
diff --git a/user-guide/latest/in-memory-cache.html b/user-guide/latest/in-memory-cache.html
index 83c3f14529..fb24085df5 100644
--- a/user-guide/latest/in-memory-cache.html
+++ b/user-guide/latest/in-memory-cache.html
@@ -812,11 +812,13 @@ of Comet’s configuration:</p>
</pre></div>
</div>
<p>It has to be set before the <code class="docutils literal notranslate"><span class="pre">SparkContext</span></code> starts. Comet’s driver plugin chooses
-<code class="docutils literal notranslate"><span class="pre">spark.sql.cache.serializer</span></code> once, while the context is initializing, so a session that started
-with the default goes on using Spark’s cache format however the config is set afterwards. The
-plugin installs Comet’s serializer only if <code class="docutils literal notranslate"><span class="pre">spark.comet.enabled</span></code> and <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.enabled</span></code>
-are enabled at that point too, because an application that starts without native execution could
-not scan Comet’s format natively.</p>
+<code class="docutils literal notranslate"><span class="pre">spark.sql.cache.serializer</span></code> once, while the context is initializing, so an application keeps the
+cache format it started with however the config is set afterwards. The plugin installs Comet’s
+serializer only if <code class="docutils literal notranslate"><span class="pre">spark.comet.enabled</span></code> and <code class="docutils literal notranslate"><span class="pre">spark.comet.exec.enabled</span></code> are enabled at that point
+too, because an application that starts without native execution could not scan Comet’s format
+natively. It also keeps Spark’s format when Comet shuffle is enabled but <code class="docutils literal notranslate"><span class="pre">spark.shuffle.manager</span></code> is
+not one of Comet’s shuffle managers, since Comet then disables itself, and when Kryo would reject
+Comet’s format; see <a class="reference internal" href="#kryo">Kryo</a>.</p>
<section id="what-changes-when-it-is-enabled">
<h2>What changes when it is enabled<a class="headerlink" href="#what-changes-when-it-is-enabled" title="Link to this heading">#</a></h2>
<p>With Comet’s serializer installed as <code class="docutils literal notranslate"><span class="pre">spark.sql.cache.serializer</span></code>:</p>
@@ -1027,10 +1029,18 @@ comparison with Spark’s own cache format. That comparison is under <a class="r
</div>
<p>Comet cannot set <code class="docutils literal notranslate"><span class="pre">spark.kryo.registrator</span></code> for you the way it sets <code class="docutils literal notranslate"><span class="pre">spark.sql.cache.serializer</span></code>:
<code class="docutils literal notranslate"><span class="pre">KryoSerializer</span></code> reads it when <code class="docutils literal notranslate"><span class="pre">SparkEnv</span></code> builds the serializer, which happens before any plugin
-runs. Without it, caching fails with a “Class is not registered” error that does not name this
-feature. Comet’s driver plugin warns at startup when it sees Kryo, <code class="docutils literal notranslate"><span class="pre">registrationRequired</span></code>, and no
-registrator. Native broadcast needs the same registrator even when the cache is disabled; see
+runs. Without it, Kryo would reject Comet’s cached batch with a “Class is not registered” error
+that does not name this feature. So when Kryo requires registration and has not registered
+Comet’s cached batch, Comet’s driver plugin does not install Comet’s serializer, and caches stay in
+Spark’s format. Registrations made another way, through a registrator of the application’s own or
+<code class="docutils literal notranslate"><span class="pre">spark.kryo.classesToRegister</span></code>, count as well. The plugin warns at startup when Kryo requires
+registration and has not registered every class <code class="docutils literal notranslate"><span class="pre">CometKryoRegistrator</span></code> registers. An application
+that sets <code class="docutils literal notranslate"><span class="pre">spark.sql.cache.serializer</span></code> to Comet’s serializer itself gets the error instead. Native
+broadcast needs the same registrator even when the cache is disabled; see
<a class="reference internal" href="installation.html#kryo-serialization"><span class="std std-ref">Kryo serialization</span></a>.</p>
+<p>Spark registers its own cached batch with Kryo only from Spark 4.1, so on earlier versions caching
+in either format under <code class="docutils literal notranslate"><span class="pre">registrationRequired</span></code> needs a registrator. <code class="docutils literal notranslate"><span class="pre">CometKryoRegistrator</span></code> registers
+Spark’s cached batch too.</p>
</section>
<section id="limitations">
<h2>Limitations<a class="headerlink" href="#limitations" title="Link to this heading">#</a></h2>
diff --git a/user-guide/latest/installation.html b/user-guide/latest/installation.html
index ed46210182..71734e676d 100644
--- a/user-guide/latest/installation.html
+++ b/user-guide/latest/installation.html
@@ -1041,11 +1041,12 @@ allocation for native execution. See <a class="reference internal" href="tuning/
</div>
<p>Without it, any query that uses Comet’s native broadcast exchange, which is enabled by default,
fails with Kryo’s “Class is not registered” error, for example on the first broadcast hash join.
-The <a class="reference internal" href="in-memory-cache.html#kryo"><span class="std std-ref">in-memory cache</span></a> needs the same registrator. Set it before the
-<code class="docutils literal notranslate"><span class="pre">SparkContext</span></code> is created: <code class="docutils literal notranslate"><span class="pre">KryoSerializer</span></code> reads it before Comet’s plugin runs, so Comet cannot
-add it for you. <code class="docutils literal notranslate"><span class="pre">spark.kryo.registrator</span></code> accepts a comma-separated list, so an application with
-its own registrator can list both. Comet logs a warning at startup when Kryo requires registration
-and this registrator is missing.</p>
+Comet’s <a class="reference internal" href="in-memory-cache.html#kryo"><span class="std std-ref">in-memory cache</span></a> format needs the same registrations, and while
+Kryo has not registered Comet’s cached batch, Comet’s plugin keeps caches in Spark’s format. Set it
+before the <code class="docutils literal notranslate"><span class="pre">SparkContext</span></code> is created: <code class="docutils literal notranslate"><span class="pre">KryoSerializer</span></code> reads it before Comet’s plugin runs, so
+Comet cannot add it for you. <code class="docutils literal notranslate"><span class="pre">spark.kryo.registrator</span></code> accepts a comma-separated list, so an
+application with its own registrator can list both. Comet logs a warning at startup when Kryo
+requires registration and has not registered the classes this registrator covers.</p>
</section>
</section>
</section>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org