You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/20 08:06:14 UTC
(datafusion-comet) 01/01: Publish built docs triggered by a48c82da53ebf13f2bea9a4735bde6917211f679
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 0e06131ea5ac96691164562cba83987665e3229c
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Tue Oct 20 08:06:06 2026 +0000
Publish built docs triggered by a48c82da53ebf13f2bea9a4735bde6917211f679
---
...id-09341aca03757d9d3637a2eaff73472d472d0673.svg | 2 +-
.../expression-audits/string_funcs.md.txt | 28 ++++-
.../expressions/spark-3.4/string.md.txt | 8 +-
.../expressions/spark-3.5/string.md.txt | 8 +-
.../expressions/spark-4.0/string.md.txt | 8 +-
.../expressions/spark-4.1/string.md.txt | 8 +-
.../expressions/spark-4.2/string.md.txt | 8 +-
_sources/user-guide/latest/expressions.md.txt | 2 +-
_sources/user-guide/latest/iceberg.md.txt | 4 +-
.../expression-audits/string_funcs.html | 134 ++++++++++++++++++++-
searchindex.js | 2 +-
.../expressions/spark-3.4/string.html | 8 +-
.../expressions/spark-3.5/string.html | 8 +-
.../expressions/spark-4.0/string.html | 8 +-
.../expressions/spark-4.1/string.html | 8 +-
.../expressions/spark-4.2/string.html | 8 +-
user-guide/latest/expressions.html | 2 +-
user-guide/latest/iceberg.html | 4 +-
18 files changed, 225 insertions(+), 33 deletions(-)
diff --git a/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg b/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
index d56cf61be8..0973a5e659 100644
--- a/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
+++ b/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 1014" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-sv [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 1014" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-sv [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/expression-audits/string_funcs.md.txt b/_sources/contributor-guide/expression-audits/string_funcs.md.txt
index 12721470b4..36cd79613f 100644
--- a/_sources/contributor-guide/expression-audits/string_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/string_funcs.md.txt
@@ -169,8 +169,34 @@
## replace
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
-- Spark 3.5.8 (audited 2026-05-27): baseline. `StringReplace(src, search, replace)`; when `search` is empty, Spark returns `src` unchanged (short-circuit on `search.numBytes == 0`). DataFusion `replace` instead inserts `replace` between every character, so `CometStringReplace` reports `Compatible` with a `NativeOptIn` and runs Spark's own generated code inside the Comet pipeline by default. The native DataFusion `replace` is used only when `spark.comet.expression.StringReplace.allowIncom [...]
+- Spark 3.5.8 (audited 2026-05-27): baseline. `StringReplace(src, search, replace)`; when `search` is empty, Spark returns `src` unchanged (short-circuit on `search.numBytes == 0`). Earlier DataFusion versions inserted `replace` between every character, so Comet routed the expression through Spark's generated code by default.
- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.StringReplace.exec`; semantics unchanged for `UTF8_BINARY`. Non-default collations not honoured by Comet ([#4496](https://github.com/apache/datafusion-comet/issues/4496)).
+- DataFusion 55.1.0 (implementation reviewed 2026-10-05, [#5354](https://github.com/apache/datafusion-comet/issues/5354)): an empty search now returns the source unchanged. A direct source column and non-null literal search/replacement use its scalar fast path, which does not broadcast the literals into Arrow `Utf8` arrays. With a replacement column, its general path broadcasts the search literal into an Arrow array, so Comet limits that search to 256 bytes and the configured batch's bro [...]
+- Benchmark (2026-10-05): `CometStringExpressionBenchmark` on Apple M4, macOS 26.7.1, JDK 21.0.12, Spark 4.1, release build, 65,536 rows, one run per route. The same query and Parquet inputs were used with only the default `replace` routing changed. These end-to-end figures include scan and projection; 10% match density is approximate.
+
+ | Match density | Dispatcher ns/row | Native ns/row | Dispatcher M rows/s | Native M rows/s | Native speedup |
+ | ------------- | ----------------: | ------------: | ------------------: | --------------: | -------------: |
+ | 0% | 207.5 | 149.5 | 4.8 | 6.7 | 1.39× |
+ | ~10% | 277.5 | 141.8 | 3.6 | 7.1 | 1.96× |
+ | 100% | 309.9 | 170.5 | 3.2 | 5.9 | 1.82× |
+
+- Column replacement benchmark rerun (2026-10-05): same Apple M4, Spark 4.1 release build and 65,536-row setup, one run. Spark versus native Comet measured 263.5 versus 149.9 ns/row (3.8 versus 6.7 M rows/s) at 0% matches, 225.2 versus 136.9 (4.4 versus 7.3 M rows/s) at about 10%, and 286.5 versus 164.9 (3.5 versus 6.1 M rows/s) at 100%. End-to-end scan and projection times.
+- Column replacement dispatcher baseline (2026-10-05): same machine and setup, with only the default route temporarily forced through the JVM codegen dispatcher. Dispatcher versus native measured 184.4 versus 149.9 ns/row (5.4 versus 6.7 M rows/s, 1.23× native speedup) at 0% matches, 243.3 versus 136.9 (4.1 versus 7.3 M rows/s, 1.78×) at about 10%, and 281.5 versus 164.9 (3.6 versus 6.1 M rows/s, 1.71×) at 100%. The dispatcher and native runs were separate single runs; benchmark timing i [...]
+- Long search literal benchmark (2026-10-05): Apple M4, macOS 26.7.1, JDK 21.0.12, Spark 4.1 release build, 65,536 rows, one run per route and case. Source strings were 2,048 bytes; search literals were 16, 256, or 1,024 ASCII bytes with 0%, about 10%, or 100% match density. Each run includes scan and projection. Dispatcher / native ns per row and native speedup over dispatcher:
+
+ | Search bytes | Match density | Dispatcher ns/row | Native ns/row | Dispatcher M rows/s | Native M rows/s | Native speedup |
+ | -----------: | ------------- | ----------------: | ------------: | ------------------: | --------------: | -------------: |
+ | 16 | 0% | 1,734.6 | 723.6 | 0.6 | 1.4 | 2.40× |
+ | 16 | ~10% | 3,071.5 | 715.8 | 0.3 | 1.4 | 4.29× |
+ | 16 | 100% | 2,667.7 | 715.1 | 0.4 | 1.4 | 3.73× |
+ | 256 | 0% | 2,823.3 | 1,270.9 | 0.4 | 0.8 | 2.22× |
+ | 256 | ~10% | 2,802.2 | 1,280.0 | 0.4 | 0.8 | 2.19× |
+ | 256 | 100% | 2,234.7 | 1,464.3 | 0.4 | 0.7 | 1.53× |
+ | 1,024 | 0% | 1,847.0 | 3,239.8 | 0.5 | 0.3 | 0.57× |
+ | 1,024 | ~10% | 1,808.6 | 3,287.6 | 0.6 | 0.3 | 0.55× |
+ | 1,024 | 100% | 1,195.0 | 4,004.1 | 0.8 | 0.2 | 0.30× |
+
+ These results led to the 256-byte default limit for replacement-column routing. Runs are independent single benchmark invocations, so treat small differences as directional rather than statistically conclusive.
## right
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/string.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/string.md.txt
index 9c2ea5ad3d..450de93c12 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/string.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/string.md.txt
@@ -157,9 +157,13 @@ The following differences from Spark are always present and do not require any a
## StringReplace
-By default, `StringReplace` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StringReplace.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
+The following cases use Comet's native implementation by default:
+
+- Native by default for a direct string column with a non-empty, valid UTF-8 search literal and either a replacement literal (both at most 64 KiB) or a replacement column (search at most 256 bytes and at most 8 MiB when broadcast per batch), under UTF8_BINARY collation. Other cases use the JVM codegen dispatcher by default.
+
+For applicable cases that are not selected for native execution automatically, `StringReplace` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringReplace.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark:
-- Produces different results from Spark when the search string is empty
+- Unrestricted native arguments can change malformed UTF-8 results or Spark's NULL short-circuit behavior, ignore non-default collations, or overflow Arrow offsets when scalars are broadcast
## StringSplit
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/string.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/string.md.txt
index 9c2ea5ad3d..450de93c12 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/string.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/string.md.txt
@@ -157,9 +157,13 @@ The following differences from Spark are always present and do not require any a
## StringReplace
-By default, `StringReplace` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StringReplace.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
+The following cases use Comet's native implementation by default:
+
+- Native by default for a direct string column with a non-empty, valid UTF-8 search literal and either a replacement literal (both at most 64 KiB) or a replacement column (search at most 256 bytes and at most 8 MiB when broadcast per batch), under UTF8_BINARY collation. Other cases use the JVM codegen dispatcher by default.
+
+For applicable cases that are not selected for native execution automatically, `StringReplace` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringReplace.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark:
-- Produces different results from Spark when the search string is empty
+- Unrestricted native arguments can change malformed UTF-8 results or Spark's NULL short-circuit behavior, ignore non-default collations, or overflow Arrow offsets when scalars are broadcast
## StringSplit
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/string.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/string.md.txt
index cf2f0047b1..bda80ae358 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/string.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/string.md.txt
@@ -165,9 +165,13 @@ The following differences from Spark are always present and do not require any a
## StringReplace
-By default, `StringReplace` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StringReplace.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
+The following cases use Comet's native implementation by default:
+
+- Native by default for a direct string column with a non-empty, valid UTF-8 search literal and either a replacement literal (both at most 64 KiB) or a replacement column (search at most 256 bytes and at most 8 MiB when broadcast per batch), under UTF8_BINARY collation. Other cases use the JVM codegen dispatcher by default.
+
+For applicable cases that are not selected for native execution automatically, `StringReplace` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringReplace.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark:
-- Produces different results from Spark when the search string is empty
+- Unrestricted native arguments can change malformed UTF-8 results or Spark's NULL short-circuit behavior, ignore non-default collations, or overflow Arrow offsets when scalars are broadcast
## StringSplit
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/string.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/string.md.txt
index cf2f0047b1..bda80ae358 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/string.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/string.md.txt
@@ -165,9 +165,13 @@ The following differences from Spark are always present and do not require any a
## StringReplace
-By default, `StringReplace` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StringReplace.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
+The following cases use Comet's native implementation by default:
+
+- Native by default for a direct string column with a non-empty, valid UTF-8 search literal and either a replacement literal (both at most 64 KiB) or a replacement column (search at most 256 bytes and at most 8 MiB when broadcast per batch), under UTF8_BINARY collation. Other cases use the JVM codegen dispatcher by default.
+
+For applicable cases that are not selected for native execution automatically, `StringReplace` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringReplace.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark:
-- Produces different results from Spark when the search string is empty
+- Unrestricted native arguments can change malformed UTF-8 results or Spark's NULL short-circuit behavior, ignore non-default collations, or overflow Arrow offsets when scalars are broadcast
## StringSplit
diff --git a/_sources/user-guide/latest/compatibility/expressions/spark-4.2/string.md.txt b/_sources/user-guide/latest/compatibility/expressions/spark-4.2/string.md.txt
index cf2f0047b1..bda80ae358 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.2/string.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.2/string.md.txt
@@ -165,9 +165,13 @@ The following differences from Spark are always present and do not require any a
## StringReplace
-By default, `StringReplace` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set `spark.comet.expression.StringReplace.allowIncompatible=true` to opt into Comet's native implementation instead, which has the following differences from Spark:
+The following cases use Comet's native implementation by default:
+
+- Native by default for a direct string column with a non-empty, valid UTF-8 search literal and either a replacement literal (both at most 64 KiB) or a replacement column (search at most 256 bytes and at most 8 MiB when broadcast per batch), under UTF8_BINARY collation. Other cases use the JVM codegen dispatcher by default.
+
+For applicable cases that are not selected for native execution automatically, `StringReplace` is evaluated in the JVM using Spark's own code-generated implementation (run inside the Comet pipeline) by default. Set `spark.comet.expression.StringReplace.allowIncompatible=true` to explicitly select Comet's native implementation, which has the following differences from Spark:
-- Produces different results from Spark when the search string is empty
+- Unrestricted native arguments can change malformed UTF-8 results or Spark's NULL short-circuit behavior, ignore non-default collations, or overflow Arrow offsets when scalars are broadcast
## StringSplit
diff --git a/_sources/user-guide/latest/expressions.md.txt b/_sources/user-guide/latest/expressions.md.txt
index 221c2b2d2e..dd18a4cdcc 100644
--- a/_sources/user-guide/latest/expressions.md.txt
+++ b/_sources/user-guide/latest/expressions.md.txt
@@ -605,7 +605,7 @@ to Spark ([#2837](https://github.com/apache/datafusion-comet/issues/2837)). Enab
| `regexp_replace` | ✅ | Hybrid | |
| `regexp_substr` | ✅ | — | Rewrites to `nullif(regexp_extract(...), '')`; `regexp_extract` routes through the JVM codegen dispatcher by default |
| `repeat` | ✅ | Native | |
-| `replace` | ✅ | Hybrid | |
+| `replace` | ✅ | Hybrid | Native by default for a direct string column with a non-empty, valid UTF-8 search literal and either a replacement literal (both at most 64 KiB) or a replacement column (search at most 256 bytes and at most 8 MiB when broadcast per batch), under UTF8_BINARY collation; other cases use the JVM codegen dispatcher by default |
| `right` | ✅ | Native | |
| `rpad` | ✅ | Hybrid | String inputs use the native kernel with a column string and literal padding; literal strings and column padding use codegen dispatch. Binary inputs use codegen dispatch. |
| `rtrim` | ✅ | Hybrid | Non-UTF8_BINARY collated input with a trim string routes through the JVM codegen dispatcher; other input runs natively |
diff --git a/_sources/user-guide/latest/iceberg.md.txt b/_sources/user-guide/latest/iceberg.md.txt
index 3e99afff0a..994a7c14d4 100644
--- a/_sources/user-guide/latest/iceberg.md.txt
+++ b/_sources/user-guide/latest/iceberg.md.txt
@@ -196,6 +196,7 @@ The following scenarios will fall back to the JVM Iceberg reader:
projected is read natively)
- Encrypted tables with 192-bit data keys (no AES-192-GCM in the underlying crypto)
- Delete files in a format other than Parquet or Puffin (Avro or ORC positional/equality deletes)
+- Equality deletes keyed on a field nested in a struct
- Tables backed by Avro or ORC data files (only Parquet is accelerated)
- Queries that use `input_file_name()`, `input_file_block_start()` or `input_file_block_length()`
- Scans whose data or delete files span more than one S3 bucket (the native reader uses one
@@ -822,9 +823,6 @@ The physical differences that every native data file has, and why they are safe,
([#7033](https://github.com/apache/datafusion-comet/issues/7033)).
- **`VERSION AS OF` reads of a column whose type was promoted after the snapshot**
([#6993](https://github.com/apache/datafusion-comet/issues/6993)).
-- **An equality delete keyed on a nested field.** Every query the native scan runs on the table
- fails with `field not found`, even one that reads the key
- ([#6782](https://github.com/apache/datafusion-comet/issues/6782)).
- **Column names that only newer Unicode case data folds together**, such as a pair of Vithkuqi
letters, which Unicode 14 added. Queries on such a table fail with a duplicate-field error
([#5540](https://github.com/apache/datafusion-comet/issues/5540)).
diff --git a/contributor-guide/expression-audits/string_funcs.html b/contributor-guide/expression-audits/string_funcs.html
index aeb75b1d91..8bc6b58726 100644
--- a/contributor-guide/expression-audits/string_funcs.html
+++ b/contributor-guide/expression-audits/string_funcs.html
@@ -756,10 +756,140 @@ under the License.
</section>
<section id="replace">
<h2>replace<a class="headerlink" href="#replace" title="Link to this heading">#</a></h2>
-<ul class="simple">
+<ul>
<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.</p></li>
-<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">StringReplace(src,</span> <span class="pre">search,</span> <span class="pre">replace)</span></code>; when <code class="docutils literal notranslate"><span class="pre">search</span></code> is empty, Spark returns <code class="docutils literal notranslate"><span class="pre">src</span></code> unchanged (short-circuit on <code class="docutils literal notranslate"><span class="pre"> [...]
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">StringReplace(src,</span> <span class="pre">search,</span> <span class="pre">replace)</span></code>; when <code class="docutils literal notranslate"><span class="pre">search</span></code> is empty, Spark returns <code class="docutils literal notranslate"><span class="pre">src</span></code> unchanged (short-circuit on <code class="docutils literal notranslate"><span class="pre"> [...]
<li><p>Spark 4.0.1 (audited 2026-05-27): routes through <code class="docutils literal notranslate"><span class="pre">CollationSupport.StringReplace.exec</span></code>; semantics unchanged for <code class="docutils literal notranslate"><span class="pre">UTF8_BINARY</span></code>. Non-default collations not honoured by Comet (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4496">#4496</a>).</p></li>
+<li><p>DataFusion 55.1.0 (implementation reviewed 2026-10-05, <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5354">#5354</a>): an empty search now returns the source unchanged. A direct source column and non-null literal search/replacement use its scalar fast path, which does not broadcast the literals into Arrow <code class="docutils literal notranslate"><span class="pre">Utf8</span></code> arrays. With a replacement column, its general path broadc [...]
+<li><p>Benchmark (2026-10-05): <code class="docutils literal notranslate"><span class="pre">CometStringExpressionBenchmark</span></code> on Apple M4, macOS 26.7.1, JDK 21.0.12, Spark 4.1, release build, 65,536 rows, one run per route. The same query and Parquet inputs were used with only the default <code class="docutils literal notranslate"><span class="pre">replace</span></code> routing changed. These end-to-end figures include scan and projection; 10% match density is approximate.</p>
+<div class="pst-scrollable-table-container"><table class="table">
+<thead>
+<tr class="row-odd"><th class="head"><p>Match density</p></th>
+<th class="head text-right"><p>Dispatcher ns/row</p></th>
+<th class="head text-right"><p>Native ns/row</p></th>
+<th class="head text-right"><p>Dispatcher M rows/s</p></th>
+<th class="head text-right"><p>Native M rows/s</p></th>
+<th class="head text-right"><p>Native speedup</p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="row-even"><td><p>0%</p></td>
+<td class="text-right"><p>207.5</p></td>
+<td class="text-right"><p>149.5</p></td>
+<td class="text-right"><p>4.8</p></td>
+<td class="text-right"><p>6.7</p></td>
+<td class="text-right"><p>1.39×</p></td>
+</tr>
+<tr class="row-odd"><td><p>~10%</p></td>
+<td class="text-right"><p>277.5</p></td>
+<td class="text-right"><p>141.8</p></td>
+<td class="text-right"><p>3.6</p></td>
+<td class="text-right"><p>7.1</p></td>
+<td class="text-right"><p>1.96×</p></td>
+</tr>
+<tr class="row-even"><td><p>100%</p></td>
+<td class="text-right"><p>309.9</p></td>
+<td class="text-right"><p>170.5</p></td>
+<td class="text-right"><p>3.2</p></td>
+<td class="text-right"><p>5.9</p></td>
+<td class="text-right"><p>1.82×</p></td>
+</tr>
+</tbody>
+</table>
+</div>
+</li>
+<li><p>Column replacement benchmark rerun (2026-10-05): same Apple M4, Spark 4.1 release build and 65,536-row setup, one run. Spark versus native Comet measured 263.5 versus 149.9 ns/row (3.8 versus 6.7 M rows/s) at 0% matches, 225.2 versus 136.9 (4.4 versus 7.3 M rows/s) at about 10%, and 286.5 versus 164.9 (3.5 versus 6.1 M rows/s) at 100%. End-to-end scan and projection times.</p></li>
+<li><p>Column replacement dispatcher baseline (2026-10-05): same machine and setup, with only the default route temporarily forced through the JVM codegen dispatcher. Dispatcher versus native measured 184.4 versus 149.9 ns/row (5.4 versus 6.7 M rows/s, 1.23× native speedup) at 0% matches, 243.3 versus 136.9 (4.1 versus 7.3 M rows/s, 1.78×) at about 10%, and 281.5 versus 164.9 (3.6 versus 6.1 M rows/s, 1.71×) at 100%. The dispatcher and native runs were separate single runs; benchmark tim [...]
+<li><p>Long search literal benchmark (2026-10-05): Apple M4, macOS 26.7.1, JDK 21.0.12, Spark 4.1 release build, 65,536 rows, one run per route and case. Source strings were 2,048 bytes; search literals were 16, 256, or 1,024 ASCII bytes with 0%, about 10%, or 100% match density. Each run includes scan and projection. Dispatcher / native ns per row and native speedup over dispatcher:</p>
+<div class="pst-scrollable-table-container"><table class="table">
+<thead>
+<tr class="row-odd"><th class="head text-right"><p>Search bytes</p></th>
+<th class="head"><p>Match density</p></th>
+<th class="head text-right"><p>Dispatcher ns/row</p></th>
+<th class="head text-right"><p>Native ns/row</p></th>
+<th class="head text-right"><p>Dispatcher M rows/s</p></th>
+<th class="head text-right"><p>Native M rows/s</p></th>
+<th class="head text-right"><p>Native speedup</p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="row-even"><td class="text-right"><p>16</p></td>
+<td><p>0%</p></td>
+<td class="text-right"><p>1,734.6</p></td>
+<td class="text-right"><p>723.6</p></td>
+<td class="text-right"><p>0.6</p></td>
+<td class="text-right"><p>1.4</p></td>
+<td class="text-right"><p>2.40×</p></td>
+</tr>
+<tr class="row-odd"><td class="text-right"><p>16</p></td>
+<td><p>~10%</p></td>
+<td class="text-right"><p>3,071.5</p></td>
+<td class="text-right"><p>715.8</p></td>
+<td class="text-right"><p>0.3</p></td>
+<td class="text-right"><p>1.4</p></td>
+<td class="text-right"><p>4.29×</p></td>
+</tr>
+<tr class="row-even"><td class="text-right"><p>16</p></td>
+<td><p>100%</p></td>
+<td class="text-right"><p>2,667.7</p></td>
+<td class="text-right"><p>715.1</p></td>
+<td class="text-right"><p>0.4</p></td>
+<td class="text-right"><p>1.4</p></td>
+<td class="text-right"><p>3.73×</p></td>
+</tr>
+<tr class="row-odd"><td class="text-right"><p>256</p></td>
+<td><p>0%</p></td>
+<td class="text-right"><p>2,823.3</p></td>
+<td class="text-right"><p>1,270.9</p></td>
+<td class="text-right"><p>0.4</p></td>
+<td class="text-right"><p>0.8</p></td>
+<td class="text-right"><p>2.22×</p></td>
+</tr>
+<tr class="row-even"><td class="text-right"><p>256</p></td>
+<td><p>~10%</p></td>
+<td class="text-right"><p>2,802.2</p></td>
+<td class="text-right"><p>1,280.0</p></td>
+<td class="text-right"><p>0.4</p></td>
+<td class="text-right"><p>0.8</p></td>
+<td class="text-right"><p>2.19×</p></td>
+</tr>
+<tr class="row-odd"><td class="text-right"><p>256</p></td>
+<td><p>100%</p></td>
+<td class="text-right"><p>2,234.7</p></td>
+<td class="text-right"><p>1,464.3</p></td>
+<td class="text-right"><p>0.4</p></td>
+<td class="text-right"><p>0.7</p></td>
+<td class="text-right"><p>1.53×</p></td>
+</tr>
+<tr class="row-even"><td class="text-right"><p>1,024</p></td>
+<td><p>0%</p></td>
+<td class="text-right"><p>1,847.0</p></td>
+<td class="text-right"><p>3,239.8</p></td>
+<td class="text-right"><p>0.5</p></td>
+<td class="text-right"><p>0.3</p></td>
+<td class="text-right"><p>0.57×</p></td>
+</tr>
+<tr class="row-odd"><td class="text-right"><p>1,024</p></td>
+<td><p>~10%</p></td>
+<td class="text-right"><p>1,808.6</p></td>
+<td class="text-right"><p>3,287.6</p></td>
+<td class="text-right"><p>0.6</p></td>
+<td class="text-right"><p>0.3</p></td>
+<td class="text-right"><p>0.55×</p></td>
+</tr>
+<tr class="row-even"><td class="text-right"><p>1,024</p></td>
+<td><p>100%</p></td>
+<td class="text-right"><p>1,195.0</p></td>
+<td class="text-right"><p>4,004.1</p></td>
+<td class="text-right"><p>0.8</p></td>
+<td class="text-right"><p>0.2</p></td>
+<td class="text-right"><p>0.30×</p></td>
+</tr>
+</tbody>
+</table>
+</div>
+<p>These results led to the 256-byte default limit for replacement-column routing. Runs are independent single benchmark invocations, so treat small differences as directional rather than statistically conclusive.</p>
+</li>
</ul>
</section>
<section id="right">
diff --git a/searchindex.js b/searchindex.js
index 15e6241436..3a5e6de503 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[63, "id1"]], "%": [[61, "id1"]], "&": [[51, "id1"]], "*": [[61, "id2"]], "+": [[61, "id3"]], "-": [[61, "id4"]], "/": [[61, "id5"]], "1. Format Your Code": [[48, "format-your-code"]], "1. Install Comet": [[70, "install-comet"], [82, "install-comet"]], "1. Native Operators (nativeExecs map)": [[31, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[48, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[70, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[63, "id1"]], "%": [[61, "id1"]], "&": [[51, "id1"]], "*": [[61, "id2"]], "+": [[61, "id3"]], "-": [[61, "id4"]], "/": [[61, "id5"]], "1. Format Your Code": [[48, "format-your-code"]], "1. Install Comet": [[70, "install-comet"], [82, "install-comet"]], "1. Native Operators (nativeExecs map)": [[31, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[48, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[70, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/compatibility/expressions/spark-3.4/string.html b/user-guide/latest/compatibility/expressions/spark-3.4/string.html
index 538e574b04..e2c26ef03c 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.4/string.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.4/string.html
@@ -1009,9 +1009,13 @@ under the License.
</section>
<section id="stringreplace">
<h2>StringReplace<a class="headerlink" href="#stringreplace" title="Link to this heading">#</a></h2>
-<p>By default, <code class="docutils literal notranslate"><span class="pre">StringReplace</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.StringReplace.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
+<p>The following cases use Comet’s native implementation by default:</p>
+<ul class="simple">
+<li><p>Native by default for a direct string column with a non-empty, valid UTF-8 search literal and either a replacement literal (both at most 64 KiB) or a replacement column (search at most 256 bytes and at most 8 MiB when broadcast per batch), under UTF8_BINARY collation. Other cases use the JVM codegen dispatcher by default.</p></li>
+</ul>
+<p>For applicable cases that are not selected for native execution automatically, <code class="docutils literal notranslate"><span class="pre">StringReplace</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline) by default. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.StringReplace.allowIncompatible=true</span></code> to explicitly select Comet’s native implementation, which has the [...]
<ul class="simple">
-<li><p>Produces different results from Spark when the search string is empty</p></li>
+<li><p>Unrestricted native arguments can change malformed UTF-8 results or Spark’s NULL short-circuit behavior, ignore non-default collations, or overflow Arrow offsets when scalars are broadcast</p></li>
</ul>
</section>
<section id="stringsplit">
diff --git a/user-guide/latest/compatibility/expressions/spark-3.5/string.html b/user-guide/latest/compatibility/expressions/spark-3.5/string.html
index f157c047b2..483b57a4d1 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.5/string.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.5/string.html
@@ -1009,9 +1009,13 @@ under the License.
</section>
<section id="stringreplace">
<h2>StringReplace<a class="headerlink" href="#stringreplace" title="Link to this heading">#</a></h2>
-<p>By default, <code class="docutils literal notranslate"><span class="pre">StringReplace</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.StringReplace.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
+<p>The following cases use Comet’s native implementation by default:</p>
+<ul class="simple">
+<li><p>Native by default for a direct string column with a non-empty, valid UTF-8 search literal and either a replacement literal (both at most 64 KiB) or a replacement column (search at most 256 bytes and at most 8 MiB when broadcast per batch), under UTF8_BINARY collation. Other cases use the JVM codegen dispatcher by default.</p></li>
+</ul>
+<p>For applicable cases that are not selected for native execution automatically, <code class="docutils literal notranslate"><span class="pre">StringReplace</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline) by default. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.StringReplace.allowIncompatible=true</span></code> to explicitly select Comet’s native implementation, which has the [...]
<ul class="simple">
-<li><p>Produces different results from Spark when the search string is empty</p></li>
+<li><p>Unrestricted native arguments can change malformed UTF-8 results or Spark’s NULL short-circuit behavior, ignore non-default collations, or overflow Arrow offsets when scalars are broadcast</p></li>
</ul>
</section>
<section id="stringsplit">
diff --git a/user-guide/latest/compatibility/expressions/spark-4.0/string.html b/user-guide/latest/compatibility/expressions/spark-4.0/string.html
index 5551fbeba8..11065bd7b9 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.0/string.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.0/string.html
@@ -1018,9 +1018,13 @@ under the License.
</section>
<section id="stringreplace">
<h2>StringReplace<a class="headerlink" href="#stringreplace" title="Link to this heading">#</a></h2>
-<p>By default, <code class="docutils literal notranslate"><span class="pre">StringReplace</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.StringReplace.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
+<p>The following cases use Comet’s native implementation by default:</p>
+<ul class="simple">
+<li><p>Native by default for a direct string column with a non-empty, valid UTF-8 search literal and either a replacement literal (both at most 64 KiB) or a replacement column (search at most 256 bytes and at most 8 MiB when broadcast per batch), under UTF8_BINARY collation. Other cases use the JVM codegen dispatcher by default.</p></li>
+</ul>
+<p>For applicable cases that are not selected for native execution automatically, <code class="docutils literal notranslate"><span class="pre">StringReplace</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline) by default. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.StringReplace.allowIncompatible=true</span></code> to explicitly select Comet’s native implementation, which has the [...]
<ul class="simple">
-<li><p>Produces different results from Spark when the search string is empty</p></li>
+<li><p>Unrestricted native arguments can change malformed UTF-8 results or Spark’s NULL short-circuit behavior, ignore non-default collations, or overflow Arrow offsets when scalars are broadcast</p></li>
</ul>
</section>
<section id="stringsplit">
diff --git a/user-guide/latest/compatibility/expressions/spark-4.1/string.html b/user-guide/latest/compatibility/expressions/spark-4.1/string.html
index d7c98456a5..e570067905 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.1/string.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.1/string.html
@@ -1018,9 +1018,13 @@ under the License.
</section>
<section id="stringreplace">
<h2>StringReplace<a class="headerlink" href="#stringreplace" title="Link to this heading">#</a></h2>
-<p>By default, <code class="docutils literal notranslate"><span class="pre">StringReplace</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.StringReplace.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
+<p>The following cases use Comet’s native implementation by default:</p>
+<ul class="simple">
+<li><p>Native by default for a direct string column with a non-empty, valid UTF-8 search literal and either a replacement literal (both at most 64 KiB) or a replacement column (search at most 256 bytes and at most 8 MiB when broadcast per batch), under UTF8_BINARY collation. Other cases use the JVM codegen dispatcher by default.</p></li>
+</ul>
+<p>For applicable cases that are not selected for native execution automatically, <code class="docutils literal notranslate"><span class="pre">StringReplace</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline) by default. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.StringReplace.allowIncompatible=true</span></code> to explicitly select Comet’s native implementation, which has the [...]
<ul class="simple">
-<li><p>Produces different results from Spark when the search string is empty</p></li>
+<li><p>Unrestricted native arguments can change malformed UTF-8 results or Spark’s NULL short-circuit behavior, ignore non-default collations, or overflow Arrow offsets when scalars are broadcast</p></li>
</ul>
</section>
<section id="stringsplit">
diff --git a/user-guide/latest/compatibility/expressions/spark-4.2/string.html b/user-guide/latest/compatibility/expressions/spark-4.2/string.html
index ba79da6076..e7da2132e9 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.2/string.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.2/string.html
@@ -1018,9 +1018,13 @@ under the License.
</section>
<section id="stringreplace">
<h2>StringReplace<a class="headerlink" href="#stringreplace" title="Link to this heading">#</a></h2>
-<p>By default, <code class="docutils literal notranslate"><span class="pre">StringReplace</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline), which matches Spark exactly. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.StringReplace.allowIncompatible=true</span></code> to opt into Comet’s native implementation instead, which has the following differences from Spark:</p>
+<p>The following cases use Comet’s native implementation by default:</p>
+<ul class="simple">
+<li><p>Native by default for a direct string column with a non-empty, valid UTF-8 search literal and either a replacement literal (both at most 64 KiB) or a replacement column (search at most 256 bytes and at most 8 MiB when broadcast per batch), under UTF8_BINARY collation. Other cases use the JVM codegen dispatcher by default.</p></li>
+</ul>
+<p>For applicable cases that are not selected for native execution automatically, <code class="docutils literal notranslate"><span class="pre">StringReplace</span></code> is evaluated in the JVM using Spark’s own code-generated implementation (run inside the Comet pipeline) by default. Set <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.StringReplace.allowIncompatible=true</span></code> to explicitly select Comet’s native implementation, which has the [...]
<ul class="simple">
-<li><p>Produces different results from Spark when the search string is empty</p></li>
+<li><p>Unrestricted native arguments can change malformed UTF-8 results or Spark’s NULL short-circuit behavior, ignore non-default collations, or overflow Arrow offsets when scalars are broadcast</p></li>
</ul>
</section>
<section id="stringsplit">
diff --git a/user-guide/latest/expressions.html b/user-guide/latest/expressions.html
index 397be0cb7a..83f47cc34e 100644
--- a/user-guide/latest/expressions.html
+++ b/user-guide/latest/expressions.html
@@ -3216,7 +3216,7 @@ to Spark (<a class="reference external" href="https://github.com/apache/datafusi
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">replace</span></code></p></td>
<td><p>✅</p></td>
<td><p>Hybrid</p></td>
-<td><p></p></td>
+<td><p>Native by default for a direct string column with a non-empty, valid UTF-8 search literal and either a replacement literal (both at most 64 KiB) or a replacement column (search at most 256 bytes and at most 8 MiB when broadcast per batch), under UTF8_BINARY collation; other cases use the JVM codegen dispatcher by default</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">right</span></code></p></td>
<td><p>✅</p></td>
diff --git a/user-guide/latest/iceberg.html b/user-guide/latest/iceberg.html
index 210ff91c5c..79b32d55bc 100644
--- a/user-guide/latest/iceberg.html
+++ b/user-guide/latest/iceberg.html
@@ -1009,6 +1009,7 @@ configure Spark to use a REST catalog with Comet’s native Iceberg scan:</p>
projected is read natively)</p></li>
<li><p>Encrypted tables with 192-bit data keys (no AES-192-GCM in the underlying crypto)</p></li>
<li><p>Delete files in a format other than Parquet or Puffin (Avro or ORC positional/equality deletes)</p></li>
+<li><p>Equality deletes keyed on a field nested in a struct</p></li>
<li><p>Tables backed by Avro or ORC data files (only Parquet is accelerated)</p></li>
<li><p>Queries that use <code class="docutils literal notranslate"><span class="pre">input_file_name()</span></code>, <code class="docutils literal notranslate"><span class="pre">input_file_block_start()</span></code> or <code class="docutils literal notranslate"><span class="pre">input_file_block_length()</span></code></p></li>
<li><p>Scans whose data or delete files span more than one S3 bucket (the native reader uses one
@@ -1666,9 +1667,6 @@ produce without any schema change
(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/7033">#7033</a>).</p></li>
<li><p><strong><code class="docutils literal notranslate"><span class="pre">VERSION</span> <span class="pre">AS</span> <span class="pre">OF</span></code> reads of a column whose type was promoted after the snapshot</strong>
(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/6993">#6993</a>).</p></li>
-<li><p><strong>An equality delete keyed on a nested field.</strong> Every query the native scan runs on the table
-fails with <code class="docutils literal notranslate"><span class="pre">field</span> <span class="pre">not</span> <span class="pre">found</span></code>, even one that reads the key
-(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/6782">#6782</a>).</p></li>
<li><p><strong>Column names that only newer Unicode case data folds together</strong>, such as a pair of Vithkuqi
letters, which Unicode 14 added. Queries on such a table fail with a duplicate-field error
(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5540">#5540</a>).</p></li>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org