You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/05/28 23:01:54 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by 850e684fd40460edb2e07e4259f25076d4303cfc
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 8fe782511e Publish built docs triggered by 850e684fd40460edb2e07e4259f25076d4303cfc
8fe782511e is described below
commit 8fe782511ee623bec19d9befc5ce8f2e6ad36975
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Thu May 28 23:01:50 2026 +0000
Publish built docs triggered by 850e684fd40460edb2e07e4259f25076d4303cfc
---
.../spark_expressions_support.md.txt | 11 +++++++++++
contributor-guide/spark_expressions_support.html | 19 ++++++++++++++++++-
searchindex.js | 2 +-
3 files changed, 30 insertions(+), 2 deletions(-)
diff --git a/_sources/contributor-guide/spark_expressions_support.md.txt b/_sources/contributor-guide/spark_expressions_support.md.txt
index 9de592c988..2e8be9d966 100644
--- a/_sources/contributor-guide/spark_expressions_support.md.txt
+++ b/_sources/contributor-guide/spark_expressions_support.md.txt
@@ -205,6 +205,17 @@
- [ ] binary
- [ ] boolean
- [x] cast
+ - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8 modulo `Cast.canUpCast` refactored to delegate to `UpCastRule.canUpCast`.
+ - Spark 3.5.8 (audited 2026-05-27): baseline. `Cast(child, dataType, timeZoneId, evalMode)`; eval modes are `LEGACY`, `ANSI`, `TRY`. The legacy `Cast.canCast` matrix and the `Cast.canAnsiCast` matrix decide acceptance per type pair. Comet routes via `CometCast` (`spark/src/main/scala/org/apache/comet/expressions/CometCast.scala`) using a per-source-type support matrix that returns `Compatible`, `Incompatible(reason)`, or `Unsupported(reason)`; literal children are short-circuited to `C [...]
+ - Spark 4.0.1 (audited 2026-05-27): `VariantType` added; `StringType` literals replaced with `_: StringType` to accommodate collated strings. `(TimestampType, ByteType|ShortType|IntegerType)` added to `canAnsiCast`. `NullIntolerant` -> `nullIntolerant: Boolean` refactor. New `ToPrettyString.BinaryFormatter` semantics for `Binary -> String` are replicated natively via `spark_binary_formatter`. Numeric-to-numeric matrix unchanged.
+ - Spark 4.1.1 (audited 2026-05-27): `TimeType` added; many `TimeType` arms in `canCast`/`canAnsiCast`. Geospatial `GeographyType` / `GeometryType` types added with their own conversion rules. Numeric-to-numeric matrix unchanged.
+ - Known divergences and gaps:
+ - `CAST(<binary> AS STRING)` uses `unsafe { String::from_utf8_unchecked }` in native code, which is undefined behaviour for non-UTF8 inputs (https://github.com/apache/datafusion-comet/issues/4488).
+ - Spark 4.0 collated `StringType` is not explicitly guarded; pattern equality is expected to keep collated-string casts falling back, but there is no test (https://github.com/apache/datafusion-comet/issues/4489; umbrella #2190).
+ - Spark 4.1 `TimeType` casts have no explicit `Unsupported` arm; they fall back implicitly but do not appear in the auto-generated compatibility doc (https://github.com/apache/datafusion-comet/issues/4490).
+ - `CAST(<map> AS <map>)` falls back to Spark even though native `cast_map_to_map` exists (https://github.com/apache/datafusion-comet/issues/4491).
+ - `spark.sql.legacy.castComplexTypesToString.enabled=true` is not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4492).
+ - `CAST(<float|double> AS DECIMAL)` rounding may differ from Spark (`Incompatible`, gated by `spark.comet.expression.Cast.allowIncompatible`, tracked at https://github.com/apache/datafusion-comet/issues/1371).
- [ ] date
- [ ] decimal
- [ ] double
diff --git a/contributor-guide/spark_expressions_support.html b/contributor-guide/spark_expressions_support.html
index 60d5ce05ab..e8ff26bfe0 100644
--- a/contributor-guide/spark_expressions_support.html
+++ b/contributor-guide/spark_expressions_support.html
@@ -737,7 +737,24 @@ per-function details are documented in the
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> bigint</p></li>
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> binary</p></li>
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> boolean</p></li>
-<li class="task-list-item"><p><input class="task-list-item-checkbox" checked="checked" disabled="disabled" type="checkbox"> cast</p></li>
+<li class="task-list-item"><p><input class="task-list-item-checkbox" checked="checked" disabled="disabled" type="checkbox"> cast</p>
+<ul>
+<li><p>Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8 modulo <code class="docutils literal notranslate"><span class="pre">Cast.canUpCast</span></code> refactored to delegate to <code class="docutils literal notranslate"><span class="pre">UpCastRule.canUpCast</span></code>.</p></li>
+<li><p>Spark 3.5.8 (audited 2026-05-27): baseline. <code class="docutils literal notranslate"><span class="pre">Cast(child,</span> <span class="pre">dataType,</span> <span class="pre">timeZoneId,</span> <span class="pre">evalMode)</span></code>; eval modes are <code class="docutils literal notranslate"><span class="pre">LEGACY</span></code>, <code class="docutils literal notranslate"><span class="pre">ANSI</span></code>, <code class="docutils literal notranslate"><span class="pre">TRY</s [...]
+<li><p>Spark 4.0.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">VariantType</span></code> added; <code class="docutils literal notranslate"><span class="pre">StringType</span></code> literals replaced with <code class="docutils literal notranslate"><span class="pre">_:</span> <span class="pre">StringType</span></code> to accommodate collated strings. <code class="docutils literal notranslate"><span class="pre">(TimestampType,</span> <span class="pre" [...]
+<li><p>Spark 4.1.1 (audited 2026-05-27): <code class="docutils literal notranslate"><span class="pre">TimeType</span></code> added; many <code class="docutils literal notranslate"><span class="pre">TimeType</span></code> arms in <code class="docutils literal notranslate"><span class="pre">canCast</span></code>/<code class="docutils literal notranslate"><span class="pre">canAnsiCast</span></code>. Geospatial <code class="docutils literal notranslate"><span class="pre">GeographyType</span> [...]
+<li><p>Known divergences and gaps:</p>
+<ul>
+<li><p><code class="docutils literal notranslate"><span class="pre">CAST(<binary></span> <span class="pre">AS</span> <span class="pre">STRING)</span></code> uses <code class="docutils literal notranslate"><span class="pre">unsafe</span> <span class="pre">{</span> <span class="pre">String::from_utf8_unchecked</span> <span class="pre">}</span></code> in native code, which is undefined behaviour for non-UTF8 inputs (https://github.com/apache/datafusion-comet/issues/4488).</p></li>
+<li><p>Spark 4.0 collated <code class="docutils literal notranslate"><span class="pre">StringType</span></code> is not explicitly guarded; pattern equality is expected to keep collated-string casts falling back, but there is no test (https://github.com/apache/datafusion-comet/issues/4489; umbrella #2190).</p></li>
+<li><p>Spark 4.1 <code class="docutils literal notranslate"><span class="pre">TimeType</span></code> casts have no explicit <code class="docutils literal notranslate"><span class="pre">Unsupported</span></code> arm; they fall back implicitly but do not appear in the auto-generated compatibility doc (https://github.com/apache/datafusion-comet/issues/4490).</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">CAST(<map></span> <span class="pre">AS</span> <span class="pre"><map>)</span></code> falls back to Spark even though native <code class="docutils literal notranslate"><span class="pre">cast_map_to_map</span></code> exists (https://github.com/apache/datafusion-comet/issues/4491).</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">spark.sql.legacy.castComplexTypesToString.enabled=true</span></code> is not honoured by Comet (https://github.com/apache/datafusion-comet/issues/4492).</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">CAST(<float|double></span> <span class="pre">AS</span> <span class="pre">DECIMAL)</span></code> rounding may differ from Spark (<code class="docutils literal notranslate"><span class="pre">Incompatible</span></code>, gated by <code class="docutils literal notranslate"><span class="pre">spark.comet.expression.Cast.allowIncompatible</span></code>, tracked at https://github.com/apache/datafusion-comet/issues/1371).</p></li>
+</ul>
+</li>
+</ul>
+</li>
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> date</p></li>
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> decimal</p></li>
<li class="task-list-item"><p><input class="task-list-item-checkbox" disabled="disabled" type="checkbox"> double</p></li>
diff --git a/searchindex.js b/searchindex.js
index db4da7da9c..fa7a6a3ed5 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Format Your Code": [[34, "format-your-code"]], "1. Install Comet": [[36, "install-comet"], [45, "install-comet"]], "1. Native Operators (nativeExecs map)": [[23, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[34, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[36, "clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[45, "clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[23, "sink-operators-sinks [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Format Your Code": [[34, "format-your-code"]], "1. Install Comet": [[36, "install-comet"], [45, "install-comet"]], "1. Native Operators (nativeExecs map)": [[23, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[34, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[36, "clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[45, "clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[23, "sink-operators-sinks [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org