You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/05/04 15:15:28 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by d12c5c9cd6e807f93f46d8d543f8dbaec6d83a39
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new e8bae6ca2 Publish built docs triggered by d12c5c9cd6e807f93f46d8d543f8dbaec6d83a39
e8bae6ca2 is described below
commit e8bae6ca23714469646c000c9fdda501737fd93a
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Mon May 4 15:15:24 2026 +0000
Publish built docs triggered by d12c5c9cd6e807f93f46d8d543f8dbaec6d83a39
---
_sources/user-guide/latest/compatibility/scans.md.txt | 9 +++++++++
.../user-guide/latest/compatibility/spark-versions.md.txt | 11 +++++++++++
searchindex.js | 2 +-
user-guide/latest/compatibility/scans.html | 9 +++++++++
user-guide/latest/compatibility/spark-versions.html | 13 +++++++++++++
5 files changed, 43 insertions(+), 1 deletion(-)
diff --git a/_sources/user-guide/latest/compatibility/scans.md.txt b/_sources/user-guide/latest/compatibility/scans.md.txt
index 27ed20c19..d68c59d56 100644
--- a/_sources/user-guide/latest/compatibility/scans.md.txt
+++ b/_sources/user-guide/latest/compatibility/scans.md.txt
@@ -57,6 +57,15 @@ The following shared limitation may produce incorrect results without falling ba
written using the Proleptic Gregorian calendar. This may produce incorrect results for dates before
October 15, 1582.
+The following shared limitation raises an error at scan time rather than falling back to Spark:
+
+- Invalid UTF-8 bytes in `STRING` columns. Spark permits arbitrary byte sequences in a `STRING`
+ column (for example from `CAST(X'C1' AS STRING)`), but Comet's native execution path is built on
+ Arrow, whose string type is strictly UTF-8. Reading a Parquet file whose `STRING` column contains
+ non-UTF-8 bytes fails with `Parquet error: encountered non UTF-8 data`. Disable Comet for the
+ query, or cast the column to `BINARY` before persisting, if you need to preserve non-UTF-8 bytes.
+ See [#4121](https://github.com/apache/datafusion-comet/issues/4121).
+
## `native_datafusion` Limitations
The `native_datafusion` scan has some additional limitations, mostly related to Parquet metadata. All of these
diff --git a/_sources/user-guide/latest/compatibility/spark-versions.md.txt b/_sources/user-guide/latest/compatibility/spark-versions.md.txt
index 6569c3540..115b1595b 100644
--- a/_sources/user-guide/latest/compatibility/spark-versions.md.txt
+++ b/_sources/user-guide/latest/compatibility/spark-versions.md.txt
@@ -51,6 +51,17 @@ Spark 4.1 support is experimental and intended for development and testing only.
in production.
```
+### Known Limitations
+
+- **`NullType` columns in Parquet files**
+ ([#4199](https://github.com/apache/datafusion-comet/issues/4199)): Spark encodes a `NullType`
+ column as a Parquet `BOOLEAN` physical type annotated with `LogicalType::Unknown`. The Rust
+ `parquet` crate that Comet depends on accepts `Unknown` only when paired with `INT32` and rejects
+ any other physical type with `Parquet error: Cannot annotate Unknown from BOOLEAN for field '<name>'`.
+ Any attempt to read a Parquet file that contains a `NullType` column fails at decode time before
+ Comet's scan runs. Workaround: project the column away, cast it to a concrete type before
+ persisting, or read the file with Comet disabled for that query.
+
## Spark 4.2 (Experimental)
Spark 4.2.0-preview4 is provided as experimental support with Java 17 and Scala 2.13.
diff --git a/searchindex.js b/searchindex.js
index 6221cdc3f..857c11524 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Format Your Code": [[14, "format-your-code"]], "1. Install Comet": [[16, "install-comet"], [24, "install-comet"]], "1. Native Operators (nativeExecs map)": [[4, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[14, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[16, "clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[24, "clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[4, "sink-operators-sinks-m [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Format Your Code": [[14, "format-your-code"]], "1. Install Comet": [[16, "install-comet"], [24, "install-comet"]], "1. Native Operators (nativeExecs map)": [[4, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[14, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[16, "clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[24, "clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[4, "sink-operators-sinks-m [...]
\ No newline at end of file
diff --git a/user-guide/latest/compatibility/scans.html b/user-guide/latest/compatibility/scans.html
index 2c6687e8e..171fec1df 100644
--- a/user-guide/latest/compatibility/scans.html
+++ b/user-guide/latest/compatibility/scans.html
@@ -519,6 +519,15 @@ Spark 3.0 (which used a hybrid Julian/Gregorian calendar), dates/timestamps will
written using the Proleptic Gregorian calendar. This may produce incorrect results for dates before
October 15, 1582.</p></li>
</ul>
+<p>The following shared limitation raises an error at scan time rather than falling back to Spark:</p>
+<ul class="simple">
+<li><p>Invalid UTF-8 bytes in <code class="docutils literal notranslate"><span class="pre">STRING</span></code> columns. Spark permits arbitrary byte sequences in a <code class="docutils literal notranslate"><span class="pre">STRING</span></code>
+column (for example from <code class="docutils literal notranslate"><span class="pre">CAST(X'C1'</span> <span class="pre">AS</span> <span class="pre">STRING)</span></code>), but Comet’s native execution path is built on
+Arrow, whose string type is strictly UTF-8. Reading a Parquet file whose <code class="docutils literal notranslate"><span class="pre">STRING</span></code> column contains
+non-UTF-8 bytes fails with <code class="docutils literal notranslate"><span class="pre">Parquet</span> <span class="pre">error:</span> <span class="pre">encountered</span> <span class="pre">non</span> <span class="pre">UTF-8</span> <span class="pre">data</span></code>. Disable Comet for the
+query, or cast the column to <code class="docutils literal notranslate"><span class="pre">BINARY</span></code> before persisting, if you need to preserve non-UTF-8 bytes.
+See <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4121">#4121</a>.</p></li>
+</ul>
</section>
<section id="native-datafusion-limitations">
<h2><code class="docutils literal notranslate"><span class="pre">native_datafusion</span></code> Limitations<a class="headerlink" href="#native-datafusion-limitations" title="Link to this heading">#</a></h2>
diff --git a/user-guide/latest/compatibility/spark-versions.html b/user-guide/latest/compatibility/spark-versions.html
index f49571b47..6637339da 100644
--- a/user-guide/latest/compatibility/spark-versions.html
+++ b/user-guide/latest/compatibility/spark-versions.html
@@ -503,6 +503,19 @@ support. Non-default collated strings are not yet supported by Comet and will fa
<p>Spark 4.1 support is experimental and intended for development and testing only. It should not be used
in production.</p>
</div>
+<section id="id1">
+<h3>Known Limitations<a class="headerlink" href="#id1" title="Link to this heading">#</a></h3>
+<ul class="simple">
+<li><p><strong><code class="docutils literal notranslate"><span class="pre">NullType</span></code> columns in Parquet files</strong>
+(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4199">#4199</a>): Spark encodes a <code class="docutils literal notranslate"><span class="pre">NullType</span></code>
+column as a Parquet <code class="docutils literal notranslate"><span class="pre">BOOLEAN</span></code> physical type annotated with <code class="docutils literal notranslate"><span class="pre">LogicalType::Unknown</span></code>. The Rust
+<code class="docutils literal notranslate"><span class="pre">parquet</span></code> crate that Comet depends on accepts <code class="docutils literal notranslate"><span class="pre">Unknown</span></code> only when paired with <code class="docutils literal notranslate"><span class="pre">INT32</span></code> and rejects
+any other physical type with <code class="docutils literal notranslate"><span class="pre">Parquet</span> <span class="pre">error:</span> <span class="pre">Cannot</span> <span class="pre">annotate</span> <span class="pre">Unknown</span> <span class="pre">from</span> <span class="pre">BOOLEAN</span> <span class="pre">for</span> <span class="pre">field</span> <span class="pre">'<name>'</span></code>.
+Any attempt to read a Parquet file that contains a <code class="docutils literal notranslate"><span class="pre">NullType</span></code> column fails at decode time before
+Comet’s scan runs. Workaround: project the column away, cast it to a concrete type before
+persisting, or read the file with Comet disabled for that query.</p></li>
+</ul>
+</section>
</section>
<section id="spark-4-2-experimental">
<h2>Spark 4.2 (Experimental)<a class="headerlink" href="#spark-4-2-experimental" title="Link to this heading">#</a></h2>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org