You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/24 09:18:07 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by 5d59317630e19d22cc9be5dbdae59ae1b87b4f41
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 0ac0e63cec Publish built docs triggered by 5d59317630e19d22cc9be5dbdae59ae1b87b4f41
0ac0e63cec is described below
commit 0ac0e63cec3e763e3ef4cd214df2e16d90a70fa2
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Thu Sep 24 09:18:02 2026 +0000
Publish built docs triggered by 5d59317630e19d22cc9be5dbdae59ae1b87b4f41
---
.../mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg | 2 +-
.../expression-audits/generator_funcs.md.txt | 4 ++--
_sources/user-guide/latest/compatibility/scans.md.txt | 14 ++++++++++++++
contributor-guide/expression-audits/generator_funcs.html | 4 ++--
searchindex.js | 2 +-
user-guide/latest/compatibility/scans.html | 14 ++++++++++++++
6 files changed, 34 insertions(+), 6 deletions(-)
diff --git a/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg b/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
index 882fb1c923..5342794c8d 100644
--- a/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
+++ b/_images/mermaid-013012468fa27c9e9fdb0a14196d357c619c68a8.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 942" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 942" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/expression-audits/generator_funcs.md.txt b/_sources/contributor-guide/expression-audits/generator_funcs.md.txt
index 4d160c6ac3..3023cfaa77 100644
--- a/_sources/contributor-guide/expression-audits/generator_funcs.md.txt
+++ b/_sources/contributor-guide/expression-audits/generator_funcs.md.txt
@@ -27,7 +27,7 @@
## explode_outer
-- Same `CometExplodeExec` path as `explode`. Compatible for array inputs; empty and NULL arrays both emit one null-valued row per Spark's `outer` semantics via the `ListEmptyToNullExpr` planner bridge (works around [datafusion#19053](https://github.com/apache/datafusion/issues/19053)). Map inputs fall back.
+- Same `CometExplodeExec` path as `explode`. Compatible for array inputs; empty and NULL arrays both emit one null-valued row per Spark's `outer` semantics, which the planner requests as DataFusion's `NullHandling::PreserveAndExpandEmpty`. Map inputs fall back.
## posexplode
@@ -35,6 +35,6 @@
## posexplode_outer
-- Same `CometExplodeExec` path as `posexplode`. Compatible for array inputs; empty and NULL arrays both emit one row with null `pos` and null `value` per Spark's `outer` semantics via the `ListEmptyToNullExpr` planner bridge (works around [datafusion#19053](https://github.com/apache/datafusion/issues/19053)).
+- Same `CometExplodeExec` path as `posexplode`. Compatible for array inputs; empty and NULL arrays both emit one row with null `pos` and null `value` per Spark's `outer` semantics, which the planner requests as DataFusion's `NullHandling::PreserveAndExpandEmpty`.
[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/_sources/user-guide/latest/compatibility/scans.md.txt b/_sources/user-guide/latest/compatibility/scans.md.txt
index 36d245992e..d479e75ffd 100644
--- a/_sources/user-guide/latest/compatibility/scans.md.txt
+++ b/_sources/user-guide/latest/compatibility/scans.md.txt
@@ -62,6 +62,20 @@ The following limitation may produce incorrect results without falling back to S
The following limitations raise an error at scan time rather than falling back to Spark:
+- Selecting a field by name when multiple physical siblings match, including inside structs,
+ arrays, and maps. Comet raises a duplicate-field error instead of resolving the collision.
+ Checks cover referenced columns, including predicates; unselected roots do not prevent
+ reading a unique field by name or field ID. Exact-name projections of unique children in
+ structs and arrays of structs remain supported. Casts that cannot use this pruning reject
+ byte-identical duplicate siblings anywhere in the decoded physical subtree, including maps.
+ Field-ID resolution retains precedence, but selecting a byte-identically duplicated physical
+ root name still raises a duplicate-field error, even when the requested field is renamed.
+ Names in separate groups do not collide. Spark may read a duplicate-bearing file with an
+ explicit schema in case-sensitive mode, but its choice of sibling depends on the field shape
+ and can produce unexpected values. Spark rejects schema inference from a single file with
+ duplicate names; inference across files can depend on merge order.
+ Resolution is tracked in [#5884](https://github.com/apache/datafusion-comet/issues/5884),
+ with mixed-type behavior in [#5964](https://github.com/apache/datafusion-comet/issues/5964).
- Invalid UTF-8 bytes in `STRING` columns. Spark permits arbitrary byte sequences in a `STRING`
column (for example from `CAST(X'C1' AS STRING)`), but Comet's native execution path is built on
Arrow, whose string type is strictly UTF-8. Reading a Parquet file whose `STRING` column contains
diff --git a/contributor-guide/expression-audits/generator_funcs.html b/contributor-guide/expression-audits/generator_funcs.html
index fec672aedd..b8edf61745 100644
--- a/contributor-guide/expression-audits/generator_funcs.html
+++ b/contributor-guide/expression-audits/generator_funcs.html
@@ -563,7 +563,7 @@ under the License.
<section id="explode-outer">
<h2>explode_outer<a class="headerlink" href="#explode-outer" title="Link to this heading">#</a></h2>
<ul class="simple">
-<li><p>Same <code class="docutils literal notranslate"><span class="pre">CometExplodeExec</span></code> path as <code class="docutils literal notranslate"><span class="pre">explode</span></code>. Compatible for array inputs; empty and NULL arrays both emit one null-valued row per Spark’s <code class="docutils literal notranslate"><span class="pre">outer</span></code> semantics via the <code class="docutils literal notranslate"><span class="pre">ListEmptyToNullExpr</span></code> planner b [...]
+<li><p>Same <code class="docutils literal notranslate"><span class="pre">CometExplodeExec</span></code> path as <code class="docutils literal notranslate"><span class="pre">explode</span></code>. Compatible for array inputs; empty and NULL arrays both emit one null-valued row per Spark’s <code class="docutils literal notranslate"><span class="pre">outer</span></code> semantics, which the planner requests as DataFusion’s <code class="docutils literal notranslate"><span class="pre">NullHan [...]
</ul>
</section>
<section id="posexplode">
@@ -575,7 +575,7 @@ under the License.
<section id="posexplode-outer">
<h2>posexplode_outer<a class="headerlink" href="#posexplode-outer" title="Link to this heading">#</a></h2>
<ul class="simple">
-<li><p>Same <code class="docutils literal notranslate"><span class="pre">CometExplodeExec</span></code> path as <code class="docutils literal notranslate"><span class="pre">posexplode</span></code>. Compatible for array inputs; empty and NULL arrays both emit one row with null <code class="docutils literal notranslate"><span class="pre">pos</span></code> and null <code class="docutils literal notranslate"><span class="pre">value</span></code> per Spark’s <code class="docutils literal not [...]
+<li><p>Same <code class="docutils literal notranslate"><span class="pre">CometExplodeExec</span></code> path as <code class="docutils literal notranslate"><span class="pre">posexplode</span></code>. Compatible for array inputs; empty and NULL arrays both emit one row with null <code class="docutils literal notranslate"><span class="pre">pos</span></code> and null <code class="docutils literal notranslate"><span class="pre">value</span></code> per Spark’s <code class="docutils literal not [...]
</ul>
</section>
</section>
diff --git a/searchindex.js b/searchindex.js
index 44c9c4dcc2..911d28dca5 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&": [[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/": [[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install Comet": [[62, "install-comet"], [73, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&": [[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/": [[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install Comet": [[62, "install-comet"], [73, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/compatibility/scans.html b/user-guide/latest/compatibility/scans.html
index c993b3a28e..62481d9c02 100644
--- a/user-guide/latest/compatibility/scans.html
+++ b/user-guide/latest/compatibility/scans.html
@@ -818,6 +818,20 @@ ignores <code class="docutils literal notranslate"><span class="pre">spark.sql.p
</ul>
<p>The following limitations raise an error at scan time rather than falling back to Spark:</p>
<ul class="simple">
+<li><p>Selecting a field by name when multiple physical siblings match, including inside structs,
+arrays, and maps. Comet raises a duplicate-field error instead of resolving the collision.
+Checks cover referenced columns, including predicates; unselected roots do not prevent
+reading a unique field by name or field ID. Exact-name projections of unique children in
+structs and arrays of structs remain supported. Casts that cannot use this pruning reject
+byte-identical duplicate siblings anywhere in the decoded physical subtree, including maps.
+Field-ID resolution retains precedence, but selecting a byte-identically duplicated physical
+root name still raises a duplicate-field error, even when the requested field is renamed.
+Names in separate groups do not collide. Spark may read a duplicate-bearing file with an
+explicit schema in case-sensitive mode, but its choice of sibling depends on the field shape
+and can produce unexpected values. Spark rejects schema inference from a single file with
+duplicate names; inference across files can depend on merge order.
+Resolution is tracked in <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5884">#5884</a>,
+with mixed-type behavior in <a class="reference external" href="https://github.com/apache/datafusion-comet/issues/5964">#5964</a>.</p></li>
<li><p>Invalid UTF-8 bytes in <code class="docutils literal notranslate"><span class="pre">STRING</span></code> columns. Spark permits arbitrary byte sequences in a <code class="docutils literal notranslate"><span class="pre">STRING</span></code>
column (for example from <code class="docutils literal notranslate"><span class="pre">CAST(X'C1'</span> <span class="pre">AS</span> <span class="pre">STRING)</span></code>), but Comet’s native execution path is built on
Arrow, whose string type is strictly UTF-8. Reading a Parquet file whose <code class="docutils literal notranslate"><span class="pre">STRING</span></code> column contains
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org