You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/23 13:50:37 UTC
(datafusion-comet) 01/01: Publish built docs triggered by c56f6a0209dba0600103a37fac7ca9b006639857
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 89bfdd2374c8610c0ec95a1dbd48f17faef2f431
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Fri Oct 23 13:50:29 2026 +0000
Publish built docs triggered by c56f6a0209dba0600103a37fac7ca9b006639857
---
...rmaid-09341aca03757d9d3637a2eaff73472d472d0673.svg | 2 +-
.../contributor-guide/adding_a_new_operator.md.txt | 5 ++++-
.../user-guide/latest/compatibility/operators.md.txt | 19 +++++++++++++------
contributor-guide/adding_a_new_operator.html | 5 ++++-
searchindex.js | 2 +-
user-guide/latest/compatibility/operators.html | 18 ++++++++++++------
6 files changed, 35 insertions(+), 16 deletions(-)
diff --git a/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg b/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
index e648e56016..fad96cf938 100644
--- a/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
+++ b/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 1014" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-sv [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 1014" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-sv [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/adding_a_new_operator.md.txt b/_sources/contributor-guide/adding_a_new_operator.md.txt
index e142aaa499..0c36921a94 100644
--- a/_sources/contributor-guide/adding_a_new_operator.md.txt
+++ b/_sources/contributor-guide/adding_a_new_operator.md.txt
@@ -372,7 +372,10 @@ ordering must keep it natively, and the native operator must say so by returning
`maintains_input_order` for that child, as DataFusion's filter and projection do. The planner
declares the order through an operator only when the operator says it keeps it. An operator that
cannot keep the order natively must not report it, or has to be added to `mayReportFalseOrder` in
-`operators.scala`, which makes the hash join serde send no order over it.
+`operators.scala`. The hash join serde then sends no order over it, and an outer hash join above it
+that has to keep that order falls back to Spark. Comet's hash joins are the exception within one
+native plan: an outer hash join over one of them stays native, even over one that declared no
+order, since DataFusion keeps the order it can see there.
A leaf that reports an ordering is where the order starts, so the planner has to know that the
native operator produces its rows in that order. It knows that of an input from the JVM and of a
diff --git a/_sources/user-guide/latest/compatibility/operators.md.txt b/_sources/user-guide/latest/compatibility/operators.md.txt
index 01d2a2ef67..8785874c6f 100644
--- a/_sources/user-guide/latest/compatibility/operators.md.txt
+++ b/_sources/user-guide/latest/compatibility/operators.md.txt
@@ -132,12 +132,19 @@ express falls back to Spark and records a fallback reason (see
[Understanding Comet Plans](../understanding-comet-plans.md)), because Comet cannot tell whether
an operator above the join relies on the order.
-Where the order Spark reports might not hold, Comet declares nothing and an outer join can return
-its unmatched rows out of order. The output of a nested loop join
-([#6898](https://github.com/apache/datafusion-comet/issues/6898)) is one such case, and so is a
-cache or checkpoint that one wrote. Comet also declares nothing when it cannot follow the
-order back to its source, as with an RDD checkpoint, and over some caches written by an outer hash
-join whose order does hold ([#6922](https://github.com/apache/datafusion-comet/issues/6922)).
+An outer join falls back to Spark when Comet cannot check the order Spark reports and that order
+reaches the join through an input from the JVM or through a nested loop join. Spark keeps the rows
+in the order they arrive. One such input is an RDD checkpoint. It reaches a native join only with
+`spark.comet.convert.rdd.enabled=true`, which defaults to `false`. Without that setting the
+checkpoint stays a Spark input and the join stays in Spark anyway. Another is a cache written by an
+outer hash join whose own probe input was not a scan
+([#6922](https://github.com/apache/datafusion-comet/issues/6922)). The output of a nested loop
+join can already be out of order when it reaches the join
+([#6898](https://github.com/apache/datafusion-comet/issues/6898)).
+
+When the order Comet cannot check comes only from another native hash join in the same plan, the
+join does not fall back. It runs natively and declares no order, since DataFusion keeps the order
+it can see there.
## Window Functions
diff --git a/contributor-guide/adding_a_new_operator.html b/contributor-guide/adding_a_new_operator.html
index 2a72972382..0b47daa6ce 100644
--- a/contributor-guide/adding_a_new_operator.html
+++ b/contributor-guide/adding_a_new_operator.html
@@ -893,7 +893,10 @@ ordering must keep it natively, and the native operator must say so by returning
<code class="docutils literal notranslate"><span class="pre">maintains_input_order</span></code> for that child, as DataFusion’s filter and projection do. The planner
declares the order through an operator only when the operator says it keeps it. An operator that
cannot keep the order natively must not report it, or has to be added to <code class="docutils literal notranslate"><span class="pre">mayReportFalseOrder</span></code> in
-<code class="docutils literal notranslate"><span class="pre">operators.scala</span></code>, which makes the hash join serde send no order over it.</p>
+<code class="docutils literal notranslate"><span class="pre">operators.scala</span></code>. The hash join serde then sends no order over it, and an outer hash join above it
+that has to keep that order falls back to Spark. Comet’s hash joins are the exception within one
+native plan: an outer hash join over one of them stays native, even over one that declared no
+order, since DataFusion keeps the order it can see there.</p>
<p>A leaf that reports an ordering is where the order starts, so the planner has to know that the
native operator produces its rows in that order. It knows that of an input from the JVM and of a
scan of files, which reads the files of each partition one after the other, and the native range
diff --git a/searchindex.js b/searchindex.js
index 220cdc9670..b6aaf90154 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[63, "id1"]], "%": [[61, "id1"]], "&": [[51, "id1"]], "*": [[61, "id2"]], "+": [[61, "id3"]], "-": [[61, "id4"]], "/": [[61, "id5"]], "1. Format Your Code": [[48, "format-your-code"]], "1. Install Comet": [[71, "install-comet"], [83, "install-comet"]], "1. Native Operators (nativeExecs map)": [[31, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[48, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[71, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[63, "id1"]], "%": [[61, "id1"]], "&": [[51, "id1"]], "*": [[61, "id2"]], "+": [[61, "id3"]], "-": [[61, "id4"]], "/": [[61, "id5"]], "1. Format Your Code": [[48, "format-your-code"]], "1. Install Comet": [[71, "install-comet"], [83, "install-comet"]], "1. Native Operators (nativeExecs map)": [[31, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[48, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[71, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/compatibility/operators.html b/user-guide/latest/compatibility/operators.html
index 8803716fba..e97d28a564 100644
--- a/user-guide/latest/compatibility/operators.html
+++ b/user-guide/latest/compatibility/operators.html
@@ -948,12 +948,18 @@ rows in place, so the join does not need the rest. An outer join whose first sor
express falls back to Spark and records a fallback reason (see
<a class="reference internal" href="../understanding-comet-plans.html"><span class="std std-doc">Understanding Comet Plans</span></a>), because Comet cannot tell whether
an operator above the join relies on the order.</p>
-<p>Where the order Spark reports might not hold, Comet declares nothing and an outer join can return
-its unmatched rows out of order. The output of a nested loop join
-(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/6898">#6898</a>) is one such case, and so is a
-cache or checkpoint that one wrote. Comet also declares nothing when it cannot follow the
-order back to its source, as with an RDD checkpoint, and over some caches written by an outer hash
-join whose order does hold (<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/6922">#6922</a>).</p>
+<p>An outer join falls back to Spark when Comet cannot check the order Spark reports and that order
+reaches the join through an input from the JVM or through a nested loop join. Spark keeps the rows
+in the order they arrive. One such input is an RDD checkpoint. It reaches a native join only with
+<code class="docutils literal notranslate"><span class="pre">spark.comet.convert.rdd.enabled=true</span></code>, which defaults to <code class="docutils literal notranslate"><span class="pre">false</span></code>. Without that setting the
+checkpoint stays a Spark input and the join stays in Spark anyway. Another is a cache written by an
+outer hash join whose own probe input was not a scan
+(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/6922">#6922</a>). The output of a nested loop
+join can already be out of order when it reaches the join
+(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/6898">#6898</a>).</p>
+<p>When the order Comet cannot check comes only from another native hash join in the same plan, the
+join does not fall back. It runs natively and declares no order, since DataFusion keeps the order
+it can see there.</p>
</section>
</section>
<section id="window-functions">
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org