You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/29 16:51:56 UTC
(datafusion-comet) branch asf-site updated (d897981a46 -> 4da9480675)
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
discard d897981a46 Publish built docs triggered by aee5e06133ca8e2fc1b1337ca2ac8cf3cb0514f3
new 4da9480675 Publish built docs triggered by 669fbde38e326b3076821ab9604db4f21b61e930
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (d897981a46)
\
N -- N -- N refs/heads/asf-site (4da9480675)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
...id-94327707e90a580b1e5f7b5fcb5974635945f9dc.svg | 2 +-
_sources/index.md.txt | 6 +-
_static/images/comet-overview.svg | 74 ++++++++++++++++++++++
index.html | 6 +-
4 files changed, 81 insertions(+), 7 deletions(-)
create mode 100644 _static/images/comet-overview.svg
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 01/01: Publish built docs triggered by 669fbde38e326b3076821ab9604db4f21b61e930
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 4da9480675c4ee3b41b1298594729d9438b93dec
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Tue Sep 29 16:51:47 2026 +0000
Publish built docs triggered by 669fbde38e326b3076821ab9604db4f21b61e930
---
...id-94327707e90a580b1e5f7b5fcb5974635945f9dc.svg | 2 +-
_sources/contributor-guide/ffi.md.txt | 12 ++++
_sources/index.md.txt | 6 +-
_static/images/comet-overview.svg | 74 ++++++++++++++++++++++
contributor-guide/ffi.html | 13 ++++
index.html | 6 +-
searchindex.js | 2 +-
7 files changed, 107 insertions(+), 8 deletions(-)
diff --git a/_images/mermaid-94327707e90a580b1e5f7b5fcb5974635945f9dc.svg b/_images/mermaid-94327707e90a580b1e5f7b5fcb5974635945f9dc.svg
index f42c010823..0fd6c93162 100644
--- a/_images/mermaid-94327707e90a580b1e5f7b5fcb5974635945f9dc.svg
+++ b/_images/mermaid-94327707e90a580b1e5f7b5fcb5974635945f9dc.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/ffi.md.txt b/_sources/contributor-guide/ffi.md.txt
index e56e5ac0f5..ecb812af41 100644
--- a/_sources/contributor-guide/ffi.md.txt
+++ b/_sources/contributor-guide/ffi.md.txt
@@ -256,6 +256,18 @@ pub extern "system" fn Java_..._exportVector(
}
```
+### Array Offsets
+
+Arrow Java's C Data import ignores `ArrowArray.offset` at every level
+([apache/arrow-java#88](https://github.com/apache/arrow-java/issues/88)) and reads each buffer from its start. arrow-rs
+folds a slice into the buffers for almost every type, but a sliced `BooleanArray` keeps its bit offset, and a struct
+exports offset 0 even when its children are sliced. So every array native exports to the JVM first goes through
+`zero_offsets` (in `native/common/src/ffi_offsets.rs`), which re-slices boolean bitmaps to start at bit 0 at every
+level and shares every other buffer. `move_to_spark` applies it to executed batches and decoded shuffle blocks, and
+`JvmScalarUdfExpr` applies it to the inputs of the JVM UDF bridge. A new native to JVM export path has to call it too,
+or sliced booleans reach the JVM misaligned
+([#6288](https://github.com/apache/datafusion-comet/issues/6288)).
+
### Wrapper Object Lifecycle (Native → JVM)
```
diff --git a/_sources/index.md.txt b/_sources/index.md.txt
index facd9b8f0b..5bcaedea8b 100644
--- a/_sources/index.md.txt
+++ b/_sources/index.md.txt
@@ -104,10 +104,10 @@ Runs your existing Spark queries on the Apache DataFusion native engine, no code
<p class="comet-feature__body">Comet tightly integrates with the core Apache DataFusion project, leveraging its powerful execution engine. The diagram below shows how the Comet plugin intercepts Spark physical plans, translates supported operators into a protocol-buffer representation, and hands them to the Apache DataFusion native engine for execution.</p>
</div>
<figure class="comet-feature__figure">
-<img src="_static/images/comet-overview.png"
- width="2569" height="2006"
+<img src="_static/images/comet-overview.svg"
+ width="880" height="640"
loading="lazy" decoding="async"
- alt="Comet architecture overview diagram showing the bridge between Apache Spark and Apache DataFusion" />
+ alt="Spark hands its physical plan to the Comet plugin, which sends the operators it supports to the Apache DataFusion native engine as a protobuf plan. DataFusion reads Parquet and Iceberg data directly and returns the results to Spark as Arrow columnar batches." />
<figcaption>Comet Overview</figcaption>
</figure>
<p class="comet-feature__links">
diff --git a/_static/images/comet-overview.svg b/_static/images/comet-overview.svg
new file mode 100644
index 0000000000..4783d02ac6
--- /dev/null
+++ b/_static/images/comet-overview.svg
@@ -0,0 +1,74 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="880" height="640" viewBox="0 0 880 640" role="img" aria-labelledby="title desc">
+ <title id="title">How Comet runs a Spark query on Apache DataFusion</title>
+ <desc id="desc">Spark plans the query as usual. The Comet plugin replaces the operators it supports with native ones and sends them to the Apache DataFusion native engine as a protobuf plan. DataFusion reads Parquet and Iceberg data, executes the plan, and returns the results to Spark as Arrow columnar batches. Anything Comet does not support keeps running in Spark.</desc>
+ <style>
+ .sans { font-family: Inter, -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif; }
+ </style>
+ <defs>
+ <marker id="arrowHead" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="5" markerHeight="5" orient="auto-start-reverse">
+ <path d="M0,0 L10,5 L0,10 z" fill="#444"/>
+ </marker>
+ </defs>
+
+ <!-- Apache Spark -->
+ <rect x="20" y="20" width="840" height="100" rx="14" fill="#FDE7A7" stroke="#D9AE2E" stroke-width="1.5"/>
+ <text class="sans" x="44" y="60" font-size="21" font-weight="700" fill="#222">Apache Spark</text>
+ <text class="sans" x="44" y="92" font-size="15" fill="#444">Plans and schedules your SQL and DataFrame queries, as usual</text>
+ <rect x="784" y="40" width="56" height="26" rx="13" fill="#ffffff" stroke="#D9AE2E" stroke-width="1.5"/>
+ <text class="sans" x="812" y="58" font-size="13" font-weight="700" fill="#555" text-anchor="middle">JVM</text>
+
+ <!-- Comet plugin -->
+ <rect x="20" y="190" width="600" height="110" rx="14" fill="#C8EBB5" stroke="#6BAF4E" stroke-width="1.5"/>
+ <text class="sans" x="44" y="230" font-size="21" font-weight="700" fill="#222">Comet plugin</text>
+ <text class="sans" font-size="15" fill="#444">
+ <tspan x="44" y="260">Replaces each operator it supports with a native one.</tspan>
+ <tspan x="44" y="282">Anything else keeps running in Spark.</tspan>
+ </text>
+ <rect x="544" y="210" width="56" height="26" rx="13" fill="#ffffff" stroke="#6BAF4E" stroke-width="1.5"/>
+ <text class="sans" x="572" y="228" font-size="13" font-weight="700" fill="#555" text-anchor="middle">JVM</text>
+
+ <!-- Apache DataFusion -->
+ <rect x="20" y="370" width="840" height="144" rx="14" fill="#C8EBB5" stroke="#6BAF4E" stroke-width="1.5"/>
+ <text class="sans" x="44" y="410" font-size="21" font-weight="700" fill="#222">Apache DataFusion native engine</text>
+ <rect x="724" y="390" width="116" height="26" rx="13" fill="#ffffff" stroke="#6BAF4E" stroke-width="1.5"/>
+ <text class="sans" x="782" y="408" font-size="13" font-weight="700" fill="#555" text-anchor="middle">Native · Rust</text>
+ <text class="sans" x="44" y="442" font-size="15" fill="#444">Vectorized, columnar execution on Apache Arrow</text>
+ <g class="sans" font-size="15" font-weight="600" fill="#2F5E1E" text-anchor="middle">
+ <rect x="44" y="460" width="76" height="32" rx="8" fill="#ffffff" stroke="#6BAF4E" stroke-width="1.5"/>
+ <text x="82" y="481">Scan</text>
+ <rect x="132" y="460" width="80" height="32" rx="8" fill="#ffffff" stroke="#6BAF4E" stroke-width="1.5"/>
+ <text x="172" y="481">Filter</text>
+ <rect x="224" y="460" width="92" height="32" rx="8" fill="#ffffff" stroke="#6BAF4E" stroke-width="1.5"/>
+ <text x="270" y="481">Project</text>
+ <rect x="328" y="460" width="68" height="32" rx="8" fill="#ffffff" stroke="#6BAF4E" stroke-width="1.5"/>
+ <text x="362" y="481">Join</text>
+ <rect x="408" y="460" width="112" height="32" rx="8" fill="#ffffff" stroke="#6BAF4E" stroke-width="1.5"/>
+ <text x="464" y="481">Aggregate</text>
+ <rect x="532" y="460" width="68" height="32" rx="8" fill="#ffffff" stroke="#6BAF4E" stroke-width="1.5"/>
+ <text x="566" y="481">Sort</text>
+ <rect x="612" y="460" width="92" height="32" rx="8" fill="#ffffff" stroke="#6BAF4E" stroke-width="1.5"/>
+ <text x="658" y="481">Shuffle</text>
+ </g>
+
+ <!-- Plan goes down on the left -->
+ <line x1="180" y1="122" x2="180" y2="186" stroke="#444" stroke-width="2.5" marker-end="url(#arrowHead)"/>
+ <text class="sans" x="196" y="160" font-size="15" font-weight="700" fill="#222">Physical plan</text>
+ <line x1="180" y1="302" x2="180" y2="366" stroke="#444" stroke-width="2.5" marker-end="url(#arrowHead)"/>
+ <text class="sans" x="196" y="340" font-size="15" font-weight="700" fill="#222">Native plan <tspan font-weight="400" fill="#444">(protobuf)</tspan></text>
+
+ <!-- Results come back up on the right -->
+ <line x1="680" y1="368" x2="680" y2="124" stroke="#444" stroke-width="2.5" marker-end="url(#arrowHead)"/>
+ <text class="sans" font-size="15" fill="#222">
+ <tspan x="698" y="238" font-weight="700">Results as Arrow</tspan>
+ <tspan x="698" y="258" font-weight="700">columnar batches</tspan>
+ </text>
+
+ <!-- Data -->
+ <line x1="82" y1="562" x2="82" y2="518" stroke="#444" stroke-width="2.5" marker-end="url(#arrowHead)"/>
+ <path d="M48,574 a34,10 0 0,1 68,0 v38 a34,10 0 0,1 -68,0 z" fill="#D5EAF7" stroke="#5B8DB8" stroke-width="1.5"/>
+ <path d="M48,574 a34,10 0 0,0 68,0" fill="none" stroke="#5B8DB8" stroke-width="1.5"/>
+ <text class="sans" font-size="15" fill="#444">
+ <tspan x="134" y="590" font-weight="700" fill="#222">Parquet files and Apache Iceberg tables</tspan>
+ <tspan x="134" y="612">Read directly by the native engine</tspan>
+ </text>
+</svg>
diff --git a/contributor-guide/ffi.html b/contributor-guide/ffi.html
index 02ea485557..1d1cf85a14 100644
--- a/contributor-guide/ffi.html
+++ b/contributor-guide/ffi.html
@@ -781,6 +781,18 @@ until then.</p>
</pre></div>
</div>
</section>
+<section id="array-offsets">
+<h3>Array Offsets<a class="headerlink" href="#array-offsets" title="Link to this heading">#</a></h3>
+<p>Arrow Java’s C Data import ignores <code class="docutils literal notranslate"><span class="pre">ArrowArray.offset</span></code> at every level
+(<a class="reference external" href="https://github.com/apache/arrow-java/issues/88">apache/arrow-java#88</a>) and reads each buffer from its start. arrow-rs
+folds a slice into the buffers for almost every type, but a sliced <code class="docutils literal notranslate"><span class="pre">BooleanArray</span></code> keeps its bit offset, and a struct
+exports offset 0 even when its children are sliced. So every array native exports to the JVM first goes through
+<code class="docutils literal notranslate"><span class="pre">zero_offsets</span></code> (in <code class="docutils literal notranslate"><span class="pre">native/common/src/ffi_offsets.rs</span></code>), which re-slices boolean bitmaps to start at bit 0 at every
+level and shares every other buffer. <code class="docutils literal notranslate"><span class="pre">move_to_spark</span></code> applies it to executed batches and decoded shuffle blocks, and
+<code class="docutils literal notranslate"><span class="pre">JvmScalarUdfExpr</span></code> applies it to the inputs of the JVM UDF bridge. A new native to JVM export path has to call it too,
+or sliced booleans reach the JVM misaligned
+(<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/6288">#6288</a>).</p>
+</section>
<section id="wrapper-object-lifecycle-native-jvm">
<h3>Wrapper Object Lifecycle (Native → JVM)<a class="headerlink" href="#wrapper-object-lifecycle-native-jvm" title="Link to this heading">#</a></h3>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>Time Native Memory JVM Heap Data location
@@ -950,6 +962,7 @@ t4 Batch handle released ArrowBuf freed Data freed
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#native-jvm-data-flow-cometexeciterator">Native → JVM Data Flow (CometExecIterator)</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">Architecture</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#ffi-transfer-process">FFI Transfer Process</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#array-offsets">Array Offsets</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#wrapper-object-lifecycle-native-jvm">Wrapper Object Lifecycle (Native → JVM)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#release-callbacks">Release Callbacks</a></li>
</ul>
diff --git a/index.html b/index.html
index 813a61479b..13ca8bf26a 100644
--- a/index.html
+++ b/index.html
@@ -468,10 +468,10 @@ Runs your existing Spark queries on the Apache DataFusion native engine, no code
<p class="comet-feature__body">Comet tightly integrates with the core Apache DataFusion project, leveraging its powerful execution engine. The diagram below shows how the Comet plugin intercepts Spark physical plans, translates supported operators into a protocol-buffer representation, and hands them to the Apache DataFusion native engine for execution.</p>
</div>
<figure class="comet-feature__figure">
-<img src="_static/images/comet-overview.png"
- width="2569" height="2006"
+<img src="_static/images/comet-overview.svg"
+ width="880" height="640"
loading="lazy" decoding="async"
- alt="Comet architecture overview diagram showing the bridge between Apache Spark and Apache DataFusion" />
+ alt="Spark hands its physical plan to the Comet plugin, which sends the operators it supports to the Apache DataFusion native engine as a protobuf plan. DataFusion reads Parquet and Iceberg data directly and returns the results to Spark as Arrow columnar batches." />
<figcaption>Comet Overview</figcaption>
</figure>
<p class="comet-feature__links">
diff --git a/searchindex.js b/searchindex.js
index 77e9b3a5cd..14a14601cd 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[57, "id1"]], "%": [[55, "id1"]], "&": [[45, "id1"]], "*": [[55, "id2"]], "+": [[55, "id3"]], "-": [[55, "id4"]], "/": [[55, "id5"]], "1. Format Your Code": [[42, "format-your-code"]], "1. Install Comet": [[63, "install-comet"], [75, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[42, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[63, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[57, "id1"]], "%": [[55, "id1"]], "&": [[45, "id1"]], "*": [[55, "id2"]], "+": [[55, "id3"]], "-": [[55, "id4"]], "/": [[55, "id5"]], "1. Format Your Code": [[42, "format-your-code"]], "1. Install Comet": [[63, "install-comet"], [75, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[42, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[63, "clone-iceberg-and-apply- [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org