You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/08/29 20:18:53 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by e0fa3ccb16d4c9b3a03a38b09af6ed0fb242e027
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 7f51889e8a Publish built docs triggered by e0fa3ccb16d4c9b3a03a38b09af6ed0fb242e027
7f51889e8a is described below
commit 7f51889e8a39dba4b6085d56ab8e3d6244624aa9
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Sat Aug 29 20:18:50 2026 +0000
Publish built docs triggered by e0fa3ccb16d4c9b3a03a38b09af6ed0fb242e027
---
_sources/user-guide/latest/pyarrow-udfs.md.txt | 26 +++++++++++---------------
searchindex.js | 2 +-
user-guide/latest/pyarrow-udfs.html | 26 +++++++++++---------------
3 files changed, 23 insertions(+), 31 deletions(-)
diff --git a/_sources/user-guide/latest/pyarrow-udfs.md.txt b/_sources/user-guide/latest/pyarrow-udfs.md.txt
index ff493de23c..14975c42c3 100644
--- a/_sources/user-guide/latest/pyarrow-udfs.md.txt
+++ b/_sources/user-guide/latest/pyarrow-udfs.md.txt
@@ -203,18 +203,14 @@ on the unoptimized path.
session time zone such a UDF can diverge from the unoptimized path. Set
`spark.comet.exec.pyarrowUDF.enabled=false` for those UDFs.
- `spark.sql.execution.arrow.useLargeVarTypes=true` is not supported. With this conf enabled,
- Spark widens `StringType` and `BinaryType` to Arrow's 8-byte-offset variants in the
- destination IPC root, while Comet's source vectors always use 4-byte offsets. The buffer-copy
- path cannot bridge that mismatch, so `EliminateRedundantTransitions` skips the rewrite and
- vanilla Spark handles the operation.
-- Each batch is copied twice on the JVM side: once from Comet's vectors into Spark's
- destination IPC root (per-buffer `setBytes`), and a second time inside the IPC writer when
- `VectorUnloader` / `MessageSerializer.serialize` walks the root and writes bytes to the
- pipe to the Python worker. The pipe write is structural (Spark's transport to Python is
- fork + pipe + Arrow IPC, so the buffer bytes must reach the pipe at least once); dropping
- the first copy by serialising directly from Comet's vectors is tracked in
- [#4294](https://github.com/apache/datafusion-comet/issues/4294). Even after that,
- true zero-copy at the JVM boundary is blocked because Comet's source `FieldVector`s are
- imported from native via Arrow C Data Interface (their buffers route `release` through FFI),
- while Spark's destination IPC root is a child of `ArrowUtils.rootAllocator`. The two
- reference managers cannot share buffers via `TransferPair`.
+ Spark supplies `large_string` and `large_binary` input columns with 8-byte offsets. Native
+ Comet vectors use 4-byte offsets, and direct serialization advertises their matching `string`
+ and `binary` types. This produces a valid IPC stream, but does not preserve the input types
+ requested by the configuration. `EliminateRedundantTransitions` therefore skips the rewrite
+ and vanilla Spark handles the operation. Comet can read `large_string` and `large_binary`
+ columns returned by a Python worker; that output support does not widen the input vectors.
+- Comet writes input Arrow IPC record batches directly from its existing vector buffers. The
+ only additional Arrow buffer is the validity bitmap for the non-null struct that wraps the
+ input columns. Writing the IPC bytes to the Python worker's pipe still requires one copy;
+ that copy is inherent to Spark's process-based Python transport. This path does not transfer
+ buffers between Arrow allocators or change their ownership.
diff --git a/searchindex.js b/searchindex.js
index 0bfa42c789..d6e8cc7bee 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[55, "id1"]], "%": [[53, "id1"]], "&": [[43, "id1"]], "*": [[53, "id2"]], "+": [[53, "id3"]], "-": [[53, "id4"]], "/": [[53, "id5"]], "1. Format Your Code": [[40, "format-your-code"]], "1. Install Comet": [[61, "install-comet"], [71, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[40, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[61, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[55, "id1"]], "%": [[53, "id1"]], "&": [[43, "id1"]], "*": [[53, "id2"]], "+": [[53, "id3"]], "-": [[53, "id4"]], "/": [[53, "id5"]], "1. Format Your Code": [[40, "format-your-code"]], "1. Install Comet": [[61, "install-comet"], [71, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[40, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[61, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/pyarrow-udfs.html b/user-guide/latest/pyarrow-udfs.html
index e1ced7b8c0..7d7291f233 100644
--- a/user-guide/latest/pyarrow-udfs.html
+++ b/user-guide/latest/pyarrow-udfs.html
@@ -967,21 +967,17 @@ UDF that reads the Arrow field’s time zone or localizes to wall-clock time (fo
session time zone such a UDF can diverge from the unoptimized path. Set
<code class="docutils literal notranslate"><span class="pre">spark.comet.exec.pyarrowUDF.enabled=false</span></code> for those UDFs.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">spark.sql.execution.arrow.useLargeVarTypes=true</span></code> is not supported. With this conf enabled,
-Spark widens <code class="docutils literal notranslate"><span class="pre">StringType</span></code> and <code class="docutils literal notranslate"><span class="pre">BinaryType</span></code> to Arrow’s 8-byte-offset variants in the
-destination IPC root, while Comet’s source vectors always use 4-byte offsets. The buffer-copy
-path cannot bridge that mismatch, so <code class="docutils literal notranslate"><span class="pre">EliminateRedundantTransitions</span></code> skips the rewrite and
-vanilla Spark handles the operation.</p></li>
-<li><p>Each batch is copied twice on the JVM side: once from Comet’s vectors into Spark’s
-destination IPC root (per-buffer <code class="docutils literal notranslate"><span class="pre">setBytes</span></code>), and a second time inside the IPC writer when
-<code class="docutils literal notranslate"><span class="pre">VectorUnloader</span></code> / <code class="docutils literal notranslate"><span class="pre">MessageSerializer.serialize</span></code> walks the root and writes bytes to the
-pipe to the Python worker. The pipe write is structural (Spark’s transport to Python is
-fork + pipe + Arrow IPC, so the buffer bytes must reach the pipe at least once); dropping
-the first copy by serialising directly from Comet’s vectors is tracked in
-<a class="reference external" href="https://github.com/apache/datafusion-comet/issues/4294">#4294</a>. Even after that,
-true zero-copy at the JVM boundary is blocked because Comet’s source <code class="docutils literal notranslate"><span class="pre">FieldVector</span></code>s are
-imported from native via Arrow C Data Interface (their buffers route <code class="docutils literal notranslate"><span class="pre">release</span></code> through FFI),
-while Spark’s destination IPC root is a child of <code class="docutils literal notranslate"><span class="pre">ArrowUtils.rootAllocator</span></code>. The two
-reference managers cannot share buffers via <code class="docutils literal notranslate"><span class="pre">TransferPair</span></code>.</p></li>
+Spark supplies <code class="docutils literal notranslate"><span class="pre">large_string</span></code> and <code class="docutils literal notranslate"><span class="pre">large_binary</span></code> input columns with 8-byte offsets. Native
+Comet vectors use 4-byte offsets, and direct serialization advertises their matching <code class="docutils literal notranslate"><span class="pre">string</span></code>
+and <code class="docutils literal notranslate"><span class="pre">binary</span></code> types. This produces a valid IPC stream, but does not preserve the input types
+requested by the configuration. <code class="docutils literal notranslate"><span class="pre">EliminateRedundantTransitions</span></code> therefore skips the rewrite
+and vanilla Spark handles the operation. Comet can read <code class="docutils literal notranslate"><span class="pre">large_string</span></code> and <code class="docutils literal notranslate"><span class="pre">large_binary</span></code>
+columns returned by a Python worker; that output support does not widen the input vectors.</p></li>
+<li><p>Comet writes input Arrow IPC record batches directly from its existing vector buffers. The
+only additional Arrow buffer is the validity bitmap for the non-null struct that wraps the
+input columns. Writing the IPC bytes to the Python worker’s pipe still requires one copy;
+that copy is inherent to Spark’s process-based Python transport. This path does not transfer
+buffers between Arrow allocators or change their ownership.</p></li>
</ul>
</section>
</section>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org