You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "sunchao (via GitHub)" <gi...@apache.org> on 2026/08/15 17:27:34 UTC

[PR] perf: serialize Python input directly from Comet Arrow vectors [datafusion-comet]

sunchao opened a new pull request, #5368:
URL: https://github.com/apache/datafusion-comet/pull/5368

   ## Why are the changes needed?
   
   Closes #4383.
   
   The Spark 4.x Comet Python runner currently copies every source Arrow column into an intermediate vector tree before serializing each batch for `mapInArrow` and `mapInPandas`. This extra per-batch allocation and memory copy is avoidable: the existing Comet vectors already have the Arrow physical layout required by the Python worker.
   
   ## What changes were proposed in this PR?
   
   - Serialize existing Comet `FieldVector` buffers directly into Arrow IPC record batches while retaining the existing schema-only stream root.
   - Allocate only the validity bitmap for the non-null struct that wraps the input columns, and retain/release borrowed native-backed Arrow buffers correctly on success and failure.
   - Preserve nested struct/list/map layouts, nullability, variadic-buffer metadata, empty batches, Spark 3.5 fallback, and incompatible large-variable-width fallback behavior.
   - Add focused JVM ownership/layout/error-cleanup coverage and multi-batch PySpark worker regressions; register the new Scala suite in both Linux and macOS CI manifests.
   - Update the Python UDF documentation and make the benchmark build profile match its documented PySpark version.
   
   ## How was this PR tested?
   
   - `JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home mvn -B -Pspark-4.0 -Pscala-2.13 -DwildcardSuites=org.apache.spark.sql.execution.python.CometArrowPythonRunnerSuite test`: full reactor succeeded; all 5 focused Arrow regression tests and 23 additional JVM unit tests passed.
   - Spark 4.1 focused Arrow regression suite: 5 tests passed; Spark 3.5 compatibility compilation succeeded.
   - Real PySpark 4.0 accelerated/fallback worker regressions, including nested structs/lists/maps, nulls, and six changing source batches of sizes `[7, 7, 7, 7, 7, 2]`.
   - `python3 dev/ci/check-suites.py`: all 178 Linux/macOS suite registrations passed.
   - Maven ScalaStyle and Spotless checks passed; `ruff check spark/src/test/resources/pyspark/benchmark_pyarrow_udf.py` passed.
   - `make -n release PROFILES='-Pspark-4.0 -Pscala-2.13'` confirmed benchmark build/profile alignment.
   - Local wide-schema benchmark measured approximately 1.24x faster `mapInArrow` and 1.27x faster `mapInPandas` (workload-dependent).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: serialize Python input directly from Comet Arrow vectors [datafusion-comet]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #5368:
URL: https://github.com/apache/datafusion-comet/pull/5368#discussion_r3883490357


##########
docs/source/user-guide/latest/pyarrow-udfs.md:
##########
@@ -203,18 +203,14 @@ on the unoptimized path.
   session time zone such a UDF can diverge from the unoptimized path. Set
   `spark.comet.exec.pyarrowUDF.enabled=false` for those UDFs.
 - `spark.sql.execution.arrow.useLargeVarTypes=true` is not supported. With this conf enabled,
-  Spark widens `StringType` and `BinaryType` to Arrow's 8-byte-offset variants in the
-  destination IPC root, while Comet's source vectors always use 4-byte offsets. The buffer-copy
-  path cannot bridge that mismatch, so `EliminateRedundantTransitions` skips the rewrite and
-  vanilla Spark handles the operation.
-- Each batch is copied twice on the JVM side: once from Comet's vectors into Spark's
-  destination IPC root (per-buffer `setBytes`), and a second time inside the IPC writer when
-  `VectorUnloader` / `MessageSerializer.serialize` walks the root and writes bytes to the
-  pipe to the Python worker. The pipe write is structural (Spark's transport to Python is
-  fork + pipe + Arrow IPC, so the buffer bytes must reach the pipe at least once); dropping
-  the first copy by serialising directly from Comet's vectors is tracked in
-  [#4294](https://github.com/apache/datafusion-comet/issues/4294). Even after that,
-  true zero-copy at the JVM boundary is blocked because Comet's source `FieldVector`s are
-  imported from native via Arrow C Data Interface (their buffers route `release` through FFI),
-  while Spark's destination IPC root is a child of `ArrowUtils.rootAllocator`. The two
-  reference managers cannot share buffers via `TransferPair`.
+  Spark supplies `large_string` and `large_binary` input columns with 8-byte offsets. Native

Review Comment:
   mmm, does that mean we need to map `large_string` to `LargeStringArray` in native? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: serialize Python input directly from Comet Arrow vectors [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5368:
URL: https://github.com/apache/datafusion-comet/pull/5368#issuecomment-5464622622

   Merged to master, thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: serialize Python input directly from Comet Arrow vectors [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao merged PR #5368:
URL: https://github.com/apache/datafusion-comet/pull/5368


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: serialize Python input directly from Comet Arrow vectors [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5368:
URL: https://github.com/apache/datafusion-comet/pull/5368#discussion_r3883029263


##########
spark/src/test/spark-4.x/org/apache/spark/sql/execution/python/CometArrowPythonRunnerSuite.scala:
##########
@@ -0,0 +1,329 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.execution.python
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, IOException}
+import java.nio.ByteBuffer
+import java.nio.channels.{Channels, WritableByteChannel}
+
+import scala.jdk.CollectionConverters._
+
+import org.scalatest.funsuite.AnyFunSuite
+import org.scalatest.matchers.should.Matchers
+
+import org.apache.arrow.memory.{BufferAllocator, RootAllocator}
+import org.apache.arrow.vector.{FieldVector, IntVector, NullVector, VarCharVector, VectorSchemaRoot}
+import org.apache.arrow.vector.complex.{ListVector, MapVector, StructVector}
+import org.apache.arrow.vector.ipc.{ArrowStreamReader, ArrowStreamWriter, WriteChannel}
+import org.apache.arrow.vector.types.pojo.{ArrowType, Field, FieldType, Schema}
+import org.apache.spark.sql.execution.python.CometArrowPythonRunnerBase.serializeBatch
+
+class CometArrowPythonRunnerSuite extends AnyFunSuite with Matchers {
+
+  private def withWriter(
+      childFields: Seq[Field],
+      allocator: BufferAllocator,
+      channel: WritableByteChannel)(f: WritableByteChannel => Unit): Unit = {
+    val structField = new Field(
+      "struct",
+      new FieldType(false, ArrowType.Struct.INSTANCE, null),
+      childFields.asJava)
+    val root = VectorSchemaRoot.create(new Schema(Seq(structField).asJava), allocator)
+    val writer = new ArrowStreamWriter(root, null, channel)
+    try {
+      writer.start()
+      f(channel)
+      writer.end()
+    } finally {
+      writer.close()
+      root.close()
+    }
+  }
+
+  private def withReader(bytes: Array[Byte])(f: ArrowStreamReader => Unit): Unit = {
+    val allocator = new RootAllocator(Long.MaxValue)
+    val reader = new ArrowStreamReader(new ByteArrayInputStream(bytes), allocator)
+    try {
+      f(reader)
+    } finally {
+      reader.close()
+      allocator.close()
+    }
+  }
+
+  test("direct batches retain borrowed buffers without copying them into the writer allocator") {

Review Comment:
   All five tests here build their source vectors with a plain `RootAllocator`. In production the source is always FFI-imported, where the buffers are owned by `ReferenceCountedArrowArray` and the validity bitmap is synthesized by `BitVectorHelper.loadValidityBuffer` at import time. That is precisely the reference manager that the refcount and `getAllocatedMemory` assertions are trying to pin down.
   
   Would you add one case that exports a vector with `Data.exportVector` and re-imports it into a second allocator before calling `serializeBatch`? I checked this locally against real Comet scan batches and the counts do balance, so this is a coverage question rather than a bug.



##########
spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala:
##########
@@ -112,10 +112,9 @@ case class EliminateRedundantTransitions(session: SparkSession)
       // 4.1+ matches the renamed `MapInArrowExec`.
       //
       // Falls back to vanilla Spark when `spark.sql.execution.arrow.useLargeVarTypes` is enabled:
-      // CometArrowPythonRunnerBase.copyVector does raw `setBytes` on each Arrow buffer, but Comet's
-      // source string/binary vectors always use 4-byte offsets while the destination root is
-      // allocated with 8-byte offsets when this conf is on. The buffer counts match but the
-      // offset width does not, so a direct memcpy would corrupt the offsets.
+      // Comet's source string/binary vectors use 4-byte offsets, while Spark expects 8-byte
+      // offsets when this conf is on. Direct IPC serialization cannot change the source vectors'
+      // physical layout, so forwarding them under Spark's widened schema would corrupt the stream.

Review Comment:
   The comment says forwarding Comet's 4-byte-offset vectors under Spark's widened schema "would corrupt the stream". That was true of `copyVector`, but after this change the advertised schema comes from `sourceVectors.map(_.getField)`, so the header and the buffers agree and the stream stays internally consistent.
   
   I think the fallback should still stay, but for different reasons: the worker would see `string` where `useLargeVarTypes=true` promises `large_string`, and the worker's `large_*` output would then come back through `ArrowStreamReader` into `CometVector.getVector`. Could you update the comment here and the matching bullet in `pyarrow-udfs.md` to say that instead?



##########
spark/src/main/spark-4.x/org/apache/spark/sql/execution/python/CometArrowPythonRunnerBase.scala:
##########
@@ -167,49 +168,37 @@ private[python] trait CometArrowPythonRunnerBase
 
         val cometBatch = currentGroup.next()
         val startData = dataOut.size()
+        val sourceVectors = (0 until cometBatch.numCols()).map { i =>
+          cometBatch
+            .column(i)
+            .asInstanceOf[CometDecodedVector]
+            .getValueVector
+            .asInstanceOf[FieldVector]
+        }
 
         if (arrowWriter == null) {
-          // Build the destination struct root once, sized to the first batch's child fields.
+          // Build the schema-only struct root once from the first batch's child fields.
           // mapInArrow/mapInPandas exchange the columns under a single non-nullable struct.
           // Comet's FFI-imported vectors leave the Arrow Field name null, so restore the real
           // column names from the input schema (the worker reads columns by name, and shaded
-          // Arrow rejects a null field name). The field types and child structure are kept as-is
-          // so copyVector still walks the source and destination trees in lockstep. Keeping the
-          // type as-is also means a TimestampType reaches the worker with Comet's UTC time zone
+          // Arrow rejects a null field name). Keep the field types and child structure as-is so
+          // the advertised schema matches the source buffers. Keeping the type as-is also means
+          // a TimestampType reaches the worker with Comet's UTC time zone
           // rather than the session zone vanilla Spark would label it with; this is a documented
           // limitation (see pyarrow-udfs.md), not a value difference, since the stored instant is
           // identical.
           val childNames = inputStructType.fieldNames
-          val childFields = (0 until cometBatch.numCols()).map { i =>
-            val vecField =
-              cometBatch.column(i).asInstanceOf[CometDecodedVector].getValueVector.getField
-            renamed(vecField, childNames(i), forceNullable = true)
+          val childFields = sourceVectors.zipWithIndex.map { case (vector, i) =>
+            renamed(vector.getField, childNames(i), forceNullable = true)
           }
           startWriter(childFields, dataOut)

Review Comment:
   `copyVector` used to fail loudly via its `require(srcBufs.size == dstBufs.size)` if a later batch's layout stopped matching the destination tree. Now that the schema header is written once from the first batch and each subsequent batch's buffers go straight to the wire, a layout change would instead produce a well-formed record batch that the worker silently misreads.
   
   I could not find a path where Comet changes layout mid-partition, so this may well be unreachable, but would you consider capturing the child fields in `startWriter` and asserting `sourceVectors.map(_.getField)` still matches them per batch? It keeps the fail-loud behaviour for about three lines.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: serialize Python input directly from Comet Arrow vectors [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5368:
URL: https://github.com/apache/datafusion-comet/pull/5368#discussion_r3884203202


##########
spark/src/test/resources/pyspark/benchmark_pyarrow_udf.py:
##########
@@ -84,6 +84,8 @@ def _build_spark() -> SparkSession:
         .config("spark.plugins", "org.apache.spark.CometPlugin")
         .config("spark.comet.enabled", "true")
         .config("spark.comet.exec.enabled", "true")
+        # Keep Comet's scan and execution rules active with Spark's default shuffle manager.
+        .config("spark.comet.shuffle.enabled", "false")

Review Comment:
   Updated in 5dd4782c. The benchmark now configures `CometShuffleManager` and leaves Comet shuffle enabled at its default, instead of disabling shuffle to keep the extension active with Spark's default manager.
   
   Smoke-tested all three workloads with both `mapInArrow` and `mapInPandas`, in vanilla and optimized modes, on Spark 4.0.4. All 12 combinations completed and the existing plan assertions passed. This was a small functional run, not a new performance measurement.
   



##########
spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala:
##########
@@ -112,10 +112,10 @@ case class EliminateRedundantTransitions(session: SparkSession)
       // 4.1+ matches the renamed `MapInArrowExec`.
       //
       // Falls back to vanilla Spark when `spark.sql.execution.arrow.useLargeVarTypes` is enabled:
-      // CometArrowPythonRunnerBase.copyVector does raw `setBytes` on each Arrow buffer, but Comet's
-      // source string/binary vectors always use 4-byte offsets while the destination root is
-      // allocated with 8-byte offsets when this conf is on. The buffer counts match but the
-      // offset width does not, so a direct memcpy would corrupt the offsets.
+      // Native Comet string/binary vectors use 4-byte offsets. The IPC schema follows these

Review Comment:
   The old comment described raw buffer copying in `copyVector`, which this PR removes. I updated the explanation for direct serialization: the IPC schema matches the source buffers, so the stream is valid, but it would expose `string` / `binary` instead of the large input types requested by `useLargeVarTypes=true`.
   
   Only the explanation changed here; the existing fallback behavior is unchanged.
   



##########
docs/source/user-guide/latest/pyarrow-udfs.md:
##########
@@ -203,18 +203,14 @@ on the unoptimized path.
   session time zone such a UDF can diverge from the unoptimized path. Set
   `spark.comet.exec.pyarrowUDF.enabled=false` for those UDFs.
 - `spark.sql.execution.arrow.useLargeVarTypes=true` is not supported. With this conf enabled,
-  Spark widens `StringType` and `BinaryType` to Arrow's 8-byte-offset variants in the
-  destination IPC root, while Comet's source vectors always use 4-byte offsets. The buffer-copy
-  path cannot bridge that mismatch, so `EliminateRedundantTransitions` skips the rewrite and
-  vanilla Spark handles the operation.
-- Each batch is copied twice on the JVM side: once from Comet's vectors into Spark's
-  destination IPC root (per-buffer `setBytes`), and a second time inside the IPC writer when
-  `VectorUnloader` / `MessageSerializer.serialize` walks the root and writes bytes to the
-  pipe to the Python worker. The pipe write is structural (Spark's transport to Python is
-  fork + pipe + Arrow IPC, so the buffer bytes must reach the pipe at least once); dropping
-  the first copy by serialising directly from Comet's vectors is tracked in
-  [#4294](https://github.com/apache/datafusion-comet/issues/4294). Even after that,
-  true zero-copy at the JVM boundary is blocked because Comet's source `FieldVector`s are
-  imported from native via Arrow C Data Interface (their buffers route `release` through FFI),
-  while Spark's destination IPC root is a child of `ArrowUtils.rootAllocator`. The two
-  reference managers cannot share buffers via `TransferPair`.
+  Spark supplies `large_string` and `large_binary` input columns with 8-byte offsets. Native

Review Comment:
   `large_string` corresponds to Arrow's `LargeUtf8` / `LargeStringArray`, but no new native mapping is needed for this PR because the large-type configuration still falls back to Spark.
   
   Supporting it on the accelerated path would require producing or converting inputs to actual 64-bit-offset arrays, not just changing the schema label. Comet already handles large string/binary output from Python on the JVM side; that is separate from widening the inputs sent to Python. I have left that additional support out of this PR and kept the fallback.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: serialize Python input directly from Comet Arrow vectors [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5368:
URL: https://github.com/apache/datafusion-comet/pull/5368#issuecomment-5456745987

   Thanks for the detailed review and the additional probes. Addressed the follow-ups in 1de4491c.
   
   I removed the ineffective Scala end-to-end test. CometMapInBatchSuite now keeps its five plan-rule checks, while actual worker execution is covered by test_pyarrow_udf.py, which exercises the accelerated and fallback plans and checks their results. All 83 Python cases passed locally on Spark 4.0.4, including source-batch turnover and chained UDFs.
   
   The three inline requests are addressed as described in their threads. I also added your isolated JVM measurements to the PR description with attribution, explicitly separated from the earlier Comet-versus-Spark end-to-end numbers and marked as not independently reproduced in this update.
   
   Validation: 12 focused JVM tests passed (7 Arrow serialization tests and 5 plan-rule tests), the Spark 4.0 Maven reactor packaged successfully, ScalaStyle and Spotless passed, and all 178 suite-registration checks passed. The FFI tests also passed with no pre-existing Maven temporary directory. The local Spark 4.1 rerun was blocked by dependency-mirror timeouts before the Spark-module compilation/tests; the existing CI matrix covers both Spark profiles.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: serialize Python input directly from Comet Arrow vectors [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5368:
URL: https://github.com/apache/datafusion-comet/pull/5368#discussion_r3816957727


##########
spark/src/main/spark-4.x/org/apache/spark/sql/execution/python/CometArrowPythonRunnerBase.scala:
##########
@@ -107,9 +109,8 @@ private[python] trait CometArrowPythonRunnerBase
       private val allocator =
         CometArrowAllocator.newChildAllocator(s"stdout writer for $pythonExec", 0, Long.MaxValue)
       private var currentGroup: Iterator[ColumnarBatch] = _
-      private var arrowWriter: ArrowStreamWriter = _
+      private var arrowWriter: CometArrowPythonRunnerBase.DirectArrowStreamWriter = _

Review Comment:
   Updated in 64a0114a7. The runner now uses a standard `ArrowStreamWriter` for the stream schema and end marker, and `DirectArrowStreamWriter` has been removed.



##########
spark/src/main/spark-4.x/org/apache/spark/sql/execution/python/CometArrowPythonRunnerBase.scala:
##########
@@ -167,49 +170,37 @@ private[python] trait CometArrowPythonRunnerBase
 
         val cometBatch = currentGroup.next()
         val startData = dataOut.size()
+        val sourceVectors = (0 until cometBatch.numCols()).map { i =>
+          cometBatch
+            .column(i)
+            .asInstanceOf[CometDecodedVector]
+            .getValueVector
+            .asInstanceOf[FieldVector]
+        }
 
         if (arrowWriter == null) {
-          // Build the destination struct root once, sized to the first batch's child fields.
+          // Build the schema-only struct root once from the first batch's child fields.
           // mapInArrow/mapInPandas exchange the columns under a single non-nullable struct.
           // Comet's FFI-imported vectors leave the Arrow Field name null, so restore the real
           // column names from the input schema (the worker reads columns by name, and shaded
-          // Arrow rejects a null field name). The field types and child structure are kept as-is
-          // so copyVector still walks the source and destination trees in lockstep. Keeping the
-          // type as-is also means a TimestampType reaches the worker with Comet's UTC time zone
+          // Arrow rejects a null field name). Keep the field types and child structure as-is so
+          // the advertised schema matches the source buffers. Keeping the type as-is also means
+          // a TimestampType reaches the worker with Comet's UTC time zone
           // rather than the session zone vanilla Spark would label it with; this is a documented
           // limitation (see pyarrow-udfs.md), not a value difference, since the stored instant is
           // identical.
           val childNames = inputStructType.fieldNames
-          val childFields = (0 until cometBatch.numCols()).map { i =>
-            val vecField =
-              cometBatch.column(i).asInstanceOf[CometDecodedVector].getValueVector.getField
-            renamed(vecField, childNames(i), forceNullable = true)
+          val childFields = sourceVectors.zipWithIndex.map { case (vector, i) =>
+            renamed(vector.getField, childNames(i), forceNullable = true)
           }
           startWriter(childFields, dataOut)
         }
 
-        var i = 0
-        while (i < cometBatch.numCols()) {
-          val src = cometBatch
-            .column(i)
-            .asInstanceOf[CometDecodedVector]
-            .getValueVector
-            .asInstanceOf[FieldVector]
-          val dst = structVec.getChildByOrdinal(i).asInstanceOf[FieldVector]
-          copyVector(src, dst)
-          i += 1
-        }
-        val numRows = cometBatch.numRows()
-        structVec.setValueCount(numRows)
-        // Mark every row of the struct non-null (all-1 validity). The validity buffer is freshly
-        // allocated and zero-initialised, so without this Python would see an all-null struct.
-        val validityBytes = (numRows + 7) / 8
-        Platform.setMemory(
-          structVec.getValidityBuffer.memoryAddress(),
-          0xff.toByte,
-          validityBytes)
-        writeRoot.setRowCount(numRows)
-        arrowWriter.writeBatch()
+        CometArrowPythonRunnerBase.writeDirectBatch(
+          arrowWriter,
+          sourceVectors,
+          cometBatch.numRows(),
+          allocator)

Review Comment:
   Updated in 64a0114a7. Each batch now gets a fresh `WriteChannel` over `dataOut` and is written with `MessageSerializer.serialize(writeChannel, wrappedBatch)`, matching the Spark 4.1 implementation. I kept the batch assembly and retain/release logic in a small `serializeBatch` helper so the allocator, nested/null, multi-batch, and failure-cleanup regression tests can exercise it directly. All five tests pass on both Spark 4.0 and Spark 4.1.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: serialize Python input directly from Comet Arrow vectors [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5368:
URL: https://github.com/apache/datafusion-comet/pull/5368#discussion_r3883361482


##########
spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala:
##########
@@ -112,10 +112,9 @@ case class EliminateRedundantTransitions(session: SparkSession)
       // 4.1+ matches the renamed `MapInArrowExec`.
       //
       // Falls back to vanilla Spark when `spark.sql.execution.arrow.useLargeVarTypes` is enabled:
-      // CometArrowPythonRunnerBase.copyVector does raw `setBytes` on each Arrow buffer, but Comet's
-      // source string/binary vectors always use 4-byte offsets while the destination root is
-      // allocated with 8-byte offsets when this conf is on. The buffer counts match but the
-      // offset width does not, so a direct memcpy would corrupt the offsets.
+      // Comet's source string/binary vectors use 4-byte offsets, while Spark expects 8-byte
+      // offsets when this conf is on. Direct IPC serialization cannot change the source vectors'
+      // physical layout, so forwarding them under Spark's widened schema would corrupt the stream.

Review Comment:
   Updated in 1de4491c. The code comment, user guide, and fallback test docstring now explain that the IPC schema matches the source vectors, so the stream is valid, but it does not provide the large input types requested by the configuration. The fallback stays in place to preserve that input contract.
   
   I also clarified that Comet already reads large_string / large_binary output through Utils and CometPlainVector; that output support does not widen the input vectors.
   
   



##########
spark/src/main/spark-4.x/org/apache/spark/sql/execution/python/CometArrowPythonRunnerBase.scala:
##########
@@ -167,49 +168,37 @@ private[python] trait CometArrowPythonRunnerBase
 
         val cometBatch = currentGroup.next()
         val startData = dataOut.size()
+        val sourceVectors = (0 until cometBatch.numCols()).map { i =>
+          cometBatch
+            .column(i)
+            .asInstanceOf[CometDecodedVector]
+            .getValueVector
+            .asInstanceOf[FieldVector]
+        }
 
         if (arrowWriter == null) {
-          // Build the destination struct root once, sized to the first batch's child fields.
+          // Build the schema-only struct root once from the first batch's child fields.
           // mapInArrow/mapInPandas exchange the columns under a single non-nullable struct.
           // Comet's FFI-imported vectors leave the Arrow Field name null, so restore the real
           // column names from the input schema (the worker reads columns by name, and shaded
-          // Arrow rejects a null field name). The field types and child structure are kept as-is
-          // so copyVector still walks the source and destination trees in lockstep. Keeping the
-          // type as-is also means a TimestampType reaches the worker with Comet's UTC time zone
+          // Arrow rejects a null field name). Keep the field types and child structure as-is so
+          // the advertised schema matches the source buffers. Keeping the type as-is also means
+          // a TimestampType reaches the worker with Comet's UTC time zone
           // rather than the session zone vanilla Spark would label it with; this is a documented
           // limitation (see pyarrow-udfs.md), not a value difference, since the stored instant is
           // identical.
           val childNames = inputStructType.fieldNames
-          val childFields = (0 until cometBatch.numCols()).map { i =>
-            val vecField =
-              cometBatch.column(i).asInstanceOf[CometDecodedVector].getValueVector.getField
-            renamed(vecField, childNames(i), forceNullable = true)
+          val childFields = sourceVectors.zipWithIndex.map { case (vector, i) =>
+            renamed(vector.getField, childNames(i), forceNullable = true)
           }
           startWriter(childFields, dataOut)

Review Comment:
   Added in 1de4491c. The writer snapshots the first batch's raw Arrow fields and requires every subsequent batch to match before serializing it. I compare raw fields with raw fields because the advertised fields have their names and nullability normalized.
   
   The full Spark 4.0 Python suite passed, including the multiple-batch, nested-source, and chained-UDF cases.
   
   



##########
spark/src/test/spark-4.x/org/apache/spark/sql/execution/python/CometArrowPythonRunnerSuite.scala:
##########
@@ -0,0 +1,329 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.execution.python
+
+import java.io.{ByteArrayInputStream, ByteArrayOutputStream, IOException}
+import java.nio.ByteBuffer
+import java.nio.channels.{Channels, WritableByteChannel}
+
+import scala.jdk.CollectionConverters._
+
+import org.scalatest.funsuite.AnyFunSuite
+import org.scalatest.matchers.should.Matchers
+
+import org.apache.arrow.memory.{BufferAllocator, RootAllocator}
+import org.apache.arrow.vector.{FieldVector, IntVector, NullVector, VarCharVector, VectorSchemaRoot}
+import org.apache.arrow.vector.complex.{ListVector, MapVector, StructVector}
+import org.apache.arrow.vector.ipc.{ArrowStreamReader, ArrowStreamWriter, WriteChannel}
+import org.apache.arrow.vector.types.pojo.{ArrowType, Field, FieldType, Schema}
+import org.apache.spark.sql.execution.python.CometArrowPythonRunnerBase.serializeBatch
+
+class CometArrowPythonRunnerSuite extends AnyFunSuite with Matchers {
+
+  private def withWriter(
+      childFields: Seq[Field],
+      allocator: BufferAllocator,
+      channel: WritableByteChannel)(f: WritableByteChannel => Unit): Unit = {
+    val structField = new Field(
+      "struct",
+      new FieldType(false, ArrowType.Struct.INSTANCE, null),
+      childFields.asJava)
+    val root = VectorSchemaRoot.create(new Schema(Seq(structField).asJava), allocator)
+    val writer = new ArrowStreamWriter(root, null, channel)
+    try {
+      writer.start()
+      f(channel)
+      writer.end()
+    } finally {
+      writer.close()
+      root.close()
+    }
+  }
+
+  private def withReader(bytes: Array[Byte])(f: ArrowStreamReader => Unit): Unit = {
+    val allocator = new RootAllocator(Long.MaxValue)
+    val reader = new ArrowStreamReader(new ByteArrayInputStream(bytes), allocator)
+    try {
+      f(reader)
+    } finally {
+      reader.close()
+      allocator.close()
+    }
+  }
+
+  test("direct batches retain borrowed buffers without copying them into the writer allocator") {

Review Comment:
   Added two FFI cases in 1de4491c: successful serialization and an injected write failure. Each exports with Data.exportVector, imports into a second allocator, verifies the shared data address, and closes the original vector before serialization so the imported ownership keeps the buffers alive.
   
   Both cases check balanced buffer reference counts, unchanged allocator usage, readable imported values, and zero outstanding allocations after the imported vector is closed. A 16 KiB payload is serialized with a 1 KiB writer allocator to rule out a payload copy. All seven Arrow tests pass on Spark 4.0.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: serialize Python input directly from Comet Arrow vectors [datafusion-comet]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #5368:
URL: https://github.com/apache/datafusion-comet/pull/5368#discussion_r3815605210


##########
spark/src/main/spark-4.x/org/apache/spark/sql/execution/python/CometArrowPythonRunnerBase.scala:
##########
@@ -107,9 +109,8 @@ private[python] trait CometArrowPythonRunnerBase
       private val allocator =
         CometArrowAllocator.newChildAllocator(s"stdout writer for $pythonExec", 0, Long.MaxValue)
       private var currentGroup: Iterator[ColumnarBatch] = _
-      private var arrowWriter: ArrowStreamWriter = _
+      private var arrowWriter: CometArrowPythonRunnerBase.DirectArrowStreamWriter = _

Review Comment:
   ```suggestion
         private var arrowWriter: ArrowStreamWriter = _
   ```



##########
spark/src/main/spark-4.x/org/apache/spark/sql/execution/python/CometArrowPythonRunnerBase.scala:
##########
@@ -167,49 +170,37 @@ private[python] trait CometArrowPythonRunnerBase
 
         val cometBatch = currentGroup.next()
         val startData = dataOut.size()
+        val sourceVectors = (0 until cometBatch.numCols()).map { i =>
+          cometBatch
+            .column(i)
+            .asInstanceOf[CometDecodedVector]
+            .getValueVector
+            .asInstanceOf[FieldVector]
+        }
 
         if (arrowWriter == null) {
-          // Build the destination struct root once, sized to the first batch's child fields.
+          // Build the schema-only struct root once from the first batch's child fields.
           // mapInArrow/mapInPandas exchange the columns under a single non-nullable struct.
           // Comet's FFI-imported vectors leave the Arrow Field name null, so restore the real
           // column names from the input schema (the worker reads columns by name, and shaded
-          // Arrow rejects a null field name). The field types and child structure are kept as-is
-          // so copyVector still walks the source and destination trees in lockstep. Keeping the
-          // type as-is also means a TimestampType reaches the worker with Comet's UTC time zone
+          // Arrow rejects a null field name). Keep the field types and child structure as-is so
+          // the advertised schema matches the source buffers. Keeping the type as-is also means
+          // a TimestampType reaches the worker with Comet's UTC time zone
           // rather than the session zone vanilla Spark would label it with; this is a documented
           // limitation (see pyarrow-udfs.md), not a value difference, since the stored instant is
           // identical.
           val childNames = inputStructType.fieldNames
-          val childFields = (0 until cometBatch.numCols()).map { i =>
-            val vecField =
-              cometBatch.column(i).asInstanceOf[CometDecodedVector].getValueVector.getField
-            renamed(vecField, childNames(i), forceNullable = true)
+          val childFields = sourceVectors.zipWithIndex.map { case (vector, i) =>
+            renamed(vector.getField, childNames(i), forceNullable = true)
           }
           startWriter(childFields, dataOut)
         }
 
-        var i = 0
-        while (i < cometBatch.numCols()) {
-          val src = cometBatch
-            .column(i)
-            .asInstanceOf[CometDecodedVector]
-            .getValueVector
-            .asInstanceOf[FieldVector]
-          val dst = structVec.getChildByOrdinal(i).asInstanceOf[FieldVector]
-          copyVector(src, dst)
-          i += 1
-        }
-        val numRows = cometBatch.numRows()
-        structVec.setValueCount(numRows)
-        // Mark every row of the struct non-null (all-1 validity). The validity buffer is freshly
-        // allocated and zero-initialised, so without this Python would see an all-null struct.
-        val validityBytes = (numRows + 7) / 8
-        Platform.setMemory(
-          structVec.getValidityBuffer.memoryAddress(),
-          0xff.toByte,
-          validityBytes)
-        writeRoot.setRowCount(numRows)
-        arrowWriter.writeBatch()
+        CometArrowPythonRunnerBase.writeDirectBatch(
+          arrowWriter,
+          sourceVectors,
+          cometBatch.numRows(),
+          allocator)

Review Comment:
   then use `MessageSerializer.serialize`
   ```suggestion
           // This root only borrows the Comet vectors; do not close it.
   	  val sourceRoot =
   	    new VectorSchemaRoot(sourceVectors.map(_.getField).asJava, sourceVectors.asJava, numRows)
   	  val sourceBatch = new VectorUnloader(sourceRoot).getRecordBatch
   	
   	  try {
   	    val validityBytes = (numRows.toLong + 7L) / 8L
   	    val structValidity = allocator.buffer(validityBytes)
   	
   	    try {
   	      if (validityBytes > 0) {
   	        structValidity.setOne(0L, validityBytes)
   	      }
   	      structValidity.writerIndex(validityBytes)
   	
   	      val nodes = new ArrayList[ArrowFieldNode](sourceBatch.getNodes.size() + 1)
   	      nodes.add(new ArrowFieldNode(numRows, 0))
   	      nodes.addAll(sourceBatch.getNodes)
   	
   	      val buffers = new ArrayList[ArrowBuf](sourceBatch.getBuffers.size() + 1)
   	      buffers.add(structValidity)
   	      buffers.addAll(sourceBatch.getBuffers)
   	
   	      val wrappedBatch = new ArrowRecordBatch(
   	        numRows,
   	        nodes,
   	        buffers,
   	        sourceBatch.getBodyCompression,
   	        sourceBatch.getVariadicBufferCounts,
   	        true)
   	
   	      try {
   	        MessageSerializer.serialize(
   	          new WriteChannel(Channels.newChannel(dataOut)),
   	          wrappedBatch)
   	      } finally {
   	        wrappedBatch.close()
   	      }
   	    } finally {
   	      structValidity.close()
   	    }
   	  } finally {
   	    sourceBatch.close()
   	  }
   	
   	  pythonMetrics("pythonDataSent") += dataOut.size() - startData
   	  true
   }
   ```
   
   
   
   ref: https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowInput.scala#L134-L166



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: serialize Python input directly from Comet Arrow vectors [datafusion-comet]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #5368:
URL: https://github.com/apache/datafusion-comet/pull/5368#discussion_r3883477086


##########
spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala:
##########
@@ -112,10 +112,10 @@ case class EliminateRedundantTransitions(session: SparkSession)
       // 4.1+ matches the renamed `MapInArrowExec`.
       //
       // Falls back to vanilla Spark when `spark.sql.execution.arrow.useLargeVarTypes` is enabled:
-      // CometArrowPythonRunnerBase.copyVector does raw `setBytes` on each Arrow buffer, but Comet's
-      // source string/binary vectors always use 4-byte offsets while the destination root is
-      // allocated with 8-byte offsets when this conf is on. The buffer counts match but the
-      // offset width does not, so a direct memcpy would corrupt the offsets.
+      // Native Comet string/binary vectors use 4-byte offsets. The IPC schema follows these

Review Comment:
   not sure why this was changed, it doesn't relate to PR 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: serialize Python input directly from Comet Arrow vectors [datafusion-comet]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #5368:
URL: https://github.com/apache/datafusion-comet/pull/5368#discussion_r3883464553


##########
spark/src/test/resources/pyspark/benchmark_pyarrow_udf.py:
##########
@@ -84,6 +84,8 @@ def _build_spark() -> SparkSession:
         .config("spark.plugins", "org.apache.spark.CometPlugin")
         .config("spark.comet.enabled", "true")
         .config("spark.comet.exec.enabled", "true")
+        # Keep Comet's scan and execution rules active with Spark's default shuffle manager.
+        .config("spark.comet.shuffle.enabled", "false")

Review Comment:
   why shuffle enabled false? Earlier we assessed that this mode should be only for testing mode



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org