You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/04/12 15:40:06 UTC
[PR] feat: avoid FFI import/export when passing batches between native plans [datafusion-comet]
andygrove opened a new pull request, #3930:
URL: https://github.com/apache/datafusion-comet/pull/3930
## Which issue does this PR close?
Closes #3925.
## Rationale for this change
When Comet has a native ShuffleWriter and a native child plan, batches get created in native code, then exported to JVM via Arrow FFI, then imported back to native for the shuffle writer. The JVM never reads the data, so the FFI round-trip is unnecessary overhead -- 4 JNI/FFI boundary crossings per batch with data copying.
## What changes are included in this PR?
Introduces a "batch stash" optimization that passes batches as opaque `u64` handles through the JVM instead of doing full Arrow FFI export/import. This reduces the per-batch overhead from 4 FFI boundary crossings to 2 lightweight JNI calls passing a single `long`.
**New components:**
- **`BatchStash`** (`native/core/src/execution/batch_stash.rs`) -- global `Mutex<HashMap<u64, RecordBatch>>` registry. The child plan stashes its output batch, and the shuffle writer's ScanExec retrieves it by handle.
- **`CometHandleBatchIterator`** (Java + Rust JNI bridge) -- passes handles between the two native execution contexts through the JVM.
- **`executePlanBatchHandle`** JNI function -- like `executePlan` but stashes the output batch instead of FFI-exporting it. Shared execution logic is extracted into `execute_plan_impl` with an `OutputMode` enum to avoid duplication.
- **`CometShuffleWriterInputIterator`** -- preserves the `CometExecIterator` reference through Spark's shuffle dependency RDD so the shuffle writer can detect native input.
**Modified components:**
- **`CometExecIterator`** -- gains stash mode (`enableStashMode()`, `nextHandle()`) for producing handles instead of `ColumnarBatch`.
- **`ScanExec`** -- gains `handle_mode` flag. When true, retrieves batches from the BatchStash instead of via Arrow FFI import.
- **`Scan` protobuf message** -- new `bool batch_stash_handle` field signals handle mode to the native planner.
- **`CometNativeShuffleWriter`** -- detects native child plans automatically and enables the stash path.
**Detection is automatic** -- no configuration needed. When a `CometExecIterator` feeds into a native shuffle writer, the optimization activates. Non-native child plans fall back to the existing FFI path.
## How are these changes tested?
- Existing `CometNativeShuffleSuite` (22 tests) passes -- validates correctness of the stash path since all native shuffle queries now use it.
- Existing `CometShuffleSuite` passes -- validates the fallback FFI path for columnar shuffle.
- New Rust unit tests for `BatchStash` (stash/take semantics, handle uniqueness, cleanup).
- Clippy clean, formatted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: avoid FFI import/export when passing batches between native plans [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #3930:
URL: https://github.com/apache/datafusion-comet/pull/3930#discussion_r3075019394
##########
native/core/src/execution/operators/scan.rs:
##########
@@ -532,6 +574,21 @@ impl Stream for ScanStream<'_> {
let maybe_batch = self.build_record_batch(columns, *num_rows);
Poll::Ready(Some(maybe_batch))
}
+ InputBatch::Complete(batch) => {
+ self.baseline_metrics.record_output(batch.num_rows());
+ let columns = batch.columns();
+ let num_rows = batch.num_rows();
+ if columns.len() == self.schema.fields().len() {
+ // Column counts match. Use build_record_batch to handle any
+ // type differences (e.g., timestamp timezone casting).
+ let maybe_batch = self.build_record_batch(columns, num_rows);
+ Poll::Ready(Some(maybe_batch))
+ } else {
+ // Column count mismatch (e.g., empty schema scan).
+ // Return the stashed batch as-is since it's already valid.
+ Poll::Ready(Some(Ok(batch.clone())))
Review Comment:
Updated. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: avoid FFI import/export when passing batches between native plans [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed pull request #3930: perf: avoid FFI import/export when passing batches between native plans
URL: https://github.com/apache/datafusion-comet/pull/3930
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: avoid FFI import/export when passing batches between native plans [datafusion-comet]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #3930:
URL: https://github.com/apache/datafusion-comet/pull/3930#discussion_r3091296157
##########
spark/src/main/scala/org/apache/spark/sql/comet/operators.scala:
##########
@@ -370,6 +370,29 @@ object CometExec {
encryptedFilePaths)
}
+ /**
+ * Create a CometExecIterator with pre-built input iterators (e.g., CometHandleBatchIterator).
+ * Bypasses the normal CometBatchIterator wrapping.
+ */
+ def getCometIteratorWithHandleInputs(
+ handleInputs: Array[Object],
Review Comment:
If this method only accepts `CometHandleBatchIterator` input, it would be better to set it to strong type. `handleInputs: Array[CometHandleBatchIterator]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: avoid FFI import/export when passing batches between native plans [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #3930:
URL: https://github.com/apache/datafusion-comet/pull/3930#discussion_r3070656227
##########
native/core/src/execution/batch_stash.rs:
##########
@@ -0,0 +1,121 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Global registry for passing RecordBatch values between native execution contexts
+//! via opaque u64 handles, without Arrow FFI serialization.
+
+use arrow::record_batch::RecordBatch;
+use once_cell::sync::Lazy;
+use std::collections::HashMap;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::Mutex;
+
+/// Counter for generating unique handles.
+static NEXT_HANDLE: AtomicU64 = AtomicU64::new(1);
+
+/// Global stash mapping handles to RecordBatch values.
+static STASH: Lazy<Mutex<HashMap<u64, RecordBatch>>> = Lazy::new(|| Mutex::new(HashMap::new()));
+
+/// Store a RecordBatch in the global stash and return a unique handle.
+pub(crate) fn stash(batch: RecordBatch) -> u64 {
+ let handle = NEXT_HANDLE.fetch_add(1, Ordering::Relaxed);
+ STASH
+ .lock()
+ .expect("batch_stash lock poisoned")
Review Comment:
Probably return an error instead of panic? i.e., `lock().unwrap_or_else(|e| e.into_inner())`.
##########
native/core/src/execution/operators/scan.rs:
##########
@@ -532,6 +574,21 @@ impl Stream for ScanStream<'_> {
let maybe_batch = self.build_record_batch(columns, *num_rows);
Poll::Ready(Some(maybe_batch))
}
+ InputBatch::Complete(batch) => {
+ self.baseline_metrics.record_output(batch.num_rows());
+ let columns = batch.columns();
+ let num_rows = batch.num_rows();
+ if columns.len() == self.schema.fields().len() {
+ // Column counts match. Use build_record_batch to handle any
+ // type differences (e.g., timestamp timezone casting).
+ let maybe_batch = self.build_record_batch(columns, num_rows);
Review Comment:
Why the batch is complete, why need to build_record_batch?
```
/// A complete RecordBatch retrieved from the BatchStash. Bypasses
/// `build_record_batch` since the batch is already fully formed.
Complete(RecordBatch),
```
##########
native/core/src/execution/operators/scan.rs:
##########
@@ -532,6 +574,21 @@ impl Stream for ScanStream<'_> {
let maybe_batch = self.build_record_batch(columns, *num_rows);
Poll::Ready(Some(maybe_batch))
}
+ InputBatch::Complete(batch) => {
+ self.baseline_metrics.record_output(batch.num_rows());
+ let columns = batch.columns();
+ let num_rows = batch.num_rows();
+ if columns.len() == self.schema.fields().len() {
+ // Column counts match. Use build_record_batch to handle any
+ // type differences (e.g., timestamp timezone casting).
+ let maybe_batch = self.build_record_batch(columns, num_rows);
Review Comment:
And in shuffle_scan.rs, Complete batch is returned directly. It looks inconsistent.
##########
native/core/src/execution/batch_stash.rs:
##########
@@ -0,0 +1,121 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Global registry for passing RecordBatch values between native execution contexts
+//! via opaque u64 handles, without Arrow FFI serialization.
+
+use arrow::record_batch::RecordBatch;
+use once_cell::sync::Lazy;
+use std::collections::HashMap;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::Mutex;
+
+/// Counter for generating unique handles.
+static NEXT_HANDLE: AtomicU64 = AtomicU64::new(1);
+
+/// Global stash mapping handles to RecordBatch values.
+static STASH: Lazy<Mutex<HashMap<u64, RecordBatch>>> = Lazy::new(|| Mutex::new(HashMap::new()));
Review Comment:
Does this must be a global one? Any leak risk? Do we need some cleanup to remove the content?
##########
native/core/src/execution/operators/scan.rs:
##########
@@ -532,6 +574,21 @@ impl Stream for ScanStream<'_> {
let maybe_batch = self.build_record_batch(columns, *num_rows);
Poll::Ready(Some(maybe_batch))
}
+ InputBatch::Complete(batch) => {
+ self.baseline_metrics.record_output(batch.num_rows());
+ let columns = batch.columns();
+ let num_rows = batch.num_rows();
+ if columns.len() == self.schema.fields().len() {
+ // Column counts match. Use build_record_batch to handle any
+ // type differences (e.g., timestamp timezone casting).
+ let maybe_batch = self.build_record_batch(columns, num_rows);
+ Poll::Ready(Some(maybe_batch))
+ } else {
+ // Column count mismatch (e.g., empty schema scan).
+ // Return the stashed batch as-is since it's already valid.
+ Poll::Ready(Some(Ok(batch.clone())))
Review Comment:
It might be more clear in semantics here to `take` it instead of `clone`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: avoid FFI import/export when passing batches between native plans [datafusion-comet]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #3930:
URL: https://github.com/apache/datafusion-comet/pull/3930#discussion_r3091301286
##########
spark/src/main/scala/org/apache/comet/CometExecIterator.scala:
##########
@@ -79,14 +80,22 @@ class CometExecIterator(
private val taskAttemptId = TaskContext.get().taskAttemptId
private val taskCPUs = TaskContext.get().cpus()
private val cometTaskMemoryManager = new CometTaskMemoryManager(id, taskAttemptId)
+ // When true, executePlan stashes output batches natively and returns handles
+ // instead of exporting via Arrow FFI. Used when output feeds a native ShuffleWriter.
+ private var stashMode: Boolean = false
+ private var pendingHandle: Long = -1L
// Build a mixed array of iterators: CometShuffleBlockIterator for shuffle
// scan indices, CometBatchIterator for regular scan indices.
- private val inputIterators: Array[Object] = inputs.zipWithIndex.map {
- case (_, idx) if shuffleBlockIterators.contains(idx) =>
- shuffleBlockIterators(idx).asInstanceOf[Object]
- case (iterator, _) =>
- new CometBatchIterator(iterator, nativeUtil).asInstanceOf[Object]
- }.toArray
+ private val inputIterators: Array[Object] = if (handleInputs.nonEmpty) {
+ handleInputs
Review Comment:
assert inputs.isEmpty
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: avoid FFI import/export when passing batches between native plans [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #3930:
URL: https://github.com/apache/datafusion-comet/pull/3930#issuecomment-4603471347
Closed in favor of https://github.com/apache/datafusion-comet/pull/4507
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: avoid FFI import/export when passing batches between native plans [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #3930:
URL: https://github.com/apache/datafusion-comet/pull/3930#issuecomment-4237253731
Before I dive too deep into the review, I think the PR description is a bit lacking in high level design and assumptions. For example, "Introduces a "batch stash" optimization that passes batches as opaque u64 handles through the JVM instead of doing full Arrow FFI export/import": so to steal some C++ lingo, we're basically stashing pointers into u64 and reinterpret casting as a pointer, and we can do this because we know more things about the life cycle of these batches than the Arrow FFI interface is designed for? The FFI interface provides safety guarantees and call back support, managing life cycle. We know things about the life cycle since we control the application on both sides of the FFI boundary, so we can bypass some of the protections that Arrow FFI otherwise affords?
Is that the high level context?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: avoid FFI import/export when passing batches between native plans [datafusion-comet]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #3930:
URL: https://github.com/apache/datafusion-comet/pull/3930#discussion_r3091153214
##########
native/core/src/execution/batch_stash.rs:
##########
@@ -0,0 +1,121 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Global registry for passing RecordBatch values between native execution contexts
+//! via opaque u64 handles, without Arrow FFI serialization.
+
+use arrow::record_batch::RecordBatch;
+use once_cell::sync::Lazy;
+use std::collections::HashMap;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::Mutex;
+
+/// Counter for generating unique handles.
+static NEXT_HANDLE: AtomicU64 = AtomicU64::new(1);
+
+/// Global stash mapping handles to RecordBatch values.
+static STASH: Lazy<Mutex<HashMap<u64, RecordBatch>>> = Lazy::new(|| Mutex::new(HashMap::new()));
Review Comment:
This could indeed lead to leaks if the task fails. We could perhaps maintain it in `execution_context` and clean it up during `releasePlan`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: avoid FFI import/export when passing batches between native plans [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #3930:
URL: https://github.com/apache/datafusion-comet/pull/3930#discussion_r3075011313
##########
native/core/src/execution/batch_stash.rs:
##########
@@ -0,0 +1,121 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Global registry for passing RecordBatch values between native execution contexts
+//! via opaque u64 handles, without Arrow FFI serialization.
+
+use arrow::record_batch::RecordBatch;
+use once_cell::sync::Lazy;
+use std::collections::HashMap;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::Mutex;
+
+/// Counter for generating unique handles.
+static NEXT_HANDLE: AtomicU64 = AtomicU64::new(1);
+
+/// Global stash mapping handles to RecordBatch values.
+static STASH: Lazy<Mutex<HashMap<u64, RecordBatch>>> = Lazy::new(|| Mutex::new(HashMap::new()));
+
+/// Store a RecordBatch in the global stash and return a unique handle.
+pub(crate) fn stash(batch: RecordBatch) -> u64 {
+ let handle = NEXT_HANDLE.fetch_add(1, Ordering::Relaxed);
+ STASH
+ .lock()
+ .expect("batch_stash lock poisoned")
Review Comment:
Updated. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: avoid FFI import/export when passing batches between native plans [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #3930:
URL: https://github.com/apache/datafusion-comet/pull/3930#issuecomment-4260641399
I moved this to draft while I address feedback and do more testing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: avoid FFI import/export when passing batches between native plans [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #3930:
URL: https://github.com/apache/datafusion-comet/pull/3930#issuecomment-4238690721
> Before I dive too deep into the review, I think the PR description is a bit lacking in high level design and assumptions. For example, "Introduces a "batch stash" optimization that passes batches as opaque u64 handles through the JVM instead of doing full Arrow FFI export/import": so to steal some C++ lingo, we're basically stashing pointers into u64 and reinterpret casting as a pointer, and we can do this because we know more things about the life cycle of these batches than the Arrow FFI interface is designed for? The FFI interface provides safety guarantees and call back support, managing life cycle. We know things about the life cycle since we control the application on both sides of the FFI boundary, so we can bypass some of the protections that Arrow FFI otherwise affords?
>
> Is that the high level context?
I added new sections to the PR description to try and summarize this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: avoid FFI import/export when passing batches between native plans [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #3930:
URL: https://github.com/apache/datafusion-comet/pull/3930#discussion_r3093740923
##########
native/core/src/execution/batch_stash.rs:
##########
@@ -0,0 +1,121 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Global registry for passing RecordBatch values between native execution contexts
+//! via opaque u64 handles, without Arrow FFI serialization.
+
+use arrow::record_batch::RecordBatch;
+use once_cell::sync::Lazy;
+use std::collections::HashMap;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::Mutex;
+
+/// Counter for generating unique handles.
+static NEXT_HANDLE: AtomicU64 = AtomicU64::new(1);
+
+/// Global stash mapping handles to RecordBatch values.
+static STASH: Lazy<Mutex<HashMap<u64, RecordBatch>>> = Lazy::new(|| Mutex::new(HashMap::new()));
Review Comment:
I am working on moving the state into the execution plan. I agree that this is a better approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: avoid FFI import/export when passing batches between native plans [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #3930:
URL: https://github.com/apache/datafusion-comet/pull/3930#discussion_r3069679337
##########
native/core/src/execution/operators/scan.rs:
##########
@@ -77,6 +77,9 @@ pub struct ScanExec {
baseline_metrics: BaselineMetrics,
/// Whether native code can assume ownership of batches that it receives
arrow_ffi_safe: bool,
+ /// When true, input comes from a CometHandleBatchIterator and batches are
+ /// retrieved from the BatchStash instead of via Arrow FFI import.
+ pub handle_mode: bool,
Review Comment:
We should probably combine `arrow_ffi_safe` and `handle_mode` and use an enum, but I would prefer to do that as a follow on PR. Something like:
```
enum TransferMode {
FfiSafe,
FfiUnsafe,
Handle
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: avoid FFI import/export when passing batches between native plans [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #3930:
URL: https://github.com/apache/datafusion-comet/pull/3930#discussion_r3070050339
##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -448,6 +448,16 @@ object CometConf extends ShimCometConf {
.intConf
.createWithDefault(1)
+ val COMET_SHUFFLE_BATCH_STASH_ENABLED: ConfigEntry[Boolean] =
Review Comment:
Added config for now just in case we discover bugs, but I plan on removing this config in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org