You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "goutamadwant (via GitHub)" <gi...@apache.org> on 2026/08/29 04:44:03 UTC
[PR] perf: add direct native reads for broadcast exchange [datafusion-comet]
goutamadwant opened a new pull request, #5538:
URL: https://github.com/apache/datafusion-comet/pull/5538
## Which issue does this PR close?
Closes #3762.
## Rationale for this change
Broadcast exchange input currently crosses the JVM twice before native execution: the JVM decompresses and deserializes Arrow IPC, then exports the resulting arrays through Arrow FFI. Comet can read the codec-prefixed IPC blocks directly and avoid that work, as it already does for direct shuffle reads.
## What changes are included in this PR?
- Add the experimental `spark.comet.exec.broadcast.directRead.enabled` configuration, disabled by default.
- Serialize direct-read broadcast batches as codec-prefixed LZ4 Arrow IPC while keeping them decodable by JVM consumers.
- Add a `BroadcastScan` protobuf operator and native `BroadcastScanExec` planning path that reuse the existing direct block protocol.
- Wire broadcast blocks through AQE and exchange-reuse boundaries without creating Arrow stream inputs.
- Keep direct-read mode in broadcast exchange equality and canonicalization so exchanges using incompatible serialization modes cannot be reused together.
- Preserve independently serialized blocks instead of coalescing a native broadcast beyond the JNI protocol's per-block `Integer.MAX_VALUE` limit.
- Preserve the legacy broadcast serialization and Arrow FFI path when the configuration is disabled.
- Close any already resolved streams and block iterators when input-slot resolution or native plan construction fails.
## How are these changes tested?
- `make core`
- `cargo check -p datafusion-comet`
- `cargo fmt --all -- --check`
- `cargo clippy -p datafusion-comet --all-targets -- -D warnings`
- `cargo test -p datafusion-comet test_broadcast_scan_has_distinct_plan_name`
- `./mvnw test -Dtest=none -Dsuites=org.apache.spark.sql.comet.util.UtilsSuite -DfailIfNoTests=false`
- `./mvnw test -Dtest=none -Dsuites=org.apache.comet.exec.CometNativeShuffleSuite -DfailIfNoTests=false`
- `./mvnw test -Dtest=none -Dsuites=org.apache.comet.exec.CometExecSuite -DfailIfNoTests=false`
The regression coverage checks legacy serialization with the feature disabled, native codec framing with the feature enabled, JVM decoding compatibility, AQE enabled and disabled, exchange identity across serialization modes, the direct block-size boundary without a large allocation, native-plan failure cleanup, empty build-side handling, and Spark-equivalent join answers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: add direct native reads for broadcast exchange [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5538:
URL: https://github.com/apache/datafusion-comet/pull/5538#issuecomment-5876671147
This is a light fully automated review since there are so many PRs open.
1. The direct path skips the invalid UTF-8 decoding that the FFI path does. `ScanExec::import_column` (`native/core/src/execution/operators/scan.rs:173`) runs every JVM-imported column through `decode_string_arrays`, while `BlockScanExec::get_next` (`native/core/src/execution/operators/shuffle_scan.rs:250-283`) decodes with `skip_validation` and passes the columns straight on. Shuffle blocks are encoded natively from strings that were already decoded, but broadcast blocks are written by Arrow Java from whatever the exchange's child produced. With `spark.comet.exec.localTableScan.enabled=true`, a broadcast join against `VALUES (1, CAST(X'80' AS STRING)) AS v(k, s)` serializes the `CometLocalTableScanExec` output directly, so the raw `0x80` byte reaches native unchecked. Tracing it through, `length(v.s)` above the join goes to `SparkLengthFunc`'s `chars().count()` and returns 0, where Spark and the legacy path both return 1. The in-memory cache and `sparkToColumnar` leaves build the
ir strings with the same `ArrowWriter`. Could the broadcast kind run `decode_string_arrays` on its columns, with a test along those lines?
2. `spark/src/test/scala/org/apache/spark/sql/benchmark/CometBroadcastDirectReadBenchmark.scala:65-72` builds both tables from `spark.range`. `RangeExec` only becomes Comet through `spark.comet.sparkToColumnar.enabled`, which defaults to `false` and isn't set here, and `CometExecRule` only converts the projection and the `BroadcastExchangeExec` above it when their children are native. So I think both cases time the same all-Spark plan. Would it make sense to write the tables to Parquet first, the way `CometExecBenchmark` uses `prepareTable`, and check that the plan contains a `BroadcastScan` before posting numbers? Once this is merged with `main`, the `getSparkSession` override will also need `spark.comet.exec.onHeap.enabled=true` (or off-heap memory), since #6195 stops Comet from loading at all without one of them.
3. `docs/source/contributor-guide/native_shuffle.md:236-259` still names `findShuffleScanIndices`, `shuffleScanIndices` and `ShuffleScanExec`, and says every other slot gets an `ArrowArrayStream`. `docs/source/contributor-guide/ffi.md:105-109` also describes a `ShuffleScan` slot as the only one that skips the FFI boundary. Could this PR update both for the broadcast path? It would also help if the new config's doc at `spark/src/main/scala/org/apache/comet/CometConf.scala:232` said that the block codec comes from `spark.comet.shuffle.compression.codec` and `spark.shuffle.compress`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: add direct native reads for broadcast exchange [datafusion-comet]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #5538:
URL: https://github.com/apache/datafusion-comet/pull/5538#discussion_r3999199901
##########
spark/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala:
##########
@@ -358,26 +400,36 @@ object Utils extends CometTypeShim with Logging {
* a single Arrow IPC stream.
*/
def coalesceBroadcastBatches(
- input: Iterator[ChunkedByteBuffer]): (Array[ChunkedByteBuffer], Long, Long) = {
+ input: Iterator[ChunkedByteBuffer],
+ nativeIpc: Boolean = false): (Array[ChunkedByteBuffer], Long, Long) = {
Review Comment:
Alongside TPC-H, could the benchmark include many small build-side batches and report producer serialization, driver coalescing, allocation/GC, and total query time with direct read off/on?
Coalescing still decodes each input stream on the JVM and serializes the combined batch again, so consumer-only timings would miss the allocation cost raised in the LZ4 thread. These measurements would also help decide whether coalescing needs a separate optimization.
##########
spark/src/main/scala/org/apache/spark/sql/comet/CometBroadcastExchangeExec.scala:
##########
@@ -85,7 +87,7 @@ case class CometBroadcastExchangeExec(
"number of coalesced rows for broadcast"))
override def doCanonicalize(): SparkPlan = {
- CometBroadcastExchangeExec(null, null, mode, child.canonicalized)
+ CometBroadcastExchangeExec(null, null, mode, child.canonicalized, directRead)
Review Comment:
Keeping `directRead` in canonicalization seems necessary: `decodeBatches` accepts both formats, but the native block reader requires codec-prefixed IPC. In non-AQE execution, exchange reuse runs after columnar conversion, so the consumer can already contain a `BroadcastScan` when its exchange is replaced.
Could we document this distinction and add a reuse regression covering a native join and JVM DPP consumer? The DPP construction sites should also carry the format choice explicitly so they match the join's exchange.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: add direct native reads for broadcast exchange [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5538:
URL: https://github.com/apache/datafusion-comet/pull/5538#discussion_r4043916900
##########
spark/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala:
##########
@@ -358,26 +400,36 @@ object Utils extends CometTypeShim with Logging {
* a single Arrow IPC stream.
*/
def coalesceBroadcastBatches(
- input: Iterator[ChunkedByteBuffer]): (Array[ChunkedByteBuffer], Long, Long) = {
+ input: Iterator[ChunkedByteBuffer],
+ nativeIpc: Boolean = false): (Array[ChunkedByteBuffer], Long, Long) = {
val buffers = input.filterNot(_.size == 0).toArray
if (buffers.isEmpty) {
return (Array.empty, 0L, 0L)
}
+ val totalInputBytes = buffers.foldLeft(0L) { (total, buffer) =>
+ if (Long.MaxValue - total < buffer.size) Long.MaxValue else total + buffer.size
+ }
+ if (shouldSkipDirectBroadcastCoalesce(nativeIpc, totalInputBytes)) {
Review Comment:
[P2] Check the coalesced output against the JNI block limit
The sum of the compressed inputs is not an upper bound on the compressed output: concatenating Arrow columns can remove the short-distance matches LZ4 used within each input batch. With direct read enabled, a large hinted broadcast can therefore pass this guard and still produce a single block that `CometBroadcastBlockIterator.hasNext()` rejects.
I reproduced this with the same Arrow 18.3 / LZ4 appender and writer operations on correlated BIGINT columns: the separate IPC streams total **551,523,846 bytes**, but the coalesced stream is **2,147,620,180 bytes**, above `Integer.MAX_VALUE`. The row count and resulting broadcast are below the existing row and default 8 GiB broadcast limits. The unchanged iterator rejects that measured size. This is a component reproduction, not a full Spark query run.
Please also validate the actual compressed output size and return the original blocks when it exceeds the JNI limit, or emit bounded independent IPC batches. The current test only exercises the input-size predicate, so it misses this case.
##########
spark/src/main/java/org/apache/comet/CometBroadcastBlockIterator.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+
+/** Provides codec-prefixed Arrow IPC broadcast blocks to native code via JNI. */
+public final class CometBroadcastBlockIterator extends CometShuffleBlockIterator {
+
+ private static final int INITIAL_BUFFER_SIZE = 128 * 1024;
+
+ private Iterator<ByteBuffer[]> blocks;
+ private ByteBuffer dataBuf = ByteBuffer.allocateDirect(INITIAL_BUFFER_SIZE);
+ private boolean closed = false;
+ private int currentBlockLength = 0;
+
+ public CometBroadcastBlockIterator(Iterator<ByteBuffer[]> blocks) {
+ // Native uses the same block-iterator JNI protocol for shuffle and broadcast inputs. The
+ // superclass stream is never read because every protocol method is overridden here.
+ super(new ByteArrayInputStream(new byte[0]));
+ this.blocks = blocks;
+ }
+
+ @Override
+ public int hasNext() throws IOException {
+ if (closed) {
+ return -1;
+ }
+
+ ByteBuffer[] block = null;
+ long blockSize = 0;
+ while (blocks.hasNext() && block == null) {
+ ByteBuffer[] candidate = blocks.next();
+ long candidateSize = 0;
+ for (ByteBuffer chunk : candidate) {
+ candidateSize += chunk.remaining();
+ }
+ if (candidateSize > 0) {
+ block = candidate;
+ blockSize = candidateSize;
+ }
+ }
+ if (block == null) {
+ close();
+ return -1;
+ }
+
+ if (blockSize > Integer.MAX_VALUE) {
+ throw new IllegalStateException(
+ "Native broadcast block size of "
+ + blockSize
+ + " exceeds the direct-read maximum of "
+ + Integer.MAX_VALUE
+ + " bytes");
+ }
+
+ currentBlockLength = (int) blockSize;
+ if (dataBuf.capacity() < currentBlockLength) {
+ long doubled = Math.max((long) dataBuf.capacity() * 2L, blockSize);
+ dataBuf = ByteBuffer.allocateDirect((int) Math.min(doubled, Integer.MAX_VALUE));
+ }
+
+ dataBuf.clear();
+ dataBuf.limit(currentBlockLength);
+ for (ByteBuffer chunk : block) {
+ dataBuf.put(chunk.duplicate());
+ }
+ return currentBlockLength;
+ }
+
+ @Override
+ public ByteBuffer getBuffer() {
+ return dataBuf;
+ }
+
+ @Override
+ public int getCurrentBlockLength() {
+ return currentBlockLength;
+ }
+
+ @Override
+ public void close() throws IOException {
+ if (!closed) {
+ closed = true;
+ blocks = null;
Review Comment:
[P2] Drop the direct broadcast buffer when the iterator closes
`close()` clears the source blocks but retains this class's `dataBuf`. After a coalesced broadcast is consumed, that buffer holds a complete compressed copy of the build relation per consumer task. `CometExecIterator` retains the block iterator while the join continues probing, so this direct allocation stays live alongside the decoded build data even after EOF; `super.close()` cannot clear the subclass buffer.
Compiling the unchanged iterator classes and feeding a 32 MiB block leaves **33,685,504 bytes** of direct memory after EOF and forced GC. Explicit close has the same result. Dropping only the subclass's `dataBuf` reference, while keeping the closed iterator alive, reclaims exactly **33,554,432 bytes**. This verifies retention, not an end-to-end OOM.
Please drop the staging-buffer reference on close/EOF and cover buffer growth followed by close. Native decoding finishes consuming the compressed bytes before the next `hasNext()` call, so the buffer is no longer needed at EOF.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: add direct native reads for broadcast exchange [datafusion-comet]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #5538:
URL: https://github.com/apache/datafusion-comet/pull/5538#issuecomment-5884775878
@andygrove @sunchao i addressed the review feedback in the latest push.. fixed broadcast UTF-8 handling and the Spark 4.2 LZ4 conflict, updated the docs, and made the benchmark verify both native paths. Pls review and let me know. thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: add direct native reads for broadcast exchange [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5538:
URL: https://github.com/apache/datafusion-comet/pull/5538#discussion_r4125143461
##########
spark/pom.xml:
##########
@@ -168,6 +168,25 @@ under the License.
<groupId>org.apache.arrow</groupId>
<artifactId>arrow-c-data</artifactId>
</dependency>
+ <!-- Direct native broadcast IPC uses the same codec-prefixed formats as native shuffle. -->
+ <dependency>
+ <groupId>org.lz4</groupId>
Review Comment:
[P2] Select the LZ4 coordinates per Spark profile. With `-Pspark-4.2`, Spark supplies `at.yawk.lz4:lz4-java:1.11.0`. This unconditional dependency adds `org.lz4:lz4-java:1.8.0` alongside it. Maven retains both because their group IDs differ, but they contain differing `net.jpountz.*` classes. Spark 4.2 should continue to build with direct reads disabled, yet root-reactor `validate` now fails at `BanDuplicateClasses`, blocking builds and tests for that profile regardless of the feature flag. Please use Spark 4.2's LZ4 fork for that profile and retain the older coordinates only for profiles that require them.
Evidence: Reproduced on the unchanged head with Maven 3.9.6/JDK 17 using root-reactor `mvn -Pspark-4.2 validate`. `comet-spark-spark4.2_2.13` fails with `Rule 2: org.codehaus.mojo.extraenforcer.dependencies.BanDuplicateClasses`, naming both LZ4 artifacts and duplicates including `LZ4FrameOutputStream.class` and `LZ4Factory.class`. An isolated reproduction using the published jars and the repository's exact rule also fails. Its control containing only `at.yawk.lz4:lz4-java:1.11.0` passes. Spark v4.2.0's root and core POMs confirm the fork dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: add direct native reads for broadcast exchange [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5538:
URL: https://github.com/apache/datafusion-comet/pull/5538#discussion_r3886893616
##########
spark/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala:
##########
@@ -49,6 +50,8 @@ import org.apache.comet.shims.CometTypeShim
import org.apache.comet.vector.CometVector
object Utils extends CometTypeShim with Logging {
+ private val NATIVE_IPC_LZ4_PREFIX = Array[Byte](0x4c, 0x5a, 0x34, 0x5f)
Review Comment:
Should the direct-read format follow the configured codec rather than always writing LZ4? Native `read_ipc_compressed` handles `SNAP`, `ZSTD` and `NONE` as well as `LZ4_`, and `spark.comet.exec.shuffle.compression.codec` already exists with a `zstd`/`lz4`/`snappy` check. As it stands someone who configured zstd silently gets LZ4 for broadcast.
##########
spark/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -229,6 +229,16 @@ object CometConf extends ShimCometConf {
createExecEnabledConfig("broadcastHashJoin", defaultValue = true)
val COMET_EXEC_BROADCAST_EXCHANGE_ENABLED: ConfigEntry[Boolean] =
createExecEnabledConfig("broadcastExchange", defaultValue = true)
+ val COMET_EXEC_BROADCAST_DIRECT_READ_ENABLED: ConfigEntry[Boolean] =
+ conf(s"$COMET_EXEC_CONFIG_PREFIX.broadcast.directRead.enabled")
Review Comment:
The shuffle equivalent is `spark.comet.shuffle.directRead.enabled`. #4986 deliberately moved these out of `spark.comet.exec.*` and kept the old name only as an alternative, so this lands back under the prefix we are moving away from. Should it be `spark.comet.broadcast.directRead.enabled`? Easier to settle now than to add a deprecation alias once it has shipped.
##########
spark/src/main/java/org/apache/comet/CometBroadcastBlockIterator.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+
+/** Provides codec-prefixed Arrow IPC broadcast blocks to native code via JNI. */
+public final class CometBroadcastBlockIterator extends CometShuffleBlockIterator {
Review Comment:
Extending `CometShuffleBlockIterator` and handing it a throwaway `ByteArrayInputStream` means the superclass constructor still allocates a 128KB direct `ByteBuffer` and a 16-byte header buffer per instance, neither of which this class ever touches, and there is one iterator per consumer partition.
Since every protocol method is overridden anyway, would it be cleaner to extract a small interface for the JNI block protocol and have both classes implement it? The native `comet_shuffle_block_iterator` bridge would keep working, and the inheritance would stop implying shared state that is not there.
##########
spark/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala:
##########
@@ -332,12 +348,38 @@ object Utils extends CometTypeShim with Logging {
return Iterator.empty
}
- // use Spark's compression codec (LZ4 by default) and not Comet's compression
- val codec = CompressionCodec.createCodec(SparkEnv.get.conf)
- val cbbis = bytes.toInputStream()
- val ins = new DataInputStream(codec.compressedInputStream(cbbis))
// batches are in Arrow IPC format
- new ArrowReaderIterator(Channels.newChannel(ins), source)
+ new ArrowReaderIterator(Channels.newChannel(compressedIpcInputStream(bytes)), source)
+ }
+
+ private def compressedIpcOutputStream(
+ output: ChunkedByteBufferOutputStream,
+ nativeIpc: Boolean): DataOutputStream = {
+ if (nativeIpc) {
+ output.write(NATIVE_IPC_LZ4_PREFIX)
+ new DataOutputStream(new net.jpountz.lz4.LZ4FrameOutputStream(output))
+ } else {
+ val codec = CompressionCodec.createCodec(SparkEnv.get.conf)
+ new DataOutputStream(codec.compressedOutputStream(output))
+ }
+ }
+
+ /**
+ * Opens either the existing Spark-codec stream or the codec-prefixed direct-read stream. This
+ * keeps non-native broadcast consumers compatible when direct read is enabled.
+ */
+ private def compressedIpcInputStream(bytes: ChunkedByteBuffer): InputStream = {
+ val input = new PushbackInputStream(bytes.toInputStream(), NATIVE_IPC_LZ4_PREFIX.length)
+ val prefix = new Array[Byte](NATIVE_IPC_LZ4_PREFIX.length)
+ val read = input.read(prefix)
Review Comment:
`input.read(prefix)` can come back with fewer than 4 bytes, and a short read here falls through to the Spark codec branch on what is actually a native-IPC stream. I do not think it can happen in practice because the chunks are 1MB, but `input.readNBytes(4)` would take the question off the table.
##########
native/core/src/execution/operators/shuffle_scan.rs:
##########
@@ -219,6 +275,14 @@ impl ShuffleScanExec {
.map(|col| unpack_dictionary(col))
.collect();
+ debug_assert_eq!(
Review Comment:
This looks redundant. `check_column_count` already returns a real error on the non-validating path, `validate_remote_schema` covers the validating one, and `cast_and_stamp_schema` checks again in `poll_next`. In release builds the assert compiles away entirely, and it is the only reason `kind` has to be threaded down into `get_next`. Can it come out?
##########
spark/src/main/scala/org/apache/spark/sql/comet/CometBroadcastExchangeExec.scala:
##########
@@ -64,7 +65,8 @@ case class CometBroadcastExchangeExec(
originalPlan: SparkPlan,
override val output: Seq[Attribute],
mode: BroadcastMode,
- override val child: SparkPlan)
+ override val child: SparkPlan,
+ directRead: Boolean = CometConf.COMET_EXEC_BROADCAST_DIRECT_READ_ENABLED.get())
Review Comment:
Two questions about the conf-reading default here.
There are two existing construction sites that do not pass this parameter and will silently inherit whatever the conf says at that moment: `CometExecRule.scala:527`, which builds the exchange for `CometSubqueryBroadcastExec`, and `CometPlanAdaptiveDynamicPruningFilters.scala:279`, whose comment says the fresh exchange has to have the same canonical form as the join's exchange so AQE's stage cache produces a `ReusedExchangeExec`. Now that `doCanonicalize` includes `directRead`, that reuse depends on a session conf resolving identically at two different points in planning, one of which is on an AQE thread. Dropping the default so the compiler forces each site to state its value would make the coupling explicit. My DPP test passes today so this is latent rather than live, but it is cheap to close off.
Separately, what does including `directRead` in `doCanonicalize` protect against? The serialized format is self-describing and `decodeBatches` sniffs the prefix, so two exchanges with different modes look reusable to me anyway.
##########
spark/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala:
##########
@@ -332,12 +348,38 @@ object Utils extends CometTypeShim with Logging {
return Iterator.empty
}
- // use Spark's compression codec (LZ4 by default) and not Comet's compression
- val codec = CompressionCodec.createCodec(SparkEnv.get.conf)
- val cbbis = bytes.toInputStream()
- val ins = new DataInputStream(codec.compressedInputStream(cbbis))
// batches are in Arrow IPC format
- new ArrowReaderIterator(Channels.newChannel(ins), source)
+ new ArrowReaderIterator(Channels.newChannel(compressedIpcInputStream(bytes)), source)
+ }
+
+ private def compressedIpcOutputStream(
+ output: ChunkedByteBufferOutputStream,
+ nativeIpc: Boolean): DataOutputStream = {
+ if (nativeIpc) {
+ output.write(NATIVE_IPC_LZ4_PREFIX)
+ new DataOutputStream(new net.jpountz.lz4.LZ4FrameOutputStream(output))
Review Comment:
`LZ4FrameOutputStream(output)` defaults to `BLOCKSIZE.SIZE_4MB`. I measured allocation against lz4-java 1.8.0 and it comes to about 12.6MB of heap per instance, versus about 135KB with `SIZE_64KB`. `serializeBatches` builds one of these per `ColumnarBatch`, and Spark's `LZ4CompressionCodec` on the path this replaces uses a 32KB block by default, so this is roughly a 90x jump in per-batch allocation.
The scaladoc on `coalesceBroadcastBatches` just below describes 200K tiny batches arriving from a 400-task, 500-partition shuffle, which is exactly the shape that would hurt. Could we pass `LZ4FrameOutputStream.BLOCKSIZE.SIZE_64KB` here? That fixes the read side at the same time, since `LZ4FrameInputStream` sizes its buffers from the frame's `BD` byte.
##########
spark/src/main/scala/org/apache/spark/sql/comet/CometBroadcastExchangeExec.scala:
##########
@@ -111,7 +113,11 @@ case class CometBroadcastExchangeExec(
private def getByteArrayRdd(plan: SparkPlan): RDD[(Long, ChunkedByteBuffer)] = {
plan.executeColumnar().mapPartitionsInternal { iter =>
- Utils.serializeBatches(iter)
+ if (directRead) {
+ Utils.serializeBroadcastBatches(iter)
Review Comment:
Reading `directRead` inside the lambda makes it capture the plan node. `mapPartitionsInternal` does not run `sc.clean`, and I confirmed against the compiled class that the closure is now `$anonfun$getByteArrayRdd$1(CometBroadcastExchangeExec, Iterator)` where before it captured nothing. That ships the whole exchange, including `originalPlan` and the child subtree, with every build-side task.
Could you hoist `val useNativeIpc = directRead` above the closure and branch on that instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: add direct native reads for broadcast exchange [datafusion-comet]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #5538:
URL: https://github.com/apache/datafusion-comet/pull/5538#discussion_r3999119823
##########
native/core/src/execution/operators/shuffle_scan.rs:
##########
@@ -47,11 +47,35 @@ use std::{
use super::scan::InputBatch;
-/// ShuffleScanExec reads compressed shuffle blocks from JVM via JNI and decodes them natively.
-/// Unlike ScanExec which receives Arrow arrays via FFI, ShuffleScanExec receives raw compressed
-/// bytes from CometShuffleBlockIterator and decodes them using read_ipc_compressed().
+/// Identifies the JVM source using the shared direct-block protocol.
+#[derive(Debug, Clone, Copy)]
+enum BlockScanKind {
+ Shuffle,
+ Broadcast,
+}
+
+impl BlockScanKind {
+ fn execution_plan_name(self) -> &'static str {
+ match self {
+ Self::Shuffle => "ShuffleScanExec",
+ Self::Broadcast => "BroadcastScanExec",
+ }
+ }
+
+ fn input_name(self) -> &'static str {
+ match self {
+ Self::Shuffle => "shuffle",
+ Self::Broadcast => "broadcast",
+ }
+ }
+}
+
+/// Reads codec-prefixed compressed Arrow IPC blocks from JVM via JNI and decodes them natively.
+/// Shuffle and broadcast scans share the same iterator protocol while retaining distinct plan
+/// names for explain output and diagnostics.
#[derive(Debug, Clone)]
pub struct ShuffleScanExec {
Review Comment:
```suggestion
pub struct BlockScanExec {
```
rename to `BlockScanExec` as it now reads codec-prefixed IPC blocks from either shuffle or broadcast.
##########
native/core/src/execution/operators/shuffle_scan.rs:
##########
@@ -79,6 +103,33 @@ impl ShuffleScanExec {
exec_context_id: i64,
input_source: Option<Arc<Global<JObject<'static>>>>,
data_types: Vec<DataType>,
+ ) -> Result<Self, CometError> {
+ Self::new_with_kind(
+ exec_context_id,
+ input_source,
+ data_types,
+ BlockScanKind::Shuffle,
+ )
+ }
+
+ pub fn new_broadcast(
+ exec_context_id: i64,
+ input_source: Option<Arc<Global<JObject<'static>>>>,
+ data_types: Vec<DataType>,
+ ) -> Result<Self, CometError> {
+ Self::new_with_kind(
+ exec_context_id,
+ input_source,
+ data_types,
+ BlockScanKind::Broadcast,
+ )
+ }
+
+ fn new_with_kind(
+ exec_context_id: i64,
+ input_source: Option<Arc<Global<JObject<'static>>>>,
+ data_types: Vec<DataType>,
+ kind: BlockScanKind,
Review Comment:
```suggestion
pub fn new(
exec_context_id: i64,
input_source: Option<Arc<Global<JObject<'static>>>>,
data_types: Vec<DataType>,
kind: BlockScanKind,
```
remove the selected constructor and only keep new\_with_kind and rename new_with_kind to new?
##########
native/core/src/execution/operators/shuffle_scan.rs:
##########
Review Comment:
```suggestion
pub fn new_with_shuffle(
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: add direct native reads for broadcast exchange [datafusion-comet]
Posted by "goutamadwant (via GitHub) via github" <gi...@datafusion.apache.org>.
goutamadwant commented on PR #5538:
URL: https://github.com/apache/datafusion-comet/pull/5538#issuecomment-6801114374
Thanks. I reproduced the buffer retention and have a local change that explicitly releases the compressed input buffer, including failure paths.
I also noticed #7016 now shares decoded broadcasts through CometBroadcastValue. Should direct reads only apply when shared decoding is disabled, or should this path be reworked around that ownership first? I have kept the changes local while that direction is clarified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org