You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/19 09:09:29 UTC
(datafusion-comet) 01/01: perf: start bypass shuffle partition writers on small pages that grow (#7059)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-7059-5ed5f6f39bb9e946179a51bc6bf11a1a291cbef0
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 3c7bee7067e80c8c4dfb131ca4068b4da5f88fbe
Author: Andy Grove <ag...@apache.org>
AuthorDate: Mon Oct 19 09:09:11 2026 +0000
perf: start bypass shuffle partition writers on small pages that grow (#7059)
* perf: start bypass shuffle partition writers on small pages that grow
The bypass-merge JVM columnar shuffle writer gives every partition its own
CometDiskBlockWriter, and each one held a page of Spark's page size as soon
as it buffered a row. A task's memory holds only 16 to 32 such pages, so a
shuffle with more partitions than that made the writers spill each other
about one row at a time: REPARTITION(64) over 3M rows wrote 2.48 GB in
266 s instead of 156 MB in about a second.
Each partition writer now starts a batch on a 64 KiB page and doubles each
further page up to the old page size, so a partition costs kilobytes until
its rows need more, and a batch takes a logarithmic number of pages, well
below the 8,192 pages Spark allows a task. The sort-based writer keeps
Spark's page size.
Closes #7058.
* test: cover growing pages under memory pressure; update manyPartitions comment
---
docs/source/contributor-guide/jvm_shuffle.md | 8 +-
.../shuffle/CometBypassMergeSortShuffleWriter.java | 18 +-
.../execution/shuffle/CometDiskBlockWriter.java | 30 ++++
.../sql/comet/execution/shuffle/SpillWriter.java | 13 +-
.../sql/benchmark/CometShuffleBenchmark.scala | 7 +-
.../shuffle/CometDiskBlockWriterSuite.scala | 195 ++++++++++++++++++++-
6 files changed, 258 insertions(+), 13 deletions(-)
diff --git a/docs/source/contributor-guide/jvm_shuffle.md b/docs/source/contributor-guide/jvm_shuffle.md
index badbaba19a..711e57ea9d 100644
--- a/docs/source/contributor-guide/jvm_shuffle.md
+++ b/docs/source/contributor-guide/jvm_shuffle.md
@@ -147,7 +147,7 @@ to protobuf and implements the partitioning logic in native code.
Selection logic in `CometShuffleManager.shouldBypassMergeSort()`:
-- Uses bypass if partitions < threshold AND partitions × cores ≤ max threads
+- Uses bypass if partitions ≤ threshold AND partitions × cores ≤ max threads
- Otherwise uses sort-based to avoid OOM from many concurrent writers
### Writers
@@ -220,6 +220,12 @@ writes the same Arrow IPC block format as native shuffle, so direct read applies
`CometShuffleExternalSorter` spills at `spark.comet.shuffle.jvm.spillThreshold` alone, which
defaults to `Int.MaxValue`, so on the sort path that trigger is effectively off by default.
- `CometDiskBlockWriter` coordinates spilling across all partition writers (largest first)
+- The bypass writer's partition writers each hold a page as soon as they buffer a row. Each one
+ starts a batch on a 64 KiB page and doubles each further page, up to Spark's page size
+ (`spark.buffer.pageSize`, capped at 128 MB). Spark sizes its page so that a task fits only 16 to
+ 32 of them, so pages of that size would make the writers spill each other about one row at a
+ time once a shuffle has more partitions than that. The sort-based writer has a single sorter per
+ task, and uses Spark's page size from the first page.
## Configuration
diff --git a/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/CometBypassMergeSortShuffleWriter.java b/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/CometBypassMergeSortShuffleWriter.java
index 24929648b1..96a7733dc0 100644
--- a/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/CometBypassMergeSortShuffleWriter.java
+++ b/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/CometBypassMergeSortShuffleWriter.java
@@ -169,12 +169,16 @@ final class CometBypassMergeSortShuffleWriter<K, V> extends ShuffleWriter<K, V>
final String checksumAlgorithm = getChecksumAlgorithm(conf);
- allocator =
- CometShuffleMemoryAllocator.getInstance(
- memoryManager,
- Math.min(
- CometShuffleExternalSorter.MAXIMUM_PAGE_SIZE_BYTES,
- memoryManager.pageSizeBytes()));
+ // Every partition writer holds a page as soon as it buffers a row. Spark sizes its page so
+ // that a task fits only 16 to 32 of them, so pages of that size would make the writers spill
+ // each other about one row at a time once there are more partitions than that. Each writer
+ // starts small and grows towards Spark's page size instead.
+ final long maxPageSize =
+ Math.min(
+ CometShuffleExternalSorter.MAXIMUM_PAGE_SIZE_BYTES, memoryManager.pageSizeBytes());
+ final long initialPageSize =
+ Math.min(CometDiskBlockWriter.INITIAL_PAGE_SIZE_BYTES, maxPageSize);
+ allocator = CometShuffleMemoryAllocator.getInstance(memoryManager, initialPageSize);
// This task's disk writers. Under memory pressure a writer spills its sibling writers in
// this list, never those of other tasks.
@@ -189,6 +193,8 @@ final class CometBypassMergeSortShuffleWriter<K, V> extends ShuffleWriter<K, V>
new CometDiskBlockWriter(
file,
allocator,
+ initialPageSize,
+ maxPageSize,
taskContext,
serializer,
schema,
diff --git a/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/CometDiskBlockWriter.java b/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/CometDiskBlockWriter.java
index 72ae0030b8..cbd9fa6379 100644
--- a/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/CometDiskBlockWriter.java
+++ b/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/CometDiskBlockWriter.java
@@ -75,9 +75,23 @@ public final class CometDiskBlockWriter {
// Copied from Spark `org.apache.spark.shuffle.sort.PackedRecordPointer.MAXIMUM_PAGE_SIZE_BYTES`
static final int MAXIMUM_PAGE_SIZE_BYTES = 1 << 27;
+ /**
+ * The size of the first page a partition writer takes for each batch. A task runs one writer per
+ * partition, and a writer holds at least one page as soon as it has a row, so this is what a
+ * partition costs before its rows need more. Each further page of the batch is twice the size of
+ * the one before, up to the writer's maximum page size.
+ */
+ static final long INITIAL_PAGE_SIZE_BYTES = 64 * 1024;
+
/** The Comet allocator used to allocate pages. */
private final CometShuffleMemoryAllocatorTrait allocator;
+ /** The size of the first page of each batch, which `allocator` rounds every request up to. */
+ private final long initialPageSize;
+
+ /** The largest page this writer asks for when it grows, unless a single row needs more. */
+ private final long maxPageSize;
+
/** The serializer used to write rows to memory page. */
private final SerializerInstance serializer;
@@ -127,6 +141,8 @@ public final class CometDiskBlockWriter {
CometDiskBlockWriter(
File file,
CometShuffleMemoryAllocatorTrait allocator,
+ long initialPageSize,
+ long maxPageSize,
TaskContext taskContext,
SerializerInstance serializer,
StructType schema,
@@ -137,6 +153,8 @@ public final class CometDiskBlockWriter {
this.nativeLib = new Native();
this.currentWriters = taskWriters;
this.allocator = allocator;
+ this.initialPageSize = Math.min(initialPageSize, maxPageSize);
+ this.maxPageSize = maxPageSize;
this.taskContext = taskContext;
this.serializer = serializer;
this.schema = schema;
@@ -325,6 +343,18 @@ public final class CometDiskBlockWriter {
pageCursor += length;
}
+ /**
+ * Starts each batch on a small page and doubles each further page, so a partition with few rows
+ * costs little, and a batch of {@code n} bytes takes O(log n) pages. Spark fails a task that
+ * holds more than 8,192 pages, and fixed small pages could reach that across many partitions.
+ * The pages held are this batch's, since writing a batch out frees them.
+ */
+ @Override
+ protected long nextPageRequest(int required) {
+ int doublings = Math.min(allocatedPages.size(), 32);
+ return Math.max(required, Math.min(maxPageSize, initialPageSize << doublings));
+ }
+
int numRecords() {
return rowPartition.getNumRows();
}
diff --git a/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/SpillWriter.java b/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/SpillWriter.java
index 48cc2d41f9..5f20b4f7d8 100644
--- a/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/SpillWriter.java
+++ b/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/SpillWriter.java
@@ -145,6 +145,15 @@ public abstract class SpillWriter {
*/
protected abstract void spill(int required) throws IOException;
+ /**
+ * The number of bytes to request for this writer's next page, which must hold a record of {@code
+ * required} bytes. The allocator rounds the request up to its page size. A writer that would
+ * rather start small and grow overrides this.
+ */
+ protected long nextPageRequest(int required) {
+ return required;
+ }
+
/**
* Allocates more memory in order to insert an additional record. This will request additional
* memory from the memory manager and spill if the requested memory can not be obtained.
@@ -160,7 +169,7 @@ public abstract class SpillWriter {
|| pageCursor + required > currentPage.getBaseOffset() + currentPage.size()) {
// TODO: try to find space in previous pages
try {
- currentPage = allocator.allocate(required);
+ currentPage = allocator.allocate(nextPageRequest(required));
} catch (SparkOutOfMemoryError error) {
try {
// Cannot allocate enough memory, spill
@@ -181,7 +190,7 @@ public abstract class SpillWriter {
public void initialCurrentPage(int required) {
assert (currentPage == null);
try {
- currentPage = allocator.allocate(required);
+ currentPage = allocator.allocate(nextPageRequest(required));
} catch (SparkOutOfMemoryError e) {
logger.error("Unable to acquire {} bytes of memory", required);
throw e;
diff --git a/spark/src/test/scala/org/apache/spark/sql/benchmark/CometShuffleBenchmark.scala b/spark/src/test/scala/org/apache/spark/sql/benchmark/CometShuffleBenchmark.scala
index 0bb78d93f7..060d1c7028 100644
--- a/spark/src/test/scala/org/apache/spark/sql/benchmark/CometShuffleBenchmark.scala
+++ b/spark/src/test/scala/org/apache/spark/sql/benchmark/CometShuffleBenchmark.scala
@@ -57,9 +57,10 @@ object CometShuffleBenchmark extends CometBenchmarkBase {
/**
* High partition count for the groups that measure both a low and a high fan-out. This must
* stay above Spark's `spark.shuffle.sort.bypassMergeThreshold` (200). Below that threshold the
- * write path switches to `CometBypassMergeSortShuffleWriter`, which holds a page per partition
- * from a JVM-wide pool, so a lower value both raises the shuffle memory requirement several
- * fold and hides the JVM shuffle's degradation at high fan-out.
+ * write path switches to `CometBypassMergeSortShuffleWriter`, which writes a separate file per
+ * partition instead of sorting. The high fan-out group is meant to measure the sort-based
+ * writer, so a lower value would measure the bypass writer and hide the sort-based writer's
+ * degradation at high fan-out.
*/
private val manyPartitions = 201
diff --git a/spark/src/test/scala/org/apache/spark/sql/comet/execution/shuffle/CometDiskBlockWriterSuite.scala b/spark/src/test/scala/org/apache/spark/sql/comet/execution/shuffle/CometDiskBlockWriterSuite.scala
index 11cdcc11f8..f18e50658c 100644
--- a/spark/src/test/scala/org/apache/spark/sql/comet/execution/shuffle/CometDiskBlockWriterSuite.scala
+++ b/spark/src/test/scala/org/apache/spark/sql/comet/execution/shuffle/CometDiskBlockWriterSuite.scala
@@ -20,7 +20,9 @@
package org.apache.spark.sql.comet.execution.shuffle
import java.io.File
-import java.util.{LinkedList => JLinkedList, Properties}
+import java.util.{LinkedList => JLinkedList, Map => JMap, Properties}
+
+import scala.collection.mutable.ArrayBuffer
import org.scalatest.funsuite.AnyFunSuite
@@ -31,6 +33,7 @@ import org.apache.spark.shuffle.api.{ShuffleExecutorComponents, ShuffleMapOutput
import org.apache.spark.shuffle.api.metadata.MapOutputCommitMessage
import org.apache.spark.shuffle.comet.CometShuffleMemoryAllocator
import org.apache.spark.shuffle.sort.SpillSorter
+import org.apache.spark.shuffle.sort.io.LocalDiskShuffleExecutorComponents
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.{UnsafeProjection, UnsafeRow}
import org.apache.spark.sql.execution.UnsafeRowSerializer
@@ -91,6 +94,8 @@ class CometDiskBlockWriterSuite extends AnyFunSuite {
new CometDiskBlockWriter(
new File(tempDir, s"taskA-partition$partition"),
allocatorA,
+ pageSize,
+ pageSize,
taskContextA,
serializer,
schema,
@@ -104,6 +109,8 @@ class CometDiskBlockWriterSuite extends AnyFunSuite {
val writerB = new CometDiskBlockWriter(
fileB,
allocatorB,
+ pageSize,
+ pageSize,
taskContextB,
serializer,
schema,
@@ -216,6 +223,192 @@ class CometDiskBlockWriterSuite extends AnyFunSuite {
}
}
+ test("bypass writers with more partitions than full pages fit write without spilling") {
+ // Every partition writer holds a page as soon as it has a row. With pages of Spark's page
+ // size, a task fits only a few of them, so with more partitions than that each new page
+ // spilled a sibling writer holding about one row (#7058).
+ val conf = new SparkConf()
+ .setMaster("local[1]")
+ .setAppName("CometDiskBlockWriterSuite")
+ .set("spark.memory.offHeap.enabled", "true")
+ .set("spark.memory.offHeap.size", "1g")
+ .set("spark.buffer.pageSize", "1m")
+ val sc = new SparkContext(conf)
+ val memoryManager = new TestMemoryManager(conf)
+ // Room for eight pages of Spark's size, for 32 partitions whose rows need far less.
+ memoryManager.limit(8 * 1024 * 1024)
+ val tmm = new TaskMemoryManager(memoryManager, 0L)
+ try {
+ val taskContext = newTaskContext(tmm, 0L)
+ val partitions = 32
+ val rowsPerPartition = 100
+ val partitioner = new Partitioner {
+ override def numPartitions: Int = partitions
+ override def getPartition(key: Any): Int = key.asInstanceOf[Int] % partitions
+ }
+ val dep = new CometShuffleDependency[Int, UnsafeRow, UnsafeRow](
+ _rdd = sc.parallelize(Seq.empty[(Int, UnsafeRow)], 1),
+ partitioner = partitioner,
+ serializer = new UnsafeRowSerializer(1),
+ schema = Some(schema),
+ decodeTime = new SQLMetric("nsTiming"))
+ val shuffleExecutorComponents = new LocalDiskShuffleExecutorComponents(conf)
+ shuffleExecutorComponents.initializeExecutor(
+ sc.applicationId,
+ "driver",
+ JMap.of[String, String]())
+ val writer = new CometBypassMergeSortShuffleWriter[Int, UnsafeRow](
+ SparkEnv.get.blockManager,
+ tmm,
+ taskContext,
+ new CometBypassMergeSortShuffleHandle[Int, UnsafeRow](dep.shuffleId, dep),
+ 0L,
+ conf,
+ taskContext.taskMetrics.shuffleWriteMetrics,
+ shuffleExecutorComponents,
+ null)
+
+ val toUnsafe = UnsafeProjection.create(schema)
+ // Round-robin, so that each partition gets a row before any gets a second one.
+ val rows = (0 until partitions * rowsPerPartition).iterator.map { i =>
+ (i % partitions, toUnsafe(InternalRow(new Array[Byte](100))))
+ }
+ writer.write(rows)
+ assert(writer.stop(true).isDefined)
+
+ // No writer had to make room for another, so each partition was written once, on close.
+ assert(taskContext.taskMetrics.memoryBytesSpilled == 0)
+ assert(taskContext.taskMetrics.diskBytesSpilled == 0)
+ val writeMetrics = taskContext.taskMetrics.shuffleWriteMetrics
+ assert(writeMetrics.recordsWritten == partitions * rowsPerPartition)
+ assert(writer.getPartitionLengths.forall(_ > 0))
+ assert(tmm.getMemoryConsumptionForThisTask == 0)
+ } finally {
+ sc.stop()
+ tmm.cleanUpAllAllocatedMemory()
+ }
+ }
+
+ test("a partition writer's pages double from a small first page up to the page size") {
+ val conf = new SparkConf()
+ .set("spark.memory.offHeap.enabled", "true")
+ .set("spark.memory.offHeap.size", "1g")
+ val memoryManager = new TestMemoryManager(conf)
+ memoryManager.limit(64L * 1024 * 1024)
+ val tmm = new TaskMemoryManager(memoryManager, 0L)
+ val taskContext = newTaskContext(tmm, 0L)
+ val kib = 1024L
+ val initialPageSize = 64 * kib
+ val maxPageSize = 1024 * kib
+ val allocator = CometShuffleMemoryAllocator.getInstance(tmm, initialPageSize)
+ val tempDir = Utils.createTempDir()
+ try {
+ val writer = new CometDiskBlockWriter(
+ new File(tempDir, "partition0"),
+ allocator,
+ initialPageSize,
+ maxPageSize,
+ taskContext,
+ new UnsafeRowSerializer(1).newInstance(),
+ schema,
+ new ShuffleWriteMetrics,
+ conf,
+ false,
+ new JLinkedList[CometDiskBlockWriter]())
+ val toUnsafe = UnsafeProjection.create(schema)
+ def insertOne(): Unit = writer.insertRow(toUnsafe(InternalRow(new Array[Byte](1000))), 0)
+
+ // The memory the writer holds after each page it takes.
+ val held = ArrayBuffer.empty[Long]
+ while (held.length < 7) {
+ insertOne()
+ if (!held.lastOption.contains(writer.getActiveMemoryUsage)) {
+ held += writer.getActiveMemoryUsage
+ }
+ }
+ // 64, 128, 256 and 512 KiB pages, then pages of the 1 MiB maximum.
+ assert(held == Seq(64, 192, 448, 960, 1984, 3008, 4032).map(_ * kib))
+
+ // Writing out a full batch frees its pages, and the next batch starts small again.
+ while (writer.getOutputRecords == 0) {
+ insertOne()
+ }
+ assert(writer.getActiveMemoryUsage == initialPageSize)
+ writer.close()
+ assert(allocator.getUsed == 0)
+ } finally {
+ Utils.deleteRecursively(tempDir)
+ tmm.cleanUpAllAllocatedMemory()
+ }
+ }
+
+ test("growing pages under real memory pressure spill but still write full batches") {
+ // The partitions' batches together do not fit, so writers must spill, and a growing writer
+ // whose next page is refused spills itself and restarts at the small page. Spilling is
+ // legitimate here, but it must not degrade to writing about one row per batch (#7058).
+ val conf = new SparkConf()
+ .set("spark.memory.offHeap.enabled", "true")
+ .set("spark.memory.offHeap.size", "1g")
+ val memoryManager = new TestMemoryManager(conf)
+ val kib = 1024L
+ val initialPageSize = 64 * kib
+ val maxPageSize = 1024 * kib
+ // 32 partitions fill about 8 MiB below, so this holds well under half of the batches.
+ memoryManager.limit(3 * 1024 * kib)
+ val tmm = new TaskMemoryManager(memoryManager, 0L)
+ val taskContext = newTaskContext(tmm, 0L)
+ val allocator = CometShuffleMemoryAllocator.getInstance(tmm, initialPageSize)
+ val tempDir = Utils.createTempDir()
+ try {
+ val partitions = 32
+ val rowsPerPartition = 2000
+ val currentWriters = new JLinkedList[CometDiskBlockWriter]()
+ val serializer = new UnsafeRowSerializer(1).newInstance()
+ val writers = (0 until partitions).map { partition =>
+ new CometDiskBlockWriter(
+ new File(tempDir, s"partition$partition"),
+ allocator,
+ initialPageSize,
+ maxPageSize,
+ taskContext,
+ serializer,
+ schema,
+ new ShuffleWriteMetrics,
+ conf,
+ false,
+ currentWriters)
+ }
+
+ val toUnsafe = UnsafeProjection.create(schema)
+ // Round-robin, so that each partition gets a row before any gets a second one. A write
+ // under memory pressure can be of any writer, so each insert checks all of them.
+ val written = Array.fill(partitions)(0L)
+ var batches = 0L
+ var rowsInBatches = 0L
+ for (_ <- 0 until rowsPerPartition; partition <- 0 until partitions) {
+ writers(partition).insertRow(toUnsafe(InternalRow(new Array[Byte](100))), 0)
+ for (i <- 0 until partitions if writers(i).getOutputRecords > written(i)) {
+ batches += 1
+ rowsInBatches += writers(i).getOutputRecords - written(i)
+ written(i) = writers(i).getOutputRecords
+ }
+ }
+ writers.foreach(_.close())
+
+ assert(writers.map(_.getOutputRecords).sum == partitions * rowsPerPartition)
+ assert(taskContext.taskMetrics.memoryBytesSpilled > 0)
+ assert(taskContext.taskMetrics.diskBytesSpilled > 0)
+ assert(tmm.getMemoryConsumptionForThisTask == 0)
+ assert(allocator.getUsed == 0)
+ // Writing a page's worth of rows is far more than one row. Pages of Spark's size, when
+ // every writer held one from its first row, gave about one row per batch here.
+ assert(batches > 0 && rowsInBatches / batches >= 100, s"$rowsInBatches rows in $batches")
+ } finally {
+ Utils.deleteRecursively(tempDir)
+ tmm.cleanUpAllAllocatedMemory()
+ }
+ }
+
test("a fatal error during write() frees the task's buffered pages") {
// Spark's ShuffleWriteProcessor only calls stop(false) when write() throws an Exception, so a
// fatal error such as SparkOutOfMemoryError skips it. write() itself must therefore free the
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org