You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/19 09:09:29 UTC

(datafusion-comet) 01/01: perf: start bypass shuffle partition writers on small pages that grow (#7059)

This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-7059-5ed5f6f39bb9e946179a51bc6bf11a1a291cbef0
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git

commit 3c7bee7067e80c8c4dfb131ca4068b4da5f88fbe
Author: Andy Grove <ag...@apache.org>
AuthorDate: Mon Oct 19 09:09:11 2026 +0000

    perf: start bypass shuffle partition writers on small pages that grow (#7059)
    
    * perf: start bypass shuffle partition writers on small pages that grow
    
    The bypass-merge JVM columnar shuffle writer gives every partition its own
    CometDiskBlockWriter, and each one held a page of Spark's page size as soon
    as it buffered a row. A task's memory holds only 16 to 32 such pages, so a
    shuffle with more partitions than that made the writers spill each other
    about one row at a time: REPARTITION(64) over 3M rows wrote 2.48 GB in
    266 s instead of 156 MB in about a second.
    
    Each partition writer now starts a batch on a 64 KiB page and doubles each
    further page up to the old page size, so a partition costs kilobytes until
    its rows need more, and a batch takes a logarithmic number of pages, well
    below the 8,192 pages Spark allows a task. The sort-based writer keeps
    Spark's page size.
    
    Closes #7058.
    
    * test: cover growing pages under memory pressure; update manyPartitions comment
---
 docs/source/contributor-guide/jvm_shuffle.md       |   8 +-
 .../shuffle/CometBypassMergeSortShuffleWriter.java |  18 +-
 .../execution/shuffle/CometDiskBlockWriter.java    |  30 ++++
 .../sql/comet/execution/shuffle/SpillWriter.java   |  13 +-
 .../sql/benchmark/CometShuffleBenchmark.scala      |   7 +-
 .../shuffle/CometDiskBlockWriterSuite.scala        | 195 ++++++++++++++++++++-
 6 files changed, 258 insertions(+), 13 deletions(-)

diff --git a/docs/source/contributor-guide/jvm_shuffle.md b/docs/source/contributor-guide/jvm_shuffle.md
index badbaba19a..711e57ea9d 100644
--- a/docs/source/contributor-guide/jvm_shuffle.md
+++ b/docs/source/contributor-guide/jvm_shuffle.md
@@ -147,7 +147,7 @@ to protobuf and implements the partitioning logic in native code.
 
 Selection logic in `CometShuffleManager.shouldBypassMergeSort()`:
 
-- Uses bypass if partitions < threshold AND partitions × cores ≤ max threads
+- Uses bypass if partitions ≤ threshold AND partitions × cores ≤ max threads
 - Otherwise uses sort-based to avoid OOM from many concurrent writers
 
 ### Writers
@@ -220,6 +220,12 @@ writes the same Arrow IPC block format as native shuffle, so direct read applies
   `CometShuffleExternalSorter` spills at `spark.comet.shuffle.jvm.spillThreshold` alone, which
   defaults to `Int.MaxValue`, so on the sort path that trigger is effectively off by default.
 - `CometDiskBlockWriter` coordinates spilling across all partition writers (largest first)
+- The bypass writer's partition writers each hold a page as soon as they buffer a row. Each one
+  starts a batch on a 64 KiB page and doubles each further page, up to Spark's page size
+  (`spark.buffer.pageSize`, capped at 128 MB). Spark sizes its page so that a task fits only 16 to
+  32 of them, so pages of that size would make the writers spill each other about one row at a
+  time once a shuffle has more partitions than that. The sort-based writer has a single sorter per
+  task, and uses Spark's page size from the first page.
 
 ## Configuration
 
diff --git a/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/CometBypassMergeSortShuffleWriter.java b/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/CometBypassMergeSortShuffleWriter.java
index 24929648b1..96a7733dc0 100644
--- a/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/CometBypassMergeSortShuffleWriter.java
+++ b/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/CometBypassMergeSortShuffleWriter.java
@@ -169,12 +169,16 @@ final class CometBypassMergeSortShuffleWriter<K, V> extends ShuffleWriter<K, V>
 
       final String checksumAlgorithm = getChecksumAlgorithm(conf);
 
-      allocator =
-          CometShuffleMemoryAllocator.getInstance(
-              memoryManager,
-              Math.min(
-                  CometShuffleExternalSorter.MAXIMUM_PAGE_SIZE_BYTES,
-                  memoryManager.pageSizeBytes()));
+      // Every partition writer holds a page as soon as it buffers a row. Spark sizes its page so
+      // that a task fits only 16 to 32 of them, so pages of that size would make the writers spill
+      // each other about one row at a time once there are more partitions than that. Each writer
+      // starts small and grows towards Spark's page size instead.
+      final long maxPageSize =
+          Math.min(
+              CometShuffleExternalSorter.MAXIMUM_PAGE_SIZE_BYTES, memoryManager.pageSizeBytes());
+      final long initialPageSize =
+          Math.min(CometDiskBlockWriter.INITIAL_PAGE_SIZE_BYTES, maxPageSize);
+      allocator = CometShuffleMemoryAllocator.getInstance(memoryManager, initialPageSize);
 
       // This task's disk writers. Under memory pressure a writer spills its sibling writers in
       // this list, never those of other tasks.
@@ -189,6 +193,8 @@ final class CometBypassMergeSortShuffleWriter<K, V> extends ShuffleWriter<K, V>
             new CometDiskBlockWriter(
                 file,
                 allocator,
+                initialPageSize,
+                maxPageSize,
                 taskContext,
                 serializer,
                 schema,
diff --git a/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/CometDiskBlockWriter.java b/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/CometDiskBlockWriter.java
index 72ae0030b8..cbd9fa6379 100644
--- a/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/CometDiskBlockWriter.java
+++ b/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/CometDiskBlockWriter.java
@@ -75,9 +75,23 @@ public final class CometDiskBlockWriter {
   // Copied from Spark `org.apache.spark.shuffle.sort.PackedRecordPointer.MAXIMUM_PAGE_SIZE_BYTES`
   static final int MAXIMUM_PAGE_SIZE_BYTES = 1 << 27;
 
+  /**
+   * The size of the first page a partition writer takes for each batch. A task runs one writer per
+   * partition, and a writer holds at least one page as soon as it has a row, so this is what a
+   * partition costs before its rows need more. Each further page of the batch is twice the size of
+   * the one before, up to the writer's maximum page size.
+   */
+  static final long INITIAL_PAGE_SIZE_BYTES = 64 * 1024;
+
   /** The Comet allocator used to allocate pages. */
   private final CometShuffleMemoryAllocatorTrait allocator;
 
+  /** The size of the first page of each batch, which `allocator` rounds every request up to. */
+  private final long initialPageSize;
+
+  /** The largest page this writer asks for when it grows, unless a single row needs more. */
+  private final long maxPageSize;
+
   /** The serializer used to write rows to memory page. */
   private final SerializerInstance serializer;
 
@@ -127,6 +141,8 @@ public final class CometDiskBlockWriter {
   CometDiskBlockWriter(
       File file,
       CometShuffleMemoryAllocatorTrait allocator,
+      long initialPageSize,
+      long maxPageSize,
       TaskContext taskContext,
       SerializerInstance serializer,
       StructType schema,
@@ -137,6 +153,8 @@ public final class CometDiskBlockWriter {
     this.nativeLib = new Native();
     this.currentWriters = taskWriters;
     this.allocator = allocator;
+    this.initialPageSize = Math.min(initialPageSize, maxPageSize);
+    this.maxPageSize = maxPageSize;
     this.taskContext = taskContext;
     this.serializer = serializer;
     this.schema = schema;
@@ -325,6 +343,18 @@ public final class CometDiskBlockWriter {
       pageCursor += length;
     }
 
+    /**
+     * Starts each batch on a small page and doubles each further page, so a partition with few rows
+     * costs little, and a batch of {@code n} bytes takes O(log n) pages. Spark fails a task that
+     * holds more than 8,192 pages, and fixed small pages could reach that across many partitions.
+     * The pages held are this batch's, since writing a batch out frees them.
+     */
+    @Override
+    protected long nextPageRequest(int required) {
+      int doublings = Math.min(allocatedPages.size(), 32);
+      return Math.max(required, Math.min(maxPageSize, initialPageSize << doublings));
+    }
+
     int numRecords() {
       return rowPartition.getNumRows();
     }
diff --git a/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/SpillWriter.java b/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/SpillWriter.java
index 48cc2d41f9..5f20b4f7d8 100644
--- a/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/SpillWriter.java
+++ b/spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/SpillWriter.java
@@ -145,6 +145,15 @@ public abstract class SpillWriter {
    */
   protected abstract void spill(int required) throws IOException;
 
+  /**
+   * The number of bytes to request for this writer's next page, which must hold a record of {@code
+   * required} bytes. The allocator rounds the request up to its page size. A writer that would
+   * rather start small and grow overrides this.
+   */
+  protected long nextPageRequest(int required) {
+    return required;
+  }
+
   /**
    * Allocates more memory in order to insert an additional record. This will request additional
    * memory from the memory manager and spill if the requested memory can not be obtained.
@@ -160,7 +169,7 @@ public abstract class SpillWriter {
         || pageCursor + required > currentPage.getBaseOffset() + currentPage.size()) {
       // TODO: try to find space in previous pages
       try {
-        currentPage = allocator.allocate(required);
+        currentPage = allocator.allocate(nextPageRequest(required));
       } catch (SparkOutOfMemoryError error) {
         try {
           // Cannot allocate enough memory, spill
@@ -181,7 +190,7 @@ public abstract class SpillWriter {
   public void initialCurrentPage(int required) {
     assert (currentPage == null);
     try {
-      currentPage = allocator.allocate(required);
+      currentPage = allocator.allocate(nextPageRequest(required));
     } catch (SparkOutOfMemoryError e) {
       logger.error("Unable to acquire {} bytes of memory", required);
       throw e;
diff --git a/spark/src/test/scala/org/apache/spark/sql/benchmark/CometShuffleBenchmark.scala b/spark/src/test/scala/org/apache/spark/sql/benchmark/CometShuffleBenchmark.scala
index 0bb78d93f7..060d1c7028 100644
--- a/spark/src/test/scala/org/apache/spark/sql/benchmark/CometShuffleBenchmark.scala
+++ b/spark/src/test/scala/org/apache/spark/sql/benchmark/CometShuffleBenchmark.scala
@@ -57,9 +57,10 @@ object CometShuffleBenchmark extends CometBenchmarkBase {
   /**
    * High partition count for the groups that measure both a low and a high fan-out. This must
    * stay above Spark's `spark.shuffle.sort.bypassMergeThreshold` (200). Below that threshold the
-   * write path switches to `CometBypassMergeSortShuffleWriter`, which holds a page per partition
-   * from a JVM-wide pool, so a lower value both raises the shuffle memory requirement several
-   * fold and hides the JVM shuffle's degradation at high fan-out.
+   * write path switches to `CometBypassMergeSortShuffleWriter`, which writes a separate file per
+   * partition instead of sorting. The high fan-out group is meant to measure the sort-based
+   * writer, so a lower value would measure the bypass writer and hide the sort-based writer's
+   * degradation at high fan-out.
    */
   private val manyPartitions = 201
 
diff --git a/spark/src/test/scala/org/apache/spark/sql/comet/execution/shuffle/CometDiskBlockWriterSuite.scala b/spark/src/test/scala/org/apache/spark/sql/comet/execution/shuffle/CometDiskBlockWriterSuite.scala
index 11cdcc11f8..f18e50658c 100644
--- a/spark/src/test/scala/org/apache/spark/sql/comet/execution/shuffle/CometDiskBlockWriterSuite.scala
+++ b/spark/src/test/scala/org/apache/spark/sql/comet/execution/shuffle/CometDiskBlockWriterSuite.scala
@@ -20,7 +20,9 @@
 package org.apache.spark.sql.comet.execution.shuffle
 
 import java.io.File
-import java.util.{LinkedList => JLinkedList, Properties}
+import java.util.{LinkedList => JLinkedList, Map => JMap, Properties}
+
+import scala.collection.mutable.ArrayBuffer
 
 import org.scalatest.funsuite.AnyFunSuite
 
@@ -31,6 +33,7 @@ import org.apache.spark.shuffle.api.{ShuffleExecutorComponents, ShuffleMapOutput
 import org.apache.spark.shuffle.api.metadata.MapOutputCommitMessage
 import org.apache.spark.shuffle.comet.CometShuffleMemoryAllocator
 import org.apache.spark.shuffle.sort.SpillSorter
+import org.apache.spark.shuffle.sort.io.LocalDiskShuffleExecutorComponents
 import org.apache.spark.sql.catalyst.InternalRow
 import org.apache.spark.sql.catalyst.expressions.{UnsafeProjection, UnsafeRow}
 import org.apache.spark.sql.execution.UnsafeRowSerializer
@@ -91,6 +94,8 @@ class CometDiskBlockWriterSuite extends AnyFunSuite {
         new CometDiskBlockWriter(
           new File(tempDir, s"taskA-partition$partition"),
           allocatorA,
+          pageSize,
+          pageSize,
           taskContextA,
           serializer,
           schema,
@@ -104,6 +109,8 @@ class CometDiskBlockWriterSuite extends AnyFunSuite {
       val writerB = new CometDiskBlockWriter(
         fileB,
         allocatorB,
+        pageSize,
+        pageSize,
         taskContextB,
         serializer,
         schema,
@@ -216,6 +223,192 @@ class CometDiskBlockWriterSuite extends AnyFunSuite {
     }
   }
 
+  test("bypass writers with more partitions than full pages fit write without spilling") {
+    // Every partition writer holds a page as soon as it has a row. With pages of Spark's page
+    // size, a task fits only a few of them, so with more partitions than that each new page
+    // spilled a sibling writer holding about one row (#7058).
+    val conf = new SparkConf()
+      .setMaster("local[1]")
+      .setAppName("CometDiskBlockWriterSuite")
+      .set("spark.memory.offHeap.enabled", "true")
+      .set("spark.memory.offHeap.size", "1g")
+      .set("spark.buffer.pageSize", "1m")
+    val sc = new SparkContext(conf)
+    val memoryManager = new TestMemoryManager(conf)
+    // Room for eight pages of Spark's size, for 32 partitions whose rows need far less.
+    memoryManager.limit(8 * 1024 * 1024)
+    val tmm = new TaskMemoryManager(memoryManager, 0L)
+    try {
+      val taskContext = newTaskContext(tmm, 0L)
+      val partitions = 32
+      val rowsPerPartition = 100
+      val partitioner = new Partitioner {
+        override def numPartitions: Int = partitions
+        override def getPartition(key: Any): Int = key.asInstanceOf[Int] % partitions
+      }
+      val dep = new CometShuffleDependency[Int, UnsafeRow, UnsafeRow](
+        _rdd = sc.parallelize(Seq.empty[(Int, UnsafeRow)], 1),
+        partitioner = partitioner,
+        serializer = new UnsafeRowSerializer(1),
+        schema = Some(schema),
+        decodeTime = new SQLMetric("nsTiming"))
+      val shuffleExecutorComponents = new LocalDiskShuffleExecutorComponents(conf)
+      shuffleExecutorComponents.initializeExecutor(
+        sc.applicationId,
+        "driver",
+        JMap.of[String, String]())
+      val writer = new CometBypassMergeSortShuffleWriter[Int, UnsafeRow](
+        SparkEnv.get.blockManager,
+        tmm,
+        taskContext,
+        new CometBypassMergeSortShuffleHandle[Int, UnsafeRow](dep.shuffleId, dep),
+        0L,
+        conf,
+        taskContext.taskMetrics.shuffleWriteMetrics,
+        shuffleExecutorComponents,
+        null)
+
+      val toUnsafe = UnsafeProjection.create(schema)
+      // Round-robin, so that each partition gets a row before any gets a second one.
+      val rows = (0 until partitions * rowsPerPartition).iterator.map { i =>
+        (i % partitions, toUnsafe(InternalRow(new Array[Byte](100))))
+      }
+      writer.write(rows)
+      assert(writer.stop(true).isDefined)
+
+      // No writer had to make room for another, so each partition was written once, on close.
+      assert(taskContext.taskMetrics.memoryBytesSpilled == 0)
+      assert(taskContext.taskMetrics.diskBytesSpilled == 0)
+      val writeMetrics = taskContext.taskMetrics.shuffleWriteMetrics
+      assert(writeMetrics.recordsWritten == partitions * rowsPerPartition)
+      assert(writer.getPartitionLengths.forall(_ > 0))
+      assert(tmm.getMemoryConsumptionForThisTask == 0)
+    } finally {
+      sc.stop()
+      tmm.cleanUpAllAllocatedMemory()
+    }
+  }
+
+  test("a partition writer's pages double from a small first page up to the page size") {
+    val conf = new SparkConf()
+      .set("spark.memory.offHeap.enabled", "true")
+      .set("spark.memory.offHeap.size", "1g")
+    val memoryManager = new TestMemoryManager(conf)
+    memoryManager.limit(64L * 1024 * 1024)
+    val tmm = new TaskMemoryManager(memoryManager, 0L)
+    val taskContext = newTaskContext(tmm, 0L)
+    val kib = 1024L
+    val initialPageSize = 64 * kib
+    val maxPageSize = 1024 * kib
+    val allocator = CometShuffleMemoryAllocator.getInstance(tmm, initialPageSize)
+    val tempDir = Utils.createTempDir()
+    try {
+      val writer = new CometDiskBlockWriter(
+        new File(tempDir, "partition0"),
+        allocator,
+        initialPageSize,
+        maxPageSize,
+        taskContext,
+        new UnsafeRowSerializer(1).newInstance(),
+        schema,
+        new ShuffleWriteMetrics,
+        conf,
+        false,
+        new JLinkedList[CometDiskBlockWriter]())
+      val toUnsafe = UnsafeProjection.create(schema)
+      def insertOne(): Unit = writer.insertRow(toUnsafe(InternalRow(new Array[Byte](1000))), 0)
+
+      // The memory the writer holds after each page it takes.
+      val held = ArrayBuffer.empty[Long]
+      while (held.length < 7) {
+        insertOne()
+        if (!held.lastOption.contains(writer.getActiveMemoryUsage)) {
+          held += writer.getActiveMemoryUsage
+        }
+      }
+      // 64, 128, 256 and 512 KiB pages, then pages of the 1 MiB maximum.
+      assert(held == Seq(64, 192, 448, 960, 1984, 3008, 4032).map(_ * kib))
+
+      // Writing out a full batch frees its pages, and the next batch starts small again.
+      while (writer.getOutputRecords == 0) {
+        insertOne()
+      }
+      assert(writer.getActiveMemoryUsage == initialPageSize)
+      writer.close()
+      assert(allocator.getUsed == 0)
+    } finally {
+      Utils.deleteRecursively(tempDir)
+      tmm.cleanUpAllAllocatedMemory()
+    }
+  }
+
+  test("growing pages under real memory pressure spill but still write full batches") {
+    // The partitions' batches together do not fit, so writers must spill, and a growing writer
+    // whose next page is refused spills itself and restarts at the small page. Spilling is
+    // legitimate here, but it must not degrade to writing about one row per batch (#7058).
+    val conf = new SparkConf()
+      .set("spark.memory.offHeap.enabled", "true")
+      .set("spark.memory.offHeap.size", "1g")
+    val memoryManager = new TestMemoryManager(conf)
+    val kib = 1024L
+    val initialPageSize = 64 * kib
+    val maxPageSize = 1024 * kib
+    // 32 partitions fill about 8 MiB below, so this holds well under half of the batches.
+    memoryManager.limit(3 * 1024 * kib)
+    val tmm = new TaskMemoryManager(memoryManager, 0L)
+    val taskContext = newTaskContext(tmm, 0L)
+    val allocator = CometShuffleMemoryAllocator.getInstance(tmm, initialPageSize)
+    val tempDir = Utils.createTempDir()
+    try {
+      val partitions = 32
+      val rowsPerPartition = 2000
+      val currentWriters = new JLinkedList[CometDiskBlockWriter]()
+      val serializer = new UnsafeRowSerializer(1).newInstance()
+      val writers = (0 until partitions).map { partition =>
+        new CometDiskBlockWriter(
+          new File(tempDir, s"partition$partition"),
+          allocator,
+          initialPageSize,
+          maxPageSize,
+          taskContext,
+          serializer,
+          schema,
+          new ShuffleWriteMetrics,
+          conf,
+          false,
+          currentWriters)
+      }
+
+      val toUnsafe = UnsafeProjection.create(schema)
+      // Round-robin, so that each partition gets a row before any gets a second one. A write
+      // under memory pressure can be of any writer, so each insert checks all of them.
+      val written = Array.fill(partitions)(0L)
+      var batches = 0L
+      var rowsInBatches = 0L
+      for (_ <- 0 until rowsPerPartition; partition <- 0 until partitions) {
+        writers(partition).insertRow(toUnsafe(InternalRow(new Array[Byte](100))), 0)
+        for (i <- 0 until partitions if writers(i).getOutputRecords > written(i)) {
+          batches += 1
+          rowsInBatches += writers(i).getOutputRecords - written(i)
+          written(i) = writers(i).getOutputRecords
+        }
+      }
+      writers.foreach(_.close())
+
+      assert(writers.map(_.getOutputRecords).sum == partitions * rowsPerPartition)
+      assert(taskContext.taskMetrics.memoryBytesSpilled > 0)
+      assert(taskContext.taskMetrics.diskBytesSpilled > 0)
+      assert(tmm.getMemoryConsumptionForThisTask == 0)
+      assert(allocator.getUsed == 0)
+      // Writing a page's worth of rows is far more than one row. Pages of Spark's size, when
+      // every writer held one from its first row, gave about one row per batch here.
+      assert(batches > 0 && rowsInBatches / batches >= 100, s"$rowsInBatches rows in $batches")
+    } finally {
+      Utils.deleteRecursively(tempDir)
+      tmm.cleanUpAllAllocatedMemory()
+    }
+  }
+
   test("a fatal error during write() frees the task's buffered pages") {
     // Spark's ShuffleWriteProcessor only calls stop(false) when write() throws an Exception, so a
     // fatal error such as SparkOutOfMemoryError skips it. write() itself must therefore free the


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org