You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by vi...@apache.org on 2026/10/13 19:58:45 UTC
(spark) branch branch-4.x updated: [SPARK-60112][CORE] Use identity equality for memory consumers
This is an automated email from the ASF dual-hosted git repository.
viirya pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 2bb44d100a1e [SPARK-60112][CORE] Use identity equality for memory consumers
2bb44d100a1e is described below
commit 2bb44d100a1e79a7c263ecfcd8d395c29a03edb5
Author: Liang-Chi Hsieh <vi...@gmail.com>
AuthorDate: Tue Oct 13 12:58:26 2026 -0700
[SPARK-60112][CORE] Use identity equality for memory consumers
### What changes were proposed in this pull request?
This PR makes memory consumers identity-based, in the base class and at the places that compare them:
1. `MemoryConsumer` now defines `equals`/`hashCode` by reference. Per the Scala spec (5.3.2), a case class does not generate `equals`/`hashCode` when a base class other than `AnyRef` already has a concrete definition, so every subclass compiled against this version, including a Scala case class, gets identity equality unless it explicitly overrides them. They are not `final`, so existing external subclasses that override `equals` keep compiling.
2. `TaskMemoryManager` tracks spillable consumers in an identity-based set (`Collections.newSetFromMap(new IdentityHashMap<>(4))`) instead of a `HashSet`, so every distinct instance is tracked, offered for spilling and reported, even if a subclass overrides `equals`. The set is only used for `add`, iteration and `clear`, and the spill loops already compare the requesting consumer by reference (`c == requestingConsumer`). This is the only hash-based collection keyed by `MemoryConsumer` [...]
3. `Spillable.spill` compares `trigger` with `this` by reference (`ne`) instead of Scala `!=`, which calls `equals`.
4. The `MemoryConsumer` class Javadoc describes the identity equality and what `TaskMemoryManager` guarantees, and the `trigger` parameter of `spill(size, trigger)` now states that it must be compared by reference.
5. The comment on the `final` identity overrides that #59282 added to `HybridQueue` no longer says that `TaskMemoryManager` uses a `HashSet`; it now points at the `MemoryConsumer` contract and explains that the overrides only make it final.
(1) is the root-cause fix. (2) and (3) keep the tracking and the self-trigger check correct for subclasses that override `equals` explicitly, and for case-class consumers compiled against an earlier Spark version, which keep their generated value equality until they are rebuilt.
### Why are the changes needed?
With value-based equality, an equal but distinct consumer is silently dropped from the `HashSet`. That instance:
- is never offered for spilling, so a memory request can fail to free memory that is actually spillable, including when the consumer itself is the requester (self-spill); and
- is missing from the per-consumer memory breakdown in `UNABLE_TO_ACQUIRE_MEMORY` errors, where its memory shows up as "not attributed to a specific consumer".
In addition, a consumer whose `spill` checks for a self-triggered spill with `equals` declines to spill when an equal but distinct consumer is the requester.
`HybridRowQueue` hit both problems because it was a Scala case class extending `MemoryConsumer` and `HybridQueue.spill` checked `trigger == this`. #59282 (SPARK-60071) fixed `HybridRowQueue` specifically, and is the change backported to the maintenance branches. As suggested by dongjoon-hyun in the #59282 review, this PR fixes the problem in one place, so that no future or external consumer can reintroduce it.
### Does this PR introduce _any_ user-facing change?
No for in-tree consumers: after #59282, none of them has value equality, so their behavior is unchanged.
For external `MemoryConsumer` subclasses: a Scala case class that relied on the generated value equality has identity equality instead when compiled against this version, and distinct instances that compare equal are now all tracked, offered for spilling and reported. A subclass that overrides `equals`/`hashCode` explicitly keeps its behavior. There is no API or configuration change. The existing `Spillable` subclasses (`ExternalSorter`, `ExternalAppendOnlyMap`) already had reference [...]
### How was this patch tested?
Added four tests to `TaskMemoryManagerSuite` using a test consumer that explicitly overrides `equals`/`hashCode`/`toString` with value semantics:
- `equalButDistinctConsumersAreAllSpilled`: another consumer's request spills both equal consumers.
- `equalButDistinctConsumerCanSelfSpill`: an equal consumer that was registered second can spill itself.
- `equalButDistinctConsumersAreAllSpilledForPageAllocation`: the page allocation recovery path (`spillConsumersForPageAllocation`) spills both.
- `equalButDistinctConsumersAreAllInMemoryConsumptionBreakdown`: both amounts are listed in the breakdown even though the consumers print the same.
Added `SpillableSuite`, which drives two equal `Spillable`s through `TaskMemoryManager`: when the second one grows past the memory limit, the first one is spilled and actually releases its memory, while a self-triggered spill is still skipped.
Added `MemoryConsumerSuite`, which checks that a Scala case class extending `MemoryConsumer` has identity equality, and drives two such consumers with equal arguments through `TaskMemoryManager`: when the second one requests more memory than is free, the first one spills even though its `spill` checks `trigger == this`.
Each of these fails without the corresponding change and passes with it. Also ran the `org.apache.spark.memory` suites, `UnsafeExternalSorterSuite`, `ShuffleExternalSorterSuite`, `BytesToBytesMap{On,Off}HeapSuite`, `ExternalSorterSuite`, `ExternalSorterSpillSuite`, `ExternalAppendOnlyMapSuite` and `RowQueueSuite`, plus checkstyle for `core` and scalastyle for `core` and `sql`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
Closes #59331 from viirya/SPARK-60112.
Authored-by: Liang-Chi Hsieh <vi...@gmail.com>
Signed-off-by: Liang-Chi Hsieh <vi...@gmail.com>
(cherry picked from commit 3869f62ae94f5fe56f338908f04c23f97006d260)
Signed-off-by: Liang-Chi Hsieh <vi...@gmail.com>
---
.../org/apache/spark/memory/MemoryConsumer.java | 29 ++++-
.../org/apache/spark/memory/TaskMemoryManager.java | 10 +-
.../apache/spark/util/collection/Spillable.scala | 4 +-
.../spark/memory/TaskMemoryManagerSuite.java | 117 +++++++++++++++++++++
.../apache/spark/memory/MemoryConsumerSuite.scala | 64 +++++++++++
.../spark/util/collection/SpillableSuite.scala | 96 +++++++++++++++++
.../spark/sql/execution/python/HybridQueue.scala | 5 +-
7 files changed, 317 insertions(+), 8 deletions(-)
diff --git a/core/src/main/java/org/apache/spark/memory/MemoryConsumer.java b/core/src/main/java/org/apache/spark/memory/MemoryConsumer.java
index 63ee13756124..3ec3c5cdf084 100644
--- a/core/src/main/java/org/apache/spark/memory/MemoryConsumer.java
+++ b/core/src/main/java/org/apache/spark/memory/MemoryConsumer.java
@@ -28,6 +28,14 @@ import org.apache.spark.unsafe.memory.MemoryBlock;
* A memory consumer of {@link TaskMemoryManager} that supports spilling.
*
* Note: this only supports allocation / spilling of Tungsten memory.
+ *
+ * Memory consumers have identity equality: {@link #equals} and {@link #hashCode} are defined here
+ * by reference, so a subclass, including a Scala case class when compiled against this version,
+ * does not get value equality unless it explicitly overrides them. {@link TaskMemoryManager} also
+ * tracks consumers by identity, even if a subclass overrides {@code equals}: distinct instances are
+ * never merged into one entry, so each of them can be offered for spilling and reported in the
+ * memory usage breakdown. How much memory a spill actually releases is decided by each consumer's
+ * {@link #spill(long, MemoryConsumer)}.
*/
public abstract class MemoryConsumer {
@@ -80,7 +88,10 @@ public abstract class MemoryConsumer {
* Note: today, this only frees Tungsten-managed pages.
*
* @param size the amount of memory should be released
- * @param trigger the MemoryConsumer that trigger this spilling
+ * @param trigger the MemoryConsumer that trigger this spilling. Compare it with this consumer
+ * by reference ({@code eq}/{@code ne} in Scala), not by {@code equals}: a
+ * distinct consumer that is equal to this one does not make this a
+ * self-triggered spill.
* @return the amount of released memory in bytes
*/
public abstract long spill(long size, MemoryConsumer trigger) throws IOException;
@@ -156,6 +167,22 @@ public abstract class MemoryConsumer {
used.getAndAdd(-size);
}
+ /**
+ * Memory consumers are compared by identity. This is not final so that existing subclasses which
+ * override it keep compiling, but its presence stops Scala case classes compiled against this
+ * version from generating value equality. A case class compiled against an earlier version keeps
+ * its generated equality until it is rebuilt.
+ */
+ @Override
+ public boolean equals(Object other) {
+ return this == other;
+ }
+
+ @Override
+ public int hashCode() {
+ return System.identityHashCode(this);
+ }
+
private void throwOom(final MemoryBlock page, final long required) {
long got = 0;
if (page != null) {
diff --git a/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java b/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java
index cefebcffa2fc..3f00e425f374 100644
--- a/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java
+++ b/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java
@@ -113,10 +113,12 @@ public class TaskMemoryManager {
final MemoryMode tungstenMemoryMode;
/**
- * Tracks spillable memory consumers.
+ * Tracks spillable memory consumers. Consumers are tracked by identity rather than by
+ * {@code equals}, so that distinct consumers are all offered for spilling and reported in the
+ * memory usage breakdown even if a subclass overrides {@code equals}.
*/
@GuardedBy("this")
- private final HashSet<MemoryConsumer> consumers;
+ private final Set<MemoryConsumer> consumers;
/**
* The amount of memory that is acquired but not used.
@@ -181,7 +183,9 @@ public class TaskMemoryManager {
this.memoryManager = memoryManager;
this.tungstenMemoryAllocator = tungstenMemoryAllocator;
this.taskAttemptId = taskAttemptId;
- this.consumers = new HashSet<>();
+ // A task rarely has more than a few consumers; the default expected size of IdentityHashMap
+ // would eagerly allocate a much larger table for every task.
+ this.consumers = Collections.newSetFromMap(new IdentityHashMap<>(4));
}
/**
diff --git a/core/src/main/scala/org/apache/spark/util/collection/Spillable.scala b/core/src/main/scala/org/apache/spark/util/collection/Spillable.scala
index 17713e83bb83..a448ef5792d7 100644
--- a/core/src/main/scala/org/apache/spark/util/collection/Spillable.scala
+++ b/core/src/main/scala/org/apache/spark/util/collection/Spillable.scala
@@ -116,7 +116,9 @@ private[spark] abstract class Spillable[C](taskMemoryManager: TaskMemoryManager)
* when there is not enough memory for the task.
*/
override def spill(size: Long, trigger: MemoryConsumer): Long = {
- if (trigger != this && taskMemoryManager.getTungstenMemoryMode == MemoryMode.ON_HEAP) {
+ // Compare by reference: a distinct consumer that is equal to this one must not be mistaken
+ // for a self-triggered spill.
+ if ((trigger ne this) && taskMemoryManager.getTungstenMemoryMode == MemoryMode.ON_HEAP) {
val isSpilled = forceSpill()
if (!isSpilled) {
0L
diff --git a/core/src/test/java/org/apache/spark/memory/TaskMemoryManagerSuite.java b/core/src/test/java/org/apache/spark/memory/TaskMemoryManagerSuite.java
index 44e17d040ea3..7f061d7f43f3 100644
--- a/core/src/test/java/org/apache/spark/memory/TaskMemoryManagerSuite.java
+++ b/core/src/test/java/org/apache/spark/memory/TaskMemoryManagerSuite.java
@@ -18,6 +18,7 @@
package org.apache.spark.memory;
import java.io.IOException;
+import java.util.Objects;
import java.util.concurrent.atomic.AtomicInteger;
import org.junit.jupiter.api.Assertions;
@@ -187,6 +188,33 @@ public class TaskMemoryManagerSuite {
}
}
+ /**
+ * A consumer with value-based equality and string representation: two instances on the same
+ * manager and memory mode are equal, share a hash code and print the same, even though they
+ * track memory independently.
+ */
+ private static final class ValueEqualConsumer extends TestMemoryConsumer {
+ ValueEqualConsumer(TaskMemoryManager memoryManager) {
+ super(memoryManager);
+ }
+
+ @Override
+ public boolean equals(Object other) {
+ return other instanceof ValueEqualConsumer that &&
+ taskMemoryManager == that.taskMemoryManager && getMode() == that.getMode();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(taskMemoryManager, getMode());
+ }
+
+ @Override
+ public String toString() {
+ return "ValueEqualConsumer";
+ }
+ }
+
@Test
public void leakedPageMemoryIsDetected() {
final TaskMemoryManager manager = new TaskMemoryManager(
@@ -861,6 +889,95 @@ public class TaskMemoryManagerSuite {
Assertions.assertEquals(80, c1.getUsed()); // not spilled
}
+ @Test
+ public void equalButDistinctConsumersAreAllSpilled() {
+ final TestMemoryManager memoryManager = new TestMemoryManager(new SparkConf());
+ memoryManager.limit(100);
+ final TaskMemoryManager manager = new TaskMemoryManager(memoryManager, 0);
+
+ ValueEqualConsumer c1 = new ValueEqualConsumer(manager);
+ ValueEqualConsumer c2 = new ValueEqualConsumer(manager);
+ Assertions.assertEquals(c1, c2);
+ TestMemoryConsumer c3 = new TestMemoryConsumer(manager);
+ c1.use(50);
+ c2.use(50);
+
+ // Both equal consumers must be offered for spilling to satisfy the request.
+ c3.use(100);
+ Assertions.assertEquals(0, c1.getUsed());
+ Assertions.assertEquals(0, c2.getUsed());
+ Assertions.assertEquals(100, c3.getUsed());
+
+ c3.free(100);
+ Assertions.assertEquals(0, manager.cleanUpAllAllocatedMemory());
+ }
+
+ @Test
+ public void equalButDistinctConsumerCanSelfSpill() {
+ final TestMemoryManager memoryManager = new TestMemoryManager(new SparkConf());
+ memoryManager.limit(100);
+ final TaskMemoryManager manager = new TaskMemoryManager(memoryManager, 0);
+
+ ValueEqualConsumer c1 = new ValueEqualConsumer(manager);
+ ValueEqualConsumer c2 = new ValueEqualConsumer(manager);
+ c1.use(50);
+ c1.free(50);
+ c2.use(100);
+
+ // The requesting consumer is spilled last, but it must still be spilled when no other
+ // consumer holds memory, even if an equal consumer was registered first.
+ c2.use(50);
+ Assertions.assertEquals(0, c1.getUsed());
+ Assertions.assertEquals(50, c2.getUsed());
+
+ c2.free(50);
+ Assertions.assertEquals(0, manager.cleanUpAllAllocatedMemory());
+ }
+
+ @Test
+ public void equalButDistinctConsumersAreAllSpilledForPageAllocation() {
+ final TestMemoryManager memoryManager = new TestMemoryManager(new SparkConf());
+ memoryManager.limit(5120);
+ final TestAllocator allocator = new TestAllocator(1);
+ final TaskMemoryManager manager = new TaskMemoryManager(memoryManager, 0, allocator);
+ ValueEqualConsumer c1 = new ValueEqualConsumer(manager);
+ ValueEqualConsumer c2 = new ValueEqualConsumer(manager);
+ final PageAllocatingConsumer requestingConsumer = new PageAllocatingConsumer(manager, 4096);
+ c1.use(512);
+ c2.use(512);
+
+ // The grant succeeds without spilling, so only the allocator-failure recovery path spills.
+ final MemoryBlock page = requestingConsumer.allocate(4096);
+ Assertions.assertNotNull(page);
+ Assertions.assertEquals(0, c1.getUsed());
+ Assertions.assertEquals(0, c2.getUsed());
+
+ requestingConsumer.freeAllocatedPage(page);
+ Assertions.assertEquals(0, manager.cleanUpAllAllocatedMemory());
+ }
+
+ @Test
+ public void equalButDistinctConsumersAreAllInMemoryConsumptionBreakdown() {
+ final TestMemoryManager memoryManager = new TestMemoryManager(new SparkConf());
+ memoryManager.limit(100);
+ final TaskMemoryManager manager = new TaskMemoryManager(memoryManager, 0);
+
+ ValueEqualConsumer c1 = new ValueEqualConsumer(manager);
+ ValueEqualConsumer c2 = new ValueEqualConsumer(manager);
+ c1.use(20);
+ c2.use(30);
+
+ String breakdown = manager.getMemoryConsumptionBreakdown();
+ Assertions.assertTrue(breakdown.contains("ValueEqualConsumer: 20.0 B"), breakdown);
+ Assertions.assertTrue(breakdown.contains("ValueEqualConsumer: 30.0 B"), breakdown);
+ Assertions.assertFalse(
+ breakdown.contains("(not attributed to a specific consumer)"), breakdown);
+
+ c1.free(20);
+ c2.free(30);
+ Assertions.assertEquals(0, manager.cleanUpAllAllocatedMemory());
+ }
+
@Test
public void memoryConsumptionBreakdownIsEmptyWhenThereIsNoMemoryToReport() {
final TestMemoryManager memoryManager = new TestMemoryManager(new SparkConf());
diff --git a/core/src/test/scala/org/apache/spark/memory/MemoryConsumerSuite.scala b/core/src/test/scala/org/apache/spark/memory/MemoryConsumerSuite.scala
new file mode 100644
index 000000000000..453276536236
--- /dev/null
+++ b/core/src/test/scala/org/apache/spark/memory/MemoryConsumerSuite.scala
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.memory
+
+import org.apache.spark.{SparkConf, SparkFunSuite}
+
+class MemoryConsumerSuite extends SparkFunSuite {
+
+ test("case class consumers have identity equality") {
+ val taskMemoryManager = new TaskMemoryManager(new TestMemoryManager(new SparkConf()), 0)
+ val consumer = CaseClassConsumer(taskMemoryManager, 1)
+ val other = CaseClassConsumer(taskMemoryManager, 1)
+
+ // The case class does not generate value-based equals/hashCode because MemoryConsumer
+ // already defines them.
+ assert(consumer != other)
+ assert(consumer == consumer)
+ assert(consumer.hashCode === System.identityHashCode(consumer))
+ }
+
+ test("case class consumers with equal arguments spill for each other") {
+ val memoryManager = new TestMemoryManager(new SparkConf())
+ memoryManager.limit(100)
+ val taskMemoryManager = new TaskMemoryManager(memoryManager, 0)
+ val consumer = CaseClassConsumer(taskMemoryManager, 1)
+ val other = CaseClassConsumer(taskMemoryManager, 1)
+
+ consumer.use(60)
+ // Only 40 bytes are free, so TaskMemoryManager asks `consumer` to spill on behalf of `other`,
+ // which must not be mistaken for a self-triggered spill.
+ other.use(60)
+ assert(consumer.getUsed === 0)
+ assert(other.getUsed === 60)
+
+ other.free(60)
+ assert(taskMemoryManager.cleanUpAllAllocatedMemory() === 0)
+ }
+}
+
+private case class CaseClassConsumer(tmm: TaskMemoryManager, id: Int)
+ extends TestMemoryConsumer(tmm) {
+
+ override def spill(size: Long, trigger: MemoryConsumer): Long = {
+ // Written with `==` on purpose, to check that it calls the identity equals of
+ // MemoryConsumer. Real consumers should still compare `trigger` with `eq`, as the
+ // MemoryConsumer#spill Javadoc says.
+ if (trigger == this) 0L else super.spill(size, trigger)
+ }
+}
diff --git a/core/src/test/scala/org/apache/spark/util/collection/SpillableSuite.scala b/core/src/test/scala/org/apache/spark/util/collection/SpillableSuite.scala
new file mode 100644
index 000000000000..dd4c6038af35
--- /dev/null
+++ b/core/src/test/scala/org/apache/spark/util/collection/SpillableSuite.scala
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.collection
+
+import org.mockito.Mockito.{mock, when}
+
+import org.apache.spark.{SparkConf, SparkEnv, SparkFunSuite}
+import org.apache.spark.internal.config.SHUFFLE_SPILL_INITIAL_MEM_THRESHOLD
+import org.apache.spark.memory.{TaskMemoryManager, TestMemoryManager}
+
+class SpillableSuite extends SparkFunSuite {
+
+ private val initialMemoryThreshold = 100L
+
+ private var memoryManager: TestMemoryManager = _
+ private var taskMemoryManager: TaskMemoryManager = _
+
+ override protected def beforeEach(): Unit = {
+ super.beforeEach()
+ val conf = new SparkConf().set(SHUFFLE_SPILL_INITIAL_MEM_THRESHOLD, initialMemoryThreshold)
+ val env = mock(classOf[SparkEnv])
+ when(env.conf).thenReturn(conf)
+ SparkEnv.set(env)
+ memoryManager = new TestMemoryManager(conf)
+ taskMemoryManager = new TaskMemoryManager(memoryManager, 0)
+ }
+
+ override protected def afterEach(): Unit = {
+ try {
+ SparkEnv.set(null)
+ assert(taskMemoryManager.cleanUpAllAllocatedMemory() === 0)
+ } finally {
+ super.afterEach()
+ }
+ }
+
+ test("spill is not skipped when triggered by a distinct but equal consumer") {
+ memoryManager.limit(1000)
+ val spillable = new ValueEqualSpillable(taskMemoryManager)
+ val other = new ValueEqualSpillable(taskMemoryManager)
+ assert(spillable == other)
+
+ // Growing past the initial threshold acquires 2 * 150 - 100 = 200 bytes without spilling.
+ assert(!spillable.grow(150))
+ assert(spillable.getUsed === 200)
+
+ // `other` requests 2 * 500 - 100 = 900 bytes but only 800 are free, so TaskMemoryManager asks
+ // `spillable` to spill on behalf of `other`, which must release its memory.
+ assert(!other.grow(500))
+ assert(spillable.forceSpillCount === 1)
+ assert(spillable.getUsed === 0)
+ assert(spillable.memoryBytesSpilled === 200)
+ assert(other.getUsed === 900)
+
+ // A spill triggered by the consumer itself is still skipped.
+ assert(other.spill(Long.MaxValue, other) === 0)
+ assert(other.forceSpillCount === 0)
+
+ other.releaseMemory()
+ }
+}
+
+/** A [[Spillable]] whose instances are all equal to each other. */
+private class ValueEqualSpillable(taskMemoryManager: TaskMemoryManager)
+ extends Spillable[Unit](taskMemoryManager) {
+
+ var forceSpillCount = 0
+
+ def grow(currentMemory: Long): Boolean = maybeSpill((), currentMemory)
+
+ override protected def spill(collection: Unit): Unit = {}
+
+ override protected def forceSpill(): Boolean = {
+ forceSpillCount += 1
+ true
+ }
+
+ override def equals(other: Any): Boolean = other.isInstanceOf[ValueEqualSpillable]
+
+ override def hashCode(): Int = 0
+}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/HybridQueue.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/HybridQueue.scala
index c7f9f8817d42..1660068f1383 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/HybridQueue.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/HybridQueue.scala
@@ -62,9 +62,8 @@ abstract class HybridQueue[T, Q <: Queue[T]](
// exposed for testing
private[python] def numQueues(): Int = queues.size()
- // A memory consumer must use identity equality. TaskMemoryManager tracks consumers in a
- // HashSet, so two queues that compare equal would be tracked (and offered for spilling) as one.
- // These are final so that no subclass, including a case class, can switch to value equality.
+ // MemoryConsumer already defines identity equality (see its class Javadoc). These overrides
+ // only make it final, so that no subclass of HybridQueue can switch to value equality.
final override def equals(other: Any): Boolean = other match {
case ref: AnyRef => this eq ref
case _ => false
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org