You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "cloud-fan (via GitHub)" <gi...@apache.org> on 2026/09/24 02:36:56 UTC
[PR] [WIP][CORE] Fix NPE in BytesToBytesMap after failed reset during spill [spark]
cloud-fan opened a new pull request, #59009:
URL: https://github.com/apache/spark/pull/59009
### What changes were proposed in this pull request?
Restore the pointer array lazily when `BytesToBytesMap.reset()` fails to allocate its replacement array. Keep the map's empty-state bookkeeping consistent before reset-time allocation and distinguish a failed reset from terminal `free()` or destructive iteration. Add regression coverage for on-heap and off-heap maps, including repeated allocation failures and concurrent `safeLookup()` callers.
### Why are the changes needed?
`reset()` frees the current pointer array before allocating a new one. If allocation throws `SparkOutOfMemoryError`, the map retains a null array and stale page state. A later lookup can then throw `NullPointerException`, and an append could target a freed page. A transient allocation failure should either recover to an empty map on the next lookup or report another allocation failure.
### Does this PR introduce _any_ user-facing change?
Yes. A hash aggregation that encounters this failed-reset state can recover if memory becomes available; if memory remains unavailable, it reports `SparkOutOfMemoryError` instead of an unrelated `NullPointerException`.
### How was this patch tested?
The new regression cases in `AbstractBytesToBytesMapSuite` run through both `BytesToBytesMapOnHeapSuite` and `BytesToBytesMapOffHeapSuite`.
`git diff --check` and changed-file line length and ASCII checks passed. The focused SBT suites
could not start locally because the environment cannot resolve the Maven hosts needed for
`scala2-sbt-bridge:2.13.18` and `jackson-annotations:2.22`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac (version not recorded), Codex (GPT-6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59757][CORE] Harden BytesToBytesMap after failed reset allocation [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #59009:
URL: https://github.com/apache/spark/pull/59009#discussion_r4106948456
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -587,12 +604,50 @@ public Location lookup(Object keyBase, long keyOffset, int keyLength, int hash)
return loc;
}
+ /**
+ * Restores the hash array after a {@link #reset()} freed it but its eager reallocation failed
+ * with an OOM, leaving {@code longArray == null}. A reset() empties the map, so re-allocating at
+ * {@code initialCapacity} here is correct. If memory is still unavailable this throws a
+ * SparkOutOfMemoryError instead of exposing the null array.
+ */
+ private void restoreArrayAfterFailedReset() {
+ if (freed) {
+ throw new IllegalStateException("BytesToBytesMap has already been freed");
+ }
+ if (longArrayRecoveryRequired) {
+ // Only the failed-reset state is recoverable here: the map is empty, so a fresh
+ // initial-capacity array is correct. `destructiveIterator != null` means destructive
+ // iteration has begun, after which map operations are illegal.
+ //
+ // allocate() acquires the TaskMemoryManager monitor while this monitor is held. The memory
+ // manager may call spill() while holding its monitor, so spill() and anything reachable
+ // from it must not acquire this monitor.
+ synchronized (this) {
Review Comment:
Addressed in 60eea7415ee. Concurrent lazy recovery and all map synchronization were removed, so lookup no longer allocates while holding a map monitor and there is no new lock-order dependency.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -587,12 +604,50 @@ public Location lookup(Object keyBase, long keyOffset, int keyLength, int hash)
return loc;
}
+ /**
+ * Restores the hash array after a {@link #reset()} freed it but its eager reallocation failed
+ * with an OOM, leaving {@code longArray == null}. A reset() empties the map, so re-allocating at
+ * {@code initialCapacity} here is correct. If memory is still unavailable this throws a
+ * SparkOutOfMemoryError instead of exposing the null array.
+ */
+ private void restoreArrayAfterFailedReset() {
+ if (freed) {
Review Comment:
Addressed in 60eea7415ee. Both volatile flags are gone. The hot path now performs one ordinary read of a single lifecycle state; map mutation is still prohibited during concurrent safeLookup() calls.
##########
core/src/test/scala/org/apache/spark/memory/TestMemoryManager.scala:
##########
@@ -33,27 +34,40 @@ class TestMemoryManager(conf: SparkConf)
private var available = Long.MaxValue
@GuardedBy("this")
private val memoryForTask = mutable.HashMap[Long, Long]().withDefaultValue(0L)
+ private val beforeNextExecutionMemoryGrant = new AtomicReference[Runnable]()
+
+ /** Runs one test action before the next grant, outside this manager's accounting lock. */
Review Comment:
Obsolete after 60eea7415ee. The TestMemoryManager hook was removed entirely along with the concurrent-recovery test.
##########
core/src/test/scala/org/apache/spark/memory/TestMemoryManager.scala:
##########
@@ -33,27 +34,40 @@ class TestMemoryManager(conf: SparkConf)
private var available = Long.MaxValue
@GuardedBy("this")
private val memoryForTask = mutable.HashMap[Long, Long]().withDefaultValue(0L)
+ private val beforeNextExecutionMemoryGrant = new AtomicReference[Runnable]()
+
+ /** Runs one test action before the next grant, outside this manager's accounting lock. */
+ def runBeforeNextExecutionMemoryGrant(action: Runnable): Unit = {
Review Comment:
Addressed in 60eea7415ee by removing the shared TestMemoryManager hook entirely. The new explicit-reset tests need no allocation interleaving hook.
##########
core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java:
##########
@@ -875,4 +879,362 @@ public void freeAfterFailedReset() {
}
}
+ @Test
+ public void lookupAfterFreeDoesNotRestoreArray() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ map.free();
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ map.reset();
+ assertEquals(0, map.getNumDataPages());
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ assertThrows(
+ IllegalStateException.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void lookupAfterDestructiveIterationDoesNotRestoreArray() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ map.destructiveIterator();
+ memoryManager.limit(PAGE_SIZE_BYTES);
+
+ assertThrows(
+ IllegalStateException.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void resetAfterDestructiveIterationFails() {
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ map.destructiveIterator();
+ assertThrows(IllegalStateException.class, map::reset);
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void arrayAccessorsRecoverAfterFailedReset() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ assertEquals(512L, map.getArray().size());
+
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ assertEquals(256, map.maxNumKeysIndex());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapAfterFailedReset() {
+ // A reset() whose eager longArray reallocation OOMs leaves the map with longArray == null (and,
+ // because the trailing reset() bookkeeping never runs, a stale currentPage). A subsequent
+ // lookup then NPEs in safeLookup (this.longArray is null), and a later append would write into
+ // a freed page. The map must instead recover to a usable empty state.
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ final long[] key = new long[]{1L};
+ final long[] value = new long[]{42L};
+ BytesToBytesMap.Location loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ loc.append(key, Platform.LONG_ARRAY_OFFSET, 8, value, Platform.LONG_ARRAY_OFFSET, 8);
+
+ // Force OOM on the reallocation inside reset(), leaving longArray == null.
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+
+ // The map must recover: lookup must not NPE, it must report an empty map, and it must be
+ // usable again (append into a fresh page, then read the value back).
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ Assertions.assertFalse(loc.isDefined());
+ Assertions.assertEquals(0, map.numValues());
+ Assertions.assertTrue(
+ loc.append(key, Platform.LONG_ARRAY_OFFSET, 8, value, Platform.LONG_ARRAY_OFFSET, 8));
+ Assertions.assertEquals(1, map.getNumDataPages());
+ loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ Assertions.assertTrue(loc.isDefined());
+ Assertions.assertEquals(42L, Platform.getLong(loc.getValueBase(), loc.getValueOffset()));
+ Assertions.assertEquals(1, map.numValues());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapWithSemanticKeyOperationsAfterFailedReset() {
+ memoryManager.limit(5000);
+ BytesToBytesMap.KeyOperationsFactory caseInsensitiveWithConstantHash =
+ () -> new BytesToBytesMap.KeyOperations() {
+ @Override
+ public int hash(Object base, long offset, int length) {
+ return 0;
+ }
+
+ @Override
+ public boolean equals(
+ Object leftBase,
+ long leftOffset,
+ int leftLength,
+ Object rightBase,
+ long rightOffset,
+ int rightLength) {
+ return Character.toLowerCase(Platform.getByte(leftBase, leftOffset)) ==
+ Character.toLowerCase(Platform.getByte(rightBase, rightOffset));
+ }
+ };
+ BytesToBytesMap map = new BytesToBytesMap(
+ taskMemoryManager,
+ blockManager,
+ serializerManager,
+ 256,
+ 0.5,
+ 4000,
+ caseInsensitiveWithConstantHash);
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ memoryManager.limit(PAGE_SIZE_BYTES);
+
+ byte[] storedKey = new byte[8];
+ storedKey[0] = 'A';
+ byte[] equivalentKey = new byte[16];
+ equivalentKey[0] = 'a';
+ byte[] value = new byte[8];
+ Platform.putLong(value, Platform.BYTE_ARRAY_OFFSET, 42L);
+
+ BytesToBytesMap.Location loc =
+ map.lookup(storedKey, Platform.BYTE_ARRAY_OFFSET, storedKey.length);
+ assertFalse(loc.isDefined());
+ assertTrue(loc.append(
+ storedKey,
+ Platform.BYTE_ARRAY_OFFSET,
+ storedKey.length,
+ value,
+ Platform.BYTE_ARRAY_OFFSET,
+ value.length));
+
+ loc = map.lookup(equivalentKey, Platform.BYTE_ARRAY_OFFSET, equivalentKey.length);
+ assertTrue(loc.isDefined());
+ assertEquals(storedKey.length, loc.getKeyLength());
+ assertEquals(42L, Platform.getLong(loc.getValueBase(), loc.getValueOffset()));
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapAfterFailedResetRestoresGrowthState() {
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ // A sub-one-key growth threshold makes canGrowArray observable through append(): after a
+ // failed grow, no new key is accepted until reset restores the flag.
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 64, 0.01, 4000);
+ try {
+ for (long i = 0; i < 2; i++) {
+ final long[] key = new long[]{i};
+ if (i == 1) {
+ // The data page already exists, so this OOM is consumed by growAndRehash().
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ }
+ assertTrue(map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8).append(
+ key, Platform.LONG_ARRAY_OFFSET, 8, key, Platform.LONG_ARRAY_OFFSET, 8));
+ }
+
+ final long[] blockedKey = new long[]{2L};
+ assertFalse(map.lookup(blockedKey, Platform.LONG_ARRAY_OFFSET, 8).append(
+ blockedKey,
+ Platform.LONG_ARRAY_OFFSET,
+ 8,
+ blockedKey,
+ Platform.LONG_ARRAY_OFFSET,
+ 8));
+
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ for (long i = 10; i < 12; i++) {
+ final long[] key = new long[]{i};
+ final long[] value = new long[]{i * 10};
+ BytesToBytesMap.Location loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ assertFalse(loc.isDefined());
+ assertTrue(loc.append(
+ key, Platform.LONG_ARRAY_OFFSET, 8, value, Platform.LONG_ARRAY_OFFSET, 8));
+ loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ assertTrue(loc.isDefined());
+ assertEquals(i * 10, Platform.getLong(loc.getValueBase(), loc.getValueOffset()));
+ }
+ assertEquals(2, map.numKeys());
+ assertEquals(2, map.numValues());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapAfterFailedResetPropagatesRepeatedOom() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+
+ final long[] key = new long[]{1L};
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(
+ SparkOutOfMemoryError.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ BytesToBytesMap.Location loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ assertFalse(loc.isDefined());
+ assertEquals(0, map.numKeys());
+ assertEquals(0, map.numValues());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapAfterRepeatedFailedReset() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ final long[] key = new long[]{1L};
+ BytesToBytesMap.Location loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ assertFalse(loc.isDefined());
+ assertEquals(0, map.numKeys());
+ assertEquals(0, map.numValues());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void safeLookupAfterFailedReset() throws Exception {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ ExecutorService executor = Executors.newFixedThreadPool(2);
+ CountDownLatch releaseRecoveryAllocation = new CountDownLatch(1);
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ final long[] key = new long[]{1L};
+ CountDownLatch recoveryAllocationEntered = new CountDownLatch(1);
+ memoryManager.runBeforeNextExecutionMemoryGrant(() -> {
+ recoveryAllocationEntered.countDown();
+ try {
+ if (!releaseRecoveryAllocation.await(30, TimeUnit.SECONDS)) {
+ throw new AssertionError("Recovery allocation was not released");
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new AssertionError("Recovery allocation was interrupted", e);
+ }
+ });
+
+ Future<Boolean> firstLookup = executor.submit(() -> {
+ BytesToBytesMap.Location threadLocation = map.new Location();
+ map.safeLookup(key, Platform.LONG_ARRAY_OFFSET, 8, threadLocation);
+ return threadLocation.isDefined();
+ });
+ assertTrue(recoveryAllocationEntered.await(10, TimeUnit.SECONDS));
+
+ AtomicReference<Thread> secondWorker = new AtomicReference<>();
+ CountDownLatch secondStarted = new CountDownLatch(1);
+ Future<Boolean> secondLookup = executor.submit(() -> {
+ BytesToBytesMap.Location threadLocation = map.new Location();
+ secondWorker.set(Thread.currentThread());
+ secondStarted.countDown();
+ map.safeLookup(key, Platform.LONG_ARRAY_OFFSET, 8, threadLocation);
+ return threadLocation.isDefined();
+ });
+ assertTrue(secondStarted.await(10, TimeUnit.SECONDS));
+
+ // The first lookup is held during allocation. Verify that the second has reached the
+ // synchronized recovery gate before allowing the first to publish the replacement array.
+ boolean contended = false;
+ long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10);
+ ThreadMXBean threadBean = ManagementFactory.getThreadMXBean();
+ while (System.nanoTime() < deadline && !secondLookup.isDone()) {
+ ThreadInfo info = threadBean.getThreadInfo(secondWorker.get().getId());
+ if (info != null && info.getThreadState() == Thread.State.BLOCKED &&
+ info.getLockInfo() != null &&
+ info.getLockInfo().getClassName().equals(BytesToBytesMap.class.getName()) &&
+ info.getLockInfo().getIdentityHashCode() == System.identityHashCode(map)) {
+ contended = true;
+ break;
+ }
+ TimeUnit.MILLISECONDS.sleep(1);
+ }
+ assertTrue(contended, "Second lookup never contended for recovery");
+ assertFalse(firstLookup.isDone());
+ releaseRecoveryAllocation.countDown();
+
+ assertFalse(firstLookup.get(10, TimeUnit.SECONDS));
+ assertFalse(secondLookup.get(10, TimeUnit.SECONDS));
+ assertEquals(0, map.getNumDataPages());
+ assertEquals(
+ map.getTotalMemoryConsumption(), taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ BytesToBytesMap.Location location = map.new Location();
+ map.safeLookup(key, Platform.LONG_ARRAY_OFFSET, 8, location, 42);
+ assertFalse(location.isDefined());
+ } finally {
+ releaseRecoveryAllocation.countDown();
+ executor.shutdownNow();
Review Comment:
Obsolete after 60eea7415ee. The executor-based concurrent recovery test was removed with the lazy-recovery design, so the shutdown race no longer exists.
##########
core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java:
##########
@@ -875,4 +879,362 @@ public void freeAfterFailedReset() {
}
}
+ @Test
+ public void lookupAfterFreeDoesNotRestoreArray() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ map.free();
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ map.reset();
+ assertEquals(0, map.getNumDataPages());
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ assertThrows(
+ IllegalStateException.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void lookupAfterDestructiveIterationDoesNotRestoreArray() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ map.destructiveIterator();
+ memoryManager.limit(PAGE_SIZE_BYTES);
+
+ assertThrows(
+ IllegalStateException.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void resetAfterDestructiveIterationFails() {
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ map.destructiveIterator();
+ assertThrows(IllegalStateException.class, map::reset);
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void arrayAccessorsRecoverAfterFailedReset() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ assertEquals(512L, map.getArray().size());
+
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ assertEquals(256, map.maxNumKeysIndex());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapAfterFailedReset() {
+ // A reset() whose eager longArray reallocation OOMs leaves the map with longArray == null (and,
+ // because the trailing reset() bookkeeping never runs, a stale currentPage). A subsequent
+ // lookup then NPEs in safeLookup (this.longArray is null), and a later append would write into
+ // a freed page. The map must instead recover to a usable empty state.
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ final long[] key = new long[]{1L};
+ final long[] value = new long[]{42L};
+ BytesToBytesMap.Location loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ loc.append(key, Platform.LONG_ARRAY_OFFSET, 8, value, Platform.LONG_ARRAY_OFFSET, 8);
Review Comment:
Addressed in 60eea7415ee. The setup append is now wrapped in assertTrue(), preserving the stale-page regression coverage if memory-grant behavior changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59757][CORE] Harden BytesToBytesMap after failed reset allocation [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #59009:
URL: https://github.com/apache/spark/pull/59009#issuecomment-5860380259
Could you fix the compilation failure, @cloud-fan ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59757][CORE] Harden BytesToBytesMap after failed reset allocation [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #59009:
URL: https://github.com/apache/spark/pull/59009#discussion_r4106947029
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -1109,30 +1173,55 @@ public int getNumDataPages() {
}
/**
- * Returns the underline long[] of longArray.
+ * Returns the underlying long array. If the pointer-array allocation in a previous
+ * {@link #reset()} failed, this method retries that allocation before returning.
+ *
+ * @throws SparkOutOfMemoryError if the pointer array still cannot be allocated
*/
public LongArray getArray() {
+ restoreArrayAfterFailedReset();
assert(longArray != null);
return longArray;
}
/**
- * Reset this map to initialized state.
+ * Reset this map to initialized state. If {@link #free()} has been called, the map cannot be
+ * reinitialized and this method is a no-op. If the replacement pointer array cannot be
+ * allocated, the map remains empty and the next lookup, {@link #getArray()}, or
Review Comment:
Addressed in 60eea7415ee. Failed reset no longer self-heals through lookup. The map remains empty and unusable until the caller explicitly retries reset(), so catching the reset OOM and continuing with lookup fails fast instead of silently discarding sorter-held data.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -612,9 +668,14 @@ public void safeLookup(Object keyBase, long keyOffset, int keyLength, Location l
*
* The provided hash is ignored when this map has configured key operations. Each thread must
* supply its own {@link Location}. Probe statistics may be inaccurate under concurrent lookup,
- * and the map must not be modified concurrently.
+ * and the map must not otherwise be modified concurrently. After {@link #reset()} has completed
+ * exceptionally because its pointer-array allocation failed, concurrent calls may safely retry
+ * that allocation.
+ *
+ * @throws SparkOutOfMemoryError if a pointer-array allocation retried after reset still fails
*/
public void safeLookup(Object keyBase, long keyOffset, int keyLength, Location loc, int hash) {
+ restoreArrayAfterFailedReset();
Review Comment:
Addressed in 60eea7415ee. lookup() and safeLookup() no longer allocate during recovery or throw SparkOutOfMemoryError from that recovery path. They perform a state check and fail with IllegalStateException after a failed reset; only reset() allocates.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -587,12 +604,50 @@ public Location lookup(Object keyBase, long keyOffset, int keyLength, int hash)
return loc;
}
+ /**
+ * Restores the hash array after a {@link #reset()} freed it but its eager reallocation failed
+ * with an OOM, leaving {@code longArray == null}. A reset() empties the map, so re-allocating at
+ * {@code initialCapacity} here is correct. If memory is still unavailable this throws a
+ * SparkOutOfMemoryError instead of exposing the null array.
+ */
+ private void restoreArrayAfterFailedReset() {
+ if (freed) {
+ throw new IllegalStateException("BytesToBytesMap has already been freed");
+ }
+ if (longArrayRecoveryRequired) {
+ // Only the failed-reset state is recoverable here: the map is empty, so a fresh
+ // initial-capacity array is correct. `destructiveIterator != null` means destructive
+ // iteration has begun, after which map operations are illegal.
+ //
+ // allocate() acquires the TaskMemoryManager monitor while this monitor is held. The memory
+ // manager may call spill() while holding its monitor, so spill() and anything reachable
+ // from it must not acquire this monitor.
+ synchronized (this) {
+ if (freed) {
+ throw new IllegalStateException("BytesToBytesMap has already been freed");
+ }
+ if (destructiveIterator != null) {
Review Comment:
Addressed in 60eea7415ee. A single lifecycle state is checked by all array-dependent operations regardless of history. Lookups, accessors, and iterators after destructive iteration now consistently throw IllegalStateException, and operationsAfterDestructiveIterationFail covers the normal destructive path.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -1109,30 +1173,55 @@ public int getNumDataPages() {
}
/**
- * Returns the underline long[] of longArray.
+ * Returns the underlying long array. If the pointer-array allocation in a previous
+ * {@link #reset()} failed, this method retries that allocation before returning.
+ *
+ * @throws SparkOutOfMemoryError if the pointer array still cannot be allocated
*/
public LongArray getArray() {
+ restoreArrayAfterFailedReset();
assert(longArray != null);
return longArray;
}
/**
- * Reset this map to initialized state.
+ * Reset this map to initialized state. If {@link #free()} has been called, the map cannot be
+ * reinitialized and this method is a no-op. If the replacement pointer array cannot be
+ * allocated, the map remains empty and the next lookup, {@link #getArray()}, or
+ * {@link #maxNumKeysIndex()} retries the allocation.
+ *
+ * @throws SparkOutOfMemoryError if the replacement pointer array cannot be allocated
+ * @throws IllegalStateException if destructive iteration has begun
*/
- public void reset() {
+ public synchronized void reset() {
+ if (freed) {
Review Comment:
Addressed in 60eea7415ee. reset() after free() now throws IllegalStateException, as does reset() after destructive iteration. operationsAfterFreeFail covers reset and the array-dependent accessors.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -1109,30 +1173,55 @@ public int getNumDataPages() {
}
/**
- * Returns the underline long[] of longArray.
+ * Returns the underlying long array. If the pointer-array allocation in a previous
+ * {@link #reset()} failed, this method retries that allocation before returning.
+ *
+ * @throws SparkOutOfMemoryError if the pointer array still cannot be allocated
*/
public LongArray getArray() {
+ restoreArrayAfterFailedReset();
assert(longArray != null);
return longArray;
}
/**
- * Reset this map to initialized state.
+ * Reset this map to initialized state. If {@link #free()} has been called, the map cannot be
+ * reinitialized and this method is a no-op. If the replacement pointer array cannot be
+ * allocated, the map remains empty and the next lookup, {@link #getArray()}, or
+ * {@link #maxNumKeysIndex()} retries the allocation.
+ *
+ * @throws SparkOutOfMemoryError if the replacement pointer array cannot be allocated
+ * @throws IllegalStateException if destructive iteration has begun
*/
- public void reset() {
+ public synchronized void reset() {
Review Comment:
I used the explicit-reset alternative in 60eea7415ee. Rather than retaining and zeroing a potentially large old pointer array, which would work against reset releasing memory, reset frees the old storage and enters RESET_FAILED if replacement allocation fails. Only a later explicit reset can recover. This also removes all lazy-recovery machinery.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59757][CORE] Harden BytesToBytesMap after failed reset allocation [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #59009:
URL: https://github.com/apache/spark/pull/59009#discussion_r4106950513
##########
core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java:
##########
@@ -875,4 +879,362 @@ public void freeAfterFailedReset() {
}
}
+ @Test
+ public void lookupAfterFreeDoesNotRestoreArray() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ map.free();
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ map.reset();
+ assertEquals(0, map.getNumDataPages());
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ assertThrows(
+ IllegalStateException.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void lookupAfterDestructiveIterationDoesNotRestoreArray() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ map.destructiveIterator();
+ memoryManager.limit(PAGE_SIZE_BYTES);
+
+ assertThrows(
+ IllegalStateException.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void resetAfterDestructiveIterationFails() {
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ map.destructiveIterator();
+ assertThrows(IllegalStateException.class, map::reset);
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void arrayAccessorsRecoverAfterFailedReset() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ assertEquals(512L, map.getArray().size());
+
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ assertEquals(256, map.maxNumKeysIndex());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapAfterFailedReset() {
+ // A reset() whose eager longArray reallocation OOMs leaves the map with longArray == null (and,
Review Comment:
Addressed in 60eea7415ee. The stale pre-fix narrative was replaced with the invariant being tested: failed reset clears old page state and lookup must not allocate implicitly.
##########
core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java:
##########
@@ -875,4 +879,362 @@ public void freeAfterFailedReset() {
}
}
+ @Test
+ public void lookupAfterFreeDoesNotRestoreArray() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ map.free();
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ map.reset();
+ assertEquals(0, map.getNumDataPages());
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ assertThrows(
+ IllegalStateException.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void lookupAfterDestructiveIterationDoesNotRestoreArray() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ map.destructiveIterator();
+ memoryManager.limit(PAGE_SIZE_BYTES);
+
+ assertThrows(
+ IllegalStateException.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void resetAfterDestructiveIterationFails() {
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ map.destructiveIterator();
+ assertThrows(IllegalStateException.class, map::reset);
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void arrayAccessorsRecoverAfterFailedReset() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ assertEquals(512L, map.getArray().size());
+
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ assertEquals(256, map.maxNumKeysIndex());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapAfterFailedReset() {
+ // A reset() whose eager longArray reallocation OOMs leaves the map with longArray == null (and,
+ // because the trailing reset() bookkeeping never runs, a stale currentPage). A subsequent
+ // lookup then NPEs in safeLookup (this.longArray is null), and a later append would write into
+ // a freed page. The map must instead recover to a usable empty state.
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ final long[] key = new long[]{1L};
+ final long[] value = new long[]{42L};
+ BytesToBytesMap.Location loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ loc.append(key, Platform.LONG_ARRAY_OFFSET, 8, value, Platform.LONG_ARRAY_OFFSET, 8);
+
+ // Force OOM on the reallocation inside reset(), leaving longArray == null.
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+
+ // The map must recover: lookup must not NPE, it must report an empty map, and it must be
+ // usable again (append into a fresh page, then read the value back).
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ Assertions.assertFalse(loc.isDefined());
+ Assertions.assertEquals(0, map.numValues());
+ Assertions.assertTrue(
+ loc.append(key, Platform.LONG_ARRAY_OFFSET, 8, value, Platform.LONG_ARRAY_OFFSET, 8));
+ Assertions.assertEquals(1, map.getNumDataPages());
+ loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ Assertions.assertTrue(loc.isDefined());
+ Assertions.assertEquals(42L, Platform.getLong(loc.getValueBase(), loc.getValueOffset()));
+ Assertions.assertEquals(1, map.numValues());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapWithSemanticKeyOperationsAfterFailedReset() {
+ memoryManager.limit(5000);
+ BytesToBytesMap.KeyOperationsFactory caseInsensitiveWithConstantHash =
Review Comment:
Addressed in 60eea7415ee. The duplicated semantic-key case and repeated failed-reset fixtures were removed. Coverage is consolidated into two failed-reset tests plus focused terminal-state tests, reducing the added test code to 141 lines.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -140,6 +140,7 @@ public interface KeyOperationsFactory {
* while position {@code 2 * i + 1} in the array holds key's full 32-bit hashcode.
*/
@Nullable private LongArray longArray;
+
Review Comment:
Addressed in 60eea7415ee. The extra blank line is removed, so the TODO is directly below longArray again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59757][CORE] Harden BytesToBytesMap after failed reset allocation [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #59009:
URL: https://github.com/apache/spark/pull/59009#issuecomment-5848374048
@dongjoon-hyun I also removed the earlier-iteration sentence from the PR description. The lifecycle follow-up is in 50c8f342c668; Java checkstyle and git diff --check pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59757][CORE] Harden BytesToBytesMap after failed reset allocation [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #59009:
URL: https://github.com/apache/spark/pull/59009#issuecomment-5882140651
the k8s failure is unrelated, thanks for the review, merging to master/4.x!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59757][CORE] Harden BytesToBytesMap after failed reset allocation [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #59009:
URL: https://github.com/apache/spark/pull/59009#discussion_r4108469002
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -581,18 +609,28 @@ public Location lookup(Object keyBase, long keyOffset, int keyLength) {
*
* This function always returns the same {@link Location} instance to avoid object allocation.
* This function is not thread-safe.
+ *
+ * @throws IllegalStateException if the map is not ready for use
*/
public Location lookup(Object keyBase, long keyOffset, int keyLength, int hash) {
safeLookup(keyBase, keyOffset, keyLength, loc, hash);
return loc;
}
+ private void ensureReady() {
Review Comment:
`Location.append()` is not guarded. With a `Location` obtained before a failed `reset()` or `free()`, `append()` only has `assert (longArray != null)`, so without `-ea` it acquires a new data page first and then hits an NPE at `longArray.get(...)`, leaving that page in `dataPages`. Since `append()` runs only for new keys, adding `ensureReady()` there should be cheap. Alternatively, documenting that a stale `Location` is unsupported would be enough.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -1109,30 +1153,46 @@ public int getNumDataPages() {
}
/**
- * Returns the underline long[] of longArray.
+ * Returns the underlying long array.
+ *
+ * @throws IllegalStateException if the map is not ready for use
*/
public LongArray getArray() {
+ ensureReady();
assert(longArray != null);
return longArray;
}
/**
- * Reset this map to initialized state.
+ * Resets this map to initialized state. If the replacement pointer array cannot be allocated,
+ * the map remains empty but cannot be used until a later call to this method succeeds.
+ *
+ * @throws SparkOutOfMemoryError if the replacement pointer array cannot be allocated
+ * @throws IllegalStateException if the map has been freed or destructive iteration has begun
*/
public void reset() {
+ if (state == MapState.FREED || state == MapState.DESTRUCTIVE) {
+ throw new IllegalStateException("BytesToBytesMap cannot be reset in state " + state);
+ }
+ state = MapState.RESET_FAILED;
updatePeakMemoryUsed();
+ // Put the map into its empty state before allocating so that a failed allocation cannot leave
+ // stale page state. A caller may retry reset(), but cannot use the map until reset succeeds.
Review Comment:
This (and the javadoc) still advertises retrying `reset()`. For the only in-tree caller, `UnsafeKVExternalSorter`, a failed `reset()` also drops the sorter holding the spilled groups, so a caller that retries `reset()` and keeps aggregating would silently produce a result without the pre-spill groups. Could we add a sentence that a caller retrying `reset()` is responsible for the validity of any state kept outside the map (e.g., spilled data)?
##########
core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java:
##########
@@ -875,4 +875,178 @@ public void freeAfterFailedReset() {
}
}
+ @Test
+ public void operationsAfterFreeFail() {
+ memoryManager.limit(PAGE_SIZE_BYTES);
Review Comment:
nit. This `memoryManager.limit(PAGE_SIZE_BYTES)` does not seem to be needed for this test.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -547,17 +566,24 @@ public Location next() {
*
* The returned iterator is NOT thread-safe. If the map is modified while iterating over it,
* the behavior of the returned iterator is undefined.
+ *
+ * @throws IllegalStateException if a reset failed, destructive iteration has begun, or the map
+ * has been freed
*/
public MapIteratorWithKeyIndex iteratorWithKeyIndex() {
+ ensureReady();
return new MapIteratorWithKeyIndex();
}
/**
* The maximum number of allowed keys index.
*
* The value of allowed keys index is in the range of [0, maxNumKeysIndex - 1].
+ *
+ * @throws IllegalStateException if the map is not ready for use
Review Comment:
nit. The `@throws` wording is inconsistent: the iterator methods say "if a reset failed, destructive iteration has begun, or the map has been freed", while `lookup` / `safeLookup` / `getArray` / `maxNumKeysIndex` say "if the map is not ready for use". Could we use one form?
##########
core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java:
##########
@@ -875,4 +875,178 @@ public void freeAfterFailedReset() {
}
}
+ @Test
+ public void operationsAfterFreeFail() {
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ map.free();
+ assertThrows(IllegalStateException.class, map::reset);
+ assertThrows(
+ IllegalStateException.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+ assertThrows(IllegalStateException.class, map::getArray);
+ assertThrows(IllegalStateException.class, map::maxNumKeysIndex);
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void configuredKeyOperationsRejectLookupAfterFree() {
+ BytesToBytesMap.KeyOperationsFactory keyOperationsFactory =
Review Comment:
nit. This anonymous `KeyOperations` duplicates the pattern of the existing `caseInsensitiveWithConstantHash` fixture in this suite. It may be worth sharing a fixture, although the existing one is test-local.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59757][CORE] Harden BytesToBytesMap after failed reset allocation [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #59009:
URL: https://github.com/apache/spark/pull/59009#discussion_r4093349286
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -1117,22 +1157,34 @@ public LongArray getArray() {
}
/**
- * Reset this map to initialized state.
+ * Reset this map to initialized state. If {@link #free()} has been called, the map cannot be
+ * reinitialized and this method is a no-op.
*/
- public void reset() {
+ public synchronized void reset() {
+ if (freed) {
+ return;
+ }
updatePeakMemoryUsed();
+ // Put the map into its empty state up front so that if the allocate() below fails with an OOM,
+ // the map is left consistently empty (longArray == null, no stale currentPage) rather than
+ // half-reset. A subsequent lookup then restores the array lazily (restoreArrayAfterFailedReset)
+ // instead of dereferencing a null longArray or writing into a freed currentPage.
numKeys = 0;
numValues = 0;
- freeArray(longArray);
+ canGrowArray = true;
+ currentPage = null;
+ pageCursor = 0;
+ if (longArray != null) {
+ freeArray(longArray);
+ }
longArray = null;
while (dataPages.size() > 0) {
MemoryBlock dataPage = dataPages.removeLast();
freePage(dataPage);
}
+ longArrayRecoveryRequired = true;
Review Comment:
I kept the flag placement unchanged intentionally. Recovery is safe only after all old data pages have been removed. If `freePage()` unexpectedly failed midway, setting the flag earlier could let a later lookup allocate a fresh pointer array while stale pages remain in `dataPages`; subsequent iteration could then expose stale records. Since `freeArray()` and `freePage()` are not normal failure points, I prefer not to mark a partially cleaned map as recoverable.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -587,6 +599,29 @@ public Location lookup(Object keyBase, long keyOffset, int keyLength, int hash)
return loc;
}
+ /**
+ * Restores the hash array after a {@link #reset()} freed it but its eager reallocation failed
+ * with an OOM, leaving {@code longArray == null}. A reset() empties the map, so re-allocating at
+ * {@code initialCapacity} here is correct. Without this, a subsequent lookup dereferences the
+ * null array in {@code safeLookup} (or an assertion error with -ea). If memory is still
+ * unavailable this throws a SparkOutOfMemoryError instead.
+ */
+ private void restoreArrayAfterFailedReset() {
+ if (!freed && longArrayRecoveryRequired) {
+ // Only the failed-reset state is recoverable here: the map is empty, so a fresh
+ // initial-capacity array is correct. `destructiveIterator != null` means destructive
+ // iteration has begun, after which map operations are illegal. Do not re-allocate the
+ // array in that terminal state; leave the illegal lookup to fail loudly.
+ synchronized (this) {
Review Comment:
Addressed in a380c1961e3. I added the lock-order invariant beside the synchronized recovery block: allocation enters `TaskMemoryManager` while holding the map monitor, so `spill()` and anything reachable from it must not acquire the map monitor.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -587,6 +599,29 @@ public Location lookup(Object keyBase, long keyOffset, int keyLength, int hash)
return loc;
}
+ /**
+ * Restores the hash array after a {@link #reset()} freed it but its eager reallocation failed
+ * with an OOM, leaving {@code longArray == null}. A reset() empties the map, so re-allocating at
+ * {@code initialCapacity} here is correct. Without this, a subsequent lookup dereferences the
+ * null array in {@code safeLookup} (or an assertion error with -ea). If memory is still
+ * unavailable this throws a SparkOutOfMemoryError instead.
+ */
+ private void restoreArrayAfterFailedReset() {
Review Comment:
Addressed in a380c1961e3. Both `getArray()` and `maxNumKeysIndex()` now invoke failed-reset recovery. `arrayAccessorsRecoverAfterFailedReset` independently exercises each entry point after a failed reset.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -615,6 +651,7 @@ public void safeLookup(Object keyBase, long keyOffset, int keyLength, Location l
* and the map must not be modified concurrently.
*/
public void safeLookup(Object keyBase, long keyOffset, int keyLength, Location loc, int hash) {
+ restoreArrayAfterFailedReset();
Review Comment:
Addressed in a380c1961e3. The `reset()` and both `safeLookup()` overloads now document failed-reset recovery and the possible `SparkOutOfMemoryError`. I also documented recovery on the two public array accessors.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -587,6 +599,29 @@ public Location lookup(Object keyBase, long keyOffset, int keyLength, int hash)
return loc;
}
+ /**
+ * Restores the hash array after a {@link #reset()} freed it but its eager reallocation failed
+ * with an OOM, leaving {@code longArray == null}. A reset() empties the map, so re-allocating at
+ * {@code initialCapacity} here is correct. Without this, a subsequent lookup dereferences the
+ * null array in {@code safeLookup} (or an assertion error with -ea). If memory is still
+ * unavailable this throws a SparkOutOfMemoryError instead.
+ */
+ private void restoreArrayAfterFailedReset() {
+ if (!freed && longArrayRecoveryRequired) {
+ // Only the failed-reset state is recoverable here: the map is empty, so a fresh
+ // initial-capacity array is correct. `destructiveIterator != null` means destructive
+ // iteration has begun, after which map operations are illegal. Do not re-allocate the
+ // array in that terminal state; leave the illegal lookup to fail loudly.
+ synchronized (this) {
+ if (!freed && longArray == null && destructiveIterator == null &&
Review Comment:
Addressed in a380c1961e3. I added `lookupAfterDestructiveIterationDoesNotRestoreArray`; the recovery path now fails explicitly with `IllegalStateException` and the test also verifies that no array memory was allocated.
##########
core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java:
##########
@@ -875,4 +879,308 @@ public void freeAfterFailedReset() {
}
}
+ @Test
+ public void lookupAfterFreeDoesNotRestoreArray() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ map.free();
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ map.reset();
+ assertEquals(0, map.getNumDataPages());
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ assertThrows(
+ AssertionError.class,
Review Comment:
Addressed in a380c1961e3. Lookup after `free()` now throws an explicit `IllegalStateException`, and the regression test no longer depends on JVM assertions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59757][CORE] Harden BytesToBytesMap after failed reset allocation [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #59009:
URL: https://github.com/apache/spark/pull/59009#discussion_r4104095287
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -612,9 +667,13 @@ public void safeLookup(Object keyBase, long keyOffset, int keyLength, Location l
*
* The provided hash is ignored when this map has configured key operations. Each thread must
* supply its own {@link Location}. Probe statistics may be inaccurate under concurrent lookup,
- * and the map must not be modified concurrently.
+ * and the map must not otherwise be modified concurrently. Concurrent calls may safely retry a
+ * pointer-array allocation that failed during {@link #reset()}.
+ *
+ * @throws SparkOutOfMemoryError if a pointer-array allocation retried after reset still fails
*/
public void safeLookup(Object keyBase, long keyOffset, int keyLength, Location loc, int hash) {
+ restoreArrayAfterFailedReset();
Review Comment:
Thanks — hardening is the right framing, and marking the user-facing change as No, with the note that no in-tree caller catches the error, resolves this from my side.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -1109,30 +1171,50 @@ public int getNumDataPages() {
}
/**
- * Returns the underline long[] of longArray.
+ * Returns the underlying long array. If the pointer-array allocation in a previous
+ * {@link #reset()} failed, this method retries that allocation before returning.
+ *
+ * @throws SparkOutOfMemoryError if the pointer array still cannot be allocated
*/
public LongArray getArray() {
+ restoreArrayAfterFailedReset();
assert(longArray != null);
return longArray;
}
/**
- * Reset this map to initialized state.
+ * Reset this map to initialized state. If {@link #free()} has been called, the map cannot be
+ * reinitialized and this method is a no-op. If the replacement pointer array cannot be
+ * allocated, the map remains empty and the next lookup, {@link #getArray()}, or
+ * {@link #maxNumKeysIndex()} retries the allocation.
+ *
+ * @throws SparkOutOfMemoryError if the replacement pointer array cannot be allocated
*/
- public void reset() {
+ public synchronized void reset() {
+ if (freed) {
+ return;
+ }
updatePeakMemoryUsed();
+ // Put the map into its empty state up front so that if the allocate() below fails with an OOM,
+ // the map is left consistently empty (longArray == null, no stale currentPage) rather than
+ // half-reset. A subsequent lookup then restores the array lazily (restoreArrayAfterFailedReset)
+ // instead of dereferencing a null longArray or writing into a freed currentPage.
numKeys = 0;
numValues = 0;
- freeArray(longArray);
+ canGrowArray = true;
+ currentPage = null;
+ pageCursor = 0;
+ if (longArray != null) {
+ freeArray(longArray);
+ }
longArray = null;
while (dataPages.size() > 0) {
MemoryBlock dataPage = dataPages.removeLast();
freePage(dataPage);
}
+ longArrayRecoveryRequired = true;
Review Comment:
Agreed on both counts: the narrowed contract (concurrent retry only after `reset()` has returned exceptionally) covers the concern, and keeping the flag after page cleanup is the safer choice given a mid-cleanup `freePage()` failure. Withdrawing the move suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59757][CORE] Harden BytesToBytesMap after failed reset allocation [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #59009:
URL: https://github.com/apache/spark/pull/59009#discussion_r4104097035
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -140,6 +140,18 @@ public interface KeyOperationsFactory {
* while position {@code 2 * i + 1} in the array holds key's full 32-bit hashcode.
*/
@Nullable private LongArray longArray;
+
+ /**
+ * Whether a completed reset needs to restore longArray after its allocation failed.
+ *
+ * Volatile so that clearing this flag safely publishes the replacement array to concurrent
+ * safeLookup callers.
+ */
+ private volatile boolean longArrayRecoveryRequired = false;
+
+ /** Whether {@link #free()} has permanently released this map. */
+ private volatile boolean freed = false;
Review Comment:
Confirmed on 64eaea7 — the TODO sits under `longArray` again with the two flags below it. Resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59757][CORE] Harden BytesToBytesMap after failed reset allocation [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #59009:
URL: https://github.com/apache/spark/pull/59009#issuecomment-5882155599
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/5a05bff0c84bc840285a6e013d79fee7ad453e78
- merged into branch-4.x https://github.com/apache/spark/commit/e0fc8decdc6e4e5fdf2ba6c6675f2b40a6d303d1
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59757][CORE] Fix NPE in BytesToBytesMap after failed reset during spill [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #59009:
URL: https://github.com/apache/spark/pull/59009#discussion_r4090886566
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -587,6 +599,29 @@ public Location lookup(Object keyBase, long keyOffset, int keyLength, int hash)
return loc;
}
+ /**
+ * Restores the hash array after a {@link #reset()} freed it but its eager reallocation failed
+ * with an OOM, leaving {@code longArray == null}. A reset() empties the map, so re-allocating at
+ * {@code initialCapacity} here is correct. Without this, a subsequent lookup dereferences the
+ * null array in {@code safeLookup} (or an assertion error with -ea). If memory is still
+ * unavailable this throws a SparkOutOfMemoryError instead.
+ */
+ private void restoreArrayAfterFailedReset() {
+ if (!freed && longArrayRecoveryRequired) {
+ // Only the failed-reset state is recoverable here: the map is empty, so a fresh
+ // initial-capacity array is correct. `destructiveIterator != null` means destructive
+ // iteration has begun, after which map operations are illegal. Do not re-allocate the
+ // array in that terminal state; leave the illegal lookup to fail loudly.
+ synchronized (this) {
+ if (!freed && longArray == null && destructiveIterator == null &&
Review Comment:
**Finding 2.** This guard has no test. I removed `destructiveIterator == null` from the condition and re-ran both suites on your head commit: all 25 tests in each stayed green, so nothing currently pins it.
This closes it, alongside the existing `lookupAfterFreeDoesNotRestoreArray`:
```java
@Test
public void lookupAfterDestructiveIterationDoesNotRestoreArray() {
memoryManager.limit(5000);
BytesToBytesMap map =
new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
final long[] key = new long[]{1L};
try {
memoryManager.markExecutionAsOutOfMemoryOnce();
assertThrows(SparkOutOfMemoryError.class, map::reset);
map.destructiveIterator();
memoryManager.limit(PAGE_SIZE_BYTES);
assertThrows(
AssertionError.class,
() -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
} finally {
map.free();
}
}
```
I ran it both ways. On your head it passes. With `destructiveIterator == null` deleted it fails with `Expected java.lang.AssertionError to be thrown, but nothing was thrown`, and the memory assertion would catch the stray array as well.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -615,6 +651,7 @@ public void safeLookup(Object keyBase, long keyOffset, int keyLength, Location l
* and the map must not be modified concurrently.
*/
public void safeLookup(Object keyBase, long keyOffset, int keyLength, Location loc, int hash) {
+ restoreArrayAfterFailedReset();
Review Comment:
**Finding 3.** Two javadoc gaps on public methods, both about behaviour this line introduces.
`safeLookup`'s doc right above says "The map must not be modified concurrently", but this call makes the lookup itself modify the map. It allocates a `LongArray`, registers it with the `TaskMemoryManager`, and rewrites `mask` and `growthThreshold`. It can also throw `SparkOutOfMemoryError`, which nobody expects from a read method. `UnsafeHashedRelation.get` and `getValue` reach `safeLookup` on the broadcast-join probe path (`sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:235`), so it is worth stating even though those maps are never reset.
`reset()`'s doc at `:1160` gained the `free()` sentence but not the one that matters most here. A failed reallocation now leaves the map empty and the next lookup restores it. That contract currently lives only on the private `restoreArrayAfterFailedReset`, where a caller will not look for it.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -587,6 +599,29 @@ public Location lookup(Object keyBase, long keyOffset, int keyLength, int hash)
return loc;
}
+ /**
+ * Restores the hash array after a {@link #reset()} freed it but its eager reallocation failed
+ * with an OOM, leaving {@code longArray == null}. A reset() empties the map, so re-allocating at
+ * {@code initialCapacity} here is correct. Without this, a subsequent lookup dereferences the
+ * null array in {@code safeLookup} (or an assertion error with -ea). If memory is still
+ * unavailable this throws a SparkOutOfMemoryError instead.
+ */
+ private void restoreArrayAfterFailedReset() {
Review Comment:
**Finding 5.** The recovery only runs on the `safeLookup` paths, so the failed-reset state still reaches two other public methods with `longArray == null`.
- **`getArray()`**: `assert(longArray != null)` is a no-op without `-ea`, so it returns null. `UnsafeKVExternalSorter` calls it first thing (`sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeKVExternalSorter.java:109`) and then `pointerArray.size()`, which is the NPE this PR is fixing on a different entry point.
- **`maxNumKeysIndex()`**: dereferences `longArray` directly.
`iterator()`, `destructiveIterator()` and `iteratorWithKeyIndex()` are safe, because `numValues == 0` after the reset keeps them from touching the array.
Calling `restoreArrayAfterFailedReset()` at the top of `getArray()` and `maxNumKeysIndex()` covers those two. The alternative is to state in the javadoc that only lookups self-heal, so a caller knows to do a lookup first.
##########
core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java:
##########
@@ -875,4 +879,308 @@ public void freeAfterFailedReset() {
}
}
+ @Test
+ public void lookupAfterFreeDoesNotRestoreArray() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ map.free();
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ map.reset();
+ assertEquals(0, map.getNumDataPages());
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ assertThrows(
+ AssertionError.class,
Review Comment:
**Finding 7.** This pins a JVM assertion, so it only describes the `-ea` build. Production runs without `-ea`, where the same lookup falls through to `longArray.get(pos * 2)` and throws a bare NPE. That is the failure mode the PR set out to remove.
Now that `freed` is tracked, making it explicit costs nothing. `restoreArrayAfterFailedReset` already reads `freed` on the fast path, so it can just be:
```java
if (freed) {
throw new IllegalStateException("BytesToBytesMap has already been freed");
}
if (longArrayRecoveryRequired) {
...
}
```
and the test asserts `IllegalStateException` instead. If you would rather not add a throw on the lookup path, a comment saying the `AssertionError` only happens under `-ea` would at least record what production does.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -587,6 +599,29 @@ public Location lookup(Object keyBase, long keyOffset, int keyLength, int hash)
return loc;
}
+ /**
+ * Restores the hash array after a {@link #reset()} freed it but its eager reallocation failed
+ * with an OOM, leaving {@code longArray == null}. A reset() empties the map, so re-allocating at
+ * {@code initialCapacity} here is correct. Without this, a subsequent lookup dereferences the
+ * null array in {@code safeLookup} (or an assertion error with -ea). If memory is still
+ * unavailable this throws a SparkOutOfMemoryError instead.
+ */
+ private void restoreArrayAfterFailedReset() {
+ if (!freed && longArrayRecoveryRequired) {
+ // Only the failed-reset state is recoverable here: the map is empty, so a fresh
+ // initial-capacity array is correct. `destructiveIterator != null` means destructive
+ // iteration has begun, after which map operations are illegal. Do not re-allocate the
+ // array in that terminal state; leave the illegal lookup to fail loudly.
+ synchronized (this) {
Review Comment:
**Finding 4.** This block holds the map's monitor across `allocate()`, which goes through `TaskMemoryManager.allocatePage` and can spill other consumers to disk. That establishes the lock order BytesToBytesMap -> TaskMemoryManager. `TaskMemoryManager.spillConsumer` uses the opposite order, calling `consumerToSpill.spill(...)` while holding the TaskMemoryManager monitor (`core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java:313`).
It does not deadlock today, and I checked why. `BytesToBytesMap.spill` is not synchronized, and `MapIterator.spill` takes the iterator's monitor rather than the map's, so nothing reachable from `spillConsumer` ever waits on this monitor. `avoidDeadlock` still passes with `reset()` and `free()` synchronized.
That is the same hazard `advanceToNextPage` spells out for `MapIterator` under SPARK-26265, and three new things now depend on it quietly: the `synchronized` here, on `reset()`, and on `free()`. A comment beside this `synchronized` recording the invariant ("nothing invoked under the TaskMemoryManager monitor may take this monitor") would stop the next change from breaking it silently.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -1117,22 +1157,34 @@ public LongArray getArray() {
}
/**
- * Reset this map to initialized state.
+ * Reset this map to initialized state. If {@link #free()} has been called, the map cannot be
+ * reinitialized and this method is a no-op.
*/
- public void reset() {
+ public synchronized void reset() {
+ if (freed) {
+ return;
+ }
updatePeakMemoryUsed();
+ // Put the map into its empty state up front so that if the allocate() below fails with an OOM,
+ // the map is left consistently empty (longArray == null, no stale currentPage) rather than
+ // half-reset. A subsequent lookup then restores the array lazily (restoreArrayAfterFailedReset)
+ // instead of dereferencing a null longArray or writing into a freed currentPage.
numKeys = 0;
numValues = 0;
- freeArray(longArray);
+ canGrowArray = true;
+ currentPage = null;
+ pageCursor = 0;
+ if (longArray != null) {
+ freeArray(longArray);
+ }
longArray = null;
while (dataPages.size() > 0) {
MemoryBlock dataPage = dataPages.removeLast();
freePage(dataPage);
}
+ longArrayRecoveryRequired = true;
Review Comment:
**Finding 6.** The invariant the whole recovery rests on is "`longArray == null` means recovery is required". It breaks at `:1180` and is only re-established here, five statements later. If anything throws in between (a `freeArray` or `freePage` failure), the map is left with a null array and the flag still `false`, which disarms the recovery for good.
Arming it before the array is dropped removes the window:
```java
longArrayRecoveryRequired = true;
if (longArray != null) {
freeArray(longArray);
}
longArray = null;
while (dataPages.size() > 0) {
MemoryBlock dataPage = dataPages.removeLast();
freePage(dataPage);
}
allocate(initialCapacity);
longArrayRecoveryRequired = false;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59757][CORE] Harden BytesToBytesMap after failed reset allocation [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #59009:
URL: https://github.com/apache/spark/pull/59009#issuecomment-5813679172
Thanks for tracing the production callers. Addressed in a380c1961e3: I updated the title and description to frame this as low-level `BytesToBytesMap` contract hardening, changed the user-facing answer to No, and explicitly noted that no in-tree caller currently catches the `SparkOutOfMemoryError` from `reset()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59757][CORE] Harden BytesToBytesMap after failed reset allocation [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #59009:
URL: https://github.com/apache/spark/pull/59009#discussion_r4093507166
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -1109,30 +1171,50 @@ public int getNumDataPages() {
}
/**
- * Returns the underline long[] of longArray.
+ * Returns the underlying long array. If the pointer-array allocation in a previous
+ * {@link #reset()} failed, this method retries that allocation before returning.
+ *
+ * @throws SparkOutOfMemoryError if the pointer array still cannot be allocated
*/
public LongArray getArray() {
+ restoreArrayAfterFailedReset();
assert(longArray != null);
return longArray;
}
/**
- * Reset this map to initialized state.
+ * Reset this map to initialized state. If {@link #free()} has been called, the map cannot be
+ * reinitialized and this method is a no-op. If the replacement pointer array cannot be
+ * allocated, the map remains empty and the next lookup, {@link #getArray()}, or
+ * {@link #maxNumKeysIndex()} retries the allocation.
+ *
+ * @throws SparkOutOfMemoryError if the replacement pointer array cannot be allocated
*/
- public void reset() {
+ public synchronized void reset() {
+ if (freed) {
+ return;
+ }
Review Comment:
**Finding 8.** The description says destructive iteration stays terminal, but only `restoreArrayAfterFailedReset` checks `destructiveIterator`, and only when a recovery is pending. `reset()` never checks it. After `destructiveIterator()`, a `reset()` that allocates successfully leaves `longArrayRecoveryRequired == false`, so the gate never runs and the map is fully usable again.
I checked it on this head: `destructiveIterator()`, then `reset()`, `lookup()` and `append()` all succeed. On master the same `reset()` failed with an NPE in `freeArray(null)`. The new `longArray != null` check below is what lets it through now.
It also leaves `destructiveIterator` set. From reading, a `spill()` triggered by another consumer would then still go to the old iterator, which spills and frees the reused map's data pages.
Failing the same way as the recovery gate closes it:
```suggestion
if (freed) {
return;
}
if (destructiveIterator != null) {
throw new IllegalStateException(
"BytesToBytesMap cannot be used after destructiveIterator() has been called");
}
```
And a test for it:
```java
@Test
public void resetAfterDestructiveIterationFails() {
memoryManager.limit(PAGE_SIZE_BYTES);
BytesToBytesMap map =
new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
try {
map.destructiveIterator();
assertThrows(IllegalStateException.class, map::reset);
assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
} finally {
map.free();
}
}
```
I ran it both ways. It passes with the check above (56 tests green across both suites). Without the check it fails with `Expected java.lang.IllegalStateException to be thrown, but nothing was thrown`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59757][CORE] Harden BytesToBytesMap after failed reset allocation [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #59009:
URL: https://github.com/apache/spark/pull/59009#discussion_r4112172378
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -547,17 +566,24 @@ public Location next() {
*
* The returned iterator is NOT thread-safe. If the map is modified while iterating over it,
* the behavior of the returned iterator is undefined.
+ *
+ * @throws IllegalStateException if a reset failed, destructive iteration has begun, or the map
+ * has been freed
*/
public MapIteratorWithKeyIndex iteratorWithKeyIndex() {
+ ensureReady();
return new MapIteratorWithKeyIndex();
}
/**
* The maximum number of allowed keys index.
*
* The value of allowed keys index is in the range of [0, maxNumKeysIndex - 1].
+ *
+ * @throws IllegalStateException if the map is not ready for use
Review Comment:
Addressed in 50c8f342c668. The array-dependent operations now use the same explicit lifecycle-state wording in their IllegalStateException documentation.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -581,18 +609,28 @@ public Location lookup(Object keyBase, long keyOffset, int keyLength) {
*
* This function always returns the same {@link Location} instance to avoid object allocation.
* This function is not thread-safe.
+ *
+ * @throws IllegalStateException if the map is not ready for use
*/
public Location lookup(Object keyBase, long keyOffset, int keyLength, int hash) {
safeLookup(keyBase, keyOffset, keyLength, loc, hash);
return loc;
}
+ private void ensureReady() {
Review Comment:
Addressed in 50c8f342c668. Location.append() now calls ensureReady() before allocation or memory access. I also added coverage for stale Location handles after failed reset and after free.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -1109,30 +1153,46 @@ public int getNumDataPages() {
}
/**
- * Returns the underline long[] of longArray.
+ * Returns the underlying long array.
+ *
+ * @throws IllegalStateException if the map is not ready for use
*/
public LongArray getArray() {
+ ensureReady();
assert(longArray != null);
return longArray;
}
/**
- * Reset this map to initialized state.
+ * Resets this map to initialized state. If the replacement pointer array cannot be allocated,
+ * the map remains empty but cannot be used until a later call to this method succeeds.
+ *
+ * @throws SparkOutOfMemoryError if the replacement pointer array cannot be allocated
+ * @throws IllegalStateException if the map has been freed or destructive iteration has begun
*/
public void reset() {
+ if (state == MapState.FREED || state == MapState.DESTRUCTIVE) {
+ throw new IllegalStateException("BytesToBytesMap cannot be reset in state " + state);
+ }
+ state = MapState.RESET_FAILED;
updatePeakMemoryUsed();
+ // Put the map into its empty state before allocating so that a failed allocation cannot leave
+ // stale page state. A caller may retry reset(), but cannot use the map until reset succeeds.
Review Comment:
Addressed in 50c8f342c668. The reset() javadoc now clarifies that a successful retry restores only the map, and callers remain responsible for external state such as spilled data.
##########
core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java:
##########
@@ -875,4 +875,178 @@ public void freeAfterFailedReset() {
}
}
+ @Test
+ public void operationsAfterFreeFail() {
+ memoryManager.limit(PAGE_SIZE_BYTES);
Review Comment:
Addressed in 50c8f342c668. I removed the unnecessary memoryManager limit from operationsAfterFreeFail.
##########
core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java:
##########
@@ -875,4 +875,178 @@ public void freeAfterFailedReset() {
}
}
+ @Test
+ public void operationsAfterFreeFail() {
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ map.free();
+ assertThrows(IllegalStateException.class, map::reset);
+ assertThrows(
+ IllegalStateException.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+ assertThrows(IllegalStateException.class, map::getArray);
+ assertThrows(IllegalStateException.class, map::maxNumKeysIndex);
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void configuredKeyOperationsRejectLookupAfterFree() {
+ BytesToBytesMap.KeyOperationsFactory keyOperationsFactory =
Review Comment:
Thanks. I left this one unchanged. The existing fixture also tracks creation and thread ownership for its concurrency assertions, while this test only needs a configured factory whose operations must never be reached. Keeping the small fixture local avoids coupling those test concerns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59757][CORE] Fix NPE in BytesToBytesMap after failed reset during spill [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #59009:
URL: https://github.com/apache/spark/pull/59009#issuecomment-5807093735
cc @LuciferYang @peter-toth
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59757][CORE] Harden BytesToBytesMap after failed reset allocation [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #59009:
URL: https://github.com/apache/spark/pull/59009#discussion_r4106026327
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -1109,30 +1173,55 @@ public int getNumDataPages() {
}
/**
- * Returns the underline long[] of longArray.
+ * Returns the underlying long array. If the pointer-array allocation in a previous
+ * {@link #reset()} failed, this method retries that allocation before returning.
+ *
+ * @throws SparkOutOfMemoryError if the pointer array still cannot be allocated
*/
public LongArray getArray() {
+ restoreArrayAfterFailedReset();
assert(longArray != null);
return longArray;
}
/**
- * Reset this map to initialized state.
+ * Reset this map to initialized state. If {@link #free()} has been called, the map cannot be
+ * reinitialized and this method is a no-op. If the replacement pointer array cannot be
+ * allocated, the map remains empty and the next lookup, {@link #getArray()}, or
+ * {@link #maxNumKeysIndex()} retries the allocation.
+ *
+ * @throws SparkOutOfMemoryError if the replacement pointer array cannot be allocated
+ * @throws IllegalStateException if destructive iteration has begun
*/
- public void reset() {
+ public synchronized void reset() {
+ if (freed) {
Review Comment:
`reset()` after `free()` is now a silent no-op, while `lookup` / `getArray` / `maxNumKeysIndex` throw `IllegalStateException` on a freed map. Before this PR, `reset()` after `free()` failed immediately (NPE in `freeArray(null)`). Now a caller that resets a freed map by mistake gets no signal at the point of the mistake, and `numKeys()` / `numValues()` stay stale after a "successful" reset. Also, because this check runs first, `destructiveIterator(); free(); reset();` returns normally, which contradicts the new `@throws IllegalStateException if destructive iteration has begun`. Would throwing here be more consistent?
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -587,12 +604,50 @@ public Location lookup(Object keyBase, long keyOffset, int keyLength, int hash)
return loc;
}
+ /**
+ * Restores the hash array after a {@link #reset()} freed it but its eager reallocation failed
+ * with an OOM, leaving {@code longArray == null}. A reset() empties the map, so re-allocating at
+ * {@code initialCapacity} here is correct. If memory is still unavailable this throws a
+ * SparkOutOfMemoryError instead of exposing the null array.
+ */
+ private void restoreArrayAfterFailedReset() {
+ if (freed) {
+ throw new IllegalStateException("BytesToBytesMap has already been freed");
+ }
+ if (longArrayRecoveryRequired) {
+ // Only the failed-reset state is recoverable here: the map is empty, so a fresh
+ // initial-capacity array is correct. `destructiveIterator != null` means destructive
+ // iteration has begun, after which map operations are illegal.
+ //
+ // allocate() acquires the TaskMemoryManager monitor while this monitor is held. The memory
+ // manager may call spill() while holding its monitor, so spill() and anything reachable
+ // from it must not acquire this monitor.
+ synchronized (this) {
+ if (freed) {
+ throw new IllegalStateException("BytesToBytesMap has already been freed");
+ }
+ if (destructiveIterator != null) {
Review Comment:
This `destructiveIterator != null` check only runs when `longArrayRecoveryRequired` is true, so the same illegal call behaves differently depending on history. After `destructiveIterator()` on a map that never had a failed reset, `lookup()` falls through to `assert(longArray != null)` (an NPE with assertions off), and `getArray()` silently returns `null`. After a failed reset, the same calls throw this `IllegalStateException`. `lookupAfterDestructiveIterationDoesNotRestoreArray` covers only the second case. Should the check sit outside the recovery-only branch, or be dropped from here?
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -612,9 +668,14 @@ public void safeLookup(Object keyBase, long keyOffset, int keyLength, Location l
*
* The provided hash is ignored when this map has configured key operations. Each thread must
* supply its own {@link Location}. Probe statistics may be inaccurate under concurrent lookup,
- * and the map must not be modified concurrently.
+ * and the map must not otherwise be modified concurrently. After {@link #reset()} has completed
+ * exceptionally because its pointer-array allocation failed, concurrent calls may safely retry
+ * that allocation.
+ *
+ * @throws SparkOutOfMemoryError if a pointer-array allocation retried after reset still fails
*/
public void safeLookup(Object keyBase, long keyOffset, int keyLength, Location loc, int hash) {
+ restoreArrayAfterFailedReset();
Review Comment:
With this call, `lookup()` / `safeLookup()` can now allocate memory, trigger spills of other consumers, and throw `SparkOutOfMemoryError`. `UnsafeFixedWidthAggregationMap.getAggregationBufferFromUnsafeRow` assumes memory exhaustion shows up only as `loc.append(...)` returning `false` (it returns `null` so the caller can call `destructAndCreateExternalSorter()`). In recovery mode the OOM escapes from the lookup instead and bypasses that spill fallback. Also, only the `safeLookup` javadocs gained `@throws`; the two `lookup(...)` overloads do not document it.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -1109,30 +1173,55 @@ public int getNumDataPages() {
}
/**
- * Returns the underline long[] of longArray.
+ * Returns the underlying long array. If the pointer-array allocation in a previous
+ * {@link #reset()} failed, this method retries that allocation before returning.
+ *
+ * @throws SparkOutOfMemoryError if the pointer array still cannot be allocated
*/
public LongArray getArray() {
+ restoreArrayAfterFailedReset();
assert(longArray != null);
return longArray;
}
/**
- * Reset this map to initialized state.
+ * Reset this map to initialized state. If {@link #free()} has been called, the map cannot be
+ * reinitialized and this method is a no-op. If the replacement pointer array cannot be
+ * allocated, the map remains empty and the next lookup, {@link #getArray()}, or
Review Comment:
This contract (the map stays empty and the next lookup recovers it) can silently drop data when combined with the only in-tree `reset()` caller. `UnsafeKVExternalSorter`'s constructor first spills every map record into a local sorter via `createWithExistingInMemorySorter(...)` and calls `map.reset()` last. If `reset()` throws `SparkOutOfMemoryError` there, the constructor fails and the sorter holding the spilled groups never reaches `TungstenAggregationIterator` / `HashAggregateExec`. A caller that follows this documented contract (catch the OOM and keep using the map) then aggregates into a freshly recovered empty map, and every pre-spill group is missing from the result. Before this PR, the next lookup failed with an NPE and the task failed.
No in-tree caller catches this OOM today, so this is latent. But the new contract turns a loud failure into a silent wrong result for the one existing `reset()` call site. Could we either not advertise reuse after a failed `reset()`, or document that `UnsafeKVExternalSorter` callers must not recover?
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -587,12 +604,50 @@ public Location lookup(Object keyBase, long keyOffset, int keyLength, int hash)
return loc;
}
+ /**
+ * Restores the hash array after a {@link #reset()} freed it but its eager reallocation failed
+ * with an OOM, leaving {@code longArray == null}. A reset() empties the map, so re-allocating at
+ * {@code initialCapacity} here is correct. If memory is still unavailable this throws a
+ * SparkOutOfMemoryError instead of exposing the null array.
+ */
+ private void restoreArrayAfterFailedReset() {
+ if (freed) {
+ throw new IllegalStateException("BytesToBytesMap has already been freed");
+ }
+ if (longArrayRecoveryRequired) {
+ // Only the failed-reset state is recoverable here: the map is empty, so a fresh
+ // initial-capacity array is correct. `destructiveIterator != null` means destructive
+ // iteration has begun, after which map operations are illegal.
+ //
+ // allocate() acquires the TaskMemoryManager monitor while this monitor is held. The memory
+ // manager may call spill() while holding its monitor, so spill() and anything reachable
+ // from it must not acquire this monitor.
+ synchronized (this) {
Review Comment:
Is there a real caller for the concurrent-recovery contract? The maps probed concurrently through `safeLookup` are broadcast `UnsafeHashedRelation` maps (`HashedRelation.scala`), and those are never reset. The only `reset()` caller (`UnsafeKVExternalSorter`) uses a task-private map. Meanwhile, this block takes the `TaskMemoryManager` monitor while holding the map monitor, which is the reverse of `TaskMemoryManager.acquireExecutionMemory` -> `consumer.spill()`. Deadlock-freedom now depends only on the comment that `spill()` must never take this monitor, a SPARK-26265-style hazard for future edits.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -1109,30 +1173,55 @@ public int getNumDataPages() {
}
/**
- * Returns the underline long[] of longArray.
+ * Returns the underlying long array. If the pointer-array allocation in a previous
+ * {@link #reset()} failed, this method retries that allocation before returning.
+ *
+ * @throws SparkOutOfMemoryError if the pointer array still cannot be allocated
*/
public LongArray getArray() {
+ restoreArrayAfterFailedReset();
assert(longArray != null);
return longArray;
}
/**
- * Reset this map to initialized state.
+ * Reset this map to initialized state. If {@link #free()} has been called, the map cannot be
+ * reinitialized and this method is a no-op. If the replacement pointer array cannot be
+ * allocated, the map remains empty and the next lookup, {@link #getArray()}, or
+ * {@link #maxNumKeysIndex()} retries the allocation.
+ *
+ * @throws SparkOutOfMemoryError if the replacement pointer array cannot be allocated
+ * @throws IllegalStateException if destructive iteration has begun
*/
- public void reset() {
+ public synchronized void reset() {
Review Comment:
The root cause seems to be that `reset()` discards the old pointer array before securing a replacement. How about fixing the ordering instead of accepting a `null` array and recovering lazily? Keep the counter/page reset as this PR does, then:
```java
LongArray old = longArray;
try {
allocate(initialCapacity);
freeArray(old);
} catch (SparkOutOfMemoryError e) {
old.zeroOut(); // or skip allocation entirely when old is already initial capacity
}
```
`allocate()` assigns `longArray` / `growthThreshold` / `mask` only after `allocateArray` returns, so a failed allocation leaves the old fields consistent. At `reset()` time the array borrowed by `UnsafeKVExternalSorter` is no longer referenced (`inMemSorter = null` after the spill, and that in-memory sorter has a `null` consumer). The map would always stay valid, and the two volatile flags, `restoreArrayAfterFailedReset()` with its four call sites, `synchronized reset()` / `free()`, the `TestMemoryManager` hook, and the ThreadMXBean test would all be unnecessary. The cost is briefly holding the old array plus a small initial-capacity array.
##########
core/src/test/scala/org/apache/spark/memory/TestMemoryManager.scala:
##########
@@ -33,27 +34,40 @@ class TestMemoryManager(conf: SparkConf)
private var available = Long.MaxValue
@GuardedBy("this")
private val memoryForTask = mutable.HashMap[Long, Long]().withDefaultValue(0L)
+ private val beforeNextExecutionMemoryGrant = new AtomicReference[Runnable]()
+
+ /** Runs one test action before the next grant, outside this manager's accounting lock. */
+ def runBeforeNextExecutionMemoryGrant(action: Runnable): Unit = {
Review Comment:
Instead of adding a public hook and restructuring `acquireExecutionMemory` in this shared test class (used by ~26 test files), could the suite use the existing pattern of an anonymous `TaskMemoryManager` subclass that injects a one-shot action? Examples: `ShuffleExternalSorterSuite.scala` (`new TaskMemoryManager(memoryManager, 0) { override def acquireExecutionMemory(...) }`, and the `spillOnGrowthAllocation` one-shot flag) and `TaskMemoryManagerSuite.java`. `AbstractBytesToBytesMapSuite.taskMemoryManager` is non-final, so the tearDown leak check still works. As a bonus, the action would then run outside the `TaskMemoryManager` monitor.
##########
core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java:
##########
@@ -875,4 +879,362 @@ public void freeAfterFailedReset() {
}
}
+ @Test
+ public void lookupAfterFreeDoesNotRestoreArray() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ map.free();
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ map.reset();
+ assertEquals(0, map.getNumDataPages());
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ assertThrows(
+ IllegalStateException.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void lookupAfterDestructiveIterationDoesNotRestoreArray() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ map.destructiveIterator();
+ memoryManager.limit(PAGE_SIZE_BYTES);
+
+ assertThrows(
+ IllegalStateException.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void resetAfterDestructiveIterationFails() {
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ map.destructiveIterator();
+ assertThrows(IllegalStateException.class, map::reset);
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void arrayAccessorsRecoverAfterFailedReset() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ assertEquals(512L, map.getArray().size());
+
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ assertEquals(256, map.maxNumKeysIndex());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapAfterFailedReset() {
+ // A reset() whose eager longArray reallocation OOMs leaves the map with longArray == null (and,
+ // because the trailing reset() bookkeeping never runs, a stale currentPage). A subsequent
+ // lookup then NPEs in safeLookup (this.longArray is null), and a later append would write into
+ // a freed page. The map must instead recover to a usable empty state.
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ final long[] key = new long[]{1L};
+ final long[] value = new long[]{42L};
+ BytesToBytesMap.Location loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ loc.append(key, Platform.LONG_ARRAY_OFFSET, 8, value, Platform.LONG_ARRAY_OFFSET, 8);
+
+ // Force OOM on the reallocation inside reset(), leaving longArray == null.
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+
+ // The map must recover: lookup must not NPE, it must report an empty map, and it must be
+ // usable again (append into a fresh page, then read the value back).
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ Assertions.assertFalse(loc.isDefined());
+ Assertions.assertEquals(0, map.numValues());
+ Assertions.assertTrue(
+ loc.append(key, Platform.LONG_ARRAY_OFFSET, 8, value, Platform.LONG_ARRAY_OFFSET, 8));
+ Assertions.assertEquals(1, map.getNumDataPages());
+ loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ Assertions.assertTrue(loc.isDefined());
+ Assertions.assertEquals(42L, Platform.getLong(loc.getValueBase(), loc.getValueOffset()));
+ Assertions.assertEquals(1, map.numValues());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapWithSemanticKeyOperationsAfterFailedReset() {
+ memoryManager.limit(5000);
+ BytesToBytesMap.KeyOperationsFactory caseInsensitiveWithConstantHash =
+ () -> new BytesToBytesMap.KeyOperations() {
+ @Override
+ public int hash(Object base, long offset, int length) {
+ return 0;
+ }
+
+ @Override
+ public boolean equals(
+ Object leftBase,
+ long leftOffset,
+ int leftLength,
+ Object rightBase,
+ long rightOffset,
+ int rightLength) {
+ return Character.toLowerCase(Platform.getByte(leftBase, leftOffset)) ==
+ Character.toLowerCase(Platform.getByte(rightBase, rightOffset));
+ }
+ };
+ BytesToBytesMap map = new BytesToBytesMap(
+ taskMemoryManager,
+ blockManager,
+ serializerManager,
+ 256,
+ 0.5,
+ 4000,
+ caseInsensitiveWithConstantHash);
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ memoryManager.limit(PAGE_SIZE_BYTES);
+
+ byte[] storedKey = new byte[8];
+ storedKey[0] = 'A';
+ byte[] equivalentKey = new byte[16];
+ equivalentKey[0] = 'a';
+ byte[] value = new byte[8];
+ Platform.putLong(value, Platform.BYTE_ARRAY_OFFSET, 42L);
+
+ BytesToBytesMap.Location loc =
+ map.lookup(storedKey, Platform.BYTE_ARRAY_OFFSET, storedKey.length);
+ assertFalse(loc.isDefined());
+ assertTrue(loc.append(
+ storedKey,
+ Platform.BYTE_ARRAY_OFFSET,
+ storedKey.length,
+ value,
+ Platform.BYTE_ARRAY_OFFSET,
+ value.length));
+
+ loc = map.lookup(equivalentKey, Platform.BYTE_ARRAY_OFFSET, equivalentKey.length);
+ assertTrue(loc.isDefined());
+ assertEquals(storedKey.length, loc.getKeyLength());
+ assertEquals(42L, Platform.getLong(loc.getValueBase(), loc.getValueOffset()));
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapAfterFailedResetRestoresGrowthState() {
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ // A sub-one-key growth threshold makes canGrowArray observable through append(): after a
+ // failed grow, no new key is accepted until reset restores the flag.
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 64, 0.01, 4000);
+ try {
+ for (long i = 0; i < 2; i++) {
+ final long[] key = new long[]{i};
+ if (i == 1) {
+ // The data page already exists, so this OOM is consumed by growAndRehash().
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ }
+ assertTrue(map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8).append(
+ key, Platform.LONG_ARRAY_OFFSET, 8, key, Platform.LONG_ARRAY_OFFSET, 8));
+ }
+
+ final long[] blockedKey = new long[]{2L};
+ assertFalse(map.lookup(blockedKey, Platform.LONG_ARRAY_OFFSET, 8).append(
+ blockedKey,
+ Platform.LONG_ARRAY_OFFSET,
+ 8,
+ blockedKey,
+ Platform.LONG_ARRAY_OFFSET,
+ 8));
+
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ for (long i = 10; i < 12; i++) {
+ final long[] key = new long[]{i};
+ final long[] value = new long[]{i * 10};
+ BytesToBytesMap.Location loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ assertFalse(loc.isDefined());
+ assertTrue(loc.append(
+ key, Platform.LONG_ARRAY_OFFSET, 8, value, Platform.LONG_ARRAY_OFFSET, 8));
+ loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ assertTrue(loc.isDefined());
+ assertEquals(i * 10, Platform.getLong(loc.getValueBase(), loc.getValueOffset()));
+ }
+ assertEquals(2, map.numKeys());
+ assertEquals(2, map.numValues());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapAfterFailedResetPropagatesRepeatedOom() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+
+ final long[] key = new long[]{1L};
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(
+ SparkOutOfMemoryError.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ BytesToBytesMap.Location loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ assertFalse(loc.isDefined());
+ assertEquals(0, map.numKeys());
+ assertEquals(0, map.numValues());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapAfterRepeatedFailedReset() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ final long[] key = new long[]{1L};
+ BytesToBytesMap.Location loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ assertFalse(loc.isDefined());
+ assertEquals(0, map.numKeys());
+ assertEquals(0, map.numValues());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void safeLookupAfterFailedReset() throws Exception {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ ExecutorService executor = Executors.newFixedThreadPool(2);
+ CountDownLatch releaseRecoveryAllocation = new CountDownLatch(1);
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ final long[] key = new long[]{1L};
+ CountDownLatch recoveryAllocationEntered = new CountDownLatch(1);
+ memoryManager.runBeforeNextExecutionMemoryGrant(() -> {
+ recoveryAllocationEntered.countDown();
+ try {
+ if (!releaseRecoveryAllocation.await(30, TimeUnit.SECONDS)) {
+ throw new AssertionError("Recovery allocation was not released");
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new AssertionError("Recovery allocation was interrupted", e);
+ }
+ });
+
+ Future<Boolean> firstLookup = executor.submit(() -> {
+ BytesToBytesMap.Location threadLocation = map.new Location();
+ map.safeLookup(key, Platform.LONG_ARRAY_OFFSET, 8, threadLocation);
+ return threadLocation.isDefined();
+ });
+ assertTrue(recoveryAllocationEntered.await(10, TimeUnit.SECONDS));
+
+ AtomicReference<Thread> secondWorker = new AtomicReference<>();
+ CountDownLatch secondStarted = new CountDownLatch(1);
+ Future<Boolean> secondLookup = executor.submit(() -> {
+ BytesToBytesMap.Location threadLocation = map.new Location();
+ secondWorker.set(Thread.currentThread());
+ secondStarted.countDown();
+ map.safeLookup(key, Platform.LONG_ARRAY_OFFSET, 8, threadLocation);
+ return threadLocation.isDefined();
+ });
+ assertTrue(secondStarted.await(10, TimeUnit.SECONDS));
+
+ // The first lookup is held during allocation. Verify that the second has reached the
+ // synchronized recovery gate before allowing the first to publish the replacement array.
+ boolean contended = false;
+ long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10);
+ ThreadMXBean threadBean = ManagementFactory.getThreadMXBean();
+ while (System.nanoTime() < deadline && !secondLookup.isDone()) {
+ ThreadInfo info = threadBean.getThreadInfo(secondWorker.get().getId());
+ if (info != null && info.getThreadState() == Thread.State.BLOCKED &&
+ info.getLockInfo() != null &&
+ info.getLockInfo().getClassName().equals(BytesToBytesMap.class.getName()) &&
+ info.getLockInfo().getIdentityHashCode() == System.identityHashCode(map)) {
+ contended = true;
+ break;
+ }
+ TimeUnit.MILLISECONDS.sleep(1);
+ }
+ assertTrue(contended, "Second lookup never contended for recovery");
+ assertFalse(firstLookup.isDone());
+ releaseRecoveryAllocation.countDown();
+
+ assertFalse(firstLookup.get(10, TimeUnit.SECONDS));
+ assertFalse(secondLookup.get(10, TimeUnit.SECONDS));
+ assertEquals(0, map.getNumDataPages());
+ assertEquals(
+ map.getTotalMemoryConsumption(), taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ BytesToBytesMap.Location location = map.new Location();
+ map.safeLookup(key, Platform.LONG_ARRAY_OFFSET, 8, location, 42);
+ assertFalse(location.isDefined());
+ } finally {
+ releaseRecoveryAllocation.countDown();
+ executor.shutdownNow();
Review Comment:
`shutdownNow()` does not wait for the workers. If an assertion above fails (e.g. `assertTrue(contended, ...)` on a loaded CI), the released second worker can already be in the lock-free probe loop of `safeLookup` when `map.free()` frees and nulls `longArray`. It then hits an NPE, or in the off-heap suite reads freed native memory. The success path is fine because both futures are `get()`-ed first. Adding `executor.awaitTermination(...)` before `map.free()` would close this window.
##########
core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java:
##########
@@ -875,4 +879,362 @@ public void freeAfterFailedReset() {
}
}
+ @Test
+ public void lookupAfterFreeDoesNotRestoreArray() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ map.free();
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ map.reset();
+ assertEquals(0, map.getNumDataPages());
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ assertThrows(
+ IllegalStateException.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void lookupAfterDestructiveIterationDoesNotRestoreArray() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ map.destructiveIterator();
+ memoryManager.limit(PAGE_SIZE_BYTES);
+
+ assertThrows(
+ IllegalStateException.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void resetAfterDestructiveIterationFails() {
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ map.destructiveIterator();
+ assertThrows(IllegalStateException.class, map::reset);
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void arrayAccessorsRecoverAfterFailedReset() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ assertEquals(512L, map.getArray().size());
+
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ assertEquals(256, map.maxNumKeysIndex());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapAfterFailedReset() {
+ // A reset() whose eager longArray reallocation OOMs leaves the map with longArray == null (and,
+ // because the trailing reset() bookkeeping never runs, a stale currentPage). A subsequent
+ // lookup then NPEs in safeLookup (this.longArray is null), and a later append would write into
+ // a freed page. The map must instead recover to a usable empty state.
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ final long[] key = new long[]{1L};
+ final long[] value = new long[]{42L};
+ BytesToBytesMap.Location loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ loc.append(key, Platform.LONG_ARRAY_OFFSET, 8, value, Platform.LONG_ARRAY_OFFSET, 8);
Review Comment:
This setup `append` is what creates the data page the stale-`currentPage` regression check depends on, but its return value is ignored. With `limit(5000)`, the 4096-byte pointer array leaves 904 bytes, so it only succeeds because `TestMemoryManager` grants a partial page. If that ever changes and `append` returns `false`, no page exists before `reset()`, and all later assertions still pass while the test no longer covers the `currentPage` / `pageCursor` reordering. Could we wrap it in `assertTrue(...)`?
##########
core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java:
##########
@@ -875,4 +879,362 @@ public void freeAfterFailedReset() {
}
}
+ @Test
+ public void lookupAfterFreeDoesNotRestoreArray() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ map.free();
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ map.reset();
+ assertEquals(0, map.getNumDataPages());
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ assertThrows(
+ IllegalStateException.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void lookupAfterDestructiveIterationDoesNotRestoreArray() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ map.destructiveIterator();
+ memoryManager.limit(PAGE_SIZE_BYTES);
+
+ assertThrows(
+ IllegalStateException.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void resetAfterDestructiveIterationFails() {
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ map.destructiveIterator();
+ assertThrows(IllegalStateException.class, map::reset);
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void arrayAccessorsRecoverAfterFailedReset() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ assertEquals(512L, map.getArray().size());
+
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ assertEquals(256, map.maxNumKeysIndex());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapAfterFailedReset() {
+ // A reset() whose eager longArray reallocation OOMs leaves the map with longArray == null (and,
+ // because the trailing reset() bookkeeping never runs, a stale currentPage). A subsequent
+ // lookup then NPEs in safeLookup (this.longArray is null), and a later append would write into
+ // a freed page. The map must instead recover to a usable empty state.
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ final long[] key = new long[]{1L};
+ final long[] value = new long[]{42L};
+ BytesToBytesMap.Location loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ loc.append(key, Platform.LONG_ARRAY_OFFSET, 8, value, Platform.LONG_ARRAY_OFFSET, 8);
+
+ // Force OOM on the reallocation inside reset(), leaving longArray == null.
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+
+ // The map must recover: lookup must not NPE, it must report an empty map, and it must be
+ // usable again (append into a fresh page, then read the value back).
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ Assertions.assertFalse(loc.isDefined());
+ Assertions.assertEquals(0, map.numValues());
+ Assertions.assertTrue(
+ loc.append(key, Platform.LONG_ARRAY_OFFSET, 8, value, Platform.LONG_ARRAY_OFFSET, 8));
+ Assertions.assertEquals(1, map.getNumDataPages());
+ loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ Assertions.assertTrue(loc.isDefined());
+ Assertions.assertEquals(42L, Platform.getLong(loc.getValueBase(), loc.getValueOffset()));
+ Assertions.assertEquals(1, map.numValues());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapWithSemanticKeyOperationsAfterFailedReset() {
+ memoryManager.limit(5000);
+ BytesToBytesMap.KeyOperationsFactory caseInsensitiveWithConstantHash =
Review Comment:
This `KeyOperationsFactory` (constant hash 0, case-insensitive first byte) and the `'A'` / `'a'` fixtures duplicate `supportsSemanticKeyOperations`. The `limit(5000)` / `new BytesToBytesMap(..., 256, 0.5, 4000)` / `markExecutionAsOutOfMemoryOnce()` / `assertThrows(SparkOutOfMemoryError.class, map::reset)` block is also repeated in about eight new tests, with 256/4000 implicitly tied to the expected `512L` and `256` elsewhere. A shared factory plus a small `newMapWithFailedReset()` helper would remove a good amount of duplication.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -587,12 +604,50 @@ public Location lookup(Object keyBase, long keyOffset, int keyLength, int hash)
return loc;
}
+ /**
+ * Restores the hash array after a {@link #reset()} freed it but its eager reallocation failed
+ * with an OOM, leaving {@code longArray == null}. A reset() empties the map, so re-allocating at
+ * {@code initialCapacity} here is correct. If memory is still unavailable this throws a
+ * SparkOutOfMemoryError instead of exposing the null array.
+ */
+ private void restoreArrayAfterFailedReset() {
+ if (freed) {
Review Comment:
This adds two volatile reads (`freed`, then `longArrayRecoveryRequired`) to every `lookup` / `safeLookup`, which is the per-row hot path of hash aggregation (`UnsafeFixedWidthAggregationMap`) and hash-join probes (`UnsafeHashedRelation`). On AArch64 these become serial `ldar` loads ahead of the probe. One read is enough: if `free()` set `longArrayRecoveryRequired = true` instead of clearing it (or with a single volatile state such as READY / NEEDS_RECOVERY / FREED), the locked slow path would throw the same `IllegalStateException` through its inner `freed` check, with the same safe publication. `longArray == null &&` at L633 is also always true under the monitor. The impact is probably small; I haven't measured it.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -140,6 +140,7 @@ public interface KeyOperationsFactory {
* while position {@code 2 * i + 1} in the array holds key's full 32-bit hashcode.
*/
@Nullable private LongArray longArray;
+
Review Comment:
Nit: this hunk only adds a blank line between `longArray` and the TODO that documents it. It looks like a leftover from moving the new fields; could we drop it so the TODO stays directly under its field?
##########
core/src/test/scala/org/apache/spark/memory/TestMemoryManager.scala:
##########
@@ -33,27 +34,40 @@ class TestMemoryManager(conf: SparkConf)
private var available = Long.MaxValue
@GuardedBy("this")
private val memoryForTask = mutable.HashMap[Long, Long]().withDefaultValue(0L)
+ private val beforeNextExecutionMemoryGrant = new AtomicReference[Runnable]()
+
+ /** Runs one test action before the next grant, outside this manager's accounting lock. */
Review Comment:
Nit: the action does run outside `TestMemoryManager`'s own lock, but it is invoked from `TaskMemoryManager.acquireExecutionMemory` inside `synchronized (this)`. So a blocking action holds the `TaskMemoryManager` monitor, and in `safeLookupAfterFailedReset` the `BytesToBytesMap` monitor as well. The current test is fine because the main thread doesn't touch the `TaskMemoryManager` while the action is blocked. But a future test whose blocked action waits on a thread that uses the same `TaskMemoryManager` would hang until the latch timeout. Could the doc mention this?
##########
core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java:
##########
@@ -875,4 +879,362 @@ public void freeAfterFailedReset() {
}
}
+ @Test
+ public void lookupAfterFreeDoesNotRestoreArray() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ map.free();
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ map.reset();
+ assertEquals(0, map.getNumDataPages());
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ assertThrows(
+ IllegalStateException.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void lookupAfterDestructiveIterationDoesNotRestoreArray() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ map.destructiveIterator();
+ memoryManager.limit(PAGE_SIZE_BYTES);
+
+ assertThrows(
+ IllegalStateException.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void resetAfterDestructiveIterationFails() {
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ map.destructiveIterator();
+ assertThrows(IllegalStateException.class, map::reset);
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void arrayAccessorsRecoverAfterFailedReset() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager, 256, 0.5, 4000);
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ assertEquals(512L, map.getArray().size());
+
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ assertEquals(256, map.maxNumKeysIndex());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapAfterFailedReset() {
+ // A reset() whose eager longArray reallocation OOMs leaves the map with longArray == null (and,
Review Comment:
This comment describes the pre-fix behavior in the present tense. After this PR, the bookkeeping (`currentPage = null; pageCursor = 0;`) runs before `allocate()` and `safeLookup` recovers first, so "the trailing reset() bookkeeping never runs" and "A subsequent lookup then NPEs" are no longer true. Maybe rephrase as "Before this fix, ..." or just state the invariant being tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59757][CORE] Harden BytesToBytesMap after failed reset allocation [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #59009:
URL: https://github.com/apache/spark/pull/59009#discussion_r4093738228
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -612,9 +667,13 @@ public void safeLookup(Object keyBase, long keyOffset, int keyLength, Location l
*
* The provided hash is ignored when this map has configured key operations. Each thread must
* supply its own {@link Location}. Probe statistics may be inaccurate under concurrent lookup,
- * and the map must not be modified concurrently.
+ * and the map must not otherwise be modified concurrently. Concurrent calls may safely retry a
+ * pointer-array allocation that failed during {@link #reset()}.
+ *
+ * @throws SparkOutOfMemoryError if a pointer-array allocation retried after reset still fails
*/
public void safeLookup(Object keyBase, long keyOffset, int keyLength, Location loc, int hash) {
+ restoreArrayAfterFailedReset();
Review Comment:
In production `reset()` has a single caller, `UnsafeKVExternalSorter:179`, and its call sites fall into two shapes, neither of which reaches reuse after a failed reset. The two mid-loop spills do re-look-up right after spilling (`TungstenAggregationIterator:247` then `:258`; generated `HashAggregateExec:993` then `:997`), but `SparkOutOfMemoryError` is an `Error`, `NonFatal` does not match it, and neither path has a catch, so the re-lookup line never runs. The two drain sites (`TungstenAggregationIterator:282`, `HashAggregateExec:344`) call `free()` immediately after spilling and never look up again.
The recovery logic itself reads fine to me. What I cannot match is the stated user-facing change, "a hash aggregation that reaches this failed-reset state can continue": with today's callers the task ends with `SparkOutOfMemoryError` either way. Did I miss a caller that keeps looking up after `reset()` throws, or is this intended as hardening? If it is hardening, one sentence in the description covers it.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -140,6 +140,18 @@ public interface KeyOperationsFactory {
* while position {@code 2 * i + 1} in the array holds key's full 32-bit hashcode.
*/
@Nullable private LongArray longArray;
+
+ /**
+ * Whether a completed reset needs to restore longArray after its allocation failed.
+ *
+ * Volatile so that clearing this flag safely publishes the replacement array to concurrent
+ * safeLookup callers.
+ */
+ private volatile boolean longArrayRecoveryRequired = false;
+
+ /** Whether {@link #free()} has permanently released this map. */
+ private volatile boolean freed = false;
Review Comment:
The TODO block starting at `:155` is about the 32-bit hashcode kept in `longArray`'s odd slots ("we're wasting 32 bits of space here", and later "this array will fit in cache"); it used to sit directly under the field at `:142`. With the two new flags inserted in between, that "here" now reads as if it were about `freed`.
Moving the two flags below the TODO block fixes it. AGENTS.md puts it as: place a new member with the code it belongs with, rather than splitting a section of code that belonged together.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -1109,30 +1171,50 @@ public int getNumDataPages() {
}
/**
- * Returns the underline long[] of longArray.
+ * Returns the underlying long array. If the pointer-array allocation in a previous
+ * {@link #reset()} failed, this method retries that allocation before returning.
+ *
+ * @throws SparkOutOfMemoryError if the pointer array still cannot be allocated
*/
public LongArray getArray() {
+ restoreArrayAfterFailedReset();
assert(longArray != null);
return longArray;
}
/**
- * Reset this map to initialized state.
+ * Reset this map to initialized state. If {@link #free()} has been called, the map cannot be
+ * reinitialized and this method is a no-op. If the replacement pointer array cannot be
+ * allocated, the map remains empty and the next lookup, {@link #getArray()}, or
+ * {@link #maxNumKeysIndex()} retries the allocation.
+ *
+ * @throws SparkOutOfMemoryError if the replacement pointer array cannot be allocated
*/
- public void reset() {
+ public synchronized void reset() {
+ if (freed) {
+ return;
+ }
updatePeakMemoryUsed();
+ // Put the map into its empty state up front so that if the allocate() below fails with an OOM,
+ // the map is left consistently empty (longArray == null, no stale currentPage) rather than
+ // half-reset. A subsequent lookup then restores the array lazily (restoreArrayAfterFailedReset)
+ // instead of dereferencing a null longArray or writing into a freed currentPage.
numKeys = 0;
numValues = 0;
- freeArray(longArray);
+ canGrowArray = true;
+ currentPage = null;
+ pageCursor = 0;
+ if (longArray != null) {
+ freeArray(longArray);
+ }
longArray = null;
while (dataPages.size() > 0) {
MemoryBlock dataPage = dataPages.removeLast();
freePage(dataPage);
}
+ longArrayRecoveryRequired = true;
Review Comment:
I cannot find the combination this concurrent recovery is for. The only map that is ever `reset()` is the aggregation map, which is task-private and reached only through `lookup()`. The only map read by concurrent `safeLookup` is `UnsafeHashedRelation.binaryMap`, where `asReadOnlyCopy` hands each thread a wrapper around one shared map, and that one is only ever `free()`d, never reset. Point me at a usage I did not scan.
The flag is also published after the array is released: `:1210` nulls `longArray` and only `:1215` sets the flag, so a `safeLookup` landing in between gets neither. The contract at `:647` rules this concurrency out anyway, so it cannot happen today.
Either narrow the comment to callers arriving after `reset()` returns, or move `:1215` above `:1207`. The move costs nothing on the success path, since `reset()` holds the monitor throughout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59757][CORE] Harden BytesToBytesMap after failed reset allocation [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #59009: [SPARK-59757][CORE] Harden BytesToBytesMap after failed reset allocation
URL: https://github.com/apache/spark/pull/59009
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59757][CORE] Harden BytesToBytesMap after failed reset allocation [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #59009:
URL: https://github.com/apache/spark/pull/59009#issuecomment-5834965343
Thank you for working on this, @cloud-fan. Moving the `canGrowArray` / `currentPage` / `pageCursor` reset ahead of the allocation is a good fix for the stale-page issue.
Here is a summary of my review comments above.
**Design**
- The root cause looks like `reset()` discarding the old pointer array before securing a replacement. Allocating first and falling back to `old.zeroOut()` on OOM would keep the map always valid. It would also remove the lazy-recovery machinery: the volatile flags, `restoreArrayAfterFailedReset()`, `synchronized reset()` / `free()`, the new map -> `TaskMemoryManager` lock order, and the `TestMemoryManager` hook.
- The concurrent-recovery contract has no in-tree caller. The maps probed concurrently (broadcast `UnsafeHashedRelation`) are never reset.
- The only `reset()` caller is `UnsafeKVExternalSorter`. If a caller there recovered from the OOM, it would silently lose the spilled groups. This is latent, since no in-tree caller catches that OOM today.
**Behavior**
- `lookup()` / `safeLookup()` can now allocate memory and throw OOM. That bypasses the spill fallback in `UnsafeFixedWidthAggregationMap`, which expects `append()` to return `false` instead.
- The `destructiveIterator` check runs only in recovery mode.
- `reset()` after `free()` is a silent no-op. That is inconsistent with the other methods and with the new `@throws`.
**Performance**
- Every lookup on the hash aggregation / hash join hot paths now does two volatile reads, where one would be enough.
**Tests**
- `safeLookupAfterFailedReset` calls `map.free()` without an `awaitTermination` first.
- The setup `append` in `reuseMapAfterFailedReset` is unchecked.
- One test comment describes the pre-fix behavior.
- Some fixtures are duplicated.
- The `TestMemoryManager` hook's doc is incomplete, and the existing anonymous `TaskMemoryManager` subclass pattern could be reused instead.
**Minor**
- There is a stray blank line in `BytesToBytesMap`.
- The PR description gives the test count as 54, but it should be 56.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59757][CORE] Harden BytesToBytesMap after failed reset allocation [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #59009:
URL: https://github.com/apache/spark/pull/59009#discussion_r4103085396
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -1109,30 +1171,50 @@ public int getNumDataPages() {
}
/**
- * Returns the underline long[] of longArray.
+ * Returns the underlying long array. If the pointer-array allocation in a previous
+ * {@link #reset()} failed, this method retries that allocation before returning.
+ *
+ * @throws SparkOutOfMemoryError if the pointer array still cannot be allocated
*/
public LongArray getArray() {
+ restoreArrayAfterFailedReset();
assert(longArray != null);
return longArray;
}
/**
- * Reset this map to initialized state.
+ * Reset this map to initialized state. If {@link #free()} has been called, the map cannot be
+ * reinitialized and this method is a no-op. If the replacement pointer array cannot be
+ * allocated, the map remains empty and the next lookup, {@link #getArray()}, or
+ * {@link #maxNumKeysIndex()} retries the allocation.
+ *
+ * @throws SparkOutOfMemoryError if the replacement pointer array cannot be allocated
*/
- public void reset() {
+ public synchronized void reset() {
+ if (freed) {
+ return;
+ }
Review Comment:
Addressed in 64eaea7f1b9. reset() now throws IllegalStateException once destructive iteration has begun, before any reset bookkeeping or allocation. resetAfterDestructiveIterationFails pins the terminal state. Both map suites pass: 28 tests each, 56 total.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -1109,30 +1171,50 @@ public int getNumDataPages() {
}
/**
- * Returns the underline long[] of longArray.
+ * Returns the underlying long array. If the pointer-array allocation in a previous
+ * {@link #reset()} failed, this method retries that allocation before returning.
+ *
+ * @throws SparkOutOfMemoryError if the pointer array still cannot be allocated
*/
public LongArray getArray() {
+ restoreArrayAfterFailedReset();
assert(longArray != null);
return longArray;
}
/**
- * Reset this map to initialized state.
+ * Reset this map to initialized state. If {@link #free()} has been called, the map cannot be
+ * reinitialized and this method is a no-op. If the replacement pointer array cannot be
+ * allocated, the map remains empty and the next lookup, {@link #getArray()}, or
+ * {@link #maxNumKeysIndex()} retries the allocation.
+ *
+ * @throws SparkOutOfMemoryError if the replacement pointer array cannot be allocated
*/
- public void reset() {
+ public synchronized void reset() {
+ if (freed) {
+ return;
+ }
updatePeakMemoryUsed();
+ // Put the map into its empty state up front so that if the allocate() below fails with an OOM,
+ // the map is left consistently empty (longArray == null, no stale currentPage) rather than
+ // half-reset. A subsequent lookup then restores the array lazily (restoreArrayAfterFailedReset)
+ // instead of dereferencing a null longArray or writing into a freed currentPage.
numKeys = 0;
numValues = 0;
- freeArray(longArray);
+ canGrowArray = true;
+ currentPage = null;
+ pageCursor = 0;
+ if (longArray != null) {
+ freeArray(longArray);
+ }
longArray = null;
while (dataPages.size() > 0) {
MemoryBlock dataPage = dataPages.removeLast();
freePage(dataPage);
}
+ longArrayRecoveryRequired = true;
Review Comment:
Agreed that there is no current in-tree caller combining reset() with concurrent safeLookup(). The intent is to preserve safeLookup's thread-safe contract for callers that catch a failed reset. In 64eaea7f1b9 I clarified that concurrent retry is supported only after reset() has completed exceptionally; overlap with reset() remains prohibited. I kept the flag after page cleanup because moving it earlier could make a partially cleaned map recoverable if freePage() unexpectedly failed, allowing stale pages to remain visible.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -612,9 +667,13 @@ public void safeLookup(Object keyBase, long keyOffset, int keyLength, Location l
*
* The provided hash is ignored when this map has configured key operations. Each thread must
* supply its own {@link Location}. Probe statistics may be inaccurate under concurrent lookup,
- * and the map must not be modified concurrently.
+ * and the map must not otherwise be modified concurrently. Concurrent calls may safely retry a
+ * pointer-array allocation that failed during {@link #reset()}.
+ *
+ * @throws SparkOutOfMemoryError if a pointer-array allocation retried after reset still fails
*/
public void safeLookup(Object keyBase, long keyOffset, int keyLength, Location loc, int hash) {
+ restoreArrayAfterFailedReset();
Review Comment:
You did not miss a caller. This is intended as hardening of BytesToBytesMap's contract for callers that recover; current in-tree hash aggregation still propagates SparkOutOfMemoryError from reset(). The PR description now says there is no in-tree production caller that catches it and marks the user-facing change as No.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -140,6 +140,18 @@ public interface KeyOperationsFactory {
* while position {@code 2 * i + 1} in the array holds key's full 32-bit hashcode.
*/
@Nullable private LongArray longArray;
+
+ /**
+ * Whether a completed reset needs to restore longArray after its allocation failed.
+ *
+ * Volatile so that clearing this flag safely publishes the replacement array to concurrent
+ * safeLookup callers.
+ */
+ private volatile boolean longArrayRecoveryRequired = false;
+
+ /** Whether {@link #free()} has permanently released this map. */
+ private volatile boolean freed = false;
Review Comment:
Addressed in 64eaea7f1b9. The longArray TODO is directly below the field again, and the two new state flags now follow the complete TODO block.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org