You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ganesha S (Jira)" <ji...@apache.org> on 2026/08/03 03:58:00 UTC
[jira] [Updated] (SPARK-58505) Attach per-consumer memory breakdown to UNABLE_TO_ACQUIRE_MEMORY task errors
[ https://issues.apache.org/jira/browse/SPARK-58505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ganesha S updated SPARK-58505:
------------------------------
Description:
h3. Problem
When a task cannot acquire execution memory, Spark throws a SparkOutOfMemoryError
with error class UNABLE_TO_ACQUIRE_MEMORY:
{code:java}
Unable to acquire 8388608 bytes of memory, got 2097152.{code}
This message, which propagates into the TaskFailedReason and surfaces on the driver and in the Spark UI, reports only the requested and received byte counts. It does not indicate which operator was holding the memory, so it gives no direct signal about
the cause of the OOM.
`TaskMemoryManager.showMemoryUsage()` does compute a per-MemoryConsumer breakdown, but it writes that breakdown only to the executor logs. Recovering it after a failure means locating the correct executor's logs and correlating by task-attempt id, which is often impractical (logs rotated/aggregated, executor lost) and is not accessible to the driver or to programmatic/automated diagnosis.
h3. Proposal
Attach the same per-consumer attribution that showMemoryUsage() already logs to the `UNABLE_TO_ACQUIRE_MEMORY` error itself, so it travels with the task failure reason to the driver and the UI. The message becomes, for example:
{code:java}
Unable to acquire 8388608 bytes of memory, got 2097152.
Memory used by task 4211 grouped by consumer:
org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@1a2b: 456.0 MiB
org.apache.spark.unsafe.map.BytesToBytesMap@3c4d: 12.0 MiB
(not attributed to a specific consumer): 3.0 MiB{code}
Consumers are listed largest-first (the most likely culprit surfaces first), followed by the bytes not attributable to any specific consumer.
was:
h3. Problem
When a task cannot acquire execution memory, Spark throws a SparkOutOfMemoryError
with error class UNABLE_TO_ACQUIRE_MEMORY:
```
Unable to acquire 8388608 bytes of memory, got 2097152.
```
This message, which propagates into the TaskFailedReason and surfaces on the
driver and in the Spark UI, reports only the requested and received byte counts. It
does not indicate which operator was holding the memory, so it gives no direct signal about
the cause of the OOM.
`TaskMemoryManager.showMemoryUsage()` does compute a per-MemoryConsumer breakdown, but it
writes that breakdown only to the executor logs. Recovering it after a failure means
locating the correct executor's logs and correlating by task-attempt id, which is often
impractical (logs rotated/aggregated, executor lost) and is not accessible to the driver
or to programmatic/automated diagnosis.
h3. Proposal
Attach the same per-consumer attribution that showMemoryUsage() already logs to the
`UNABLE_TO_ACQUIRE_MEMORY` error itself, so it travels with the task failure reason to the driver and the UI. The message becomes, for example:
```
Unable to acquire 8388608 bytes of memory, got 2097152.
Memory used by task 4211 grouped by consumer:
org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@1a2b: 456.0 MiB
org.apache.spark.unsafe.map.BytesToBytesMap@3c4d: 12.0 MiB
(not attributed to a specific consumer): 3.0 MiB
```
Consumers are listed largest-first (the most likely culprit surfaces first), followed by
the bytes not attributable to any specific consumer.
> Attach per-consumer memory breakdown to UNABLE_TO_ACQUIRE_MEMORY task errors
> ----------------------------------------------------------------------------
>
> Key: SPARK-58505
> URL: https://issues.apache.org/jira/browse/SPARK-58505
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 4.2.0
> Reporter: Ganesha S
> Priority: Major
>
> h3. Problem
> When a task cannot acquire execution memory, Spark throws a SparkOutOfMemoryError
> with error class UNABLE_TO_ACQUIRE_MEMORY:
> {code:java}
> Unable to acquire 8388608 bytes of memory, got 2097152.{code}
> This message, which propagates into the TaskFailedReason and surfaces on the driver and in the Spark UI, reports only the requested and received byte counts. It does not indicate which operator was holding the memory, so it gives no direct signal about
> the cause of the OOM.
> `TaskMemoryManager.showMemoryUsage()` does compute a per-MemoryConsumer breakdown, but it writes that breakdown only to the executor logs. Recovering it after a failure means locating the correct executor's logs and correlating by task-attempt id, which is often impractical (logs rotated/aggregated, executor lost) and is not accessible to the driver or to programmatic/automated diagnosis.
> h3. Proposal
> Attach the same per-consumer attribution that showMemoryUsage() already logs to the `UNABLE_TO_ACQUIRE_MEMORY` error itself, so it travels with the task failure reason to the driver and the UI. The message becomes, for example:
> {code:java}
> Unable to acquire 8388608 bytes of memory, got 2097152.
> Memory used by task 4211 grouped by consumer:
> org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@1a2b: 456.0 MiB
> org.apache.spark.unsafe.map.BytesToBytesMap@3c4d: 12.0 MiB
> (not attributed to a specific consumer): 3.0 MiB{code}
> Consumers are listed largest-first (the most likely culprit surfaces first), followed by the bytes not attributable to any specific consumer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org