[jira] [Created] (SPARK-25008) Add memory mode info to showMemoryUsage in TaskMemoryManager

["Ankur Gupta (JIRA)" <ji...@apache.org>]

Ankur Gupta created SPARK-25008:
-----------------------------------

             Summary: Add memory mode info to showMemoryUsage in TaskMemoryManager
                 Key: SPARK-25008
                 URL: https://issues.apache.org/jira/browse/SPARK-25008
             Project: Spark
          Issue Type: Task
          Components: Spark Core
    Affects Versions: 2.3.0
            Reporter: Ankur Gupta


TaskMemoryManager prints the current memory usage information before throwing an OOM exception which is helpful in debugging issues. This log does not have the memory mode information which can be also useful to quickly determine which memory users need to increase.

This JIRA is to add that information to showMemoryUsage method of TaskMemoryManager.

Current logs:
{code}
18/07/03 17:57:16 INFO memory.TaskMemoryManager: Memory used in task 318
18/07/03 17:57:16 INFO memory.TaskMemoryManager: Acquired by org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@7f084d1b: 1024.0 KB
18/07/03 17:57:16 INFO memory.TaskMemoryManager: Acquired by org.apache.spark.shuffle.sort.ShuffleExternalSorter@713d50f2: 32.0 KB
18/07/03 17:57:16 INFO memory.TaskMemoryManager: 0 bytes of memory were used by task 318 but are not associated with specific consumers
18/07/03 17:57:16 INFO memory.TaskMemoryManager: 1081344 bytes of memory are used for execution and 306201016 bytes of memory are used for storage
18/07/03 17:57:16 ERROR executor.Executor: Exception in task 86.0 in stage 49.0 (TID 318)
java.lang.OutOfMemoryError: Unable to acquire 326284160 bytes of memory, got 3112960
 at org.apache.spark.memory.MemoryConsumer.allocatePage(MemoryConsumer.java:127)
 at org.apache.spark.shuffle.sort.ShuffleExternalSorter.acquireNewPageIfNecessary(ShuffleExternalSorter.java:359)
 at org.apache.spark.shuffle.sort.ShuffleExternalSorter.insertRecord(ShuffleExternalSorter.java:382)
 at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.insertRecordIntoSorter(UnsafeShuffleWriter.java:246)
 at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:167)
 at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
 at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
 at org.apache.spark.scheduler.Task.run(Task.scala:108)
 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org