spark git commit: [SPARK-18557] Downgrade confusing memory leak warning message

[hv...@apache.org]

Repository: spark
Updated Branches:
  refs/heads/master 84284e8c8 -> 9785ed40d


[SPARK-18557] Downgrade confusing memory leak warning message

## What changes were proposed in this pull request?
TaskMemoryManager has a memory leak detector that gets called at task completion callback and checks whether any memory has not been released. If they are not released by the time the callback is invoked, TaskMemoryManager releases them.

The current error message says something like the following:
```
WARN  [Executor task launch worker-0]
org.apache.spark.memory.TaskMemoryManager - leak 16.3 MB memory from
org.apache.spark.unsafe.map.BytesToBytesMap33fb6a15
In practice, there are multiple reasons why these can be triggered in the normal code path (e.g. limit, or task failures), and the fact that these messages are log means the "leak" is fixed by TaskMemoryManager.
```

To not confuse users, this patch downgrade the message from warning to debug level, and avoids using the word "leak" since it is not actually a leak.

## How was this patch tested?
N/A - this is a simple logging improvement.

Author: Reynold Xin <rx...@databricks.com>

Closes #15989 from rxin/SPARK-18557.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9785ed40
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9785ed40
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9785ed40

Branch: refs/heads/master
Commit: 9785ed40d7fe4e1fcd440e55706519c6e5f8d6b1
Parents: 84284e8
Author: Reynold Xin <rx...@databricks.com>
Authored: Wed Nov 23 04:22:26 2016 -0800
Committer: Herman van Hovell <hv...@databricks.com>
Committed: Wed Nov 23 04:22:26 2016 -0800

----------------------------------------------------------------------
 .../src/main/java/org/apache/spark/memory/TaskMemoryManager.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/9785ed40/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java
----------------------------------------------------------------------
diff --git a/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java b/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java
index 1a700aa..c40974b 100644
--- a/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java
+++ b/core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java
@@ -378,14 +378,14 @@ public class TaskMemoryManager {
       for (MemoryConsumer c: consumers) {
         if (c != null && c.getUsed() > 0) {
           // In case of failed task, it's normal to see leaked memory
-          logger.warn("leak " + Utils.bytesToString(c.getUsed()) + " memory from " + c);
+          logger.debug("unreleased " + Utils.bytesToString(c.getUsed()) + " memory from " + c);
         }
       }
       consumers.clear();
 
       for (MemoryBlock page : pageTable) {
         if (page != null) {
-          logger.warn("leak a page: " + page + " in task " + taskAttemptId);
+          logger.debug("unreleased page: " + page + " in task " + taskAttemptId);
           memoryManager.tungstenMemoryAllocator().free(page);
         }
       }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org