[spark] branch branch-2.4 updated: [SPARK-27558][CORE] Gracefully cleanup task when it fails with OOM exception

[sr...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

srowen pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git


View the commit online:
https://github.com/apache/spark/commit/12ec3387af1ab5d0e5687f9b8277b3c96b7cc11f

The following commit(s) were added to refs/heads/branch-2.4 by this push:
     new 12ec338  [SPARK-27558][CORE] Gracefully cleanup task when it fails with OOM exception
12ec338 is described below

commit 12ec3387af1ab5d0e5687f9b8277b3c96b7cc11f
Author: yudovin <ar...@profitero.com>
AuthorDate: Wed Nov 20 18:18:29 2019 -0600

    [SPARK-27558][CORE] Gracefully cleanup task when it fails with OOM exception
    
    ### What changes were proposed in this pull request?
    When a task fails with OOM exception, the UnsafeInMemorySorter.array could be null. In the meanwhile, the cleanupResources() on task completion would call UnsafeInMemorySorter.getMemoryUsage in turn, and that lead to another NPE thrown.
    ### Why are the changes needed?
    Check if array is null in UnsafeInMemorySorter.getMemoryUsage and it should help to avoid NPE.
    ### Does this PR introduce any user-facing change?
    No
    ### How was this patch tested?
    It was tested manually.
    
    Closes #26606 from ayudovin/fix-npe-in-listener-2.4.
    
    Authored-by: yudovin <ar...@profitero.com>
    Signed-off-by: Sean Owen <se...@databricks.com>
---
 .../spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java       | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java b/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
index 75690ae..aedc7ec 100644
--- a/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
+++ b/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java
@@ -205,6 +205,10 @@ public final class UnsafeInMemorySorter {
   }
 
   public long getMemoryUsage() {
+    if (array == null) {
+      return 0L;
+    }
+
     return array.size() * 8;
   }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org