[spark] branch master updated: [SPARK-27333][TEST] Update thread audit whitelist to skip broadcast-exchange-.*, process reaper and StatisticsDataReferenceCleaner threads

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 9eb896c  [SPARK-27333][TEST] Update thread audit whitelist to skip broadcast-exchange-.*, process reaper and StatisticsDataReferenceCleaner threads
9eb896c is described below

commit 9eb896cc3b67499148dd7c0b7ac694ba2fd9dd32
Author: “attilapiros” <pi...@gmail.com>
AuthorDate: Sun Mar 31 17:33:31 2019 -0700

    [SPARK-27333][TEST] Update thread audit whitelist to skip broadcast-exchange-.*, process reaper and StatisticsDataReferenceCleaner threads
    
    ## What changes were proposed in this pull request?
    
    Update thread audit whitelist to skip threads of the global broadcast exchange thread pool, process reaper and Hadoop FS statistics data reference cleaner thread.
    
    ## How was this patch tested?
    
    Via existing UT using broadcast exchange via `sbt` i.e:
    
    ```
    > project sql
    > testOnly *.SessionStateSuite -- -z "fork new sessions and run query on inherited table"
    ```
    
    Before (wrapped long line for manually to save horizontal scrolling for reviewers):
    
    ```
    ===== POSSIBLE THREAD LEAK IN SUITE o.a.s.sql.SessionStateSuite,
    thread names: broadcast-exchange-6, broadcast-exchange-0,
    broadcast-exchange-2, broadcast-exchange-5, broadcast-exchange-7,
    broadcast-exchange-4, broadcast-exchange-1, process reaper, broadcast-exchange-3,
     org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner =====
    ```
    
    After this change no possible thread leak detected.
    
    Closes #24244 from attilapiros/thread-audit-minor.
    
    Authored-by: “attilapiros” <pi...@gmail.com>
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
 .../test/scala/org/apache/spark/ThreadAudit.scala    | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/core/src/test/scala/org/apache/spark/ThreadAudit.scala b/core/src/test/scala/org/apache/spark/ThreadAudit.scala
index 6b91162..44d1f22 100644
--- a/core/src/test/scala/org/apache/spark/ThreadAudit.scala
+++ b/core/src/test/scala/org/apache/spark/ThreadAudit.scala
@@ -77,7 +77,25 @@ trait ThreadAudit extends Logging {
      * asynchronously. In each case proper stopping is checked manually.
      */
     "shuffle-client.*",
-    "shuffle-server.*"
+    "shuffle-server.*",
+
+    /**
+     * Global cleaner thread that manage statistics data references of Hadoop filesystems.
+     * This is excluded because their lifecycle is handled by Hadoop and spark has no explicit
+     * effect on it.
+     */
+    "org.apache.hadoop.fs.FileSystem\\$Statistics\\$StatisticsDataReferenceCleaner",
+
+    /**
+     * A global thread pool for broadcast exchange executions.
+     */
+    "broadcast-exchange.*",
+
+    /**
+     * A thread started by JRE to support safe parallel execution of waitFor() and exitStatus()
+     * methods to forked subprocesses.
+     */
+    "process reaper"
   )
   private var threadNamesSnapshot: Set[String] = Set.empty
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org