[spark] branch master updated: [SPARK-26960][ML] Wait for listener bus to clear in MLEventsSuite to reduce test flakiness

[gu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new be1cadf  [SPARK-26960][ML] Wait for listener bus to clear in MLEventsSuite to reduce test flakiness
be1cadf is described below

commit be1cadf16dc70e22eae144b3dfce9e269ef95acc
Author: Joseph K. Bradley <jo...@databricks.com>
AuthorDate: Fri Feb 22 10:08:16 2019 +0800

    [SPARK-26960][ML] Wait for listener bus to clear in MLEventsSuite to reduce test flakiness
    
    ## What changes were proposed in this pull request?
    
    This patch aims to address flakiness I've observed in MLEventsSuite in these tests:
    *  test("pipeline read/write events")
    *  test("pipeline model read/write events")
    
    The issue is in the "read/write events" tests, which work as follows:
    * write
    * wait until we see at least 1 write-related SparkListenerEvent
    * read
    * wait until we see at least 1 read-related SparkListenerEvent
    
    The problem is that the last step does NOT allow any write-related SparkListenerEvents, but some of those events may be delayed enough that they are seen in this last step. We should ideally add logic before "read" to wait until the listener events are cleared/complete. Looking into other SparkListener tests, we need to use `sc.listenerBus.waitUntilEmpty(TIMEOUT)`.
    
    This patch adds the waitUntilEmpty() call.
    
    ## How was this patch tested?
    
    It's a test!
    
    Closes #23863 from jkbradley/SPARK-26960.
    
    Authored-by: Joseph K. Bradley <jo...@databricks.com>
    Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
 mllib/src/test/scala/org/apache/spark/ml/MLEventsSuite.scala | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/mllib/src/test/scala/org/apache/spark/ml/MLEventsSuite.scala b/mllib/src/test/scala/org/apache/spark/ml/MLEventsSuite.scala
index 80ae0c7..4fe69b6 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/MLEventsSuite.scala
+++ b/mllib/src/test/scala/org/apache/spark/ml/MLEventsSuite.scala
@@ -239,6 +239,7 @@ class MLEventsSuite
       events.map(JsonProtocol.sparkEventToJson).foreach { event =>
         assert(JsonProtocol.sparkEventFromJson(event).isInstanceOf[MLEvent])
       }
+      sc.listenerBus.waitUntilEmpty(timeoutMillis = 10000)
 
       events.clear()
       val pipelineReader = Pipeline.read
@@ -297,6 +298,7 @@ class MLEventsSuite
       events.map(JsonProtocol.sparkEventToJson).foreach { event =>
         assert(JsonProtocol.sparkEventFromJson(event).isInstanceOf[MLEvent])
       }
+      sc.listenerBus.waitUntilEmpty(timeoutMillis = 10000)
 
       events.clear()
       val pipelineModelReader = PipelineModel.read


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org