[GitHub] jkbradley opened a new pull request #23863: [SPARK-26960][ML] Wait for listener bus to clear in MLEventsSuite to reduce test flakiness

[GitBox <gi...@apache.org>]

jkbradley opened a new pull request #23863: [SPARK-26960][ML] Wait for listener bus to clear in MLEventsSuite to reduce test flakiness
URL: https://github.com/apache/spark/pull/23863
 
 
   ## What changes were proposed in this pull request?
   
   This patch aims to address flakiness I've observed in MLEventsSuite in these tests:
   *  test("pipeline read/write events")
   *  test("pipeline model read/write events")
   
   The issue is in the "read/write events" tests, which work as follows:
   * write
   * wait until we see at least 1 write-related SparkListenerEvent
   * read
   * wait until we see at least 1 read-related SparkListenerEvent
   
   The problem is that the last step does NOT allow any write-related SparkListenerEvents, but some of those events may be delayed enough that they are seen in this last step. We should ideally add logic before "read" to wait until the listener events are cleared/complete. Looking into other SparkListener tests, we need to use `sc.listenerBus.waitUntilEmpty(TIMEOUT)`.
   
   This patch adds the waitUntilEmpty() call.
   
   ## How was this patch tested?
   
   It's a test!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org