You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2019/07/17 09:49:18 UTC
[spark] branch branch-2.3 updated: [SPARK-28418][PYTHON][SQL] Wait
for event process in 'test_query_execution_listener_on_collect'
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-2.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.3 by this push:
new 10a8ee9 [SPARK-28418][PYTHON][SQL] Wait for event process in 'test_query_execution_listener_on_collect'
10a8ee9 is described below
commit 10a8ee9846e878897d067b796bf79522c4662de3
Author: HyukjinKwon <gu...@apache.org>
AuthorDate: Wed Jul 17 18:44:11 2019 +0900
[SPARK-28418][PYTHON][SQL] Wait for event process in 'test_query_execution_listener_on_collect'
It fixes a flaky test:
```
ERROR [0.164s]: test_query_execution_listener_on_collect (pyspark.sql.tests.test_dataframe.QueryExecutionListenerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jenkins/python/pyspark/sql/tests/test_dataframe.py", line 758, in test_query_execution_listener_on_collect
"The callback from the query execution listener should be called after 'collect'")
AssertionError: The callback from the query execution listener should be called after 'collect'
```
Seems it can be failed because the event was somehow delayed but checked first.
Manually.
Closes #25177 from HyukjinKwon/SPARK-28418.
Authored-by: HyukjinKwon <gu...@apache.org>
Signed-off-by: HyukjinKwon <gu...@apache.org>
(cherry picked from commit 66179fa8426324e11819e04af4cf3eabf9f2627f)
Signed-off-by: HyukjinKwon <gu...@apache.org>
---
python/pyspark/sql/tests.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/python/pyspark/sql/tests.py b/python/pyspark/sql/tests.py
index dc5ed19..3696fca 100644
--- a/python/pyspark/sql/tests.py
+++ b/python/pyspark/sql/tests.py
@@ -3073,6 +3073,7 @@ class QueryExecutionListenerTests(unittest.TestCase, SQLTestUtils):
self.spark._jvm.OnSuccessCall.isCalled(),
"The callback from the query execution listener should not be called before 'collect'")
self.spark.sql("SELECT * FROM range(1)").collect()
+ self.spark.sparkContext._jsc.sc().listenerBus().waitUntilEmpty(10000)
self.assertTrue(
self.spark._jvm.OnSuccessCall.isCalled(),
"The callback from the query execution listener should be called after 'collect'")
@@ -3087,6 +3088,7 @@ class QueryExecutionListenerTests(unittest.TestCase, SQLTestUtils):
"The callback from the query execution listener should not be "
"called before 'toPandas'")
self.spark.sql("SELECT * FROM range(1)").toPandas()
+ self.spark.sparkContext._jsc.sc().listenerBus().waitUntilEmpty(10000)
self.assertTrue(
self.spark._jvm.OnSuccessCall.isCalled(),
"The callback from the query execution listener should be called after 'toPandas'")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org