You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by at...@apache.org on 2025/07/10 18:16:58 UTC
(spark) branch master updated: [SPARK-52731][SS][PYTHON][TESTS] Fix flaky test_listener_events in 'pyspark.sql.tests.streaming.test_streaming_listener'
This is an automated email from the ASF dual-hosted git repository.
attilapiros pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 0aace88342e6 [SPARK-52731][SS][PYTHON][TESTS] Fix flaky test_listener_events in 'pyspark.sql.tests.streaming.test_streaming_listener'
0aace88342e6 is described below
commit 0aace88342e63f18cb79ae889fc010520ec2f90f
Author: Rishi <sr...@gmail.com>
AuthorDate: Thu Jul 10 11:16:46 2025 -0700
[SPARK-52731][SS][PYTHON][TESTS] Fix flaky test_listener_events in 'pyspark.sql.tests.streaming.test_streaming_listener'
### What changes were proposed in this pull request?
Fix a flaky test_listener_events in `pyspark.sql.tests.streaming.test_streaming_listener`
### Why are the changes needed?
It may cause python test failures even when nothing in your code is wrong
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manually. I reproduced the flaky failure more consistently by adding a `time.sleep` for a few seconds in `onQueryTerminated` method in `TestListenerV1 `class in `test_listener_events`. With this fix, the flakyness went away.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #51423 from sririshindra/master-SPARK-52731.
Authored-by: Rishi <sr...@gmail.com>
Signed-off-by: attilapiros <pi...@gmail.com>
---
python/pyspark/sql/tests/streaming/test_streaming_listener.py | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/python/pyspark/sql/tests/streaming/test_streaming_listener.py b/python/pyspark/sql/tests/streaming/test_streaming_listener.py
index 934b1463a15c..bb485dae6bdc 100644
--- a/python/pyspark/sql/tests/streaming/test_streaming_listener.py
+++ b/python/pyspark/sql/tests/streaming/test_streaming_listener.py
@@ -400,9 +400,17 @@ class StreamingListenerTests(StreamingListenerTestsMixin, ReusedSQLTestCase):
# Check query terminated with exception
from pyspark.sql.functions import col, udf
+ start_event = None
+ progress_event = None
+ terminated_event = None
bad_udf = udf(lambda x: 1 / 0)
q = df.select(bad_udf(col("value"))).writeStream.format("noop").start()
- time.sleep(5)
+ wait_count = 0
+ while terminated_event is None:
+ time.sleep(0.5)
+ wait_count = wait_count + 1
+ if wait_count > 100:
+ self.fail("Not getting terminated event after 50 seconds")
q.stop()
self.spark.sparkContext._jsc.sc().listenerBus().waitUntilEmpty()
self.check_terminated_event(terminated_event, "ZeroDivisionError")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org