You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/08/20 12:16:11 UTC
(spark) branch branch-4.x updated: [MINOR][PYTHON][TEST] Retry PySpark logger UDF log assertion
This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new d5292ae68131 [MINOR][PYTHON][TEST] Retry PySpark logger UDF log assertion
d5292ae68131 is described below
commit d5292ae681315a96531c554035d140e9e1813822
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Thu Aug 20 20:15:27 2026 +0800
[MINOR][PYTHON][TEST] Retry PySpark logger UDF log assertion
### What changes were proposed in this pull request?
This PR retries the `python_worker_logs()` assertion in
`test_udf_with_pyspark_logger`.
The UDF result assertion still runs once. Only the follow-up read from the Python
worker log TVF is retried, using the existing `eventually` helper.
### Why are the changes needed?
The log assertion can be flaky when a Python worker log row is not visible at
the moment the test reads `python_worker_logs()`. Retrying the log-table
assertion keeps the expected rows unchanged while tolerating delayed log
visibility.
This was observed in https://github.com/apache/spark/pull/58124 at
https://github.com/zhengruifeng/spark/actions/runs/32337222716/job/96337527156.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Not run.
Checked locally:
```
git diff --check
awk 'length>100 && $0 !~ /^[[:space:]]*(import|package) / && $0 !~ /https?:\/\// {print FILENAME":"FNR": "length" chars"}' python/pyspark/sql/tests/test_udf.py
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #58162 from zhengruifeng/fix-pyspark-logger-flake.
Authored-by: Ruifeng Zheng <ru...@apache.org>
Signed-off-by: Ruifeng Zheng <ru...@apache.org>
(cherry picked from commit e0f5b469f6607c6ab647cf671ebd69ad6240e9a1)
Signed-off-by: Ruifeng Zheng <ru...@apache.org>
---
python/pyspark/sql/tests/test_udf.py | 33 ++++++++++++++++++---------------
1 file changed, 18 insertions(+), 15 deletions(-)
diff --git a/python/pyspark/sql/tests/test_udf.py b/python/pyspark/sql/tests/test_udf.py
index 5701fb6dcda9..eeb1b474cac0 100644
--- a/python/pyspark/sql/tests/test_udf.py
+++ b/python/pyspark/sql/tests/test_udf.py
@@ -54,7 +54,7 @@ from pyspark.testing.sqlutils import (
test_compiled,
test_not_compiled_message,
)
-from pyspark.testing.utils import assertDataFrameEqual, timeout
+from pyspark.testing.utils import assertDataFrameEqual, eventually, timeout
from pyspark.util import is_remote_only
@@ -1706,20 +1706,23 @@ class BaseUDFTestsMixin:
[Row(result=str(i)) for i in range(2)],
)
- logs = self.spark.tvf.python_worker_logs()
-
- assertDataFrameEqual(
- logs.select("level", "msg", "context", "logger"),
- [
- Row(
- level="WARNING",
- msg="PySparkLogger test",
- context={"func_name": my_udf.__name__, "x": str(i)},
- logger="PySparkLogger",
- )
- for i in range(2)
- ],
- )
+ @eventually(timeout=10, catch_assertions=True)
+ def check_logs():
+ logs = self.spark.tvf.python_worker_logs()
+ assertDataFrameEqual(
+ logs.select("level", "msg", "context", "logger"),
+ [
+ Row(
+ level="WARNING",
+ msg="PySparkLogger test",
+ context={"func_name": my_udf.__name__, "x": str(i)},
+ logger="PySparkLogger",
+ )
+ for i in range(2)
+ ],
+ )
+
+ check_logs()
class UDFTests(BaseUDFTestsMixin, ReusedSQLTestCase):
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org