You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/08/25 19:07:00 UTC

[PR] [SPARK-59007][PYTHON][TESTS] Fix flaky Arrow grouped map worker logging tests [spark]

Yicong-Huang opened a new pull request, #58289:
URL: https://github.com/apache/spark/pull/58289

   ### What changes were proposed in this pull request?
   
   This PR fixes flakiness in the two worker-logging tests of `ApplyInArrowTests`, `test_apply_in_arrow_with_logging` and `test_apply_in_arrow_iter_with_logging`, by wrapping the log read-and-assert step in `@eventually` so it polls until the asynchronously captured logs become visible. The `applyInArrow` call that produces the logs stays outside the poll so it runs exactly once.
   
   ### Why are the changes needed?
   
   `spark.tvf.python_worker_logs()` reads log records that Python workers emit on their stdout. On the JVM side those records are drained by a per-worker `RedirectThread` (`PythonWorkerLogCapture`), and the corresponding log block is only saved to the `BlockManager` once that thread reads the trailing marker line and closes the writer. This stdout drain is asynchronous with respect to the query result, which is returned over a separate socket channel. As a result, `python_worker_logs()` invoked immediately after the query completes can observe zero blocks, and the assertion fails with `[DIFFERENT_ROWS]` (actual empty vs. the expected WARNING rows). Polling the read side waits out this race without changing the feature, which captures worker logs on a best-effort basis. The producing UDF is intentionally kept outside the poll: re-running it would append more log rows and never match the expected count.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   Built with `build/sbt -Phive package` and ran the affected tests locally:
   
   ```
   python/run-tests --testnames 'pyspark.sql.tests.arrow.test_arrow_grouped_map'
   ```
   
   Both logging tests and the full `test_arrow_grouped_map` module pass.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   No.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59007][PYTHON][TESTS] Fix flaky Arrow grouped map worker logging tests [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #58289:
URL: https://github.com/apache/spark/pull/58289#issuecomment-5428916556

   Thank you @Yicong-Huang and @zhengruifeng!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59007][PYTHON][TESTS] Fix flaky Arrow grouped map worker logging tests [spark]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang closed pull request #58289: [SPARK-59007][PYTHON][TESTS] Fix flaky Arrow grouped map worker logging tests
URL: https://github.com/apache/spark/pull/58289


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59007][PYTHON][TESTS] Fix flaky Arrow grouped map worker logging tests [spark]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #58289:
URL: https://github.com/apache/spark/pull/58289#issuecomment-5446423427

   **Merge Summary:**
   - merged into master https://github.com/apache/spark/commit/02cb44fe44acaa12e02c939e0cee85e347e0a114
   - merged into branch-4.x https://github.com/apache/spark/commit/03473687b6e6bc37a0b2c64f9b32a667729e2133
   
   *Posted by `merge_spark_pr.py`*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org