You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "antban (via GitHub)" <gi...@apache.org> on 2025/02/11 13:24:11 UTC
[PR] Log exception on Python runner termination [spark]
antban opened a new pull request, #49890:
URL: https://github.com/apache/spark/pull/49890
### What changes were proposed in this pull request?
There are situations, when python worker may throw exceptions that are not covered by any other kind of logging (for example custom implementation/unexpected throw from existing code)
In these situations it is hard to debug which exactly issue happened, as there is no stack trace in logs and no actual message from exception.
To address this issue exception is added for the logging call (rare exception situation while stopping python worker)
### Why are the changes needed?
To effectively debug rare situations with worker termination.
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
no need
### Was this patch authored or co-authored using generative AI tooling?
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51608] Log exception on Python runner termination [spark]
Posted by "antban (via GitHub)" <gi...@apache.org>.
antban commented on PR #49890:
URL: https://github.com/apache/spark/pull/49890#issuecomment-2754049685
@allisonwang-db done, can you please take a look if I satisfied all the requirements? https://issues.apache.org/jira/browse/SPARK-51608
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Log exception on Python runner termination [spark]
Posted by "antban (via GitHub)" <gi...@apache.org>.
antban commented on PR #49890:
URL: https://github.com/apache/spark/pull/49890#issuecomment-2750792182
@allisonwang-db Hi, sorry, lost track of it. Will do asap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Log exception on Python runner termination [spark]
Posted by "allisonwang-db (via GitHub)" <gi...@apache.org>.
allisonwang-db commented on PR #49890:
URL: https://github.com/apache/spark/pull/49890#issuecomment-2702249616
@antban Thanks for contributing. Could you create a JIRA ticket and add it to the PR title?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51608][PYTHON] Log exception on Python runner termination [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin closed pull request #49890: [SPARK-51608][PYTHON] Log exception on Python runner termination
URL: https://github.com/apache/spark/pull/49890
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51608][PYTHON] Log exception on Python runner termination [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #49890:
URL: https://github.com/apache/spark/pull/49890#issuecomment-2759931645
@antban Could you enable GHA following [the instruction](
https://github.com/apache/spark/pull/49890/checks?check_run_id=37026676740) and retrigger the tests, just in case? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51608][PYTHON] Log exception on Python runner termination [spark]
Posted by "antban (via GitHub)" <gi...@apache.org>.
antban commented on code in PR #49890:
URL: https://github.com/apache/spark/pull/49890#discussion_r2018219051
##########
core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:
##########
@@ -274,7 +274,7 @@ private[spark] abstract class BasePythonRunner[IN, OUT](
worker.stop()
} catch {
case e: Exception =>
- logWarning("Failed to stop worker")
+ logWarning("Failed to stop worker", e)
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51608][PYTHON] Log exception on Python runner termination [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #49890:
URL: https://github.com/apache/spark/pull/49890#discussion_r2017773133
##########
core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:
##########
@@ -274,7 +274,7 @@ private[spark] abstract class BasePythonRunner[IN, OUT](
worker.stop()
} catch {
case e: Exception =>
- logWarning("Failed to stop worker")
+ logWarning("Failed to stop worker", e)
Review Comment:
Shall we use `log" ... "`, even though there is no `MDC` embedded?
```suggestion
logWarning(log"Failed to stop worker", e)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51608][PYTHON] Log exception on Python runner termination [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #49890:
URL: https://github.com/apache/spark/pull/49890#issuecomment-2762351169
Thanks! merging to master/4.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org