You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ueshin (via GitHub)" <gi...@apache.org> on 2025/02/07 02:21:08 UTC
[PR] [SPARK-51127][PYTHON] Kill the Python worker on idle timeout [spark]
ueshin opened a new pull request, #49843:
URL: https://github.com/apache/spark/pull/49843
### What changes were proposed in this pull request?
Kills the Python worker on idle timeout introduced at #49818.
- Spark conf: `spark.python.worker.killOnIdleTimeout` (default `false`)
Whether Spark should terminate the Python worker process when the idle timeout (as defined by spark.python.worker.idleTimeoutSeconds) is reached. If enabled, Spark will terminate the Python worker process in addition to logging the status.
- SQL conf: `spark.sql.execution.pyspark.udf.killOnIdleTimeout`
The same as `spark.python.worker.killOnIdleTimeout`, but this is a runtime conf for Python UDFs. Falls back to `spark.python.worker.killOnIdleTimeout`.
### Why are the changes needed?
When the Python worker gets stuck, currently there is no way to resume the execution.
### Does this PR introduce _any_ user-facing change?
When `spark.python.worker.idleTimeoutSeconds` is set to a positive number, and set `spark.python.worker.killOnIdleTimeout` to `true`, once it detects a Python worker as stuck, the executor will try to kill the process, which will cause a task failure but Spark will retry the task.
### How was this patch tested?
Added the related tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51127][PYTHON] Kill the Python worker on idle timeout [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #49843:
URL: https://github.com/apache/spark/pull/49843#issuecomment-2652279974
Thank you! Feel free to choose what is required for you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51127][PYTHON] Kill the Python worker on idle timeout [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #49843:
URL: https://github.com/apache/spark/pull/49843#issuecomment-2644408761
Thanks! merging to master/4.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51127][PYTHON] Kill the Python worker on idle timeout [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #49843:
URL: https://github.com/apache/spark/pull/49843#issuecomment-2644408621
The remaining test failures are not related to this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51127][PYTHON] Kill the Python worker on idle timeout [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #49843:
URL: https://github.com/apache/spark/pull/49843#issuecomment-2652281482
This should be a new feature. I'll just revert it. Thanks for pointing out!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51127][PYTHON] Kill the Python worker on idle timeout [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #49843:
URL: https://github.com/apache/spark/pull/49843#issuecomment-2652277357
@dongjoon-hyun Oops, my bad. I thought we still had time.
Let me revert it from 4.0, and submit a follow-up PR to update the `since` for the config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51127][PYTHON] Kill the Python worker on idle timeout [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #49843:
URL: https://github.com/apache/spark/pull/49843#issuecomment-2652292310
Also the follow-up https://github.com/apache/spark/pull/49895.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51127][PYTHON] Kill the Python worker on idle timeout [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #49843:
URL: https://github.com/apache/spark/pull/49843#issuecomment-2652278745
If you think this could be a bug fix, you can update JIRA issue type too, @ueshin .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51127][PYTHON] Kill the Python worker on idle timeout [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #49843:
URL: https://github.com/apache/spark/pull/49843#issuecomment-2652283316
Got it. Thank you so much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51127][PYTHON] Kill the Python worker on idle timeout [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #49843:
URL: https://github.com/apache/spark/pull/49843#issuecomment-2652164381
Hi, @ueshin . If you don't mind, may I ask why this is backported to `branch-4.0`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51127][PYTHON] Kill the Python worker on idle timeout [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on PR #49843:
URL: https://github.com/apache/spark/pull/49843#issuecomment-2652285455
Reverted https://github.com/apache/spark/commit/a94faa738fcb602df6d641a79bf2a168224a52f5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51127][PYTHON] Kill the Python worker on idle timeout [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin closed pull request #49843: [SPARK-51127][PYTHON] Kill the Python worker on idle timeout
URL: https://github.com/apache/spark/pull/49843
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org