You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "wbo4958 (via GitHub)" <gi...@apache.org> on 2026/02/03 05:01:16 UTC
[PR] [SPARK-55326][PYTHON][Connect] Release remote session when SPARK_CONNECT_RELEASE_SESSION_ON_EXIT is set [spark]
wbo4958 opened a new pull request, #54106:
URL: https://github.com/apache/spark/pull/54106
### What changes were proposed in this pull request?
This PR adds an _on_exit handler to SparkConnectClient that is registered with Python's atexit module. When enabled via the SPARK_CONNECT_RELEASE_SESSION_ON_EXIT environment variable, the client will automatically
### Why are the changes needed?
Currently, when a PySpark Connect client process exits without explicitly calling `spark.stop()`, the session may remain active on the server side, consuming resources unnecessarily. This change provides an opt-in mechanism to automatically release the session during process exit
### Does this PR introduce _any_ user-facing change?
Yes. Users can now set the environment variable `SPARK_CONNECT_RELEASE_SESSION_ON_EXIT=true` to enable automatic session release when the Python process exits.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
Yes, co-authored with claude-4.5-opus-high
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55326][PYTHON][CONNECT] Release remote session when SPARK_CONNECT_RELEASE_SESSION_ON_EXIT is set [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on PR #54106:
URL: https://github.com/apache/spark/pull/54106#issuecomment-3987854634
Hi @HyukjinKwon, Could you be able to cherry-pick this PR to branch-4.0 and branch-4.1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55326][PYTHON][Connect] Release remote session when SPARK_CONNECT_RELEASE_SESSION_ON_EXIT is set [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #54106:
URL: https://github.com/apache/spark/pull/54106#issuecomment-3839068096
## JIRA Issue Information
=== Improvement SPARK-55326 ===
Summary: Add an optional connect session release in pyspark
Assignee: None
Status: Open
Affected: ["4.2"]
---
*This comment was automatically generated by GitHub Actions*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55326][PYTHON][CONNECT] Release remote session when SPARK_CONNECT_RELEASE_SESSION_ON_EXIT is set [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #54106:
URL: https://github.com/apache/spark/pull/54106#discussion_r2801674667
##########
python/pyspark/sql/connect/client/core.py:
##########
@@ -755,8 +755,11 @@ def __init__(
self._release_futures: weakref.WeakSet[concurrent.futures.Future] = weakref.WeakSet()
- # cleanup ml cache if possible
- atexit.register(self._cleanup_ml_cache)
+ self._release_session_on_exit = os.getenv(
+ "SPARK_CONNECT_RELEASE_SESSION_ON_EXIT", "false"
Review Comment:
I am fine with this but I wonder if we should at least document this `SPARK_CONNECT_RELEASE_SESSION_ON_EXIT` somewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55326][PYTHON][CONNECT] Release remote session when SPARK_CONNECT_RELEASE_SESSION_ON_EXIT is set [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on PR #54106:
URL: https://github.com/apache/spark/pull/54106#issuecomment-3893822648
Hi @HyukjinKwon, @hvanhovell, @tgravescs, Could you help review it. Thx very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55326][PYTHON][CONNECT] Release remote session when SPARK_CONNECT_RELEASE_SESSION_ON_EXIT is set [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #54106: [SPARK-55326][PYTHON][CONNECT] Release remote session when SPARK_CONNECT_RELEASE_SESSION_ON_EXIT is set
URL: https://github.com/apache/spark/pull/54106
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55326][PYTHON][CONNECT] Release remote session when SPARK_CONNECT_RELEASE_SESSION_ON_EXIT is set [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #54106:
URL: https://github.com/apache/spark/pull/54106#issuecomment-3987912554
Nah, let's don't backport. It's not really a bug fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55326][PYTHON][Connect] Release remote session when SPARK_CONNECT_RELEASE_SESSION_ON_EXIT is set [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on PR #54106:
URL: https://github.com/apache/spark/pull/54106#issuecomment-3839190286
Hi @HyukjinKwon, @hvanhovell, @tgravescs, Could you help review it. Thx very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55326][PYTHON][CONNECT] Release remote session when SPARK_CONNECT_RELEASE_SESSION_ON_EXIT is set [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #54106:
URL: https://github.com/apache/spark/pull/54106#discussion_r2825020834
##########
python/pyspark/sql/connect/client/core.py:
##########
@@ -755,8 +755,11 @@ def __init__(
self._release_futures: weakref.WeakSet[concurrent.futures.Future] = weakref.WeakSet()
- # cleanup ml cache if possible
- atexit.register(self._cleanup_ml_cache)
+ self._release_session_on_exit = os.getenv(
+ "SPARK_CONNECT_RELEASE_SESSION_ON_EXIT", "false"
Review Comment:
ah ok. I see there are already some instances undocumented. Let's probably document them in a separate PR .. I am fine with this PR as is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55326][PYTHON][CONNECT] Release remote session when SPARK_CONNECT_RELEASE_SESSION_ON_EXIT is set [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #54106:
URL: https://github.com/apache/spark/pull/54106#issuecomment-3987827982
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55326][PYTHON][CONNECT] Release remote session when SPARK_CONNECT_RELEASE_SESSION_ON_EXIT is set [spark]
Posted by "tgravescs (via GitHub)" <gi...@apache.org>.
tgravescs commented on code in PR #54106:
URL: https://github.com/apache/spark/pull/54106#discussion_r2817773454
##########
python/pyspark/sql/connect/client/core.py:
##########
@@ -755,8 +755,11 @@ def __init__(
self._release_futures: weakref.WeakSet[concurrent.futures.Future] = weakref.WeakSet()
- # cleanup ml cache if possible
- atexit.register(self._cleanup_ml_cache)
+ self._release_session_on_exit = os.getenv(
+ "SPARK_CONNECT_RELEASE_SESSION_ON_EXIT", "false"
Review Comment:
I agree, that would be nice. the documentation seems a bit scattered right now, do you have recommendation on where to put this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55326][PYTHON][CONNECT] Release remote session when SPARK_CONNECT_RELEASE_SESSION_ON_EXIT is set [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on code in PR #54106:
URL: https://github.com/apache/spark/pull/54106#discussion_r2870713975
##########
python/pyspark/sql/connect/client/core.py:
##########
@@ -755,8 +755,11 @@ def __init__(
self._release_futures: weakref.WeakSet[concurrent.futures.Future] = weakref.WeakSet()
- # cleanup ml cache if possible
- atexit.register(self._cleanup_ml_cache)
+ self._release_session_on_exit = os.getenv(
+ "SPARK_CONNECT_RELEASE_SESSION_ON_EXIT", "false"
Review Comment:
Yeah, Sounds a separate PR is suitable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55326][PYTHON][CONNECT] Release remote session when SPARK_CONNECT_RELEASE_SESSION_ON_EXIT is set [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on PR #54106:
URL: https://github.com/apache/spark/pull/54106#issuecomment-3982384238
Sorry for the late response due to a long holiday.
Hi @HyukjinKwon @tgravescs, Could you help merge this PR. Thx very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org