You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2023/08/02 04:50:14 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request, #42289: [SPARK-44631][CONNECT][CORE] Remove session-based directory when the isolated session cache is evicted
HyukjinKwon opened a new pull request, #42289:
URL: https://github.com/apache/spark/pull/42289
### What changes were proposed in this pull request?
This PR proposes to remove session-based directories when the isolated session is evicted from the cache.
### Why are the changes needed?
SPARK-44078 added the cache for isolated sessions, and SPARK-44348 added the session-based directory for isolation.
When the isolated session cache is evicted, we should remove the session-based directory so it doesn't fail when the same session is used, see also https://github.com/apache/spark/pull/41625#discussion_r1251427466
### Does this PR introduce _any_ user-facing change?
No to end users since the feature has not been released yet.
### How was this patch tested?
I manually tested as described in https://github.com/apache/spark/pull/41292. Especially, I reduced the TTL to few minutes, and tested as below at the last step:
```python
spark.range(10).select(plug_one("id")).show()
spark.range(10).select(plug_one("id")).show()
# Wait few minutes
spark.range(10).select(plug_one("id")).show()
```
I verified that the same session can be added back to the cache, and creates the directory with the same name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #42289: [SPARK-44631][CONNECT][CORE] Remove session-based directory when the isolated session cache is evicted
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #42289:
URL: https://github.com/apache/spark/pull/42289#issuecomment-1661492010
cc @vicennial and @hvanhovell mind taking a look please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #42289: [SPARK-44631][CONNECT][CORE] Remove session-based directory when the isolated session cache is evicted
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #42289: [SPARK-44631][CONNECT][CORE] Remove session-based directory when the isolated session cache is evicted
URL: https://github.com/apache/spark/pull/42289
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #42289: [SPARK-44631][CONNECT][CORE] Remove session-based directory when the isolated session cache is evicted
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #42289:
URL: https://github.com/apache/spark/pull/42289#issuecomment-1661681292
Merged to master and branch-3.5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org