You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/06/18 21:04:44 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request #32962: [SPARK-35303][SPARK-35498][PYTHON][FOLLOW-UP] Refactor inheritable thread logic, and use it in codebase for pinned thread mode
HyukjinKwon opened a new pull request #32962:
URL: https://github.com/apache/spark/pull/32962
### What changes were proposed in this pull request?
This PR is a followup of https://github.com/apache/spark/pull/32429 and https://github.com/apache/spark/pull/32644.
I was thinking about creating separate PRs but decided to include all in this PR because it shares the same context, and should be easier to review together.
This PR includes:
- Use `InheritableThread` and `inheritable_thread_target` in the current code base to prevent potential resource leak (since we enabled pinned thread mode by default now at https://github.com/apache/spark/pull/32429)
- Enable pinned thread mode by default. The initial PR missed some please at `inheritable_thread_target`.
- Change `InheritableThread`:
- Copy local properties when `InheritableThread.start` is called to mimic JVM behaviour. Previously it was copied when `InheritableThread` instance was created.
- Clean Py4J connections on `__del__` when garbage-collecting instead of each target function invocation. This is a regression from https://github.com/apache/spark/pull/32644.
### Why are the changes needed?
To mimic the JVM behaviour about thread lifecycle
### Does this PR introduce _any_ user-facing change?
Ideally no. One possible case is that users use `InheritableThread` with pinned thread mode enabled.
In this case, the local properties will be copied when starting the thread instead of defining the `InheritableThread` object.
This is a small difference that wouldn't likely affect end users.
It doesn't affect about cleaning logic because Spark 3.1 already cleans up in `__del__`. It was mistakenly fixed in https://github.com/apache/spark/pull/32644.
### How was this patch tested?
Existing tests should cover this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32962: [SPARK-35303][SPARK-35498][PYTHON][FOLLOW-UP] Refactor inheritable thread logic, and use it in codebase for pinned thread mode
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32962:
URL: https://github.com/apache/spark/pull/32962#issuecomment-863851453
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32962: [SPARK-35303][SPARK-35498][PYTHON][FOLLOW-UP] Refactor inheritable thread logic, and use it in codebase for pinned thread mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32962:
URL: https://github.com/apache/spark/pull/32962#issuecomment-863851453
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32962: [SPARK-35303][SPARK-35498][PYTHON][FOLLOW-UP] Copy local properties when starting the thread, and use inheritable thread in the current codebase
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32962:
URL: https://github.com/apache/spark/pull/32962#discussion_r654728144
##########
File path: python/pyspark/util.py
##########
@@ -305,35 +307,22 @@ def inheritable_thread_target(f):
>>> Thread(target=inheritable_thread_target(target_func)).start() # doctest: +SKIP
"""
from pyspark import SparkContext
- if os.environ.get("PYSPARK_PIN_THREAD", "false").lower() == "true":
- # Here's when the pinned-thread mode (PYSPARK_PIN_THREAD) is on.
- sc = SparkContext._active_spark_context
- # Get local properties from main thread
- properties = sc._jsc.sc().getLocalProperties().clone()
+ if os.environ.get("PYSPARK_PIN_THREAD", "true").lower() == "true":
Review comment:
Oh yeah, it was a mistake from #32429, and actually I meant it by:
> #32429 missed one place at inheritable_thread_target (https://github.com/apache/spark/blob/master/python/pyspark/util.py#L308).
Let me me more explicit on that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32962: [SPARK-35303][SPARK-35498][PYTHON][FOLLOW-UP] Refactor inheritable thread logic, and use it in codebase for pinned thread mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32962:
URL: https://github.com/apache/spark/pull/32962#issuecomment-863887638
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32962: [SPARK-35303][SPARK-35498][PYTHON][FOLLOW-UP] Copy local properties when starting the thread, and use inheritable thread in the current codebase
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32962:
URL: https://github.com/apache/spark/pull/32962#issuecomment-864491492
Thanks, @dongjoon-hyun.
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32962: [SPARK-35303][SPARK-35498][PYTHON][FOLLOW-UP] Copy local properties when starting the thread, and use inheritable thread in the current codebase
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32962:
URL: https://github.com/apache/spark/pull/32962#discussion_r654728013
##########
File path: python/pyspark/util.py
##########
@@ -305,35 +307,22 @@ def inheritable_thread_target(f):
>>> Thread(target=inheritable_thread_target(target_func)).start() # doctest: +SKIP
"""
from pyspark import SparkContext
- if os.environ.get("PYSPARK_PIN_THREAD", "false").lower() == "true":
- # Here's when the pinned-thread mode (PYSPARK_PIN_THREAD) is on.
- sc = SparkContext._active_spark_context
- # Get local properties from main thread
- properties = sc._jsc.sc().getLocalProperties().clone()
+ if os.environ.get("PYSPARK_PIN_THREAD", "true").lower() == "true":
Review comment:
This seems to be the change of default value when `PYSPARK_PIN_THREAD` is not set. Do we need to mention this in the PR description?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32962: [SPARK-35303][SPARK-35498][PYTHON][FOLLOW-UP] Refactor inheritable thread logic, and use it in codebase for pinned thread mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32962:
URL: https://github.com/apache/spark/pull/32962#issuecomment-863887638
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #32962: [SPARK-35303][SPARK-35498][PYTHON][FOLLOW-UP] Copy local properties when starting the thread, and use inheritable thread in the current codebase
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #32962:
URL: https://github.com/apache/spark/pull/32962
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org