You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "clee704 (via GitHub)" <gi...@apache.org> on 2023/10/05 03:52:14 UTC

[PR] [SPARK-45417][PYTHON] Make InheritableThread inherit the active session [spark]

clee704 opened a new pull request, #43224:
URL: https://github.com/apache/spark/pull/43224

   ### What changes were proposed in this pull request?
   Make `InheritableThread` and `inheritable_thread_target` inherit the active session.
   
   ### Why are the changes needed?
   Sometimes the active session can be null in Python threads, even with `InheritableThread` or `inheritable_thread_target`.
   
   Example:
   
   ```python
   # repro.py
   # run as: bin/spark-submit repro.py
   from multiprocessing.pool import ThreadPool
   from pyspark import inheritable_thread_target
   from pyspark.sql import SparkSession
   
   spark = SparkSession.builder.appName("Test").getOrCreate()
   spark.sparkContext.setLogLevel("ERROR")
   
   def f(i, spark):
       print(f"{i} spark = {spark}")
       print(f"{i} active session = {SparkSession.getActiveSession()}")
       print(f"{i} local property foo = {spark.sparkContext.getLocalProperty('foo')}")
       spark = SparkSession.builder.appName("Test").getOrCreate()
       print(f"{i} spark = {spark}")
       print(f"{i} active session = {SparkSession.getActiveSession()}")
   
   pool = ThreadPool(4)
   spark.sparkContext.setLocalProperty("foo", "bar")
   pool.starmap(inheritable_thread_target(f), [(i, spark) for i in range(4)])
   ```
   
   ### Does this PR introduce _any_ user-facing change?
   Yes. When using `InheritableThread` or `inheritable_thread_target`, the active session will be copied from the parent thread.
   
   ### How was this patch tested?
   `python/run-tests --testnames pyspark.tests.test_pin_thread`
   
   ### Was this patch authored or co-authored using generative AI tooling?
   No
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-45417][PYTHON] Make InheritableThread inherit the active session [spark]

Posted by "clee704 (via GitHub)" <gi...@apache.org>.
clee704 commented on PR #43224:
URL: https://github.com/apache/spark/pull/43224#issuecomment-1749350308

   Opened a PR against master here: https://github.com/apache/spark/pull/43231


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-45417][PYTHON] Make InheritableThread inherit the active session [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #43224:
URL: https://github.com/apache/spark/pull/43224#issuecomment-1747998509

   Thanks for taking a look. Mind opening a PR against `master` branch? so we can backport the change to `branch-3.5`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-45417][PYTHON] Make InheritableThread inherit the active session [spark]

Posted by "clee704 (via GitHub)" <gi...@apache.org>.
clee704 closed pull request #43224: [SPARK-45417][PYTHON] Make InheritableThread inherit the active session
URL: https://github.com/apache/spark/pull/43224


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org