You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/03/16 05:47:45 UTC

[GitHub] [spark] HyukjinKwon opened a new pull request #35871: [SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.4

HyukjinKwon opened a new pull request #35871:
URL: https://github.com/apache/spark/pull/35871


   ### What changes were proposed in this pull request?
   
   This PR upgrade Py4J 0.10.9.4, with relevant documentation changes.
   
   ### Why are the changes needed?
   
   Py4J 0.10.9.4 has a resource leak issue when pinned thread mode is enabled - it's enabled by default in PySpark at https://github.com/apache/spark/commit/41af409b7bcfe1b3960274c0b3085bcc1f9d1c98.
   We worked around this by enforcing users to use `InheritableThread` or `inhteritable_thread_target` as a workaround.
   After upgrading, we don't need to enforce users anymore because it automatically cleans up, see also https://github.com/py4j/py4j/pull/471
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes, users don't have to use `InheritableThread` or `inhteritable_thread_target` to avoid resource leaking problem anymore.
   
   ### How was this patch tested?
   
   CI in this PR should test it out.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #35871: [SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.4

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #35871:
URL: https://github.com/apache/spark/pull/35871#issuecomment-1068904745


   Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #35871: [SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.4

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #35871:
URL: https://github.com/apache/spark/pull/35871#issuecomment-1071504849


   Sorry but let me create reverting PRs for this.
   - https://github.com/apache/spark/pull/35903 (Apache Spark 3.4)
   - https://github.com/apache/spark/pull/35904 (Apache Spark 3.3)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #35871: [SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.4

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #35871:
URL: https://github.com/apache/spark/pull/35871#issuecomment-1071838091


   I made a fix, and fixing CI now (https://github.com/py4j/py4j/pull/477). I will make another release, and upgrade it back soon.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] wangyum commented on pull request #35871: [SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.4

Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #35871:
URL: https://github.com/apache/spark/pull/35871#issuecomment-1068760904


   Py4J 0.10.9.4 has a resource leak issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #35871: [SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.4

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #35871:
URL: https://github.com/apache/spark/pull/35871#issuecomment-1071504849


   Sorry but let me create reverting PRs for this.
   - https://github.com/apache/spark/pull/35903 (Apache Spark 3.4)
   - https://github.com/apache/spark/pull/35904 (Apache Spark 3.3)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #35871: [SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.4

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #35871:
URL: https://github.com/apache/spark/pull/35871#issuecomment-1071586400


   Yeah I'm making another fix and release now. We can revert it for now too. Thanks for checking this!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #35871: [SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.4

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35871:
URL: https://github.com/apache/spark/pull/35871#issuecomment-1071504849


   Sorry but let me create reverting PRs for this.
   - https://github.com/apache/spark/pull/35903


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon closed pull request #35871: [SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.4

Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #35871:
URL: https://github.com/apache/spark/pull/35871


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #35871: [SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.4

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #35871:
URL: https://github.com/apache/spark/pull/35871#issuecomment-1068767672


   Oops, I meant 0.10.9.3. Just fixed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #35871: [SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.4

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #35871:
URL: https://github.com/apache/spark/pull/35871#issuecomment-1071586400






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #35871: [SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.4

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35871:
URL: https://github.com/apache/spark/pull/35871#issuecomment-1071924726


   Thank you!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #35871: [SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.4

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35871:
URL: https://github.com/apache/spark/pull/35871#issuecomment-1071500516


   It seems that Py4J has no test coverage for Python 3.10.
   - https://www.py4j.org/download.html
   <img width="722" alt="Screen Shot 2022-03-17 at 2 08 31 PM" src="https://user-images.githubusercontent.com/9700541/158895548-555f831b-ce6a-4eda-9193-352d84284c6e.png">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #35871: [SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.4

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35871:
URL: https://github.com/apache/spark/pull/35871#issuecomment-1071508432


   For Apache Spark 3.2, I'm fine because we didn't support Python 3.10 before.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #35871: [SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.4

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35871:
URL: https://github.com/apache/spark/pull/35871#issuecomment-1071500516






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #35871: [SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.4

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #35871:
URL: https://github.com/apache/spark/pull/35871#issuecomment-1068756646


   cc @WeichenXu123 FYI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #35871: [SPARK-38563][PYTHON] Upgrade to Py4J 0.10.9.4

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #35871:
URL: https://github.com/apache/spark/pull/35871#issuecomment-1068907860


   Merged to master, branch-3.3 and branch-3.2.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org