You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/16 21:12:13 UTC
[GitHub] [spark] shanyu opened a new pull request #27598: [SPARK-30845] Do
not upload local pyspark archives for spark-submit on Yarn
shanyu opened a new pull request #27598: [SPARK-30845] Do not upload local pyspark archives for spark-submit on Yarn
URL: https://github.com/apache/spark/pull/27598
### What changes were proposed in this pull request?
Use spark-submit to submit a pyspark app on Yarn, and set this in spark-env.sh:
export PYSPARK_ARCHIVES_PATH=local:/opt/spark/python/lib/pyspark.zip,local:/opt/spark/python/lib/py4j-0.10.7-src.zip
You can see that these local archives are still uploaded to Yarn distributed cache:
yarn.Client: Uploading resource file:/opt/spark/python/lib/pyspark.zip -> hdfs://myhdfs/user/test1/.sparkStaging/application_1581024490249_0001/pyspark.zip
This PR fix this issue by checking the files specified in PYSPARK_ARCHIVES_PATH, if they are local archives, don't distribute to Yarn dist cache.
### Why are the changes needed?
For pyspark appp to support local pyspark archives set in PYSPARK_ARCHIVES_PATH.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Existing tests and manual tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27598: [SPARK-30845] Do
not upload local pyspark archives for spark-submit on Yarn
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27598: [SPARK-30845] Do not upload local pyspark archives for spark-submit on Yarn
URL: https://github.com/apache/spark/pull/27598#issuecomment-586754912
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27598: [SPARK-30845] Do
not upload local pyspark archives for spark-submit on Yarn
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27598: [SPARK-30845] Do not upload local pyspark archives for spark-submit on Yarn
URL: https://github.com/apache/spark/pull/27598#issuecomment-586755149
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27598:
[SPARK-30845] Do not upload local pyspark archives for spark-submit on Yarn
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27598: [SPARK-30845] Do not upload local pyspark archives for spark-submit on Yarn
URL: https://github.com/apache/spark/pull/27598#issuecomment-586754912
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org