You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/07/02 21:55:41 UTC
[GitHub] [spark] hopper-signifyd opened a new pull request #28985: [SPARK-25262][K8S][2.4] Allow SPARK_LOCAL_DIRS to be tmpfs backed on K8S
hopper-signifyd opened a new pull request #28985:
URL: https://github.com/apache/spark/pull/28985
### What changes were proposed in this pull request?
This is a backport of changes from SPARK-25262, PR #22323
I plan to also backport SPARK-28042, PR #24879 after this is landed. That diff will also include a test confirming that related issue [SPARK-31666](https://issues.apache.org/jira/browse/SPARK-31666) is fixed.
### Why are the changes needed?
Running Spark on Kubernetes and not being able to use mounted volumes as local storage causes issues that prevent Spark jobs from starting. I've seen this on AWS EKS, but I've been able to reproduce it with a basic spark-submit command on a standard K8S cluster. Upgrading to 3.0 just to fix this bug is more hassle than it's worth for some organizations.
### Does this PR introduce _any_ user-facing change?
Technically, yes. This adds the spark.kubernetes.local.dirs.tmpfs back to Spark 2.4 from Spark 3. However, there's no "breaking changes" per se.
### How was this patch tested?
The tests were backported. Also, we've been running our own custom Spark 2.4.5 build with this patch applied at my org for the past few months.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28985: [SPARK-25262][K8S][2.4] Allow SPARK_LOCAL_DIRS to be tmpfs backed on K8S
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28985:
URL: https://github.com/apache/spark/pull/28985#issuecomment-653931804
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28985: [SPARK-25262][K8S][2.4] Allow SPARK_LOCAL_DIRS to be tmpfs backed on K8S
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28985:
URL: https://github.com/apache/spark/pull/28985#issuecomment-653931804
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28985: [SPARK-25262][K8S][2.4] Allow SPARK_LOCAL_DIRS to be tmpfs backed on K8S
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28985:
URL: https://github.com/apache/spark/pull/28985#issuecomment-653250384
I closed SPARK-31666 as `Not A Problem` because Apache Spark 2.4's SPARK-23529 is not designed for the duplicate volume name. It works as designed and your configuration is not supported by Kubernetes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #28985: [SPARK-25262][K8S][2.4] Allow SPARK_LOCAL_DIRS to be tmpfs backed on K8S
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #28985:
URL: https://github.com/apache/spark/pull/28985#issuecomment-653250384
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hopper-signifyd closed pull request #28985: [SPARK-25262][K8S][2.4] Allow SPARK_LOCAL_DIRS to be tmpfs backed on K8S
Posted by GitBox <gi...@apache.org>.
hopper-signifyd closed pull request #28985:
URL: https://github.com/apache/spark/pull/28985
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hopper-signifyd commented on pull request #28985: [SPARK-25262][K8S][2.4] Allow SPARK_LOCAL_DIRS to be tmpfs backed on K8S
Posted by GitBox <gi...@apache.org>.
hopper-signifyd commented on pull request #28985:
URL: https://github.com/apache/spark/pull/28985#issuecomment-653242054
As best as I can tell, both patches are required for fixing SPARK-31666. My plan was to add the additional unit test in the PR for backporting SPARK-28042. If I add the additional unit test to this PR, it will fail.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #28985: [SPARK-25262][K8S][2.4] Allow SPARK_LOCAL_DIRS to be tmpfs backed on K8S
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #28985:
URL: https://github.com/apache/spark/pull/28985#issuecomment-653250384
I closed SPARK-31666 as `Not A Problem` because Apache Spark 2.4's SPARK-23529 was not designed for the duplicate volume name. It works as designed and your configuration is not supported by Kubernetes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28985: [SPARK-25262][K8S][2.4] Allow SPARK_LOCAL_DIRS to be tmpfs backed on K8S
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28985:
URL: https://github.com/apache/spark/pull/28985#issuecomment-653932080
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org