You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/25 18:37:15 UTC

[GitHub] [spark] dongjoon-hyun opened a new pull request #29873: [SPARK-32997][K8S] Support dynamic PVC creation and deletion in K8s driver

dongjoon-hyun opened a new pull request #29873:
URL: https://github.com/apache/spark/pull/29873


   ### What changes were proposed in this pull request?
   
   This PR aims to support dynamic PVC creation and deletion in K8s driver.
   
   ### Why are the changes needed?
   
   Like executors, driver also needs larger PVC.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. This is a new feature.
   
   ### How was this patch tested?
   
   Pass the newly added test case.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #29873: [SPARK-32997][K8S] Support dynamic PVC creation and deletion in K8s driver

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29873:
URL: https://github.com/apache/spark/pull/29873#issuecomment-699171718


   Could you review this, please, @dbtsai and @viirya ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun closed pull request #29873: [SPARK-32997][K8S] Support dynamic PVC creation and deletion in K8s driver

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #29873:
URL: https://github.com/apache/spark/pull/29873


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29873: [SPARK-32997][K8S] Support dynamic PVC creation and deletion in K8s driver

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29873:
URL: https://github.com/apache/spark/pull/29873#issuecomment-699120696






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #29873: [SPARK-32997][K8S] Support dynamic PVC creation and deletion in K8s driver

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29873:
URL: https://github.com/apache/spark/pull/29873#issuecomment-699217663


   Thank you, @viirya . Merged to master.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29873: [SPARK-32997][K8S] Support dynamic PVC creation and deletion in K8s driver

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29873:
URL: https://github.com/apache/spark/pull/29873#issuecomment-699095830






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29873: [SPARK-32997][K8S] Support dynamic PVC creation and deletion in K8s driver

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29873:
URL: https://github.com/apache/spark/pull/29873#discussion_r495337096



##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/MountVolumesFeatureStep.scala
##########
@@ -66,32 +66,30 @@ private[spark] class MountVolumesFeatureStep(conf: KubernetesConf)
         case KubernetesPVCVolumeConf(claimNameTemplate, storageClass, size) =>
           val claimName = conf match {
             case c: KubernetesExecutorConf =>
-              val claimName = claimNameTemplate
+              claimNameTemplate
                 .replaceAll(PVC_ON_DEMAND,
                   s"${conf.resourceNamePrefix}-exec-${c.executorId}$PVC_POSTFIX-$i")
                 .replaceAll(ENV_EXECUTOR_ID, c.executorId)
-
-              if (storageClass.isDefined && size.isDefined) {
-                additionalResources.append(new PersistentVolumeClaimBuilder()
-                  .withKind(PVC)
-                  .withApiVersion("v1")
-                  .withNewMetadata()
-                    .withName(claimName)
-                    .endMetadata()
-                  .withNewSpec()
-                    .withStorageClassName(storageClass.get)
-                    .withAccessModes(PVC_ACCESS_MODE)
-                    .withResources(new ResourceRequirementsBuilder()
-                      .withRequests(Map("storage" -> new Quantity(size.get)).asJava).build())
-                    .endSpec()
-                  .build())
-              }
-
-              claimName
-
             case _ =>

Review comment:
       Late to review. Should we match `KubernetesDriverConf` and add anther default case?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29873: [SPARK-32997][K8S] Support dynamic PVC creation and deletion in K8s driver

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29873:
URL: https://github.com/apache/spark/pull/29873#issuecomment-699090277


   **[Test build #129121 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129121/testReport)** for PR 29873 at commit [`90c552c`](https://github.com/apache/spark/commit/90c552ce4da20f49f6bdc536c5557c3fbf2e98ea).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29873: [SPARK-32997][K8S] Support dynamic PVC creation and deletion in K8s driver

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29873:
URL: https://github.com/apache/spark/pull/29873#discussion_r495337096



##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/MountVolumesFeatureStep.scala
##########
@@ -66,32 +66,30 @@ private[spark] class MountVolumesFeatureStep(conf: KubernetesConf)
         case KubernetesPVCVolumeConf(claimNameTemplate, storageClass, size) =>
           val claimName = conf match {
             case c: KubernetesExecutorConf =>
-              val claimName = claimNameTemplate
+              claimNameTemplate
                 .replaceAll(PVC_ON_DEMAND,
                   s"${conf.resourceNamePrefix}-exec-${c.executorId}$PVC_POSTFIX-$i")
                 .replaceAll(ENV_EXECUTOR_ID, c.executorId)
-
-              if (storageClass.isDefined && size.isDefined) {
-                additionalResources.append(new PersistentVolumeClaimBuilder()
-                  .withKind(PVC)
-                  .withApiVersion("v1")
-                  .withNewMetadata()
-                    .withName(claimName)
-                    .endMetadata()
-                  .withNewSpec()
-                    .withStorageClassName(storageClass.get)
-                    .withAccessModes(PVC_ACCESS_MODE)
-                    .withResources(new ResourceRequirementsBuilder()
-                      .withRequests(Map("storage" -> new Quantity(size.get)).asJava).build())
-                    .endSpec()
-                  .build())
-              }
-
-              claimName
-
             case _ =>

Review comment:
       Late to review. minor, should we match `KubernetesDriverConf` and add anther default case?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29873: [SPARK-32997][K8S] Support dynamic PVC creation and deletion in K8s driver

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29873:
URL: https://github.com/apache/spark/pull/29873#issuecomment-699095678


   **[Test build #129121 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129121/testReport)** for PR 29873 at commit [`90c552c`](https://github.com/apache/spark/commit/90c552ce4da20f49f6bdc536c5557c3fbf2e98ea).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29873: [SPARK-32997][K8S] Support dynamic PVC creation and deletion in K8s driver

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29873:
URL: https://github.com/apache/spark/pull/29873#issuecomment-699095830






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29873: [SPARK-32997][K8S] Support dynamic PVC creation and deletion in K8s driver

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29873:
URL: https://github.com/apache/spark/pull/29873#issuecomment-699110188


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33740/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29873: [SPARK-32997][K8S] Support dynamic PVC creation and deletion in K8s driver

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29873:
URL: https://github.com/apache/spark/pull/29873#issuecomment-699120696






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29873: [SPARK-32997][K8S] Support dynamic PVC creation and deletion in K8s driver

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29873:
URL: https://github.com/apache/spark/pull/29873#issuecomment-699090277


   **[Test build #129121 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129121/testReport)** for PR 29873 at commit [`90c552c`](https://github.com/apache/spark/commit/90c552ce4da20f49f6bdc536c5557c3fbf2e98ea).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #29873: [SPARK-32997][K8S] Support dynamic PVC creation and deletion in K8s driver

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29873:
URL: https://github.com/apache/spark/pull/29873#issuecomment-699135395


   Could you review this please, @holdenk ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29873: [SPARK-32997][K8S] Support dynamic PVC creation and deletion in K8s driver

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29873:
URL: https://github.com/apache/spark/pull/29873#issuecomment-699120684


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33740/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org