You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/04/30 06:16:52 UTC

[GitHub] [spark] dongjoon-hyun opened a new pull request #32406: [SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi

dongjoon-hyun opened a new pull request #32406:
URL: https://github.com/apache/spark/pull/32406


   ### What changes were proposed in this pull request?
   
   Since SPARK-22757, `KubernetesUtils` is used as an important utility class for all K8s module. This issue aims to promote it to DeveloperApi to maintain it officially in a backward compatible way.
   
   ### Why are the changes needed?
   
   Apache Spark provides an extensible external cluster manager framework. To have `ExternalClusterManager` for K8s, `KubernetesUtils` class is very useful and stable. To promote it to K8s developer API, we can maintain this more sustainable way.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes, but this is new API additions.
   
   ### How was this patch tested?
   
   Pass the CIs.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun closed pull request #32406: [SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #32406:
URL: https://github.com/apache/spark/pull/32406


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32406: [SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32406:
URL: https://github.com/apache/spark/pull/32406#discussion_r624082880



##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -325,6 +351,7 @@ private[spark] object KubernetesUtils extends Logging {
     }
   }
 
+  @Since("3.0.0")
   def buildPodWithServiceAccount(serviceAccount: Option[String], pod: SparkPod): Option[Pod] = {

Review comment:
       It's not released yet, @attilapiros ! :)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32406: [SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32406:
URL: https://github.com/apache/spark/pull/32406#issuecomment-829884332


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138101/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #32406: [SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #32406:
URL: https://github.com/apache/spark/pull/32406#issuecomment-830286584


   Then, I'll merge this with 2 approvals. :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32406: [SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32406:
URL: https://github.com/apache/spark/pull/32406#issuecomment-829877710


   **[Test build #138101 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138101/testReport)** for PR 32406 at commit [`c405cab`](https://github.com/apache/spark/commit/c405cab30e2cb7a729192b8a121a43a18324a5ca).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32406: [SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32406:
URL: https://github.com/apache/spark/pull/32406#issuecomment-829909676


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42621/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #32406: [SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #32406:
URL: https://github.com/apache/spark/pull/32406#issuecomment-830284264


   Thank you, @viirya and @attilapiros .
   Although some APIs are very stable since `2.3.0`, this PR starts to annotate these APIs as `@Unstable` at Apache Spark 3.2.0. This is just a beginning to give a better dev experience in K8s area.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32406: [SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32406:
URL: https://github.com/apache/spark/pull/32406#discussion_r624082497



##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -75,11 +86,13 @@ private[spark] object KubernetesUtils extends Logging {
     }
   }
 
+  @Since("2.3.0")
   def requireNandDefined(opt1: Option[_], opt2: Option[_], errMessage: String): Unit = {
     opt1.foreach { _ => require(opt2.isEmpty, errMessage) }
     opt2.foreach { _ => require(opt1.isEmpty, errMessage) }
   }
 
+  @Since("3.2.0")
   def loadPodFromTemplate(

Review comment:
       Please see the following PR description.
   > In this PR, Since annotations denote the last function signature changes because these are going to become public at Apache Spark 3.2.0.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32406: [SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32406:
URL: https://github.com/apache/spark/pull/32406#issuecomment-829909702


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42621/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32406: [SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32406:
URL: https://github.com/apache/spark/pull/32406#discussion_r624081888



##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -75,11 +86,13 @@ private[spark] object KubernetesUtils extends Logging {
     }
   }
 
+  @Since("2.3.0")
   def requireNandDefined(opt1: Option[_], opt2: Option[_], errMessage: String): Unit = {
     opt1.foreach { _ => require(opt2.isEmpty, errMessage) }
     opt2.foreach { _ => require(opt1.isEmpty, errMessage) }
   }
 
+  @Since("3.2.0")
   def loadPodFromTemplate(

Review comment:
       As I mentioned in the PR title, the signature is change recently (2fa792aa64), @attilapiros . 
   I know that because I changed it. :)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32406: [SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32406:
URL: https://github.com/apache/spark/pull/32406#discussion_r624081888



##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -75,11 +86,13 @@ private[spark] object KubernetesUtils extends Logging {
     }
   }
 
+  @Since("2.3.0")
   def requireNandDefined(opt1: Option[_], opt2: Option[_], errMessage: String): Unit = {
     opt1.foreach { _ => require(opt2.isEmpty, errMessage) }
     opt2.foreach { _ => require(opt1.isEmpty, errMessage) }
   }
 
+  @Since("3.2.0")
   def loadPodFromTemplate(

Review comment:
       As I mentioned in the PR title, the signature is change recently, @attilapiros . 
   I know that because I changed it. :)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32406: [SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32406:
URL: https://github.com/apache/spark/pull/32406#issuecomment-829884163


   **[Test build #138101 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138101/testReport)** for PR 32406 at commit [`c405cab`](https://github.com/apache/spark/commit/c405cab30e2cb7a729192b8a121a43a18324a5ca).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #32406: [SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #32406:
URL: https://github.com/apache/spark/pull/32406#issuecomment-830173759


   Hi, @viirya . Could you review this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] attilapiros commented on a change in pull request #32406: [SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi

Posted by GitBox <gi...@apache.org>.
attilapiros commented on a change in pull request #32406:
URL: https://github.com/apache/spark/pull/32406#discussion_r624012224



##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -75,11 +86,13 @@ private[spark] object KubernetesUtils extends Logging {
     }
   }
 
+  @Since("2.3.0")
   def requireNandDefined(opt1: Option[_], opt2: Option[_], errMessage: String): Unit = {
     opt1.foreach { _ => require(opt2.isEmpty, errMessage) }
     opt2.foreach { _ => require(opt1.isEmpty, errMessage) }
   }
 
+  @Since("3.2.0")
   def loadPodFromTemplate(

Review comment:
       This was introduced by https://issues.apache.org/jira/browse/SPARK-24434.
   Which went into Spark 3.0.0.
   
   https://github.com/apache/spark/blob/branch-3.0/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L82
   
   

##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -325,6 +351,7 @@ private[spark] object KubernetesUtils extends Logging {
     }
   }
 
+  @Since("3.0.0")
   def buildPodWithServiceAccount(serviceAccount: Option[String], pod: SparkPod): Option[Pod] = {

Review comment:
       This was backported to 2.4.8:
   https://github.com/apache/spark/blob/branch-2.4/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L66




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32406: [SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32406:
URL: https://github.com/apache/spark/pull/32406#discussion_r624084543



##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -325,6 +351,7 @@ private[spark] object KubernetesUtils extends Logging {
     }
   }
 
+  @Since("3.0.0")
   def buildPodWithServiceAccount(serviceAccount: Option[String], pod: SparkPod): Option[Pod] = {

Review comment:
       2.4.8 RC3 will fail again. Please see this ancient bug fix.
   - https://github.com/apache/spark/pull/32404#issuecomment-829894069




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #32406: [SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32406:
URL: https://github.com/apache/spark/pull/32406#issuecomment-829877710


   **[Test build #138101 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138101/testReport)** for PR 32406 at commit [`c405cab`](https://github.com/apache/spark/commit/c405cab30e2cb7a729192b8a121a43a18324a5ca).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #32406: [SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #32406:
URL: https://github.com/apache/spark/pull/32406#issuecomment-829874108


   Hi, @attilapiros . Could you review this PR, please?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #32406: [SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #32406:
URL: https://github.com/apache/spark/pull/32406#issuecomment-830286488


   Thank you so much for your review, @viirya and @attilapiros !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32406: [SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32406:
URL: https://github.com/apache/spark/pull/32406#issuecomment-829906928


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42621/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32406: [SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32406:
URL: https://github.com/apache/spark/pull/32406#issuecomment-829909702


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42621/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32406: [SPARK-35280][K8S] Promote KubernetesUtils to DeveloperApi

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32406:
URL: https://github.com/apache/spark/pull/32406#issuecomment-829884332


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138101/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org