You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/03/02 01:18:42 UTC

[GitHub] [spark] dongjoon-hyun opened a new pull request #35704: [SPARK-38383][K8S] Support APP_ID and EXECUTOR_ID placeholder in annotations

dongjoon-hyun opened a new pull request #35704:
URL: https://github.com/apache/spark/pull/35704


   ### What changes were proposed in this pull request?
   
   This PR aims to support `APP_ID` and `EXECUTOR_ID` placeholder in K8s annotation in the same way we did for `EXECUTOR_JAVA_OPTIONS`.
   
   ### Why are the changes needed?
   
   Although Apache Spark provides `spark-app-id` already, some custom schedulers are not able to recognize them.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No because the pattern strings are very specific.
   
   ### How was this patch tested?
   
   Pass the CIs.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #35704: [SPARK-38383][K8S] Support `APP_ID` and `EXECUTOR_ID` placeholder in annotations

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35704:
URL: https://github.com/apache/spark/pull/35704#issuecomment-1056135521


   Could you review this when you have some time, please, @viirya ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35704: [SPARK-38383][K8S] Support `APP_ID` and `EXECUTOR_ID` placeholder in annotations

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35704:
URL: https://github.com/apache/spark/pull/35704#discussion_r817324443



##########
File path: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala
##########
@@ -82,12 +82,14 @@ private[spark] trait BasicTestsSuite { k8sSuite: KubernetesSuite =>
       .set("spark.kubernetes.driver.label.label2", "label2-value")
       .set("spark.kubernetes.driver.annotation.annotation1", "annotation1-value")
       .set("spark.kubernetes.driver.annotation.annotation2", "annotation2-value")
+      .set("spark.kubernetes.driver.annotation.yunikorn.apache.org/app-id", "{{APP_ID}}")
       .set("spark.kubernetes.driverEnv.ENV1", "VALUE1")
       .set("spark.kubernetes.driverEnv.ENV2", "VALUE2")
       .set("spark.kubernetes.executor.label.label1", "label1-value")
       .set("spark.kubernetes.executor.label.label2", "label2-value")
       .set("spark.kubernetes.executor.annotation.annotation1", "annotation1-value")
       .set("spark.kubernetes.executor.annotation.annotation2", "annotation2-value")
+      .set("spark.kubernetes.executor.annotation.yunikorn.apache.org/app-id", "{{APP_ID}}")

Review comment:
       This integration test covers `driver` and `executor` cases together.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #35704: [SPARK-38383][K8S] Support APP_ID and EXECUTOR_ID placeholder in annotations

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35704:
URL: https://github.com/apache/spark/pull/35704#issuecomment-1056044359


   cc @yangwwei


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #35704: [SPARK-38383][K8S] Support `APP_ID` and `EXECUTOR_ID` placeholder in annotations

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35704:
URL: https://github.com/apache/spark/pull/35704#issuecomment-1056175735


   Thank you so much, @viirya .
   Merged to master for Apache Spark 3.3.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun closed pull request #35704: [SPARK-38383][K8S] Support `APP_ID` and `EXECUTOR_ID` placeholder in annotations

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #35704:
URL: https://github.com/apache/spark/pull/35704


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org