You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/12/22 07:53:53 UTC

[GitHub] [spark] AngersZhuuuu opened a new pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay

AngersZhuuuu opened a new pull request #34982:
URL: https://github.com/apache/spark/pull/34982


   ### What changes were proposed in this pull request?
   Spark will request yarn cluster metrics and print a log about nodemanager number, it's not so important and this rpc is always slow
   ![image](https://user-images.githubusercontent.com/46485123/147055954-30698764-b313-419f-8759-772ad9f301ff.png)
   
   We can make it as debug level
   
   
   ### Why are the changes needed?
   Avoid unnecessary delay when submit application.
   
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   
   ### How was this patch tested?
   Not need
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-999360090


   **[Test build #146473 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146473/testReport)** for PR 34982 at commit [`bafb1ce`](https://github.com/apache/spark/commit/bafb1ce7551fca8dd46de6899e3b8bc0befa8854).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-999394770


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146473/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-999437889


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50949/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] asfgit closed pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay

Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #34982:
URL: https://github.com/apache/spark/pull/34982


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-999360000


   ping @tgravescs @mridulm 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] tgravescs commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay

Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-1007504670


   I guess I'm fine with this, I kind of assume this is because your yarn resource manager is overloaded or unresponsive though again?  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-999394770


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146473/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-999399590


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50949/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] tgravescs commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay

Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-1007504670


   I guess I'm fine with this, I kind of assume this is because your yarn resource manager is overloaded or unresponsive though again?  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-999360090


   **[Test build #146473 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146473/testReport)** for PR 34982 at commit [`bafb1ce`](https://github.com/apache/spark/commit/bafb1ce7551fca8dd46de6899e3b8bc0befa8854).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu removed a comment on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu removed a comment on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-1009743419


   Gentle ping 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-1001842359


   Also ping @srowen 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-1009743419


   Gentle ping 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-1007945556


   > I guess I'm fine with this, I kind of assume this is because your yarn resource manager is overloaded or unresponsive though again?
   
   Yea, yarn's performance is not so stable. With this patch, performance become better ==


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-1014073619


   ping @srowen @tgravescs 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-999437854


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50949/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-999437889


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50949/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-999377163


   **[Test build #146473 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146473/testReport)** for PR 34982 at commit [`bafb1ce`](https://github.com/apache/spark/commit/bafb1ce7551fca8dd46de6899e3b8bc0befa8854).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AngersZhuuuu commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay

Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-1004486678


   ping @tgravescs


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] mridulm commented on a change in pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay

Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #34982:
URL: https://github.com/apache/spark/pull/34982#discussion_r786342436



##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
##########
@@ -183,8 +183,10 @@ private[spark] class Client(
       yarnClient.init(hadoopConf)
       yarnClient.start()
 
-      logInfo("Requesting a new application from cluster with %d NodeManagers"
-        .format(yarnClient.getYarnClusterMetrics.getNumNodeManagers))
+      if (log.isDebugEnabled) {
+        logDebug("Requesting a new application from cluster with %d NodeManagers"
+          .format(yarnClient.getYarnClusterMetrics.getNumNodeManagers))
+      }

Review comment:
       super nit: we dont need `isDebugEnabled` :-)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] mridulm commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay

Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-1014969982


   Merging to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org