You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/12/22 07:53:53 UTC
[GitHub] [spark] AngersZhuuuu opened a new pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay
AngersZhuuuu opened a new pull request #34982:
URL: https://github.com/apache/spark/pull/34982
### What changes were proposed in this pull request?
Spark will request yarn cluster metrics and print a log about nodemanager number, it's not so important and this rpc is always slow

We can make it as debug level
### Why are the changes needed?
Avoid unnecessary delay when submit application.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Not need
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-999360090
**[Test build #146473 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146473/testReport)** for PR 34982 at commit [`bafb1ce`](https://github.com/apache/spark/commit/bafb1ce7551fca8dd46de6899e3b8bc0befa8854).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-999394770
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146473/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-999437889
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50949/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] asfgit closed pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #34982:
URL: https://github.com/apache/spark/pull/34982
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-999360000
ping @tgravescs @mridulm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-1007504670
I guess I'm fine with this, I kind of assume this is because your yarn resource manager is overloaded or unresponsive though again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-999394770
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146473/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-999399590
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50949/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-1007504670
I guess I'm fine with this, I kind of assume this is because your yarn resource manager is overloaded or unresponsive though again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-999360090
**[Test build #146473 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146473/testReport)** for PR 34982 at commit [`bafb1ce`](https://github.com/apache/spark/commit/bafb1ce7551fca8dd46de6899e3b8bc0befa8854).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu removed a comment on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu removed a comment on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-1009743419
Gentle ping
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-1001842359
Also ping @srowen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-1009743419
Gentle ping
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-1007945556
> I guess I'm fine with this, I kind of assume this is because your yarn resource manager is overloaded or unresponsive though again?
Yea, yarn's performance is not so stable. With this patch, performance become better ==
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-1014073619
ping @srowen @tgravescs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-999437854
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50949/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-999437889
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50949/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-999377163
**[Test build #146473 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146473/testReport)** for PR 34982 at commit [`bafb1ce`](https://github.com/apache/spark/commit/bafb1ce7551fca8dd46de6899e3b8bc0befa8854).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-1004486678
ping @tgravescs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #34982:
URL: https://github.com/apache/spark/pull/34982#discussion_r786342436
##########
File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
##########
@@ -183,8 +183,10 @@ private[spark] class Client(
yarnClient.init(hadoopConf)
yarnClient.start()
- logInfo("Requesting a new application from cluster with %d NodeManagers"
- .format(yarnClient.getYarnClusterMetrics.getNumNodeManagers))
+ if (log.isDebugEnabled) {
+ logDebug("Requesting a new application from cluster with %d NodeManagers"
+ .format(yarnClient.getYarnClusterMetrics.getNumNodeManagers))
+ }
Review comment:
super nit: we dont need `isDebugEnabled` :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #34982: [SPARK-37712][YARN] Spark request yarn cluster metrics slow cause delay
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #34982:
URL: https://github.com/apache/spark/pull/34982#issuecomment-1014969982
Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org