You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/19 06:17:18 UTC
[GitHub] [spark] AngersZhuuuu opened a new pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
AngersZhuuuu opened a new pull request #34326:
URL: https://github.com/apache/spark/pull/34326
### What changes were proposed in this pull request?
Add metrics system to history server.
### Why are the changes needed?
Add metrics to History Server
### Does this PR introduce _any_ user-facing change?
User can use history server monitor data
### How was this patch tested?
Added UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946485296
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48879/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946586097
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48883/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946411227
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48873/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946410604
**[Test build #144400 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144400/testReport)** for PR 34326 at commit [`ac64269`](https://github.com/apache/spark/commit/ac6426945750f58b894e53aaf613c8ed5aee1e55).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-972580431
**[Test build #145376 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145376/testReport)** for PR 34326 at commit [`e173b55`](https://github.com/apache/spark/commit/e173b5518fcf3ffd513af14d534a29767357c20b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971489742
**[Test build #145330 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145330/testReport)** for PR 34326 at commit [`0daf56b`](https://github.com/apache/spark/commit/0daf56bb47246e23aee0f7debb1f9d191c440e77).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971587683
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145330/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-999432545
Gentle ping @dongjoon-hyun @HyukjinKwon @tgravescs @HeartSaVioR @srowen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #34326:
URL: https://github.com/apache/spark/pull/34326#discussion_r780369492
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala
##########
@@ -157,6 +161,15 @@ class HistoryServer(
contextHandler.setContextPath(HistoryServer.UI_PATH_PREFIX)
contextHandler.addServlet(new ServletHolder(loaderServlet), "/*")
attachHandler(contextHandler)
+
+ // Register history server source to history server metrics system.
+ cacheMetrics.init()
Review comment:
why are we calling init here? Doesn't this end up getting called twice?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971489742
**[Test build #145330 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145330/testReport)** for PR 34326 at commit [`0daf56b`](https://github.com/apache/spark/commit/0daf56bb47246e23aee0f7debb1f9d191c440e77).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971585866
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49801/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946536228
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48879/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946522838
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48879/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971242247
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49778/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971352560
**[Test build #145326 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145326/testReport)** for PR 34326 at commit [`f2670c0`](https://github.com/apache/spark/commit/f2670c0fd7f3b8c1303c18a58be2a232e23264ea).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-973774051
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145430/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu removed a comment on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-987684859
Any suggestion? Also ping @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-1014073504
ping @tgravescs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-973762355
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49903/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-972692222
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145376/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971412814
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145323/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971587683
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145330/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-973726375
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49903/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-973101824
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145395/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946607877
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946607877
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144409/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-947297598
**[Test build #144439 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144439/testReport)** for PR 34326 at commit [`719dda2`](https://github.com/apache/spark/commit/719dda2df069d16c09bfa8db8c32e1855b22f4fe).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-972924514
**[Test build #145395 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145395/testReport)** for PR 34326 at commit [`818eba6`](https://github.com/apache/spark/commit/818eba69f703deaca110076d99c6830f5af481b2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-972645591
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49849/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-972632176
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49849/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-947344961
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48912/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946614713
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48883/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-947320959
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48912/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946449199
**[Test build #144404 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144404/testReport)** for PR 34326 at commit [`719dda2`](https://github.com/apache/spark/commit/719dda2df069d16c09bfa8db8c32e1855b22f4fe).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946449233
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144404/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946445029
**[Test build #144404 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144404/testReport)** for PR 34326 at commit [`719dda2`](https://github.com/apache/spark/commit/719dda2df069d16c09bfa8db8c32e1855b22f4fe).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946445029
**[Test build #144404 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144404/testReport)** for PR 34326 at commit [`719dda2`](https://github.com/apache/spark/commit/719dda2df069d16c09bfa8db8c32e1855b22f4fe).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971118865
**[Test build #145304 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145304/testReport)** for PR 34326 at commit [`7bcb63c`](https://github.com/apache/spark/commit/7bcb63ca63cac86f97309bb0d1ffbeef7558c977).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971121895
**[Test build #145305 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145305/testReport)** for PR 34326 at commit [`7015710`](https://github.com/apache/spark/commit/7015710d13e405880f04bc71f3d9112d33ec3223).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] thejdeep commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
thejdeep commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-970253324
@AngersZhuuuu This would be helpful and is a clean implementation. Any chance to complete work on this ?
We can plugin these metrics to any Dropwizard supported monitoring infra and get insights into the performance of SHS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971122928
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145305/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971121895
**[Test build #145305 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145305/testReport)** for PR 34326 at commit [`7015710`](https://github.com/apache/spark/commit/7015710d13e405880f04bc71f3d9112d33ec3223).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971201373
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49775/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971204980
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971444121
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-947372863
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144439/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946492255
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-947289464
> > ### Why are the changes needed?
> > Add metrics to History Server
>
> Let's describe _why_. Why do we need this matrix?
>
> > ### Does this PR introduce _any_ user-facing change?
> > User can use history server monitor data
>
> How can users monitor which data?
Yeah, I know, still under organization, so add `WIP` tag
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] thejdeep commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
thejdeep commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-993836333
Gentle ping on this, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-987753978
cc @HeartSaVioR too FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-973680863
**[Test build #145430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145430/testReport)** for PR 34326 at commit [`06ae18f`](https://github.com/apache/spark/commit/06ae18f9a14164485436226f794dba5c6328ba83).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-973774051
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145430/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-973680863
**[Test build #145430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145430/testReport)** for PR 34326 at commit [`06ae18f`](https://github.com/apache/spark/commit/06ae18f9a14164485436226f794dba5c6328ba83).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-973769698
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49903/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-1033813586
gentle ping @tgravescs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971411982
**[Test build #145323 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145323/testReport)** for PR 34326 at commit [`c8cb0fc`](https://github.com/apache/spark/commit/c8cb0fc4444cdc33138044de3d7f6dcd02bfee7e).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971125205
**[Test build #145307 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145307/testReport)** for PR 34326 at commit [`13af762`](https://github.com/apache/spark/commit/13af762191f48e7b8f527056481e3e3eb7eab0a9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971120055
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145304/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971439050
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49797/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] thejdeep commented on a change in pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
thejdeep commented on a change in pull request #34326:
URL: https://github.com/apache/spark/pull/34326#discussion_r752222837
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerSource.scala
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import com.codahale.metrics.{Gauge, MetricRegistry, Timer}
+
+import org.apache.spark.metrics.source.Source
+
+private[spark] class HistoryServerSource(val history: ApplicationHistoryProvider) extends Source {
+ override val sourceName: String = "historyServer"
+ override val metricRegistry: MetricRegistry = new MetricRegistry()
+
+ metricRegistry.register(MetricRegistry.name("uncompleted"), new Gauge[Int] {
Review comment:
incompleted instead of uncompleted ?
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala
##########
@@ -157,6 +161,14 @@ class HistoryServer(
contextHandler.setContextPath(HistoryServer.UI_PATH_PREFIX)
contextHandler.addServlet(new ServletHolder(loaderServlet), "/*")
attachHandler(contextHandler)
+
+ // Register history server source to history server metrics system.
+ cacheMetrics.init()
+ metricsSystem.registerSource(provider.getHistoryServerSource())
+ metricsSystem.registerSource(cacheMetrics)
+ metricsSystem.start(false)
Review comment:
can we add a comment about why we are passing false here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971454519
**[Test build #145326 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145326/testReport)** for PR 34326 at commit [`f2670c0`](https://github.com/apache/spark/commit/f2670c0fd7f3b8c1303c18a58be2a232e23264ea).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971584036
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49801/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-973084694
**[Test build #145395 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145395/testReport)** for PR 34326 at commit [`818eba6`](https://github.com/apache/spark/commit/818eba69f703deaca110076d99c6830f5af481b2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-972924514
**[Test build #145395 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145395/testReport)** for PR 34326 at commit [`818eba6`](https://github.com/apache/spark/commit/818eba69f703deaca110076d99c6830f5af481b2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946494117
**[Test build #144409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144409/testReport)** for PR 34326 at commit [`719dda2`](https://github.com/apache/spark/commit/719dda2df069d16c09bfa8db8c32e1855b22f4fe).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946411248
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48873/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946414342
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144400/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946494117
**[Test build #144409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144409/testReport)** for PR 34326 at commit [`719dda2`](https://github.com/apache/spark/commit/719dda2df069d16c09bfa8db8c32e1855b22f4fe).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-947285225
> ### Why are the changes needed?
> Add metrics to History Server
Let's describe _why_. Why do we need this matrix?
> ### Does this PR introduce _any_ user-facing change?
> User can use history server monitor data
How can users monitor which data?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-947297598
**[Test build #144439 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144439/testReport)** for PR 34326 at commit [`719dda2`](https://github.com/apache/spark/commit/719dda2df069d16c09bfa8db8c32e1855b22f4fe).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhouyejoe commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
zhouyejoe commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-1027353355
> * If we don't set `spark.history.store.path`, it will use memory to store app info and it's not belong to LevelDB metrics, right?
Yes, correct. In this case, it will only use the in memory key value store.
> * Only when support hybrid store disk backend, then we have in-memory LevelDBs, right?
I don't quite get the question here. When hybrid mode is enabled, the in-memory store used during parsing is called [InMemoryStore](https://github.com/apache/spark/blob/master/common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java#L45), not the actual LevelDB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971094669
> @AngersZhuuuu This would be helpful and is a clean implementation. Any chance to complete work on this ?
>
> We can plugin these metrics to any Dropwizard supported monitoring infra and get insights into the performance of SHS.
Yea, will start this this week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971315087
**[Test build #145323 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145323/testReport)** for PR 34326 at commit [`c8cb0fc`](https://github.com/apache/spark/commit/c8cb0fc4444cdc33138044de3d7f6dcd02bfee7e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971352560
**[Test build #145326 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145326/testReport)** for PR 34326 at commit [`f2670c0`](https://github.com/apache/spark/commit/f2670c0fd7f3b8c1303c18a58be2a232e23264ea).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971326096
@thejdeep @HyukjinKwon Update pr desc and add unit test to show the result. Hope for your review and can suggest if I have miss some thing that can be added to metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-973037748
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49868/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-973101824
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145395/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971198007
**[Test build #145307 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145307/testReport)** for PR 34326 at commit [`13af762`](https://github.com/apache/spark/commit/13af762191f48e7b8f527056481e3e3eb7eab0a9).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971402597
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49797/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971437635
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49794/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971125205
**[Test build #145307 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145307/testReport)** for PR 34326 at commit [`13af762`](https://github.com/apache/spark/commit/13af762191f48e7b8f527056481e3e3eb7eab0a9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #34326:
URL: https://github.com/apache/spark/pull/34326#discussion_r780655043
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala
##########
@@ -157,6 +161,15 @@ class HistoryServer(
contextHandler.setContextPath(HistoryServer.UI_PATH_PREFIX)
contextHandler.addServlet(new ServletHolder(loaderServlet), "/*")
attachHandler(contextHandler)
+
+ // Register history server source to history server metrics system.
+ cacheMetrics.init()
Review comment:
> why are we calling init here? Doesn't this end up getting called twice?
Only called once here, before this method is never called. I am also confused about this..
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala
##########
@@ -157,6 +161,15 @@ class HistoryServer(
contextHandler.setContextPath(HistoryServer.UI_PATH_PREFIX)
contextHandler.addServlet(new ServletHolder(loaderServlet), "/*")
attachHandler(contextHandler)
+
+ // Register history server source to history server metrics system.
+ cacheMetrics.init()
Review comment:
> why are we calling init here? Doesn't this end up getting called twice?
Only called once here. At before this method is never called. I am also confused about this..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-947344961
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48912/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946635090
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48883/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-947278219
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-973683363
ping @HyukjinKwon also ping @mridulm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971122928
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145305/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971167215
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49774/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971205263
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49778/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971315087
**[Test build #145323 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145323/testReport)** for PR 34326 at commit [`c8cb0fc`](https://github.com/apache/spark/commit/c8cb0fc4444cdc33138044de3d7f6dcd02bfee7e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971455615
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145326/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-972685918
**[Test build #145376 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145376/testReport)** for PR 34326 at commit [`e173b55`](https://github.com/apache/spark/commit/e173b5518fcf3ffd513af14d534a29767357c20b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-972580431
**[Test build #145376 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145376/testReport)** for PR 34326 at commit [`e173b55`](https://github.com/apache/spark/commit/e173b5518fcf3ffd513af14d534a29767357c20b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-972692222
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145376/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-972975636
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49868/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-973037748
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49868/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971534976
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49801/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-972607484
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49849/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-974314111
+CC @shardulm94, @zhouyejoe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-973769698
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49903/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946411248
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48873/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-947285225
> ### Why are the changes needed?
> Add metrics to History Server
Let's describe _why_. Why do we need this matrix?
> ### Does this PR introduce _any_ user-facing change?
> User can use history server monitor data
How users can monitor which data?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946536228
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48879/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-947362830
**[Test build #144439 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144439/testReport)** for PR 34326 at commit [`719dda2`](https://github.com/apache/spark/commit/719dda2df069d16c09bfa8db8c32e1855b22f4fe).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #34326:
URL: https://github.com/apache/spark/pull/34326#discussion_r780369492
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala
##########
@@ -157,6 +161,15 @@ class HistoryServer(
contextHandler.setContextPath(HistoryServer.UI_PATH_PREFIX)
contextHandler.addServlet(new ServletHolder(loaderServlet), "/*")
attachHandler(contextHandler)
+
+ // Register history server source to history server metrics system.
+ cacheMetrics.init()
Review comment:
why are we calling init here? Doesn't this end up getting called twice?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-1006317362
> It is great to see that more metrics are being added to History server. Updating the listing.ldb is one of the major factors impacting the History server performance, there are several new metrics tracked for listing.ldb. But there is also another factor, which is generating each individual application levelDB when there is a user request. There are some metrics tracking the LevelDB generating time and loading time in this PR. Can you help add some more metrics, such as how many applications are being parsed to generated their own LevelDBs, and how many LevelDBs are currently stored in Disk, and what is the total size of them. Thanks.
Update a change. And I want confirm with you that:
1. If we don't set `spark.history.store.path`, it will use memory to store app info and it's not belong to LevelDB metrics, right?
2. Only when support hybrid store disk backend, then we have in-memory LevelDBs, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-987684859
Any suggestion? Also ping @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-1081357770
Any suggestion? @tgravescs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu removed a comment on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-1033813586
gentle ping @tgravescs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971118865
**[Test build #145304 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145304/testReport)** for PR 34326 at commit [`7bcb63c`](https://github.com/apache/spark/commit/7bcb63ca63cac86f97309bb0d1ffbeef7558c977).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971242247
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49778/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971120055
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145304/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971221777
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49778/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971390976
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49794/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-973773382
**[Test build #145430 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145430/testReport)** for PR 34326 at commit [`06ae18f`](https://github.com/apache/spark/commit/06ae18f9a14164485436226f794dba5c6328ba83).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946414294
**[Test build #144400 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144400/testReport)** for PR 34326 at commit [`ac64269`](https://github.com/apache/spark/commit/ac6426945750f58b894e53aaf613c8ed5aee1e55).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946414342
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144400/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946449233
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144404/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946606892
**[Test build #144409 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144409/testReport)** for PR 34326 at commit [`719dda2`](https://github.com/apache/spark/commit/719dda2df069d16c09bfa8db8c32e1855b22f4fe).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-947372863
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144439/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-1004554015
Gentle ping on this, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-1007968668
> at a high level seems fine, I need to take a more detailed look at a few parts. It also needs docs.
Doc added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] thejdeep commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
thejdeep commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-1027219993
Gentle ping on this. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971444120
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971120039
**[Test build #145304 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145304/testReport)** for PR 34326 at commit [`7bcb63c`](https://github.com/apache/spark/commit/7bcb63ca63cac86f97309bb0d1ffbeef7558c977).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public class JavaModuleOptions `
* `class SparkConf(object):`
* `class ProbabilisticClassifier(Classifier, _ProbabilisticClassifierParams, metaclass=ABCMeta):`
* `class ProbabilisticClassificationModel(`
* `class _JavaProbabilisticClassifier(ProbabilisticClassifier, _JavaClassifier, metaclass=ABCMeta):`
* `class _JavaProbabilisticClassificationModel(`
* `class _LinearSVCParams(`
* `class LinearSVCModel(`
* `class _LogisticRegressionParams(`
* `class LogisticRegression(`
* `class LogisticRegressionModel(`
* `class BinaryLogisticRegressionSummary(_BinaryClassificationSummary, LogisticRegressionSummary):`
* `class BinaryLogisticRegressionTrainingSummary(`
* `class DecisionTreeClassifier(`
* `class DecisionTreeClassificationModel(`
* `class RandomForestClassifier(`
* `class RandomForestClassificationModel(`
* `class RandomForestClassificationTrainingSummary(`
* `class BinaryRandomForestClassificationTrainingSummary(`
* `class GBTClassifier(`
* `class GBTClassificationModel(`
* `class NaiveBayes(`
* `class NaiveBayesModel(`
* `class _MultilayerPerceptronParams(`
* `class MultilayerPerceptronClassifier(`
* `class MultilayerPerceptronClassificationModel(`
* `class MultilayerPerceptronClassificationTrainingSummary(`
* `class FMClassifier(`
* `class FMClassificationModel(`
* `class _GaussianMixtureParams(`
* `class GaussianMixtureModel(`
* `class _KMeansParams(`
* `class KMeansModel(`
* `class _BisectingKMeansParams(`
* `class BisectingKMeansModel(`
* `class PowerIterationClustering(`
* `class BinaryClassificationEvaluator(`
* `class RegressionEvaluator(`
* `class MulticlassClassificationEvaluator(`
* `class MultilabelClassificationEvaluator(`
* `class ClusteringEvaluator(`
* `class RankingEvaluator(`
* `class Binarizer(`
* `class BucketedRandomProjectionLSH(`
* `class BucketedRandomProjectionLSHModel(`
* `class Bucketizer(`
* `class ElementwiseProduct(`
* `class FeatureHasher(`
* `class HashingTF(`
* `class _OneHotEncoderParams(`
* `class PolynomialExpansion(`
* `class QuantileDiscretizer(`
* `class _StringIndexerParams(`
* `class StopWordsRemover(`
* `class VectorAssembler(`
* `class VectorSizeHint(`
* `class VarianceThresholdSelector(`
* `class VarianceThresholdSelectorModel(`
* `class UnivariateFeatureSelector(`
* `class UnivariateFeatureSelectorModel(`
* `class _LinearRegressionParams(`
* `class LinearRegressionModel(`
* `class IsotonicRegression(`
* `class IsotonicRegressionModel(JavaModel, _IsotonicRegressionParams, JavaMLWritable, JavaMLReadable):`
* `class DecisionTreeRegressor(`
* `class RandomForestRegressor(`
* `class _AFTSurvivalRegressionParams(`
* `class AFTSurvivalRegression(`
* `class AFTSurvivalRegressionModel(`
* `class _GeneralizedLinearRegressionParams(`
* `class GeneralizedLinearRegression(`
* `class GeneralizedLinearRegressionModel(`
* `class _FactorizationMachinesParams(`
* `class FMRegressionModel(`
* `class CrossValidator(`
* `class TrainValidationSplit(`
* ` + \"class name `
* `class MultivariateGaussian(NamedTuple):`
* `class PandasAPIOnSparkAdviceWarning(Warning):`
* `class ResultIterable(Iterable[T]):`
* `class ArrowStreamUDFSerializer(ArrowStreamSerializer):`
* `class SparkJobInfo(NamedTuple):`
* `class SparkStageInfo(NamedTuple):`
* `public class NumericHistogram `
* ` * The Coord class defines a histogram bin, which is just an (x,y) pair.`
* ` public static class Coord implements Comparable `
* `trait ExpressionBuilder `
* `trait FunctionExpressionBuilder `
* `case class ExpressionStats(expr: Expression)(var useCount: Int)`
* `case class HistogramNumeric(`
* `trait ExtractValue extends Expression with NullIntolerant `
* `trait PadExpressionBuilderBase extends ExpressionBuilder `
* `case class StringLPad(str: Expression, len: Expression, pad: Expression)`
* `case class BinaryLPad(str: Expression, len: Expression, pad: Expression, child: Expression)`
* `case class BinaryRPad(str: Expression, len: Expression, pad: Expression, child: Expression)`
* `case class PythonMapInArrow(`
* `case class CreateIndex(`
* `case class DropIndex(`
* `public class ColumnIOUtil `
* `public class OrcColumnStatistics `
* `public class OrcFooterReader `
* `case class ParquetColumn(`
* `case class CreateIndexExec(`
* `case class DropIndexExec(`
* `case class PushedDownOperators(`
* `case class TableSampleInfo(`
* `trait MapInBatchExec extends UnaryExecNode `
* `case class PythonMapInArrowExec(`
* `class RatePerMicroBatchProvider extends SimpleTableProvider with DataSourceRegister `
* `class RatePerMicroBatchTable(`
* `class RatePerMicroBatchStream(`
* `case class RatePerMicroBatchStreamOffset(offset: Long, timestamp: Long) extends Offset `
* `case class RatePerMicroBatchStreamInputPartition(`
* `class RatePerMicroBatchStreamPartitionReader(`
* `class SparkUDFExpressionBuilder extends FunctionExpressionBuilder `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971122910
**[Test build #145305 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145305/testReport)** for PR 34326 at commit [`7015710`](https://github.com/apache/spark/commit/7015710d13e405880f04bc71f3d9112d33ec3223).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971200258
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49774/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971204980
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971412814
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145323/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971455615
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145326/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971168048
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49775/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971586366
**[Test build #145330 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145330/testReport)** for PR 34326 at commit [`0daf56b`](https://github.com/apache/spark/commit/0daf56bb47246e23aee0f7debb1f9d191c440e77).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-971585866
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49801/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhouyejoe commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
zhouyejoe commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-1005121529
It is great to see that more metrics are being added to History server.
Updating the listing.ldb is one of the major factors impacting the History server performance, there are several new metrics tracked for listing.ldb. But there is also another factor, which is generating each individual application levelDB when there is a user request. There are some metrics tracking the LevelDB generating time and loading time in this PR. Can you help add some more metrics, such as how many applications are being parsed to generated their own LevelDBs, and how many LevelDBs are currently stored in Disk, and what is the total size of them. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-947343572
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48912/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-972645591
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49849/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #34326:
URL: https://github.com/apache/spark/pull/34326#discussion_r752267434
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerSource.scala
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import com.codahale.metrics.{Gauge, MetricRegistry, Timer}
+
+import org.apache.spark.metrics.source.Source
+
+private[spark] class HistoryServerSource(val history: ApplicationHistoryProvider) extends Source {
+ override val sourceName: String = "historyServer"
+ override val metricRegistry: MetricRegistry = new MetricRegistry()
+
+ metricRegistry.register(MetricRegistry.name("uncompleted"), new Gauge[Int] {
Review comment:
> incompleted
Done
##########
File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala
##########
@@ -157,6 +161,14 @@ class HistoryServer(
contextHandler.setContextPath(HistoryServer.UI_PATH_PREFIX)
contextHandler.addServlet(new ServletHolder(loaderServlet), "/*")
attachHandler(contextHandler)
+
+ // Register history server source to history server metrics system.
+ cacheMetrics.init()
+ metricsSystem.registerSource(provider.getHistoryServerSource())
+ metricsSystem.registerSource(cacheMetrics)
+ metricsSystem.start(false)
Review comment:
> can we add a comment about why we are passing false here ?
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-973034670
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49868/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34326: [WIP][SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-946410604
**[Test build #144400 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144400/testReport)** for PR 34326 at commit [`ac64269`](https://github.com/apache/spark/commit/ac6426945750f58b894e53aaf613c8ed5aee1e55).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #34326:
URL: https://github.com/apache/spark/pull/34326#issuecomment-997564772
Can any committer review this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org