You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/07/26 16:03:12 UTC
[GitHub] [spark] xkrogen opened a new pull request #33524: [SPARK-35259][SHUFFLE][3.0] Update ExternalBlockHandler Timer variables to expose correct units
xkrogen opened a new pull request #33524:
URL: https://github.com/apache/spark/pull/33524
`ExternalBlockHandler` exposes 2 metrics which are Dropwizard `Timer` metrics, and are named with a `millis` suffix:
```
private final Timer openBlockRequestLatencyMillis = new Timer();
private final Timer registerExecutorRequestLatencyMillis = new Timer();
```
However these Dropwizard Timers by default use nanoseconds ([documentation](https://metrics.dropwizard.io/3.2.3/getting-started.html#timers)).
This PR adds a new `Timer` subclass, `TimerWithCustomTimeUnit`, which accepts a `TimeUnit` at creation time and exposes timing information using this time unit when values are read. Internally, values are still stored with nanosecond-level precision. The `Timer` metrics within `ExternalBlockHandler` are updated to use the new class with milliseconds as the unit.
This introduces a user-facing change as the values of the metrics themselves have changed, to expose milliseconds instead of nanoseconds. Note that this does not affect metrics such as `openBlockRequestLatencyMillis_count` or `openBlockRequestLatencyMillis_rate1`, only the `Snapshot`-related metrics (`max`, `median`, percentiles, etc.). For the YARN case, these metrics are not exposed prior to 3.2, so there is no change. It was possible for the nanosecond values to be consumed by some other metrics reporter reading the Dropwizard metrics directly, but I'm not aware of any such usages, so this PR is primarily code cleanup and to make branch 3.0 match with 3.2/master.
Note that this differs from the master/3.2 version (#33116) primarily because there are no changes needed in `YarnShuffleServiceMetrics`.
Unit tests have been updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33524: [SPARK-35259][SHUFFLE][3.0] Update ExternalBlockHandler Timer variables to expose correct units
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33524:
URL: https://github.com/apache/spark/pull/33524#issuecomment-886961595
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141644/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33524: [SPARK-35259][SHUFFLE][3.0] Update ExternalBlockHandler Timer variables to expose correct units
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33524:
URL: https://github.com/apache/spark/pull/33524#issuecomment-886845233
**[Test build #141644 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141644/testReport)** for PR 33524 at commit [`4ccedc1`](https://github.com/apache/spark/commit/4ccedc17f2ca5d78302794150811151793908c51).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33524: [SPARK-35259][SHUFFLE][3.0] Update ExternalBlockHandler Timer variables to expose correct units
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33524:
URL: https://github.com/apache/spark/pull/33524#issuecomment-886961595
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141644/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen closed pull request #33524: [SPARK-35259][SHUFFLE][3.0] Update ExternalBlockHandler Timer variables to expose correct units
Posted by GitBox <gi...@apache.org>.
xkrogen closed pull request #33524:
URL: https://github.com/apache/spark/pull/33524
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33524: [SPARK-35259][SHUFFLE][3.0] Update ExternalBlockHandler Timer variables to expose correct units
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33524:
URL: https://github.com/apache/spark/pull/33524#issuecomment-886943311
**[Test build #141644 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141644/testReport)** for PR 33524 at commit [`4ccedc1`](https://github.com/apache/spark/commit/4ccedc17f2ca5d78302794150811151793908c51).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public class TimerWithCustomTimeUnit extends Timer `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33524: [SPARK-35259][SHUFFLE][3.0] Update ExternalBlockHandler Timer variables to expose correct units
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33524:
URL: https://github.com/apache/spark/pull/33524#issuecomment-886874559
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46160/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33524: [SPARK-35259][SHUFFLE][3.0] Update ExternalBlockHandler Timer variables to expose correct units
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33524:
URL: https://github.com/apache/spark/pull/33524#issuecomment-886892069
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46160/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xkrogen commented on pull request #33524: [SPARK-35259][SHUFFLE][3.0] Update ExternalBlockHandler Timer variables to expose correct units
Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #33524:
URL: https://github.com/apache/spark/pull/33524#issuecomment-886830044
fyi @Ngone51
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33524: [SPARK-35259][SHUFFLE][3.0] Update ExternalBlockHandler Timer variables to expose correct units
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33524:
URL: https://github.com/apache/spark/pull/33524#issuecomment-886892097
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46160/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33524: [SPARK-35259][SHUFFLE][3.0] Update ExternalBlockHandler Timer variables to expose correct units
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33524:
URL: https://github.com/apache/spark/pull/33524#issuecomment-886892097
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46160/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33524: [SPARK-35259][SHUFFLE][3.0] Update ExternalBlockHandler Timer variables to expose correct units
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33524:
URL: https://github.com/apache/spark/pull/33524#issuecomment-886845233
**[Test build #141644 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141644/testReport)** for PR 33524 at commit [`4ccedc1`](https://github.com/apache/spark/commit/4ccedc17f2ca5d78302794150811151793908c51).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org