You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Hai Zhou UTC+8 (JIRA)" <ji...@apache.org> on 2017/10/08 12:44:00 UTC
[jira] [Comment Edited] (FLINK-7608) LatencyGauge change to
histogram metric
[ https://issues.apache.org/jira/browse/FLINK-7608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16196096#comment-16196096 ]
Hai Zhou UTC+8 edited comment on FLINK-7608 at 10/8/17 12:43 PM:
-----------------------------------------------------------------
Thanks [~rmetzger], [~StephanEwen] for the suggestion.
true, i think that we need to re-discuss the choice of which histogram implementation for latency statistics.
*Option #1*
Currently, using histograms is *DescriptiveStatistics* from commons-math3.jar. [source | https://github.com/apache/commons-math/blob/dcaed2dc2838cfbb05d4bb783c5740668e659262/src/main/java/org/apache/commons/math4/stat/descriptive/DescriptiveStatistics.java].
DescriptiveStatistics maintains the input data in memory and has the capability of producing "rolling" statistics computed from a "window" consisting of the most recently added values.
* Add a value with O(1) time complexity.
* Call min、 max、 mean method once with O(N) time complexity.
* Call Percentile method(use kthSelector algorithm(binary heap)) once with O(N log2 N) time complexity.
*Option #2*
Dropwizard's *SlidingWindowReservoirs* [source | https://github.com/dropwizard/metrics/blob/4.0-development/metrics-core/src/main/java/com/codahale/metrics/SlidingTimeWindowReservoir.java].
A histogram with a sliding window reservoir produces quantiles which are representative of the past N measurements.
* Add value with O(1) time complexity.
* When reporter metrics, need do a snapshot (need Arrays.sort) with O(N log2 N) time complexity.
* After snapshot, Call min、 max、 mean、 percentile method once with O(1) time complexity.
*My summary:*
SlidingWindowReservoirs is much less time cost than the DescriptiveStatistics time in our use scene, and the implementation of the code is also simpler. replacement is also very simple.
If we use SlidingWindowReservoirs, we only need to make the following changes
replace new DescriptiveStatistics(size) to
new DropwizardHistogramWrapper(new com.codahale.metrics.Histogram(new SlidingWindowReservoir(size)))
[~rmetzger], Let me know your opinions?
was (Author: yew1eb):
Thanks [~rmetzger], [~StephanEwen] for the suggestion.
true, i think that we need to re-discuss the choice of which histogram implementation for latency statistics.
*Option #1*
Currently, using histograms is *DescriptiveStatistics* from commons-math3.jar. [source | https://github.com/apache/commons-math/blob/dcaed2dc2838cfbb05d4bb783c5740668e659262/src/main/java/org/apache/commons/math4/stat/descriptive/DescriptiveStatistics.java].
DescriptiveStatistics maintains the input data in memory and has the capability of producing "rolling" statistics computed from a "window" consisting of the most recently added values.
* Add a value with O(1) time complexity.
* Call min、 max、 mean method once with O(n) time complexity.
* Call Percentile method(use kthSelector algorithm(binary heap)) once with O(N log2 N) time complexity.
*Option #2*
Dropwizard's *SlidingWindowReservoirs* [source | https://github.com/dropwizard/metrics/blob/4.0-development/metrics-core/src/main/java/com/codahale/metrics/SlidingTimeWindowReservoir.java].
A histogram with a sliding window reservoir produces quantiles which are representative of the past N measurements.
* Add value with O(1) time complexity.
* When reporter metrics, need do a snapshot (need Arrays.sort) with O(N log2 N) time complexity.
* After snapshot, Call min、 max、 mean、 percentile method once with O(1) time complexity.
*My summary:*
SlidingWindowReservoirs is much less time cost than the DescriptiveStatistics time in our use scene, and the implementation of the code is also simpler.
If we use SlidingWindowReservoirs, we only need to make the following changes
replace new DescriptiveStatistics(size) to
new DropwizardHistogramWrapper(new com.codahale.metrics.Histogram(new SlidingWindowReservoir(size)))
[~rmetzger], Let me know your opinions?
> LatencyGauge change to histogram metric
> ----------------------------------------
>
> Key: FLINK-7608
> URL: https://issues.apache.org/jira/browse/FLINK-7608
> Project: Flink
> Issue Type: Bug
> Components: Metrics
> Reporter: Hai Zhou UTC+8
> Assignee: Hai Zhou UTC+8
> Priority: Blocker
> Fix For: 1.4.0, 1.3.3
>
>
> I used slf4jReporter[https://issues.apache.org/jira/browse/FLINK-4831] to export metrics the log file.
> I found:
> {noformat}
> -- Gauges ---------------------------------------------------------------------
> ......
> zhouhai-mbp.taskmanager.f3fd3a269c8c3da4e8319c8f6a201a57.Flink Streaming Job.Map.0.latency:
> value={LatencySourceDescriptor{vertexID=1, subtaskIndex=-1}={p99=116.0, p50=59.5, min=11.0, max=116.0, p95=116.0, mean=61.833333333333336}}
> zhouhai-mbp.taskmanager.f3fd3a269c8c3da4e8319c8f6a201a57.Flink Streaming Job.Sink- Unnamed.0.latency:
> value={LatencySourceDescriptor{vertexID=1, subtaskIndex=0}={p99=195.0, p50=163.5, min=115.0, max=195.0, p95=195.0, mean=161.0}}
> ......
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)