You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Nico Kruber (Jira)" <ji...@apache.org> on 2019/08/21 15:09:00 UTC
[jira] [Closed] (FLINK-12983) Replace descriptive histogram's
storage back-end
[ https://issues.apache.org/jira/browse/FLINK-12983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nico Kruber closed FLINK-12983.
-------------------------------
Fix Version/s: 1.10.0
Resolution: Fixed
merged to master via f57a615
> Replace descriptive histogram's storage back-end
> ------------------------------------------------
>
> Key: FLINK-12983
> URL: https://issues.apache.org/jira/browse/FLINK-12983
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Metrics
> Reporter: Nico Kruber
> Assignee: Nico Kruber
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {{DescriptiveStatistics}} relies on their {{ResizableDoubleArray}} for storing double values for their histograms. However, this is constantly resizing an internal array and seems to have quite some overhead.
> Additionally, we're not using {{SynchronizedDescriptiveStatistics}} which, according to its docs, we should. Currently, we seem to be somewhat safe because {{ResizableDoubleArray}} has some synchronized parts but these are scheduled to go away with commons.math version 4.
> Internal tests with the current implementation, one based on a linear array of twice the histogram size (and moving values back to the start once the window reaches the end), and one using a circular array (wrapping around with flexible start position) has shown these numbers using the optimised code from FLINK-10236, FLINK-12981, and FLINK-12982:
> # only adding values to the histogram
> {code}
> Benchmark Mode Cnt Score Error Units
> HistogramBenchmarks.dropwizardHistogramAdd thrpt 30 47985.359 ± 25.847 ops/ms
> HistogramBenchmarks.descriptiveHistogramAdd thrpt 30 70158.792 ± 276.858 ops/ms
> --- with FLINK-10236, FLINK-12981, and FLINK-12982 ---
> HistogramBenchmarks.descriptiveHistogramAdd thrpt 30 75303.040 ± 475.355 ops/ms
> HistogramBenchmarks.descrHistogramCircularAdd thrpt 30 200906.902 ± 384.483 ops/ms
> HistogramBenchmarks.descrHistogramLinearAdd thrpt 30 189788.728 ± 233.283 ops/ms
> {code}
> # after adding each value, also retrieving a common set of metrics:
> {code}
> Benchmark Mode Cnt Score Error Units
> HistogramBenchmarks.dropwizardHistogram thrpt 30 400.274 ± 4.930 ops/ms
> HistogramBenchmarks.descriptiveHistogram thrpt 30 124.533 ± 1.060 ops/ms
> --- with FLINK-10236, FLINK-12981, and FLINK-12982 ---
> HistogramBenchmarks.descriptiveHistogram thrpt 30 251.895 ± 1.809 ops/ms
> HistogramBenchmarks.descrHistogramCircular thrpt 30 301.068 ± 2.077 ops/ms
> HistogramBenchmarks.descrHistogramLinear thrpt 30 234.050 ± 5.485 ops/ms
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)