You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "penguin." <bx...@126.com> on 2021/05/17 12:47:53 UTC
How to get throughput and processing latency when testing Flink
with hibench?
When using hibench to test Flink, the jobs submitted are built-in applications of hibench, that is, the code logic of programs like wordcount cannot be changed.
How can I get the throughput and processing delay of Flink?
In addition, in the /report/hibench.report file of hibench, we can't get the throughput information of Flink. The contents of /report/hibench.report are as follows:
| Type | Date | Time | Input_data_size | Duration(s) | Throunghput(bytes/s) | Throunghput/node |
| FlinkFixWindow | 2021-05-17 | 16:17:58 | 0 | 88.599 | 0 | 0 |
| FlinkFixWindow | 2021-05-17 | 16:33:12 | 0 | 872.982 | 0 | 0 |
| FlinkFixWindow | 2021-05-17 | 16:53:29 | 0 | 301.524 | 0 | 0 |
| FlinkFixWindow | 2021-05-17 | 17:09:22 | 0 | 880.098 | 0 | 0 |
Re: How to get throughput and processing latency when testing Flink
with hibench?
Posted by Nicolaus Weidner <ni...@data-artisans.com>.
Hi,
Flink allows to enable latency tracking [1] and exposes several metrics
that might be what you are looking for [2, look for e.g. "numRecordsIn" or
"numBytesIn"]. You can query these metrics using the REST API [3] or by
registering a metrics reporter [4] that exposes them. As noted in the other
email, I cannot speak about what hibench does and why it cannot show any
throughput information for Flink. Even Flink 1.0.3 seems to expose some
metrics (just going by the docs [5] here, I was not involved back then).
Best wishes,
Nico
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/metrics/#latency-tracking
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/metrics/#io
[3]
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/rest_api/
[4]
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/metric_reporters/
[5]
https://ci.apache.org/projects/flink/flink-docs-release-1.0/internals/monitoring_rest_api.html
On Mon, May 17, 2021 at 2:49 PM penguin. <bx...@126.com> wrote:
> When using hibench to test Flink, the jobs submitted are built-in
> applications of hibench, that is, the code logic of programs like wordcount
> cannot be changed.
>
> How can I get the throughput and processing delay of Flink?
>
> In addition, in the /report/hibench.report file of hibench, we can't get
> the throughput information of Flink. The contents of
> /report/hibench.report are as follows:
> TypeDateTimeInput_data_sizeDuration(s)Throunghput(bytes/s)Throunghput/node
> FlinkFixWindow 2021-05-17 16:17:58 0 88.599 0 0
> FlinkFixWindow 2021-05-17 16:33:12 0 872.982 0 0
> FlinkFixWindow 2021-05-17 16:53:29 0 301.524 0 0
> FlinkFixWindow 2021-05-17 17:09:22 0 880.098 0 0
>
>
>
>