You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Piotr Nowojski (JIRA)" <ji...@apache.org> on 2019/08/12 10:21:00 UTC

[jira] [Commented] (FLINK-12818) Improve stability of twoInputMapSink benchmark

    [ https://issues.apache.org/jira/browse/FLINK-12818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905051#comment-16905051 ] 

Piotr Nowojski commented on FLINK-12818:
----------------------------------------

Thanks for the investigation. I tried running with your configuration and with disabled checkpointing, but my local results were even less stable:

{noformat}
# Run progress: 0.00% complete, ETA 00:00:24
# Fork: 1 of 3
# Warmup Iteration   1: 5428.001 ops/ms
# Warmup Iteration   2: 9326.627 ops/ms
# Warmup Iteration   3: 10375.921 ops/ms
Iteration   1: 11318.975 ops/ms
Iteration   2: 10919.210 ops/ms
Iteration   3: 11385.420 ops/ms
Iteration   4: 11588.595 ops/ms
Iteration   5: 11699.230 ops/ms

# Run progress: 33.33% complete, ETA 00:00:44
# Fork: 2 of 3
# Warmup Iteration   1: 6764.317 ops/ms
# Warmup Iteration   2: 12866.327 ops/ms
# Warmup Iteration   3: 13283.575 ops/ms
Iteration   1: 18927.883 ops/ms
Iteration   2: 18005.842 ops/ms
Iteration   3: 16681.179 ops/ms
Iteration   4: 20368.397 ops/ms
Iteration   5: 16163.679 ops/ms

# Run progress: 66.67% complete, ETA 00:00:18
# Fork: 3 of 3
# Warmup Iteration   1: 6622.793 ops/ms
# Warmup Iteration   2: 12608.261 ops/ms
# Warmup Iteration   3: 14369.082 ops/ms
Iteration   1: 14280.980 ops/ms
Iteration   2: 14649.605 ops/ms
Iteration   3: 14222.255 ops/ms
Iteration   4: 14735.759 ops/ms
Iteration   5: 14227.849 ops/ms
{noformat}

Also disabling the checkpointing also removes benchmark coverage in a code that has changed significantly between selectable and non-selectable TwoInputOperators.

I think if we want to investigate this further, we would need to print compiled byte code and try to understand the difference.

Alternatively until this is confirmed as a production code issue, we can just try to live with this benchmark instability and relay on long term trends.

> Improve stability of twoInputMapSink benchmark
> ----------------------------------------------
>
>                 Key: FLINK-12818
>                 URL: https://issues.apache.org/jira/browse/FLINK-12818
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Benchmarks
>            Reporter: Piotr Nowojski
>            Priority: Critical
>         Attachments: RecordWriter-emit.png
>
>
> The {{twoInputMapSink}} benchmark is very unstable over time:
> http://codespeed.dak8s.net:8000/timeline/#/?exe=1&ben=twoInputMapSink&env=2&revs=200&equid=off&quarts=on&extr=on
> It should be fixed, otherwise the benchmark can not be used.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)