You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "Geode Integration (Jira)" <ji...@apache.org> on 2021/09/10 20:49:00 UTC
[jira] [Commented] (GEODE-9302) Benchmark instability in
PartitionedPutStringBenchmark
[ https://issues.apache.org/jira/browse/GEODE-9302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17413386#comment-17413386 ]
Geode Integration commented on GEODE-9302:
------------------------------------------
Seen in [benchmark-base #175|https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/benchmark-base/builds/175].
> Benchmark instability in PartitionedPutStringBenchmark
> ------------------------------------------------------
>
> Key: GEODE-9302
> URL: https://issues.apache.org/jira/browse/GEODE-9302
> Project: Geode
> Issue Type: Bug
> Components: benchmarks
> Affects Versions: 1.15.0
> Reporter: Donal Evans
> Priority: Major
>
> A benchmark failure due to the recently-introduced PartitionedPutStringBenchmark was observed:
> {noformat}
> This is ITERATION 1 of benchmarking against baseline.
> P2pPartitionedGetBenchmark avg ops/sec Baseline: 853001.60 Test: 867151.67 Difference: +1.7%
> avg latency Baseline: 842007.55 Test: 828545.06 Difference: -1.6%
> P2pPartitionedPutBenchmark avg ops/sec Baseline: 128283.47 Test: 126510.92 Difference: -1.4%
> avg latency Baseline: 5785619.62 Test: 5915913.49 Difference: +2.3%
> P2pPartitionedPutBytesBenchmark avg ops/sec Baseline: 175658.08 Test: 174865.97 Difference: -0.5%
> avg latency Baseline: 4130071.43 Test: 4130753.09 Difference: +0.0%
> PartitionedFunctionExecutionBenchmark avg ops/sec Baseline: 254788.26 Test: 268132.99 Difference: +5.2%
> avg latency Baseline: 846158.41 Test: 804199.42 Difference: -5.0%
> PartitionedFunctionExecutionWithArgumentsBenchmark avg ops/sec Baseline: 278669.87 Test: 281504.58 Difference: +1.0%
> avg latency Baseline: 1031826.82 Test: 1021314.54 Difference: -1.0%
> PartitionedFunctionExecutionWithFiltersBenchmark avg ops/sec Baseline: 372204.82 Test: 348815.81 Difference: -6.3%
> avg latency Baseline: 1545217.38 Test: 1649706.37 Difference: +6.8%
> PartitionedGetBenchmark avg ops/sec Baseline: 823740.09 Test: 819044.99 Difference: -0.6%
> avg latency Baseline: 872172.75 Test: 877580.02 Difference: +0.6%
> PartitionedGetLongBenchmark avg ops/sec Baseline: 1047221.43 Test: 1045565.89 Difference: -0.2%
> avg latency Baseline: 685757.55 Test: 687005.43 Difference: +0.2%
> PartitionedGetStringBenchmark avg ops/sec Baseline: 1055904.14 Test: 1045420.73 Difference: -1.0%
> avg latency Baseline: 680031.44 Test: 687045.15 Difference: +1.0%
> PartitionedIndexedQueryBenchmark avg ops/sec Baseline: 31596.35 Test: 31653.48 Difference: +0.2%
> avg latency Baseline: 18221302.10 Test: 18216097.86 Difference: -0.0%
> PartitionedNonIndexedQueryBenchmark avg ops/sec Baseline: 95.78 Test: 100.35 Difference: +4.8%
> avg latency Baseline: 750871203.78 Test: 716853923.95 Difference: -4.5%
> PartitionedPutAllBenchmark avg ops/sec Baseline: 8675.75 Test: 8628.10 Difference: -0.5%
> avg latency Baseline: 16595044.73 Test: 16685258.91 Difference: +0.5%
> PartitionedPutAllLongBenchmark avg ops/sec Baseline: 1382.38 Test: 1380.50 Difference: -0.1%
> avg latency Baseline: 104866853.92 Test: 104775538.34 Difference: -0.1%
> PartitionedPutBenchmark avg ops/sec Baseline: 491790.40 Test: 479926.75 Difference: -2.4%
> avg latency Baseline: 1461947.23 Test: 1497519.77 Difference: +2.4%
> PartitionedPutBytesBenchmark avg ops/sec Baseline: 472520.77 Test: 475046.43 Difference: +0.5%
> avg latency Baseline: 1523521.43 Test: 1515515.20 Difference: -0.5%
> PartitionedPutLongBenchmark avg ops/sec Baseline: 412720.03 Test: 389975.92 Difference: -5.5%
> avg latency Baseline: 1740407.45 Test: 1842985.87 Difference: +5.9%
> PartitionedPutStringBenchmark avg ops/sec Baseline: 430083.15 Test: 402523.17 Difference: -6.4%
> avg latency Baseline: 1671189.90 Test: 1786949.71 Difference: +6.9%
> ReplicatedFunctionExecutionBenchmark avg ops/sec Baseline: 513699.75 Test: 514378.00 Difference: +0.1%
> avg latency Baseline: 418897.75 Test: 418374.49 Difference: -0.1%
> ReplicatedFunctionExecutionWithArgumentsBenchmark avg ops/sec Baseline: 525004.44 Test: 522304.20 Difference: -0.5%
> avg latency Baseline: 2192090.72 Test: 2203176.97 Difference: +0.5%
> ReplicatedFunctionExecutionWithFiltersBenchmark avg ops/sec Baseline: 540464.25 Test: 532795.86 Difference: -1.4%
> avg latency Baseline: 1330260.37 Test: 1349440.25 Difference: +1.4%
> ReplicatedGetBenchmark avg ops/sec Baseline: 827036.49 Test: 817813.56 Difference: -1.1%
> avg latency Baseline: 868818.39 Test: 878720.34 Difference: +1.1%
> ReplicatedGetLongBenchmark avg ops/sec Baseline: 1085165.62 Test: 1071977.64 Difference: -1.2%
> avg latency Baseline: 661809.29 Test: 670064.70 Difference: +1.2%
> ReplicatedIndexedQueryBenchmark avg ops/sec Baseline: 32216.21 Test: 31293.98 Difference: -2.9%
> avg latency Baseline: 17873159.19 Test: 18437420.16 Difference: +3.2%
> ReplicatedNonIndexedQueryBenchmark avg ops/sec Baseline: 51.62 Test: 47.74 Difference: -7.5%
> avg latency Baseline: 1390417763.11 Test: 1505134279.26 Difference: +8.3%
> ReplicatedPutAllBenchmark avg ops/sec Baseline: 788.90 Test: 814.00 Difference: +3.2%
> avg latency Baseline: 182735327.71 Test: 177046841.08 Difference: -3.1%
> ReplicatedPutAllLongBenchmark avg ops/sec Baseline: 832.81 Test: 826.93 Difference: -0.7%
> avg latency Baseline: 173063206.58 Test: 174233069.86 Difference: +0.7%
> ReplicatedPutBenchmark avg ops/sec Baseline: 490054.06 Test: 494297.13 Difference: +0.9%
> avg latency Baseline: 1466451.86 Test: 1454784.46 Difference: -0.8%
> ReplicatedPutLongBenchmark avg ops/sec Baseline: 457689.80 Test: 457973.53 Difference: +0.1%
> avg latency Baseline: 1571708.57 Test: 1569573.85 Difference: -0.1%
> This is ITERATION 2 of benchmarking against baseline.
> PartitionedFunctionExecutionWithFiltersBenchmark avg ops/sec Baseline: 395993.05 Test: 358748.93 Difference: -9.4%
> avg latency Baseline: 1452604.27 Test: 1603625.58 Difference: +10.4%
> PartitionedPutLongBenchmark avg ops/sec Baseline: 422650.17 Test: 415015.59 Difference: -1.8%
> avg latency Baseline: 1702486.44 Test: 1731996.79 Difference: +1.7%
> PartitionedPutStringBenchmark avg ops/sec Baseline: 429296.09 Test: 391046.61 Difference: -8.9%
> avg latency Baseline: 1675896.07 Test: 1837288.98 Difference: +9.6%
> ReplicatedNonIndexedQueryBenchmark avg ops/sec Baseline: 49.02 Test: 50.55 Difference: +3.1%
> avg latency Baseline: 1464289182.72 Test: 1420848686.90 Difference: -3.0%
> This is ITERATION 3 of benchmarking against baseline.
> PartitionedFunctionExecutionWithFiltersBenchmark avg ops/sec Baseline: 407931.35 Test: 434871.25 Difference: +6.6%
> avg latency Baseline: 1410246.93 Test: 1322795.29 Difference: -6.2%
> PartitionedPutStringBenchmark avg ops/sec Baseline: 427102.32 Test: 402755.46 Difference: -5.7%
> avg latency Baseline: 1682556.85 Test: 1786603.33 Difference: +6.2%
> This is ITERATION 4 of benchmarking against baseline.
> PartitionedPutStringBenchmark avg ops/sec Baseline: 434609.00 Test: 402475.52 Difference: -7.4%
> avg latency Baseline: 1656172.72 Test: 1785229.42 Difference: +7.8%
> This is ITERATION 5 of benchmarking against baseline.
> PartitionedPutStringBenchmark avg ops/sec Baseline: 423047.66 Test: 402475.29 Difference: -4.9%
> avg latency Baseline: 1700791.11 Test: 1787257.59 Difference: +5.1%
> {noformat}
> Looking at some previous runs of the benchmarks since this particular benchmark was introduced, it appears that there is some instability in this benchmark:
> {noformat}
> PartitionedPutStringBenchmark
> average ops/second Baseline: 384659.34 Test: 375511.88 Difference: -2.4%
> average latency Baseline: 1868722.73 Test: 1918780.45 Difference: +2.7%
> average ops/second Baseline: 434900.93 Test: 391933.67 Difference: -9.9%
> average latency Baseline: 1656702.07 Test: 1830827.53 Difference: +10.5%
> average ops/second Baseline: 419108.79 Test: 401341.06 Difference: -4.2%
> average latency Baseline: 1712897.05 Test: 1791919.35 Difference: +4.6%
> average ops/second Baseline: 421598.58 Test: 399128.86 Difference: -5.3%
> average latency Baseline: 1707314.91 Test: 1801691.41 Difference: +5.5%
> average ops/second Baseline: 424445.58 Test: 403798.53 Difference: -4.9%
> average latency Baseline: 1694539.23 Test: 1781948.40 Difference: +5.2%
> average ops/second Baseline: 417622.25 Test: 388290.32 Difference: -7.0%
> average latency Baseline: 1721608.57 Test: 1850134.14 Difference: +7.5%
> average ops/second Baseline: 413538.43 Test: 396040.63 Difference: -4.2%
> average latency Baseline: 1738858.16 Test: 1820714.96 Difference: +4.7%
> average ops/second Baseline: 363423.19 Test: 394963.41 Difference: +8.7%
> average latency Baseline: 1981399.62 Test: 1822056.94 Difference: -8.0%
> average ops/second Baseline: 369520.84 Test: 394239.61 Difference: +6.7%
> average latency Baseline: 1948697.59 Test: 1824966.98 Difference: -6.3%
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)