You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "Mark Hanson (Jira)" <ji...@apache.org> on 2019/10/02 18:35:00 UTC
[jira] [Assigned] (GEODE-7269) Benchmarks failure
[ https://issues.apache.org/jira/browse/GEODE-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Hanson reassigned GEODE-7269:
----------------------------------
Assignee: Mark Hanson
> Benchmarks failure
> ------------------
>
> Key: GEODE-7269
> URL: https://issues.apache.org/jira/browse/GEODE-7269
> Project: Geode
> Issue Type: Bug
> Components: benchmarks
> Reporter: Mark Hanson
> Assignee: Mark Hanson
> Priority: Major
>
> Benchmarks failed in [https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/Benchmark/builds/607]
>
> {noformat}
> > Task :harness:analyzeRun
> Running analyzer
> Comparing test result at /tmp/build/2ea77113/results/benchmarks-apache-develop-main-Benchmark-607-98877/branch/benchmarks_20191002113518 to baseline at /tmp/build/2ea77113/results/benchmarks-apache-develop-main-Benchmark-607-98877/baseline/benchmarks_20191002092613
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
> org.apache.geode.benchmark.tests.PartitionedFunctionExecutionWithArgumentsBenchmark
> average ops/second Baseline: 165126.55 Test: 163322.19 Difference: -1.1%
> ops/second standard error Baseline: 481.54 Test: 394.45 Difference: -18.1%
> ops/second standard deviation Baseline: 8326.61 Test: 6832.03 Difference: -17.9%
> YS 99th percentile latency Baseline: 1906.00 Test: 1905.00 Difference: -0.1%
> median latency Baseline: 633855.00 Test: 630271.00 Difference: -0.6%
> 90th percentile latency Baseline: 1326079.00 Test: 1367039.00 Difference: +3.1%
> 99th percentile latency Baseline: 4591615.00 Test: 4644863.00 Difference: +1.2%
> 99.9th percentile latency Baseline: 15359999.00 Test: 16809983.00 Difference: +9.4%
> average latency Baseline: 871151.53 Test: 880810.13 Difference: +1.1%
> latency standard deviation Baseline: 1050482.32 Test: 1100814.13 Difference: +4.8%
> latency standard error Baseline: 149.28 Test: 157.30 Difference: +5.4%
> org.apache.geode.benchmark.tests.PartitionedFunctionExecutionWithFiltersBenchmark
> average ops/second Baseline: 459731.05 Test: 453392.16 Difference: -1.4%
> ops/second standard error Baseline: 394.19 Test: 332.85 Difference: -15.6%
> ops/second standard deviation Baseline: 6816.12 Test: 5765.11 Difference: -15.4%
> YS 99th percentile latency Baseline: 1405.00 Test: 1405.00 Difference: +0.0%
> median latency Baseline: 449791.00 Test: 459519.00 Difference: +2.2%
> 90th percentile latency Baseline: 943615.00 Test: 961023.00 Difference: +1.8%
> 99th percentile latency Baseline: 3616767.00 Test: 3614719.00 Difference: -0.1%
> 99.9th percentile latency Baseline: 10559487.00 Test: 10551295.00 Difference: -0.1%
> average latency Baseline: 625412.03 Test: 634121.49 Difference: +1.4%
> latency standard deviation Baseline: 788269.89 Test: 788318.28 Difference: +0.0%
> latency standard error Baseline: 67.13 Test: 67.60 Difference: +0.7%
> org.apache.geode.benchmark.tests.PartitionedGetBenchmark
> average ops/second Baseline: 841000.72 Test: 828146.58 Difference: -1.5%
> ops/second standard error Baseline: 558.63 Test: 507.87 Difference: -9.1%
> ops/second standard deviation Baseline: 9659.55 Test: 8781.92 Difference: -9.1%
> YS 99th percentile latency Baseline: 806.00 Test: 805.00 Difference: -0.1%
> median latency Baseline: 332287.00 Test: 339967.00 Difference: +2.3%
> 90th percentile latency Baseline: 443391.00 Test: 452351.00 Difference: +2.0%
> 99th percentile latency Baseline: 2672639.00 Test: 2662399.00 Difference: -0.4%
> 99.9th percentile latency Baseline: 8089599.00 Test: 7974911.00 Difference: -1.4%
> average latency Baseline: 426878.72 Test: 433532.06 Difference: +1.6%
> latency standard deviation Baseline: 546763.26 Test: 542115.50 Difference: -0.9%
> latency standard error Baseline: 34.43 Test: 34.40 Difference: -0.1%
> org.apache.geode.benchmark.tests.PartitionedGetLongBenchmark
> average ops/second Baseline: 1108020.75 Test: 1107517.03 Difference: -0.0%
> ops/second standard error Baseline: 508.14 Test: 306.97 Difference: -39.6%
> ops/second standard deviation Baseline: 8801.18 Test: 5307.94 Difference: -39.7%
> YS 99th percentile latency Baseline: 902.00 Test: 902.00 Difference: +0.0%
> median latency Baseline: 276735.00 Test: 276223.00 Difference: -0.2%
> 90th percentile latency Baseline: 412159.00 Test: 412159.00 Difference: +0.0%
> 99th percentile latency Baseline: 1497087.00 Test: 1497087.00 Difference: +0.0%
> 99.9th percentile latency Baseline: 4624383.00 Test: 4833279.00 Difference: +4.5%
> average latency Baseline: 323752.23 Test: 324056.03 Difference: +0.1%
> latency standard deviation Baseline: 325710.03 Test: 337221.72 Difference: +3.5%
> latency standard error Baseline: 17.87 Test: 18.51 Difference: +3.6%
> org.apache.geode.benchmark.tests.PartitionedIndexedQueryBenchmark
> average ops/second Baseline: 18134.76 Test: 18259.64 Difference: +0.7%
> ops/second standard error Baseline: 5.64 Test: 10.68 Difference: +89.4%
> ops/second standard deviation Baseline: 97.49 Test: 184.64 Difference: +89.4%
> YS 99th percentile latency Baseline: 20092.87 Test: 20094.31 Difference: +0.0%
> median latency Baseline: 11075583.00 Test: 13991935.00 Difference: +26.3%
> 90th percentile latency Baseline: 32178175.00 Test: 22986751.00 Difference: -28.6%
> 99th percentile latency Baseline: 85786623.00 Test: 59506687.00 Difference: -30.6%
> 99.9th percentile latency Baseline: 103284735.00 Test: 69271551.00 Difference: -32.9%
> average latency Baseline: 15878192.88 Test: 15776565.53 Difference: -0.6%
> latency standard deviation Baseline: 16509200.47 Test: 10274770.09 Difference: -37.8%
> latency standard error Baseline: 7080.93 Test: 4392.16 Difference: -38.0%
> org.apache.geode.benchmark.tests.PartitionedPutAllBenchmark
> average ops/second Baseline: 660.51 Test: 592.93 Difference: -10.2%
> ops/second standard error Baseline: 3.94 Test: 13.05 Difference: +230.8%
> ops/second standard deviation Baseline: 67.17 Test: 219.13 Difference: +226.2%
> YS 99th percentile latency Baseline: 20099.00 Test: 20099.00 Difference: +0.0%
> median latency Baseline: 65306623.00 Test: 64684031.00 Difference: -1.0%
> 90th percentile latency Baseline: 222298111.00 Test: 221118463.00 Difference: -0.5%
> 99th percentile latency Baseline: 332136447.00 Test: 371982335.00 Difference: +12.0%
> 99.9th percentile latency Baseline: 418381823.00 Test: 8623489023.00 Difference: +1961.2%
> average latency Baseline: 112946954.05 Test: 128847254.66 Difference: +14.1%
> latency standard deviation Baseline: 82763295.05 Test: 388106155.06 Difference: +368.9%
> latency standard error Baseline: 189315.85 Test: 948056.57 Difference: +400.8%
> org.apache.geode.benchmark.tests.PartitionedPutAllLongBenchmark
> average ops/second Baseline: 661.43 Test: 637.88 Difference: -3.6%
> ops/second standard error Baseline: 4.03 Test: 4.12 Difference: +2.2%
> ops/second standard deviation Baseline: 68.83 Test: 70.35 Difference: +2.2%
> YS 99th percentile latency Baseline: 20099.00 Test: 20099.00 Difference: +0.0%
> median latency Baseline: 64290815.00 Test: 65765375.00 Difference: +2.3%
> 90th percentile latency Baseline: 215220223.00 Test: 227934207.00 Difference: +5.9%
> 99th percentile latency Baseline: 333185023.00 Test: 360710143.00 Difference: +8.3%
> 99.9th percentile latency Baseline: 425197567.00 Test: 438042623.00 Difference: +3.0%
> average latency Baseline: 112212752.63 Test: 116032713.49 Difference: +3.4%
> latency standard deviation Baseline: 82581968.17 Test: 86699325.73 Difference: +5.0%
> latency standard error Baseline: 188292.70 Test: 200999.53 Difference: +6.7%
> org.apache.geode.benchmark.tests.PartitionedPutBenchmark
> average ops/second Baseline: 442387.48 Test: 438685.31 Difference: -0.8%
> ops/second standard error Baseline: 1156.06 Test: 1238.08 Difference: +7.1%
> ops/second standard deviation Baseline: 19990.22 Test: 21408.48 Difference: +7.1%
> YS 99th percentile latency Baseline: 1703.00 Test: 1605.00 Difference: -5.8%
> median latency Baseline: 533503.00 Test: 532991.00 Difference: -0.1%
> 90th percentile latency Baseline: 1041407.00 Test: 1031679.00 Difference: -0.9%
> 99th percentile latency Baseline: 3389439.00 Test: 3647487.00 Difference: +7.6%
> 99.9th percentile latency Baseline: 71041023.00 Test: 69730303.00 Difference: -1.8%
> average latency Baseline: 812619.22 Test: 819287.36 Difference: +0.8%
> latency standard deviation Baseline: 3162083.45 Test: 3172446.86 Difference: +0.3%
> latency standard error Baseline: 274.51 Test: 276.55 Difference: +0.7%
> org.apache.geode.benchmark.tests.PartitionedPutLongBenchmark
> average ops/second Baseline: 435576.83 Test: 430206.87 Difference: -1.2%
> ops/second standard error Baseline: 942.64 Test: 1029.65 Difference: +9.2%
> ops/second standard deviation Baseline: 16299.70 Test: 17804.26 Difference: +9.2%
> YS 99th percentile latency Baseline: 1704.00 Test: 1604.00 Difference: -5.9%
> median latency Baseline: 530431.00 Test: 534527.00 Difference: +0.8%
> 90th percentile latency Baseline: 1041919.00 Test: 1027583.00 Difference: -1.4%
> 99th percentile latency Baseline: 3418111.00 Test: 3627007.00 Difference: +6.1%
> 99.9th percentile latency Baseline: 79429631.00 Test: 78839807.00 Difference: -0.7%
> average latency Baseline: 825364.59 Test: 835694.56 Difference: +1.3%
> latency standard deviation Baseline: 3513247.60 Test: 3521391.08 Difference: +0.2%
> latency standard error Baseline: 307.37 Test: 310.02 Difference: +0.9%
> org.apache.geode.benchmark.tests.ReplicatedFunctionExecutionBenchmark
> average ops/second Baseline: 425796.94 Test: 426212.98 Difference: +0.1%
> ops/second standard error Baseline: 203.05 Test: 176.30 Difference: -13.2%
> ops/second standard deviation Baseline: 3511.14 Test: 3048.55 Difference: -13.2%
> YS 99th percentile latency Baseline: 551.00 Test: 550.00 Difference: -0.2%
> median latency Baseline: 237951.00 Test: 238207.00 Difference: +0.1%
> 90th percentile latency Baseline: 316927.00 Test: 319999.00 Difference: +1.0%
> 99th percentile latency Baseline: 486143.00 Test: 474367.00 Difference: -2.4%
> 99.9th percentile latency Baseline: 1098751.00 Test: 778239.00 Difference: -29.2%
> average latency Baseline: 252639.01 Test: 252393.64 Difference: -0.1%
> latency standard deviation Baseline: 148085.78 Test: 140822.70 Difference: -4.9%
> latency standard error Baseline: 13.10 Test: 12.45 Difference: -5.0%
> org.apache.geode.benchmark.tests.ReplicatedFunctionExecutionWithArgumentsBenchmark
> average ops/second Baseline: 602363.99 Test: 618818.13 Difference: +2.7%
> ops/second standard error Baseline: 585.52 Test: 440.33 Difference: -24.8%
> ops/second standard deviation Baseline: 10124.58 Test: 7614.06 Difference: -24.8%
> YS 99th percentile latency Baseline: 1505.00 Test: 1504.00 Difference: -0.1%
> median latency Baseline: 715775.00 Test: 740351.00 Difference: +3.4%
> 90th percentile latency Baseline: 1096703.00 Test: 1108991.00 Difference: +1.1%
> 99th percentile latency Baseline: 9166847.00 Test: 6455295.00 Difference: -29.6%
> 99.9th percentile latency Baseline: 22953983.00 Test: 24051711.00 Difference: +4.8%
> average latency Baseline: 954315.26 Test: 928885.19 Difference: -2.7%
> latency standard deviation Baseline: 1699506.42 Test: 1593965.78 Difference: -6.2%
> latency standard error Baseline: 126.44 Test: 117.00 Difference: -7.5%
> org.apache.geode.benchmark.tests.ReplicatedFunctionExecutionWithFiltersBenchmark
> average ops/second Baseline: 598263.31 Test: 603567.04 Difference: +0.9%
> ops/second standard error Baseline: 436.82 Test: 417.76 Difference: -4.4%
> ops/second standard deviation Baseline: 7553.34 Test: 7223.80 Difference: -4.4%
> YS 99th percentile latency Baseline: 1103.00 Test: 1104.00 Difference: +0.1%
> median latency Baseline: 470271.00 Test: 493311.00 Difference: +4.9%
> 90th percentile latency Baseline: 719871.00 Test: 721919.00 Difference: +0.3%
> 99th percentile latency Baseline: 4100095.00 Test: 3348479.00 Difference: -18.3%
> 99.9th percentile latency Baseline: 13074431.00 Test: 11190271.00 Difference: -14.4%
> average latency Baseline: 600157.39 Test: 594966.11 Difference: -0.9%
> latency standard deviation Baseline: 914725.70 Test: 740635.84 Difference: -19.0%
> latency standard error Baseline: 68.28 Test: 55.05 Difference: -19.4%
> org.apache.geode.benchmark.tests.ReplicatedGetBenchmark
> average ops/second Baseline: 870190.68 Test: 860972.41 Difference: -1.1%
> ops/second standard error Baseline: 541.64 Test: 493.00 Difference: -9.0%
> ops/second standard deviation Baseline: 9365.80 Test: 8524.83 Difference: -9.0%
> YS 99th percentile latency Baseline: 804.00 Test: 805.00 Difference: +0.1%
> median latency Baseline: 324351.00 Test: 324863.00 Difference: +0.2%
> 90th percentile latency Baseline: 427519.00 Test: 425471.00 Difference: -0.5%
> 99th percentile latency Baseline: 2635775.00 Test: 2840575.00 Difference: +7.8%
> 99.9th percentile latency Baseline: 7774207.00 Test: 8126463.00 Difference: +4.5%
> average latency Baseline: 412569.73 Test: 416977.59 Difference: +1.1%
> latency standard deviation Baseline: 523805.97 Test: 556261.72 Difference: +6.2%
> latency standard error Baseline: 32.42 Test: 34.62 Difference: +6.8%
> org.apache.geode.benchmark.tests.ReplicatedGetLongBenchmark
> average ops/second Baseline: 1115957.04 Test: 1121417.41 Difference: +0.5%
> ops/second standard error Baseline: 502.85 Test: 564.03 Difference: +12.2%
> ops/second standard deviation Baseline: 8695.02 Test: 9752.91 Difference: +12.2%
> YS 99th percentile latency Baseline: 902.00 Test: 901.00 Difference: -0.1%
> median latency Baseline: 272895.00 Test: 266239.00 Difference: -2.4%
> 90th percentile latency Baseline: 412671.00 Test: 411647.00 Difference: -0.2%
> 99th percentile latency Baseline: 1494015.00 Test: 1508351.00 Difference: +1.0%
> 99.9th percentile latency Baseline: 4435967.00 Test: 4808703.00 Difference: +8.4%
> average latency Baseline: 321439.30 Test: 319877.94 Difference: -0.5%
> latency standard deviation Baseline: 325471.93 Test: 348138.79 Difference: +7.0%
> latency standard error Baseline: 17.79 Test: 18.98 Difference: +6.7%
> org.apache.geode.benchmark.tests.ReplicatedIndexedQueryBenchmark
> average ops/second Baseline: 18228.89 Test: 18244.84 Difference: +0.1%
> ops/second standard error Baseline: 7.23 Test: 10.20 Difference: +41.2%
> ops/second standard deviation Baseline: 125.14 Test: 176.40 Difference: +41.0%
> YS 99th percentile latency Baseline: 20093.46 Test: 20093.42 Difference: -0.0%
> median latency Baseline: 14000127.00 Test: 14303231.00 Difference: +2.2%
> 90th percentile latency Baseline: 23330815.00 Test: 22151167.00 Difference: -5.1%
> 99th percentile latency Baseline: 59703295.00 Test: 55902207.00 Difference: -6.4%
> 99.9th percentile latency Baseline: 69468159.00 Test: 65863679.00 Difference: -5.2%
> average latency Baseline: 15793710.81 Test: 15788088.06 Difference: -0.0%
> latency standard deviation Baseline: 10392676.29 Test: 9415504.26 Difference: -9.4%
> latency standard error Baseline: 4445.10 Test: 4026.08 Difference: -9.4%
> org.apache.geode.benchmark.tests.ReplicatedPutAllBenchmark
> average ops/second Baseline: 669.13 Test: 670.15 Difference: +0.2%
> ops/second standard error Baseline: 4.77 Test: 4.52 Difference: -5.3%
> ops/second standard deviation Baseline: 81.85 Test: 77.50 Difference: -5.3%
> YS 99th percentile latency Baseline: 20099.00 Test: 20099.00 Difference: +0.0%
> median latency Baseline: 79429631.00 Test: 77070335.00 Difference: -3.0%
> 90th percentile latency Baseline: 217055231.00 Test: 217448447.00 Difference: +0.2%
> 99th percentile latency Baseline: 347865087.00 Test: 356515839.00 Difference: +2.5%
> 99.9th percentile latency Baseline: 405012479.00 Test: 445644799.00 Difference: +10.0%
> average latency Baseline: 109546138.69 Test: 109117201.07 Difference: -0.4%
> latency standard deviation Baseline: 75949075.20 Test: 77910756.92 Difference: +2.6%
> latency standard error Baseline: 171109.43 Test: 175165.73 Difference: +2.4%
> org.apache.geode.benchmark.tests.ReplicatedPutAllLongBenchmark
> average ops/second Baseline: 673.75 Test: 681.35 Difference: +1.1%
> ops/second standard error Baseline: 4.59 Test: 4.81 Difference: +4.7%
> ops/second standard deviation Baseline: 78.88 Test: 82.76 Difference: +4.9%
> YS 99th percentile latency Baseline: 20099.00 Test: 20099.00 Difference: +0.0%
> median latency Baseline: 73334783.00 Test: 74448895.00 Difference: +1.5%
> 90th percentile latency Baseline: 221380607.00 Test: 215089151.00 Difference: -2.8%
> 99th percentile latency Baseline: 372244479.00 Test: 355991551.00 Difference: -4.4%
> 99.9th percentile latency Baseline: 434372607.00 Test: 477626367.00 Difference: +10.0%
> average latency Baseline: 108819632.70 Test: 107443738.57 Difference: -1.3%
> latency standard deviation Baseline: 80782678.15 Test: 78388444.16 Difference: -3.0%
> latency standard error Baseline: 181377.55 Test: 174863.16 Difference: -3.6%
> org.apache.geode.benchmark.tests.ReplicatedPutBenchmark
> average ops/second Baseline: 453381.24 Test: 453594.38 Difference: +0.0%
> ops/second standard error Baseline: 655.88 Test: 608.38 Difference: -7.2%
> ops/second standard deviation Baseline: 11341.21 Test: 10519.91 Difference: -7.2%
> YS 99th percentile latency Baseline: 1702.00 Test: 1702.00 Difference: +0.0%
> median latency Baseline: 525823.00 Test: 524031.00 Difference: -0.3%
> 90th percentile latency Baseline: 1099775.00 Test: 1114111.00 Difference: +1.3%
> 99th percentile latency Baseline: 2881535.00 Test: 2961407.00 Difference: +2.8%
> 99.9th percentile latency Baseline: 77266943.00 Test: 73203711.00 Difference: -5.3%
> average latency Baseline: 792895.77 Test: 792565.55 Difference: -0.0%
> latency standard deviation Baseline: 3174758.31 Test: 3045130.83 Difference: -4.1%
> latency standard error Baseline: 272.24 Test: 261.07 Difference: -4.1%
> org.apache.geode.benchmark.tests.ReplicatedPutLongBenchmark
> average ops/second Baseline: 440979.16 Test: 444911.43 Difference: +0.9%
> ops/second standard error Baseline: 1076.73 Test: 724.78 Difference: -32.7%
> ops/second standard deviation Baseline: 18618.44 Test: 12532.59 Difference: -32.7%
> YS 99th percentile latency Baseline: 1702.00 Test: 1702.00 Difference: +0.0%
> median latency Baseline: 522751.00 Test: 525823.00 Difference: +0.6%
> 90th percentile latency Baseline: 1124351.00 Test: 1111039.00 Difference: -1.2%
> 99th percentile latency Baseline: 3012607.00 Test: 2938879.00 Difference: -2.4%
> 99.9th percentile latency Baseline: 85065727.00 Test: 80347135.00 Difference: -5.5%
> average latency Baseline: 815307.00 Test: 807777.30 Difference: -0.9%
> latency standard deviation Baseline: 3621053.66 Test: 3505761.25 Difference: -3.2%
> latency standard error Baseline: 314.86 Test: 303.47 Difference: -3.6%
> BENCHMARK FAILED: org.apache.geode.benchmark.tests.PartitionedPutAllBenchmark average latency is 5% worse than baseline. {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)