[jira] [Commented] (AVRO-2269) Improve variances seen across Perf.java runs

["Zoltan Farkas (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AVRO-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16688925#comment-16688925 ] 

Zoltan Farkas commented on AVRO-2269:
-------------------------------------

instead or re-inventing the wheel...Perf.java should be rewritten to use JMH (https://java-performance.info/jmh/) ..
see https://github.com/zolyfarkas/benchmarks for an example example on how to use JMH
(there is also a benchmark for one of my experiments:
https://github.com/zolyfarkas/benchmarks/blob/master/src/test/java/org/spf4j/avro/GenericRecordBenchmark.java)
the example shows how you can run them with a profiler Java flight recorder, stack sampler, so that you can even have some data to look at and optimize...


> Improve variances seen across Perf.java runs
> --------------------------------------------
>
>                 Key: AVRO-2269
>                 URL: https://issues.apache.org/jira/browse/AVRO-2269
>             Project: Apache Avro
>          Issue Type: Test
>          Components: java
>            Reporter: Raymie Stata
>            Assignee: Raymie Stata
>            Priority: Major
>
> In attempting to use Perf.java to show that proposed performance changes actually improved performance, different runs of Perf.java using the exact same code base resulted variances of 5% or greater – and often 10% or greater – for about half the test cases. With variance this high within a code base, it's impossible to tell if a proposed "improved" code base indeed improves performance. I will post to the wiki and elsewhere some documents and scripts I developed to reduce this variance. This JIRA is for changes to Perf.java that reduce the variance. Specifically:
>  * Access the {{reader}} and {{writer}} instance variables directly in the inner-loop for {{SpecificTest}}, as well as switched to a "reuse" object for reading records, rather than constructing fresh objects for each read. Both helped to significantly reduce variance for {{FooBarSpecificRecordTestWrite}}, a major target of recent performance-improvement efforts.
>  * Switched to {{DirectBinaryEncoder}} instead of {{BufferedBinaryEncoder}} for write tests. Although this slowed writer-tests a bit, it reduced variance a lot, especially for performance tests of primitives like booleans, making it a better choice for measuring the performance-impact of code changes.
>  * Started the timer of a test after the encoder/decoder for the test is constructed, rather than before. Helps a little.
>  * Added the ability to output the _minimum_ runtime of a test case across multiple cycles (vs the total runtime across all cycles). This was inspired by JVMSpec, which used to use a minimum.  I was able to reduce the variance of total runtime enough to obviate the need for this metric, but since it's helpful diagnostically, I left it in.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)