[jira] [Commented] (AVRO-2269) Improve usability of Perf.java

["Raymie Stata (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AVRO-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16698096#comment-16698096 ] 

Raymie Stata commented on AVRO-2269:
------------------------------------

This work has changed direction.  The focus shifted away from variance and towards usability.  I've updated the subject and description accordingly.

> Improve usability of Perf.java
> ------------------------------
>
>                 Key: AVRO-2269
>                 URL: https://issues.apache.org/jira/browse/AVRO-2269
>             Project: Apache Avro
>          Issue Type: Test
>          Components: java
>            Reporter: Raymie Stata
>            Assignee: Raymie Stata
>            Priority: Major
>
> In attempting to use Perf.java to show that proposed performance changes actually improved performance, different runs of Perf.java using the exact same code base resulted variances of 5% or greater – and often 10% or greater – for about half the test cases. With variance this high within a code base, it's impossible to tell if a proposed "improved" code base indeed improves performance. I will post to the wiki and elsewhere some documents and scripts I developed to reduce this variance. This JIRA is for changes to Perf.java that reduce the variance. Specifically:
>  * Access the {{reader}} and {{writer}} instance variables directly in the inner-loop for {{SpecificTest}}, as well as switched to a "reuse" object for reading records, rather than constructing fresh objects for each read. Both helped to significantly reduce variance for {{FooBarSpecificRecordTestWrite}}, a major target of recent performance-improvement efforts.
>  * Switched to {{DirectBinaryEncoder}} instead of {{BufferedBinaryEncoder}} for write tests. Although this slowed writer-tests a bit, it reduced variance a lot, especially for performance tests of primitives like booleans, making it a better choice for measuring the performance-impact of code changes.
>  * Started the timer of a test after the encoder/decoder for the test is constructed, rather than before. Helps a little.
>  * Added the ability to output the _minimum_ runtime of a test case across multiple cycles (vs the total runtime across all cycles). This was inspired by JVMSpec, which used to use a minimum.  I was able to reduce the variance of total runtime enough to obviate the need for this metric, but since it's helpful diagnostically, I left it in.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)