You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Raymie Stata (JIRA)" <ji...@apache.org> on 2018/11/16 02:23:00 UTC

[jira] [Created] (AVRO-2269) Improve variances seen across Perf.java runs

Raymie Stata created AVRO-2269:
----------------------------------

             Summary: Improve variances seen across Perf.java runs
                 Key: AVRO-2269
                 URL: https://issues.apache.org/jira/browse/AVRO-2269
             Project: Apache Avro
          Issue Type: Test
          Components: java
            Reporter: Raymie Stata
            Assignee: Raymie Stata


In attempting to use Perf.java to show that proposed performance changes actually improved performance, different runs of Perf.java using the exact same code base resulted variances of 5% or greater – and often 10% or greater – for about half the test cases. With variance this high within a code base, it's impossible to tell if a proposed "improved" code base indeed improves performance. I will post to the wiki and elsewhere some documents and scripts I developed to reduce this variance. This JIRA is for changes to Perf.java that reduce the variance. Specifically:
 * Access the {{reader}} and {{writer}} instance variables directly in the inner-loop for {{SpecificTest}}, as well as switched to a "reuse" object for reading records, rather than constructing fresh objects for each read. Both helped to significantly reduce variance for {{FooBarSpecificRecordTestWrite}}, a major target of recent performance-improvement efforts.

 * Switched to {{DirectBinaryEncoder}} instead of {{BufferedBinaryEncoder}} for write tests. Although this slowed writer-tests a bit, it reduced variance a lot, especially for performance tests of primitives like booleans, making it a better choice for measuring the performance-impact of code changes.

 * Started the timer of a test after the encoder/decoder for the test is constructed, rather than before. Helps a little.

 * Added the ability to output the _minimum_ runtime of a test case across multiple cycles (vs the total runtime across all cycles). This was inspired by JVMSpec, which used to use a minimum.  I was able to reduce the variance of total runtime enough to obviate the need for this metric, but since it's helpful diagnostically, I left it in.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)