[jira] [Created] (ARROW-9239) [C++][Developer] Implement tool to compile benchmark C++ code from master branch against older codebase revisions

["Wes McKinney (Jira)" <ji...@apache.org>]

Wes McKinney created ARROW-9239:
-----------------------------------

             Summary: [C++][Developer] Implement tool to compile benchmark C++ code from master branch against older codebase revisions
                 Key: ARROW-9239
                 URL: https://issues.apache.org/jira/browse/ARROW-9239
             Project: Apache Arrow
          Issue Type: New Feature
          Components: C++, Developer Tools
            Reporter: Wes McKinney


Whenever a benchmark configuration is changed, the results from that benchmark become non-comparable with prior revisions of the benchmark before the change. This impacts "archery benchmark diff" and any other workflow that prepares benchmark comparisons

I suggest the creation of a tool that 

* Backports -benchmark.cc changes along with known dependent files (like stuff in arrow/testing) to a branch based on a target commit to benchmark
* Compile as many of the benchmarks as possible (some may not compile due to API changes)
* Run benchmarks and collect their data

I've already done this manually in a number of cases to compare performance in PRs where there are changes to the -benchmark.cc files, but it would be better to have a more structured tool that simplifies this process



--
This message was sent by Atlassian Jira
(v8.3.4#803005)