You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@aurora.apache.org by Bill Farner <wf...@apache.org> on 2015/04/03 03:43:36 UTC
Review Request 32802: Add a benchmark for getRoleSummary.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32802/
-----------------------------------------------------------
Review request for Aurora and Maxim Khutornenko.
Repository: aurora
Description
-------
The first of several benchmarks to exercise the thrift API. The real motivation here is to have performance data to assess the impact of migrating the task store to H2. I'm targeting the `getRoleSummary` RPC specifically here as i consider it the likely candidate to suffer.
One 'trick' i employed here was JSON for input parameters. I took this approach since i really wanted to explore specific axes of growth without the full outer product of each individual parameters.
Some changes along the way:
- upgraded to jmh 1.7.1
- moved some JMH run configuration parameters out of the gradle config and into benchmark classes. This allows us to define run configurations for each microbenchmark
- changed the benchmark mode for existing and new tests to throughput. We had several benchmarks whose iteration time was measuring in hundreds of microseconds, which is too high resolution to capture performance variation. By running iterations for a set amount of time and counting iterations accomplished, we can capture more reliable data.
Diffs
-----
build.gradle 66dbdfde830c81c9b2291d2b8391f1ccac94b485
src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 5309e8140fff411da30ee87c1b3b1a55d6fdaeeb
src/jmh/java/org/apache/aurora/benchmark/Tasks.java 491c6871ba7a4c76a4eb084afb2627a3f93db8df
src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/thrift/ThriftModule.java c65e0218211b797ad50e2ac62e71136f967e5f5d
Diff: https://reviews.apache.org/r/32802/diff/
Testing
-------
Will post jmh output shortly.
Thanks,
Bill Farner
Re: Review Request 32802: Add a benchmark for getRoleSummary.
Posted by Bill Farner <wf...@apache.org>.
> On April 3, 2015, 6:03 p.m., Maxim Khutornenko wrote:
> > src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java, line 125
> > <https://reviews.apache.org/r/32802/diff/1/?file=914385#file914385line125>
> >
> > Perhaps just hardcode this in Tasks.Builder() instead and drop the builder method? I don't see why we would ever want to change this value.
The problem is that it can't be hard-coded, since the underlying builder has no state. In this case, the tasks added on :114 will have colliding IDs with the ones added on :120, so some adjustment is necessary to augment them.
- Bill
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32802/#review78800
-----------------------------------------------------------
On April 3, 2015, 2:19 a.m., Bill Farner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32802/
> -----------------------------------------------------------
>
> (Updated April 3, 2015, 2:19 a.m.)
>
>
> Review request for Aurora and Maxim Khutornenko.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> The first of several benchmarks to exercise the thrift API. The real motivation here is to have performance data to assess the impact of migrating the task store to H2. I'm targeting the `getRoleSummary` RPC specifically here as i consider it the likely candidate to suffer.
>
> One 'trick' i employed here was JSON for input parameters. I took this approach since i really wanted to explore specific axes of growth without the full outer product of each individual parameters.
>
> Some changes along the way:
> - upgraded to jmh 1.7.1
> - moved some JMH run configuration parameters out of the gradle config and into benchmark classes. This allows us to define run configurations for each microbenchmark
> - changed the benchmark mode for existing and new tests to throughput. We had several benchmarks whose iteration time was measuring in hundreds of microseconds, which is too high resolution to capture performance variation. By running iterations for a set amount of time and counting iterations accomplished, we can capture more reliable data.
>
>
> Diffs
> -----
>
> build.gradle 66dbdfde830c81c9b2291d2b8391f1ccac94b485
> src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 5309e8140fff411da30ee87c1b3b1a55d6fdaeeb
> src/jmh/java/org/apache/aurora/benchmark/Tasks.java 491c6871ba7a4c76a4eb084afb2627a3f93db8df
> src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/thrift/ThriftModule.java c65e0218211b797ad50e2ac62e71136f967e5f5d
>
> Diff: https://reviews.apache.org/r/32802/diff/
>
>
> Testing
> -------
>
> FYI numbers and stddev are not useful in the data captured below, as my machine was doing other intensive things while running this. Output is more useful as an example of how the data and run configurations are presented.
>
> ```
> # Run complete. Total time: 00:18:51
>
> Benchmark (testConfiguration) Mode Cnt Score Error Units
> SchedulingBenchmarks.ClusterFullUtilizationBenchmark.runBenchmark N/A thrpt 10 499824.589 ± 86328.152 ops/s
> SchedulingBenchmarks.InsufficientResourcesSchedulingBenchmark.runBenchmark N/A thrpt 10 40482.762 ± 5394.306 ops/s
> SchedulingBenchmarks.LimitConstraintMismatchSchedulingBenchmark.runBenchmark N/A thrpt 10 2933.958 ± 577.571 ops/s
> SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark N/A thrpt 10 220.079 ± 28.943 ops/s
> SchedulingBenchmarks.ValueConstraintMismatchSchedulingBenchmark.runBenchmark N/A thrpt 10 41948.230 ± 1906.106 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"roles": 1} thrpt 5 1978.614 ± 736.704 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"roles": 10} thrpt 5 143.465 ± 85.579 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"roles": 100} thrpt 5 4.003 ± 2.742 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"roles": 500} thrpt 5 1.964 ± 0.569 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"jobs": 1} thrpt 5 4073.564 ± 3343.602 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"jobs": 10} thrpt 5 4401.662 ± 3943.372 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"jobs": 100} thrpt 5 4327.726 ± 726.900 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"jobs": 500} thrpt 5 2473.799 ± 5396.188 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"instances": 1} thrpt 5 7351.543 ± 13368.279 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"instances": 10} thrpt 5 9496.478 ± 3066.478 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"instances": 100} thrpt 5 4640.480 ± 3307.831 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"instances": 1000} thrpt 5 429.841 ± 191.521 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"instances": 10000} thrpt 5 23.327 ± 7.011 ops/s
> ```
>
>
> Thanks,
>
> Bill Farner
>
>
Re: Review Request 32802: Add a benchmark for getRoleSummary.
Posted by Maxim Khutornenko <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32802/#review78800
-----------------------------------------------------------
Ship it!
src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java
<https://reviews.apache.org/r/32802/#comment127824>
Nice! I could not make this work through annotations in earlier versions of jmh/gradle plugin.
src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java
<https://reviews.apache.org/r/32802/#comment127844>
Perhaps just hardcode this in Tasks.Builder() instead and drop the builder method? I don't see why we would ever want to change this value.
- Maxim Khutornenko
On April 3, 2015, 2:19 a.m., Bill Farner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32802/
> -----------------------------------------------------------
>
> (Updated April 3, 2015, 2:19 a.m.)
>
>
> Review request for Aurora and Maxim Khutornenko.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> The first of several benchmarks to exercise the thrift API. The real motivation here is to have performance data to assess the impact of migrating the task store to H2. I'm targeting the `getRoleSummary` RPC specifically here as i consider it the likely candidate to suffer.
>
> One 'trick' i employed here was JSON for input parameters. I took this approach since i really wanted to explore specific axes of growth without the full outer product of each individual parameters.
>
> Some changes along the way:
> - upgraded to jmh 1.7.1
> - moved some JMH run configuration parameters out of the gradle config and into benchmark classes. This allows us to define run configurations for each microbenchmark
> - changed the benchmark mode for existing and new tests to throughput. We had several benchmarks whose iteration time was measuring in hundreds of microseconds, which is too high resolution to capture performance variation. By running iterations for a set amount of time and counting iterations accomplished, we can capture more reliable data.
>
>
> Diffs
> -----
>
> build.gradle 66dbdfde830c81c9b2291d2b8391f1ccac94b485
> src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 5309e8140fff411da30ee87c1b3b1a55d6fdaeeb
> src/jmh/java/org/apache/aurora/benchmark/Tasks.java 491c6871ba7a4c76a4eb084afb2627a3f93db8df
> src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/thrift/ThriftModule.java c65e0218211b797ad50e2ac62e71136f967e5f5d
>
> Diff: https://reviews.apache.org/r/32802/diff/
>
>
> Testing
> -------
>
> FYI numbers and stddev are not useful in the data captured below, as my machine was doing other intensive things while running this. Output is more useful as an example of how the data and run configurations are presented.
>
> ```
> # Run complete. Total time: 00:18:51
>
> Benchmark (testConfiguration) Mode Cnt Score Error Units
> SchedulingBenchmarks.ClusterFullUtilizationBenchmark.runBenchmark N/A thrpt 10 499824.589 ± 86328.152 ops/s
> SchedulingBenchmarks.InsufficientResourcesSchedulingBenchmark.runBenchmark N/A thrpt 10 40482.762 ± 5394.306 ops/s
> SchedulingBenchmarks.LimitConstraintMismatchSchedulingBenchmark.runBenchmark N/A thrpt 10 2933.958 ± 577.571 ops/s
> SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark N/A thrpt 10 220.079 ± 28.943 ops/s
> SchedulingBenchmarks.ValueConstraintMismatchSchedulingBenchmark.runBenchmark N/A thrpt 10 41948.230 ± 1906.106 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"roles": 1} thrpt 5 1978.614 ± 736.704 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"roles": 10} thrpt 5 143.465 ± 85.579 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"roles": 100} thrpt 5 4.003 ± 2.742 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"roles": 500} thrpt 5 1.964 ± 0.569 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"jobs": 1} thrpt 5 4073.564 ± 3343.602 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"jobs": 10} thrpt 5 4401.662 ± 3943.372 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"jobs": 100} thrpt 5 4327.726 ± 726.900 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"jobs": 500} thrpt 5 2473.799 ± 5396.188 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"instances": 1} thrpt 5 7351.543 ± 13368.279 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"instances": 10} thrpt 5 9496.478 ± 3066.478 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"instances": 100} thrpt 5 4640.480 ± 3307.831 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"instances": 1000} thrpt 5 429.841 ± 191.521 ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"instances": 10000} thrpt 5 23.327 ± 7.011 ops/s
> ```
>
>
> Thanks,
>
> Bill Farner
>
>
Re: Review Request 32802: Add a benchmark for getRoleSummary.
Posted by Bill Farner <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32802/
-----------------------------------------------------------
(Updated April 3, 2015, 2:19 a.m.)
Review request for Aurora and Maxim Khutornenko.
Repository: aurora
Description
-------
The first of several benchmarks to exercise the thrift API. The real motivation here is to have performance data to assess the impact of migrating the task store to H2. I'm targeting the `getRoleSummary` RPC specifically here as i consider it the likely candidate to suffer.
One 'trick' i employed here was JSON for input parameters. I took this approach since i really wanted to explore specific axes of growth without the full outer product of each individual parameters.
Some changes along the way:
- upgraded to jmh 1.7.1
- moved some JMH run configuration parameters out of the gradle config and into benchmark classes. This allows us to define run configurations for each microbenchmark
- changed the benchmark mode for existing and new tests to throughput. We had several benchmarks whose iteration time was measuring in hundreds of microseconds, which is too high resolution to capture performance variation. By running iterations for a set amount of time and counting iterations accomplished, we can capture more reliable data.
Diffs
-----
build.gradle 66dbdfde830c81c9b2291d2b8391f1ccac94b485
src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 5309e8140fff411da30ee87c1b3b1a55d6fdaeeb
src/jmh/java/org/apache/aurora/benchmark/Tasks.java 491c6871ba7a4c76a4eb084afb2627a3f93db8df
src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/thrift/ThriftModule.java c65e0218211b797ad50e2ac62e71136f967e5f5d
Diff: https://reviews.apache.org/r/32802/diff/
Testing (updated)
-------
FYI numbers and stddev are not useful in the data captured below, as my machine was doing other intensive things while running this. Output is more useful as an example of how the data and run configurations are presented.
```
# Run complete. Total time: 00:18:51
Benchmark (testConfiguration) Mode Cnt Score Error Units
SchedulingBenchmarks.ClusterFullUtilizationBenchmark.runBenchmark N/A thrpt 10 499824.589 ± 86328.152 ops/s
SchedulingBenchmarks.InsufficientResourcesSchedulingBenchmark.runBenchmark N/A thrpt 10 40482.762 ± 5394.306 ops/s
SchedulingBenchmarks.LimitConstraintMismatchSchedulingBenchmark.runBenchmark N/A thrpt 10 2933.958 ± 577.571 ops/s
SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark N/A thrpt 10 220.079 ± 28.943 ops/s
SchedulingBenchmarks.ValueConstraintMismatchSchedulingBenchmark.runBenchmark N/A thrpt 10 41948.230 ± 1906.106 ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"roles": 1} thrpt 5 1978.614 ± 736.704 ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"roles": 10} thrpt 5 143.465 ± 85.579 ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"roles": 100} thrpt 5 4.003 ± 2.742 ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"roles": 500} thrpt 5 1.964 ± 0.569 ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"jobs": 1} thrpt 5 4073.564 ± 3343.602 ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"jobs": 10} thrpt 5 4401.662 ± 3943.372 ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"jobs": 100} thrpt 5 4327.726 ± 726.900 ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"jobs": 500} thrpt 5 2473.799 ± 5396.188 ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"instances": 1} thrpt 5 7351.543 ± 13368.279 ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"instances": 10} thrpt 5 9496.478 ± 3066.478 ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"instances": 100} thrpt 5 4640.480 ± 3307.831 ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"instances": 1000} thrpt 5 429.841 ± 191.521 ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run {"instances": 10000} thrpt 5 23.327 ± 7.011 ops/s
```
Thanks,
Bill Farner
Re: Review Request 32802: Add a benchmark for getRoleSummary.
Posted by Aurora ReviewBot <wf...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32802/#review78756
-----------------------------------------------------------
This patch does not apply cleanly on master (a7b95d9), do you need to rebase?
I will refresh this build result if you post a review containing "@ReviewBot retry"
- Aurora ReviewBot
On April 3, 2015, 1:43 a.m., Bill Farner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32802/
> -----------------------------------------------------------
>
> (Updated April 3, 2015, 1:43 a.m.)
>
>
> Review request for Aurora and Maxim Khutornenko.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> The first of several benchmarks to exercise the thrift API. The real motivation here is to have performance data to assess the impact of migrating the task store to H2. I'm targeting the `getRoleSummary` RPC specifically here as i consider it the likely candidate to suffer.
>
> One 'trick' i employed here was JSON for input parameters. I took this approach since i really wanted to explore specific axes of growth without the full outer product of each individual parameters.
>
> Some changes along the way:
> - upgraded to jmh 1.7.1
> - moved some JMH run configuration parameters out of the gradle config and into benchmark classes. This allows us to define run configurations for each microbenchmark
> - changed the benchmark mode for existing and new tests to throughput. We had several benchmarks whose iteration time was measuring in hundreds of microseconds, which is too high resolution to capture performance variation. By running iterations for a set amount of time and counting iterations accomplished, we can capture more reliable data.
>
>
> Diffs
> -----
>
> build.gradle 66dbdfde830c81c9b2291d2b8391f1ccac94b485
> src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 5309e8140fff411da30ee87c1b3b1a55d6fdaeeb
> src/jmh/java/org/apache/aurora/benchmark/Tasks.java 491c6871ba7a4c76a4eb084afb2627a3f93db8df
> src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/thrift/ThriftModule.java c65e0218211b797ad50e2ac62e71136f967e5f5d
>
> Diff: https://reviews.apache.org/r/32802/diff/
>
>
> Testing
> -------
>
> Will post jmh output shortly.
>
>
> Thanks,
>
> Bill Farner
>
>