You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@aurora.apache.org by Bill Farner <wf...@apache.org> on 2015/06/19 00:40:01 UTC
Review Request 35630: DbTaskStore perf: add a task store API to list
task job keys.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35630/
-----------------------------------------------------------
Review request for Aurora and Maxim Khutornenko.
Bugs: AURORA-1298
https://issues.apache.org/jira/browse/AURORA-1298
Repository: aurora
Description
-------
DbTaskStore perf: add a task store API to list task job keys.
Diffs
-----
src/main/java/org/apache/aurora/scheduler/storage/ForwardingStore.java 5242eba2e1a5b35ba3d321f96e2c6670afed4c11
src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 2768e6eafa51f6df78073715a7caf10bb7adba25
src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 76f65dab5a5cb8d97ada962a06a7d0e191739119
src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java 9903675c9d5ee34aebbc52c85fbf02929422a3bf
src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 3a9de60d7e743634815b83587bd61814d883bf86
src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 30e579c46168b64bdc4ba2416d49cf34b4e5c5f6
src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml c3ab3eb9bd6289f73db13e7a7978b5d47621bb52
src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 0f3a1a0ffde16f629422caf868fadb59be70e418
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java a854ed3d99af08fb46e436eb1bc00d8fb312278e
Diff: https://reviews.apache.org/r/35630/diff/
Testing
-------
GetRoleSummaryBenchmark results using MemTaskStore:
```
{"roles": 1} 6234.156 ± 41.303
{"roles": 10} 448.273 ± 130.781
{"roles": 100} 13.768 ± 2.186
{"roles": 500} 2.017 ± 3.046
{"jobs": 1} 7252.866 ± 295.430
{"jobs": 10} 6034.267 ± 256.936
{"jobs": 100} 6081.036 ± 272.341
{"jobs": 500} 6108.709 ± 59.118
{"instances": 1} 14591.270 ± 2589.012
{"instances": 10} 11790.114 ± 1595.596
{"instances": 100} 6332.318 ± 177.585
{"instances": 1000} 1078.874 ± 25.261
{"instances": 10000} 23.874 ± 0.911
```
Results using DbTaskStore with this patch:
```
{"roles": 1} 450002.057 ± 54941.670
{"roles": 10} 111124.857 ± 5335.829
{"roles": 100} 12377.483 ± 800.637
{"roles": 500} 2388.166 ± 101.003
{"jobs": 1} 431646.943 ± 34172.295
{"jobs": 10} 424383.191 ± 19939.399
{"jobs": 100} 417483.093 ± 16342.521
{"jobs": 500} 392792.143 ± 27064.583
{"instances": 1} 454656.616 ± 21980.375
{"instances": 10} 460502.356 ± 19182.511
{"instances": 100} 444265.855 ± 28006.036
{"instances": 1000} 419068.664 ± 49878.394
{"instances": 10000} 404739.919 ± 13857.501
```
Seems like we're able to take pretty good advantage of caching in myBatis and/or H2, at which point performance mostly varies with the response size.
Thanks,
Bill Farner
Re: Review Request 35630: DbTaskStore perf: add a task store API to
list task job keys.
Posted by Maxim Khutornenko <ma...@apache.org>.
> On June 18, 2015, 11:08 p.m., Maxim Khutornenko wrote:
> > src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml, line 167
> > <https://reviews.apache.org/r/35630/diff/1/?file=987636#file987636line167>
> >
> > Is DISTINCT necessary given the Set result type?
>
> Bill Farner wrote:
> Functionally it's not necessary, but why not dedupe as low in the stack as we can? If the database is on a separate host, for example, we save on the network.
There is already a UNIQUE constraint on the job_keys table, so DISTINCT is not really necessary anyway.
- Maxim
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35630/#review88452
-----------------------------------------------------------
On June 18, 2015, 10:39 p.m., Bill Farner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35630/
> -----------------------------------------------------------
>
> (Updated June 18, 2015, 10:39 p.m.)
>
>
> Review request for Aurora and Maxim Khutornenko.
>
>
> Bugs: AURORA-1298
> https://issues.apache.org/jira/browse/AURORA-1298
>
>
> Repository: aurora
>
>
> Description
> -------
>
> DbTaskStore perf: add a task store API to list task job keys.
>
>
> Diffs
> -----
>
> src/main/java/org/apache/aurora/scheduler/storage/ForwardingStore.java 5242eba2e1a5b35ba3d321f96e2c6670afed4c11
> src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 2768e6eafa51f6df78073715a7caf10bb7adba25
> src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 76f65dab5a5cb8d97ada962a06a7d0e191739119
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java 9903675c9d5ee34aebbc52c85fbf02929422a3bf
> src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 3a9de60d7e743634815b83587bd61814d883bf86
> src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 30e579c46168b64bdc4ba2416d49cf34b4e5c5f6
> src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml c3ab3eb9bd6289f73db13e7a7978b5d47621bb52
> src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 0f3a1a0ffde16f629422caf868fadb59be70e418
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java a854ed3d99af08fb46e436eb1bc00d8fb312278e
>
> Diff: https://reviews.apache.org/r/35630/diff/
>
>
> Testing
> -------
>
> GetRoleSummaryBenchmark results using MemTaskStore:
> ```
> {"roles": 1} 6234.156 ± 41.303
> {"roles": 10} 448.273 ± 130.781
> {"roles": 100} 13.768 ± 2.186
> {"roles": 500} 2.017 ± 3.046
> {"jobs": 1} 7252.866 ± 295.430
> {"jobs": 10} 6034.267 ± 256.936
> {"jobs": 100} 6081.036 ± 272.341
> {"jobs": 500} 6108.709 ± 59.118
> {"instances": 1} 14591.270 ± 2589.012
> {"instances": 10} 11790.114 ± 1595.596
> {"instances": 100} 6332.318 ± 177.585
> {"instances": 1000} 1078.874 ± 25.261
> {"instances": 10000} 23.874 ± 0.911
> ```
>
> Results using DbTaskStore with this patch:
> ```
> {"roles": 1} 450002.057 ± 54941.670
> {"roles": 10} 111124.857 ± 5335.829
> {"roles": 100} 12377.483 ± 800.637
> {"roles": 500} 2388.166 ± 101.003
> {"jobs": 1} 431646.943 ± 34172.295
> {"jobs": 10} 424383.191 ± 19939.399
> {"jobs": 100} 417483.093 ± 16342.521
> {"jobs": 500} 392792.143 ± 27064.583
> {"instances": 1} 454656.616 ± 21980.375
> {"instances": 10} 460502.356 ± 19182.511
> {"instances": 100} 444265.855 ± 28006.036
> {"instances": 1000} 419068.664 ± 49878.394
> {"instances": 10000} 404739.919 ± 13857.501
> ```
>
> Seems like we're able to take pretty good advantage of caching in myBatis and/or H2, at which point performance mostly varies with the response size.
>
>
> Thanks,
>
> Bill Farner
>
>
Re: Review Request 35630: DbTaskStore perf: add a task store API to
list task job keys.
Posted by Bill Farner <wf...@apache.org>.
> On June 18, 2015, 11:08 p.m., Maxim Khutornenko wrote:
> > src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml, line 167
> > <https://reviews.apache.org/r/35630/diff/1/?file=987636#file987636line167>
> >
> > Is DISTINCT necessary given the Set result type?
>
> Bill Farner wrote:
> Functionally it's not necessary, but why not dedupe as low in the stack as we can? If the database is on a separate host, for example, we save on the network.
>
> Maxim Khutornenko wrote:
> There is already a UNIQUE constraint on the job_keys table, so DISTINCT is not really necessary anyway.
Closing the loop from offline discussion - the UNIQUE constraint is not relevant since we're doing an INNER JOIN against job_keys.
- Bill
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35630/#review88452
-----------------------------------------------------------
On June 18, 2015, 10:39 p.m., Bill Farner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35630/
> -----------------------------------------------------------
>
> (Updated June 18, 2015, 10:39 p.m.)
>
>
> Review request for Aurora and Maxim Khutornenko.
>
>
> Bugs: AURORA-1298
> https://issues.apache.org/jira/browse/AURORA-1298
>
>
> Repository: aurora
>
>
> Description
> -------
>
> DbTaskStore perf: add a task store API to list task job keys.
>
>
> Diffs
> -----
>
> src/main/java/org/apache/aurora/scheduler/storage/ForwardingStore.java 5242eba2e1a5b35ba3d321f96e2c6670afed4c11
> src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 2768e6eafa51f6df78073715a7caf10bb7adba25
> src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 76f65dab5a5cb8d97ada962a06a7d0e191739119
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java 9903675c9d5ee34aebbc52c85fbf02929422a3bf
> src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 3a9de60d7e743634815b83587bd61814d883bf86
> src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 30e579c46168b64bdc4ba2416d49cf34b4e5c5f6
> src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml c3ab3eb9bd6289f73db13e7a7978b5d47621bb52
> src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 0f3a1a0ffde16f629422caf868fadb59be70e418
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java a854ed3d99af08fb46e436eb1bc00d8fb312278e
>
> Diff: https://reviews.apache.org/r/35630/diff/
>
>
> Testing
> -------
>
> GetRoleSummaryBenchmark results using MemTaskStore:
> ```
> {"roles": 1} 6234.156 ± 41.303
> {"roles": 10} 448.273 ± 130.781
> {"roles": 100} 13.768 ± 2.186
> {"roles": 500} 2.017 ± 3.046
> {"jobs": 1} 7252.866 ± 295.430
> {"jobs": 10} 6034.267 ± 256.936
> {"jobs": 100} 6081.036 ± 272.341
> {"jobs": 500} 6108.709 ± 59.118
> {"instances": 1} 14591.270 ± 2589.012
> {"instances": 10} 11790.114 ± 1595.596
> {"instances": 100} 6332.318 ± 177.585
> {"instances": 1000} 1078.874 ± 25.261
> {"instances": 10000} 23.874 ± 0.911
> ```
>
> Results using DbTaskStore with this patch:
> ```
> {"roles": 1} 450002.057 ± 54941.670
> {"roles": 10} 111124.857 ± 5335.829
> {"roles": 100} 12377.483 ± 800.637
> {"roles": 500} 2388.166 ± 101.003
> {"jobs": 1} 431646.943 ± 34172.295
> {"jobs": 10} 424383.191 ± 19939.399
> {"jobs": 100} 417483.093 ± 16342.521
> {"jobs": 500} 392792.143 ± 27064.583
> {"instances": 1} 454656.616 ± 21980.375
> {"instances": 10} 460502.356 ± 19182.511
> {"instances": 100} 444265.855 ± 28006.036
> {"instances": 1000} 419068.664 ± 49878.394
> {"instances": 10000} 404739.919 ± 13857.501
> ```
>
> Seems like we're able to take pretty good advantage of caching in myBatis and/or H2, at which point performance mostly varies with the response size.
>
>
> Thanks,
>
> Bill Farner
>
>
Re: Review Request 35630: DbTaskStore perf: add a task store API to
list task job keys.
Posted by Bill Farner <wf...@apache.org>.
> On June 18, 2015, 11:08 p.m., Maxim Khutornenko wrote:
> > src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml, line 167
> > <https://reviews.apache.org/r/35630/diff/1/?file=987636#file987636line167>
> >
> > Is DISTINCT necessary given the Set result type?
Functionally it's not necessary, but why not dedupe as low in the stack as we can? If the database is on a separate host, for example, we save on the network.
- Bill
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35630/#review88452
-----------------------------------------------------------
On June 18, 2015, 10:39 p.m., Bill Farner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35630/
> -----------------------------------------------------------
>
> (Updated June 18, 2015, 10:39 p.m.)
>
>
> Review request for Aurora and Maxim Khutornenko.
>
>
> Bugs: AURORA-1298
> https://issues.apache.org/jira/browse/AURORA-1298
>
>
> Repository: aurora
>
>
> Description
> -------
>
> DbTaskStore perf: add a task store API to list task job keys.
>
>
> Diffs
> -----
>
> src/main/java/org/apache/aurora/scheduler/storage/ForwardingStore.java 5242eba2e1a5b35ba3d321f96e2c6670afed4c11
> src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 2768e6eafa51f6df78073715a7caf10bb7adba25
> src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 76f65dab5a5cb8d97ada962a06a7d0e191739119
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java 9903675c9d5ee34aebbc52c85fbf02929422a3bf
> src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 3a9de60d7e743634815b83587bd61814d883bf86
> src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 30e579c46168b64bdc4ba2416d49cf34b4e5c5f6
> src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml c3ab3eb9bd6289f73db13e7a7978b5d47621bb52
> src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 0f3a1a0ffde16f629422caf868fadb59be70e418
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java a854ed3d99af08fb46e436eb1bc00d8fb312278e
>
> Diff: https://reviews.apache.org/r/35630/diff/
>
>
> Testing
> -------
>
> GetRoleSummaryBenchmark results using MemTaskStore:
> ```
> {"roles": 1} 6234.156 ± 41.303
> {"roles": 10} 448.273 ± 130.781
> {"roles": 100} 13.768 ± 2.186
> {"roles": 500} 2.017 ± 3.046
> {"jobs": 1} 7252.866 ± 295.430
> {"jobs": 10} 6034.267 ± 256.936
> {"jobs": 100} 6081.036 ± 272.341
> {"jobs": 500} 6108.709 ± 59.118
> {"instances": 1} 14591.270 ± 2589.012
> {"instances": 10} 11790.114 ± 1595.596
> {"instances": 100} 6332.318 ± 177.585
> {"instances": 1000} 1078.874 ± 25.261
> {"instances": 10000} 23.874 ± 0.911
> ```
>
> Results using DbTaskStore with this patch:
> ```
> {"roles": 1} 450002.057 ± 54941.670
> {"roles": 10} 111124.857 ± 5335.829
> {"roles": 100} 12377.483 ± 800.637
> {"roles": 500} 2388.166 ± 101.003
> {"jobs": 1} 431646.943 ± 34172.295
> {"jobs": 10} 424383.191 ± 19939.399
> {"jobs": 100} 417483.093 ± 16342.521
> {"jobs": 500} 392792.143 ± 27064.583
> {"instances": 1} 454656.616 ± 21980.375
> {"instances": 10} 460502.356 ± 19182.511
> {"instances": 100} 444265.855 ± 28006.036
> {"instances": 1000} 419068.664 ± 49878.394
> {"instances": 10000} 404739.919 ± 13857.501
> ```
>
> Seems like we're able to take pretty good advantage of caching in myBatis and/or H2, at which point performance mostly varies with the response size.
>
>
> Thanks,
>
> Bill Farner
>
>
Re: Review Request 35630: DbTaskStore perf: add a task store API to
list task job keys.
Posted by Maxim Khutornenko <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35630/#review88452
-----------------------------------------------------------
Ship it!
src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml (line 167)
<https://reviews.apache.org/r/35630/#comment140987>
Is DISTINCT necessary given the Set result type?
- Maxim Khutornenko
On June 18, 2015, 10:39 p.m., Bill Farner wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35630/
> -----------------------------------------------------------
>
> (Updated June 18, 2015, 10:39 p.m.)
>
>
> Review request for Aurora and Maxim Khutornenko.
>
>
> Bugs: AURORA-1298
> https://issues.apache.org/jira/browse/AURORA-1298
>
>
> Repository: aurora
>
>
> Description
> -------
>
> DbTaskStore perf: add a task store API to list task job keys.
>
>
> Diffs
> -----
>
> src/main/java/org/apache/aurora/scheduler/storage/ForwardingStore.java 5242eba2e1a5b35ba3d321f96e2c6670afed4c11
> src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 2768e6eafa51f6df78073715a7caf10bb7adba25
> src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 76f65dab5a5cb8d97ada962a06a7d0e191739119
> src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java 9903675c9d5ee34aebbc52c85fbf02929422a3bf
> src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 3a9de60d7e743634815b83587bd61814d883bf86
> src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 30e579c46168b64bdc4ba2416d49cf34b4e5c5f6
> src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml c3ab3eb9bd6289f73db13e7a7978b5d47621bb52
> src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 0f3a1a0ffde16f629422caf868fadb59be70e418
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java a854ed3d99af08fb46e436eb1bc00d8fb312278e
>
> Diff: https://reviews.apache.org/r/35630/diff/
>
>
> Testing
> -------
>
> GetRoleSummaryBenchmark results using MemTaskStore:
> ```
> {"roles": 1} 6234.156 ± 41.303
> {"roles": 10} 448.273 ± 130.781
> {"roles": 100} 13.768 ± 2.186
> {"roles": 500} 2.017 ± 3.046
> {"jobs": 1} 7252.866 ± 295.430
> {"jobs": 10} 6034.267 ± 256.936
> {"jobs": 100} 6081.036 ± 272.341
> {"jobs": 500} 6108.709 ± 59.118
> {"instances": 1} 14591.270 ± 2589.012
> {"instances": 10} 11790.114 ± 1595.596
> {"instances": 100} 6332.318 ± 177.585
> {"instances": 1000} 1078.874 ± 25.261
> {"instances": 10000} 23.874 ± 0.911
> ```
>
> Results using DbTaskStore with this patch:
> ```
> {"roles": 1} 450002.057 ± 54941.670
> {"roles": 10} 111124.857 ± 5335.829
> {"roles": 100} 12377.483 ± 800.637
> {"roles": 500} 2388.166 ± 101.003
> {"jobs": 1} 431646.943 ± 34172.295
> {"jobs": 10} 424383.191 ± 19939.399
> {"jobs": 100} 417483.093 ± 16342.521
> {"jobs": 500} 392792.143 ± 27064.583
> {"instances": 1} 454656.616 ± 21980.375
> {"instances": 10} 460502.356 ± 19182.511
> {"instances": 100} 444265.855 ± 28006.036
> {"instances": 1000} 419068.664 ± 49878.394
> {"instances": 10000} 404739.919 ± 13857.501
> ```
>
> Seems like we're able to take pretty good advantage of caching in myBatis and/or H2, at which point performance mostly varies with the response size.
>
>
> Thanks,
>
> Bill Farner
>
>