You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Ben Mahler <be...@gmail.com> on 2014/03/05 01:26:29 UTC
Review Request 18758: Performance optimizations to the Registrar.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18758/
-----------------------------------------------------------
Review request for mesos, Benjamin Hindman and Vinod Kone.
Bugs: MESOS-764
https://issues.apache.org/jira/browse/MESOS-764
Repository: mesos-git
Description
-------
The bottleneck in the Registrar is the copying of the Registry. For a large number of slaves and operations, this becomes prohibitively expensive. This patch therefore eliminates copying when applying the operations. Optimizations to Future also eliminated copying in _update.
Further optimizations will be explored for larger clusters, this was currently tested with 10,000 slaves.
Diffs
-----
src/master/registrar.cpp 37337c07b24a96e71910b7c83085d159361a1188
Diff: https://reviews.apache.org/r/18758/diff/
Testing
-------
Prior to this patch and the Future::get optimizations, the performance benchmark I wrote took over an hour for 10,000 slaves.
With these optimizations 10,000 slaves can be handled quite efficiently:
Key numbers with gcc 4.1.2 on a 2.4GHz 12 core machine:
I0304 23:37:06.024024 33235 registrar_tests.cpp:229] Admitted 10,000 slaves in 10.04498586secs
I0304 23:37:15.853411 33235 registrar_tests.cpp:237] Readmitted 10,000 slaves in 9.829011689secs
I0304 23:37:15.876322 33235 registrar_tests.cpp:243] Recovered 10,000 slaves in 22.715892ms
I0304 23:37:19.453868 33235 registrar_tests.cpp:251] Removed 10,000 slaves in 3.577429332secs
[ RUN ] RegistrarTest.DISABLED_performance
I0304 23:36:55.685456 33261 registrar.cpp:218] Recovering registrar
I0304 23:36:55.801048 33252 registrar.cpp:381] Attempting to update the 'registry'
Applied 1 operations in: 1.235673ms
Stored Registry in: 36.199255ms
I0304 23:36:55.837477 33271 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 66813ns
I0304 23:36:55.972564 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 1 operations in: 1.573455ms
Stored Registry in: 30.215382ms
I0304 23:36:56.002907 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 113129ns
I0304 23:36:56.003094 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 146 operations in: 8.579203ms
Stored Registry in: 174.710644ms
I0304 23:36:56.179323 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 146 promises in: 4.656175ms
I0304 23:36:56.184154 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 574 operations in: 73.828182ms
Stored Registry in: 527.540062ms
I0304 23:36:56.744689 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 574 promises in: 13.873625ms
I0304 23:36:56.760572 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 2400 operations in: 907.070761ms
Stored Registry in: 268.604653ms
I0304 23:36:57.778074 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 2400 promises in: 69.867013ms
I0304 23:36:57.858157 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 6879 operations in: 7.910334133secs
Stored Registry in: 990.759579ms
I0304 23:37:05.795042 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 6879 promises in: 221.701612ms
I0304 23:37:06.024024 33235 registrar_tests.cpp:229] Admitted 10,000 slaves in 10.04498586secs
I0304 23:37:06.125613 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 1 operations in: 852.076566ms
Stored Registry in: 1.384153523secs
I0304 23:37:07.576215 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 191.668241ms
I0304 23:37:07.768013 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 993 operations in: 923.901654ms
Stored Registry in: 953.378897ms
I0304 23:37:08.827229 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 993 promises in: 164.437762ms
I0304 23:37:08.992169 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 9006 operations in: 6.081593589secs
Stored Registry in: 896.142182ms
I0304 23:37:15.397238 33257 registrar.cpp:425] Successfully updated 'registry'
Transitioned 9006 promises in: 427.267246ms
I0304 23:37:15.853411 33235 registrar_tests.cpp:237] Readmitted 10,000 slaves in 9.829011689secs
I0304 23:37:15.876085 33257 registrar.cpp:218] Recovering registrar
I0304 23:37:15.876322 33235 registrar_tests.cpp:243] Recovered 10,000 slaves in 22.715892ms
I0304 23:37:15.876656 33264 registrar.cpp:381] Attempting to update the 'registry'
Applied 1 operations in: 819.303422ms
Stored Registry in: 1.383689258secs
I0304 23:37:17.329473 33264 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 189.505006ms
I0304 23:37:17.541276 33264 registrar.cpp:381] Attempting to update the 'registry'
Applied 1 operations in: 921.761221ms
Stored Registry in: 986.688995ms
I0304 23:37:18.588706 33264 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 179.771104ms
I0304 23:37:18.768606 33264 registrar.cpp:381] Attempting to update the 'registry'
Applied 9998 operations in: 441.652539ms
Stored Registry in: 209.866548ms
I0304 23:37:19.298931 33265 registrar.cpp:425] Successfully updated 'registry'
Transitioned 9998 promises in: 154.775647ms
I0304 23:37:19.453868 33235 registrar_tests.cpp:251] Removed 10,000 slaves in 3.577429332secs
[ OK ] RegistrarTest.DISABLED_performance (23791 ms)
Thanks,
Ben Mahler
Re: Review Request 18758: Performance optimizations to the Registrar.
Posted by Dominic Hamon <dh...@twopensource.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18758/#review36197
-----------------------------------------------------------
Ship it!
Ship It!
- Dominic Hamon
On March 4, 2014, 4:26 p.m., Ben Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18758/
> -----------------------------------------------------------
>
> (Updated March 4, 2014, 4:26 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Vinod Kone.
>
>
> Bugs: MESOS-764
> https://issues.apache.org/jira/browse/MESOS-764
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> The bottleneck in the Registrar is the copying of the Registry. For a large number of slaves and operations, this becomes prohibitively expensive. This patch therefore eliminates copying when applying the operations. Optimizations to Future also eliminated copying in _update.
>
> Further optimizations will be explored for larger clusters, this was currently tested with 10,000 slaves.
>
>
> Diffs
> -----
>
> src/master/registrar.cpp 37337c07b24a96e71910b7c83085d159361a1188
>
> Diff: https://reviews.apache.org/r/18758/diff/
>
>
> Testing
> -------
>
> Prior to this patch and the Future::get optimizations, the performance benchmark I wrote took over an hour for 10,000 slaves.
>
> With these optimizations 10,000 slaves can be handled quite efficiently:
>
> Key numbers with gcc 4.1.2 on a 2.4GHz 12 core machine:
>
> I0304 23:37:06.024024 33235 registrar_tests.cpp:229] Admitted 10,000 slaves in 10.04498586secs
> I0304 23:37:15.853411 33235 registrar_tests.cpp:237] Readmitted 10,000 slaves in 9.829011689secs
> I0304 23:37:15.876322 33235 registrar_tests.cpp:243] Recovered 10,000 slaves in 22.715892ms
> I0304 23:37:19.453868 33235 registrar_tests.cpp:251] Removed 10,000 slaves in 3.577429332secs
>
>
> [ RUN ] RegistrarTest.DISABLED_performance
> I0304 23:36:55.685456 33261 registrar.cpp:218] Recovering registrar
> I0304 23:36:55.801048 33252 registrar.cpp:381] Attempting to update the 'registry'
> Applied 1 operations in: 1.235673ms
> Stored Registry in: 36.199255ms
> I0304 23:36:55.837477 33271 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 1 promises in: 66813ns
> I0304 23:36:55.972564 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 1 operations in: 1.573455ms
> Stored Registry in: 30.215382ms
> I0304 23:36:56.002907 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 1 promises in: 113129ns
> I0304 23:36:56.003094 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 146 operations in: 8.579203ms
> Stored Registry in: 174.710644ms
> I0304 23:36:56.179323 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 146 promises in: 4.656175ms
> I0304 23:36:56.184154 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 574 operations in: 73.828182ms
> Stored Registry in: 527.540062ms
> I0304 23:36:56.744689 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 574 promises in: 13.873625ms
> I0304 23:36:56.760572 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 2400 operations in: 907.070761ms
> Stored Registry in: 268.604653ms
> I0304 23:36:57.778074 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 2400 promises in: 69.867013ms
> I0304 23:36:57.858157 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 6879 operations in: 7.910334133secs
> Stored Registry in: 990.759579ms
> I0304 23:37:05.795042 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 6879 promises in: 221.701612ms
> I0304 23:37:06.024024 33235 registrar_tests.cpp:229] Admitted 10,000 slaves in 10.04498586secs
> I0304 23:37:06.125613 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 1 operations in: 852.076566ms
> Stored Registry in: 1.384153523secs
> I0304 23:37:07.576215 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 1 promises in: 191.668241ms
> I0304 23:37:07.768013 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 993 operations in: 923.901654ms
> Stored Registry in: 953.378897ms
> I0304 23:37:08.827229 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 993 promises in: 164.437762ms
> I0304 23:37:08.992169 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 9006 operations in: 6.081593589secs
> Stored Registry in: 896.142182ms
> I0304 23:37:15.397238 33257 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 9006 promises in: 427.267246ms
> I0304 23:37:15.853411 33235 registrar_tests.cpp:237] Readmitted 10,000 slaves in 9.829011689secs
> I0304 23:37:15.876085 33257 registrar.cpp:218] Recovering registrar
> I0304 23:37:15.876322 33235 registrar_tests.cpp:243] Recovered 10,000 slaves in 22.715892ms
> I0304 23:37:15.876656 33264 registrar.cpp:381] Attempting to update the 'registry'
> Applied 1 operations in: 819.303422ms
> Stored Registry in: 1.383689258secs
> I0304 23:37:17.329473 33264 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 1 promises in: 189.505006ms
> I0304 23:37:17.541276 33264 registrar.cpp:381] Attempting to update the 'registry'
> Applied 1 operations in: 921.761221ms
> Stored Registry in: 986.688995ms
> I0304 23:37:18.588706 33264 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 1 promises in: 179.771104ms
> I0304 23:37:18.768606 33264 registrar.cpp:381] Attempting to update the 'registry'
> Applied 9998 operations in: 441.652539ms
> Stored Registry in: 209.866548ms
> I0304 23:37:19.298931 33265 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 9998 promises in: 154.775647ms
> I0304 23:37:19.453868 33235 registrar_tests.cpp:251] Removed 10,000 slaves in 3.577429332secs
> [ OK ] RegistrarTest.DISABLED_performance (23791 ms)
>
>
> Thanks,
>
> Ben Mahler
>
>
Re: Review Request 18758: Performance optimizations to the Registrar.
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18758/#review36207
-----------------------------------------------------------
Ship it!
Ship It!
- Vinod Kone
On March 5, 2014, 12:26 a.m., Ben Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18758/
> -----------------------------------------------------------
>
> (Updated March 5, 2014, 12:26 a.m.)
>
>
> Review request for mesos, Benjamin Hindman and Vinod Kone.
>
>
> Bugs: MESOS-764
> https://issues.apache.org/jira/browse/MESOS-764
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> The bottleneck in the Registrar is the copying of the Registry. For a large number of slaves and operations, this becomes prohibitively expensive. This patch therefore eliminates copying when applying the operations. Optimizations to Future also eliminated copying in _update.
>
> Further optimizations will be explored for larger clusters, this was currently tested with 10,000 slaves.
>
>
> Diffs
> -----
>
> src/master/registrar.cpp 37337c07b24a96e71910b7c83085d159361a1188
>
> Diff: https://reviews.apache.org/r/18758/diff/
>
>
> Testing
> -------
>
> Prior to this patch and the Future::get optimizations, the performance benchmark I wrote took over an hour for 10,000 slaves.
>
> With these optimizations 10,000 slaves can be handled quite efficiently:
>
> Key numbers with gcc 4.1.2 on a 2.4GHz 12 core machine:
>
> I0304 23:37:06.024024 33235 registrar_tests.cpp:229] Admitted 10,000 slaves in 10.04498586secs
> I0304 23:37:15.853411 33235 registrar_tests.cpp:237] Readmitted 10,000 slaves in 9.829011689secs
> I0304 23:37:15.876322 33235 registrar_tests.cpp:243] Recovered 10,000 slaves in 22.715892ms
> I0304 23:37:19.453868 33235 registrar_tests.cpp:251] Removed 10,000 slaves in 3.577429332secs
>
>
> [ RUN ] RegistrarTest.DISABLED_performance
> I0304 23:36:55.685456 33261 registrar.cpp:218] Recovering registrar
> I0304 23:36:55.801048 33252 registrar.cpp:381] Attempting to update the 'registry'
> Applied 1 operations in: 1.235673ms
> Stored Registry in: 36.199255ms
> I0304 23:36:55.837477 33271 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 1 promises in: 66813ns
> I0304 23:36:55.972564 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 1 operations in: 1.573455ms
> Stored Registry in: 30.215382ms
> I0304 23:36:56.002907 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 1 promises in: 113129ns
> I0304 23:36:56.003094 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 146 operations in: 8.579203ms
> Stored Registry in: 174.710644ms
> I0304 23:36:56.179323 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 146 promises in: 4.656175ms
> I0304 23:36:56.184154 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 574 operations in: 73.828182ms
> Stored Registry in: 527.540062ms
> I0304 23:36:56.744689 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 574 promises in: 13.873625ms
> I0304 23:36:56.760572 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 2400 operations in: 907.070761ms
> Stored Registry in: 268.604653ms
> I0304 23:36:57.778074 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 2400 promises in: 69.867013ms
> I0304 23:36:57.858157 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 6879 operations in: 7.910334133secs
> Stored Registry in: 990.759579ms
> I0304 23:37:05.795042 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 6879 promises in: 221.701612ms
> I0304 23:37:06.024024 33235 registrar_tests.cpp:229] Admitted 10,000 slaves in 10.04498586secs
> I0304 23:37:06.125613 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 1 operations in: 852.076566ms
> Stored Registry in: 1.384153523secs
> I0304 23:37:07.576215 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 1 promises in: 191.668241ms
> I0304 23:37:07.768013 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 993 operations in: 923.901654ms
> Stored Registry in: 953.378897ms
> I0304 23:37:08.827229 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 993 promises in: 164.437762ms
> I0304 23:37:08.992169 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 9006 operations in: 6.081593589secs
> Stored Registry in: 896.142182ms
> I0304 23:37:15.397238 33257 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 9006 promises in: 427.267246ms
> I0304 23:37:15.853411 33235 registrar_tests.cpp:237] Readmitted 10,000 slaves in 9.829011689secs
> I0304 23:37:15.876085 33257 registrar.cpp:218] Recovering registrar
> I0304 23:37:15.876322 33235 registrar_tests.cpp:243] Recovered 10,000 slaves in 22.715892ms
> I0304 23:37:15.876656 33264 registrar.cpp:381] Attempting to update the 'registry'
> Applied 1 operations in: 819.303422ms
> Stored Registry in: 1.383689258secs
> I0304 23:37:17.329473 33264 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 1 promises in: 189.505006ms
> I0304 23:37:17.541276 33264 registrar.cpp:381] Attempting to update the 'registry'
> Applied 1 operations in: 921.761221ms
> Stored Registry in: 986.688995ms
> I0304 23:37:18.588706 33264 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 1 promises in: 179.771104ms
> I0304 23:37:18.768606 33264 registrar.cpp:381] Attempting to update the 'registry'
> Applied 9998 operations in: 441.652539ms
> Stored Registry in: 209.866548ms
> I0304 23:37:19.298931 33265 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 9998 promises in: 154.775647ms
> I0304 23:37:19.453868 33235 registrar_tests.cpp:251] Removed 10,000 slaves in 3.577429332secs
> [ OK ] RegistrarTest.DISABLED_performance (23791 ms)
>
>
> Thanks,
>
> Ben Mahler
>
>
Re: Review Request 18758: Performance optimizations to the Registrar.
Posted by Benjamin Hindman <be...@berkeley.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18758/#review36284
-----------------------------------------------------------
Ship it!
Ship It!
- Benjamin Hindman
On March 5, 2014, 12:26 a.m., Ben Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18758/
> -----------------------------------------------------------
>
> (Updated March 5, 2014, 12:26 a.m.)
>
>
> Review request for mesos, Benjamin Hindman and Vinod Kone.
>
>
> Bugs: MESOS-764
> https://issues.apache.org/jira/browse/MESOS-764
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> The bottleneck in the Registrar is the copying of the Registry. For a large number of slaves and operations, this becomes prohibitively expensive. This patch therefore eliminates copying when applying the operations. Optimizations to Future also eliminated copying in _update.
>
> Further optimizations will be explored for larger clusters, this was currently tested with 10,000 slaves.
>
>
> Diffs
> -----
>
> src/master/registrar.cpp 37337c07b24a96e71910b7c83085d159361a1188
>
> Diff: https://reviews.apache.org/r/18758/diff/
>
>
> Testing
> -------
>
> Prior to this patch and the Future::get optimizations, the performance benchmark I wrote took over an hour for 10,000 slaves.
>
> With these optimizations 10,000 slaves can be handled quite efficiently:
>
> Key numbers with gcc 4.1.2 on a 2.4GHz 12 core machine:
>
> I0304 23:37:06.024024 33235 registrar_tests.cpp:229] Admitted 10,000 slaves in 10.04498586secs
> I0304 23:37:15.853411 33235 registrar_tests.cpp:237] Readmitted 10,000 slaves in 9.829011689secs
> I0304 23:37:15.876322 33235 registrar_tests.cpp:243] Recovered 10,000 slaves in 22.715892ms
> I0304 23:37:19.453868 33235 registrar_tests.cpp:251] Removed 10,000 slaves in 3.577429332secs
>
>
> [ RUN ] RegistrarTest.DISABLED_performance
> I0304 23:36:55.685456 33261 registrar.cpp:218] Recovering registrar
> I0304 23:36:55.801048 33252 registrar.cpp:381] Attempting to update the 'registry'
> Applied 1 operations in: 1.235673ms
> Stored Registry in: 36.199255ms
> I0304 23:36:55.837477 33271 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 1 promises in: 66813ns
> I0304 23:36:55.972564 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 1 operations in: 1.573455ms
> Stored Registry in: 30.215382ms
> I0304 23:36:56.002907 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 1 promises in: 113129ns
> I0304 23:36:56.003094 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 146 operations in: 8.579203ms
> Stored Registry in: 174.710644ms
> I0304 23:36:56.179323 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 146 promises in: 4.656175ms
> I0304 23:36:56.184154 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 574 operations in: 73.828182ms
> Stored Registry in: 527.540062ms
> I0304 23:36:56.744689 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 574 promises in: 13.873625ms
> I0304 23:36:56.760572 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 2400 operations in: 907.070761ms
> Stored Registry in: 268.604653ms
> I0304 23:36:57.778074 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 2400 promises in: 69.867013ms
> I0304 23:36:57.858157 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 6879 operations in: 7.910334133secs
> Stored Registry in: 990.759579ms
> I0304 23:37:05.795042 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 6879 promises in: 221.701612ms
> I0304 23:37:06.024024 33235 registrar_tests.cpp:229] Admitted 10,000 slaves in 10.04498586secs
> I0304 23:37:06.125613 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 1 operations in: 852.076566ms
> Stored Registry in: 1.384153523secs
> I0304 23:37:07.576215 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 1 promises in: 191.668241ms
> I0304 23:37:07.768013 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 993 operations in: 923.901654ms
> Stored Registry in: 953.378897ms
> I0304 23:37:08.827229 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 993 promises in: 164.437762ms
> I0304 23:37:08.992169 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 9006 operations in: 6.081593589secs
> Stored Registry in: 896.142182ms
> I0304 23:37:15.397238 33257 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 9006 promises in: 427.267246ms
> I0304 23:37:15.853411 33235 registrar_tests.cpp:237] Readmitted 10,000 slaves in 9.829011689secs
> I0304 23:37:15.876085 33257 registrar.cpp:218] Recovering registrar
> I0304 23:37:15.876322 33235 registrar_tests.cpp:243] Recovered 10,000 slaves in 22.715892ms
> I0304 23:37:15.876656 33264 registrar.cpp:381] Attempting to update the 'registry'
> Applied 1 operations in: 819.303422ms
> Stored Registry in: 1.383689258secs
> I0304 23:37:17.329473 33264 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 1 promises in: 189.505006ms
> I0304 23:37:17.541276 33264 registrar.cpp:381] Attempting to update the 'registry'
> Applied 1 operations in: 921.761221ms
> Stored Registry in: 986.688995ms
> I0304 23:37:18.588706 33264 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 1 promises in: 179.771104ms
> I0304 23:37:18.768606 33264 registrar.cpp:381] Attempting to update the 'registry'
> Applied 9998 operations in: 441.652539ms
> Stored Registry in: 209.866548ms
> I0304 23:37:19.298931 33265 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 9998 promises in: 154.775647ms
> I0304 23:37:19.453868 33235 registrar_tests.cpp:251] Removed 10,000 slaves in 3.577429332secs
> [ OK ] RegistrarTest.DISABLED_performance (23791 ms)
>
>
> Thanks,
>
> Ben Mahler
>
>
Re: Review Request 18758: Performance optimizations to the Registrar.
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18758/
-----------------------------------------------------------
(Updated March 7, 2014, 9:36 p.m.)
Review request for mesos, Benjamin Hindman and Vinod Kone.
Changes
-------
Rebased. NNFR.
Bugs: MESOS-764
https://issues.apache.org/jira/browse/MESOS-764
Repository: mesos-git
Description
-------
The bottleneck in the Registrar is the copying of the Registry. For a large number of slaves and operations, this becomes prohibitively expensive. This patch therefore eliminates copying when applying the operations. Optimizations to Future also eliminated copying in _update.
Further optimizations will be explored for larger clusters, this was currently tested with 10,000 slaves.
Diffs (updated)
-----
src/master/registrar.cpp 37337c07b24a96e71910b7c83085d159361a1188
Diff: https://reviews.apache.org/r/18758/diff/
Testing
-------
Prior to this patch and the Future::get optimizations, the performance benchmark I wrote took over an hour for 10,000 slaves.
With these optimizations 10,000 slaves can be handled quite efficiently:
Key numbers with gcc 4.1.2 on a 2.4GHz 12 core machine:
I0304 23:37:06.024024 33235 registrar_tests.cpp:229] Admitted 10,000 slaves in 10.04498586secs
I0304 23:37:15.853411 33235 registrar_tests.cpp:237] Readmitted 10,000 slaves in 9.829011689secs
I0304 23:37:15.876322 33235 registrar_tests.cpp:243] Recovered 10,000 slaves in 22.715892ms
I0304 23:37:19.453868 33235 registrar_tests.cpp:251] Removed 10,000 slaves in 3.577429332secs
[ RUN ] RegistrarTest.DISABLED_performance
I0304 23:36:55.685456 33261 registrar.cpp:218] Recovering registrar
I0304 23:36:55.801048 33252 registrar.cpp:381] Attempting to update the 'registry'
Applied 1 operations in: 1.235673ms
Stored Registry in: 36.199255ms
I0304 23:36:55.837477 33271 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 66813ns
I0304 23:36:55.972564 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 1 operations in: 1.573455ms
Stored Registry in: 30.215382ms
I0304 23:36:56.002907 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 113129ns
I0304 23:36:56.003094 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 146 operations in: 8.579203ms
Stored Registry in: 174.710644ms
I0304 23:36:56.179323 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 146 promises in: 4.656175ms
I0304 23:36:56.184154 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 574 operations in: 73.828182ms
Stored Registry in: 527.540062ms
I0304 23:36:56.744689 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 574 promises in: 13.873625ms
I0304 23:36:56.760572 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 2400 operations in: 907.070761ms
Stored Registry in: 268.604653ms
I0304 23:36:57.778074 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 2400 promises in: 69.867013ms
I0304 23:36:57.858157 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 6879 operations in: 7.910334133secs
Stored Registry in: 990.759579ms
I0304 23:37:05.795042 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 6879 promises in: 221.701612ms
I0304 23:37:06.024024 33235 registrar_tests.cpp:229] Admitted 10,000 slaves in 10.04498586secs
I0304 23:37:06.125613 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 1 operations in: 852.076566ms
Stored Registry in: 1.384153523secs
I0304 23:37:07.576215 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 191.668241ms
I0304 23:37:07.768013 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 993 operations in: 923.901654ms
Stored Registry in: 953.378897ms
I0304 23:37:08.827229 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 993 promises in: 164.437762ms
I0304 23:37:08.992169 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 9006 operations in: 6.081593589secs
Stored Registry in: 896.142182ms
I0304 23:37:15.397238 33257 registrar.cpp:425] Successfully updated 'registry'
Transitioned 9006 promises in: 427.267246ms
I0304 23:37:15.853411 33235 registrar_tests.cpp:237] Readmitted 10,000 slaves in 9.829011689secs
I0304 23:37:15.876085 33257 registrar.cpp:218] Recovering registrar
I0304 23:37:15.876322 33235 registrar_tests.cpp:243] Recovered 10,000 slaves in 22.715892ms
I0304 23:37:15.876656 33264 registrar.cpp:381] Attempting to update the 'registry'
Applied 1 operations in: 819.303422ms
Stored Registry in: 1.383689258secs
I0304 23:37:17.329473 33264 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 189.505006ms
I0304 23:37:17.541276 33264 registrar.cpp:381] Attempting to update the 'registry'
Applied 1 operations in: 921.761221ms
Stored Registry in: 986.688995ms
I0304 23:37:18.588706 33264 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 179.771104ms
I0304 23:37:18.768606 33264 registrar.cpp:381] Attempting to update the 'registry'
Applied 9998 operations in: 441.652539ms
Stored Registry in: 209.866548ms
I0304 23:37:19.298931 33265 registrar.cpp:425] Successfully updated 'registry'
Transitioned 9998 promises in: 154.775647ms
I0304 23:37:19.453868 33235 registrar_tests.cpp:251] Removed 10,000 slaves in 3.577429332secs
[ OK ] RegistrarTest.DISABLED_performance (23791 ms)
Thanks,
Ben Mahler
Re: Review Request 18758: Performance optimizations to the Registrar.
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18758/
-----------------------------------------------------------
(Updated March 6, 2014, 8:28 p.m.)
Review request for mesos, Benjamin Hindman and Vinod Kone.
Changes
-------
Re-added dependency.
Bugs: MESOS-764
https://issues.apache.org/jira/browse/MESOS-764
Repository: mesos-git
Description
-------
The bottleneck in the Registrar is the copying of the Registry. For a large number of slaves and operations, this becomes prohibitively expensive. This patch therefore eliminates copying when applying the operations. Optimizations to Future also eliminated copying in _update.
Further optimizations will be explored for larger clusters, this was currently tested with 10,000 slaves.
Diffs
-----
src/master/registrar.cpp 37337c07b24a96e71910b7c83085d159361a1188
Diff: https://reviews.apache.org/r/18758/diff/
Testing
-------
Prior to this patch and the Future::get optimizations, the performance benchmark I wrote took over an hour for 10,000 slaves.
With these optimizations 10,000 slaves can be handled quite efficiently:
Key numbers with gcc 4.1.2 on a 2.4GHz 12 core machine:
I0304 23:37:06.024024 33235 registrar_tests.cpp:229] Admitted 10,000 slaves in 10.04498586secs
I0304 23:37:15.853411 33235 registrar_tests.cpp:237] Readmitted 10,000 slaves in 9.829011689secs
I0304 23:37:15.876322 33235 registrar_tests.cpp:243] Recovered 10,000 slaves in 22.715892ms
I0304 23:37:19.453868 33235 registrar_tests.cpp:251] Removed 10,000 slaves in 3.577429332secs
[ RUN ] RegistrarTest.DISABLED_performance
I0304 23:36:55.685456 33261 registrar.cpp:218] Recovering registrar
I0304 23:36:55.801048 33252 registrar.cpp:381] Attempting to update the 'registry'
Applied 1 operations in: 1.235673ms
Stored Registry in: 36.199255ms
I0304 23:36:55.837477 33271 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 66813ns
I0304 23:36:55.972564 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 1 operations in: 1.573455ms
Stored Registry in: 30.215382ms
I0304 23:36:56.002907 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 113129ns
I0304 23:36:56.003094 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 146 operations in: 8.579203ms
Stored Registry in: 174.710644ms
I0304 23:36:56.179323 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 146 promises in: 4.656175ms
I0304 23:36:56.184154 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 574 operations in: 73.828182ms
Stored Registry in: 527.540062ms
I0304 23:36:56.744689 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 574 promises in: 13.873625ms
I0304 23:36:56.760572 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 2400 operations in: 907.070761ms
Stored Registry in: 268.604653ms
I0304 23:36:57.778074 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 2400 promises in: 69.867013ms
I0304 23:36:57.858157 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 6879 operations in: 7.910334133secs
Stored Registry in: 990.759579ms
I0304 23:37:05.795042 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 6879 promises in: 221.701612ms
I0304 23:37:06.024024 33235 registrar_tests.cpp:229] Admitted 10,000 slaves in 10.04498586secs
I0304 23:37:06.125613 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 1 operations in: 852.076566ms
Stored Registry in: 1.384153523secs
I0304 23:37:07.576215 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 191.668241ms
I0304 23:37:07.768013 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 993 operations in: 923.901654ms
Stored Registry in: 953.378897ms
I0304 23:37:08.827229 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 993 promises in: 164.437762ms
I0304 23:37:08.992169 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 9006 operations in: 6.081593589secs
Stored Registry in: 896.142182ms
I0304 23:37:15.397238 33257 registrar.cpp:425] Successfully updated 'registry'
Transitioned 9006 promises in: 427.267246ms
I0304 23:37:15.853411 33235 registrar_tests.cpp:237] Readmitted 10,000 slaves in 9.829011689secs
I0304 23:37:15.876085 33257 registrar.cpp:218] Recovering registrar
I0304 23:37:15.876322 33235 registrar_tests.cpp:243] Recovered 10,000 slaves in 22.715892ms
I0304 23:37:15.876656 33264 registrar.cpp:381] Attempting to update the 'registry'
Applied 1 operations in: 819.303422ms
Stored Registry in: 1.383689258secs
I0304 23:37:17.329473 33264 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 189.505006ms
I0304 23:37:17.541276 33264 registrar.cpp:381] Attempting to update the 'registry'
Applied 1 operations in: 921.761221ms
Stored Registry in: 986.688995ms
I0304 23:37:18.588706 33264 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 179.771104ms
I0304 23:37:18.768606 33264 registrar.cpp:381] Attempting to update the 'registry'
Applied 9998 operations in: 441.652539ms
Stored Registry in: 209.866548ms
I0304 23:37:19.298931 33265 registrar.cpp:425] Successfully updated 'registry'
Transitioned 9998 promises in: 154.775647ms
I0304 23:37:19.453868 33235 registrar_tests.cpp:251] Removed 10,000 slaves in 3.577429332secs
[ OK ] RegistrarTest.DISABLED_performance (23791 ms)
Thanks,
Ben Mahler
Re: Review Request 18758: Performance optimizations to the Registrar.
Posted by Mesos ReviewBot <de...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18758/#review36421
-----------------------------------------------------------
Bad patch!
Reviews applied: [18758]
Failed command: git apply --index 18758.patch
Error:
error: patch failed: src/master/registrar.cpp:108
error: src/master/registrar.cpp: patch does not apply
- Mesos ReviewBot
On March 6, 2014, 8:13 p.m., Ben Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18758/
> -----------------------------------------------------------
>
> (Updated March 6, 2014, 8:13 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Vinod Kone.
>
>
> Bugs: MESOS-764
> https://issues.apache.org/jira/browse/MESOS-764
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> The bottleneck in the Registrar is the copying of the Registry. For a large number of slaves and operations, this becomes prohibitively expensive. This patch therefore eliminates copying when applying the operations. Optimizations to Future also eliminated copying in _update.
>
> Further optimizations will be explored for larger clusters, this was currently tested with 10,000 slaves.
>
>
> Diffs
> -----
>
> src/master/registrar.cpp 37337c07b24a96e71910b7c83085d159361a1188
>
> Diff: https://reviews.apache.org/r/18758/diff/
>
>
> Testing
> -------
>
> Prior to this patch and the Future::get optimizations, the performance benchmark I wrote took over an hour for 10,000 slaves.
>
> With these optimizations 10,000 slaves can be handled quite efficiently:
>
> Key numbers with gcc 4.1.2 on a 2.4GHz 12 core machine:
>
> I0304 23:37:06.024024 33235 registrar_tests.cpp:229] Admitted 10,000 slaves in 10.04498586secs
> I0304 23:37:15.853411 33235 registrar_tests.cpp:237] Readmitted 10,000 slaves in 9.829011689secs
> I0304 23:37:15.876322 33235 registrar_tests.cpp:243] Recovered 10,000 slaves in 22.715892ms
> I0304 23:37:19.453868 33235 registrar_tests.cpp:251] Removed 10,000 slaves in 3.577429332secs
>
>
> [ RUN ] RegistrarTest.DISABLED_performance
> I0304 23:36:55.685456 33261 registrar.cpp:218] Recovering registrar
> I0304 23:36:55.801048 33252 registrar.cpp:381] Attempting to update the 'registry'
> Applied 1 operations in: 1.235673ms
> Stored Registry in: 36.199255ms
> I0304 23:36:55.837477 33271 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 1 promises in: 66813ns
> I0304 23:36:55.972564 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 1 operations in: 1.573455ms
> Stored Registry in: 30.215382ms
> I0304 23:36:56.002907 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 1 promises in: 113129ns
> I0304 23:36:56.003094 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 146 operations in: 8.579203ms
> Stored Registry in: 174.710644ms
> I0304 23:36:56.179323 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 146 promises in: 4.656175ms
> I0304 23:36:56.184154 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 574 operations in: 73.828182ms
> Stored Registry in: 527.540062ms
> I0304 23:36:56.744689 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 574 promises in: 13.873625ms
> I0304 23:36:56.760572 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 2400 operations in: 907.070761ms
> Stored Registry in: 268.604653ms
> I0304 23:36:57.778074 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 2400 promises in: 69.867013ms
> I0304 23:36:57.858157 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 6879 operations in: 7.910334133secs
> Stored Registry in: 990.759579ms
> I0304 23:37:05.795042 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 6879 promises in: 221.701612ms
> I0304 23:37:06.024024 33235 registrar_tests.cpp:229] Admitted 10,000 slaves in 10.04498586secs
> I0304 23:37:06.125613 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 1 operations in: 852.076566ms
> Stored Registry in: 1.384153523secs
> I0304 23:37:07.576215 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 1 promises in: 191.668241ms
> I0304 23:37:07.768013 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 993 operations in: 923.901654ms
> Stored Registry in: 953.378897ms
> I0304 23:37:08.827229 33266 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 993 promises in: 164.437762ms
> I0304 23:37:08.992169 33266 registrar.cpp:381] Attempting to update the 'registry'
> Applied 9006 operations in: 6.081593589secs
> Stored Registry in: 896.142182ms
> I0304 23:37:15.397238 33257 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 9006 promises in: 427.267246ms
> I0304 23:37:15.853411 33235 registrar_tests.cpp:237] Readmitted 10,000 slaves in 9.829011689secs
> I0304 23:37:15.876085 33257 registrar.cpp:218] Recovering registrar
> I0304 23:37:15.876322 33235 registrar_tests.cpp:243] Recovered 10,000 slaves in 22.715892ms
> I0304 23:37:15.876656 33264 registrar.cpp:381] Attempting to update the 'registry'
> Applied 1 operations in: 819.303422ms
> Stored Registry in: 1.383689258secs
> I0304 23:37:17.329473 33264 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 1 promises in: 189.505006ms
> I0304 23:37:17.541276 33264 registrar.cpp:381] Attempting to update the 'registry'
> Applied 1 operations in: 921.761221ms
> Stored Registry in: 986.688995ms
> I0304 23:37:18.588706 33264 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 1 promises in: 179.771104ms
> I0304 23:37:18.768606 33264 registrar.cpp:381] Attempting to update the 'registry'
> Applied 9998 operations in: 441.652539ms
> Stored Registry in: 209.866548ms
> I0304 23:37:19.298931 33265 registrar.cpp:425] Successfully updated 'registry'
> Transitioned 9998 promises in: 154.775647ms
> I0304 23:37:19.453868 33235 registrar_tests.cpp:251] Removed 10,000 slaves in 3.577429332secs
> [ OK ] RegistrarTest.DISABLED_performance (23791 ms)
>
>
> Thanks,
>
> Ben Mahler
>
>
Re: Review Request 18758: Performance optimizations to the Registrar.
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18758/
-----------------------------------------------------------
(Updated March 6, 2014, 8:13 p.m.)
Review request for mesos, Benjamin Hindman and Vinod Kone.
Changes
-------
Rebase, NNFR.
Bugs: MESOS-764
https://issues.apache.org/jira/browse/MESOS-764
Repository: mesos-git
Description
-------
The bottleneck in the Registrar is the copying of the Registry. For a large number of slaves and operations, this becomes prohibitively expensive. This patch therefore eliminates copying when applying the operations. Optimizations to Future also eliminated copying in _update.
Further optimizations will be explored for larger clusters, this was currently tested with 10,000 slaves.
Diffs (updated)
-----
src/master/registrar.cpp 37337c07b24a96e71910b7c83085d159361a1188
Diff: https://reviews.apache.org/r/18758/diff/
Testing
-------
Prior to this patch and the Future::get optimizations, the performance benchmark I wrote took over an hour for 10,000 slaves.
With these optimizations 10,000 slaves can be handled quite efficiently:
Key numbers with gcc 4.1.2 on a 2.4GHz 12 core machine:
I0304 23:37:06.024024 33235 registrar_tests.cpp:229] Admitted 10,000 slaves in 10.04498586secs
I0304 23:37:15.853411 33235 registrar_tests.cpp:237] Readmitted 10,000 slaves in 9.829011689secs
I0304 23:37:15.876322 33235 registrar_tests.cpp:243] Recovered 10,000 slaves in 22.715892ms
I0304 23:37:19.453868 33235 registrar_tests.cpp:251] Removed 10,000 slaves in 3.577429332secs
[ RUN ] RegistrarTest.DISABLED_performance
I0304 23:36:55.685456 33261 registrar.cpp:218] Recovering registrar
I0304 23:36:55.801048 33252 registrar.cpp:381] Attempting to update the 'registry'
Applied 1 operations in: 1.235673ms
Stored Registry in: 36.199255ms
I0304 23:36:55.837477 33271 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 66813ns
I0304 23:36:55.972564 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 1 operations in: 1.573455ms
Stored Registry in: 30.215382ms
I0304 23:36:56.002907 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 113129ns
I0304 23:36:56.003094 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 146 operations in: 8.579203ms
Stored Registry in: 174.710644ms
I0304 23:36:56.179323 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 146 promises in: 4.656175ms
I0304 23:36:56.184154 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 574 operations in: 73.828182ms
Stored Registry in: 527.540062ms
I0304 23:36:56.744689 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 574 promises in: 13.873625ms
I0304 23:36:56.760572 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 2400 operations in: 907.070761ms
Stored Registry in: 268.604653ms
I0304 23:36:57.778074 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 2400 promises in: 69.867013ms
I0304 23:36:57.858157 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 6879 operations in: 7.910334133secs
Stored Registry in: 990.759579ms
I0304 23:37:05.795042 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 6879 promises in: 221.701612ms
I0304 23:37:06.024024 33235 registrar_tests.cpp:229] Admitted 10,000 slaves in 10.04498586secs
I0304 23:37:06.125613 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 1 operations in: 852.076566ms
Stored Registry in: 1.384153523secs
I0304 23:37:07.576215 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 191.668241ms
I0304 23:37:07.768013 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 993 operations in: 923.901654ms
Stored Registry in: 953.378897ms
I0304 23:37:08.827229 33266 registrar.cpp:425] Successfully updated 'registry'
Transitioned 993 promises in: 164.437762ms
I0304 23:37:08.992169 33266 registrar.cpp:381] Attempting to update the 'registry'
Applied 9006 operations in: 6.081593589secs
Stored Registry in: 896.142182ms
I0304 23:37:15.397238 33257 registrar.cpp:425] Successfully updated 'registry'
Transitioned 9006 promises in: 427.267246ms
I0304 23:37:15.853411 33235 registrar_tests.cpp:237] Readmitted 10,000 slaves in 9.829011689secs
I0304 23:37:15.876085 33257 registrar.cpp:218] Recovering registrar
I0304 23:37:15.876322 33235 registrar_tests.cpp:243] Recovered 10,000 slaves in 22.715892ms
I0304 23:37:15.876656 33264 registrar.cpp:381] Attempting to update the 'registry'
Applied 1 operations in: 819.303422ms
Stored Registry in: 1.383689258secs
I0304 23:37:17.329473 33264 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 189.505006ms
I0304 23:37:17.541276 33264 registrar.cpp:381] Attempting to update the 'registry'
Applied 1 operations in: 921.761221ms
Stored Registry in: 986.688995ms
I0304 23:37:18.588706 33264 registrar.cpp:425] Successfully updated 'registry'
Transitioned 1 promises in: 179.771104ms
I0304 23:37:18.768606 33264 registrar.cpp:381] Attempting to update the 'registry'
Applied 9998 operations in: 441.652539ms
Stored Registry in: 209.866548ms
I0304 23:37:19.298931 33265 registrar.cpp:425] Successfully updated 'registry'
Transitioned 9998 promises in: 154.775647ms
I0304 23:37:19.453868 33235 registrar_tests.cpp:251] Removed 10,000 slaves in 3.577429332secs
[ OK ] RegistrarTest.DISABLED_performance (23791 ms)
Thanks,
Ben Mahler