You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-dev@jackrabbit.apache.org by Jukka Zitting <ju...@gmail.com> on 2013/09/25 02:08:15 UTC
Re: Some more benchmarks
Hi,
Updating this thread with the latest numbers. No major changes on
these benchmarks:
# ReadPropertyTest 90%
Jackrabbit 48
Oak-Default 39
Oak-Mongo 41
Oak-Segment 41
Oak-Tar 42
# SmallFileReadTest 90%
Jackrabbit 91
Oak-Default 19
Oak-Mongo 19
Oak-Segment 18
Oak-Tar 18
# SmallFileWriteTest 90%
Jackrabbit 386
Oak-Default 425
Oak-Mongo 963
Oak-Segment 180
Oak-Tar 88
Full details in https://gist.github.com/jukka/6693063.
I'll add a few more benchmarks to my test script.
BR,
Jukka Zitting
Re: Some more benchmarks
Posted by Stefan Guggisberg <st...@gmail.com>.
On Tue, Jul 1, 2014 at 8:37 PM, Jukka Zitting <ju...@gmail.com> wrote:
> Hi,
>
> On Tue, Jul 1, 2014 at 9:38 AM, Jukka Zitting <ju...@gmail.com> wrote:
>> I also tried including MongoMK results, but the benchmark got stuck in
>> ConcurrentReadTest. I'll re-try today and will file a bug if I can
>> reproduce the problem.
>
> I guess it was a transient problem. Here are the results with
> Oak-Mongo included:
>
> Summary (90%, lower is better)
>
> Benchmark Jackrabbit Oak-Mongo Oak-Tar
> -------------------------------------------------------------
> ReadPropertyTest 45 4 4
> SetPropertyTest 1179 2398 119
> SmallFileReadTest 47 9 7
> SmallFileWriteTest 182 530 43
> ConcurrentReadTest 1201 1247 710
> ConcurrentReadWriteTest 1900 2321 775
> ConcurrentWriteReadTest 1009 354 108
> ConcurrentWriteTest 532 553 101
wow, very impressive, congrats!
cheers
stefan
>
> I updated the gist at
> https://gist.github.com/jukka/078bd524aa0ba36b184b with full details.
>
> The general message here is to use TarMK for maximum single-node
> performance and MongoMK for scalability and throughput across multiple
> cluster nodes.
>
> BR,
>
> Jukka Zitting
Re: Some more benchmarks
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Tue, Jul 1, 2014 at 9:38 AM, Jukka Zitting <ju...@gmail.com> wrote:
> I also tried including MongoMK results, but the benchmark got stuck in
> ConcurrentReadTest. I'll re-try today and will file a bug if I can
> reproduce the problem.
I guess it was a transient problem. Here are the results with
Oak-Mongo included:
Summary (90%, lower is better)
Benchmark Jackrabbit Oak-Mongo Oak-Tar
-------------------------------------------------------------
ReadPropertyTest 45 4 4
SetPropertyTest 1179 2398 119
SmallFileReadTest 47 9 7
SmallFileWriteTest 182 530 43
ConcurrentReadTest 1201 1247 710
ConcurrentReadWriteTest 1900 2321 775
ConcurrentWriteReadTest 1009 354 108
ConcurrentWriteTest 532 553 101
I updated the gist at
https://gist.github.com/jukka/078bd524aa0ba36b184b with full details.
The general message here is to use TarMK for maximum single-node
performance and MongoMK for scalability and throughput across multiple
cluster nodes.
BR,
Jukka Zitting
Re: Some more benchmarks
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
I'm resurrecting this thread with some new findings. I re-ran many of
the benchmarks we've been following, pitting Jackrabbit 2.8.0 against
Oak 1.0.1 with TarMK. The results look pretty nice:
Summary (90%, lower is better)
Benchmark Jackrabbit Oak-Tar
--------------------------------------------------
ReadPropertyTest 45 4
SetPropertyTest 1179 119
SmallFileReadTest 47 7
SmallFileWriteTest 182 43
ConcurrentReadTest 1201 710
ConcurrentReadWriteTest 1900 775
ConcurrentWriteReadTest 1009 108
ConcurrentWriteTest 532 101
See https://gist.github.com/jukka/078bd524aa0ba36b184b for details.
I also tried including MongoMK results, but the benchmark got stuck in
ConcurrentReadTest. I'll re-try today and will file a bug if I can
reproduce the problem.
BR,
Jukka Zitting
Re: Some more benchmarks
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Tue, Sep 24, 2013 at 11:19 PM, Jukka Zitting <ju...@gmail.com> wrote:
> On Tue, Sep 24, 2013 at 10:47 PM, Jukka Zitting <ju...@gmail.com> wrote:
>> The concurrent read and read/write test cases look like more attention
>> is needed on the test code, as it's currently hard to interpret the
>> results. I'll see what I can do there.
>
> It turns out most of the reported time was going to login() calls (see
> OAK-634). I refactored the tests in revision 1526092 so that the login
> calls won't affect the performance measurements.
There were a few other systemic issues with the concurrency benchmarks
(like the background threads running at lower priority and doing less
work). I made some further improvements, and the numbers now look like
this:
# ConcurrentReadTest 90%
Jackrabbit 4132
Oak-Default 2031
Oak-Mongo 2116
Oak-Segment 2258
Oak-Tar 2580
# ConcurrentReadWriteTest 90%
Jackrabbit 3192
Oak-Default 3600
Oak-Mongo 4596
Oak-Segment 2605
Oak-Tar 2876
# ConcurrentWriteReadTest 90%
Jackrabbit 2770
Oak-Default 875
Oak-Mongo 1243
Oak-Segment 565
Oak-Tar 405
# ConcurrentWriteTest 90%
Jackrabbit 597
Oak-Default 2141
Oak-Mongo 1166
Oak-Segment 558
Oak-Tar 348
Full details in https://gist.github.com/jukka/6748243.
BR,
Jukka Zitting
Re: Some more benchmarks
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Tue, Sep 24, 2013 at 10:47 PM, Jukka Zitting <ju...@gmail.com> wrote:
> The concurrent read and read/write test cases look like more attention
> is needed on the test code, as it's currently hard to interpret the
> results. I'll see what I can do there.
It turns out most of the reported time was going to login() calls (see
OAK-634). I refactored the tests in revision 1526092 so that the login
calls won't affect the performance measurements. As a result the
numbers look much better:
# ConcurrentReadTest 90%
Jackrabbit 447
Oak-Default 286
Oak-Mongo 240
Oak-Segment 245
Oak-Tar 252
# ConcurrentReadWriteTest 90%
Jackrabbit 383
Oak-Default 263
Oak-Mongo 270
Oak-Segment 280
Oak-Tar 268
BR,
Jukka Zitting
Re: Some more benchmarks
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Tue, Sep 24, 2013 at 8:08 PM, Jukka Zitting <ju...@gmail.com> wrote:
> I'll add a few more benchmarks to my test script.
Here are results for three more benchmarks:
# SetPropertyTest 90%
Jackrabbit 740
Oak-Default 916
Oak-Mongo 5973
Oak-Segment 2386
Oak-Tar 728
# ConcurrentReadTest 90%
Jackrabbit 25656
Oak-Default 32840
Oak-Mongo 33928
Oak-Segment 35522
Oak-Tar 35354
# ConcurrentReadWriteTest 90%
Jackrabbit 19280
Oak-Default 39289
Oak-Mongo 48078
Oak-Segment 37384
Oak-Tar 37165
The SetPropertyTest measures extremely small transactions (single
property change), which makes the networking overhead in Oak-Mongo and
Oak-Segment more prominent. Other than that the results are fairly
good.
The concurrent read and read/write test cases look like more attention
is needed on the test code, as it's currently hard to interpret the
results. I'll see what I can do there.
BR,
Jukka Zitting