You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-dev@jackrabbit.apache.org by Jukka Zitting <ju...@gmail.com> on 2013/07/02 22:11:21 UTC
Re: Some more benchmarks
Hi,
On Fri, May 31, 2013 at 3:14 PM, Jukka Zitting <ju...@gmail.com> wrote:
> On Fri, Apr 26, 2013 at 2:12 PM, Jukka Zitting <ju...@gmail.com> wrote:
>> On Wed, Mar 27, 2013 at 11:41 AM, Jukka Zitting <ju...@gmail.com> wrote:
>>> Here's a few more simple benchmark results to show where we are:
>>
>> Updated numbers with latest Oak:
>
> And another one:
Here we go again:
# ReadPropertyTest 90%
Jackrabbit 48
Oak-Default 38
Oak-Mongo 39
Oak-Segment 41
Oak-Tar 40
# SmallFileReadTest 90%
Jackrabbit 94
Oak-Default 258
Oak-Mongo 421
Oak-Segment 23
Oak-Tar 20
# SmallFileWriteTest 90%
Jackrabbit 424
Oak-Default 349
Oak-Mongo 1376
Oak-Segment 257
Oak-Tar 116
For simplicy I've only included the 90th percentile figure (smaller is
better). See https://gist.github.com/jukka/5912460 for the full
details.
The ReadPropertyTest figures were again lagging behind those of
Jackrabbit, but my changes earlier today got us back to the same
range. However, we've still regressed somewhat from the level we
reached in early June.
BR,
Jukka Zitting
Re: Some more benchmarks
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Thu, Jul 4, 2013 at 1:12 PM, Jukka Zitting <ju...@gmail.com> wrote:
> On Wed, Jul 3, 2013 at 5:34 PM, Thomas Mueller <mu...@adobe.com> wrote:
>> I think it's because all binaries are loaded from the backend (no
>> caching). I bumped the blob cache size from 8 MB to 16 MB, let's see if
>> this helps.
>
> Yes, that could be it (I'll run the tests again soon to confirm).
Indeed the numbers look great now:
# SmallFileReadTest min 10% 50% 90% max N
Oak-Mongo 14 15 16 17 56 3790
BR,
Jukka Zitting
Re: Some more benchmarks
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Wed, Jul 3, 2013 at 5:34 PM, Thomas Mueller <mu...@adobe.com> wrote:
> I think it's because all binaries are loaded from the backend (no
> caching). I bumped the blob cache size from 8 MB to 16 MB, let's see if
> this helps.
Yes, that could be it (I'll run the tests again soon to confirm). The
working set of the test is about 10MB in size and scanned linearly, so
each iteration would end up flushing a cache that's less than 10MB in
size.
BR,
Jukka Zitting
Re: Some more benchmarks
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Wed, Jul 3, 2013 at 11:54 AM, Jukka Zitting <ju...@gmail.com> wrote:
> On Wed, Jul 3, 2013 at 11:22 AM, Thomas Mueller <mu...@adobe.com> wrote:
>> I usually look at "N" first :-)
>
> It's also a good measure.
Actually not that good, as only the lower limit on the amount of time
over which those N iterations happen is defined, so it's for example
not possible to compute an accurate mean execution time from the
reported N. Also, the N figure also covers the before/afterTest()
methods, which are not included in the other statistics and that which
aren't really within the scope of the functionality that a benchmark
intends to measure. The reason I originally included N in the output
was to given an idea about the statistical significance of the other
figures.
Perhaps we should replace the median (50%) or the 10th percentile (not
a very useful figure) with a more exactly calculated mean execution
time, as that would better represent the information for which N
currently only acts as a rough proxy.
BR,
Jukka Zitting
Re: Some more benchmarks
Posted by Thomas Mueller <mu...@adobe.com>.
Hi,
>I don't know what's dragging the performance in the SmallFileReadTest,
I think it's because all binaries are loaded from the backend (no
caching). I bumped the blob cache size from 8 MB to 16 MB, let's see if
this helps.
Regards,
Thomas
Re: Some more benchmarks
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Wed, Jul 3, 2013 at 11:22 AM, Thomas Mueller <mu...@adobe.com> wrote:
>> I've only included the 90th percentile
>
> I usually look at "N" first :-)
It's also a good measure. I like the 90th percentile better as it
filters out outliers that may otherwise weight pretty heavily on the
total or average execution time. Of course, as you note below, it's
good to pay attention also to such cases.
> There is one strange result: SmallFileWriteTest; Oak-Segment: 90%=257,
> max=14763 - Maybe the warmup phase is too short, or the test isn't that
> great?
Good catch. I ran the benchmark a few more times, and the max time is
always pretty high. It shouldn't be about the warmup phase, as the
time in this test should be governed by the blob I/O. I'll try to
figure out what's causing such worst case behavior.
> As for SmallFileReadTest and SmallFileWriteTest with Oak-Mongo: I think I
> know what the problem is; it doesn't seem to be related to BLOB handling
> at all (actually performance is the same without the BLOB), but partially
> related to the "split documents" that should be added in the near future.
> Also, it seems to be partially related to what the test does (repeatedly
> adding and removing the same nodes).
Right. The semantics of the SmallFileWriteTest should be the same if
the test root name was different for each test iteration, which should
avoid the slit document edge case. I adjusted the test (see patch
below), and the numbers do look a bit better but not radically so:
# SmallFileWriteTest min 10% 50% 90% max N
Oak-Mongo 577 591 708 1198 1585 33
I don't know what's dragging the performance in the SmallFileReadTest,
as there the nodes are created just once at the beginning of the
benchmark.
BR,
Jukka Zitting
----
diff --git a/oak-run/src/main/java/org/apache/jackrabbit/oak/benchmark/SmallFileWriteTest.java
b/oak-run/src/main/java/org/apache/jackrabbit/oak/benchmark/SmallFileWriteTest.java
index 7d15b00..c5f2ec8 100644
--- a/oak-run/src/main/java/org/apache/jackrabbit/oak/benchmark/SmallFileWriteTest.java
+++ b/oak-run/src/main/java/org/apache/jackrabbit/oak/benchmark/SmallFileWriteTest.java
@@ -32,6 +32,8 @@ public class SmallFileWriteTest extends AbstractTest {
private Node root;
+ private long count = 0;
+
@Override
public void beforeSuite() throws RepositoryException {
session = loginWriter();
@@ -39,7 +41,7 @@ public class SmallFileWriteTest extends AbstractTest {
@Override
public void beforeTest() throws RepositoryException {
- root = session.getRootNode().addNode("SmallFileWriteTest",
"nt:folder");
+ root = session.getRootNode().addNode("SmallFileWriteTest" +
count++, "nt:folder");
session.save();
}
Re: Some more benchmarks
Posted by Stefan Guggisberg <st...@gmail.com>.
On Tue, Jul 1, 2014 at 8:37 PM, Jukka Zitting <ju...@gmail.com> wrote:
> Hi,
>
> On Tue, Jul 1, 2014 at 9:38 AM, Jukka Zitting <ju...@gmail.com> wrote:
>> I also tried including MongoMK results, but the benchmark got stuck in
>> ConcurrentReadTest. I'll re-try today and will file a bug if I can
>> reproduce the problem.
>
> I guess it was a transient problem. Here are the results with
> Oak-Mongo included:
>
> Summary (90%, lower is better)
>
> Benchmark Jackrabbit Oak-Mongo Oak-Tar
> -------------------------------------------------------------
> ReadPropertyTest 45 4 4
> SetPropertyTest 1179 2398 119
> SmallFileReadTest 47 9 7
> SmallFileWriteTest 182 530 43
> ConcurrentReadTest 1201 1247 710
> ConcurrentReadWriteTest 1900 2321 775
> ConcurrentWriteReadTest 1009 354 108
> ConcurrentWriteTest 532 553 101
wow, very impressive, congrats!
cheers
stefan
>
> I updated the gist at
> https://gist.github.com/jukka/078bd524aa0ba36b184b with full details.
>
> The general message here is to use TarMK for maximum single-node
> performance and MongoMK for scalability and throughput across multiple
> cluster nodes.
>
> BR,
>
> Jukka Zitting
Re: Some more benchmarks
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Tue, Jul 1, 2014 at 9:38 AM, Jukka Zitting <ju...@gmail.com> wrote:
> I also tried including MongoMK results, but the benchmark got stuck in
> ConcurrentReadTest. I'll re-try today and will file a bug if I can
> reproduce the problem.
I guess it was a transient problem. Here are the results with
Oak-Mongo included:
Summary (90%, lower is better)
Benchmark Jackrabbit Oak-Mongo Oak-Tar
-------------------------------------------------------------
ReadPropertyTest 45 4 4
SetPropertyTest 1179 2398 119
SmallFileReadTest 47 9 7
SmallFileWriteTest 182 530 43
ConcurrentReadTest 1201 1247 710
ConcurrentReadWriteTest 1900 2321 775
ConcurrentWriteReadTest 1009 354 108
ConcurrentWriteTest 532 553 101
I updated the gist at
https://gist.github.com/jukka/078bd524aa0ba36b184b with full details.
The general message here is to use TarMK for maximum single-node
performance and MongoMK for scalability and throughput across multiple
cluster nodes.
BR,
Jukka Zitting
Re: Some more benchmarks
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
I'm resurrecting this thread with some new findings. I re-ran many of
the benchmarks we've been following, pitting Jackrabbit 2.8.0 against
Oak 1.0.1 with TarMK. The results look pretty nice:
Summary (90%, lower is better)
Benchmark Jackrabbit Oak-Tar
--------------------------------------------------
ReadPropertyTest 45 4
SetPropertyTest 1179 119
SmallFileReadTest 47 7
SmallFileWriteTest 182 43
ConcurrentReadTest 1201 710
ConcurrentReadWriteTest 1900 775
ConcurrentWriteReadTest 1009 108
ConcurrentWriteTest 532 101
See https://gist.github.com/jukka/078bd524aa0ba36b184b for details.
I also tried including MongoMK results, but the benchmark got stuck in
ConcurrentReadTest. I'll re-try today and will file a bug if I can
reproduce the problem.
BR,
Jukka Zitting
Re: Some more benchmarks
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Tue, Sep 24, 2013 at 11:19 PM, Jukka Zitting <ju...@gmail.com> wrote:
> On Tue, Sep 24, 2013 at 10:47 PM, Jukka Zitting <ju...@gmail.com> wrote:
>> The concurrent read and read/write test cases look like more attention
>> is needed on the test code, as it's currently hard to interpret the
>> results. I'll see what I can do there.
>
> It turns out most of the reported time was going to login() calls (see
> OAK-634). I refactored the tests in revision 1526092 so that the login
> calls won't affect the performance measurements.
There were a few other systemic issues with the concurrency benchmarks
(like the background threads running at lower priority and doing less
work). I made some further improvements, and the numbers now look like
this:
# ConcurrentReadTest 90%
Jackrabbit 4132
Oak-Default 2031
Oak-Mongo 2116
Oak-Segment 2258
Oak-Tar 2580
# ConcurrentReadWriteTest 90%
Jackrabbit 3192
Oak-Default 3600
Oak-Mongo 4596
Oak-Segment 2605
Oak-Tar 2876
# ConcurrentWriteReadTest 90%
Jackrabbit 2770
Oak-Default 875
Oak-Mongo 1243
Oak-Segment 565
Oak-Tar 405
# ConcurrentWriteTest 90%
Jackrabbit 597
Oak-Default 2141
Oak-Mongo 1166
Oak-Segment 558
Oak-Tar 348
Full details in https://gist.github.com/jukka/6748243.
BR,
Jukka Zitting
Re: Some more benchmarks
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Tue, Sep 24, 2013 at 10:47 PM, Jukka Zitting <ju...@gmail.com> wrote:
> The concurrent read and read/write test cases look like more attention
> is needed on the test code, as it's currently hard to interpret the
> results. I'll see what I can do there.
It turns out most of the reported time was going to login() calls (see
OAK-634). I refactored the tests in revision 1526092 so that the login
calls won't affect the performance measurements. As a result the
numbers look much better:
# ConcurrentReadTest 90%
Jackrabbit 447
Oak-Default 286
Oak-Mongo 240
Oak-Segment 245
Oak-Tar 252
# ConcurrentReadWriteTest 90%
Jackrabbit 383
Oak-Default 263
Oak-Mongo 270
Oak-Segment 280
Oak-Tar 268
BR,
Jukka Zitting
Re: Some more benchmarks
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Tue, Sep 24, 2013 at 8:08 PM, Jukka Zitting <ju...@gmail.com> wrote:
> I'll add a few more benchmarks to my test script.
Here are results for three more benchmarks:
# SetPropertyTest 90%
Jackrabbit 740
Oak-Default 916
Oak-Mongo 5973
Oak-Segment 2386
Oak-Tar 728
# ConcurrentReadTest 90%
Jackrabbit 25656
Oak-Default 32840
Oak-Mongo 33928
Oak-Segment 35522
Oak-Tar 35354
# ConcurrentReadWriteTest 90%
Jackrabbit 19280
Oak-Default 39289
Oak-Mongo 48078
Oak-Segment 37384
Oak-Tar 37165
The SetPropertyTest measures extremely small transactions (single
property change), which makes the networking overhead in Oak-Mongo and
Oak-Segment more prominent. Other than that the results are fairly
good.
The concurrent read and read/write test cases look like more attention
is needed on the test code, as it's currently hard to interpret the
results. I'll see what I can do there.
BR,
Jukka Zitting
Re: Some more benchmarks
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
Updating this thread with the latest numbers. No major changes on
these benchmarks:
# ReadPropertyTest 90%
Jackrabbit 48
Oak-Default 39
Oak-Mongo 41
Oak-Segment 41
Oak-Tar 42
# SmallFileReadTest 90%
Jackrabbit 91
Oak-Default 19
Oak-Mongo 19
Oak-Segment 18
Oak-Tar 18
# SmallFileWriteTest 90%
Jackrabbit 386
Oak-Default 425
Oak-Mongo 963
Oak-Segment 180
Oak-Tar 88
Full details in https://gist.github.com/jukka/6693063.
I'll add a few more benchmarks to my test script.
BR,
Jukka Zitting
Re: Some more benchmarks
Posted by Thomas Mueller <mu...@adobe.com>.
Hi,
Thanks a lot!
> I've only included the 90th percentile
I usually look at "N" first :-)
There is one strange result: SmallFileWriteTest; Oak-Segment: 90%=257,
max=14763 - Maybe the warmup phase is too short, or the test isn't that
great?
As for SmallFileReadTest and SmallFileWriteTest with Oak-Mongo: I think I
know what the problem is; it doesn't seem to be related to BLOB handling
at all (actually performance is the same without the BLOB), but partially
related to the "split documents" that should be added in the near future.
Also, it seems to be partially related to what the test does (repeatedly
adding and removing the same nodes).
Regards,
Thomas
On 7/2/13 10:11 PM, "Jukka Zitting" <ju...@gmail.com> wrote:
>Hi,
>
>On Fri, May 31, 2013 at 3:14 PM, Jukka Zitting <ju...@gmail.com>
>wrote:
>> On Fri, Apr 26, 2013 at 2:12 PM, Jukka Zitting
>><ju...@gmail.com> wrote:
>>> On Wed, Mar 27, 2013 at 11:41 AM, Jukka Zitting
>>><ju...@gmail.com> wrote:
>>>> Here's a few more simple benchmark results to show where we are:
>>>
>>> Updated numbers with latest Oak:
>>
>> And another one:
>
>Here we go again:
>
> # ReadPropertyTest 90%
> Jackrabbit 48
> Oak-Default 38
> Oak-Mongo 39
> Oak-Segment 41
> Oak-Tar 40
>
> # SmallFileReadTest 90%
> Jackrabbit 94
> Oak-Default 258
> Oak-Mongo 421
> Oak-Segment 23
> Oak-Tar 20
>
> # SmallFileWriteTest 90%
> Jackrabbit 424
> Oak-Default 349
> Oak-Mongo 1376
> Oak-Segment 257
> Oak-Tar 116
>
>For simplicy I've only included the 90th percentile figure (smaller is
>better). See https://gist.github.com/jukka/5912460 for the full
>details.
>
>The ReadPropertyTest figures were again lagging behind those of
>Jackrabbit, but my changes earlier today got us back to the same
>range. However, we've still regressed somewhat from the level we
>reached in early June.
>
>BR,
>
>Jukka Zitting