You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jackrabbit.apache.org by "Jukka Zitting (JIRA)" <ji...@apache.org> on 2010/08/03 15:07:15 UTC
[jira] Created: (JCR-2695) Jackrabbit performance test suite
Jackrabbit performance test suite
---------------------------------
Key: JCR-2695
URL: https://issues.apache.org/jira/browse/JCR-2695
Project: Jackrabbit Content Repository
Issue Type: Improvement
Components: jackrabbit-core
Reporter: Jukka Zitting
Assignee: Jukka Zitting
I'd like to set up a multi-version performance test suite inside jackrabbit-core/src/test/performance, similar to the compatibility test suite we added in JCR-2631. This performance test suite would produce comparable performance numbers for a number of simple benchmark tests across different Jackrabbit versions, including the latest snapshot.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Created: (JCR-2695) Jackrabbit performance test suite
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Tue, Aug 3, 2010 at 4:00 PM, Felix Meschberger <fm...@gmail.com> wrote:
> I agree, that this is certainly very interesting. But is this really
> something which should be located inside Jackrabbit Core ?
I moved the performance test suite outside jackrabbit-core in revision 983710.
On a related note, we might want to migrate the
jackrabbit-jcr-benchmark component from JCR commons back to Jackrabbit
trunk to avoid the need to work across multiple checkouts.
BR,
Jukka Zitting
Re: [jira] Created: (JCR-2695) Jackrabbit performance test suite
Posted by Felix Meschberger <fm...@gmail.com>.
Hi,
I agree, that this is certainly very interesting. But is this really
something which should be located inside Jackrabbit Core ?
I rather look at it as something which measures performance of a
repository from the outside. As such it is IMHO not part of a concrete
JCR implementation module like jackrabbit-core.
Regards
Felix
On 03.08.2010 15:07, Jukka Zitting (JIRA) wrote:
> Jackrabbit performance test suite
> ---------------------------------
>
> Key: JCR-2695
> URL: https://issues.apache.org/jira/browse/JCR-2695
> Project: Jackrabbit Content Repository
> Issue Type: Improvement
> Components: jackrabbit-core
> Reporter: Jukka Zitting
> Assignee: Jukka Zitting
>
>
> I'd like to set up a multi-version performance test suite inside jackrabbit-core/src/test/performance, similar to the compatibility test suite we added in JCR-2631. This performance test suite would produce comparable performance numbers for a number of simple benchmark tests across different Jackrabbit versions, including the latest snapshot.
>
>
>
[jira] Commented: (JCR-2695) Jackrabbit performance test suite
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-2695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895216#action_12895216 ]
Jukka Zitting commented on JCR-2695:
------------------------------------
Currently the jcr-benchmark suite currently uses the commons-math class SummaryStatistics which doesn't do percentiles, but it should be pretty straightforward to switch to using DescriptiveStatistics instead. The above plots show the mean value and standard deviations. But switching to something like the median plus 10th and 90th percentiles might give a more accurate picture. Alternatively we could just do a scatter plot of the entire data set.
About the 2.x figures, the increase in session startup times is probably due to the extra JCR 2.0 features (access control?) we're starting up. Note that the performance drop starts already in 1.6 where we started introducing new JCR 2.0 features. In any case starting up a single session still takes only about 0.1ms on my computer, which seems OK to me.
The switch to using the FileDataStore by default in Jackrabbit 2.0 seems to explain the changes in file handling performance. We're pretty good with both reading and writing large files, but the handling of smaller files suffers somewhat from the way the FileDataStore uses intermediate directories. Perhaps we should consider putting all smaller binaries inside one big file instead.
> Jackrabbit performance test suite
> ---------------------------------
>
> Key: JCR-2695
> URL: https://issues.apache.org/jira/browse/JCR-2695
> Project: Jackrabbit Content Repository
> Issue Type: Improvement
> Components: jackrabbit-core
> Reporter: Jukka Zitting
> Assignee: Jukka Zitting
> Fix For: 2.2.0
>
> Attachments: bigread.png, bigwrite.png, login.png, logout.png, smallread.png, smallwrite.png
>
>
> I'd like to set up a multi-version performance test suite inside jackrabbit-core/src/test/performance, similar to the compatibility test suite we added in JCR-2631. This performance test suite would produce comparable performance numbers for a number of simple benchmark tests across different Jackrabbit versions, including the latest snapshot.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (JCR-2695) Jackrabbit performance test suite
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-2695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jukka Zitting updated JCR-2695:
-------------------------------
Attachment: login.png
logout.png
smallread.png
See the attached plots for a few example results I got from running the current basic performance tests over last night.
> Jackrabbit performance test suite
> ---------------------------------
>
> Key: JCR-2695
> URL: https://issues.apache.org/jira/browse/JCR-2695
> Project: Jackrabbit Content Repository
> Issue Type: Improvement
> Components: jackrabbit-core
> Reporter: Jukka Zitting
> Assignee: Jukka Zitting
> Attachments: bigread.png, bigwrite.png, login.png, logout.png, smallread.png, smallwrite.png
>
>
> I'd like to set up a multi-version performance test suite inside jackrabbit-core/src/test/performance, similar to the compatibility test suite we added in JCR-2631. This performance test suite would produce comparable performance numbers for a number of simple benchmark tests across different Jackrabbit versions, including the latest snapshot.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (JCR-2695) Jackrabbit performance test suite
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-2695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jukka Zitting updated JCR-2695:
-------------------------------
Attachment: smallwrite.png
bigread.png
bigwrite.png
A few more plots.
> Jackrabbit performance test suite
> ---------------------------------
>
> Key: JCR-2695
> URL: https://issues.apache.org/jira/browse/JCR-2695
> Project: Jackrabbit Content Repository
> Issue Type: Improvement
> Components: jackrabbit-core
> Reporter: Jukka Zitting
> Assignee: Jukka Zitting
> Attachments: bigread.png, bigwrite.png, login.png, logout.png, smallread.png, smallwrite.png
>
>
> I'd like to set up a multi-version performance test suite inside jackrabbit-core/src/test/performance, similar to the compatibility test suite we added in JCR-2631. This performance test suite would produce comparable performance numbers for a number of simple benchmark tests across different Jackrabbit versions, including the latest snapshot.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (JCR-2695) Jackrabbit performance test
suite
Posted by "Alexander Klimetschek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-2695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895198#action_12895198 ]
Alexander Klimetschek edited comment on JCR-2695 at 8/4/10 6:06 AM:
--------------------------------------------------------------------
Looks cool! (Well, the test suite, not necessarily the graphs for versions >= 2.0 ;-))
Interesting would also be the 90th percentile value of all runs of a certain test, instead of the mean.
was (Author: alexander.klimetschek):
Looks cool! Interesting would also be the 90th percentile value of all runs of a certain test.
> Jackrabbit performance test suite
> ---------------------------------
>
> Key: JCR-2695
> URL: https://issues.apache.org/jira/browse/JCR-2695
> Project: Jackrabbit Content Repository
> Issue Type: Improvement
> Components: jackrabbit-core
> Reporter: Jukka Zitting
> Assignee: Jukka Zitting
> Fix For: 2.2.0
>
> Attachments: bigread.png, bigwrite.png, login.png, logout.png, smallread.png, smallwrite.png
>
>
> I'd like to set up a multi-version performance test suite inside jackrabbit-core/src/test/performance, similar to the compatibility test suite we added in JCR-2631. This performance test suite would produce comparable performance numbers for a number of simple benchmark tests across different Jackrabbit versions, including the latest snapshot.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (JCR-2695) Jackrabbit performance test suite
Posted by "Alexander Klimetschek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-2695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895198#action_12895198 ]
Alexander Klimetschek commented on JCR-2695:
--------------------------------------------
Looks cool! Interesting would also be the 90th percentile value of all runs of a certain test.
> Jackrabbit performance test suite
> ---------------------------------
>
> Key: JCR-2695
> URL: https://issues.apache.org/jira/browse/JCR-2695
> Project: Jackrabbit Content Repository
> Issue Type: Improvement
> Components: jackrabbit-core
> Reporter: Jukka Zitting
> Assignee: Jukka Zitting
> Fix For: 2.2.0
>
> Attachments: bigread.png, bigwrite.png, login.png, logout.png, smallread.png, smallwrite.png
>
>
> I'd like to set up a multi-version performance test suite inside jackrabbit-core/src/test/performance, similar to the compatibility test suite we added in JCR-2631. This performance test suite would produce comparable performance numbers for a number of simple benchmark tests across different Jackrabbit versions, including the latest snapshot.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (JCR-2695) Jackrabbit performance test suite
Posted by "Thomas Mueller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-2695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920467#action_12920467 ]
Thomas Mueller commented on JCR-2695:
-------------------------------------
I created JCR-2781 to speed up the FileDataStore.
> Jackrabbit performance test suite
> ---------------------------------
>
> Key: JCR-2695
> URL: https://issues.apache.org/jira/browse/JCR-2695
> Project: Jackrabbit Content Repository
> Issue Type: Improvement
> Components: jackrabbit-core
> Reporter: Jukka Zitting
> Assignee: Jukka Zitting
> Fix For: 2.2.0
>
> Attachments: bigread.png, bigwrite.png, login.png, logout.png, SimpleSearchTest.png, smallread.png, smallwrite.png
>
>
> I'd like to set up a multi-version performance test suite inside jackrabbit-core/src/test/performance, similar to the compatibility test suite we added in JCR-2631. This performance test suite would produce comparable performance numbers for a number of simple benchmark tests across different Jackrabbit versions, including the latest snapshot.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (JCR-2695) Jackrabbit performance test suite
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-2695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jukka Zitting resolved JCR-2695.
--------------------------------
Fix Version/s: 2.2.0
Resolution: Fixed
Resolving this as fixed since we now have a basic performance test suite in place. We can use followup issues to add new test cases or to fix performance issues highlighted by existing tests.
> Jackrabbit performance test suite
> ---------------------------------
>
> Key: JCR-2695
> URL: https://issues.apache.org/jira/browse/JCR-2695
> Project: Jackrabbit Content Repository
> Issue Type: Improvement
> Components: jackrabbit-core
> Reporter: Jukka Zitting
> Assignee: Jukka Zitting
> Fix For: 2.2.0
>
> Attachments: bigread.png, bigwrite.png, login.png, logout.png, smallread.png, smallwrite.png
>
>
> I'd like to set up a multi-version performance test suite inside jackrabbit-core/src/test/performance, similar to the compatibility test suite we added in JCR-2631. This performance test suite would produce comparable performance numbers for a number of simple benchmark tests across different Jackrabbit versions, including the latest snapshot.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (JCR-2695) Jackrabbit performance test suite
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-2695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jukka Zitting updated JCR-2695:
-------------------------------
Attachment: SimpleSearchTest.png
Attached a pretty impressive simple search (100 queries on 10k nodes, each returning 100 results) performance graph. It could be that the post-1.4 figures are a result of the entire data set residing in the bundle cache.
> Jackrabbit performance test suite
> ---------------------------------
>
> Key: JCR-2695
> URL: https://issues.apache.org/jira/browse/JCR-2695
> Project: Jackrabbit Content Repository
> Issue Type: Improvement
> Components: jackrabbit-core
> Reporter: Jukka Zitting
> Assignee: Jukka Zitting
> Fix For: 2.2.0
>
> Attachments: bigread.png, bigwrite.png, login.png, logout.png, SimpleSearchTest.png, smallread.png, smallwrite.png
>
>
> I'd like to set up a multi-version performance test suite inside jackrabbit-core/src/test/performance, similar to the compatibility test suite we added in JCR-2631. This performance test suite would produce comparable performance numbers for a number of simple benchmark tests across different Jackrabbit versions, including the latest snapshot.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.