[jira] [Comment Edited] (OAK-5192) Reduce Lucene related growth of repository size

["Tommaso Teofili (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OAK-5192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072493#comment-16072493 ] 

Tommaso Teofili edited comment on OAK-5192 at 7/3/17 2:11 PM:
--------------------------------------------------------------

I've enhanced the current small benchmark so that it indexes some content, then perform a full reindex, then some content gets added and some deleted; this way we should be also able to observe how reindexing and deletion impact size.
I've performed the following test runs :

||Codec||Merge Policy||Size after first full index||Size after full reindex|| Size after add/delete||
|oakCodec|default|171.1 MB|312.9 MB|432.1 MB|
|Lucene46|default|159.4 MB|289.5 MB|429.7 MB|
|customCodec|default|152.8 MB|276.3 MB|410.4 MB|
|oakCodec|no|171.1 MB|312.9 MB|432.1 MB|
|Lucene46|no|159.4 MB|289.4 MB|400.6 MB|
|customCodec|no|152.8 MB|276.3 MB|382.8 MB|

CustomCodec is a test codec having the combination of settings that resulted in best space saving results in the previous test.


was (Author: teofili):
I've enhanced the current small benchmark so that it indexes some content, then perform a full reindex, then some content gets added and some deleted; this way we should be also able to observe how reindexing and deletion impact size.
I've performed the following test runs :

||Codec||Merge Policy||Size after first full index||Size after full reindex|| Size after add/delete||
|oakCodec|default|171.1 MB|312.9 MB|432.1 MB|
|Lucene46|default|159.4 MB|289.5 MB|429.7 MB|
|customCodec|default|152.8 MB|276.3 MB|410.4 MB|
|oakCodec|no|171.1 MB|312.9 MB|432.1 MB|
|Lucene46|no|159.4 MB|289.4 MB|400.6 MB|
|customCodec|no|152.8 MB|276.3 MB|382.8 MB|



> Reduce Lucene related growth of repository size
> -----------------------------------------------
>
>                 Key: OAK-5192
>                 URL: https://issues.apache.org/jira/browse/OAK-5192
>             Project: Jackrabbit Oak
>          Issue Type: Improvement
>          Components: lucene, segment-tar
>            Reporter: Michael Dürig
>            Assignee: Tommaso Teofili
>              Labels: perfomance, scalability
>             Fix For: 1.8, 1.7.8
>
>         Attachments: added-bytes-zoom.png, binSize100.txt, binSize16384.txt, binSizeTotal.txt, diff.txt.zip, nonBinSizeTotal.txt, OAK-5192.0.patch
>
>
> I observed Lucene indexing contributing to up to 99% of repository growth. While the size of the index itself is well inside reasonable bounds, the overall turnover of data being written and removed again can be as much as 99%. 
> In the case of the TarMK this negatively impacts overall system performance due to fast growing number of tar files / segments, bad locality of reference, cache misses/thrashing when looking up segments and vastly prolonged garbage collection cycles.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)