You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jackrabbit.apache.org by "Ard Schrijvers (JIRA)" <ji...@apache.org> on 2009/01/08 18:36:59 UTC
[jira] Created: (JCR-1931) SharedFieldCache$StringIndex memory leak
causing OOM's
SharedFieldCache$StringIndex memory leak causing OOM's
-------------------------------------------------------
Key: JCR-1931
URL: https://issues.apache.org/jira/browse/JCR-1931
Project: Jackrabbit
Issue Type: Bug
Components: query
Affects Versions: 1.5.0
Reporter: Ard Schrijvers
Assignee: Ard Schrijvers
Priority: Critical
Fix For: 1.5.1
SharedFieldCache$StringIndex is not working properly. It is meant to cache the docnumbers in lucene along with the term to sort on. The issue is twofold. I have a solution for the second one, the first one is not really solvable from jr pov, because lucene index readers are already heavily caching Terms.
Explanation of the problem:
For *each* unique property where is sorted on, a new lucene ScoreDocComparator is created (see SharedFieldComparator newComparator). This new comparator creates *per* lucene indexreader SharedFieldCache.StringIndex which is stored in a WeakHashMap with as key, the indexreader . As this indexreader almost *never* can be garbage collected (only if it is merged and thus unused after), the SharedFieldCache.StringIndex are there to be the rest of the jvm life (which is sometime short, as can be seen from the simple unittest attached). Obviously, this results pretty fast in OOM.
1) issue one: The cached terms[] in SharedFieldCache.StringIndex can become huge when you sort on a common property (date) which is present in a lot of nodes. It you sort on large properties, like 'title' this SharedFieldCache.StringIndex will quickly use hundreds of Mb for a couple of hundred of thousand of nodes with a title. This issue is already a lucene issue, as lucene already caches the terms. OTOH, I really doubt whether we should index long string values as UNTOKENIZED in lucene at all. A half working solution might be a two-step solution, where the first sort is on the first 10 chars, and only if the comparator returns 0, take the entire string to sort on
2) issue two: The cached terms[] in SharedFieldCache.StringIndex is frequently sparse, consuming an incredible amount of memory for string arrays containing mainly null values. For example (see attached unit test):
- add 1.000.000 nodes
- do a query and sort on a non existing property
- you'll loose 1.000.000 * 4 bytes ~ 4 Mb of memory
- sort on another non existing prop : another 4 Mb is lost
- do it 100 times --> 400 Mb is lost, and can't be reclaimed
I'll attach a solution which works really fine for me, still having the almost unavoidable memory absorption, but makes it much smaller. The solution is, that if < 10% of the String array is filled, i consider the array already sparse, and move to a HashMap solution. Performance does not decrease much (and in case of large sparsity increases because less memory consumption --> less gc, etc).
Perhaps it does not seem to be a common issue (certainly the unit test) but our production environments memory snapshots indicate most memory being held by the SharedFieldCache$StringIndex (and the lucene Terms, which is harder to avoid)
I'd like to see this in the 1.5.1 if others are ok with it
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (JCR-1931) SharedFieldCache$StringIndex memory
leak causing OOM's
Posted by "Marcel Reutegger (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-1931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662960#action_12662960 ]
Marcel Reutegger commented on JCR-1931:
---------------------------------------
Comments:
- can you please create a patch that only includes relevant changes? it seems there are several changes in the patch that are just line breaks at different positions.
- StringIndex#sparse should be final, it is only set in the constructor
- comment in StringIndex#isSparse still mentions 10%
Other than that, the patch looks OK.
> SharedFieldCache$StringIndex memory leak causing OOM's
> -------------------------------------------------------
>
> Key: JCR-1931
> URL: https://issues.apache.org/jira/browse/JCR-1931
> Project: Jackrabbit
> Issue Type: Bug
> Components: query
> Affects Versions: 1.5.0
> Reporter: Ard Schrijvers
> Assignee: Ard Schrijvers
> Priority: Critical
> Fix For: 1.5.1
>
> Attachments: JCR-1931.patch, OrderByOOMTest.java
>
>
> SharedFieldCache$StringIndex is not working properly. It is meant to cache the docnumbers in lucene along with the term to sort on. The issue is twofold. I have a solution for the second one, the first one is not really solvable from jr pov, because lucene index readers are already heavily caching Terms.
> Explanation of the problem:
> For *each* unique property where is sorted on, a new lucene ScoreDocComparator is created (see SharedFieldComparator newComparator). This new comparator creates *per* lucene indexreader SharedFieldCache.StringIndex which is stored in a WeakHashMap with as key, the indexreader . As this indexreader almost *never* can be garbage collected (only if it is merged and thus unused after), the SharedFieldCache.StringIndex are there to be the rest of the jvm life (which is sometime short, as can be seen from the simple unittest attached). Obviously, this results pretty fast in OOM.
> 1) issue one: The cached terms[] in SharedFieldCache.StringIndex can become huge when you sort on a common property (date) which is present in a lot of nodes. It you sort on large properties, like 'title' this SharedFieldCache.StringIndex will quickly use hundreds of Mb for a couple of hundred of thousand of nodes with a title. This issue is already a lucene issue, as lucene already caches the terms. OTOH, I really doubt whether we should index long string values as UNTOKENIZED in lucene at all. A half working solution might be a two-step solution, where the first sort is on the first 10 chars, and only if the comparator returns 0, take the entire string to sort on
> 2) issue two: The cached terms[] in SharedFieldCache.StringIndex is frequently sparse, consuming an incredible amount of memory for string arrays containing mainly null values. For example (see attached unit test):
> - add 1.000.000 nodes
> - do a query and sort on a non existing property
> - you'll loose 1.000.000 * 4 bytes ~ 4 Mb of memory
> - sort on another non existing prop : another 4 Mb is lost
> - do it 100 times --> 400 Mb is lost, and can't be reclaimed
> I'll attach a solution which works really fine for me, still having the almost unavoidable memory absorption, but makes it much smaller. The solution is, that if < 10% of the String array is filled, i consider the array already sparse, and move to a HashMap solution. Performance does not decrease much (and in case of large sparsity increases because less memory consumption --> less gc, etc).
> Perhaps it does not seem to be a common issue (certainly the unit test) but our production environments memory snapshots indicate most memory being held by the SharedFieldCache$StringIndex (and the lucene Terms, which is harder to avoid)
> I'd like to see this in the 1.5.1 if others are ok with it
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (JCR-1931) SharedFieldCache$StringIndex memory
leak causing OOM's
Posted by "Ard Schrijvers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-1931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662277#action_12662277 ]
Ard Schrijvers commented on JCR-1931:
-------------------------------------
Yes, i'll add the patch today. Rethinking the sparsity factor i think it is better to have it 'sparse' is < 1% instead of 10% is filled, though, it is still pretty heuristic.Trying to patch against the trunk, will add the patch which can be tested (with the unit test included, where with the patch no memory is lost, and without in 500 searches about 250 Mb is lost (for 100.000 nodes test))
> SharedFieldCache$StringIndex memory leak causing OOM's
> -------------------------------------------------------
>
> Key: JCR-1931
> URL: https://issues.apache.org/jira/browse/JCR-1931
> Project: Jackrabbit
> Issue Type: Bug
> Components: query
> Affects Versions: 1.5.0
> Reporter: Ard Schrijvers
> Assignee: Ard Schrijvers
> Priority: Critical
> Fix For: 1.5.1
>
> Attachments: OrderByOOMTest.java
>
>
> SharedFieldCache$StringIndex is not working properly. It is meant to cache the docnumbers in lucene along with the term to sort on. The issue is twofold. I have a solution for the second one, the first one is not really solvable from jr pov, because lucene index readers are already heavily caching Terms.
> Explanation of the problem:
> For *each* unique property where is sorted on, a new lucene ScoreDocComparator is created (see SharedFieldComparator newComparator). This new comparator creates *per* lucene indexreader SharedFieldCache.StringIndex which is stored in a WeakHashMap with as key, the indexreader . As this indexreader almost *never* can be garbage collected (only if it is merged and thus unused after), the SharedFieldCache.StringIndex are there to be the rest of the jvm life (which is sometime short, as can be seen from the simple unittest attached). Obviously, this results pretty fast in OOM.
> 1) issue one: The cached terms[] in SharedFieldCache.StringIndex can become huge when you sort on a common property (date) which is present in a lot of nodes. It you sort on large properties, like 'title' this SharedFieldCache.StringIndex will quickly use hundreds of Mb for a couple of hundred of thousand of nodes with a title. This issue is already a lucene issue, as lucene already caches the terms. OTOH, I really doubt whether we should index long string values as UNTOKENIZED in lucene at all. A half working solution might be a two-step solution, where the first sort is on the first 10 chars, and only if the comparator returns 0, take the entire string to sort on
> 2) issue two: The cached terms[] in SharedFieldCache.StringIndex is frequently sparse, consuming an incredible amount of memory for string arrays containing mainly null values. For example (see attached unit test):
> - add 1.000.000 nodes
> - do a query and sort on a non existing property
> - you'll loose 1.000.000 * 4 bytes ~ 4 Mb of memory
> - sort on another non existing prop : another 4 Mb is lost
> - do it 100 times --> 400 Mb is lost, and can't be reclaimed
> I'll attach a solution which works really fine for me, still having the almost unavoidable memory absorption, but makes it much smaller. The solution is, that if < 10% of the String array is filled, i consider the array already sparse, and move to a HashMap solution. Performance does not decrease much (and in case of large sparsity increases because less memory consumption --> less gc, etc).
> Perhaps it does not seem to be a common issue (certainly the unit test) but our production environments memory snapshots indicate most memory being held by the SharedFieldCache$StringIndex (and the lucene Terms, which is harder to avoid)
> I'd like to see this in the 1.5.1 if others are ok with it
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (JCR-1931) SharedFieldCache$StringIndex memory leak
causing OOM's
Posted by "Ard Schrijvers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-1931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ard Schrijvers updated JCR-1931:
--------------------------------
Attachment: OrderByOOMTest.java
> SharedFieldCache$StringIndex memory leak causing OOM's
> -------------------------------------------------------
>
> Key: JCR-1931
> URL: https://issues.apache.org/jira/browse/JCR-1931
> Project: Jackrabbit
> Issue Type: Bug
> Components: query
> Affects Versions: 1.5.0
> Reporter: Ard Schrijvers
> Assignee: Ard Schrijvers
> Priority: Critical
> Fix For: 1.5.1
>
> Attachments: OrderByOOMTest.java
>
>
> SharedFieldCache$StringIndex is not working properly. It is meant to cache the docnumbers in lucene along with the term to sort on. The issue is twofold. I have a solution for the second one, the first one is not really solvable from jr pov, because lucene index readers are already heavily caching Terms.
> Explanation of the problem:
> For *each* unique property where is sorted on, a new lucene ScoreDocComparator is created (see SharedFieldComparator newComparator). This new comparator creates *per* lucene indexreader SharedFieldCache.StringIndex which is stored in a WeakHashMap with as key, the indexreader . As this indexreader almost *never* can be garbage collected (only if it is merged and thus unused after), the SharedFieldCache.StringIndex are there to be the rest of the jvm life (which is sometime short, as can be seen from the simple unittest attached). Obviously, this results pretty fast in OOM.
> 1) issue one: The cached terms[] in SharedFieldCache.StringIndex can become huge when you sort on a common property (date) which is present in a lot of nodes. It you sort on large properties, like 'title' this SharedFieldCache.StringIndex will quickly use hundreds of Mb for a couple of hundred of thousand of nodes with a title. This issue is already a lucene issue, as lucene already caches the terms. OTOH, I really doubt whether we should index long string values as UNTOKENIZED in lucene at all. A half working solution might be a two-step solution, where the first sort is on the first 10 chars, and only if the comparator returns 0, take the entire string to sort on
> 2) issue two: The cached terms[] in SharedFieldCache.StringIndex is frequently sparse, consuming an incredible amount of memory for string arrays containing mainly null values. For example (see attached unit test):
> - add 1.000.000 nodes
> - do a query and sort on a non existing property
> - you'll loose 1.000.000 * 4 bytes ~ 4 Mb of memory
> - sort on another non existing prop : another 4 Mb is lost
> - do it 100 times --> 400 Mb is lost, and can't be reclaimed
> I'll attach a solution which works really fine for me, still having the almost unavoidable memory absorption, but makes it much smaller. The solution is, that if < 10% of the String array is filled, i consider the array already sparse, and move to a HashMap solution. Performance does not decrease much (and in case of large sparsity increases because less memory consumption --> less gc, etc).
> Perhaps it does not seem to be a common issue (certainly the unit test) but our production environments memory snapshots indicate most memory being held by the SharedFieldCache$StringIndex (and the lucene Terms, which is harder to avoid)
> I'd like to see this in the 1.5.1 if others are ok with it
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (JCR-1931) SharedFieldCache$StringIndex memory
leak causing OOM's
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-1931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662122#action_12662122 ]
Jukka Zitting commented on JCR-1931:
------------------------------------
Your solution sounds reasonable, do you already have a patch for it?
I'm planning to cut the 1.5.1 release early next week. Can we have this in trunk and tested before that?
> SharedFieldCache$StringIndex memory leak causing OOM's
> -------------------------------------------------------
>
> Key: JCR-1931
> URL: https://issues.apache.org/jira/browse/JCR-1931
> Project: Jackrabbit
> Issue Type: Bug
> Components: query
> Affects Versions: 1.5.0
> Reporter: Ard Schrijvers
> Assignee: Ard Schrijvers
> Priority: Critical
> Fix For: 1.5.1
>
> Attachments: OrderByOOMTest.java
>
>
> SharedFieldCache$StringIndex is not working properly. It is meant to cache the docnumbers in lucene along with the term to sort on. The issue is twofold. I have a solution for the second one, the first one is not really solvable from jr pov, because lucene index readers are already heavily caching Terms.
> Explanation of the problem:
> For *each* unique property where is sorted on, a new lucene ScoreDocComparator is created (see SharedFieldComparator newComparator). This new comparator creates *per* lucene indexreader SharedFieldCache.StringIndex which is stored in a WeakHashMap with as key, the indexreader . As this indexreader almost *never* can be garbage collected (only if it is merged and thus unused after), the SharedFieldCache.StringIndex are there to be the rest of the jvm life (which is sometime short, as can be seen from the simple unittest attached). Obviously, this results pretty fast in OOM.
> 1) issue one: The cached terms[] in SharedFieldCache.StringIndex can become huge when you sort on a common property (date) which is present in a lot of nodes. It you sort on large properties, like 'title' this SharedFieldCache.StringIndex will quickly use hundreds of Mb for a couple of hundred of thousand of nodes with a title. This issue is already a lucene issue, as lucene already caches the terms. OTOH, I really doubt whether we should index long string values as UNTOKENIZED in lucene at all. A half working solution might be a two-step solution, where the first sort is on the first 10 chars, and only if the comparator returns 0, take the entire string to sort on
> 2) issue two: The cached terms[] in SharedFieldCache.StringIndex is frequently sparse, consuming an incredible amount of memory for string arrays containing mainly null values. For example (see attached unit test):
> - add 1.000.000 nodes
> - do a query and sort on a non existing property
> - you'll loose 1.000.000 * 4 bytes ~ 4 Mb of memory
> - sort on another non existing prop : another 4 Mb is lost
> - do it 100 times --> 400 Mb is lost, and can't be reclaimed
> I'll attach a solution which works really fine for me, still having the almost unavoidable memory absorption, but makes it much smaller. The solution is, that if < 10% of the String array is filled, i consider the array already sparse, and move to a HashMap solution. Performance does not decrease much (and in case of large sparsity increases because less memory consumption --> less gc, etc).
> Perhaps it does not seem to be a common issue (certainly the unit test) but our production environments memory snapshots indicate most memory being held by the SharedFieldCache$StringIndex (and the lucene Terms, which is harder to avoid)
> I'd like to see this in the 1.5.1 if others are ok with it
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (JCR-1931) SharedFieldCache$StringIndex memory
leak causing OOM's
Posted by "Ard Schrijvers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-1931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663297#action_12663297 ]
Ard Schrijvers commented on JCR-1931:
-------------------------------------
committed in 734092
> SharedFieldCache$StringIndex memory leak causing OOM's
> -------------------------------------------------------
>
> Key: JCR-1931
> URL: https://issues.apache.org/jira/browse/JCR-1931
> Project: Jackrabbit Content Repository
> Issue Type: Bug
> Components: query
> Affects Versions: 1.5.0
> Reporter: Ard Schrijvers
> Assignee: Ard Schrijvers
> Priority: Critical
> Fix For: 1.5.1
>
> Attachments: JCR-1931.patch, OrderByOOMTest.java
>
>
> SharedFieldCache$StringIndex is not working properly. It is meant to cache the docnumbers in lucene along with the term to sort on. The issue is twofold. I have a solution for the second one, the first one is not really solvable from jr pov, because lucene index readers are already heavily caching Terms.
> Explanation of the problem:
> For *each* unique property where is sorted on, a new lucene ScoreDocComparator is created (see SharedFieldComparator newComparator). This new comparator creates *per* lucene indexreader SharedFieldCache.StringIndex which is stored in a WeakHashMap with as key, the indexreader . As this indexreader almost *never* can be garbage collected (only if it is merged and thus unused after), the SharedFieldCache.StringIndex are there to be the rest of the jvm life (which is sometime short, as can be seen from the simple unittest attached). Obviously, this results pretty fast in OOM.
> 1) issue one: The cached terms[] in SharedFieldCache.StringIndex can become huge when you sort on a common property (date) which is present in a lot of nodes. It you sort on large properties, like 'title' this SharedFieldCache.StringIndex will quickly use hundreds of Mb for a couple of hundred of thousand of nodes with a title. This issue is already a lucene issue, as lucene already caches the terms. OTOH, I really doubt whether we should index long string values as UNTOKENIZED in lucene at all. A half working solution might be a two-step solution, where the first sort is on the first 10 chars, and only if the comparator returns 0, take the entire string to sort on
> 2) issue two: The cached terms[] in SharedFieldCache.StringIndex is frequently sparse, consuming an incredible amount of memory for string arrays containing mainly null values. For example (see attached unit test):
> - add 1.000.000 nodes
> - do a query and sort on a non existing property
> - you'll loose 1.000.000 * 4 bytes ~ 4 Mb of memory
> - sort on another non existing prop : another 4 Mb is lost
> - do it 100 times --> 400 Mb is lost, and can't be reclaimed
> I'll attach a solution which works really fine for me, still having the almost unavoidable memory absorption, but makes it much smaller. The solution is, that if < 10% of the String array is filled, i consider the array already sparse, and move to a HashMap solution. Performance does not decrease much (and in case of large sparsity increases because less memory consumption --> less gc, etc).
> Perhaps it does not seem to be a common issue (certainly the unit test) but our production environments memory snapshots indicate most memory being held by the SharedFieldCache$StringIndex (and the lucene Terms, which is harder to avoid)
> I'd like to see this in the 1.5.1 if others are ok with it
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (JCR-1931) SharedFieldCache$StringIndex memory
leak causing OOM's
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-1931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663010#action_12663010 ]
Jukka Zitting commented on JCR-1931:
------------------------------------
Seems OK to me too. Ard, can you commit this with Marcel's suggetions? I'll then merge it to 1.5 for inclusion in 1.5.1.
> SharedFieldCache$StringIndex memory leak causing OOM's
> -------------------------------------------------------
>
> Key: JCR-1931
> URL: https://issues.apache.org/jira/browse/JCR-1931
> Project: Jackrabbit
> Issue Type: Bug
> Components: query
> Affects Versions: 1.5.0
> Reporter: Ard Schrijvers
> Assignee: Ard Schrijvers
> Priority: Critical
> Fix For: 1.5.1
>
> Attachments: JCR-1931.patch, OrderByOOMTest.java
>
>
> SharedFieldCache$StringIndex is not working properly. It is meant to cache the docnumbers in lucene along with the term to sort on. The issue is twofold. I have a solution for the second one, the first one is not really solvable from jr pov, because lucene index readers are already heavily caching Terms.
> Explanation of the problem:
> For *each* unique property where is sorted on, a new lucene ScoreDocComparator is created (see SharedFieldComparator newComparator). This new comparator creates *per* lucene indexreader SharedFieldCache.StringIndex which is stored in a WeakHashMap with as key, the indexreader . As this indexreader almost *never* can be garbage collected (only if it is merged and thus unused after), the SharedFieldCache.StringIndex are there to be the rest of the jvm life (which is sometime short, as can be seen from the simple unittest attached). Obviously, this results pretty fast in OOM.
> 1) issue one: The cached terms[] in SharedFieldCache.StringIndex can become huge when you sort on a common property (date) which is present in a lot of nodes. It you sort on large properties, like 'title' this SharedFieldCache.StringIndex will quickly use hundreds of Mb for a couple of hundred of thousand of nodes with a title. This issue is already a lucene issue, as lucene already caches the terms. OTOH, I really doubt whether we should index long string values as UNTOKENIZED in lucene at all. A half working solution might be a two-step solution, where the first sort is on the first 10 chars, and only if the comparator returns 0, take the entire string to sort on
> 2) issue two: The cached terms[] in SharedFieldCache.StringIndex is frequently sparse, consuming an incredible amount of memory for string arrays containing mainly null values. For example (see attached unit test):
> - add 1.000.000 nodes
> - do a query and sort on a non existing property
> - you'll loose 1.000.000 * 4 bytes ~ 4 Mb of memory
> - sort on another non existing prop : another 4 Mb is lost
> - do it 100 times --> 400 Mb is lost, and can't be reclaimed
> I'll attach a solution which works really fine for me, still having the almost unavoidable memory absorption, but makes it much smaller. The solution is, that if < 10% of the String array is filled, i consider the array already sparse, and move to a HashMap solution. Performance does not decrease much (and in case of large sparsity increases because less memory consumption --> less gc, etc).
> Perhaps it does not seem to be a common issue (certainly the unit test) but our production environments memory snapshots indicate most memory being held by the SharedFieldCache$StringIndex (and the lucene Terms, which is harder to avoid)
> I'd like to see this in the 1.5.1 if others are ok with it
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (JCR-1931) SharedFieldCache$StringIndex memory
leak causing OOM's
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-1931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jukka Zitting resolved JCR-1931.
--------------------------------
Resolution: Fixed
Thanks! Merged to the 1.5 branch in revision 734187.
> SharedFieldCache$StringIndex memory leak causing OOM's
> -------------------------------------------------------
>
> Key: JCR-1931
> URL: https://issues.apache.org/jira/browse/JCR-1931
> Project: Jackrabbit Content Repository
> Issue Type: Bug
> Components: jackrabbit-core, query
> Affects Versions: 1.5.0
> Reporter: Ard Schrijvers
> Assignee: Ard Schrijvers
> Priority: Critical
> Fix For: 1.5.1
>
> Attachments: JCR-1931.patch, OrderByOOMTest.java
>
>
> SharedFieldCache$StringIndex is not working properly. It is meant to cache the docnumbers in lucene along with the term to sort on. The issue is twofold. I have a solution for the second one, the first one is not really solvable from jr pov, because lucene index readers are already heavily caching Terms.
> Explanation of the problem:
> For *each* unique property where is sorted on, a new lucene ScoreDocComparator is created (see SharedFieldComparator newComparator). This new comparator creates *per* lucene indexreader SharedFieldCache.StringIndex which is stored in a WeakHashMap with as key, the indexreader . As this indexreader almost *never* can be garbage collected (only if it is merged and thus unused after), the SharedFieldCache.StringIndex are there to be the rest of the jvm life (which is sometime short, as can be seen from the simple unittest attached). Obviously, this results pretty fast in OOM.
> 1) issue one: The cached terms[] in SharedFieldCache.StringIndex can become huge when you sort on a common property (date) which is present in a lot of nodes. It you sort on large properties, like 'title' this SharedFieldCache.StringIndex will quickly use hundreds of Mb for a couple of hundred of thousand of nodes with a title. This issue is already a lucene issue, as lucene already caches the terms. OTOH, I really doubt whether we should index long string values as UNTOKENIZED in lucene at all. A half working solution might be a two-step solution, where the first sort is on the first 10 chars, and only if the comparator returns 0, take the entire string to sort on
> 2) issue two: The cached terms[] in SharedFieldCache.StringIndex is frequently sparse, consuming an incredible amount of memory for string arrays containing mainly null values. For example (see attached unit test):
> - add 1.000.000 nodes
> - do a query and sort on a non existing property
> - you'll loose 1.000.000 * 4 bytes ~ 4 Mb of memory
> - sort on another non existing prop : another 4 Mb is lost
> - do it 100 times --> 400 Mb is lost, and can't be reclaimed
> I'll attach a solution which works really fine for me, still having the almost unavoidable memory absorption, but makes it much smaller. The solution is, that if < 10% of the String array is filled, i consider the array already sparse, and move to a HashMap solution. Performance does not decrease much (and in case of large sparsity increases because less memory consumption --> less gc, etc).
> Perhaps it does not seem to be a common issue (certainly the unit test) but our production environments memory snapshots indicate most memory being held by the SharedFieldCache$StringIndex (and the lucene Terms, which is harder to avoid)
> I'd like to see this in the 1.5.1 if others are ok with it
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (JCR-1931) SharedFieldCache$StringIndex memory leak
causing OOM's
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-1931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jukka Zitting updated JCR-1931:
-------------------------------
Component/s: jackrabbit-core
> SharedFieldCache$StringIndex memory leak causing OOM's
> -------------------------------------------------------
>
> Key: JCR-1931
> URL: https://issues.apache.org/jira/browse/JCR-1931
> Project: Jackrabbit Content Repository
> Issue Type: Bug
> Components: jackrabbit-core, query
> Affects Versions: 1.5.0
> Reporter: Ard Schrijvers
> Assignee: Ard Schrijvers
> Priority: Critical
> Fix For: 1.5.1
>
> Attachments: JCR-1931.patch, OrderByOOMTest.java
>
>
> SharedFieldCache$StringIndex is not working properly. It is meant to cache the docnumbers in lucene along with the term to sort on. The issue is twofold. I have a solution for the second one, the first one is not really solvable from jr pov, because lucene index readers are already heavily caching Terms.
> Explanation of the problem:
> For *each* unique property where is sorted on, a new lucene ScoreDocComparator is created (see SharedFieldComparator newComparator). This new comparator creates *per* lucene indexreader SharedFieldCache.StringIndex which is stored in a WeakHashMap with as key, the indexreader . As this indexreader almost *never* can be garbage collected (only if it is merged and thus unused after), the SharedFieldCache.StringIndex are there to be the rest of the jvm life (which is sometime short, as can be seen from the simple unittest attached). Obviously, this results pretty fast in OOM.
> 1) issue one: The cached terms[] in SharedFieldCache.StringIndex can become huge when you sort on a common property (date) which is present in a lot of nodes. It you sort on large properties, like 'title' this SharedFieldCache.StringIndex will quickly use hundreds of Mb for a couple of hundred of thousand of nodes with a title. This issue is already a lucene issue, as lucene already caches the terms. OTOH, I really doubt whether we should index long string values as UNTOKENIZED in lucene at all. A half working solution might be a two-step solution, where the first sort is on the first 10 chars, and only if the comparator returns 0, take the entire string to sort on
> 2) issue two: The cached terms[] in SharedFieldCache.StringIndex is frequently sparse, consuming an incredible amount of memory for string arrays containing mainly null values. For example (see attached unit test):
> - add 1.000.000 nodes
> - do a query and sort on a non existing property
> - you'll loose 1.000.000 * 4 bytes ~ 4 Mb of memory
> - sort on another non existing prop : another 4 Mb is lost
> - do it 100 times --> 400 Mb is lost, and can't be reclaimed
> I'll attach a solution which works really fine for me, still having the almost unavoidable memory absorption, but makes it much smaller. The solution is, that if < 10% of the String array is filled, i consider the array already sparse, and move to a HashMap solution. Performance does not decrease much (and in case of large sparsity increases because less memory consumption --> less gc, etc).
> Perhaps it does not seem to be a common issue (certainly the unit test) but our production environments memory snapshots indicate most memory being held by the SharedFieldCache$StringIndex (and the lucene Terms, which is harder to avoid)
> I'd like to see this in the 1.5.1 if others are ok with it
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (JCR-1931) SharedFieldCache$StringIndex memory leak
causing OOM's
Posted by "Ard Schrijvers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-1931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ard Schrijvers updated JCR-1931:
--------------------------------
Attachment: JCR-1931.patch
suggested fix for memory usage reduction. When sorting on sparse properties, memory reduction is large. When sorting on non existing properties, memory usage almost drops to a few bytes per unique sort property, where it used to be 4* maxdoc bytes (maxdoc is lucene multireader maxdoc, so in case of 1.000.000 nodes, about 1.000.000)
> SharedFieldCache$StringIndex memory leak causing OOM's
> -------------------------------------------------------
>
> Key: JCR-1931
> URL: https://issues.apache.org/jira/browse/JCR-1931
> Project: Jackrabbit
> Issue Type: Bug
> Components: query
> Affects Versions: 1.5.0
> Reporter: Ard Schrijvers
> Assignee: Ard Schrijvers
> Priority: Critical
> Fix For: 1.5.1
>
> Attachments: JCR-1931.patch, OrderByOOMTest.java
>
>
> SharedFieldCache$StringIndex is not working properly. It is meant to cache the docnumbers in lucene along with the term to sort on. The issue is twofold. I have a solution for the second one, the first one is not really solvable from jr pov, because lucene index readers are already heavily caching Terms.
> Explanation of the problem:
> For *each* unique property where is sorted on, a new lucene ScoreDocComparator is created (see SharedFieldComparator newComparator). This new comparator creates *per* lucene indexreader SharedFieldCache.StringIndex which is stored in a WeakHashMap with as key, the indexreader . As this indexreader almost *never* can be garbage collected (only if it is merged and thus unused after), the SharedFieldCache.StringIndex are there to be the rest of the jvm life (which is sometime short, as can be seen from the simple unittest attached). Obviously, this results pretty fast in OOM.
> 1) issue one: The cached terms[] in SharedFieldCache.StringIndex can become huge when you sort on a common property (date) which is present in a lot of nodes. It you sort on large properties, like 'title' this SharedFieldCache.StringIndex will quickly use hundreds of Mb for a couple of hundred of thousand of nodes with a title. This issue is already a lucene issue, as lucene already caches the terms. OTOH, I really doubt whether we should index long string values as UNTOKENIZED in lucene at all. A half working solution might be a two-step solution, where the first sort is on the first 10 chars, and only if the comparator returns 0, take the entire string to sort on
> 2) issue two: The cached terms[] in SharedFieldCache.StringIndex is frequently sparse, consuming an incredible amount of memory for string arrays containing mainly null values. For example (see attached unit test):
> - add 1.000.000 nodes
> - do a query and sort on a non existing property
> - you'll loose 1.000.000 * 4 bytes ~ 4 Mb of memory
> - sort on another non existing prop : another 4 Mb is lost
> - do it 100 times --> 400 Mb is lost, and can't be reclaimed
> I'll attach a solution which works really fine for me, still having the almost unavoidable memory absorption, but makes it much smaller. The solution is, that if < 10% of the String array is filled, i consider the array already sparse, and move to a HashMap solution. Performance does not decrease much (and in case of large sparsity increases because less memory consumption --> less gc, etc).
> Perhaps it does not seem to be a common issue (certainly the unit test) but our production environments memory snapshots indicate most memory being held by the SharedFieldCache$StringIndex (and the lucene Terms, which is harder to avoid)
> I'd like to see this in the 1.5.1 if others are ok with it
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.