You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jackrabbit.apache.org by "Xiaohua Lu (JIRA)" <ji...@apache.org> on 2007/05/22 18:31:16 UTC
[jira] Created: (JCR-937) CacheManager max memory size
CacheManager max memory size
----------------------------
Key: JCR-937
URL: https://issues.apache.org/jira/browse/JCR-937
Project: Jackrabbit
Issue Type: Bug
Affects Versions: 1.3
Reporter: Xiaohua Lu
I have ran into OutOfMemory a couple of times with Jackrabbit (cluster, 4 nodes, each has 1G mem heap size).
After adding some debug into the CacheManager, I noticed that maxMemorySize (default to 16M) is not really honored during resizeAll check. Each individual MLRUItemStateCache seems to honor the size, but the total number/size of MLRUItemStateCache is not. If you put some print statement of totalMemoryUsed and unusedMemory, you can see that totalMemoryUsed is more than 16M and unusedMemory is negative.
The other problem we have noticed during the profiling is there are a lots of other in memory objects that are consuming memory but not included in CacheManager caches control. One example is CachingHierarchyManager which consumed 58M out of 242M through its use of PathMap. If CacheManager's maxSize can control the total cache size used by Jackrabbit, that would be easier from a management's perspective. (btw, upper_limit in CachingHierarchyManager is hardcoded and can't be control from outside)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (JCR-937) CacheManager max memory size
Posted by "Martijn Hendriks (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12537307 ]
Martijn Hendriks commented on JCR-937:
--------------------------------------
Ard, thanks for your explanation. I think you are right that a more efficient Lucene configuration would let the test pass. If I add 1000 String properties of approx 1 MB (with max 512 MB heap space) then the OOM will reappear because Jackrabbit then tries to cache all these properties due to underestimation of the retained memory of the caches. I will create a separate issue for this.
> CacheManager max memory size
> ----------------------------
>
> Key: JCR-937
> URL: https://issues.apache.org/jira/browse/JCR-937
> Project: Jackrabbit
> Issue Type: Bug
> Components: jackrabbit-core
> Affects Versions: 1.3
> Reporter: Xiaohua Lu
> Assignee: Thomas Mueller
> Priority: Minor
> Attachments: CacheManagerTest.java
>
>
> I have ran into OutOfMemory a couple of times with Jackrabbit (cluster, 4 nodes, each has 1G mem heap size).
> After adding some debug into the CacheManager, I noticed that maxMemorySize (default to 16M) is not really honored during resizeAll check. Each individual MLRUItemStateCache seems to honor the size, but the total number/size of MLRUItemStateCache is not. If you put some print statement of totalMemoryUsed and unusedMemory, you can see that totalMemoryUsed is more than 16M and unusedMemory is negative.
> The other problem we have noticed during the profiling is there are a lots of other in memory objects that are consuming memory but not included in CacheManager caches control. One example is CachingHierarchyManager which consumed 58M out of 242M through its use of PathMap. If CacheManager's maxSize can control the total cache size used by Jackrabbit, that would be easier from a management's perspective. (btw, upper_limit in CachingHierarchyManager is hardcoded and can't be control from outside)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (JCR-937) CacheManager max memory size
Posted by "Stefan Guggisberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stefan Guggisberg updated JCR-937:
----------------------------------
Component/s: core
Priority: Minor (was: Major)
> CacheManager max memory size
> ----------------------------
>
> Key: JCR-937
> URL: https://issues.apache.org/jira/browse/JCR-937
> Project: Jackrabbit
> Issue Type: Bug
> Components: core
> Affects Versions: 1.3
> Reporter: Xiaohua Lu
> Priority: Minor
>
> I have ran into OutOfMemory a couple of times with Jackrabbit (cluster, 4 nodes, each has 1G mem heap size).
> After adding some debug into the CacheManager, I noticed that maxMemorySize (default to 16M) is not really honored during resizeAll check. Each individual MLRUItemStateCache seems to honor the size, but the total number/size of MLRUItemStateCache is not. If you put some print statement of totalMemoryUsed and unusedMemory, you can see that totalMemoryUsed is more than 16M and unusedMemory is negative.
> The other problem we have noticed during the profiling is there are a lots of other in memory objects that are consuming memory but not included in CacheManager caches control. One example is CachingHierarchyManager which consumed 58M out of 242M through its use of PathMap. If CacheManager's maxSize can control the total cache size used by Jackrabbit, that would be easier from a management's perspective. (btw, upper_limit in CachingHierarchyManager is hardcoded and can't be control from outside)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (JCR-937) CacheManager max memory size
Posted by "Martijn Hendriks (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12537264 ]
Martijn Hendriks commented on JCR-937:
--------------------------------------
It seems rather trivial to extend InternalValue and BlobFileValue with a much more accurate estimation of the retained memory and to use this in PropertyState.calculateMemoryFootprint(). This would give a much better estimation of the retained size of the caches. It would also be nice to cache this size in the MLRUItemStateCache itself instead of recomputing it every time.
Should I create a new JIRA issue for this specific problem?
> CacheManager max memory size
> ----------------------------
>
> Key: JCR-937
> URL: https://issues.apache.org/jira/browse/JCR-937
> Project: Jackrabbit
> Issue Type: Bug
> Components: jackrabbit-core
> Affects Versions: 1.3
> Reporter: Xiaohua Lu
> Assignee: Thomas Mueller
> Priority: Minor
> Attachments: CacheManagerTest.java
>
>
> I have ran into OutOfMemory a couple of times with Jackrabbit (cluster, 4 nodes, each has 1G mem heap size).
> After adding some debug into the CacheManager, I noticed that maxMemorySize (default to 16M) is not really honored during resizeAll check. Each individual MLRUItemStateCache seems to honor the size, but the total number/size of MLRUItemStateCache is not. If you put some print statement of totalMemoryUsed and unusedMemory, you can see that totalMemoryUsed is more than 16M and unusedMemory is negative.
> The other problem we have noticed during the profiling is there are a lots of other in memory objects that are consuming memory but not included in CacheManager caches control. One example is CachingHierarchyManager which consumed 58M out of 242M through its use of PathMap. If CacheManager's maxSize can control the total cache size used by Jackrabbit, that would be easier from a management's perspective. (btw, upper_limit in CachingHierarchyManager is hardcoded and can't be control from outside)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (JCR-937) CacheManager max memory size
Posted by "Thomas Mueller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Mueller resolved JCR-937.
--------------------------------
Resolution: Cannot Reproduce
Assignee: Thomas Mueller
If the problem is really too many open sessions maybe we should add a limit on the number of sessions, or warn if the application did not close sessions?
I am resolving this problem until it can be reproduced. Please re-open if necessary.
> CacheManager max memory size
> ----------------------------
>
> Key: JCR-937
> URL: https://issues.apache.org/jira/browse/JCR-937
> Project: Jackrabbit
> Issue Type: Bug
> Components: core
> Affects Versions: 1.3
> Reporter: Xiaohua Lu
> Assignee: Thomas Mueller
> Priority: Minor
>
> I have ran into OutOfMemory a couple of times with Jackrabbit (cluster, 4 nodes, each has 1G mem heap size).
> After adding some debug into the CacheManager, I noticed that maxMemorySize (default to 16M) is not really honored during resizeAll check. Each individual MLRUItemStateCache seems to honor the size, but the total number/size of MLRUItemStateCache is not. If you put some print statement of totalMemoryUsed and unusedMemory, you can see that totalMemoryUsed is more than 16M and unusedMemory is negative.
> The other problem we have noticed during the profiling is there are a lots of other in memory objects that are consuming memory but not included in CacheManager caches control. One example is CachingHierarchyManager which consumed 58M out of 242M through its use of PathMap. If CacheManager's maxSize can control the total cache size used by Jackrabbit, that would be easier from a management's perspective. (btw, upper_limit in CachingHierarchyManager is hardcoded and can't be control from outside)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (JCR-937) CacheManager max memory size
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12518405 ]
Jukka Zitting commented on JCR-937:
-----------------------------------
There's a minimum size (default 128kB) per each cache that overrides the global maximum memory setting when you start having large numbers of sessions. Each session is in effect guaranteed at least a small slice of memory for caching.
Do you have an idea how many sessions you have open concurrently?
> CacheManager max memory size
> ----------------------------
>
> Key: JCR-937
> URL: https://issues.apache.org/jira/browse/JCR-937
> Project: Jackrabbit
> Issue Type: Bug
> Components: core
> Affects Versions: 1.3
> Reporter: Xiaohua Lu
> Priority: Minor
>
> I have ran into OutOfMemory a couple of times with Jackrabbit (cluster, 4 nodes, each has 1G mem heap size).
> After adding some debug into the CacheManager, I noticed that maxMemorySize (default to 16M) is not really honored during resizeAll check. Each individual MLRUItemStateCache seems to honor the size, but the total number/size of MLRUItemStateCache is not. If you put some print statement of totalMemoryUsed and unusedMemory, you can see that totalMemoryUsed is more than 16M and unusedMemory is negative.
> The other problem we have noticed during the profiling is there are a lots of other in memory objects that are consuming memory but not included in CacheManager caches control. One example is CachingHierarchyManager which consumed 58M out of 242M through its use of PathMap. If CacheManager's maxSize can control the total cache size used by Jackrabbit, that would be easier from a management's perspective. (btw, upper_limit in CachingHierarchyManager is hardcoded and can't be control from outside)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (JCR-937) CacheManager max memory size
Posted by "Thomas Mueller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12537255 ]
Thomas Mueller commented on JCR-937:
------------------------------------
Hi,
I agree the cache size calculation is not accurate. The question is, how exact does it need to be?
I don't think that the original reporter of the problem used long strings: the problem was too many caches (hundreds if not thousands). The reason for that many caches is not known, for this we need a reproducible test case.
> CacheManager max memory size
> ----------------------------
>
> Key: JCR-937
> URL: https://issues.apache.org/jira/browse/JCR-937
> Project: Jackrabbit
> Issue Type: Bug
> Components: jackrabbit-core
> Affects Versions: 1.3
> Reporter: Xiaohua Lu
> Assignee: Thomas Mueller
> Priority: Minor
> Attachments: CacheManagerTest.java
>
>
> I have ran into OutOfMemory a couple of times with Jackrabbit (cluster, 4 nodes, each has 1G mem heap size).
> After adding some debug into the CacheManager, I noticed that maxMemorySize (default to 16M) is not really honored during resizeAll check. Each individual MLRUItemStateCache seems to honor the size, but the total number/size of MLRUItemStateCache is not. If you put some print statement of totalMemoryUsed and unusedMemory, you can see that totalMemoryUsed is more than 16M and unusedMemory is negative.
> The other problem we have noticed during the profiling is there are a lots of other in memory objects that are consuming memory but not included in CacheManager caches control. One example is CachingHierarchyManager which consumed 58M out of 242M through its use of PathMap. If CacheManager's maxSize can control the total cache size used by Jackrabbit, that would be easier from a management's perspective. (btw, upper_limit in CachingHierarchyManager is hardcoded and can't be control from outside)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (JCR-937) CacheManager max memory size
Posted by "Ard Schrijvers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12537289 ]
Ard Schrijvers commented on JCR-937:
------------------------------------
"Is the Lucene memory consumption due to the extremely long String properties, or has it to do with the test setup? "
No, this is not because of long String properties, and yes, it is closely related to the test setup. You are storing nodes of roughly 1 Mb, right? Now, depending on your SearchIndex configuration, for example volatileIdleTime (how long before a memory index is flushed to disk) , bufferSize (maximum number of documents that are held in a pending queue until added to the index), minMergeDocs etc etc, it depends how large the memory lucene index will be (VolatileIndex). I really think the OOM is part of the way you have done the setup. If for example, you would replace the 1Mb String of 'aaaaa....' to a random piece of text of 1 Mb, the lucene indexing would take much longer, hence the memory index wouldn't be able to grow that large, because volatileIdleTime will be reached ealier and memory index will be persisted.
> CacheManager max memory size
> ----------------------------
>
> Key: JCR-937
> URL: https://issues.apache.org/jira/browse/JCR-937
> Project: Jackrabbit
> Issue Type: Bug
> Components: jackrabbit-core
> Affects Versions: 1.3
> Reporter: Xiaohua Lu
> Assignee: Thomas Mueller
> Priority: Minor
> Attachments: CacheManagerTest.java
>
>
> I have ran into OutOfMemory a couple of times with Jackrabbit (cluster, 4 nodes, each has 1G mem heap size).
> After adding some debug into the CacheManager, I noticed that maxMemorySize (default to 16M) is not really honored during resizeAll check. Each individual MLRUItemStateCache seems to honor the size, but the total number/size of MLRUItemStateCache is not. If you put some print statement of totalMemoryUsed and unusedMemory, you can see that totalMemoryUsed is more than 16M and unusedMemory is negative.
> The other problem we have noticed during the profiling is there are a lots of other in memory objects that are consuming memory but not included in CacheManager caches control. One example is CachingHierarchyManager which consumed 58M out of 242M through its use of PathMap. If CacheManager's maxSize can control the total cache size used by Jackrabbit, that would be easier from a management's perspective. (btw, upper_limit in CachingHierarchyManager is hardcoded and can't be control from outside)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (JCR-937) CacheManager max memory size
Posted by "Martijn Hendriks (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Martijn Hendriks updated JCR-937:
---------------------------------
Attachment: CacheManagerTest.java
Currently, the estimation of the sizes of the caches that the CacheManager manages is not very accurate. The sizes of properties are estimated in PropertyState.calculateMemoryFootprint() to be 350 + values.length * 100. This is really bad when you have a lot of non-binary properties that are significantly larger than 100 bytes.
The attached unit test gives a testcase with only one concurrent session which is closed properly. Running this test (-Xmx=512M) and checking a heap-dump triggered by an OutOfMemoryError with YourKit shows the following:
(1) The CacheManager lets the cache of the SISM grow way beyond the default max total cache size of 16 MB, namely to 157 MB.
(2) Lucene classes have a retained size of approx 323 MB.
Is the Lucene memory consumption due to the extremely long String properties, or has it to do with the test setup?
I'd like to reopen this issue, but I don't see a button for it.
> CacheManager max memory size
> ----------------------------
>
> Key: JCR-937
> URL: https://issues.apache.org/jira/browse/JCR-937
> Project: Jackrabbit
> Issue Type: Bug
> Components: jackrabbit-core
> Affects Versions: 1.3
> Reporter: Xiaohua Lu
> Assignee: Thomas Mueller
> Priority: Minor
> Attachments: CacheManagerTest.java
>
>
> I have ran into OutOfMemory a couple of times with Jackrabbit (cluster, 4 nodes, each has 1G mem heap size).
> After adding some debug into the CacheManager, I noticed that maxMemorySize (default to 16M) is not really honored during resizeAll check. Each individual MLRUItemStateCache seems to honor the size, but the total number/size of MLRUItemStateCache is not. If you put some print statement of totalMemoryUsed and unusedMemory, you can see that totalMemoryUsed is more than 16M and unusedMemory is negative.
> The other problem we have noticed during the profiling is there are a lots of other in memory objects that are consuming memory but not included in CacheManager caches control. One example is CachingHierarchyManager which consumed 58M out of 242M through its use of PathMap. If CacheManager's maxSize can control the total cache size used by Jackrabbit, that would be easier from a management's perspective. (btw, upper_limit in CachingHierarchyManager is hardcoded and can't be control from outside)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (JCR-937) CacheManager max memory size
Posted by "Thomas Mueller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/JCR-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12537293 ]
Thomas Mueller commented on JCR-937:
------------------------------------
> extend InternalValue and BlobFileValue with a much more accurate estimation of the retained memory
Yes, that would be very good. I don't think it would affect performance much.
> cache this size in the MLRUItemStateCache itself
I am not sure, but maybe the objects can change (so the size can change) while they are in the cache.
> CacheManager max memory size
> ----------------------------
>
> Key: JCR-937
> URL: https://issues.apache.org/jira/browse/JCR-937
> Project: Jackrabbit
> Issue Type: Bug
> Components: jackrabbit-core
> Affects Versions: 1.3
> Reporter: Xiaohua Lu
> Assignee: Thomas Mueller
> Priority: Minor
> Attachments: CacheManagerTest.java
>
>
> I have ran into OutOfMemory a couple of times with Jackrabbit (cluster, 4 nodes, each has 1G mem heap size).
> After adding some debug into the CacheManager, I noticed that maxMemorySize (default to 16M) is not really honored during resizeAll check. Each individual MLRUItemStateCache seems to honor the size, but the total number/size of MLRUItemStateCache is not. If you put some print statement of totalMemoryUsed and unusedMemory, you can see that totalMemoryUsed is more than 16M and unusedMemory is negative.
> The other problem we have noticed during the profiling is there are a lots of other in memory objects that are consuming memory but not included in CacheManager caches control. One example is CachingHierarchyManager which consumed 58M out of 242M through its use of PathMap. If CacheManager's maxSize can control the total cache size used by Jackrabbit, that would be easier from a management's perspective. (btw, upper_limit in CachingHierarchyManager is hardcoded and can't be control from outside)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.