You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "Kristian Waagan (JIRA)" <ji...@apache.org> on 2007/10/22 11:13:50 UTC
[jira] Issue Comment Edited: (DERBY-3130) Reduce memory footprint
of StoredRecordHeader
[ https://issues.apache.org/jira/browse/DERBY-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536603 ]
kristwaa edited comment on DERBY-3130 at 10/22/07 2:12 AM:
------------------------------------------------------------------
I ran the patch with the load I experience problems with, and it fixes the problem for my configuration.
As I mentioned earlier, the patch may not solve the problem for other configurations due to the amount of heap available and the size of the Derby page cache.
In my case Derby was able to keep under 3.5 GB of heap with a page cache of size 262144 (which with 4K pages would be around 1 GB + overhead).
Here's the start out the heap histogram:
num #instances #bytes class name
--------------------------------------
1: 33176246 1061639872 org.apache.derby.impl.store.raw.data.StoredRecordHeader
2: __683347 _961070736 [B
3: 33402612 _801662688 org.apache.derby.impl.store.raw.data.RecordId
4: __226889 _136070680 [Lorg.apache.derby.impl.store.raw.data.StoredRecordHeader;
5: __226729 __43531968 org.apache.derby.impl.store.raw.data.StoredPage
6: __237426 __40619752 [C
7: __226653 __14505792 org.apache.derby.impl.store.access.btree.LeafControlRow
8: __226890 ___9075600 org.apache.derby.iapi.services.io.FormatIdInputStream
9: __226698 ___9068064 [Lorg.apache.derby.iapi.types.DataValueDescriptor;
The total is reported like this, which I interpret to mean that the heap usage is around 2.9 GB:
Total 70589681 -1158738416
Using the 48 byte version of StoredRecordHeader would add another 500 MB, causing the heap to be filled to the rim (I haven't checked if these objects refer other objects as well) which in turn effectively brings the application to an halt. With a 32-bit VM you can't give Derby more heap, and the only option would be to reduce the page cache size.
+1 from me based on testing the patch (I haven't reviewed the code changes).
In this specific case, 8 more bytes saved per instance would sum up to 253 MB.
was (Author: kristwaa):
I ran the patch with the load I experience problems with, and it fixes the problem for my configuration.
As I mentioned earlier, the patch may not solve the problem for other configurations due to the amount of heap available and the size of the Derby page cache.
In my case Derby was able to keep under 3.5 GB of heap with a page cache of size 262144 (which with 4K pages would be around 1 GB + overhead).
Here's the start out the heap histogram:
num #instances #bytes class name
--------------------------------------
1: 33176246 1061639872 org.apache.derby.impl.store.raw.data.StoredRecordHe
ader
2: 683347 961070736 [B
3: 33402612 801662688 org.apache.derby.impl.store.raw.data.RecordId
4: 226889 136070680 [Lorg.apache.derby.impl.store.raw.data.StoredRecordH
eader;
5: 226729 43531968 org.apache.derby.impl.store.raw.data.StoredPage
6: 237426 40619752 [C
7: 226653 14505792 org.apache.derby.impl.store.access.btree.LeafControl
Row
8: 226890 9075600 org.apache.derby.iapi.services.io.FormatIdInputStrea
m
9: 226698 9068064 [Lorg.apache.derby.iapi.types.DataValueDescriptor;
The total is reported like this, which I interpret to mean that the heap usage is around 2.9 GB:
Total 70589681 -1158738416
Using the 48 byte version of StoredRecordHeader would add another 500 MB, causing the heap to be filled to the rim (I haven't checked if these objects refer other objects as well) which in turn effectively brings the application to an halt. With a 32-bit VM you can't give Derby more heap, and the only option would be to reduce the page cache size.
+1 from me based on testing the patch (I haven't reviewed the code changes).
In this specific case, 8 more bytes saved per instance would sum up to 253 MB.
> Reduce memory footprint of StoredRecordHeader
> ---------------------------------------------
>
> Key: DERBY-3130
> URL: https://issues.apache.org/jira/browse/DERBY-3130
> Project: Derby
> Issue Type: Improvement
> Components: Store
> Affects Versions: 10.4.0.0
> Reporter: Knut Anders Hatlen
> Priority: Minor
> Attachments: SmallRecordsTest.java, srh.diff
>
>
> Derby's page cache often has a memory footprint that is much larger than pageSize*pageCacheSize. One large contributor to the footprint is the array of StoredPageHeader objects in BasePage. The memory consumed by these objects can be as large as, and sometimes even larger than, the byte arrays containing the raw page data. (See for instance http://www.nabble.com/How-much-derby-need-memory--tf3307655.html.) Reducing the size of the StoredPageHeader objects could therefore reduce Derby's memory footprint significantly, especially if the page cache is large and contains many pages from tables with small records or from indices.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.