You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "Knut Anders Hatlen (JIRA)" <ji...@apache.org> on 2007/10/14 18:50:50 UTC
[jira] Created: (DERBY-3130) Reduce memory footprint of
StoredRecordHeader
Reduce memory footprint of StoredRecordHeader
---------------------------------------------
Key: DERBY-3130
URL: https://issues.apache.org/jira/browse/DERBY-3130
Project: Derby
Issue Type: Improvement
Components: Store
Affects Versions: 10.4.0.0
Reporter: Knut Anders Hatlen
Priority: Minor
Derby's page cache often has a memory footprint that is much larger than pageSize*pageCacheSize. One large contributor to the footprint is the array of StoredPageHeader objects in BasePage. The memory consumed by these objects can be as large as, and sometimes even larger than, the byte arrays containing the raw page data. (See for instance http://www.nabble.com/How-much-derby-need-memory--tf3307655.html.) Reducing the size of the StoredPageHeader objects could therefore reduce Derby's memory footprint significantly, especially if the page cache is large and contains many pages from tables with small records or from indices.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-3130) Reduce memory footprint of
StoredRecordHeader
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Knut Anders Hatlen updated DERBY-3130:
--------------------------------------
Derby Info: [Patch Available]
> Reduce memory footprint of StoredRecordHeader
> ---------------------------------------------
>
> Key: DERBY-3130
> URL: https://issues.apache.org/jira/browse/DERBY-3130
> Project: Derby
> Issue Type: Improvement
> Components: Store
> Affects Versions: 10.4.0.0
> Reporter: Knut Anders Hatlen
> Assignee: Knut Anders Hatlen
> Priority: Minor
> Attachments: d3130-1a.diff, SmallRecordsTest.java, srh.diff
>
>
> Derby's page cache often has a memory footprint that is much larger than pageSize*pageCacheSize. One large contributor to the footprint is the array of StoredPageHeader objects in BasePage. The memory consumed by these objects can be as large as, and sometimes even larger than, the byte arrays containing the raw page data. (See for instance http://www.nabble.com/How-much-derby-need-memory--tf3307655.html.) Reducing the size of the StoredPageHeader objects could therefore reduce Derby's memory footprint significantly, especially if the page cache is large and contains many pages from tables with small records or from indices.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-3130) Reduce memory footprint of
StoredRecordHeader
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Knut Anders Hatlen updated DERBY-3130:
--------------------------------------
Attachment: srh.diff
The attached patch srh.diff implements 1+2a, which is not the optimal solution, but it reduces the object size to 32 bytes. The patch is only for testing, and not ready for commit. Here's the output from "jmap -histo:live" when running SmallRecordsTest:
num #instances #bytes class name
--------------------------------------
1: 5879 4644664 [B
2: 136814 4378048 org.apache.derby.impl.store.raw.data.StoredRecordHeader
3: 22064 2226240 <constMethodKlass>
4: 22064 1769032 <methodKlass>
5: 3517 1457848 [I
6: 29809 1266424 <symbolKlass>
The memory consumed by StoredRecordHeader is reduced by ~2.2 MB.
> Reduce memory footprint of StoredRecordHeader
> ---------------------------------------------
>
> Key: DERBY-3130
> URL: https://issues.apache.org/jira/browse/DERBY-3130
> Project: Derby
> Issue Type: Improvement
> Components: Store
> Affects Versions: 10.4.0.0
> Reporter: Knut Anders Hatlen
> Priority: Minor
> Attachments: SmallRecordsTest.java, srh.diff
>
>
> Derby's page cache often has a memory footprint that is much larger than pageSize*pageCacheSize. One large contributor to the footprint is the array of StoredPageHeader objects in BasePage. The memory consumed by these objects can be as large as, and sometimes even larger than, the byte arrays containing the raw page data. (See for instance http://www.nabble.com/How-much-derby-need-memory--tf3307655.html.) Reducing the size of the StoredPageHeader objects could therefore reduce Derby's memory footprint significantly, especially if the page cache is large and contains many pages from tables with small records or from indices.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (DERBY-3130) Reduce memory footprint
of StoredRecordHeader
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536603 ]
kristwaa edited comment on DERBY-3130 at 10/22/07 2:12 AM:
------------------------------------------------------------------
I ran the patch with the load I experience problems with, and it fixes the problem for my configuration.
As I mentioned earlier, the patch may not solve the problem for other configurations due to the amount of heap available and the size of the Derby page cache.
In my case Derby was able to keep under 3.5 GB of heap with a page cache of size 262144 (which with 4K pages would be around 1 GB + overhead).
Here's the start out the heap histogram:
num #instances #bytes class name
--------------------------------------
1: 33176246 1061639872 org.apache.derby.impl.store.raw.data.StoredRecordHeader
2: __683347 _961070736 [B
3: 33402612 _801662688 org.apache.derby.impl.store.raw.data.RecordId
4: __226889 _136070680 [Lorg.apache.derby.impl.store.raw.data.StoredRecordHeader;
5: __226729 __43531968 org.apache.derby.impl.store.raw.data.StoredPage
6: __237426 __40619752 [C
7: __226653 __14505792 org.apache.derby.impl.store.access.btree.LeafControlRow
8: __226890 ___9075600 org.apache.derby.iapi.services.io.FormatIdInputStream
9: __226698 ___9068064 [Lorg.apache.derby.iapi.types.DataValueDescriptor;
The total is reported like this, which I interpret to mean that the heap usage is around 2.9 GB:
Total 70589681 -1158738416
Using the 48 byte version of StoredRecordHeader would add another 500 MB, causing the heap to be filled to the rim (I haven't checked if these objects refer other objects as well) which in turn effectively brings the application to an halt. With a 32-bit VM you can't give Derby more heap, and the only option would be to reduce the page cache size.
+1 from me based on testing the patch (I haven't reviewed the code changes).
In this specific case, 8 more bytes saved per instance would sum up to 253 MB.
was (Author: kristwaa):
I ran the patch with the load I experience problems with, and it fixes the problem for my configuration.
As I mentioned earlier, the patch may not solve the problem for other configurations due to the amount of heap available and the size of the Derby page cache.
In my case Derby was able to keep under 3.5 GB of heap with a page cache of size 262144 (which with 4K pages would be around 1 GB + overhead).
Here's the start out the heap histogram:
num #instances #bytes class name
--------------------------------------
1: 33176246 1061639872 org.apache.derby.impl.store.raw.data.StoredRecordHe
ader
2: 683347 961070736 [B
3: 33402612 801662688 org.apache.derby.impl.store.raw.data.RecordId
4: 226889 136070680 [Lorg.apache.derby.impl.store.raw.data.StoredRecordH
eader;
5: 226729 43531968 org.apache.derby.impl.store.raw.data.StoredPage
6: 237426 40619752 [C
7: 226653 14505792 org.apache.derby.impl.store.access.btree.LeafControl
Row
8: 226890 9075600 org.apache.derby.iapi.services.io.FormatIdInputStrea
m
9: 226698 9068064 [Lorg.apache.derby.iapi.types.DataValueDescriptor;
The total is reported like this, which I interpret to mean that the heap usage is around 2.9 GB:
Total 70589681 -1158738416
Using the 48 byte version of StoredRecordHeader would add another 500 MB, causing the heap to be filled to the rim (I haven't checked if these objects refer other objects as well) which in turn effectively brings the application to an halt. With a 32-bit VM you can't give Derby more heap, and the only option would be to reduce the page cache size.
+1 from me based on testing the patch (I haven't reviewed the code changes).
In this specific case, 8 more bytes saved per instance would sum up to 253 MB.
> Reduce memory footprint of StoredRecordHeader
> ---------------------------------------------
>
> Key: DERBY-3130
> URL: https://issues.apache.org/jira/browse/DERBY-3130
> Project: Derby
> Issue Type: Improvement
> Components: Store
> Affects Versions: 10.4.0.0
> Reporter: Knut Anders Hatlen
> Priority: Minor
> Attachments: SmallRecordsTest.java, srh.diff
>
>
> Derby's page cache often has a memory footprint that is much larger than pageSize*pageCacheSize. One large contributor to the footprint is the array of StoredPageHeader objects in BasePage. The memory consumed by these objects can be as large as, and sometimes even larger than, the byte arrays containing the raw page data. (See for instance http://www.nabble.com/How-much-derby-need-memory--tf3307655.html.) Reducing the size of the StoredPageHeader objects could therefore reduce Derby's memory footprint significantly, especially if the page cache is large and contains many pages from tables with small records or from indices.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3130) Reduce memory footprint of
StoredRecordHeader
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534776 ]
Knut Anders Hatlen commented on DERBY-3130:
-------------------------------------------
Thanks Kristian. You're right, it should be 1000 pages, not 4000. I'll try to create a patch you can test.
> Reduce memory footprint of StoredRecordHeader
> ---------------------------------------------
>
> Key: DERBY-3130
> URL: https://issues.apache.org/jira/browse/DERBY-3130
> Project: Derby
> Issue Type: Improvement
> Components: Store
> Affects Versions: 10.4.0.0
> Reporter: Knut Anders Hatlen
> Priority: Minor
> Attachments: SmallRecordsTest.java
>
>
> Derby's page cache often has a memory footprint that is much larger than pageSize*pageCacheSize. One large contributor to the footprint is the array of StoredPageHeader objects in BasePage. The memory consumed by these objects can be as large as, and sometimes even larger than, the byte arrays containing the raw page data. (See for instance http://www.nabble.com/How-much-derby-need-memory--tf3307655.html.) Reducing the size of the StoredPageHeader objects could therefore reduce Derby's memory footprint significantly, especially if the page cache is large and contains many pages from tables with small records or from indices.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3130) Reduce memory footprint of
StoredRecordHeader
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536651 ]
Knut Anders Hatlen commented on DERBY-3130:
-------------------------------------------
Thanks for testing the patch, Kristian. I'm glad it solved the problems for you, but I think the patch needs to be tidied before it can be committed.
> Reduce memory footprint of StoredRecordHeader
> ---------------------------------------------
>
> Key: DERBY-3130
> URL: https://issues.apache.org/jira/browse/DERBY-3130
> Project: Derby
> Issue Type: Improvement
> Components: Store
> Affects Versions: 10.4.0.0
> Reporter: Knut Anders Hatlen
> Priority: Minor
> Attachments: SmallRecordsTest.java, srh.diff
>
>
> Derby's page cache often has a memory footprint that is much larger than pageSize*pageCacheSize. One large contributor to the footprint is the array of StoredPageHeader objects in BasePage. The memory consumed by these objects can be as large as, and sometimes even larger than, the byte arrays containing the raw page data. (See for instance http://www.nabble.com/How-much-derby-need-memory--tf3307655.html.) Reducing the size of the StoredPageHeader objects could therefore reduce Derby's memory footprint significantly, especially if the page cache is large and contains many pages from tables with small records or from indices.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (DERBY-3130) Reduce memory footprint of
StoredRecordHeader
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Knut Anders Hatlen resolved DERBY-3130.
---------------------------------------
Resolution: Fixed
Fix Version/s: 10.4.1.0
Committed to 10.4 with revision 644209.
> Reduce memory footprint of StoredRecordHeader
> ---------------------------------------------
>
> Key: DERBY-3130
> URL: https://issues.apache.org/jira/browse/DERBY-3130
> Project: Derby
> Issue Type: Improvement
> Components: Store
> Affects Versions: 10.4.0.0
> Reporter: Knut Anders Hatlen
> Assignee: Knut Anders Hatlen
> Priority: Minor
> Fix For: 10.4.1.0, 10.5.0.0
>
> Attachments: d3130-1a.diff, SmallRecordsTest.java, srh.diff
>
>
> Derby's page cache often has a memory footprint that is much larger than pageSize*pageCacheSize. One large contributor to the footprint is the array of StoredPageHeader objects in BasePage. The memory consumed by these objects can be as large as, and sometimes even larger than, the byte arrays containing the raw page data. (See for instance http://www.nabble.com/How-much-derby-need-memory--tf3307655.html.) Reducing the size of the StoredPageHeader objects could therefore reduce Derby's memory footprint significantly, especially if the page cache is large and contains many pages from tables with small records or from indices.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3130) Reduce memory footprint of
StoredRecordHeader
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534771 ]
Kristian Waagan commented on DERBY-3130:
----------------------------------------
Hi Knut,
I think this is a good fix. I have gotten into trouble with larger databases when pulling in lots of indexes (normally using a SELECT COUNT(*) ). The problem for me has been that the page cache gets bigger when it is filled with index pages than when it is filled with data pages.
In some cases, the only way around it has been to reduce the page cache size, but then Derby is not allowed to utilize all the memory it could have during normal processing.
I think there is a typo in your last comment, the default page cache size is 1000 pages, not 4000.
If a patch is created I can test it out in one of my runs that experience the described problem.
> Reduce memory footprint of StoredRecordHeader
> ---------------------------------------------
>
> Key: DERBY-3130
> URL: https://issues.apache.org/jira/browse/DERBY-3130
> Project: Derby
> Issue Type: Improvement
> Components: Store
> Affects Versions: 10.4.0.0
> Reporter: Knut Anders Hatlen
> Priority: Minor
> Attachments: SmallRecordsTest.java
>
>
> Derby's page cache often has a memory footprint that is much larger than pageSize*pageCacheSize. One large contributor to the footprint is the array of StoredPageHeader objects in BasePage. The memory consumed by these objects can be as large as, and sometimes even larger than, the byte arrays containing the raw page data. (See for instance http://www.nabble.com/How-much-derby-need-memory--tf3307655.html.) Reducing the size of the StoredPageHeader objects could therefore reduce Derby's memory footprint significantly, especially if the page cache is large and contains many pages from tables with small records or from indices.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3130) Reduce memory footprint of
StoredRecordHeader
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534645 ]
Knut Anders Hatlen commented on DERBY-3130:
-------------------------------------------
>From a quick look at the class, it seems like there are at least two obvious candidates for improvement:
1) Only four bits of the status field are in use, so it could be changed from int to byte
2) The fields overflowId (int), overflowPage (long) and firstField (int) are only used if the records are overflowed. Factoring out these fields could reduce the object size when there's no overflow (which is the common case). One could either (a) create an inner class with these three fields and let StoredRecordHeader have a reference to such an object, or null if the fields are not needed, or (b) create a subclass of StoredRecordHeader which contains the extra fields.
With Sun's Java 6 running in 32-bit mode, doing 1+2a will reduce the object size from 48 bytes to 32 bytes. Doing 1+2b will reduce the size further down to 24 bytes. In 64-bit mode, the sizes are 56 bytes -> 48 bytes -> 40 bytes. (I haven't actually implemented the changes to find the numbers, only changed variable types and made variables static to see how eliminating them would affect the size).
> Reduce memory footprint of StoredRecordHeader
> ---------------------------------------------
>
> Key: DERBY-3130
> URL: https://issues.apache.org/jira/browse/DERBY-3130
> Project: Derby
> Issue Type: Improvement
> Components: Store
> Affects Versions: 10.4.0.0
> Reporter: Knut Anders Hatlen
> Priority: Minor
>
> Derby's page cache often has a memory footprint that is much larger than pageSize*pageCacheSize. One large contributor to the footprint is the array of StoredPageHeader objects in BasePage. The memory consumed by these objects can be as large as, and sometimes even larger than, the byte arrays containing the raw page data. (See for instance http://www.nabble.com/How-much-derby-need-memory--tf3307655.html.) Reducing the size of the StoredPageHeader objects could therefore reduce Derby's memory footprint significantly, especially if the page cache is large and contains many pages from tables with small records or from indices.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-3130) Reduce memory footprint of
StoredRecordHeader
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Knut Anders Hatlen updated DERBY-3130:
--------------------------------------
Attachment: d3130-1a.diff
I have cleaned up the srh.diff patch and attached it as d3130-1a.diff. Derbyall ran cleanly, and suites.All had 3 known failures (caused by DERBY-3519 and DERBY-3561). Although implementing alternative 2b would save even more space, it seems like it's a much bigger rewrite, so I don't think it's worth doing it at this point.
> Reduce memory footprint of StoredRecordHeader
> ---------------------------------------------
>
> Key: DERBY-3130
> URL: https://issues.apache.org/jira/browse/DERBY-3130
> Project: Derby
> Issue Type: Improvement
> Components: Store
> Affects Versions: 10.4.0.0
> Reporter: Knut Anders Hatlen
> Assignee: Knut Anders Hatlen
> Priority: Minor
> Attachments: d3130-1a.diff, SmallRecordsTest.java, srh.diff
>
>
> Derby's page cache often has a memory footprint that is much larger than pageSize*pageCacheSize. One large contributor to the footprint is the array of StoredPageHeader objects in BasePage. The memory consumed by these objects can be as large as, and sometimes even larger than, the byte arrays containing the raw page data. (See for instance http://www.nabble.com/How-much-derby-need-memory--tf3307655.html.) Reducing the size of the StoredPageHeader objects could therefore reduce Derby's memory footprint significantly, especially if the page cache is large and contains many pages from tables with small records or from indices.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-3130) Reduce memory footprint of
StoredRecordHeader
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Knut Anders Hatlen updated DERBY-3130:
--------------------------------------
Derby Info: (was: [Patch Available])
Fix Version/s: 10.5.0.0
Committed revision 643875.
I also plan to merge the fix to the 10.4 branch.
> Reduce memory footprint of StoredRecordHeader
> ---------------------------------------------
>
> Key: DERBY-3130
> URL: https://issues.apache.org/jira/browse/DERBY-3130
> Project: Derby
> Issue Type: Improvement
> Components: Store
> Affects Versions: 10.4.0.0
> Reporter: Knut Anders Hatlen
> Assignee: Knut Anders Hatlen
> Priority: Minor
> Fix For: 10.5.0.0
>
> Attachments: d3130-1a.diff, SmallRecordsTest.java, srh.diff
>
>
> Derby's page cache often has a memory footprint that is much larger than pageSize*pageCacheSize. One large contributor to the footprint is the array of StoredPageHeader objects in BasePage. The memory consumed by these objects can be as large as, and sometimes even larger than, the byte arrays containing the raw page data. (See for instance http://www.nabble.com/How-much-derby-need-memory--tf3307655.html.) Reducing the size of the StoredPageHeader objects could therefore reduce Derby's memory footprint significantly, especially if the page cache is large and contains many pages from tables with small records or from indices.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3130) Reduce memory footprint of
StoredRecordHeader
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536603 ]
Kristian Waagan commented on DERBY-3130:
----------------------------------------
I ran the patch with the load I experience problems with, and it fixes the problem for my configuration.
As I mentioned earlier, the patch may not solve the problem for other configurations due to the amount of heap available and the size of the Derby page cache.
In my case Derby was able to keep under 3.5 GB of heap with a page cache of size 262144 (which with 4K pages would be around 1 GB + overhead).
Here's the start out the heap histogram:
num #instances #bytes class name
--------------------------------------
1: 33176246 1061639872 org.apache.derby.impl.store.raw.data.StoredRecordHe
ader
2: 683347 961070736 [B
3: 33402612 801662688 org.apache.derby.impl.store.raw.data.RecordId
4: 226889 136070680 [Lorg.apache.derby.impl.store.raw.data.StoredRecordH
eader;
5: 226729 43531968 org.apache.derby.impl.store.raw.data.StoredPage
6: 237426 40619752 [C
7: 226653 14505792 org.apache.derby.impl.store.access.btree.LeafControl
Row
8: 226890 9075600 org.apache.derby.iapi.services.io.FormatIdInputStrea
m
9: 226698 9068064 [Lorg.apache.derby.iapi.types.DataValueDescriptor;
The total is reported like this, which I interpret to mean that the heap usage is around 2.9 GB:
Total 70589681 -1158738416
Using the 48 byte version of StoredRecordHeader would add another 500 MB, causing the heap to be filled to the rim (I haven't checked if these objects refer other objects as well) which in turn effectively brings the application to an halt. With a 32-bit VM you can't give Derby more heap, and the only option would be to reduce the page cache size.
+1 from me based on testing the patch (I haven't reviewed the code changes).
In this specific case, 8 more bytes saved per instance would sum up to 253 MB.
> Reduce memory footprint of StoredRecordHeader
> ---------------------------------------------
>
> Key: DERBY-3130
> URL: https://issues.apache.org/jira/browse/DERBY-3130
> Project: Derby
> Issue Type: Improvement
> Components: Store
> Affects Versions: 10.4.0.0
> Reporter: Knut Anders Hatlen
> Priority: Minor
> Attachments: SmallRecordsTest.java, srh.diff
>
>
> Derby's page cache often has a memory footprint that is much larger than pageSize*pageCacheSize. One large contributor to the footprint is the array of StoredPageHeader objects in BasePage. The memory consumed by these objects can be as large as, and sometimes even larger than, the byte arrays containing the raw page data. (See for instance http://www.nabble.com/How-much-derby-need-memory--tf3307655.html.) Reducing the size of the StoredPageHeader objects could therefore reduce Derby's memory footprint significantly, especially if the page cache is large and contains many pages from tables with small records or from indices.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-3130) Reduce memory footprint of
StoredRecordHeader
Posted by "Mike Matrigali (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Matrigali updated DERBY-3130:
----------------------------------
Fix Version/s: 10.3.3.0
commited 813145:
backporting fix (revision 643875) from trunk to 10.3 branch.
> Reduce memory footprint of StoredRecordHeader
> ---------------------------------------------
>
> Key: DERBY-3130
> URL: https://issues.apache.org/jira/browse/DERBY-3130
> Project: Derby
> Issue Type: Improvement
> Components: Store
> Affects Versions: 10.4.1.3
> Reporter: Knut Anders Hatlen
> Assignee: Knut Anders Hatlen
> Priority: Minor
> Fix For: 10.3.3.0, 10.4.1.3, 10.5.1.1
>
> Attachments: d3130-1a.diff, SmallRecordsTest.java, srh.diff
>
>
> Derby's page cache often has a memory footprint that is much larger than pageSize*pageCacheSize. One large contributor to the footprint is the array of StoredPageHeader objects in BasePage. The memory consumed by these objects can be as large as, and sometimes even larger than, the byte arrays containing the raw page data. (See for instance http://www.nabble.com/How-much-derby-need-memory--tf3307655.html.) Reducing the size of the StoredPageHeader objects could therefore reduce Derby's memory footprint significantly, especially if the page cache is large and contains many pages from tables with small records or from indices.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3130) Reduce memory footprint of
StoredRecordHeader
Posted by "Bryan Pendleton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534895 ]
Bryan Pendleton commented on DERBY-3130:
----------------------------------------
Could you educate us a bit more about the StoredRecordHeader? What is
the life cycle of a StoredRecordHeader? (when are they created and when
reclaimed?). I'm trying to understand why your benchmark has 136,814
instances of the StoredRecordHeader: is that because your test touched
that many records in a single transaction? Or because you touched that
many records during the entire benchmark? Or because there were 136.8 records
per page and you had 1,000 pages in memory? Or some other reason? Thanks!
> Reduce memory footprint of StoredRecordHeader
> ---------------------------------------------
>
> Key: DERBY-3130
> URL: https://issues.apache.org/jira/browse/DERBY-3130
> Project: Derby
> Issue Type: Improvement
> Components: Store
> Affects Versions: 10.4.0.0
> Reporter: Knut Anders Hatlen
> Priority: Minor
> Attachments: SmallRecordsTest.java, srh.diff
>
>
> Derby's page cache often has a memory footprint that is much larger than pageSize*pageCacheSize. One large contributor to the footprint is the array of StoredPageHeader objects in BasePage. The memory consumed by these objects can be as large as, and sometimes even larger than, the byte arrays containing the raw page data. (See for instance http://www.nabble.com/How-much-derby-need-memory--tf3307655.html.) Reducing the size of the StoredPageHeader objects could therefore reduce Derby's memory footprint significantly, especially if the page cache is large and contains many pages from tables with small records or from indices.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-3130) Reduce memory footprint of
StoredRecordHeader
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Knut Anders Hatlen updated DERBY-3130:
--------------------------------------
Attachment: SmallRecordsTest.java
Attached is a test which fills a table with 500000 small records (consisting of a single int) and than waits for the user to press enter before it finishes. With a default sized page cache (4000 pages), this is what jmap -histo:live says about the contents of the heap when all the records have been inserted (Sun Java 6, 32-bit):
num #instances #bytes class name
--------------------------------------
1: 136814 6567072 org.apache.derby.impl.store.raw.data.StoredRecordHeader
2: 5239 4568104 [B
3: 14465 1447800 <constMethodKlass>
4: 14465 1158248 <methodKlass>
5: 17416 808920 <symbolKlass>
6: 1000 559280 [Lorg.apache.derby.impl.store.raw.data.StoredRecordHeader;
The page cache is expected to contain ~4MB of byte arrays, but the StoredRecordHeader objects consume even more memory, ~6.5MB.
> Reduce memory footprint of StoredRecordHeader
> ---------------------------------------------
>
> Key: DERBY-3130
> URL: https://issues.apache.org/jira/browse/DERBY-3130
> Project: Derby
> Issue Type: Improvement
> Components: Store
> Affects Versions: 10.4.0.0
> Reporter: Knut Anders Hatlen
> Priority: Minor
> Attachments: SmallRecordsTest.java
>
>
> Derby's page cache often has a memory footprint that is much larger than pageSize*pageCacheSize. One large contributor to the footprint is the array of StoredPageHeader objects in BasePage. The memory consumed by these objects can be as large as, and sometimes even larger than, the byte arrays containing the raw page data. (See for instance http://www.nabble.com/How-much-derby-need-memory--tf3307655.html.) Reducing the size of the StoredPageHeader objects could therefore reduce Derby's memory footprint significantly, especially if the page cache is large and contains many pages from tables with small records or from indices.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (DERBY-3130) Reduce memory footprint of
StoredRecordHeader
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Knut Anders Hatlen reassigned DERBY-3130:
-----------------------------------------
Assignee: Knut Anders Hatlen
> Reduce memory footprint of StoredRecordHeader
> ---------------------------------------------
>
> Key: DERBY-3130
> URL: https://issues.apache.org/jira/browse/DERBY-3130
> Project: Derby
> Issue Type: Improvement
> Components: Store
> Affects Versions: 10.4.0.0
> Reporter: Knut Anders Hatlen
> Assignee: Knut Anders Hatlen
> Priority: Minor
> Attachments: SmallRecordsTest.java, srh.diff
>
>
> Derby's page cache often has a memory footprint that is much larger than pageSize*pageCacheSize. One large contributor to the footprint is the array of StoredPageHeader objects in BasePage. The memory consumed by these objects can be as large as, and sometimes even larger than, the byte arrays containing the raw page data. (See for instance http://www.nabble.com/How-much-derby-need-memory--tf3307655.html.) Reducing the size of the StoredPageHeader objects could therefore reduce Derby's memory footprint significantly, especially if the page cache is large and contains many pages from tables with small records or from indices.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-3130) Reduce memory footprint of
StoredRecordHeader
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534918 ]
Knut Anders Hatlen commented on DERBY-3130:
-------------------------------------------
A StoredRecordHeader object is created the first time its corresponding record is accessed after a page has been fetched into the page cache. When a page is evicted from the cache, all its StoredRecordHeader objects can be gc'ed. The number 136814 means that there are 1000 pages, on each of which 136.8 records have been accessed.
> Reduce memory footprint of StoredRecordHeader
> ---------------------------------------------
>
> Key: DERBY-3130
> URL: https://issues.apache.org/jira/browse/DERBY-3130
> Project: Derby
> Issue Type: Improvement
> Components: Store
> Affects Versions: 10.4.0.0
> Reporter: Knut Anders Hatlen
> Priority: Minor
> Attachments: SmallRecordsTest.java, srh.diff
>
>
> Derby's page cache often has a memory footprint that is much larger than pageSize*pageCacheSize. One large contributor to the footprint is the array of StoredPageHeader objects in BasePage. The memory consumed by these objects can be as large as, and sometimes even larger than, the byte arrays containing the raw page data. (See for instance http://www.nabble.com/How-much-derby-need-memory--tf3307655.html.) Reducing the size of the StoredPageHeader objects could therefore reduce Derby's memory footprint significantly, especially if the page cache is large and contains many pages from tables with small records or from indices.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.