You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-user@db.apache.org by Mattias Andersson <an...@gmail.com> on 2015/06/04 08:39:30 UTC
OutOfMemoryError
We got a 'java.lang.OutOfMemoryError: Java heap space' when using a Derby
disk database with the following schema.
CREATE TABLE TEST ( ID BIGINT NOT NULL
, NEXT _ID BIGINT
, PREV_ ID BIGINT
, TEST_XML XML);
ALTER TABLE TABLE
ADD PRIMARY KEY
(ID);
I opened the heap dump with Eclipse Memory Analyzer and found that tree
instances of org.apache.derby.impl.store.access.sort.MergeInserter occupies
290 MB of memory (that is 66 % of the total heap)
The problem seems to be that MergeInserter holds a sort buffer that
contains complete rows and the TEST_XML is about 160 kB each. MergeInserter
has an advanced algorithm to calculate the allowed size of the buffer, but
is it necessary store the complete row in the buffer, i.e. the non-indexed
values (TEST_XML is this case)?
Is there some property I can set to limit the buffer size or is it possible
to rewrite the SQL schema someway to exclude the XML type from the buffer?
Br Mattias
Following is the dominator tree from Eclipse Memory Analyzer:
org.apache.derby.impl.store.access.sort.MergeInserter @ 0xe3f72e20
|
72 | 127 533 080 | 30,76%
|- org.apache.derby.impl.store.access.sort.SortBuffer @ 0xe3f73018
| 48 | 127 532 792 | 30,76%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed5e0
| 40 | 164 384 | 0,04%
| | '- org.apache.derby.iapi.types.DataValueDescriptor[4] @
0xee3da280
| 32 | 164 344 | 0,04%
| | |- org.apache.derby.iapi.types.SQLClob @ 0xee3da2a0
| 56 | 164 240 | 0,04%
| | | |- java.lang.String @ 0xee3da2d8 < ?xml
version="1.0"?...|
24 | 164 160 | 0,04%
| | | |- char[1][] @ 0xee3da2f0
| 24 | 24 | 0,00%
| | | '- Total: 2 entries
| | |
| | |- org.apache.derby.iapi.types.SQLLongint @ 0xee3da308
| 24 | 24
| 0,00%
| | |- org.apache.derby.iapi.types.SQLLongint @ 0xee3da320
| 24 | 24 | 0,00%
| | |- org.apache.derby.iapi.types.SQLLongint @ 0xee3da338
| 24 | 24 | 0,00%
| | '- Total: 4 entries
|
| |
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed608
| 40 | 164 384 | 0,04%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed630
| 40 | 164 384
| 0,04%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed658
| 40 | 164 384 | 0,04%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed680
| 40 | 164 384 | 0,04%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed6a8
|
40 | 164 384 | 0,04%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed6d0
| 40 | 164 384 | 0,04%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed6f8
| 40 | 164 384 | 0,04%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed720
| 40 | 164 384 | 0,04%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3fedd50
| 40 | 164 384 | 0,04%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3fedd78
| 40 | 164 384 | 0,04%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3fedda0
| 40 | 164 384 | 0,04%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3feddc8
| 40 | 164 384 | 0,04%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3fedff8
| 40 | 164 384
| 0,04%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3fef5c8
| 40 | 164 384 | 0,04%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3fef5f0
| 40 | 164 384 | 0,04%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3fef618
|
40 | 164 384 | 0,04%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff01f8
| 40 | 164 384 | 0,04%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff04c8
| 40 | 164 384 | 0,04%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff0608
| 40 | 164 384 | 0,04%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff0630
| 40 | 164 384 | 0,04%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff0658
| 40 | 164 384 | 0,04%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff0680
| 40 | 164 384 | 0,04%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff06a8
| 40 | 164 384 | 0,04%
| |- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff06d0
| 40 | 164 384
| 0,04%
| '- Total: 25 of 780 entries; 755
Re: OutOfMemoryError
Posted by mike matrigali <mi...@gmail.com>.
Did you get the error when adding the primary key?
If you got the error on a different query, could you post that query, and possibly
the query plan being executed? A possible workaround might be to see if you
could create a different index such that the system could pick a query plan where
the sort is not needed.
As you mention derby does try to estimate ahead of time amount of memory, but it
has a number of problems:
1) the java interfaces do not work well for estimating amount of memory available in
the future.
2) derby estimates for blob/clob size are often wrong. I believe xml uses clob implementation
internally. I would guess the technical problem here is that derby underestimated the sort
size.
Providing a reproducible test case that you can check into JIRA will increase the likelyhood that
a developer will look at the issue.
/mikem
On 6/3/2015 11:39 PM, Mattias Andersson wrote:
>
> We got a 'java.lang.OutOfMemoryError: Java heap space' when using a Derby disk database with the following schema.
>
>
> CREATE TABLE TEST (ID BIGINT NOT NULL
>
>
> ,NEXT _ID BIGINT
>
> ,PREV_ ID BIGINT
>
> ,TEST_XML XML);
>
> ALTER TABLE TABLE
>
> ADD PRIMARY KEY
>
> (ID);
>
>
> I opened the heap dump with Eclipse Memory Analyzer and found that tree instances of
> org.apache.derby.impl.store.access.sort.MergeInserter occupies 290 MB of memory (that is 66 % of the total heap)
>
>
> The problem seems to be that MergeInserter holds a sort buffer that contains complete rows and theTEST_XML is about 160 kB each.
> MergeInserter has an advanced algorithm to calculate the allowed size of the buffer, but is it necessary store the complete row in
> the buffer, i.e. the non-indexed values (TEST_XML is this case)?
>
>
> Is there some property I can set to limit the buffer size or is it possible to rewrite the SQL schema someway to exclude the XML
> type from the buffer?
>
>
> Br Mattias
>
>
>
>
> Following is the dominator tree from Eclipse Memory Analyzer:
>
>
>
> org.apache.derby.impl.store.access.sort.MergeInserter @ 0xe3f72e20|72 |127 533 080 |30,76%
>
> |- org.apache.derby.impl.store.access.sort.SortBuffer @ 0xe3f73018|48 |127 532 792 |30,76%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed5e0|40 |164 384 |0,04%
>
> ||'- org.apache.derby.iapi.types.DataValueDescriptor[4] @ 0xee3da280|32 |164 344 |0,04%
>
> |||- org.apache.derby.iapi.types.SQLClob @ 0xee3da2a0|56 |164 240 |0,04%
>
> ||||- java.lang.String @ 0xee3da2d8< ?xml version="1.0"?...|24 |164 160 |0,04%
>
> ||||- char[1][] @ 0xee3da2f0|24 |24 |0,00%
>
> |||'- Total: 2 entries|||
>
> |||- org.apache.derby.iapi.types.SQLLongint @ 0xee3da308|24 |24 |0,00%
>
> |||- org.apache.derby.iapi.types.SQLLongint @ 0xee3da320|24 |24 |0,00%
>
> |||- org.apache.derby.iapi.types.SQLLongint @ 0xee3da338|24 |24 |0,00%
>
> ||'- Total: 4 entries|||
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed608|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed630|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed658|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed680|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed6a8|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed6d0|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed6f8|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed720|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fedd50|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fedd78|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fedda0|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3feddc8|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fedff8|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fef5c8|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fef5f0|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3fef618|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff01f8|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff04c8|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff0608|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff0630|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff0658|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff0680|40 |164 384 |0,04%
>
> ||- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff06a8|40 |164 384 |0,04%
>
> | |- org.apache.derby.impl.store.access.sort.Node @ 0xe3ff06d0|40 |164 384 |0,04%
>
> |'- Total: 25 of 780 entries; 755
>
--
email: Mike Matrigali - mikemapp1@gmail.com
linkedin: https://www.linkedin.com/in/MikeMatrigali