You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "Ramin Baradari (JIRA)" <ji...@apache.org> on 2011/09/16 15:06:08 UTC
[jira] [Created] (DERBY-5416) SYSCS_COMPRESS_TABLE causes an
OutOfMemoryError when the heap is full at call time and then gets mostly
garbage collected later on
SYSCS_COMPRESS_TABLE causes an OutOfMemoryError when the heap is full at call time and then gets mostly garbage collected later on
----------------------------------------------------------------------------------------------------------------------------------
Key: DERBY-5416
URL: https://issues.apache.org/jira/browse/DERBY-5416
Project: Derby
Issue Type: Bug
Components: Store
Affects Versions: 10.8.1.2, 10.7.1.1, 10.6.2.1
Reporter: Ramin Baradari
Priority: Critical
When compressing a table with an index that is larger than the maximum heap size and therefore cannot be hold in memory as a whole an OutOfMemoryError can occur.
For this to happen the heap usage must be close to the maximum heap size at the start of the index recreation and then while the entries are sorted a garbage collection run must clean out most of the heap. This can happen because a concurrent process releases a huge chunk of memory or just because the buffer of a previous table compression has not yet been garbage collected.
The internally used heuristics to guess when more memory can be used for the merge inserter estimates that more memory is available and then the sort buffer gets doubled. The buffer size gets doubled until the heap usage is back to the level when the merge inserter was first initialized or when the OOM occurs.
The problem lies in MergeInsert.insert(...). The check if the buffer can be doubled contains the expression "estimatedMemoryUsed < 0" where estimatedMemoryUsed is the difference in current heap usage and heap usage at initialization. Unfortunately, in the aforementioned scenario this will be true until the heap usage will reach close to maximum heap size before doubling the buffer size will be stopped.
I've tested it with 10.6.2.1, 10.7.1.1 and 10.8.1.2 but the actual bug most likely exists in prior versions too.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (DERBY-5416) SYSCS_COMPRESS_TABLE causes an
OutOfMemoryError when the heap is full at call time and then gets mostly
garbage collected later on
Posted by "Ramin Baradari (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13106042#comment-13106042 ]
Ramin Baradari commented on DERBY-5416:
---------------------------------------
Patch "compress_test_5416.patch" adds a new test method to CompressTableTest.java that reproduces the crash. The test reproduced the crash on my systems each time I ran it but the bug depends a bit on the proper timing so it might actually succeed on systems. The test is a bit slow takes about 10minutes to complete.
The test creates a table with an index that is larger in data size than the 512mb maximum heap size the test runner has. (should be about 600mb+). It then fills the heap in chunks of several megabytes until an OOM occurs that is catched and ignore. The reference to that dummy data is then released and the table compression is called.
> SYSCS_COMPRESS_TABLE causes an OutOfMemoryError when the heap is full at call time and then gets mostly garbage collected later on
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-5416
> URL: https://issues.apache.org/jira/browse/DERBY-5416
> Project: Derby
> Issue Type: Bug
> Components: Store
> Affects Versions: 10.6.2.1, 10.7.1.1, 10.8.1.2
> Reporter: Ramin Baradari
> Priority: Critical
> Attachments: compress_test_5416.patch
>
>
> When compressing a table with an index that is larger than the maximum heap size and therefore cannot be hold in memory as a whole an OutOfMemoryError can occur.
> For this to happen the heap usage must be close to the maximum heap size at the start of the index recreation and then while the entries are sorted a garbage collection run must clean out most of the heap. This can happen because a concurrent process releases a huge chunk of memory or just because the buffer of a previous table compression has not yet been garbage collected.
> The internally used heuristics to guess when more memory can be used for the merge inserter estimates that more memory is available and then the sort buffer gets doubled. The buffer size gets doubled until the heap usage is back to the level when the merge inserter was first initialized or when the OOM occurs.
> The problem lies in MergeInsert.insert(...). The check if the buffer can be doubled contains the expression "estimatedMemoryUsed < 0" where estimatedMemoryUsed is the difference in current heap usage and heap usage at initialization. Unfortunately, in the aforementioned scenario this will be true until the heap usage will reach close to maximum heap size before doubling the buffer size will be stopped.
> I've tested it with 10.6.2.1, 10.7.1.1 and 10.8.1.2 but the actual bug most likely exists in prior versions too.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (DERBY-5416) SYSCS_COMPRESS_TABLE causes an
OutOfMemoryError when the heap is full at call time and then gets mostly
garbage collected later on
Posted by "Ramin Baradari (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ramin Baradari updated DERBY-5416:
----------------------------------
Attachment: compress_test_5416.patch
> SYSCS_COMPRESS_TABLE causes an OutOfMemoryError when the heap is full at call time and then gets mostly garbage collected later on
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-5416
> URL: https://issues.apache.org/jira/browse/DERBY-5416
> Project: Derby
> Issue Type: Bug
> Components: Store
> Affects Versions: 10.6.2.1, 10.7.1.1, 10.8.1.2
> Reporter: Ramin Baradari
> Priority: Critical
> Attachments: compress_test_5416.patch
>
>
> When compressing a table with an index that is larger than the maximum heap size and therefore cannot be hold in memory as a whole an OutOfMemoryError can occur.
> For this to happen the heap usage must be close to the maximum heap size at the start of the index recreation and then while the entries are sorted a garbage collection run must clean out most of the heap. This can happen because a concurrent process releases a huge chunk of memory or just because the buffer of a previous table compression has not yet been garbage collected.
> The internally used heuristics to guess when more memory can be used for the merge inserter estimates that more memory is available and then the sort buffer gets doubled. The buffer size gets doubled until the heap usage is back to the level when the merge inserter was first initialized or when the OOM occurs.
> The problem lies in MergeInsert.insert(...). The check if the buffer can be doubled contains the expression "estimatedMemoryUsed < 0" where estimatedMemoryUsed is the difference in current heap usage and heap usage at initialization. Unfortunately, in the aforementioned scenario this will be true until the heap usage will reach close to maximum heap size before doubling the buffer size will be stopped.
> I've tested it with 10.6.2.1, 10.7.1.1 and 10.8.1.2 but the actual bug most likely exists in prior versions too.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (DERBY-5416) SYSCS_COMPRESS_TABLE causes an
OutOfMemoryError when the heap is full at call time and then gets mostly
garbage collected later on
Posted by "Kristian Waagan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13106072#comment-13106072 ]
Kristian Waagan commented on DERBY-5416:
----------------------------------------
Thanks for taking the time to write up a bug report and to provide a repro, Ramin.
FYI, we do have a separate low-memory test suite that's run with a 16 MB heap. It can be run with 'ant junit-lowmem', the suite is found in java/testing/org/apache/derbyTesting/functionTests/tests/memory
> SYSCS_COMPRESS_TABLE causes an OutOfMemoryError when the heap is full at call time and then gets mostly garbage collected later on
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-5416
> URL: https://issues.apache.org/jira/browse/DERBY-5416
> Project: Derby
> Issue Type: Bug
> Components: Store
> Affects Versions: 10.6.2.1, 10.7.1.1, 10.8.1.2
> Reporter: Ramin Baradari
> Priority: Critical
> Attachments: compress_test_5416.patch
>
>
> When compressing a table with an index that is larger than the maximum heap size and therefore cannot be hold in memory as a whole an OutOfMemoryError can occur.
> For this to happen the heap usage must be close to the maximum heap size at the start of the index recreation and then while the entries are sorted a garbage collection run must clean out most of the heap. This can happen because a concurrent process releases a huge chunk of memory or just because the buffer of a previous table compression has not yet been garbage collected.
> The internally used heuristics to guess when more memory can be used for the merge inserter estimates that more memory is available and then the sort buffer gets doubled. The buffer size gets doubled until the heap usage is back to the level when the merge inserter was first initialized or when the OOM occurs.
> The problem lies in MergeInsert.insert(...). The check if the buffer can be doubled contains the expression "estimatedMemoryUsed < 0" where estimatedMemoryUsed is the difference in current heap usage and heap usage at initialization. Unfortunately, in the aforementioned scenario this will be true until the heap usage will reach close to maximum heap size before doubling the buffer size will be stopped.
> I've tested it with 10.6.2.1, 10.7.1.1 and 10.8.1.2 but the actual bug most likely exists in prior versions too.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (DERBY-5416) SYSCS_COMPRESS_TABLE causes an
OutOfMemoryError when the heap is full at call time and then gets mostly
garbage collected later on
Posted by "Kathey Marsden (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164675#comment-13164675 ]
Kathey Marsden commented on DERBY-5416:
---------------------------------------
I wonder, is this really a bug or just a case where we simply do not have enough heap to execute the task so it fails with an OOM?
> SYSCS_COMPRESS_TABLE causes an OutOfMemoryError when the heap is full at call time and then gets mostly garbage collected later on
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-5416
> URL: https://issues.apache.org/jira/browse/DERBY-5416
> Project: Derby
> Issue Type: Bug
> Components: Store
> Affects Versions: 10.6.2.1, 10.7.1.1, 10.8.1.2
> Reporter: Ramin Baradari
> Priority: Critical
> Labels: derby_triage10_9
> Attachments: compress_test_5416.patch
>
>
> When compressing a table with an index that is larger than the maximum heap size and therefore cannot be hold in memory as a whole an OutOfMemoryError can occur.
> For this to happen the heap usage must be close to the maximum heap size at the start of the index recreation and then while the entries are sorted a garbage collection run must clean out most of the heap. This can happen because a concurrent process releases a huge chunk of memory or just because the buffer of a previous table compression has not yet been garbage collected.
> The internally used heuristics to guess when more memory can be used for the merge inserter estimates that more memory is available and then the sort buffer gets doubled. The buffer size gets doubled until the heap usage is back to the level when the merge inserter was first initialized or when the OOM occurs.
> The problem lies in MergeInsert.insert(...). The check if the buffer can be doubled contains the expression "estimatedMemoryUsed < 0" where estimatedMemoryUsed is the difference in current heap usage and heap usage at initialization. Unfortunately, in the aforementioned scenario this will be true until the heap usage will reach close to maximum heap size before doubling the buffer size will be stopped.
> I've tested it with 10.6.2.1, 10.7.1.1 and 10.8.1.2 but the actual bug most likely exists in prior versions too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (DERBY-5416) SYSCS_COMPRESS_TABLE causes an
OutOfMemoryError when the heap is full at call time and then gets mostly
garbage collected later on
Posted by "Mike Matrigali (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Matrigali updated DERBY-5416:
----------------------------------
Urgency: Normal
Any suggestions on a better way to determine if sort should use more in-memory. It seems for performance derby should be
using more memory for these operations automatically, but as this issue points out operations can fail if the choice is wrong.
Solutions should be zero-admin by default, with maybe tuning knobs to work around issues like this one where the default does
not work.
> SYSCS_COMPRESS_TABLE causes an OutOfMemoryError when the heap is full at call time and then gets mostly garbage collected later on
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-5416
> URL: https://issues.apache.org/jira/browse/DERBY-5416
> Project: Derby
> Issue Type: Bug
> Components: Store
> Affects Versions: 10.6.2.1, 10.7.1.1, 10.8.1.2
> Reporter: Ramin Baradari
> Priority: Critical
> Labels: derby_triage10_9
> Attachments: compress_test_5416.patch
>
>
> When compressing a table with an index that is larger than the maximum heap size and therefore cannot be hold in memory as a whole an OutOfMemoryError can occur.
> For this to happen the heap usage must be close to the maximum heap size at the start of the index recreation and then while the entries are sorted a garbage collection run must clean out most of the heap. This can happen because a concurrent process releases a huge chunk of memory or just because the buffer of a previous table compression has not yet been garbage collected.
> The internally used heuristics to guess when more memory can be used for the merge inserter estimates that more memory is available and then the sort buffer gets doubled. The buffer size gets doubled until the heap usage is back to the level when the merge inserter was first initialized or when the OOM occurs.
> The problem lies in MergeInsert.insert(...). The check if the buffer can be doubled contains the expression "estimatedMemoryUsed < 0" where estimatedMemoryUsed is the difference in current heap usage and heap usage at initialization. Unfortunately, in the aforementioned scenario this will be true until the heap usage will reach close to maximum heap size before doubling the buffer size will be stopped.
> I've tested it with 10.6.2.1, 10.7.1.1 and 10.8.1.2 but the actual bug most likely exists in prior versions too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira