You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "Marco (JIRA)" <ji...@apache.org> on 2019/08/11 11:55:00 UTC
[jira] [Comment Edited] (DERBY-7049) OutOfMemoryError: Compressed
class space
[ https://issues.apache.org/jira/browse/DERBY-7049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904620#comment-16904620 ]
Marco edited comment on DERBY-7049 at 8/11/19 11:54 AM:
--------------------------------------------------------
Thanks a lot!
But I won't extend DataNucleus to use such a special solution. Instead, I refactored my code to organise the IDs in ranges and then create a query like
{code:sql}
SELECT ... FROM ...
WHERE
"a0"."discriminator" = 'SsNormalFile' AND
(
("a0"."id" >= ? "a0"."id" <= ?) OR
("a0"."id" >= ? "a0"."id" <= ?) OR
("a0"."id" >= ? "a0"."id" <= ?)
)
{code}
The number of ranges in my queries is hard-coded to 10 and I simply repeat the query multiple times (in the Java code), if there are more than 10 ranges.
This reduces the quantity of unique queries (analysed via derby.log and StatementLogReadingVTI.java as explained by you above) from multiple thousand (and potentially unlimited) to less than 200. GREAT! Thanks again for this valuable hint.
However, I thought that this would solve the problem, but my friend still encountered the same exception again :-( His derby.log had a size of over 50 GB, when it happened, so that he decided to throw it away and try it again for a shorter time (only about 2 hours -- the queries should be the same). Unfortunately, though, in this smaller log (only about 5 GB), there were still only less than 200 unique queries. I'm thus not sure, whether the problem is a different one, now, or whether derby compiles new classes even though the very same query-text was already used before (and there are only roughly 200).
Is the query-class-cache maybe somehow limited? Maybe derby evicts entries from this cache, but the classes are still in the ClassLoader? Or is there maybe some other constraint on the query-class-reusing-code, e.g. a maximum query-text-length?
If there are constraints like cache-size or query-text-length, can this be configured using some system-properties?
P.S.: Sorry for my late reaction. I'm usually very busy with paid work and don't have much time to help my friend fixing his troubles with my hobby project ;-)
was (Author: nlmarco):
Thanks a lot!
But I won't extend DataNucleus to use such a special solution. Instead, I refactored my code to organise the IDs in ranges and then create a query like
{code:sql}
SELECT ... FROM ...
WHERE
"a0"."discriminator" = 'SsNormalFile' AND
(("a0"."id" >= ? "a0"."id" <= ?) OR ("a0"."id" >= ? "a0"."id" <= ?) OR ("a0"."id" >= ? "a0"."id" <= ?))
{code}
This reduces the quantity of unique queries (analysed via derby.log and StatementLogReadingVTI.java as explained by you above) from multiple thousand (and potentially unlimited) to less than 200. GREAT! Thanks again for this valuable hint.
However, I thought that this would solve the problem, but my friend still encountered the same exception again :-( His derby.log had a size of over 50 GB, when it happened, so that he decided to throw it away and try it again for a shorter time (only about 2 hours -- the queries should be the same). Unfortunately, though, in this smaller log (only about 5 GB), there were still only less than 200 unique queries. I'm thus not sure, whether the problem is a different one, now, or whether derby compiles new classes even though the very same query-text was already used before (and there are only roughly 200).
Is the query-class-cache maybe somehow limited? Maybe derby evicts entries from this cache, but the classes are still in the ClassLoader? Or is there maybe some other constraint on the query-class-reusing-code, e.g. a maximum query-text-length?
If there are constraints like cache-size or query-text-length, can this be configured using some system-properties?
P.S.: Sorry for my late reaction. I'm usually very busy with paid work and don't have much time to help my friend fixing his troubles with my hobby project ;-)
> OutOfMemoryError: Compressed class space
> ----------------------------------------
>
> Key: DERBY-7049
> URL: https://issues.apache.org/jira/browse/DERBY-7049
> Project: Derby
> Issue Type: Bug
> Components: SQL
> Affects Versions: 10.13.1.1
> Reporter: Marco
> Priority: Major
> Attachments: StatementLogReadingVTI.java
>
>
> After a few days of working with an embedded Derby database (currently version 10.13.1.1 on Oracle Java 1.8.0_201-b09), the following error occurs:
> *java.lang.OutOfMemoryError: Compressed class space*
> {code:java}
> java.lang.OutOfMemoryError: Compressed class space
> at java.lang.ClassLoader.defineClass1(Native Method) ~[na:1.8.0_201]
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763) ~[na:1.8.0_201]
> at java.lang.ClassLoader.defineClass(ClassLoader.java:642) ~[na:1.8.0_201]
> at org.apache.derby.impl.services.reflect.ReflectLoaderJava2.loadGeneratedClass(Unknown Source) ~[derby-10.13.1.1.jar:na]
> at org.apache.derby.impl.services.reflect.ReflectClassesJava2.loadGeneratedClassFromData(Unknown Source) ~[derby-10.13.1.1.jar:na]
> at org.apache.derby.impl.services.reflect.DatabaseClasses.loadGeneratedClass(Unknown Source) ~[derby-10.13.1.1.jar:na]
> at org.apache.derby.impl.services.bytecode.GClass.getGeneratedClass(Unknown Source) ~[derby-10.13.1.1.jar:na]
> at org.apache.derby.impl.sql.compile.ExpressionClassBuilder.getGeneratedClass(Unknown Source) ~[derby-10.13.1.1.jar:na]
> at org.apache.derby.impl.sql.compile.StatementNode.generate(Unknown Source) ~[derby-10.13.1.1.jar:na]
> at org.apache.derby.impl.sql.GenericStatement.prepMinion(Unknown Source) ~[derby-10.13.1.1.jar:na]
> at org.apache.derby.impl.sql.GenericStatement.prepare(Unknown Source) ~[derby-10.13.1.1.jar:na]
> at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(Unknown Source) ~[derby-10.13.1.1.jar:na]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(Unknown Source) ~[derby-10.13.1.1.jar:na]
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement42.<init>(Unknown Source) ~[derby-10.13.1.1.jar:na]
> at org.apache.derby.jdbc.Driver42.newEmbedPreparedStatement(Unknown Source) ~[derby-10.13.1.1.jar:na]
> at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source) ~[derby-10.13.1.1.jar:na]
> at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source) ~[derby-10.13.1.1.jar:na]
> at org.datanucleus.store.rdbms.datasource.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:259) ~[datanucleus-rdbms-4.0.12.jar:na]{code}
> I tried to solve the problem by periodically shutting down the database, because I read that the generated classes as well as all other allocated resources should be released when the DB is shut-down.
> I thus perform the following code once per roughly 20 minutes:
> {code:java}
> String shutdownConnectionURL = connectionURL + ";shutdown=true";
> try {
> DriverManager.getConnection(shutdownConnectionURL);
> } catch (SQLException e) {
> int errorCode = e.getErrorCode();
> if (DERBY_ERROR_CODE_SHUTDOWN_DATABASE_SUCCESSFULLY != errorCode &&
> DERBY_ERROR_CODE_SHUTDOWN_DATABASE_WAS_NOT_RUNNING != errorCode) {
> throw new RuntimeException(e);
> }
> }
> {code}
> Unfortunately, this has no effect :( The OutOfMemoryError still occurs after about 2 days. Do I assume correctly that the above code _should_ properly shut-down the database? And do I assume correctly that this shut-down should release the generated classes?
> IMHO, it is already a bug in Derby that I need to shut-down the database at all in order to prevent it from piling up generated classes. Shouldn't it already release the generated classes at the end of each transaction? But even if I really have to shut-down the DB, it is certainly a bug that the classes are still kept in "compressed class space" even after the shut-down.
> I searched the release notes and the existing bugs (here in JIRA) and did not find anything related to this {{OutOfMemoryError}}. Hence, I open this bug-report, now.
> This issue was originally reported in [subshare#74|https://github.com/subshare/subshare/issues/74], but it is IMHO clearly a Derby bug.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)