You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "Knut Anders Hatlen (JIRA)" <de...@db.apache.org> on 2006/03/22 19:54:05 UTC
[jira] Created: (DERBY-1142) Metadata calls leak memory
Metadata calls leak memory
--------------------------
Key: DERBY-1142
URL: http://issues.apache.org/jira/browse/DERBY-1142
Project: Derby
Type: Bug
Components: JDBC
Versions: 10.1.2.1, 10.2.0.0
Reporter: Knut Anders Hatlen
Priority: Minor
When calling a DatabaseMetaData method that returns a ResultSet,
memory is leaked. A loop like this (using the embedded driver)
while (true) {
ResultSet rs = dmd.getSchemas();
rs.close();
}
will eventually cause an OutOfMemoryError.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1142) Metadata calls leak memory
Posted by "Daniel John Debrunner (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1142?page=comments#action_12420085 ]
Daniel John Debrunner commented on DERBY-1142:
----------------------------------------------
An addition to the subset of changes of DERBY-827 fixed the issue, increasing the number of iterations from ~10,000 to over a milllion with no error when limited to 10Mb of heap.
One part of the change is to close single use activations in the close method of EmbedResultSet rather than waiting for some other time, need to figure out what that time was. Will attach the patch for this soon.
The second part is to mark EmbedResultSet as single use activations when they are created for meta data.
> Metadata calls leak memory
> --------------------------
>
> Key: DERBY-1142
> URL: http://issues.apache.org/jira/browse/DERBY-1142
> Project: Derby
> Type: Bug
> Components: JDBC
> Versions: 10.1.2.1, 10.2.0.0
> Reporter: Knut Anders Hatlen
> Assignee: Daniel John Debrunner
> Priority: Minor
> Attachments: metadataloop.java
>
> When calling a DatabaseMetaData method that returns a ResultSet,
> memory is leaked. A loop like this (using the embedded driver)
> while (true) {
> ResultSet rs = dmd.getSchemas();
> rs.close();
> }
> will eventually cause an OutOfMemoryError.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1142) Metadata calls leak memory
Posted by "Daniel John Debrunner (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1142?page=comments#action_12419953 ]
Daniel John Debrunner commented on DERBY-1142:
----------------------------------------------
The subset of changes from the draft patch in DERBY-827 (not 872 as I mis-typed previously) do not address this issue. I'm looking at moving those changes into Derby, but no help here :-(
> Metadata calls leak memory
> --------------------------
>
> Key: DERBY-1142
> URL: http://issues.apache.org/jira/browse/DERBY-1142
> Project: Derby
> Type: Bug
> Components: JDBC
> Versions: 10.1.2.1, 10.2.0.0
> Reporter: Knut Anders Hatlen
> Priority: Minor
> Attachments: metadataloop.java
>
> When calling a DatabaseMetaData method that returns a ResultSet,
> memory is leaked. A loop like this (using the embedded driver)
> while (true) {
> ResultSet rs = dmd.getSchemas();
> rs.close();
> }
> will eventually cause an OutOfMemoryError.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1142) Metadata calls leak memory
Posted by "Daniel John Debrunner (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1142?page=comments#action_12371598 ]
Daniel John Debrunner commented on DERBY-1142:
----------------------------------------------
May be unrelated but part of my example patch for DERBY-872 I had a feeling was a more generic issue about activations not being closed correctly in some situations, though I think it may have only been related to single use activations. It's on my list to extract that code and see if it made sense by itself and passed derbyall.
> Metadata calls leak memory
> --------------------------
>
> Key: DERBY-1142
> URL: http://issues.apache.org/jira/browse/DERBY-1142
> Project: Derby
> Type: Bug
> Components: JDBC
> Versions: 10.1.2.1, 10.2.0.0
> Reporter: Knut Anders Hatlen
> Priority: Minor
> Attachments: metadataloop.java
>
> When calling a DatabaseMetaData method that returns a ResultSet,
> memory is leaked. A loop like this (using the embedded driver)
> while (true) {
> ResultSet rs = dmd.getSchemas();
> rs.close();
> }
> will eventually cause an OutOfMemoryError.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1142) Metadata calls leak memory
Posted by "Mayuresh Nirhali (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1142?page=comments#action_12429260 ]
Mayuresh Nirhali commented on DERBY-1142:
-----------------------------------------
If the singleUseActivation is closed in the finalize method of EmdebResultSet then the memory leak is not observed even in this extreme case. While, this may not be a complete fix, but certainly one of fixes.
Please see DERBY-418 for more details.
> Metadata calls leak memory
> --------------------------
>
> Key: DERBY-1142
> URL: http://issues.apache.org/jira/browse/DERBY-1142
> Project: Derby
> Issue Type: Bug
> Components: JDBC
> Affects Versions: 10.1.2.1, 10.2.1.0
> Reporter: Knut Anders Hatlen
> Priority: Minor
> Attachments: 1142_close_single_use_activations_draft.txt, metadataloop.java
>
>
> When calling a DatabaseMetaData method that returns a ResultSet,
> memory is leaked. A loop like this (using the embedded driver)
> while (true) {
> ResultSet rs = dmd.getSchemas();
> rs.close();
> }
> will eventually cause an OutOfMemoryError.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-1142) Metadata calls leak memory
Posted by "Knut Anders Hatlen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1142?page=all ]
Knut Anders Hatlen updated DERBY-1142:
--------------------------------------
Attachment: metadataloop.java
Attached repro. With Derby 10.1.2.1 and Sun JVM 1.4.2,
OutOfMemoryError was thrown after about 80000 calls to
DatabaseMetaData.getCatalogs().
> Metadata calls leak memory
> --------------------------
>
> Key: DERBY-1142
> URL: http://issues.apache.org/jira/browse/DERBY-1142
> Project: Derby
> Type: Bug
> Components: JDBC
> Versions: 10.1.2.1, 10.2.0.0
> Reporter: Knut Anders Hatlen
> Priority: Minor
> Attachments: metadataloop.java
>
> When calling a DatabaseMetaData method that returns a ResultSet,
> memory is leaked. A loop like this (using the embedded driver)
> while (true) {
> ResultSet rs = dmd.getSchemas();
> rs.close();
> }
> will eventually cause an OutOfMemoryError.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Closed: (DERBY-1142) Metadata calls leak memory
Posted by "Knut Anders Hatlen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1142?page=all ]
Knut Anders Hatlen closed DERBY-1142.
-------------------------------------
Fix Version/s: 10.2.1.0
Resolution: Fixed
Verified that the fix for DERBY-418 also fixes the remaining problems for this issue.
> Metadata calls leak memory
> --------------------------
>
> Key: DERBY-1142
> URL: http://issues.apache.org/jira/browse/DERBY-1142
> Project: Derby
> Issue Type: Bug
> Components: JDBC
> Affects Versions: 10.1.2.1, 10.2.1.0
> Reporter: Knut Anders Hatlen
> Priority: Minor
> Fix For: 10.2.1.0
>
> Attachments: 1142_close_single_use_activations_draft.txt, metadataloop.java
>
>
> When calling a DatabaseMetaData method that returns a ResultSet,
> memory is leaked. A loop like this (using the embedded driver)
> while (true) {
> ResultSet rs = dmd.getSchemas();
> rs.close();
> }
> will eventually cause an OutOfMemoryError.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1142) Metadata calls leak memory
Posted by "Daniel John Debrunner (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1142?page=comments#action_12420461 ]
Daniel John Debrunner commented on DERBY-1142:
----------------------------------------------
Two patches applied (420656 & 421015) that stop the memory leak shown in the repro.
A leak still exists if the repro is modified to not close the ResultSet.
This is because the activations marked unused in the language connection context are only cleaned up if a commit or rollback is executed.
In this code no such call is made.
One solution is to look for unused activations when an activation is added to GenericLanguageConnectionContext (addActivation method).
A couple of possible solutions to avoid overhead when no check is needed are:
- Have Activation.markUnused() set an boolean (unsynchronized) flag in the LCC to indicate it has some unused activations
- Only perform the check if the number of activations is beyond some threshold.
Not sure how urgent it is needed to solve this issue, if the application calls commit or rollback in any way (as most applications will at some point)
then memory leak will not be seen. This repro is an extereme edge case.
> Metadata calls leak memory
> --------------------------
>
> Key: DERBY-1142
> URL: http://issues.apache.org/jira/browse/DERBY-1142
> Project: Derby
> Type: Bug
> Components: JDBC
> Versions: 10.1.2.1, 10.2.0.0
> Reporter: Knut Anders Hatlen
> Assignee: Daniel John Debrunner
> Priority: Minor
> Attachments: 1142_close_single_use_activations_draft.txt, metadataloop.java
>
> When calling a DatabaseMetaData method that returns a ResultSet,
> memory is leaked. A loop like this (using the embedded driver)
> while (true) {
> ResultSet rs = dmd.getSchemas();
> rs.close();
> }
> will eventually cause an OutOfMemoryError.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Assigned: (DERBY-1142) Metadata calls leak memory
Posted by "Daniel John Debrunner (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1142?page=all ]
Daniel John Debrunner reassigned DERBY-1142:
--------------------------------------------
Assign To: Daniel John Debrunner
> Metadata calls leak memory
> --------------------------
>
> Key: DERBY-1142
> URL: http://issues.apache.org/jira/browse/DERBY-1142
> Project: Derby
> Type: Bug
> Components: JDBC
> Versions: 10.1.2.1, 10.2.0.0
> Reporter: Knut Anders Hatlen
> Assignee: Daniel John Debrunner
> Priority: Minor
> Attachments: metadataloop.java
>
> When calling a DatabaseMetaData method that returns a ResultSet,
> memory is leaked. A loop like this (using the embedded driver)
> while (true) {
> ResultSet rs = dmd.getSchemas();
> rs.close();
> }
> will eventually cause an OutOfMemoryError.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1142) Metadata calls leak memory
Posted by "Knut Anders Hatlen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1142?page=comments#action_12371579 ]
Knut Anders Hatlen commented on DERBY-1142:
-------------------------------------------
Thanks for looking into this, Anders!
The activation is removed from acts when Activation.close() is
called. Normally, EmbedResultSet.close() calls theResults.finish()
(implemented in BasicNoPutResultSetImpl), which calls
activation.close(). From BasicNoPutResultSetImpl.finishAndRTS():
if (isTopResultSet && activation.isSingleExecution())
activation.close();
For the metadata query, isSingleExecution() returns false, hence
activation.close() isn't called when the result set is closed. It
seems like isSingleExecution() is true only if a Statement is used,
but the metadata query uses a PreparedStatement.
As you noticed, the PreparedStatement is not being closed. However, I
found this in EmbedResultSet.close():
if (forMetaData) {
LanguageConnectionContext lcc = getEmbedConnection().getLanguageConnection();
if (lcc.getActivationCount() > 1) {
// we do not want to commit here as there seems to be other
// statements/resultSets currently opened for this connection.
} else if (owningStmt != null) {
// allow the satement to commit if required.
owningStmt.resultSetClosing(this);
}
} else if (owningStmt != null) {
// allow the satement to commit if required.
owningStmt.resultSetClosing(this);
}
Could we solve the problem by invoking owningStmt.close() when
forMetaData is true and the activation count is not greater than one?
> Metadata calls leak memory
> --------------------------
>
> Key: DERBY-1142
> URL: http://issues.apache.org/jira/browse/DERBY-1142
> Project: Derby
> Type: Bug
> Components: JDBC
> Versions: 10.1.2.1, 10.2.0.0
> Reporter: Knut Anders Hatlen
> Priority: Minor
> Attachments: metadataloop.java
>
> When calling a DatabaseMetaData method that returns a ResultSet,
> memory is leaked. A loop like this (using the embedded driver)
> while (true) {
> ResultSet rs = dmd.getSchemas();
> rs.close();
> }
> will eventually cause an OutOfMemoryError.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Assigned: (DERBY-1142) Metadata calls leak memory
Posted by "Daniel John Debrunner (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1142?page=all ]
Daniel John Debrunner reassigned DERBY-1142:
--------------------------------------------
Assignee: (was: Daniel John Debrunner)
Since this is an extreme edge case now, I'm no longer working on it.
> Metadata calls leak memory
> --------------------------
>
> Key: DERBY-1142
> URL: http://issues.apache.org/jira/browse/DERBY-1142
> Project: Derby
> Issue Type: Bug
> Components: JDBC
> Affects Versions: 10.1.2.1, 10.2.0.0
> Reporter: Knut Anders Hatlen
> Priority: Minor
> Attachments: 1142_close_single_use_activations_draft.txt, metadataloop.java
>
>
> When calling a DatabaseMetaData method that returns a ResultSet,
> memory is leaked. A loop like this (using the embedded driver)
> while (true) {
> ResultSet rs = dmd.getSchemas();
> rs.close();
> }
> will eventually cause an OutOfMemoryError.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1142) Metadata calls leak memory
Posted by "Anders Morken (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1142?page=comments#action_12371561 ]
Anders Morken commented on DERBY-1142:
--------------------------------------
The direct cause of the memory usage is in the org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext class, more specifically its member "acts", which is a Vector of Activation instances. It seems one Activation is added to the vector for every dmd.getSchemas() execution, but they are never removed. I've drilled down to this using NetBeans' memory profiler and debugger and IBMs HeapRoots utility in concert. While I can hunt through heap dumps I can't say that I know enough about Derby internals yet to suggest how to fix this. I've tried explicitly closing the preparedstatement in DatabaseMetaData before returning, but to no effect. I need some time to figure out how all these things (activations, prepared statements, connections and connection contexts) fit together. =)
> Metadata calls leak memory
> --------------------------
>
> Key: DERBY-1142
> URL: http://issues.apache.org/jira/browse/DERBY-1142
> Project: Derby
> Type: Bug
> Components: JDBC
> Versions: 10.2.0.0, 10.1.2.1
> Reporter: Knut Anders Hatlen
> Priority: Minor
> Attachments: metadataloop.java
>
> When calling a DatabaseMetaData method that returns a ResultSet,
> memory is leaked. A loop like this (using the embedded driver)
> while (true) {
> ResultSet rs = dmd.getSchemas();
> rs.close();
> }
> will eventually cause an OutOfMemoryError.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1142) Metadata calls leak memory
Posted by "Knut Anders Hatlen (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1142?page=comments#action_12371700 ]
Knut Anders Hatlen commented on DERBY-1142:
-------------------------------------------
My proposed solution actually fixed the memory leak, but broke
derbyall. These tests failed:
derbyall/derbyall.fail:jdbcapi/odbc_metadata.java
derbyall/derbyall.fail:jdbcapi/bestrowidentifier.sql
derbyall/derbynetmats/derbynetmats.fail:jdbcapi/bestrowidentifier.sql
derbyall/derbynetclientmats/derbynetmats.fail:jdbcapi/bestrowidentifier.sql
> Metadata calls leak memory
> --------------------------
>
> Key: DERBY-1142
> URL: http://issues.apache.org/jira/browse/DERBY-1142
> Project: Derby
> Type: Bug
> Components: JDBC
> Versions: 10.1.2.1, 10.2.0.0
> Reporter: Knut Anders Hatlen
> Priority: Minor
> Attachments: metadataloop.java
>
> When calling a DatabaseMetaData method that returns a ResultSet,
> memory is leaked. A loop like this (using the embedded driver)
> while (true) {
> ResultSet rs = dmd.getSchemas();
> rs.close();
> }
> will eventually cause an OutOfMemoryError.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-1142) Metadata calls leak memory
Posted by "Daniel John Debrunner (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-1142?page=all ]
Daniel John Debrunner updated DERBY-1142:
-----------------------------------------
Attachment: 1142_close_single_use_activations_draft.txt
Draft patch that closes an EmbedResultSet's single use activation when the result set is closed. Previously the activations' close would have been delayed until after the EmbedResultSet was garbage collected and some activilty occured in the connection that caused the list of activations to be scanned. Comment needs some comments before being applied.
> Metadata calls leak memory
> --------------------------
>
> Key: DERBY-1142
> URL: http://issues.apache.org/jira/browse/DERBY-1142
> Project: Derby
> Type: Bug
> Components: JDBC
> Versions: 10.1.2.1, 10.2.0.0
> Reporter: Knut Anders Hatlen
> Assignee: Daniel John Debrunner
> Priority: Minor
> Attachments: 1142_close_single_use_activations_draft.txt, metadataloop.java
>
> When calling a DatabaseMetaData method that returns a ResultSet,
> memory is leaked. A loop like this (using the embedded driver)
> while (true) {
> ResultSet rs = dmd.getSchemas();
> rs.close();
> }
> will eventually cause an OutOfMemoryError.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira