You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "Knut Anders Hatlen (JIRA)" <ji...@apache.org> on 2012/04/27 10:55:58 UTC
[jira] [Created] (DERBY-5730) DataDictionaryImpl leaks references
to itself via SYSFUN_AD
Knut Anders Hatlen created DERBY-5730:
-----------------------------------------
Summary: DataDictionaryImpl leaks references to itself via SYSFUN_AD
Key: DERBY-5730
URL: https://issues.apache.org/jira/browse/DERBY-5730
Project: Derby
Issue Type: Bug
Components: Services
Affects Versions: 10.9.0.0
Reporter: Knut Anders Hatlen
DataDictionaryImpl contains a static field called SYSFUN_AD, which holds an array of AliasDescriptor objects for the functions in the SYSFUN schema. This array is lazily populated as the functions are called, and it is shared between all DataDictionaryImpl instances on the system.
The AliasDescriptors contain references to the DataDictionaryImpl that created them, so SYSFUN_AD may end up referencing indirectly to a number of different DataDictionaryImpl instances, depending on where the respective SYSFUN functions are called the first time. Once an AliasDescriptor has been added to SYSFUN_AD, it stays there until the DataDictionaryImpl class is unloaded (in most cases, until the JVM terminates). This means the array may hold references to DataDictionaryImpl instances that belong to database instances that have been shut down, and that the memory held by those database instances never becomes eligible for garbage collection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (DERBY-5730) DataDictionaryImpl leaks references
to itself via SYSFUN_AD
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269563#comment-13269563 ]
Knut Anders Hatlen commented on DERBY-5730:
-------------------------------------------
> Can you comment on expected performance degredation now that we are
> caching less, I don't know what work the caching is saving. Since we
> are still caching privately I think only expected degredation is on
> first call for booted db, though I know there are some applications
> out there that boot very often.
That's right, degradation should only be expected on the first call of
a SYSFUN function after the second db boot.
I'm not sure what degradation to expect myself, so I've written a
small test program that executes the following statement repeatedly:
values atan(tan(asin(sin(acos(cos(pi()))))))
The statement calls seven different SYSFUN functions, and I would
expect it to be very close to the worst case since VALUES statements
are among the cheapest ones to compile.
I also turned off the statement cache, so that each execution of the
statement would cause a compilation (it is during the bind phase of
the compilation the SYSFUN_AD array is accessed).
Using the test program, I compared clean trunk to a variant that was
modified to skip the caching altogether. That is, I commented out the
following line in DataDictionaryImpl.getRoutineList():
DataDictionaryImpl.SYSFUN_AD[f] = ad;
The variant that didn't cache the alias descriptor, executed the
statement 2.5% slower on average in my environment.
Since this degradation will only be observed when SYSFUN functions are
compiled for the first time in newly booted databases, I'd say it's
negligible.
One might perhaps even say that the gains of the alias descriptor
cache are so small that they don't justify the extra memory and code
complexity, but I don't think we want to go there in this JIRA.
> If there were no memory consequences does the existing
> implementation have any correctness issues?
Not that I have been able to find. The data dictionary instance is
only used in the following situations, as far as I can tell:
- At initialization time of the AliasDescriptor to find the UUID of
the SYSFUN schema. This doesn't cause any problems, since the data
dictionary instance is still alive at that time, and the UUID of the
SYSFUN schema is the same in all databases.
- When an alias is dropped, but the functions in SYSFUN can't be
dropped.
- During invalidation, but SYSFUN functions can't be invalidated.
> DataDictionaryImpl leaks references to itself via SYSFUN_AD
> -----------------------------------------------------------
>
> Key: DERBY-5730
> URL: https://issues.apache.org/jira/browse/DERBY-5730
> Project: Derby
> Issue Type: Bug
> Components: Services
> Affects Versions: 10.9.0.0
> Reporter: Knut Anders Hatlen
> Assignee: Knut Anders Hatlen
> Attachments: Sysfun.java, d5730-1a.diff
>
>
> DataDictionaryImpl contains a static field called SYSFUN_AD, which holds an array of AliasDescriptor objects for the functions in the SYSFUN schema. This array is lazily populated as the functions are called, and it is shared between all DataDictionaryImpl instances on the system.
> The AliasDescriptors contain references to the DataDictionaryImpl that created them, so SYSFUN_AD may end up referencing indirectly to a number of different DataDictionaryImpl instances, depending on where the respective SYSFUN functions are called the first time. Once an AliasDescriptor has been added to SYSFUN_AD, it stays there until the DataDictionaryImpl class is unloaded (in most cases, until the JVM terminates). This means the array may hold references to DataDictionaryImpl instances that belong to database instances that have been shut down, and that the memory held by those database instances never becomes eligible for garbage collection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (DERBY-5730) DataDictionaryImpl leaks references
to itself via SYSFUN_AD
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Knut Anders Hatlen updated DERBY-5730:
--------------------------------------
Attachment: d5730-1a.diff
The attached patch makes each DataDictionaryImpl instance have its own copy of the AliasDescriptors for the SYSFUN functions. This avoids keeping references to old dictionary instances in a shared static array, and allows gc of the instances before the DataDictionaryImpl class itself is eligible for gc.
The patch also adds a new test to the memory suite. The test reports an OOME without the fix.
All the regression tests ran cleanly with the patch.
> DataDictionaryImpl leaks references to itself via SYSFUN_AD
> -----------------------------------------------------------
>
> Key: DERBY-5730
> URL: https://issues.apache.org/jira/browse/DERBY-5730
> Project: Derby
> Issue Type: Bug
> Components: Services
> Affects Versions: 10.9.0.0
> Reporter: Knut Anders Hatlen
> Assignee: Knut Anders Hatlen
> Attachments: Sysfun.java, d5730-1a.diff
>
>
> DataDictionaryImpl contains a static field called SYSFUN_AD, which holds an array of AliasDescriptor objects for the functions in the SYSFUN schema. This array is lazily populated as the functions are called, and it is shared between all DataDictionaryImpl instances on the system.
> The AliasDescriptors contain references to the DataDictionaryImpl that created them, so SYSFUN_AD may end up referencing indirectly to a number of different DataDictionaryImpl instances, depending on where the respective SYSFUN functions are called the first time. Once an AliasDescriptor has been added to SYSFUN_AD, it stays there until the DataDictionaryImpl class is unloaded (in most cases, until the JVM terminates). This means the array may hold references to DataDictionaryImpl instances that belong to database instances that have been shut down, and that the memory held by those database instances never becomes eligible for garbage collection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (DERBY-5730) DataDictionaryImpl leaks references
to itself via SYSFUN_AD
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Knut Anders Hatlen updated DERBY-5730:
--------------------------------------
Attachment: Sysfun.java
Attaching a repro that exposes the leak.
The repro repeatedly boot a database, performs some db operations, and shuts down the database. One of the db operations it performs, is preparing a statement that calls one of the SYSFUN functions, and it is a different function in each iteration.
When running the repro with -Xmx20M, I see an OutOfMemoryError. If I comment out the preparing of the function call, the repro completes successfully with -Xmx5M.
> DataDictionaryImpl leaks references to itself via SYSFUN_AD
> -----------------------------------------------------------
>
> Key: DERBY-5730
> URL: https://issues.apache.org/jira/browse/DERBY-5730
> Project: Derby
> Issue Type: Bug
> Components: Services
> Affects Versions: 10.9.0.0
> Reporter: Knut Anders Hatlen
> Attachments: Sysfun.java
>
>
> DataDictionaryImpl contains a static field called SYSFUN_AD, which holds an array of AliasDescriptor objects for the functions in the SYSFUN schema. This array is lazily populated as the functions are called, and it is shared between all DataDictionaryImpl instances on the system.
> The AliasDescriptors contain references to the DataDictionaryImpl that created them, so SYSFUN_AD may end up referencing indirectly to a number of different DataDictionaryImpl instances, depending on where the respective SYSFUN functions are called the first time. Once an AliasDescriptor has been added to SYSFUN_AD, it stays there until the DataDictionaryImpl class is unloaded (in most cases, until the JVM terminates). This means the array may hold references to DataDictionaryImpl instances that belong to database instances that have been shut down, and that the memory held by those database instances never becomes eligible for garbage collection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (DERBY-5730) DataDictionaryImpl leaks references
to itself via SYSFUN_AD
Posted by "Mike Matrigali (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268513#comment-13268513 ]
Mike Matrigali commented on DERBY-5730:
---------------------------------------
the patch looks good to me.
Can you comment
on expected performance degredation now that we are caching less, I don't know what work the caching is saving. Since we are still caching privately I think only expected degredation is on first call for booted db, though I know there are some applications out there that boot very often. If there were no memory consequences does the existing
implementation have any correctness issues?
> DataDictionaryImpl leaks references to itself via SYSFUN_AD
> -----------------------------------------------------------
>
> Key: DERBY-5730
> URL: https://issues.apache.org/jira/browse/DERBY-5730
> Project: Derby
> Issue Type: Bug
> Components: Services
> Affects Versions: 10.9.0.0
> Reporter: Knut Anders Hatlen
> Assignee: Knut Anders Hatlen
> Attachments: Sysfun.java, d5730-1a.diff
>
>
> DataDictionaryImpl contains a static field called SYSFUN_AD, which holds an array of AliasDescriptor objects for the functions in the SYSFUN schema. This array is lazily populated as the functions are called, and it is shared between all DataDictionaryImpl instances on the system.
> The AliasDescriptors contain references to the DataDictionaryImpl that created them, so SYSFUN_AD may end up referencing indirectly to a number of different DataDictionaryImpl instances, depending on where the respective SYSFUN functions are called the first time. Once an AliasDescriptor has been added to SYSFUN_AD, it stays there until the DataDictionaryImpl class is unloaded (in most cases, until the JVM terminates). This means the array may hold references to DataDictionaryImpl instances that belong to database instances that have been shut down, and that the memory held by those database instances never becomes eligible for garbage collection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (DERBY-5730) DataDictionaryImpl leaks references
to itself via SYSFUN_AD
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Knut Anders Hatlen updated DERBY-5730:
--------------------------------------
Bug behavior facts: Crash
Marking the bug as a crash, since it's a memory leak that could potentially cause OOME.
> DataDictionaryImpl leaks references to itself via SYSFUN_AD
> -----------------------------------------------------------
>
> Key: DERBY-5730
> URL: https://issues.apache.org/jira/browse/DERBY-5730
> Project: Derby
> Issue Type: Bug
> Components: Services
> Affects Versions: 10.9.0.0
> Reporter: Knut Anders Hatlen
> Attachments: Sysfun.java
>
>
> DataDictionaryImpl contains a static field called SYSFUN_AD, which holds an array of AliasDescriptor objects for the functions in the SYSFUN schema. This array is lazily populated as the functions are called, and it is shared between all DataDictionaryImpl instances on the system.
> The AliasDescriptors contain references to the DataDictionaryImpl that created them, so SYSFUN_AD may end up referencing indirectly to a number of different DataDictionaryImpl instances, depending on where the respective SYSFUN functions are called the first time. Once an AliasDescriptor has been added to SYSFUN_AD, it stays there until the DataDictionaryImpl class is unloaded (in most cases, until the JVM terminates). This means the array may hold references to DataDictionaryImpl instances that belong to database instances that have been shut down, and that the memory held by those database instances never becomes eligible for garbage collection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (DERBY-5730) DataDictionaryImpl leaks references
to itself via SYSFUN_AD
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Knut Anders Hatlen updated DERBY-5730:
--------------------------------------
Issue & fix info: Patch Available
> DataDictionaryImpl leaks references to itself via SYSFUN_AD
> -----------------------------------------------------------
>
> Key: DERBY-5730
> URL: https://issues.apache.org/jira/browse/DERBY-5730
> Project: Derby
> Issue Type: Bug
> Components: Services
> Affects Versions: 10.9.0.0
> Reporter: Knut Anders Hatlen
> Assignee: Knut Anders Hatlen
> Attachments: Sysfun.java, d5730-1a.diff
>
>
> DataDictionaryImpl contains a static field called SYSFUN_AD, which holds an array of AliasDescriptor objects for the functions in the SYSFUN schema. This array is lazily populated as the functions are called, and it is shared between all DataDictionaryImpl instances on the system.
> The AliasDescriptors contain references to the DataDictionaryImpl that created them, so SYSFUN_AD may end up referencing indirectly to a number of different DataDictionaryImpl instances, depending on where the respective SYSFUN functions are called the first time. Once an AliasDescriptor has been added to SYSFUN_AD, it stays there until the DataDictionaryImpl class is unloaded (in most cases, until the JVM terminates). This means the array may hold references to DataDictionaryImpl instances that belong to database instances that have been shut down, and that the memory held by those database instances never becomes eligible for garbage collection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (DERBY-5730) DataDictionaryImpl leaks references
to itself via SYSFUN_AD
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Knut Anders Hatlen reassigned DERBY-5730:
-----------------------------------------
Assignee: Knut Anders Hatlen
> DataDictionaryImpl leaks references to itself via SYSFUN_AD
> -----------------------------------------------------------
>
> Key: DERBY-5730
> URL: https://issues.apache.org/jira/browse/DERBY-5730
> Project: Derby
> Issue Type: Bug
> Components: Services
> Affects Versions: 10.9.0.0
> Reporter: Knut Anders Hatlen
> Assignee: Knut Anders Hatlen
> Attachments: Sysfun.java
>
>
> DataDictionaryImpl contains a static field called SYSFUN_AD, which holds an array of AliasDescriptor objects for the functions in the SYSFUN schema. This array is lazily populated as the functions are called, and it is shared between all DataDictionaryImpl instances on the system.
> The AliasDescriptors contain references to the DataDictionaryImpl that created them, so SYSFUN_AD may end up referencing indirectly to a number of different DataDictionaryImpl instances, depending on where the respective SYSFUN functions are called the first time. Once an AliasDescriptor has been added to SYSFUN_AD, it stays there until the DataDictionaryImpl class is unloaded (in most cases, until the JVM terminates). This means the array may hold references to DataDictionaryImpl instances that belong to database instances that have been shut down, and that the memory held by those database instances never becomes eligible for garbage collection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (DERBY-5730) DataDictionaryImpl leaks references
to itself via SYSFUN_AD
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Knut Anders Hatlen resolved DERBY-5730.
---------------------------------------
Resolution: Fixed
Fix Version/s: 10.9.0.0
Issue & fix info: (was: Patch Available)
Committed revision 1335423.
> DataDictionaryImpl leaks references to itself via SYSFUN_AD
> -----------------------------------------------------------
>
> Key: DERBY-5730
> URL: https://issues.apache.org/jira/browse/DERBY-5730
> Project: Derby
> Issue Type: Bug
> Components: Services
> Affects Versions: 10.9.0.0
> Reporter: Knut Anders Hatlen
> Assignee: Knut Anders Hatlen
> Fix For: 10.9.0.0
>
> Attachments: Sysfun.java, SysfunPerftest.java, d5730-1a.diff
>
>
> DataDictionaryImpl contains a static field called SYSFUN_AD, which holds an array of AliasDescriptor objects for the functions in the SYSFUN schema. This array is lazily populated as the functions are called, and it is shared between all DataDictionaryImpl instances on the system.
> The AliasDescriptors contain references to the DataDictionaryImpl that created them, so SYSFUN_AD may end up referencing indirectly to a number of different DataDictionaryImpl instances, depending on where the respective SYSFUN functions are called the first time. Once an AliasDescriptor has been added to SYSFUN_AD, it stays there until the DataDictionaryImpl class is unloaded (in most cases, until the JVM terminates). This means the array may hold references to DataDictionaryImpl instances that belong to database instances that have been shut down, and that the memory held by those database instances never becomes eligible for garbage collection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (DERBY-5730) DataDictionaryImpl leaks references
to itself via SYSFUN_AD
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Knut Anders Hatlen updated DERBY-5730:
--------------------------------------
Attachment: SysfunPerftest.java
Attaching the program I used to test the performance impact.
> DataDictionaryImpl leaks references to itself via SYSFUN_AD
> -----------------------------------------------------------
>
> Key: DERBY-5730
> URL: https://issues.apache.org/jira/browse/DERBY-5730
> Project: Derby
> Issue Type: Bug
> Components: Services
> Affects Versions: 10.9.0.0
> Reporter: Knut Anders Hatlen
> Assignee: Knut Anders Hatlen
> Attachments: Sysfun.java, SysfunPerftest.java, d5730-1a.diff
>
>
> DataDictionaryImpl contains a static field called SYSFUN_AD, which holds an array of AliasDescriptor objects for the functions in the SYSFUN schema. This array is lazily populated as the functions are called, and it is shared between all DataDictionaryImpl instances on the system.
> The AliasDescriptors contain references to the DataDictionaryImpl that created them, so SYSFUN_AD may end up referencing indirectly to a number of different DataDictionaryImpl instances, depending on where the respective SYSFUN functions are called the first time. Once an AliasDescriptor has been added to SYSFUN_AD, it stays there until the DataDictionaryImpl class is unloaded (in most cases, until the JVM terminates). This means the array may hold references to DataDictionaryImpl instances that belong to database instances that have been shut down, and that the memory held by those database instances never becomes eligible for garbage collection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira