You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "Bergquist, Brett" <BB...@canoga.com> on 2011/08/16 14:56:43 UTC
Question on unloading in an embedded environment
I am writing a database copy utility that is built upon the dblook utility and using the SYSCS_UTIL.SYSCS_EXPORT_TABLE and SYSCS_UTIL.SYSCS_IMPORT_TABLE procedures and I have this mostly working. When running this on a large database (110Gb) I am getting an OutOfMemory exception. I have the utility setup to use up to 8192Mb of memory so somewhere there is a problem. I am running this again with the -XX:+HeapDumpOnOutOfMemoryError switch to see if I can get a picture on where it is leaking but it will take about 6 hours to get to this point.
In the mean time, assuming the error is not in my utility, I would like to potentially shutdown the database and restart it to try to work around the OOME. I am running the database engine in embedded mode and see the documentation to shutdown the database, but I also see the following in the Derby 10.8 Reference manual page 293:
"With the embedded driver, if your application shuts down Derby or calls the
DriverManager.unload method, and you then want to reload the driver, call the
Class.forName().newInstance() method to do so:"
When I look at the Javadoc for the DriverManager class, I do not find an "unload" method. So what is this documentation referring to?
Thanks
Brett
RE: Question on unloading in an embedded environment
Posted by "Bergquist, Brett" <BB...@canoga.com>.
Here is the whole application. It is still in a rough format but on a smaller database it does indeed work correctly. The only thing that it cannot do when copying the database is handle GENERATED ALWAYS columns. For these, it turns them into GENERATED BY DEFAULT and as the last step determines the next value to use and issues an ALTER TABLE statement to reset the next generated value.
It would be useful to somehow allow the SYSCS_IMPORT_TABLE to bypass the constraint that a GENERATED ALWAYS column cannot be inserted with a value or have the ability to change the column type back to GENERATED ALWAYS after the determining the correct next value.
-----Original Message-----
From: Bryan Pendleton [mailto:bpendleton.derby@gmail.com]
Sent: Wednesday, August 17, 2011 9:40 AM
To: derby-dev@db.apache.org
Subject: Re: Question on unloading in an embedded environment
> Instance Counts for All Classes (excluding platform)
>
> 38006784 instances of class org.apache.derby.impl.store.access.sort.Node
One possibility is that the sorter is confused about how much memory
is available.
The sorter is very clever, and tries to figure out whether it can perform
the sort in-memory, or whether it has to switch to an external (disk-based)
sort, using temporary files to hold the partially-sorted data.
If the sorter is confused about how much memory is available (i.e., thinks
there is more memory available than there actually is), then it might
drive the system out of memory.
This would certainly be a bug, but not a leak exactly, rather a flaw
in the internal-vs-external sort analysis code.
thanks,
bryan
P.S. I'm still wondering if your memory issues are actually external to
Derby; that is, if you've configured your JVM with memory sizes that exceed
the memory available in the underlying operating system. That would cause
Derby to attempt to allocate data structures that the JVM was willing to
allow, but which the underlying OS refused to provide.
RE: Question on unloading in an embedded environment
Posted by "Bergquist, Brett" <BB...@canoga.com>.
Open https://issues.apache.org/jira/browse/DERBY-5387 and attached the Netbeans project for the utility along with two reports from Eclipse MemoryAnalyzer to for the suspected leak.
-----Original Message-----
From: Rick Hillegas [mailto:rick.hillegas@oracle.com]
Sent: Wednesday, August 17, 2011 2:57 PM
To: derby-dev@db.apache.org
Subject: Re: Question on unloading in an embedded environment
On 8/17/11 11:03 AM, Bergquist, Brett wrote:
> I have a report generated by MemoryAnalyzer (Eclipse) tool. The report is stored as a zip file that contains the HTML and images. Can I attach this to an email?
Hi Brett,
If it's big, I would recommend attaching it to a JIRA issue.
Thanks,
-Rick
> I have attached a JPG of some of the report here.
>
>
>
> -----Original Message-----
> From: Bergquist, Brett [mailto:BBergquist@canoga.com]
> Sent: Wednesday, August 17, 2011 11:16 AM
> To: derby-dev@db.apache.org
> Subject: RE: Question on unloading in an embedded environment
>
> So this is running on Solaris 10 Java 1.6.0_22 with "-d64 -Xmx8129m -XX:+HeapDumpOnOutOfMemory" Unfortunately the heap dump is so large it take 32Gb for jhat to load it and then forever for look at anything. I am running the app again with a smaller heap to get the problem to occur quicker and with a smaller heap dump to be able to get a allocation traceback.
>
> -----Original Message-----
> From: Bergquist, Brett [mailto:BBergquist@canoga.com]
> Sent: Wednesday, August 17, 2011 10:49 AM
> To: derby-dev@db.apache.org
> Subject: RE: Question on unloading in an embedded environment
>
> This was run with 8gb of heap available. The system is a Oracle M3000 with 32Gb of real ram. It did not run out of swap while running, so it hit the heap max.
>
> Why would the sorter be called when doing a SYSCS_IMPORT_TABLE? That is all this application is doing. It exports a table from one database with SYSCS_EXPORT_TABLE and imports into the other database with SYSCS_IMPORT_TABLE. No queries or other statements are being used.
>
> -----Original Message-----
> From: Bryan Pendleton [mailto:bpendleton.derby@gmail.com]
> Sent: Wednesday, August 17, 2011 9:40 AM
> To: derby-dev@db.apache.org
> Subject: Re: Question on unloading in an embedded environment
>
>> Instance Counts for All Classes (excluding platform)
>>
>> 38006784 instances of class org.apache.derby.impl.store.access.sort.Node
> One possibility is that the sorter is confused about how much memory
> is available.
>
> The sorter is very clever, and tries to figure out whether it can perform
> the sort in-memory, or whether it has to switch to an external (disk-based)
> sort, using temporary files to hold the partially-sorted data.
>
> If the sorter is confused about how much memory is available (i.e., thinks
> there is more memory available than there actually is), then it might
> drive the system out of memory.
>
> This would certainly be a bug, but not a leak exactly, rather a flaw
> in the internal-vs-external sort analysis code.
>
> thanks,
>
> bryan
>
> P.S. I'm still wondering if your memory issues are actually external to
> Derby; that is, if you've configured your JVM with memory sizes that exceed
> the memory available in the underlying operating system. That would cause
> Derby to attempt to allocate data structures that the JVM was willing to
> allow, but which the underlying OS refused to provide.
>
>
>
>
>
>
Re: Question on unloading in an embedded environment
Posted by Rick Hillegas <ri...@oracle.com>.
On 8/17/11 11:03 AM, Bergquist, Brett wrote:
> I have a report generated by MemoryAnalyzer (Eclipse) tool. The report is stored as a zip file that contains the HTML and images. Can I attach this to an email?
Hi Brett,
If it's big, I would recommend attaching it to a JIRA issue.
Thanks,
-Rick
> I have attached a JPG of some of the report here.
>
>
>
> -----Original Message-----
> From: Bergquist, Brett [mailto:BBergquist@canoga.com]
> Sent: Wednesday, August 17, 2011 11:16 AM
> To: derby-dev@db.apache.org
> Subject: RE: Question on unloading in an embedded environment
>
> So this is running on Solaris 10 Java 1.6.0_22 with "-d64 -Xmx8129m -XX:+HeapDumpOnOutOfMemory" Unfortunately the heap dump is so large it take 32Gb for jhat to load it and then forever for look at anything. I am running the app again with a smaller heap to get the problem to occur quicker and with a smaller heap dump to be able to get a allocation traceback.
>
> -----Original Message-----
> From: Bergquist, Brett [mailto:BBergquist@canoga.com]
> Sent: Wednesday, August 17, 2011 10:49 AM
> To: derby-dev@db.apache.org
> Subject: RE: Question on unloading in an embedded environment
>
> This was run with 8gb of heap available. The system is a Oracle M3000 with 32Gb of real ram. It did not run out of swap while running, so it hit the heap max.
>
> Why would the sorter be called when doing a SYSCS_IMPORT_TABLE? That is all this application is doing. It exports a table from one database with SYSCS_EXPORT_TABLE and imports into the other database with SYSCS_IMPORT_TABLE. No queries or other statements are being used.
>
> -----Original Message-----
> From: Bryan Pendleton [mailto:bpendleton.derby@gmail.com]
> Sent: Wednesday, August 17, 2011 9:40 AM
> To: derby-dev@db.apache.org
> Subject: Re: Question on unloading in an embedded environment
>
>> Instance Counts for All Classes (excluding platform)
>>
>> 38006784 instances of class org.apache.derby.impl.store.access.sort.Node
> One possibility is that the sorter is confused about how much memory
> is available.
>
> The sorter is very clever, and tries to figure out whether it can perform
> the sort in-memory, or whether it has to switch to an external (disk-based)
> sort, using temporary files to hold the partially-sorted data.
>
> If the sorter is confused about how much memory is available (i.e., thinks
> there is more memory available than there actually is), then it might
> drive the system out of memory.
>
> This would certainly be a bug, but not a leak exactly, rather a flaw
> in the internal-vs-external sort analysis code.
>
> thanks,
>
> bryan
>
> P.S. I'm still wondering if your memory issues are actually external to
> Derby; that is, if you've configured your JVM with memory sizes that exceed
> the memory available in the underlying operating system. That would cause
> Derby to attempt to allocate data structures that the JVM was willing to
> allow, but which the underlying OS refused to provide.
>
>
>
>
>
>
RE: Question on unloading in an embedded environment
Posted by "Bergquist, Brett" <BB...@canoga.com>.
I have a report generated by MemoryAnalyzer (Eclipse) tool. The report is stored as a zip file that contains the HTML and images. Can I attach this to an email?
I have attached a JPG of some of the report here.
-----Original Message-----
From: Bergquist, Brett [mailto:BBergquist@canoga.com]
Sent: Wednesday, August 17, 2011 11:16 AM
To: derby-dev@db.apache.org
Subject: RE: Question on unloading in an embedded environment
So this is running on Solaris 10 Java 1.6.0_22 with "-d64 -Xmx8129m -XX:+HeapDumpOnOutOfMemory" Unfortunately the heap dump is so large it take 32Gb for jhat to load it and then forever for look at anything. I am running the app again with a smaller heap to get the problem to occur quicker and with a smaller heap dump to be able to get a allocation traceback.
-----Original Message-----
From: Bergquist, Brett [mailto:BBergquist@canoga.com]
Sent: Wednesday, August 17, 2011 10:49 AM
To: derby-dev@db.apache.org
Subject: RE: Question on unloading in an embedded environment
This was run with 8gb of heap available. The system is a Oracle M3000 with 32Gb of real ram. It did not run out of swap while running, so it hit the heap max.
Why would the sorter be called when doing a SYSCS_IMPORT_TABLE? That is all this application is doing. It exports a table from one database with SYSCS_EXPORT_TABLE and imports into the other database with SYSCS_IMPORT_TABLE. No queries or other statements are being used.
-----Original Message-----
From: Bryan Pendleton [mailto:bpendleton.derby@gmail.com]
Sent: Wednesday, August 17, 2011 9:40 AM
To: derby-dev@db.apache.org
Subject: Re: Question on unloading in an embedded environment
> Instance Counts for All Classes (excluding platform)
>
> 38006784 instances of class org.apache.derby.impl.store.access.sort.Node
One possibility is that the sorter is confused about how much memory
is available.
The sorter is very clever, and tries to figure out whether it can perform
the sort in-memory, or whether it has to switch to an external (disk-based)
sort, using temporary files to hold the partially-sorted data.
If the sorter is confused about how much memory is available (i.e., thinks
there is more memory available than there actually is), then it might
drive the system out of memory.
This would certainly be a bug, but not a leak exactly, rather a flaw
in the internal-vs-external sort analysis code.
thanks,
bryan
P.S. I'm still wondering if your memory issues are actually external to
Derby; that is, if you've configured your JVM with memory sizes that exceed
the memory available in the underlying operating system. That would cause
Derby to attempt to allocate data structures that the JVM was willing to
allow, but which the underlying OS refused to provide.
RE: Question on unloading in an embedded environment
Posted by "Bergquist, Brett" <BB...@canoga.com>.
So this is running on Solaris 10 Java 1.6.0_22 with "-d64 -Xmx8129m -XX:+HeapDumpOnOutOfMemory" Unfortunately the heap dump is so large it take 32Gb for jhat to load it and then forever for look at anything. I am running the app again with a smaller heap to get the problem to occur quicker and with a smaller heap dump to be able to get a allocation traceback.
-----Original Message-----
From: Bergquist, Brett [mailto:BBergquist@canoga.com]
Sent: Wednesday, August 17, 2011 10:49 AM
To: derby-dev@db.apache.org
Subject: RE: Question on unloading in an embedded environment
This was run with 8gb of heap available. The system is a Oracle M3000 with 32Gb of real ram. It did not run out of swap while running, so it hit the heap max.
Why would the sorter be called when doing a SYSCS_IMPORT_TABLE? That is all this application is doing. It exports a table from one database with SYSCS_EXPORT_TABLE and imports into the other database with SYSCS_IMPORT_TABLE. No queries or other statements are being used.
-----Original Message-----
From: Bryan Pendleton [mailto:bpendleton.derby@gmail.com]
Sent: Wednesday, August 17, 2011 9:40 AM
To: derby-dev@db.apache.org
Subject: Re: Question on unloading in an embedded environment
> Instance Counts for All Classes (excluding platform)
>
> 38006784 instances of class org.apache.derby.impl.store.access.sort.Node
One possibility is that the sorter is confused about how much memory
is available.
The sorter is very clever, and tries to figure out whether it can perform
the sort in-memory, or whether it has to switch to an external (disk-based)
sort, using temporary files to hold the partially-sorted data.
If the sorter is confused about how much memory is available (i.e., thinks
there is more memory available than there actually is), then it might
drive the system out of memory.
This would certainly be a bug, but not a leak exactly, rather a flaw
in the internal-vs-external sort analysis code.
thanks,
bryan
P.S. I'm still wondering if your memory issues are actually external to
Derby; that is, if you've configured your JVM with memory sizes that exceed
the memory available in the underlying operating system. That would cause
Derby to attempt to allocate data structures that the JVM was willing to
allow, but which the underlying OS refused to provide.
RE: Question on unloading in an embedded environment
Posted by "Bergquist, Brett" <BB...@canoga.com>.
This was run with 8gb of heap available. The system is a Oracle M3000 with 32Gb of real ram. It did not run out of swap while running, so it hit the heap max.
Why would the sorter be called when doing a SYSCS_IMPORT_TABLE? That is all this application is doing. It exports a table from one database with SYSCS_EXPORT_TABLE and imports into the other database with SYSCS_IMPORT_TABLE. No queries or other statements are being used.
-----Original Message-----
From: Bryan Pendleton [mailto:bpendleton.derby@gmail.com]
Sent: Wednesday, August 17, 2011 9:40 AM
To: derby-dev@db.apache.org
Subject: Re: Question on unloading in an embedded environment
> Instance Counts for All Classes (excluding platform)
>
> 38006784 instances of class org.apache.derby.impl.store.access.sort.Node
One possibility is that the sorter is confused about how much memory
is available.
The sorter is very clever, and tries to figure out whether it can perform
the sort in-memory, or whether it has to switch to an external (disk-based)
sort, using temporary files to hold the partially-sorted data.
If the sorter is confused about how much memory is available (i.e., thinks
there is more memory available than there actually is), then it might
drive the system out of memory.
This would certainly be a bug, but not a leak exactly, rather a flaw
in the internal-vs-external sort analysis code.
thanks,
bryan
P.S. I'm still wondering if your memory issues are actually external to
Derby; that is, if you've configured your JVM with memory sizes that exceed
the memory available in the underlying operating system. That would cause
Derby to attempt to allocate data structures that the JVM was willing to
allow, but which the underlying OS refused to provide.
Re: Question on unloading in an embedded environment
Posted by Bryan Pendleton <bp...@gmail.com>.
> Instance Counts for All Classes (excluding platform)
>
> 38006784 instances of class org.apache.derby.impl.store.access.sort.Node
One possibility is that the sorter is confused about how much memory
is available.
The sorter is very clever, and tries to figure out whether it can perform
the sort in-memory, or whether it has to switch to an external (disk-based)
sort, using temporary files to hold the partially-sorted data.
If the sorter is confused about how much memory is available (i.e., thinks
there is more memory available than there actually is), then it might
drive the system out of memory.
This would certainly be a bug, but not a leak exactly, rather a flaw
in the internal-vs-external sort analysis code.
thanks,
bryan
P.S. I'm still wondering if your memory issues are actually external to
Derby; that is, if you've configured your JVM with memory sizes that exceed
the memory available in the underlying operating system. That would cause
Derby to attempt to allocate data structures that the JVM was willing to
allow, but which the underlying OS refused to provide.
RE: Question on unloading in an embedded environment
Posted by "Bergquist, Brett" <BB...@canoga.com>.
Here is what jhat reports for the is OOME:
Instance Counts for All Classes (excluding platform)
38006784 instances of class org.apache.derby.impl.store.access.sort.Node
26547422 instances of class org.apache.derby.impl.store.raw.data.RecordId
26524427 instances of class [Lorg.apache.derby.iapi.types.DataValueDescriptor;
26524237 instances of class org.apache.derby.iapi.types.SQLTimestamp
26524235 instances of class org.apache.derby.impl.store.access.heap.HeapRowLocation
3323190 instances of class org.apache.derby.iapi.store.raw.PageKey
31521 instances of class org.apache.derby.impl.store.raw.data.StoredRecordHeader
...
I am trying to get a stack traceback of where some of these are allocated, but there is definitely a leak in Derby, probably in the implementation of the SYSCS_IMPORT_TABLE somewhere.
-----Original Message-----
From: Rick Hillegas [mailto:rick.hillegas@oracle.com]
Sent: Tuesday, August 16, 2011 9:20 AM
To: derby-dev@db.apache.org
Subject: Re: Question on unloading in an embedded environment
On 8/16/11 5:56 AM, Bergquist, Brett wrote:
> I am writing a database copy utility that is built upon the dblook utility and using the SYSCS_UTIL.SYSCS_EXPORT_TABLE and SYSCS_UTIL.SYSCS_IMPORT_TABLE procedures and I have this mostly working. When running this on a large database (110Gb) I am getting an OutOfMemory exception. I have the utility setup to use up to 8192Mb of memory so somewhere there is a problem. I am running this again with the -XX:+HeapDumpOnOutOfMemoryError switch to see if I can get a picture on where it is leaking but it will take about 6 hours to get to this point.
>
> In the mean time, assuming the error is not in my utility, I would like to potentially shutdown the database and restart it to try to work around the OOME. I am running the database engine in embedded mode and see the documentation to shutdown the database, but I also see the following in the Derby 10.8 Reference manual page 293:
>
> "With the embedded driver, if your application shuts down Derby or calls the
> DriverManager.unload method, and you then want to reload the driver, call the
> Class.forName().newInstance() method to do so:"
>
> When I look at the Javadoc for the DriverManager class, I do not find an "unload" method. So what is this documentation referring to?
Hi Brett,
That looks like an error in the documentation. I believe it is supposed
to refer to DriverManager.deregisterDriver(). I have logged
https://issues.apache.org/jira/browse/DERBY-5384 to track this.
Thanks for finding this bug,
-Rick
> Thanks
>
> Brett
>
>
>
>
>
Re: Question on unloading in an embedded environment
Posted by Rick Hillegas <ri...@oracle.com>.
On 8/16/11 5:56 AM, Bergquist, Brett wrote:
> I am writing a database copy utility that is built upon the dblook utility and using the SYSCS_UTIL.SYSCS_EXPORT_TABLE and SYSCS_UTIL.SYSCS_IMPORT_TABLE procedures and I have this mostly working. When running this on a large database (110Gb) I am getting an OutOfMemory exception. I have the utility setup to use up to 8192Mb of memory so somewhere there is a problem. I am running this again with the -XX:+HeapDumpOnOutOfMemoryError switch to see if I can get a picture on where it is leaking but it will take about 6 hours to get to this point.
>
> In the mean time, assuming the error is not in my utility, I would like to potentially shutdown the database and restart it to try to work around the OOME. I am running the database engine in embedded mode and see the documentation to shutdown the database, but I also see the following in the Derby 10.8 Reference manual page 293:
>
> "With the embedded driver, if your application shuts down Derby or calls the
> DriverManager.unload method, and you then want to reload the driver, call the
> Class.forName().newInstance() method to do so:"
>
> When I look at the Javadoc for the DriverManager class, I do not find an "unload" method. So what is this documentation referring to?
Hi Brett,
That looks like an error in the documentation. I believe it is supposed
to refer to DriverManager.deregisterDriver(). I have logged
https://issues.apache.org/jira/browse/DERBY-5384 to track this.
Thanks for finding this bug,
-Rick
> Thanks
>
> Brett
>
>
>
>
>
Re: Question on unloading in an embedded environment
Posted by Bryan Pendleton <bp...@gmail.com>.
> I am getting an OutOfMemory exception. I have the utility setup
> to use up to 8192Mb of memory so somewhere there is a problem.
> Brett
Sometimes, these errors arise externally to Java; e.g., your system
runs out of pagefile or swap space, etc.
It might be worth checking to ensure that you have adequate
system-wide resources.
thanks,
bryan