You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "Daniel John Debrunner (JIRA)" <de...@db.apache.org> on 2005/07/06 22:39:09 UTC
[jira] Created: (DERBY-444) Handle OutOfMemoryError exceptions when creating a new embedded connection
Handle OutOfMemoryError exceptions when creating a new embedded connection
--------------------------------------------------------------------------
Key: DERBY-444
URL: http://issues.apache.org/jira/browse/DERBY-444
Project: Derby
Type: Sub-task
Components: JDBC
Reporter: Daniel John Debrunner
Assigned to: Daniel John Debrunner
If an OutOfMemoryError is thrown while creating objects for a new embedded connection, reject the connection request with a SQLException and do not shutdown the system.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-444) Handle OutOfMemoryError exceptions
when creating a new embedded connection
Posted by "Bryan Pendleton (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-444?page=comments#action_12368805 ]
Bryan Pendleton commented on DERBY-444:
---------------------------------------
I'm not sure I'm following the basic thrust of this. Is there something special about the Connection object? Couldn't we just as easily happen to run out of memory when creating a Statement, or a ResultSet, or at various other places where the app calls into Derby to perform some work?
Would every class in the external API need be changed similarly to how you changed EmbedConnection and InternalDriver?
thanks,
bryan
> Handle OutOfMemoryError exceptions when creating a new embedded connection
> --------------------------------------------------------------------------
>
> Key: DERBY-444
> URL: http://issues.apache.org/jira/browse/DERBY-444
> Project: Derby
> Type: Sub-task
> Components: JDBC
> Reporter: Daniel John Debrunner
> Assignee: Daniel John Debrunner
> Attachments: derby444_draft_v1.txt
>
> If an OutOfMemoryError is thrown while creating objects for a new embedded connection, reject the connection request with a SQLException and do not shutdown the system.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Closed: (DERBY-444) Handle OutOfMemoryError exceptions when
creating a new embedded connection
Posted by "Daniel John Debrunner (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-444?page=all ]
Daniel John Debrunner closed DERBY-444:
---------------------------------------
> Handle OutOfMemoryError exceptions when creating a new embedded connection
> --------------------------------------------------------------------------
>
> Key: DERBY-444
> URL: http://issues.apache.org/jira/browse/DERBY-444
> Project: Derby
> Type: Sub-task
> Components: JDBC
> Reporter: Daniel John Debrunner
> Assignee: Daniel John Debrunner
> Fix For: 10.2.0.0
> Attachments: derby444_draft_v1.txt
>
> If an OutOfMemoryError is thrown while creating objects for a new embedded connection, reject the connection request with a SQLException and do not shutdown the system.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-444) Handle OutOfMemoryError exceptions
when creating a new embedded connection
Posted by "Daniel John Debrunner (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-444?page=comments#action_12368821 ]
Daniel John Debrunner commented on DERBY-444:
---------------------------------------------
More on the setting the low water mark. The low memory water mark is a value, proposed by this fix, that disables operations
if the JVM's free memory is lower than the mark, to allow some chance of recovery rather than wasting cpu cycles performing
an operation that is most likely doomed to throw an OutOfMemoryError (OOME). The low water mark would be set dynamically per operation type by checking the free memory at the first OOME for that operation.
Given that I can successfully open around 19,000 connections in 64Mb, that makes a just opened connection overhead on the order of 3.5kb. I was suprised when the call Runtime.getRuntime().freeMemory() returned around 100k to 400k when an OutOfMemoryError (OOME) is hit the first time opening a connection. I was expecting a number closer to the apparent overhead of the connection.
Because in a multi-threaded environment another thread could release a whole lot of memory between my thread's OOME and it calling Runtime.getRuntime().freeMemory() I was planning to ignore the low water mark if was was "bigger" than some estimate, in this case say 10k for a connection. This was to avoid a situation of setting a low water mark that was huge (due to other released memory) and subsequently disabling all connection requests.
However the reality is the freeMemory at OOME time seems much bigger than any reasonable estimate, and I'm not sure a estimate of 100k-400k for a connection seems reasonable when the overhead is 3.5k. Seems like in some JVMs the low water
mark could typically be much lower.
My new approach is to only have the low water mark for 5 seconds, then it gets discarded, any subsequent OOME for the same operation would then set a new low water mark. This means even a really bad low water mark would not break the system, but just deny service of that operation for five seconds. I picked five seconds as a reasonable time for memory to be released, say by future work in Derby.
I also tried running the memory/ConnectionHandling test without the concept of a low water mark, just throwing the
static SQLException if an OOME was thrown. This did not work, the test just hung somewhere in the opening 500
extra connections once the first OOME was hit. Thus this backing off strategy with the low water mark is some help,
it allows the JVM to spend some cpu cycles doing useful garbage collection, rather than trying operations that are doomed to fail.
> Handle OutOfMemoryError exceptions when creating a new embedded connection
> --------------------------------------------------------------------------
>
> Key: DERBY-444
> URL: http://issues.apache.org/jira/browse/DERBY-444
> Project: Derby
> Type: Sub-task
> Components: JDBC
> Reporter: Daniel John Debrunner
> Assignee: Daniel John Debrunner
> Attachments: derby444_draft_v1.txt
>
> If an OutOfMemoryError is thrown while creating objects for a new embedded connection, reject the connection request with a SQLException and do not shutdown the system.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-444) Handle OutOfMemoryError exceptions
when creating a new embedded connection
Posted by "Daniel John Debrunner (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-444?page=comments#action_12368650 ]
Daniel John Debrunner commented on DERBY-444:
---------------------------------------------
The draft patch has three components:
1) Catching OutOfMemoryErrors in the paths where a new embedded connection is obtained. Basically within
EmbeddedConnection constructor and within InternalDriver when creating the connection.
2) A new static SQLException with SQLstate 08004 that is thrown when a connection cannot be obtained due to low memory
3) A new LowMemory class that provides utiltity methods to indicate and check for low memory. Any area that
can recover from an out of memory situation can create a static instance of this class to manage its own low
memory threshold. The setLowMemory() is called when an OutOfMemoryError is thrown and records the current free
memory, that value is then used as future watermark at which point the operation (in this case opening a connection)
will not be attempted and instead a low memory situation is assumed, and the static exception thrown.
Setting the low water mark value needs some more investigation.
Future ideas include having a callback where on a low memory situation, seen by LowMemory.setLowMemory,
callbaclks coudl be invoked to get classes to reduce their memory usage, such as caches.
> Handle OutOfMemoryError exceptions when creating a new embedded connection
> --------------------------------------------------------------------------
>
> Key: DERBY-444
> URL: http://issues.apache.org/jira/browse/DERBY-444
> Project: Derby
> Type: Sub-task
> Components: JDBC
> Reporter: Daniel John Debrunner
> Assignee: Daniel John Debrunner
> Attachments: derby444_draft_v1.txt
>
> If an OutOfMemoryError is thrown while creating objects for a new embedded connection, reject the connection request with a SQLException and do not shutdown the system.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-444) Handle OutOfMemoryError exceptions when
creating a new embedded connection
Posted by "Daniel John Debrunner (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-444?page=all ]
Daniel John Debrunner updated DERBY-444:
----------------------------------------
Attachment: derby444_draft_v1.txt
Draft patch of the direction I'm working on.
The program/test memory/ConnectionHandling.java (added under this issue) creates about 19,000 connections
in a JVM limited to 64Mb before failing due to an OutOfMemoryError. The test then goes on to try to open 500
more connections (while holding the previous 19,000+ open) to see what happens.
With the existing code basically a mess happens, the initial OutOfMemoryError I think closes the database
and then each of the 500 connection opens is a (slow) boot attempt with limited memory. About 95 out of 500 pass,
but I believe all the existing connections are closed since the database was closed underneath them. The
test does have logic to count how many of its existing open connections have become closed , but with the
existing code an OutOfMemoryError is hit and no reporting takes place.
With the patch, roughly the same number (19,000+) of connections are successfully opened, but the most of
the subsequent failures due to lack of memory receive a SQLException with a 08004 SQL state. These exceptions
do not shut down the database and the 19,000 connections remain open, and can be closed with no further exceptions.
About 495 of the 500 requests once memory is full receive a 08004 SQLException, and about 5 recieve a direct OutOfMemoryError.
These dirrect OutOfMemoryErrors occur above Derby, in the DriverManager class due to object allocation there. Thus the OOME has
no effect on Derby.
Thus a summary is:
Existing code:
19,000+ connections in 64Mb, once limit is hit all are implicitly closed and a handful of future requests succeed
but any OOME shuts the database down again. System does not degrade well.
Patch
19,000+ connections in 64Mb, once limit is hit some stability is acheieved, no more connection requests succeed.
System behaves well.
Next comment will describe this draft patch
> Handle OutOfMemoryError exceptions when creating a new embedded connection
> --------------------------------------------------------------------------
>
> Key: DERBY-444
> URL: http://issues.apache.org/jira/browse/DERBY-444
> Project: Derby
> Type: Sub-task
> Components: JDBC
> Reporter: Daniel John Debrunner
> Assignee: Daniel John Debrunner
> Attachments: derby444_draft_v1.txt
>
> If an OutOfMemoryError is thrown while creating objects for a new embedded connection, reject the connection request with a SQLException and do not shutdown the system.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Resolved: (DERBY-444) Handle OutOfMemoryError exceptions
when creating a new embedded connection
Posted by "Daniel John Debrunner (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-444?page=all ]
Daniel John Debrunner resolved DERBY-444:
-----------------------------------------
Fix Version: 10.2.0.0
Resolution: Fixed
> Handle OutOfMemoryError exceptions when creating a new embedded connection
> --------------------------------------------------------------------------
>
> Key: DERBY-444
> URL: http://issues.apache.org/jira/browse/DERBY-444
> Project: Derby
> Type: Sub-task
> Components: JDBC
> Reporter: Daniel John Debrunner
> Assignee: Daniel John Debrunner
> Fix For: 10.2.0.0
> Attachments: derby444_draft_v1.txt
>
> If an OutOfMemoryError is thrown while creating objects for a new embedded connection, reject the connection request with a SQLException and do not shutdown the system.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Commented: (DERBY-444) Handle OutOfMemoryError exceptions
when creating a new embedded connection
Posted by Bryan Pendleton <bp...@amberpoint.com>.
> So, yes, one would need to do other changes in other areas, I'm just starting off in one well contained area
Excellent, that makes perfect sense.
> I do think something needs to be done in this area (DERBY-443)
Me too, and I think your approach is a good one.
Thanks for the follow-up.
bryan
[jira] Commented: (DERBY-444) Handle OutOfMemoryError exceptions
when creating a new embedded connection
Posted by "Daniel John Debrunner (JIRA)" <de...@db.apache.org>.
[ http://issues.apache.org/jira/browse/DERBY-444?page=comments#action_12368811 ]
Daniel John Debrunner commented on DERBY-444:
---------------------------------------------
This is a sub-task of
Make Derby engine robust in case of OutOfMemoryError exceptions (DERBY-443)
So, yes, one would need to do other changes in other areas, I'm just starting off in one well contained area as a way to
see what is possible and to see if I can provide some form of initial framework/useful code. Incremental development,
get something simple working quickly and build from there, rather than trying to solve everything completely.
I do think something needs to be done in this area (DERBY-443), there is the person on derby-user that put the cache-size up
to 10,,000 pages and may be hitting OutOfMemoryErrors, wouldn't it be better if Derby just didn't shutdown with his
configuration, instead kept runing after it saw an OOME with a smaller cache. This specific patch/Jira issue doesn't get us there
but I'm trying to make a start. I just chose to start with connections.
> Handle OutOfMemoryError exceptions when creating a new embedded connection
> --------------------------------------------------------------------------
>
> Key: DERBY-444
> URL: http://issues.apache.org/jira/browse/DERBY-444
> Project: Derby
> Type: Sub-task
> Components: JDBC
> Reporter: Daniel John Debrunner
> Assignee: Daniel John Debrunner
> Attachments: derby444_draft_v1.txt
>
> If an OutOfMemoryError is thrown while creating objects for a new embedded connection, reject the connection request with a SQLException and do not shutdown the system.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira