Force condition: ERROR 08000: Connection closed by unknown interrupt

[Phil Bradley <ph...@tower.ie>]

Hi,

I have an application that uses Derby 10.8.1.2 in embedded mode.
Occasionally, I run into the condition: 

java.sql.SQLNonTransientConnectionException: Connection closed by
unknown interrupt
...
Caused by: ERROR 08000: Connection closed by unknown interrupt.

My understanding is that this state occurs if a thread accessing the
database gets interrupted. As a workaround, the application has some
logic which detects when this occurs and handles it by:

1. Closing all connections
2. Shutting down the database (new DerbyShutdownRunner().shutdown())
3. Recreating the connections

At the moment, this doesn't address the problem. Specifically, even
after taking these steps, the exception gets thrown after every database
operation. It's quite possible that there's a flaw in the implementation
of the above steps but the meta issue is that I don't have a way to test
this logic as I don't have a way to force the error condition.

So, my question is, is there a way that I can reliably force the
application into the above state in order to test its ability to recover
from it? 

Thanks,
Phil

Re: Force condition: ERROR 08000: Connection closed by unknown interrupt

[Phil Bradley <ph...@tower.ie>]

Hi Knut, 

You've potentially addressed both issues here (that is, how to force the
issue to occur and how to address it when it does occur). 

Thank you very much for you help.
Regards,
Phil



On Thursday, October 20, 2011 5:25 PM, "Knut Anders Hatlen"
<kn...@oracle.com> wrote:
> "Phil Bradley" <ph...@tower.ie> writes:
> 
> >> 
> >> Please bear with me, I do not understand the following paragraph:
> >> > At the moment, this doesn't address the problem. Specifically, even
> >> > after taking these steps, the exception gets thrown after every database
> >> > operation.
> >> When I read that sentence, I get the impression that the 08000 exception 
> >> is being thrown on every JDBC call. Is that right?
> >
> > Yes, that's correct. The issue is that I cannot force the 08000
> > condition but once it occurs, every JDBC call subsequent to that fails
> > until the application is restarted. Note that I've never experienced any
> > data corruption due to this, just problems with the connection;
> > restarting the application has always resolved it.
> 
> Maybe the thread's interrupt flag is still on? Derby doesn't clear the
> flag when it detects that it has been interrupted, so if you retry
> database operations in the same thread after receiving this exception, I
> think you'll continue to see that the operations are being interrupted
> until you clear the flag, for example by calling Thread.interrupted().
> 
> > There's every possibility that there is a problem with the application
> > code; it's certainly a little more convoluted that it should be and I'm
> > trying to reduce the complexity bit by bit. For the moment though, my
> > issue is not with the 08000 error per se, it's that I have no way to
> > verify that the error handling that is invoked in response to the 08000
> > actually works. 
> 
> Maybe you can get it to fail by manually interrupting the thread before
> doing some db operations. The following code fails with a 08000 error on
> the rs.next() call in my environment:
> 
> Thread.currentThread().interrupt();
> Connection c =
> DriverManager.getConnection("jdbc:derby:memory:db;create=true");
> Statement s = c.createStatement();
> ResultSet rs = s.executeQuery("select * from sysibm.sysdummy1");
> rs.next();
> 
> -- 
> Knut Anders
> 

Re: Force condition: ERROR 08000: Connection closed by unknown interrupt

[Knut Anders Hatlen <kn...@oracle.com>]

"Phil Bradley" <ph...@tower.ie> writes:

>> 
>> Please bear with me, I do not understand the following paragraph:
>> > At the moment, this doesn't address the problem. Specifically, even
>> > after taking these steps, the exception gets thrown after every database
>> > operation.
>> When I read that sentence, I get the impression that the 08000 exception 
>> is being thrown on every JDBC call. Is that right?
>
> Yes, that's correct. The issue is that I cannot force the 08000
> condition but once it occurs, every JDBC call subsequent to that fails
> until the application is restarted. Note that I've never experienced any
> data corruption due to this, just problems with the connection;
> restarting the application has always resolved it.

Maybe the thread's interrupt flag is still on? Derby doesn't clear the
flag when it detects that it has been interrupted, so if you retry
database operations in the same thread after receiving this exception, I
think you'll continue to see that the operations are being interrupted
until you clear the flag, for example by calling Thread.interrupted().

> There's every possibility that there is a problem with the application
> code; it's certainly a little more convoluted that it should be and I'm
> trying to reduce the complexity bit by bit. For the moment though, my
> issue is not with the 08000 error per se, it's that I have no way to
> verify that the error handling that is invoked in response to the 08000
> actually works. 

Maybe you can get it to fail by manually interrupting the thread before
doing some db operations. The following code fails with a 08000 error on
the rs.next() call in my environment:

Thread.currentThread().interrupt();
Connection c = DriverManager.getConnection("jdbc:derby:memory:db;create=true");
Statement s = c.createStatement();
ResultSet rs = s.executeQuery("select * from sysibm.sysdummy1");
rs.next();

-- 
Knut Anders

Re: Force condition: ERROR 08000: Connection closed by unknown interrupt

[Phil Bradley <ph...@tower.ie>]

Hi Rick,

Thanks for your interest, I'll try to clarify.

> I'm interested in why you have chosen this workaround. If Derby's 
> interrupt handling is working correctly, then the database and all other 
> connections to it should be fine. Probably the interrupted connection is 
> fine too and you have only lost the transaction which was running in the 
> interrupted connection. Of course, Derby may not be performing the way 
> we think it should be. Could you explain why you chose this expensive 
> solution?

In previous incarnations of the application, it would simply try to
recreate the connection but even though it would receive a Connection
object, every JDBC call would fail with the same exception. The solution
I describe above was implemented when the application was using 10.4.
It's certainly not lightweight but seemed to be the only sure approach. 

My hope is that if I can force the condition to arise, then I will be in
a position to come up with error handling that works. If there are more
lightweight solutions than that described above, then even better as
long as I can prove to myself that what I've done actually works.

> 
> Please bear with me, I do not understand the following paragraph:
> > At the moment, this doesn't address the problem. Specifically, even
> > after taking these steps, the exception gets thrown after every database
> > operation.
> When I read that sentence, I get the impression that the 08000 exception 
> is being thrown on every JDBC call. Is that right?

Yes, that's correct. The issue is that I cannot force the 08000
condition but once it occurs, every JDBC call subsequent to that fails
until the application is restarted. Note that I've never experienced any
data corruption due to this, just problems with the connection;
restarting the application has always resolved it.

There's every possibility that there is a problem with the application
code; it's certainly a little more convoluted that it should be and I'm
trying to reduce the complexity bit by bit. For the moment though, my
issue is not with the 08000 error per se, it's that I have no way to
verify that the error handling that is invoked in response to the 08000
actually works. 

Thanks,
Phil

Re: Force condition: ERROR 08000: Connection closed by unknown interrupt

[Rick Hillegas <ri...@oracle.com>]

On 10/20/11 4:59 AM, Phil Bradley wrote:
> Hi,
>
> I have an application that uses Derby 10.8.1.2 in embedded mode.
> Occasionally, I run into the condition:
>
> java.sql.SQLNonTransientConnectionException: Connection closed by
> unknown interrupt
> ...
> Caused by: ERROR 08000: Connection closed by unknown interrupt.
>
> My understanding is that this state occurs if a thread accessing the
> database gets interrupted. As a workaround, the application has some
> logic which detects when this occurs and handles it by:
>
> 1. Closing all connections
> 2. Shutting down the database (new DerbyShutdownRunner().shutdown())
> 3. Recreating the connections
Hi Phil,

I'm interested in why you have chosen this workaround. If Derby's 
interrupt handling is working correctly, then the database and all other 
connections to it should be fine. Probably the interrupted connection is 
fine too and you have only lost the transaction which was running in the 
interrupted connection. Of course, Derby may not be performing the way 
we think it should be. Could you explain why you chose this expensive 
solution?

Please bear with me, I do not understand the following paragraph:
> At the moment, this doesn't address the problem. Specifically, even
> after taking these steps, the exception gets thrown after every database
> operation.
When I read that sentence, I get the impression that the 08000 exception 
is being thrown on every JDBC call. Is that right?
>   It's quite possible that there's a flaw in the implementation
> of the above steps but the meta issue is that I don't have a way to test
> this logic as I don't have a way to force the error condition.
This is what is confusing me. I thought that the previous sentence said 
that you were reliably experiencing the error, on every JDBC call.

Thanks,
-Rick
> So, my question is, is there a way that I can reliably force the
> application into the above state in order to test its ability to recover
> from it?
>
> Thanks,
> Phil
>