NetXAResource.recover() causes throwXAException on transaction branch

[Jason Free <jf...@sms-fed.com>]

I have two databases (one Derby 10.8.2.2, one Oracle 11) joining a
transaction.  I'm using Atomikos as the transaction manager.  When Atomikos
calls recover() with TMNOFLAGS to get the Xid(s) on the Derby branch it
throws an exception.  The stacktrace is below...

-------stacktrace---------
java.util.ConcurrentModificationException
    at java.util.WeakHashMap$HashIterator.remove(Unknown Source)
    at org.apache.derby.client.am.Connection.markStatementsClosed(Unknown
Source)
    at
org.apache.derby.client.am.Connection.completeChainBreakingDisconnect(Unknown
Source)
    at org.apache.derby.client.am.Agent.disconnectEvent(Unknown Source)
    at org.apache.derby.client.am.DisconnectException.<init>(Unknown Source)
    at org.apache.derby.client.am.DisconnectException.<init>(Unknown Source)
    at org.apache.derby.client.am.DisconnectException.<init>(Unknown Source)
    at org.apache.derby.client.am.DisconnectException.<init>(Unknown Source)
    at org.apache.derby.client.net.Reply.doSyntaxrmSemantics(Unknown Source)
    at org.apache.derby.client.net.Reply.readDssHeader(Unknown Source)
    at org.apache.derby.client.net.Reply.startSameIdChainParse(Unknown
Source)
    at org.apache.derby.client.net.NetStatementReply.readOpenQuery(Unknown
Source)
    at org.apache.derby.client.net.StatementReply.readOpenQuery(Unknown
Source)
    at org.apache.derby.client.net.NetStatement.readOpenQuery_(Unknown
Source)
    at org.apache.derby.client.am.Statement.readOpenQuery(Unknown Source)
    at org.apache.derby.client.am.PreparedStatement.flowExecute(Unknown
Source)
    at org.apache.derby.client.am.PreparedStatement.executeQueryX(Unknown
Source)
    at org.apache.derby.client.am.PreparedStatement.executeQuery(Unknown
Source)
    at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
    at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
    at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
...

org.apache.derby.client.am.XaException: XAER_RMFAIL : A network protocol
error was encountered and the connection has been terminated: A PROTOCOL
Data Stream Syntax Error was detected.  Reason: 0x3. Plaintext connection
attempt to an SSL enabled server?
    at org.apache.derby.client.net.NetXAResource.throwXAException(Unknown
Source)
    at org.apache.derby.client.net.NetXAResource.recover(Unknown Source)
    at
com.atomikos.datasource.xa.XATransactionalResource.recover(XATransactionalResource.java:585)
    at
com.atomikos.datasource.xa.XATransactionalResource.endRecovery(XATransactionalResource.java:670)
    at
com.atomikos.icatch.imp.TransactionServiceImp.recover(TransactionServiceImp.java:635)
    at
com.atomikos.datasource.xa.XATransactionalResource.setRecoveryService(XATransactionalResource.java:503)
    at
com.atomikos.icatch.system.Configuration.addResource(Configuration.java:404)
    at
com.atomikos.icatch.jta.TransactionImp.enlistResource(TransactionImp.java:334)
    at
org.apache.commons.dbcp.managed.TransactionContext.setSharedConnection(TransactionContext.java:89)
    at
org.apache.commons.dbcp.managed.ManagedConnection.updateTransactionStatus(ManagedConnection.java:131)
    at
org.apache.commons.dbcp.managed.ManagedConnection.<init>(ManagedConnection.java:55)
    at
org.apache.commons.dbcp.managed.ManagedDataSource.getConnection(ManagedDataSource.java:77)
    at
org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
...

Even though it says there is a protocol error, I think that's a red herring
because the server isn't SSL enabled.  I'm not sure if the real cause is
because the Derby branch at the time is only doing a select query and so
the status is XA_RDONLY but the conditional at the end of the recover()
method is only checking for XA_OK and throwing the exception for all other
cases or if it is because the transaction branches are in separate threads
and Derby is "going first" and already running the query when the TM
enlists the second database and calls recover() on all branches to
enumerate the XID(s).  The same scenario seems to work fine with MySQL,
PostgreSQL, etc. Is the Derby client thread safe in a XA transaction?

Thanks,
-Jason