[jira] [Created] (CXF-6778) Invalid replyDestination is cached after jms connection has been reset

["Jerome Waibel (JIRA)" <ji...@apache.org>]

Jerome Waibel created CXF-6778:
----------------------------------

             Summary: Invalid replyDestination is cached after jms connection has been reset
                 Key: CXF-6778
                 URL: https://issues.apache.org/jira/browse/CXF-6778
             Project: CXF
          Issue Type: Bug
          Components: JMS, Transports
    Affects Versions: 3.0.8, 3.1.5
            Reporter: Jerome Waibel
            Priority: Critical


We have a spring boot application that is doing SOAP over JMS with a tibco backend.

Whenever the JMS connection is reset our application fails to receive the answer for the SOA request until the application server is restarted.

>From the logs we see that re-establishing the jms connection is working fine:
{quote}
2016-02-05 12:59:03.458  WARN 5662 --- [TIBCO EMS TCPLink Reader (Server-608470)] o.s.j.c.CachingConnectionFactory         : Encountered a JMSException - resetting the underlying JMS Connection
javax.jms.JMSException: Connection has been terminated
        at com.tibco.tibjms.Tibjmsx.buildException(Tibjmsx.java:509)
        at com.tibco.tibjms.TibjmsConnection._invokeOnExceptionCallback(TibjmsConnection.java:2025)
        at com.tibco.tibjms.TibjmsConnection._onDisconnected(TibjmsConnection.java:2394)
        at com.tibco.tibjms.TibjmsConnection$ServerLinkEventHandler.onEventDisconnected(TibjmsConnection.java:349)
         at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:330)
         at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
2016-02-05 12:59:04.701  INFO 5662 --- [ajp-bio-18009-exec-1] o.s.j.c.CachingConnectionFactory: Established shared JMS Connection: QueueConnection[ClientId=null Connected=tcp://ems2-k:12004, URL=tcp://ems2-k:12004]
{quote}

After the connection has been reset when our application does the next SOAP call sending the request works fine, but when CXF waits for the reply the following exception occurs:
{quote}
javax.xml.ws.soap.SOAPFaultException: Timeout receiving message with correlationId b02ad4683421442db439b54797fcc7350000000000000003
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:161)
{quote}

Debugging shows that the problem seems to be located in _public Destination getReplyDestination(Session session) in org.apache.cxf.transport.jms.JMSConfiguration_. There a Destination object for receiving the answer is created once and cached forever. There is no way this cached destination will ever be dropped and recreated. This destination object (implemented by a temporary queue in our case) contains a reference to the jms connection. So after a reset of the connection it still contains the old, invalid jms connection object. This is why after a connection reset it is impossible to receive any more replies.

Using a debugger, setting a breakpoint in this method and manually setting replyDestinationDest to null forces this method to recreate the temporary queue. After that receiving replies is working again after a jms connection reset. This is of course not possible for our live environment, there all servers have to be restarted after a jms connection reset.

This behaviour was introduced in CXF3. Before upgrading we used CXF 2 where the temporary response queue seems not to be cached but created and deleted for every request. With CXF 2 our application worked fine when the jms connection was reset.

Steps for reproducing this error (sorry, no example project yet):
* Have a SOAP server
* Have a SOAP client using Spring so that CachingConnectionFactory will do a proper connection reset
* Have a SOAP over JMS broker
* Make one successful client request so that the replyDestinationDest gets created and cached.
* Force a JMS connection reset (e.g. drop connection from the broker admin console)
* Try another client request. The client will reconnect but re-use the old replyDestinationDest and never be able to receive the reply again.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)