CXF and options for async SOAP over JMS

[Pavel <pa...@gmail.com>]

Hi,

I have a few questions about SOAP over JMS implementation in CXF and
applicability to high volume applications.

my requirements are
* wdsl-first approach. (See no problems with CXF)

* it should be simple JMS transport wrapper over existing web service.
  As I understand CXF's SOAP over JMS implementation addresses this
perfectly. I can afford extra port and binding.

* It should allow for transactional request processing.
  - e.g. runtime exception in service itself shoulf cause message
redelivery, but legitimate faults will be sent to reply destination.
  I did not get to this part yet, but I assume this should be possible with
JMS transport.

and for client:

* send requests in transactional fashion
  - e.g. runtime exception later in the same transaction should roll back
sent message.

* process response in transactional fashion
  - e.g. runtime exception in response handler should cause response message
redelivery

* provide simple async API for client
  - allow working with same POJOs application would work in synchronous
request/response case.

* scale - application will be sending ~200K messages a day and thus will
receive same number of responses.
  Both service and client will be clustered.

* allow for significant response time - due to heavy and spiky load it may
sometimes take hours for service to respond. It is a batch process, so
client is ok to wait.
  - But it has to process response eventually. Application cant afford
dropping responses.

Specifics of my task is that response is self-contained and can be processed
with no knowledge of request. I.e. no need to correlate messages.
I also know that data binding will be JAXB.
**********

Before I realized there is a way to generate async interface, my plan (and
3/4 of implementation) went like this:

I extended JMSConduit with implementation that overrides sendExchange(), so
that all exchanges processed as one-way ones.
One difference is that pre-configured reply destination still added to the
outbound message.

I also extended JaxWsClientProxy and overrode invoke() implementation so
that it always calls invokeAsync().

Both pieces contain enough of code copied from original classes, so it does
not look terribly clean.

For response processing I created listener that takes original CXF client
and extracts JAXB context (endpoint -> service -> dataBinding -> context).
When message arrives it applies xpath to extract
"/soap:Envelope/soap:Body/*", deserializes results with JAXB context, and
then calls actual biz logic with resulting element.
******

While it all works and provides simple API so far, I was wondering if there
is a better way. And I found out async bindings possible (sorry, I'm new to
JAX-WS binding).

Now, I'm seriously considering dropping all my efforts in favour of standard
async binding, but a few things still bother me.
I really hope this list subscribers could help me clarify a couple of
questions.

* How do async bindings with JMS scale?
I.e. in my mind lots of messages with response time in minutes or perhaps
even hours leads to numerous listeners with numerous selectors.
Also to large correlation map.

* How reliable async binding is for high load/slow response case?
  E.g. if client goes down, correlation map is lost, so are all responses.
Or did I miss the way to recover after that?

* Does it work for static response queues?
  E.g. http://cwiki.apache.org/CXF20DOC/jms-transport.html states that
"static reply queue can not be shared by several instances of the service
client", but it is unclear if the issue is still relevant.

* What is a degree of transaction support? I.e. is it possible to roll back
message sent, or expect redelivery if response processing failed?


Also if anyone already has experience with CXF/SOAP/JMS deployment and could
share, that would be really great.

Thanks!
Pavel

Re: CXF and options for async SOAP over JMS

[Pavel <pa...@gmail.com>]

I did a prototype with Camel and so far it looks amazingly simple (in terms
of how much code/config it took) and powerful.

Unless there are hidden issues down the road, Camel just rocks.

Pavel
On Sun, Oct 4, 2009 at 4:24 PM, Pavel <pa...@gmail.com> wrote:

> Christian - thanks a lot for response.
>
> I was looking into both 2.2 and 2.2.3 sources and it seems like correlation
> map is always populated for anything but one-way exchanges:
> correlationMap.put(correlationId, exchange); <<< all logic branches except
> one way call this
>
> Also I see how sendExchange() calls JMSFactory.createJmsListener() that
> would create a brand new DefaultMessageListenerContainer per exchange. I
> wonder if thread settings would be effective with that approach.
>
> And while I see trunk has changed that to jmsTemplate.receiveSelected(), I
> realize one conceptual problem is still there: even if correlation part is
> solved, application still needs to send something in order to receive
> something.
>
> With clustered client and possible node failures in mind, I think I won't
> be able to use async implementation as is.
>
> So my options are
> * customize CXF client, like what I did. And the more I look at it the more
> I realize I extended part that was not designed for extension. I get ability
> to leverage JMS endpoint on server side though for free.
>
> * Use 2 two-way clients as you suggested. I'll look into other options
> first, as there are service consumers that would still want to call it
> synchronously.
>
> * Give up on the idea to reuse SOAP over JMS on server side and create
> another invocation layer (IL), like Camel-backed one you advised. It
> probably would be the best option, assuming that
>   - it is possible to route all messages to original web service
> implementor, even though IL has changed
>   - Exceptions/faults are somehow delivered to client too.
>   - (Optional, but nice to have) JAXB context is borrowed from endpoint
> metadata.
>
> Is it something Camel could do?
>
> Thanks a lot for your help!
> Pavel
>
>

Re: CXF and options for async SOAP over JMS

[Pavel <pa...@gmail.com>]

Christian - thanks a lot for response.

I was looking into both 2.2 and 2.2.3 sources and it seems like correlation
map is always populated for anything but one-way exchanges:
correlationMap.put(correlationId, exchange); <<< all logic branches except
one way call this

Also I see how sendExchange() calls JMSFactory.createJmsListener() that
would create a brand new DefaultMessageListenerContainer per exchange. I
wonder if thread settings would be effective with that approach.

And while I see trunk has changed that to jmsTemplate.receiveSelected(), I
realize one conceptual problem is still there: even if correlation part is
solved, application still needs to send something in order to receive
something.

With clustered client and possible node failures in mind, I think I won't be
able to use async implementation as is.

So my options are
* customize CXF client, like what I did. And the more I look at it the more
I realize I extended part that was not designed for extension. I get ability
to leverage JMS endpoint on server side though for free.

* Use 2 two-way clients as you suggested. I'll look into other options
first, as there are service consumers that would still want to call it
synchronously.

* Give up on the idea to reuse SOAP over JMS on server side and create
another invocation layer (IL), like Camel-backed one you advised. It
probably would be the best option, assuming that
  - it is possible to route all messages to original web service
implementor, even though IL has changed
  - Exceptions/faults are somehow delivered to client too.
  - (Optional, but nice to have) JAXB context is borrowed from endpoint
metadata.

Is it something Camel could do?

Thanks a lot for your help!
Pavel

On Sat, Oct 3, 2009 at 12:17 AM, Christian Schneider <
chris@die-schneider.net> wrote:

> Hi Pavel,
>
> I am not sure about the transactional part but I can help you with the
> asnyc interface.
>
> The easiest way to achieve asynch behaviour is to create a method with only
> an in message and no out message. CXF will by default make the call
> asynchronous.  The  new wsdl first example on the subversion head shows
> this.
>
> The other way to use async is to use a binding file that declares
>
> Pavel schrieb:
>
>> * How do async bindings with JMS scale?
>> I.e. in my mind lots of messages with response time in minutes or perhaps
>> even hours leads to numerous listeners with numerous selectors.
>> Also to large correlation map.
>>
>>
> The JMS transport uses a spring DefaultMessageListenerContainer. Using the
> JMSConfigFeature you can tweak it´s params.
> http://cwiki.apache.org/CXF20DOC/using-the-jmsconfigfeature.html
>
> You can define the number of listener threads and then also the number of
> threads the executor allows. So there should be no problem with overloading.
> When all listeners and executors are blocked no additional messages will be
> processed and wait in the JMS queue. You could even make sure that only one
> message is processed at a time.
>
>> * How reliable async binding is for high load/slow response case?
>>  E.g. if client goes down, correlation map is lost, so are all responses.
>> Or did I miss the way to recover after that?
>>
>>
> I think you can turn off the correlation and use the async interface. So
> any response should be processed. As you say you have the correlation
> information in your
> data this should work.
>
> If that does not work you could use two one way services. One for request
> one for reply. So there would be no correlation.
>
> BTW. In your case of purely asynchronous communication that is only
> correlated by the data it could also be an option to use simply XML over
> JMS. You can serialize and deserialize with JAXB.
> Apache Camel allows to do such things very easily. If you are interested I
> could make a small example for this case.
>
>> * Does it work for static response queues?
>>  E.g. http://cwiki.apache.org/CXF20DOC/jms-transport.html states that
>> "static reply queue can not be shared by several instances of the service
>> client", but it is unclear if the issue is still relevant.
>>
>>
> The problem was that with a static queue and an open selector the wrong
> clients would retrieve the responses. But in your case this would be
> desired.
>
>> Also if anyone already has experience with CXF/SOAP/JMS deployment and
>> could
>> share, that would be really great.
>>
>>
>>
> We use SOAP over JMS as the main service transport at EnBW (germany energy
> company). Till now it works very well. We have not that big throughput
> though.
>
> Greetings
>
> Christian
>
>
> --
>
> Christian Schneider
> ---
> http://www.liquid-reality.de
>
>

Re: CXF and options for async SOAP over JMS

[Christian Schneider <ch...@die-schneider.net>]

Hi Pavel,

I am not sure about the transactional part but I can help you with the 
asnyc interface.

The easiest way to achieve asynch behaviour is to create a method with 
only an in message and no out message. CXF will by default make the call 
asynchronous.  The  new wsdl first example on the subversion head shows 
this.

The other way to use async is to use a binding file that declares

Pavel schrieb:
> * How do async bindings with JMS scale?
> I.e. in my mind lots of messages with response time in minutes or perhaps
> even hours leads to numerous listeners with numerous selectors.
> Also to large correlation map.
>   
The JMS transport uses a spring DefaultMessageListenerContainer. Using 
the JMSConfigFeature you can tweak it´s params.
http://cwiki.apache.org/CXF20DOC/using-the-jmsconfigfeature.html

You can define the number of listener threads and then also the number 
of threads the executor allows. So there should be no problem with 
overloading. When all listeners and executors are blocked no additional 
messages will be processed and wait in the JMS queue. You could even 
make sure that only one message is processed at a time.
> * How reliable async binding is for high load/slow response case?
>   E.g. if client goes down, correlation map is lost, so are all responses.
> Or did I miss the way to recover after that?
>   
I think you can turn off the correlation and use the async interface. So 
any response should be processed. As you say you have the correlation 
information in your
data this should work.

If that does not work you could use two one way services. One for 
request one for reply. So there would be no correlation.

BTW. In your case of purely asynchronous communication that is only 
correlated by the data it could also be an option to use simply XML over 
JMS. You can serialize and deserialize with JAXB.
Apache Camel allows to do such things very easily. If you are interested 
I could make a small example for this case.
> * Does it work for static response queues?
>   E.g. http://cwiki.apache.org/CXF20DOC/jms-transport.html states that
> "static reply queue can not be shared by several instances of the service
> client", but it is unclear if the issue is still relevant.
>   
The problem was that with a static queue and an open selector the wrong 
clients would retrieve the responses. But in your case this would be 
desired.
> Also if anyone already has experience with CXF/SOAP/JMS deployment and could
> share, that would be really great.
>
>   
We use SOAP over JMS as the main service transport at EnBW (germany 
energy company). Till now it works very well. We have not that big 
throughput though.

Greetings

Christian


-- 

Christian Schneider
---
http://www.liquid-reality.de