CXF + ActiveMQ + CachingConnectionFactory: consumers stop consuming

[Pavel <pa...@gmail.com>]

Hi,

We are observing an issue with CXF/SOAP over
JMS/ActiveMQ/CachingConnectionFactory:

Application shares same Spring's CachingConnectionFactory between 3
endpoints and ~5 clients. Clients are called from enpoint web service
methods. Clients use static response queues.

Initially application works well, but at some point clients start timing
out. And ActiveMQ web console shows that
* client response queue has several consumers
* They have non-zero and non-decreasing "Dispatched Queue"
* On each client call, response is returned by target service (we can see
that in the target service log), and "dispatched queue" for one of consumers
increases. Yet at application level there are no indications of response;
nothing is logged with CXF logging interceptor.

We have a load test that reproduces the issue, and a few observations:

* With SingleConnectionFactory everything works.
* setting prefetch size to 1 increases time application works before getting
stuck.
* Setting sessionCacheSize to 1 makes problem disappear (or not reproduce
within reasonable timeframe)
* Other applications that simply expose one endpoint and do not share
connection factory with anything do not exhibit such problem.

Any ideas what could be happening? And why SingleConnectionFactory would
make a difference?
And also, is there a known negative impact from using
SingleConnectionFactory with CXF/ACtiveMQ?

Thanks,
Pavel

Re: CXF + ActiveMQ + CachingConnectionFactory: consumers stop consuming

[Gary Tully <ga...@gmail.com>]

consumer prefetch with pooled consumers/connections is hazardous, with the
default prefetch of 1000 and spring template taking a single message from a
consumer.receive before returning it to a pool. 999 messages could be
pending delivery on a consumer cached in a pooled connection. Until that
connection is retrieved form the pool, which can depend on pool size and
load, those prefetched messages will be locked out from other consumers
which can lead to starvation.

Connection pools are fine for message producers, but the prefetch (eagerly
dispatching messages to a consumer) can cause problems for consumers that
are shared.

Using prefetch=0 or not caching consumers or using a single connection can
all help.
Having seperate pools for producers and consumers makes this easier to
achieve but not sure if CXF has that option.

Prefetch is great when a consumer is long lived and dedicated, it is only
when a consumer is inadvertently shared that it needs to be fully
understood.
http://activemq.apache.org/what-is-the-prefetch-limit-for.html


On 19 June 2010 16:07, Pavel <pa...@gmail.com> wrote:

> Hi,
>
> We are observing an issue with CXF/SOAP over
> JMS/ActiveMQ/CachingConnectionFactory:
>
> Application shares same Spring's CachingConnectionFactory between 3
> endpoints and ~5 clients. Clients are called from enpoint web service
> methods. Clients use static response queues.
>
> Initially application works well, but at some point clients start timing
> out. And ActiveMQ web console shows that
> * client response queue has several consumers
> * They have non-zero and non-decreasing "Dispatched Queue"
> * On each client call, response is returned by target service (we can see
> that in the target service log), and "dispatched queue" for one of
> consumers
> increases. Yet at application level there are no indications of response;
> nothing is logged with CXF logging interceptor.
>
> We have a load test that reproduces the issue, and a few observations:
>
> * With SingleConnectionFactory everything works.
> * setting prefetch size to 1 increases time application works before
> getting
> stuck.
> * Setting sessionCacheSize to 1 makes problem disappear (or not reproduce
> within reasonable timeframe)
> * Other applications that simply expose one endpoint and do not share
> connection factory with anything do not exhibit such problem.
>
> Any ideas what could be happening? And why SingleConnectionFactory would
> make a difference?
> And also, is there a known negative impact from using
> SingleConnectionFactory with CXF/ACtiveMQ?
>
> Thanks,
> Pavel
>



-- 
http://blog.garytully.com

Open Source Integration
http://fusesource.com