Servicemix half freezed: thread dump

[Lorenzo Bolzani <l....@gmail.com>]

Hi,
sending a lot, about 70 client threads, of requests to servicemix through
http half freeze the application.

New requests from http are accepted but no response comes back.

There are no deadlock in the application.

All http threads and all cxf-se just wait here

DeliveryChannelImpl.waitForExchange(MessageExchangeImpl, Object, long,
String) line: 721


What do you think? What could bring to this situation?


Thanks, bye


Lorenzo

Re: Servicemix half freezed: thread dump

[Freeman Fang <fr...@gmail.com>]

Hi,

My comment inline.
On 2011-8-9, at 下午7:51, Lorenzo Bolzani wrote:

> 2011/8/9 Freeman Fang <fr...@gmail.com>
>
>> Hi,
>>
>> As you're using Servicemix 3.3.2, for better performance of  
>> concurrent
>> client, you need add synchronous="false" to cxf bc consumer  
>> configuration.
>> Also you need configure a big thread pool for the cxf bc and cxf se
>> component, take a look at [1] to get more details.
>>
>
>
> Hi Freeman,
> thanks for your reply. The synchronous attribute helped a lot but  
> did not
> solved the problem completely.
>
> Now I can run the application with 100 client threads and it does  
> not freeze
> using 16 as corePoolThreadSize, 256 as maximumPoolSize and 1024 as  
> queueSize
> (global settings).
>
> But if I set the corePoolSize to 4, maximum to 256 and queueSize to  
> 256 then
> the application freezes almost immediately. From the jconsole I can  
> see that
> no new threads are created beyond the initial four.
>
> Considering the rules from the thread-pools page I suppose the queue  
> is not
> full so no new threads are created. But all the cxf-se threads are  
> stopped
> here
>
> java.lang.Object.wait(Native Method)
> org 
> .apache 
> .servicemix 
> .jbi 
> .messaging 
> .DeliveryChannelImpl.waitForExchange(DeliveryChannelImpl.java:721)
> org 
> .apache 
> .servicemix 
> .jbi.messaging.DeliveryChannelImpl.sendSync(DeliveryChannelImpl.java: 
> 482)
> [...]
> org.apache.servicemix.cxfse.CxfSeEndpoint.process(CxfSeEndpoint.java: 
> 452)
>
> and everything cxf-se related stops.
>
> Looks like there is a minimum number of threads required for a  
> process to
> complete. For example 8 is enough, but 4 is not (my case). Is this  
> correct?
Yeah, if corePoolSize is too small but queueSize is large, then  
because all available threads are used up(in wait) and no more new  
thread could be created as the queueSize isn't full, so no new thread  
can notify the wait thread, so no more thread could be released, so  
deadlock happen.


>
> Considering that the number of threads does not increase how can I  
> be sure
> that I have enough for a specific complex BPEL process? Should I set  
> the
> queue lenght very low to force new threads to be created immediately  
> (I
> tryed, this works but looks like a bad hack)? Should I set the  
> corePoolSize
> very high and do a lot of testing?
You need fine-tune the threadpool configuration parameter per your  
business scale.
Set queueSize=-1 can avoid the deadlock situation in this case.

Freeman
>
> For my specific application this is a big issue. I found almost  
> nothing
> about it with google. Am I doing something wrong? For example calling
> sendSync is bad practice? Should I avoid it at all costs?
>
>
> Thanks, bye
>
> Lorenzo

---------------------------------------------
Freeman Fang

FuseSource
Email:ffang@fusesource.com
Web: fusesource.com
Twitter: freemanfang
Blog: http://freemanfang.blogspot.com

Re: Servicemix half freezed: thread dump

[Lorenzo Bolzani <l....@gmail.com>]

Freeman, Gert thanks for your replies.


Setting the queueSize to zero worked: now new threads are started as
required and closed afterward.

Setting it to -1 did not work, I report this in case you want to further
investigate this case.


For each client thread the number of cxf server threads used is about 1 and
a half (20 clients -> 30 cxf threads).


Do you think using a JMS flow for this case could require less threads? I
expect some overhead but with less threads.

I've started to study this page

http://servicemix.apache.org/nmr-flows.html

but here the JMS flow is described mostly in a cluster context.

I'll give it a try as soon as I better understand how to configure flows.


Thanks, bye

Lorenzo

Re: Servicemix half freezed: thread dump

[Gert Vanthienen <ge...@gmail.com>]

Lorenzo,


Generally speaking, if you can avoid the use of sendSync, it becomes a lot
easier for the application to scale and to be tuned.  If your application
requires you to use sendSync, every call to sendSync will block the thread
until the exchange has been handled, forcing you to ensure that the
configuration has enough threads available to handle the workload.  This
becomes especially important if you are use routes with multiple endpoints
from the same component type, as those will all need threads from the same
thread pool if you don't add endpoint specific thread configurations.

I would set the core pool size as low as possible, but keeping in mind the
minimum number of threads that you need to actually handle the exchanges
(e.g. 8 sounds like a good number in your scenario).  To allow for the
system to scale under load, you can then increase the max pool size and
lower the queue size for those particular endpoints that need it.  Those
threads will only get created on demand and will also be removed after a
timeout period to ensure the system thread count goes down again after a
period of high traffic.


Regards,

Gert Vanthienen
------------------------
FuseSource
Web: http://fusesource.com
Blog: http://gertvanthienen.blogspot.com/


On Tue, Aug 9, 2011 at 1:51 PM, Lorenzo Bolzani <l....@gmail.com> wrote:

> 2011/8/9 Freeman Fang <fr...@gmail.com>
>
> > Hi,
> >
> > As you're using Servicemix 3.3.2, for better performance of concurrent
> > client, you need add synchronous="false" to cxf bc consumer
> configuration.
> > Also you need configure a big thread pool for the cxf bc and cxf se
> > component, take a look at [1] to get more details.
> >
>
>
> Hi Freeman,
> thanks for your reply. The synchronous attribute helped a lot but did not
> solved the problem completely.
>
> Now I can run the application with 100 client threads and it does not
> freeze
> using 16 as corePoolThreadSize, 256 as maximumPoolSize and 1024 as
> queueSize
> (global settings).
>
> But if I set the corePoolSize to 4, maximum to 256 and queueSize to 256
> then
> the application freezes almost immediately. From the jconsole I can see
> that
> no new threads are created beyond the initial four.
>
> Considering the rules from the thread-pools page I suppose the queue is not
> full so no new threads are created. But all the cxf-se threads are stopped
> here
>
> java.lang.Object.wait(Native Method)
>
> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.waitForExchange(DeliveryChannelImpl.java:721)
>
> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.sendSync(DeliveryChannelImpl.java:482)
> [...]
> org.apache.servicemix.cxfse.CxfSeEndpoint.process(CxfSeEndpoint.java:452)
>
> and everything cxf-se related stops.
>
> Looks like there is a minimum number of threads required for a process to
> complete. For example 8 is enough, but 4 is not (my case). Is this correct?
>
> Considering that the number of threads does not increase how can I be sure
> that I have enough for a specific complex BPEL process? Should I set the
> queue lenght very low to force new threads to be created immediately (I
> tryed, this works but looks like a bad hack)? Should I set the corePoolSize
> very high and do a lot of testing?
>
> For my specific application this is a big issue. I found almost nothing
> about it with google. Am I doing something wrong? For example calling
> sendSync is bad practice? Should I avoid it at all costs?
>
>
> Thanks, bye
>
> Lorenzo
>

Re: Servicemix half freezed: thread dump

[Lorenzo Bolzani <l....@gmail.com>]

2011/8/9 Freeman Fang <fr...@gmail.com>

> Hi,
>
> As you're using Servicemix 3.3.2, for better performance of concurrent
> client, you need add synchronous="false" to cxf bc consumer configuration.
> Also you need configure a big thread pool for the cxf bc and cxf se
> component, take a look at [1] to get more details.
>


Hi Freeman,
thanks for your reply. The synchronous attribute helped a lot but did not
solved the problem completely.

Now I can run the application with 100 client threads and it does not freeze
using 16 as corePoolThreadSize, 256 as maximumPoolSize and 1024 as queueSize
(global settings).

But if I set the corePoolSize to 4, maximum to 256 and queueSize to 256 then
the application freezes almost immediately. From the jconsole I can see that
no new threads are created beyond the initial four.

Considering the rules from the thread-pools page I suppose the queue is not
full so no new threads are created. But all the cxf-se threads are stopped
here

java.lang.Object.wait(Native Method)
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.waitForExchange(DeliveryChannelImpl.java:721)
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.sendSync(DeliveryChannelImpl.java:482)
[...]
org.apache.servicemix.cxfse.CxfSeEndpoint.process(CxfSeEndpoint.java:452)

and everything cxf-se related stops.

Looks like there is a minimum number of threads required for a process to
complete. For example 8 is enough, but 4 is not (my case). Is this correct?

Considering that the number of threads does not increase how can I be sure
that I have enough for a specific complex BPEL process? Should I set the
queue lenght very low to force new threads to be created immediately (I
tryed, this works but looks like a bad hack)? Should I set the corePoolSize
very high and do a lot of testing?

For my specific application this is a big issue. I found almost nothing
about it with google. Am I doing something wrong? For example calling
sendSync is bad practice? Should I avoid it at all costs?


Thanks, bye

Lorenzo

Re: Servicemix half freezed: thread dump

[Freeman Fang <fr...@gmail.com>]

Hi,

As you're using Servicemix 3.3.2, for better performance of concurrent  
client, you need add synchronous="false" to cxf bc consumer  
configuration.
Also you need configure a big thread pool for the cxf bc and cxf se  
component, take a look at [1] to get more details.

[1]http://servicemix.apache.org/thread-pools.html

Freeman
On 2011-8-9, at 上午4:10, Lorenzo Bolzani wrote:

>
> Hi,
> I try to better explain the situation.
>
> I'm running Servicemix 3.3.2, jdk 1.5.0, on linux 2.6.38
>
> I have deployed a simple cxf component that does nothing at all and  
> returns a simple result class.
>
> I have a client generated from the WSDL that invokes the component  
> using several threads, up to 100 to quickly reproduce the problem.
> This client runs inside eclipse.
>
> After a few seconds requests stop to be passed to the cxf component.
>
> As attachment there is the log of DeliveryChannelImpl with level  
> DEBUG. You can see that threads gradually stop to be notified and  
> all remain waiting.
>
>
> This "freeze" is a problem we have in production with a few threads  
> and complex bpel processes.
>
>
> Let me know if and how I can provide more infos about this problem.
>
>
> Thanks, bye
>
> Lorenzo
> <servicemix-lock2.log.zip>

---------------------------------------------
Freeman Fang

FuseSource
Email:ffang@fusesource.com
Web: fusesource.com
Twitter: freemanfang
Blog: http://freemanfang.blogspot.com

Re: Servicemix half freezed: thread dump

[Lorenzo Bolzani <l....@gmail.com>]

Hi,
I try to better explain the situation.

I'm running Servicemix 3.3.2, jdk 1.5.0, on linux 2.6.38

I have deployed a simple cxf component that does nothing at all and returns
a simple result class.

I have a client generated from the WSDL that invokes the component using
several threads, up to 100 to quickly reproduce the problem.
This client runs inside eclipse.

After a few seconds requests stop to be passed to the cxf component.

As attachment there is the log of DeliveryChannelImpl with level DEBUG. You
can see that threads gradually stop to be notified and all remain waiting.


This "freeze" is a problem we have in production with a few threads and
complex bpel processes.


Let me know if and how I can provide more infos about this problem.


Thanks, bye

Lorenzo