You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@servicemix.apache.org by navigator09 <ga...@gmail.com> on 2008/12/30 14:46:42 UTC

Load testing problems with servicemix-camel component

Hi,

  I am load testing the following configuration 

   from("jbi:endpoint:http://foo.bar.org/Pipeline/Endpoint?mep=in-out").
  
to("jbi:endpoint:http://www.icenigrid.org/service/gridsam/JobSubmission/JobSubmission?mep=in-out").
   convertBodyTo(DOMSource.class).
   to("log:jobsubmissionoutput").				
  
to("jbi:endpoint:http://foo.bar.org/EventProcessor/EventProcessorendpoint").

The entire flow works well with 7 concurrent threads. But when I try with
more threads , the flow works fine till
jbi:endpoint:http://www.icenigrid.org/service/gridsam/JobSubmission/JobSubmission?mep=in-out
end point. The response from this end point is not forwrded to the event
processor end point and I get the following message

  19:02:04,500 | DEBUG | pool-flow.seda.servicemix-camel-thread-3 |
SecuredBroker            | mix.jbi.security.SecuredBroker   72 | service
name :{http://foo.bar.org}EventProcessor
19:02:04,500 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
DeliveryChannelImpl      | .messaging.DeliveryChannelImpl  693 | Waiting for
exchange ID:10.20.3.92-11e881736ec-2:19 (de1520) to be answered in
DeliveryChannel{servicemix-camel} from sendSync

Then after waiting for around 10 minutes the the thread expires ( as
configured in component.properties). Why is the event processor not being
processed properly for 10 and above concorrent threads. 

Regards,
Ganesh
-- 
View this message in context: http://www.nabble.com/Load-testing-problems-with-servicemix-camel-component-tp21217362p21217362.html
Sent from the ServiceMix - User mailing list archive at Nabble.com.


Re: Load testing problems with servicemix-camel component

Posted by Gert Vanthienen <ge...@skynet.be>.
Ganesh,

What do you mean by x concurrent threads: Are you using 7 or more 
threads for simulating concurrent client load or is it the JBI component 
thread pool size you are refering too?
Could you enabled DEBUG logging en check if all the MEPs are handled 
correctly (i.e. every exchange ends with a DONE message)?  I see you are 
using sendSync, so if some MessageExchange never end with DONE/ERROR, 
this is going to lock up threads.

Which version of ServiceMix are you using? 

Regards,

Gert

navigator09 wrote:
> Hi,
>
>   I am load testing the following configuration 
>
>    from("jbi:endpoint:http://foo.bar.org/Pipeline/Endpoint?mep=in-out").
>   
> to("jbi:endpoint:http://www.icenigrid.org/service/gridsam/JobSubmission/JobSubmission?mep=in-out").
>    convertBodyTo(DOMSource.class).
>    to("log:jobsubmissionoutput").				
>   
> to("jbi:endpoint:http://foo.bar.org/EventProcessor/EventProcessorendpoint").
>
> The entire flow works well with 7 concurrent threads. But when I try with
> more threads , the flow works fine till
> jbi:endpoint:http://www.icenigrid.org/service/gridsam/JobSubmission/JobSubmission?mep=in-out
> end point. The response from this end point is not forwrded to the event
> processor end point and I get the following message
>
>   19:02:04,500 | DEBUG | pool-flow.seda.servicemix-camel-thread-3 |
> SecuredBroker            | mix.jbi.security.SecuredBroker   72 | service
> name :{http://foo.bar.org}EventProcessor
> 19:02:04,500 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
> DeliveryChannelImpl      | .messaging.DeliveryChannelImpl  693 | Waiting for
> exchange ID:10.20.3.92-11e881736ec-2:19 (de1520) to be answered in
> DeliveryChannel{servicemix-camel} from sendSync
>
> Then after waiting for around 10 minutes the the thread expires ( as
> configured in component.properties). Why is the event processor not being
> processed properly for 10 and above concorrent threads. 
>
> Regards,
> Ganesh
>   


Re: Load testing problems with servicemix-camel component

Posted by navigator09 <ga...@gmail.com>.
Hi Freeman,

  I get your point. The number of threads created for the servicemix-camel
was insufficient to cater to the requests for the threads created by the
consumer side. So by adding qeueSize=0 and maximumPoolSize 
= -1( creating as many concurrent threads as requests) to the 
servicemix-camel component, the bottleneck is solved. 

Regards,
Ganesh


Freeman Fang wrote:
> 
> Hi,
> 
> This should be a threadPool issue.
> To avoid the hanging thread, you can set queueSize=0 and maximumPoolSize 
> = -1 for your servicemix-camel component.
> With this configuration, for new incoming request, a new thread will be 
> created, so that avoid the deadlock.
> This may cause a problem(too many threads created) when you have too 
> much concurrent request,
> but if the endpoint you see have the thread problem play the role as 
> provider, I think one solution is that you can also add thread pool on 
> consumer side so that you can control the concurrent invocation number 
> arrive the provider to avoid the resource run out problem.
> 
> For example, the work flow looks like
> external client ===> cxf bc consumer ===> your SE (servicemix-camel in 
> your case)
> Although you can't control how many concurrent invocation  comes to cxf 
> bc consumer, but you can configure the cxf bc consumer thread pool size 
> (by configure the underlying jetty)to determine how many invocation can 
> dispatch to your SE at the same time.
> This is just a example, hopefully  you can get roughly idea how things 
> can work.
> Ideally, the concurrent request should be less than the corePoolSize for 
> component when you use sendSync (which potentially have thread deadlock 
> issue when corePoolSize is run out)
> 
> Another way to avoid the deadlock is use send instead.
> 
> [1] get more details about threadPool configuration
> [1]http://servicemix.apache.org/thread-pools.html
> 
> Freeman
> 
> navigator09 wrote:
>> Hi,
>>
>>   I think there is a deadlock situation between two(6 and 16) threads and
>> the two threads will simply wait till the they time out. I have pasted
>> the
>> logs produced before servicemix stops all activity
>>
>> 19:51:23,171 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 |
>> SecuredBroker            | mix.jbi.security.SecuredBroker   66 | send
>> exchange with secure broker
>> 19:51:23,187 | INFO  | pool-flow.seda.servicemix-camel-thread-6 |
>> jobsubmissionoutput      | .apache.camel.processor.Logger   88 |
>> Exchange[BodyType:javax.xml.transform.dom.DOMSource,
>> Body:<submitJobResponse
>> xmlns="http://www.icenigrid.org/service/gridsam"
>> xmlns:ns2="http://schemas.ggf.org/jsdl/2005/11/jsdl"><JobIdentifier><ID>{"WorkingDirectory":"uploadfiles","ServerIP":"10.20.3.97","Password":"icmconct","UserName":"icmconct","JobDetails":"940:0"}</ID></JobIdentifier></submitJobResponse>]
>> 19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
>> DeliveryChannelImpl      | .messaging.DeliveryChannelImpl  458 | SendSync
>> ID:10.20.3.92-11e88445927-2:33 in DeliveryChannel{servicemix-camel}
>> 19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 |
>> SecuredBroker            | mix.jbi.security.SecuredBroker   72 | service
>> name :{http://foo.bar.org}EventProcessor
>> 19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
>> SecuredBroker            | mix.jbi.security.SecuredBroker   66 | send
>> exchange with secure broker
>> 19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 |
>> SecuredBroker            | mix.jbi.security.SecuredBroker   73 |
>> operation
>> name :{http://www.icenigrid.org/service/gridsam}submitJob
>> 19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
>> SecuredBroker            | mix.jbi.security.SecuredBroker   72 | service
>> name :{http://foo.bar.org}EventProcessor
>> 19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
>> SecuredBroker            | mix.jbi.security.SecuredBroker   73 |
>> operation
>> name :{http://www.icenigrid.org/service/gridsam}submitJob
>> 19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 |
>> SedaFlow                
>> | emix.jbi.nmr.flow.AbstractFlow  118 | Called Flow send
>> 19:51:23,359 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 |
>> DeliveryChannelImpl      | .messaging.DeliveryChannelImpl  693 | Waiting
>> for
>> exchange ID:10.20.3.92-11e88445927-2:32 (19b9669) to be answered in
>> DeliveryChannel{servicemix-camel} from sendSync
>> 19:51:23,359 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
>> SedaFlow                
>> | emix.jbi.nmr.flow.AbstractFlow  118 | Called Flow send
>> 19:51:23,375 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
>> DeliveryChannelImpl      | .messaging.DeliveryChannelImpl  693 | Waiting
>> for
>> exchange ID:10.20.3.92-11e88445927-2:33 (1afe611) to be answered in
>> DeliveryChannel{servicemix-camel} from sendSync
>>
>> How do I solve the problem?
>>
>> Regards,
>> Ganesh
>>
>> navigator09 wrote:
>>   
>>> Hi,
>>>
>>>   I am load testing the following configuration 
>>>
>>>    from("jbi:endpoint:http://foo.bar.org/Pipeline/Endpoint?mep=in-out").
>>>   
>>> to("jbi:endpoint:http://www.icenigrid.org/service/gridsam/JobSubmission/JobSubmission?mep=in-out").
>>>    convertBodyTo(DOMSource.class).
>>>    to("log:jobsubmissionoutput").				
>>>   
>>> to("jbi:endpoint:http://foo.bar.org/EventProcessor/EventProcessorendpoint").
>>>
>>> The entire flow works well with 7 concurrent threads. But when I try
>>> with
>>> more threads , the flow works fine till
>>> jbi:endpoint:http://www.icenigrid.org/service/gridsam/JobSubmission/JobSubmission?mep=in-out
>>> end point. The response from this end point is not forwrded to the event
>>> processor end point and I get the following message
>>>
>>>   19:02:04,500 | DEBUG | pool-flow.seda.servicemix-camel-thread-3 |
>>> SecuredBroker            | mix.jbi.security.SecuredBroker   72 | service
>>> name :{http://foo.bar.org}EventProcessor
>>> 19:02:04,500 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
>>> DeliveryChannelImpl      | .messaging.DeliveryChannelImpl  693 | Waiting
>>> for exchange ID:10.20.3.92-11e881736ec-2:19 (de1520) to be answered in
>>> DeliveryChannel{servicemix-camel} from sendSync
>>>
>>> Then after waiting for around 10 minutes the the thread expires ( as
>>> configured in component.properties). Why is the event processor not
>>> being
>>> processed properly for 10 and above concorrent threads. 
>>>
>>> Regards,
>>> Ganesh
>>>
>>>     
>>
>>   
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Load-testing-problems-with-servicemix-camel-component-tp21217362p21227353.html
Sent from the ServiceMix - User mailing list archive at Nabble.com.


Re: Load testing problems with servicemix-camel component

Posted by Freeman Fang <fr...@gmail.com>.
Hi,

This should be a threadPool issue.
To avoid the hanging thread, you can set queueSize=0 and maximumPoolSize 
= -1 for your servicemix-camel component.
With this configuration, for new incoming request, a new thread will be 
created, so that avoid the deadlock.
This may cause a problem(too many threads created) when you have too 
much concurrent request,
but if the endpoint you see have the thread problem play the role as 
provider, I think one solution is that you can also add thread pool on 
consumer side so that you can control the concurrent invocation number 
arrive the provider to avoid the resource run out problem.

For example, the work flow looks like
external client ===> cxf bc consumer ===> your SE (servicemix-camel in 
your case)
Although you can't control how many concurrent invocation  comes to cxf 
bc consumer, but you can configure the cxf bc consumer thread pool size 
(by configure the underlying jetty)to determine how many invocation can 
dispatch to your SE at the same time.
This is just a example, hopefully  you can get roughly idea how things 
can work.
Ideally, the concurrent request should be less than the corePoolSize for 
component when you use sendSync (which potentially have thread deadlock 
issue when corePoolSize is run out)

Another way to avoid the deadlock is use send instead.

[1] get more details about threadPool configuration
[1]http://servicemix.apache.org/thread-pools.html

Freeman

navigator09 wrote:
> Hi,
>
>   I think there is a deadlock situation between two(6 and 16) threads and
> the two threads will simply wait till the they time out. I have pasted the
> logs produced before servicemix stops all activity
>
> 19:51:23,171 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 |
> SecuredBroker            | mix.jbi.security.SecuredBroker   66 | send
> exchange with secure broker
> 19:51:23,187 | INFO  | pool-flow.seda.servicemix-camel-thread-6 |
> jobsubmissionoutput      | .apache.camel.processor.Logger   88 |
> Exchange[BodyType:javax.xml.transform.dom.DOMSource, Body:<submitJobResponse
> xmlns="http://www.icenigrid.org/service/gridsam"
> xmlns:ns2="http://schemas.ggf.org/jsdl/2005/11/jsdl"><JobIdentifier><ID>{"WorkingDirectory":"uploadfiles","ServerIP":"10.20.3.97","Password":"icmconct","UserName":"icmconct","JobDetails":"940:0"}</ID></JobIdentifier></submitJobResponse>]
> 19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
> DeliveryChannelImpl      | .messaging.DeliveryChannelImpl  458 | SendSync
> ID:10.20.3.92-11e88445927-2:33 in DeliveryChannel{servicemix-camel}
> 19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 |
> SecuredBroker            | mix.jbi.security.SecuredBroker   72 | service
> name :{http://foo.bar.org}EventProcessor
> 19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
> SecuredBroker            | mix.jbi.security.SecuredBroker   66 | send
> exchange with secure broker
> 19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 |
> SecuredBroker            | mix.jbi.security.SecuredBroker   73 | operation
> name :{http://www.icenigrid.org/service/gridsam}submitJob
> 19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
> SecuredBroker            | mix.jbi.security.SecuredBroker   72 | service
> name :{http://foo.bar.org}EventProcessor
> 19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
> SecuredBroker            | mix.jbi.security.SecuredBroker   73 | operation
> name :{http://www.icenigrid.org/service/gridsam}submitJob
> 19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 | SedaFlow                
> | emix.jbi.nmr.flow.AbstractFlow  118 | Called Flow send
> 19:51:23,359 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 |
> DeliveryChannelImpl      | .messaging.DeliveryChannelImpl  693 | Waiting for
> exchange ID:10.20.3.92-11e88445927-2:32 (19b9669) to be answered in
> DeliveryChannel{servicemix-camel} from sendSync
> 19:51:23,359 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 | SedaFlow                
> | emix.jbi.nmr.flow.AbstractFlow  118 | Called Flow send
> 19:51:23,375 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
> DeliveryChannelImpl      | .messaging.DeliveryChannelImpl  693 | Waiting for
> exchange ID:10.20.3.92-11e88445927-2:33 (1afe611) to be answered in
> DeliveryChannel{servicemix-camel} from sendSync
>
> How do I solve the problem?
>
> Regards,
> Ganesh
>
> navigator09 wrote:
>   
>> Hi,
>>
>>   I am load testing the following configuration 
>>
>>    from("jbi:endpoint:http://foo.bar.org/Pipeline/Endpoint?mep=in-out").
>>   
>> to("jbi:endpoint:http://www.icenigrid.org/service/gridsam/JobSubmission/JobSubmission?mep=in-out").
>>    convertBodyTo(DOMSource.class).
>>    to("log:jobsubmissionoutput").				
>>   
>> to("jbi:endpoint:http://foo.bar.org/EventProcessor/EventProcessorendpoint").
>>
>> The entire flow works well with 7 concurrent threads. But when I try with
>> more threads , the flow works fine till
>> jbi:endpoint:http://www.icenigrid.org/service/gridsam/JobSubmission/JobSubmission?mep=in-out
>> end point. The response from this end point is not forwrded to the event
>> processor end point and I get the following message
>>
>>   19:02:04,500 | DEBUG | pool-flow.seda.servicemix-camel-thread-3 |
>> SecuredBroker            | mix.jbi.security.SecuredBroker   72 | service
>> name :{http://foo.bar.org}EventProcessor
>> 19:02:04,500 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
>> DeliveryChannelImpl      | .messaging.DeliveryChannelImpl  693 | Waiting
>> for exchange ID:10.20.3.92-11e881736ec-2:19 (de1520) to be answered in
>> DeliveryChannel{servicemix-camel} from sendSync
>>
>> Then after waiting for around 10 minutes the the thread expires ( as
>> configured in component.properties). Why is the event processor not being
>> processed properly for 10 and above concorrent threads. 
>>
>> Regards,
>> Ganesh
>>
>>     
>
>   


Re: Load testing problems with servicemix-camel component

Posted by navigator09 <ga...@gmail.com>.
Hi,

  I think there is a deadlock situation between two(6 and 16) threads and
the two threads will simply wait till the they time out. I have pasted the
logs produced before servicemix stops all activity

19:51:23,171 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 |
SecuredBroker            | mix.jbi.security.SecuredBroker   66 | send
exchange with secure broker
19:51:23,187 | INFO  | pool-flow.seda.servicemix-camel-thread-6 |
jobsubmissionoutput      | .apache.camel.processor.Logger   88 |
Exchange[BodyType:javax.xml.transform.dom.DOMSource, Body:<submitJobResponse
xmlns="http://www.icenigrid.org/service/gridsam"
xmlns:ns2="http://schemas.ggf.org/jsdl/2005/11/jsdl"><JobIdentifier><ID>{"WorkingDirectory":"uploadfiles","ServerIP":"10.20.3.97","Password":"icmconct","UserName":"icmconct","JobDetails":"940:0"}</ID></JobIdentifier></submitJobResponse>]
19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
DeliveryChannelImpl      | .messaging.DeliveryChannelImpl  458 | SendSync
ID:10.20.3.92-11e88445927-2:33 in DeliveryChannel{servicemix-camel}
19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 |
SecuredBroker            | mix.jbi.security.SecuredBroker   72 | service
name :{http://foo.bar.org}EventProcessor
19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
SecuredBroker            | mix.jbi.security.SecuredBroker   66 | send
exchange with secure broker
19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 |
SecuredBroker            | mix.jbi.security.SecuredBroker   73 | operation
name :{http://www.icenigrid.org/service/gridsam}submitJob
19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
SecuredBroker            | mix.jbi.security.SecuredBroker   72 | service
name :{http://foo.bar.org}EventProcessor
19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
SecuredBroker            | mix.jbi.security.SecuredBroker   73 | operation
name :{http://www.icenigrid.org/service/gridsam}submitJob
19:51:23,343 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 | SedaFlow                
| emix.jbi.nmr.flow.AbstractFlow  118 | Called Flow send
19:51:23,359 | DEBUG | pool-flow.seda.servicemix-camel-thread-16 |
DeliveryChannelImpl      | .messaging.DeliveryChannelImpl  693 | Waiting for
exchange ID:10.20.3.92-11e88445927-2:32 (19b9669) to be answered in
DeliveryChannel{servicemix-camel} from sendSync
19:51:23,359 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 | SedaFlow                
| emix.jbi.nmr.flow.AbstractFlow  118 | Called Flow send
19:51:23,375 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
DeliveryChannelImpl      | .messaging.DeliveryChannelImpl  693 | Waiting for
exchange ID:10.20.3.92-11e88445927-2:33 (1afe611) to be answered in
DeliveryChannel{servicemix-camel} from sendSync

How do I solve the problem?

Regards,
Ganesh

navigator09 wrote:
> 
> Hi,
> 
>   I am load testing the following configuration 
> 
>    from("jbi:endpoint:http://foo.bar.org/Pipeline/Endpoint?mep=in-out").
>   
> to("jbi:endpoint:http://www.icenigrid.org/service/gridsam/JobSubmission/JobSubmission?mep=in-out").
>    convertBodyTo(DOMSource.class).
>    to("log:jobsubmissionoutput").				
>   
> to("jbi:endpoint:http://foo.bar.org/EventProcessor/EventProcessorendpoint").
> 
> The entire flow works well with 7 concurrent threads. But when I try with
> more threads , the flow works fine till
> jbi:endpoint:http://www.icenigrid.org/service/gridsam/JobSubmission/JobSubmission?mep=in-out
> end point. The response from this end point is not forwrded to the event
> processor end point and I get the following message
> 
>   19:02:04,500 | DEBUG | pool-flow.seda.servicemix-camel-thread-3 |
> SecuredBroker            | mix.jbi.security.SecuredBroker   72 | service
> name :{http://foo.bar.org}EventProcessor
> 19:02:04,500 | DEBUG | pool-flow.seda.servicemix-camel-thread-6 |
> DeliveryChannelImpl      | .messaging.DeliveryChannelImpl  693 | Waiting
> for exchange ID:10.20.3.92-11e881736ec-2:19 (de1520) to be answered in
> DeliveryChannel{servicemix-camel} from sendSync
> 
> Then after waiting for around 10 minutes the the thread expires ( as
> configured in component.properties). Why is the event processor not being
> processed properly for 10 and above concorrent threads. 
> 
> Regards,
> Ganesh
> 

-- 
View this message in context: http://www.nabble.com/Load-testing-problems-with-servicemix-camel-component-tp21217362p21217843.html
Sent from the ServiceMix - User mailing list archive at Nabble.com.