You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Mirek Borsky <ge...@email.cz> on 2018/07/25 19:16:14 UTC
Concurrent consumers from SEDA to CXF
Greetings,
I have encountered a strange behavior with a simple camel route and I would
like to ask for help of more experienced users. First, let me quickly show
the route:
The relevant part:
<from uri="vm:sam.logger.to.server?concurrentConsumers=10" />
<to uri="MonitoringService" />
The whole blueprint:
<?xml version="1.0" encoding="UTF-8"?>
<osgi:blueprint xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v
1.0.0 https://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd" xmlns:xsi=
"http://www.w3.org/2001/XMLSchema-instance" xmlns:cxf="http://camel.apache.
org/schema/blueprint/cxf" xmlns:osgi="http://www.osgi.org/xmlns/blueprint/v
1.0.0">
<camelContext id="TestSAMLoggerSeda-ctx" xmlns="http://camel.apache.org/
schema/blueprint" streamCache="true">
<route>
<from uri="vm:sam.logger.to.server?concurrentConsumers=10" />
<to uri="MonitoringService" />
</route>
</camelContext>
<cxf:cxfEndpoint xmlns:tns="http://www.talend.org/esb/sam/
MonitoringService/v1" id="MonitoringService" address="http://localhost:8042/
MonitoringMock" serviceName="tns:MonitoringWebServiceService" endpointName=
"tns:MonitoringWebServicePort" wsdlURL="deploy/MonitoringService.wsdl">
<cxf:properties>
<osgi:entry key="dataFormat" value="PAYLOAD"/>
</cxf:properties>
</cxf:cxfEndpoint>
</osgi:blueprint>
Now the expected behavior:
What I want to achieve is a queue from which 10 parallel threads are
processing messages, calling the cxf endpoint. The cxf endpoint takes
several seconds to answer, so I am expecting the speed of processing
somewhere about 1-2 messages per second.
Observed behavior:
Using JMX I can see X+N inflight messages in the route, where X is the
maximum number of concurrent http connections to the cxf endpoint and N is
the number of concurrent users. For some reason, the workers that are
waiting for the cxf endpoint to answer are not counted against the
concurrentConsumers limit. Only the workers that are waiting for the
connection to be established are counted against the limit of
concurrentConsumers.
Questions:
What is the correct behavior for the route?
If my configuration is incorrect, how can I achieve my target behavior?
The camel version used is 2.20.
If someone would be able to help me, and explain what I did wrong, I would
very much appreciate it.
thank you for help
kind regards
Miroslav Borský
Re: Concurrent consumers from SEDA to CXF
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Ah okay its likely because CXF can process asynchronously and you may
not have a limit on the seda/vm queue so it can buffer pending
inflight exchanges.
So try setting cxf to be synchronous, via synchronous=true on its endpoint.
On Tue, Jul 31, 2018 at 2:13 PM, Mirek Borsky <ge...@email.cz> wrote:
> Hi Claus,
>
> I would try to explain. Please ask for the clarification as needed.
>
> In the route in my example, if there will a delayer as the component that
> take 2 seconds to proceed, the route will look like this:
> <route>
> <from uri="vm:sam.logger.to.server?concurrentConsumers=10" />
> <delay><constant>2000</constant></delay>
> </route>
>
> there will be 10 inflight messages in this route when the seda queue is
> filled. The rate of processing of the messages from the seda queue will be
> around 5 messages per second. This is the expected behaviour when I set
> concurrentConsumers=10 on the seda queue..
>
> However, when instead of <delay/> there will be the cxf call to a service
> that take 2 seconds to answer, things will change. In my runtime
> nevironment, I can see 110 inflight messages in the route. Also the rate of
> processing of the messages from the seda queue is much higher, around 50 per
> second.
>
> The problem I perceive is not the number of concurrent requests for the cxf
> endpoint. Problem is the fact, that the combination of seda queue and cxf
> endpoint seems to exhaust all the available connections regardless of the
> concurrentConsumers setting.
>
> I hope I have provided the example that help you better understand the issue
> I am facing. If I have failed in doing so, please bear with me and try to
> ask differently.
>
> kind regards
> Miroslav Borský
>
>
> ---------- Původní e-mail ----------
> Od: Claus Ibsen <cl...@gmail.com>
> Komu: users@camel.apache.org
> Datum: 30. 7. 2018 17:04:45
> Předmět: Re: Concurrent consumers from SEDA to CXF
> "Hi
>
> Can you try to explain the numbers a bit more, maybe with some example
> numbers?
>
> For http connections etc, then CXF has its own set of limitations,
> specially if you call the same remote server via the same client, eg
> same cxf-endpoint in Camel.
> That may have limitations on number of concurrent requests, that can
> be lower. So try to look at the CXF documentation about this.
>
> On Wed, Jul 25, 2018 at 9:16 PM, Mirek Borsky <ge...@email.cz> wrote:
>> Greetings,
>>
>> I have encountered a strange behavior with a simple camel route and I
> would
>> like to ask for help of more experienced users. First, let me quickly show
>
>> the route:
>>
>> The relevant part:
>> <from uri="vm:sam.logger.to.server?concurrentConsumers=10" />
>> <to uri="MonitoringService" />
>>
>> The whole blueprint:
>> <?xml version="1.0" encoding="UTF-8"?>
>> <osgi:blueprint xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v
>> 1.0.0 https://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd" xmlns:xsi
> =
>> "http://www.w3.org/2001/XMLSchema-instance" xmlns:cxf="http://camel.
> apache.
>> org/schema/blueprint/cxf" xmlns:osgi="http://www.osgi.org/xmlns/blueprint/
> v
>> 1.0.0">
>> <camelContext id="TestSAMLoggerSeda-ctx" xmlns="http://camel.apache.org/
>> schema/blueprint" streamCache="true">
>> <route>
>> <from uri="vm:sam.logger.to.server?concurrentConsumers=10" />
>> <to uri="MonitoringService" />
>> </route>
>> </camelContext>
>> <cxf:cxfEndpoint xmlns:tns="http://www.talend.org/esb/sam/
>> MonitoringService/v1" id="MonitoringService" address="http://localhost:
> 8042/
>> MonitoringMock" serviceName="tns:MonitoringWebServiceService" endpointName
> =
>> "tns:MonitoringWebServicePort" wsdlURL="deploy/MonitoringService.wsdl">
>> <cxf:properties>
>> <osgi:entry key="dataFormat" value="PAYLOAD"/>
>> </cxf:properties>
>> </cxf:cxfEndpoint>
>> </osgi:blueprint>
>>
>> Now the expected behavior:
>> What I want to achieve is a queue from which 10 parallel threads are
>> processing messages, calling the cxf endpoint. The cxf endpoint takes
>> several seconds to answer, so I am expecting the speed of processing
>> somewhere about 1-2 messages per second.
>>
>> Observed behavior:
>> Using JMX I can see X+N inflight messages in the route, where X is the
>> maximum number of concurrent http connections to the cxf endpoint and N is
>
>> the number of concurrent users. For some reason, the workers that are
>> waiting for the cxf endpoint to answer are not counted against the
>> concurrentConsumers limit. Only the workers that are waiting for the
>> connection to be established are counted against the limit of
>> concurrentConsumers.
>>
>> Questions:
>> What is the correct behavior for the route?
>> If my configuration is incorrect, how can I achieve my target behavior?
>>
>> The camel version used is 2.20.
>>
>> If someone would be able to help me, and explain what I did wrong, I would
>
>> very much appreciate it.
>>
>> thank you for help
>> kind regards
>> Miroslav Borský
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
> "
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: Concurrent consumers from SEDA to CXF
Posted by Mirek Borsky <ge...@email.cz>.
Hi Claus,
I would try to explain. Please ask for the clarification as needed.
In the route in my example, if there will a delayer as the component that
take 2 seconds to proceed, the route will look like this:
<route>
<from uri="vm:sam.logger.to.server?concurrentConsumers=10" />
<delay><constant>2000</constant></delay>
</route>
there will be 10 inflight messages in this route when the seda queue is
filled. The rate of processing of the messages from the seda queue will be
around 5 messages per second. This is the expected behaviour when I set
concurrentConsumers=10 on the seda queue..
However, when instead of <delay/> there will be the cxf call to a service
that take 2 seconds to answer, things will change. In my runtime
nevironment, I can see 110 inflight messages in the route. Also the rate of
processing of the messages from the seda queue is much higher, around 50 per
second.
The problem I perceive is not the number of concurrent requests for the cxf
endpoint. Problem is the fact, that the combination of seda queue and cxf
endpoint seems to exhaust all the available connections regardless of the
concurrentConsumers setting.
I hope I have provided the example that help you better understand the issue
I am facing. If I have failed in doing so, please bear with me and try to
ask differently.
kind regards
Miroslav Borský
---------- Původní e-mail ----------
Od: Claus Ibsen <cl...@gmail.com>
Komu: users@camel.apache.org
Datum: 30. 7. 2018 17:04:45
Předmět: Re: Concurrent consumers from SEDA to CXF
"Hi
Can you try to explain the numbers a bit more, maybe with some example
numbers?
For http connections etc, then CXF has its own set of limitations,
specially if you call the same remote server via the same client, eg
same cxf-endpoint in Camel.
That may have limitations on number of concurrent requests, that can
be lower. So try to look at the CXF documentation about this.
On Wed, Jul 25, 2018 at 9:16 PM, Mirek Borsky <ge...@email.cz> wrote:
> Greetings,
>
> I have encountered a strange behavior with a simple camel route and I
would
> like to ask for help of more experienced users. First, let me quickly show
> the route:
>
> The relevant part:
> <from uri="vm:sam.logger.to.server?concurrentConsumers=10" />
> <to uri="MonitoringService" />
>
> The whole blueprint:
> <?xml version="1.0" encoding="UTF-8"?>
> <osgi:blueprint xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v
> 1.0.0 https://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd" xmlns:xsi
=
> "http://www.w3.org/2001/XMLSchema-instance" xmlns:cxf="http://camel.
apache.
> org/schema/blueprint/cxf" xmlns:osgi="http://www.osgi.org/xmlns/blueprint/
v
> 1.0.0">
> <camelContext id="TestSAMLoggerSeda-ctx" xmlns="http://camel.apache.org/
> schema/blueprint" streamCache="true">
> <route>
> <from uri="vm:sam.logger.to.server?concurrentConsumers=10" />
> <to uri="MonitoringService" />
> </route>
> </camelContext>
> <cxf:cxfEndpoint xmlns:tns="http://www.talend.org/esb/sam/
> MonitoringService/v1" id="MonitoringService" address="http://localhost:
8042/
> MonitoringMock" serviceName="tns:MonitoringWebServiceService" endpointName
=
> "tns:MonitoringWebServicePort" wsdlURL="deploy/MonitoringService.wsdl">
> <cxf:properties>
> <osgi:entry key="dataFormat" value="PAYLOAD"/>
> </cxf:properties>
> </cxf:cxfEndpoint>
> </osgi:blueprint>
>
> Now the expected behavior:
> What I want to achieve is a queue from which 10 parallel threads are
> processing messages, calling the cxf endpoint. The cxf endpoint takes
> several seconds to answer, so I am expecting the speed of processing
> somewhere about 1-2 messages per second.
>
> Observed behavior:
> Using JMX I can see X+N inflight messages in the route, where X is the
> maximum number of concurrent http connections to the cxf endpoint and N is
> the number of concurrent users. For some reason, the workers that are
> waiting for the cxf endpoint to answer are not counted against the
> concurrentConsumers limit. Only the workers that are waiting for the
> connection to be established are counted against the limit of
> concurrentConsumers.
>
> Questions:
> What is the correct behavior for the route?
> If my configuration is incorrect, how can I achieve my target behavior?
>
> The camel version used is 2.20.
>
> If someone would be able to help me, and explain what I did wrong, I would
> very much appreciate it.
>
> thank you for help
> kind regards
> Miroslav Borský
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
"
Re: Concurrent consumers from SEDA to CXF
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Can you try to explain the numbers a bit more, maybe with some example numbers?
For http connections etc, then CXF has its own set of limitations,
specially if you call the same remote server via the same client, eg
same cxf-endpoint in Camel.
That may have limitations on number of concurrent requests, that can
be lower. So try to look at the CXF documentation about this.
On Wed, Jul 25, 2018 at 9:16 PM, Mirek Borsky <ge...@email.cz> wrote:
> Greetings,
>
> I have encountered a strange behavior with a simple camel route and I would
> like to ask for help of more experienced users. First, let me quickly show
> the route:
>
> The relevant part:
> <from uri="vm:sam.logger.to.server?concurrentConsumers=10" />
> <to uri="MonitoringService" />
>
> The whole blueprint:
> <?xml version="1.0" encoding="UTF-8"?>
> <osgi:blueprint xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v
> 1.0.0 https://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd" xmlns:xsi=
> "http://www.w3.org/2001/XMLSchema-instance" xmlns:cxf="http://camel.apache.
> org/schema/blueprint/cxf" xmlns:osgi="http://www.osgi.org/xmlns/blueprint/v
> 1.0.0">
> <camelContext id="TestSAMLoggerSeda-ctx" xmlns="http://camel.apache.org/
> schema/blueprint" streamCache="true">
> <route>
> <from uri="vm:sam.logger.to.server?concurrentConsumers=10" />
> <to uri="MonitoringService" />
> </route>
> </camelContext>
> <cxf:cxfEndpoint xmlns:tns="http://www.talend.org/esb/sam/
> MonitoringService/v1" id="MonitoringService" address="http://localhost:8042/
> MonitoringMock" serviceName="tns:MonitoringWebServiceService" endpointName=
> "tns:MonitoringWebServicePort" wsdlURL="deploy/MonitoringService.wsdl">
> <cxf:properties>
> <osgi:entry key="dataFormat" value="PAYLOAD"/>
> </cxf:properties>
> </cxf:cxfEndpoint>
> </osgi:blueprint>
>
> Now the expected behavior:
> What I want to achieve is a queue from which 10 parallel threads are
> processing messages, calling the cxf endpoint. The cxf endpoint takes
> several seconds to answer, so I am expecting the speed of processing
> somewhere about 1-2 messages per second.
>
> Observed behavior:
> Using JMX I can see X+N inflight messages in the route, where X is the
> maximum number of concurrent http connections to the cxf endpoint and N is
> the number of concurrent users. For some reason, the workers that are
> waiting for the cxf endpoint to answer are not counted against the
> concurrentConsumers limit. Only the workers that are waiting for the
> connection to be established are counted against the limit of
> concurrentConsumers.
>
> Questions:
> What is the correct behavior for the route?
> If my configuration is incorrect, how can I achieve my target behavior?
>
> The camel version used is 2.20.
>
> If someone would be able to help me, and explain what I did wrong, I would
> very much appreciate it.
>
> thank you for help
> kind regards
> Miroslav Borský
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2