You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cxf.apache.org by Sergey Beryozkin <se...@progress.com> on 2008/11/17 13:12:03 UTC
JMS queues and multiple consumer threads
Hi
I have a system JMS test which involves multiple consumer threads hitting a single provider endpoint which is a JMS queue and
expecting some response.
If more than 1 consumer thread is involved than I can often see a server side being unable even to get a given invocation to the
endpoint application code.
For ex, with 2 threads involved, here's a failed test :
// S.B - this is happening on the consumer side I believe :
14000 [pool-1-thread-1] INFO org.springframework.jms.connection.SingleConnectionFactory - Established shared JMS Connection:
ActiveMQConnection {id=ID:sberyoz-4729-1226923006536-1:1,clientId=null,started=false}
// S.B : I added these printlns :
Receieve timeout******** : 500000
Receieve timeout******** : 500000
//S.B This is a server side now
17-Nov-2008 11:56:48 org.apache.cxf.transport.jms.JMSDestination onMessage
FINE: server received request:
17-Nov-2008 11:56:48 org.apache.cxf.transport.jms.JMSDestination onMessage
FINE: The Request Message is [ [B@1465ca2]
and this is basically, the println in an application code is not even hit.
Now, same test with 2 threads succeeding :
// S.B. : client side
14594 [pool-1-thread-2] INFO org.springframework.jms.connection.SingleConnectionFactory - Established shared JMS Connection:
ActiveMQConnection {id=ID:sberyoz-4775-1226923455594-1:1,clientId=null,started=false}
Receieve timeout******** : 500000
Receieve timeout******** : 500000
// S.B. The first thread has managed to get through
7-Nov-2008 12:04:17 org.apache.cxf.transport.jms.JMSDestination onMessage
INFO: The Request Message is [ [B@148c02f]
Getting continuation for Fred
Suspending invocation for Fred
Request message has been suspended
....
// S.B the second one is lucky too
17-Nov-2008 12:04:17 org.apache.cxf.transport.jms.JMSDestination onMessage
INFO: The Request Message is [ [B@115470e]
Invocation for Fred has been suspended
Can someone help me to figure why I'm seeing this transient behaviour ?
here's a wsdl configuration (which I copied from the other test's wsdl) shared by all the consumer threads and by the provider
endpoint :
<service name="HelloContinuationService">
<port binding="tns:HelloContinuationServiceSoapBinding" name="HelloContinuationPort">
<jms:clientConfig clientReceiveTimeout="500000" messageTimeToLive="500000"/>
<jms:serverConfig messageTimeToLive="500000"/>
<jms:address
jndiConnectionFactoryName="ConnectionFactory"
jndiDestinationName="dynamicQueues/test.jmstransport.text">
<jms:JMSNamingProperty name="java.naming.factory.initial" value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/>
<jms:JMSNamingProperty name="java.naming.provider.url" value="tcp://localhost:61500"/>
</jms:address>
<jms:server durableSubscriberName="CXF_subscriber"/>
</port>
</service>
Thanks, Sergey
Re: JMS queues and multiple consumer threads
Posted by Sergey Beryozkin <se...@progress.com>.
Oh, thanks, I've checked
http://cwiki.apache.org/CXF20DOC/jms-transport.html
but missed this one which is actually linked to from the above page :-)
Cheers, Sergey
----- Original Message -----
From: "Daniel Kulp" <dk...@apache.org>
To: <us...@cxf.apache.org>
Cc: "Sergey Beryozkin" <se...@progress.com>
Sent: Monday, November 17, 2008 3:39 PM
Subject: Re: JMS queues and multiple consumer threads
>
> Sergey,
>
> See:
> http://cwiki.apache.org/CXF20DOC/using-the-jmsconfigfeature.html
>
> There is a setting for concurrentConsumers and maxConcurrentConsumers.
>
> Dan
>
>
>
> On Monday 17 November 2008 9:35:44 am Sergey Beryozkin wrote:
>> Now that I believe I understand the reason for these transient failuires,
>> it was very easy to update the test to have the control threads spawned by
>> the actual server endpoint code.
>> In other words, my JMS continuations test has been originally written the
>> same way the HTTP one has been done. Control threads invoke on a service
>> endpoint and wait till they're told that the main user threads have invoked
>> on say helloService.sayHi() and have called cont.suspend(), so what they do
>> next is call helloService.resume() which results in a given suspended
>> continuation be resumed.
>>
>> For JMS this model simply does not walk for me at the moment due to the
>> fact that multiple consumer threads sharing the same client proxy end up
>> sharinh the same JMS connection too with the connection factory transiently
>> allocating a number of threads capable of utilizing that connection, as per
>> the traces I posted earlier. I'm sure this can be solved somehow and I'd
>> love to know how, but in meantime I've updated the server endpoint code to
>> do 'local' control threads, which works quite well...
>>
>> Cheers, Sergey
>>
>>
>> ----- Original Message -----
>> From: "Sergey Beryozkin" <se...@progress.com>
>> To: <us...@cxf.apache.org>
>> Sent: Monday, November 17, 2008 12:30 PM
>> Subject: Re: JMS queues and multiple consumer threads
>>
>> > Looking at the traces again, I think the problem might be to do with the
>> > spring connection factory
>> >
>> > For ex, this a trace for a failing test with 2 threads involved
>> >
>> >>> 14000 [pool-1-thread-1] INFO
>> >>> org.springframework.jms.connection.SingleConnectionFactory -
>> >>> Established shared JMS Connection:
>> >
>> > and here's the trace for a green test :
>> >> 14594 [pool-1-thread-2] INFO
>> >> org.springframework.jms.connection.SingleConnectionFactory - Established
>> >> shared JMS Connection:
>> >
>> > So I guess what happens is that if in my test the thread which expects
>> > the notification from the other thread squeezes in first, then it
>> > actually is never awaken (it does hit the application code though), while
>> > the other one is waiting on the client side.
>> >
>> > So I guess the question is how to configure a client side connection
>> > factory's thread pool ?
>> >
>> > Thanks, Sergey
>> >
>> >> Hi
>> >>
>> >> I have a system JMS test which involves multiple consumer threads
>> >> hitting a single provider endpoint which is a JMS queue and expecting
>> >> some response.
>> >>
>> >> If more than 1 consumer thread is involved than I can often see a server
>> >> side being unable even to get a given invocation to the endpoint
>> >> application code.
>> >>
>> >> For ex, with 2 threads involved, here's a failed test :
>> >>
>> >> // S.B - this is happening on the consumer side I believe :
>> >>
>> >> 14000 [pool-1-thread-1] INFO
>> >> org.springframework.jms.connection.SingleConnectionFactory - Established
>> >> shared JMS Connection: ActiveMQConnection
>> >> {id=ID:sberyoz-4729-1226923006536-1:1,clientId=null,started=false}
>> >>
>> >> // S.B : I added these printlns :
>> >> Receieve timeout******** : 500000
>> >> Receieve timeout******** : 500000
>> >> //S.B This is a server side now
>> >> 17-Nov-2008 11:56:48 org.apache.cxf.transport.jms.JMSDestination
>> >> onMessage FINE: server received request:
>> >> 17-Nov-2008 11:56:48 org.apache.cxf.transport.jms.JMSDestination
>> >> onMessage FINE: The Request Message is [ [B@1465ca2]
>> >>
>> >> and this is basically, the println in an application code is not even
>> >> hit.
>> >>
>> >> Now, same test with 2 threads succeeding :
>> >>
>> >> // S.B. : client side
>> >>
>> >> 14594 [pool-1-thread-2] INFO
>> >> org.springframework.jms.connection.SingleConnectionFactory - Established
>> >> shared JMS Connection: ActiveMQConnection
>> >> {id=ID:sberyoz-4775-1226923455594-1:1,clientId=null,started=false}
>> >> Receieve timeout******** : 500000
>> >> Receieve timeout******** : 500000
>> >> // S.B. The first thread has managed to get through
>> >> 7-Nov-2008 12:04:17 org.apache.cxf.transport.jms.JMSDestination
>> >> onMessage INFO: The Request Message is [ [B@148c02f]
>> >> Getting continuation for Fred
>> >> Suspending invocation for Fred
>> >> Request message has been suspended
>> >> ....
>> >>
>> >> // S.B the second one is lucky too
>> >> 17-Nov-2008 12:04:17 org.apache.cxf.transport.jms.JMSDestination
>> >> onMessage INFO: The Request Message is [ [B@115470e]
>> >> Invocation for Fred has been suspended
>> >>
>> >> Can someone help me to figure why I'm seeing this transient behaviour ?
>> >>
>> >> here's a wsdl configuration (which I copied from the other test's wsdl)
>> >> shared by all the consumer threads and by the provider endpoint :
>> >>
>> >> <service name="HelloContinuationService">
>> >> <port binding="tns:HelloContinuationServiceSoapBinding"
>> >> name="HelloContinuationPort"> <jms:clientConfig
>> >> clientReceiveTimeout="500000" messageTimeToLive="500000"/>
>> >> <jms:serverConfig messageTimeToLive="500000"/>
>> >> <jms:address
>> >> jndiConnectionFactoryName="ConnectionFactory"
>> >> jndiDestinationName="dynamicQueues/test.jmstransport.text">
>> >> <jms:JMSNamingProperty name="java.naming.factory.initial"
>> >> value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/>
>> >> <jms:JMSNamingProperty name="java.naming.provider.url"
>> >> value="tcp://localhost:61500"/> </jms:address>
>> >> <jms:server durableSubscriberName="CXF_subscriber"/>
>> >> </port>
>> >> </service>
>> >>
>> >> Thanks, Sergey
>
>
>
> --
> Daniel Kulp
> dkulp@apache.org
> http://dankulp.com/blog
>
Re: JMS queues and multiple consumer threads
Posted by Daniel Kulp <dk...@apache.org>.
Sergey,
See:
http://cwiki.apache.org/CXF20DOC/using-the-jmsconfigfeature.html
There is a setting for concurrentConsumers and maxConcurrentConsumers.
Dan
On Monday 17 November 2008 9:35:44 am Sergey Beryozkin wrote:
> Now that I believe I understand the reason for these transient failuires,
> it was very easy to update the test to have the control threads spawned by
> the actual server endpoint code.
> In other words, my JMS continuations test has been originally written the
> same way the HTTP one has been done. Control threads invoke on a service
> endpoint and wait till they're told that the main user threads have invoked
> on say helloService.sayHi() and have called cont.suspend(), so what they do
> next is call helloService.resume() which results in a given suspended
> continuation be resumed.
>
> For JMS this model simply does not walk for me at the moment due to the
> fact that multiple consumer threads sharing the same client proxy end up
> sharinh the same JMS connection too with the connection factory transiently
> allocating a number of threads capable of utilizing that connection, as per
> the traces I posted earlier. I'm sure this can be solved somehow and I'd
> love to know how, but in meantime I've updated the server endpoint code to
> do 'local' control threads, which works quite well...
>
> Cheers, Sergey
>
>
> ----- Original Message -----
> From: "Sergey Beryozkin" <se...@progress.com>
> To: <us...@cxf.apache.org>
> Sent: Monday, November 17, 2008 12:30 PM
> Subject: Re: JMS queues and multiple consumer threads
>
> > Looking at the traces again, I think the problem might be to do with the
> > spring connection factory
> >
> > For ex, this a trace for a failing test with 2 threads involved
> >
> >>> 14000 [pool-1-thread-1] INFO
> >>> org.springframework.jms.connection.SingleConnectionFactory -
> >>> Established shared JMS Connection:
> >
> > and here's the trace for a green test :
> >> 14594 [pool-1-thread-2] INFO
> >> org.springframework.jms.connection.SingleConnectionFactory - Established
> >> shared JMS Connection:
> >
> > So I guess what happens is that if in my test the thread which expects
> > the notification from the other thread squeezes in first, then it
> > actually is never awaken (it does hit the application code though), while
> > the other one is waiting on the client side.
> >
> > So I guess the question is how to configure a client side connection
> > factory's thread pool ?
> >
> > Thanks, Sergey
> >
> >> Hi
> >>
> >> I have a system JMS test which involves multiple consumer threads
> >> hitting a single provider endpoint which is a JMS queue and expecting
> >> some response.
> >>
> >> If more than 1 consumer thread is involved than I can often see a server
> >> side being unable even to get a given invocation to the endpoint
> >> application code.
> >>
> >> For ex, with 2 threads involved, here's a failed test :
> >>
> >> // S.B - this is happening on the consumer side I believe :
> >>
> >> 14000 [pool-1-thread-1] INFO
> >> org.springframework.jms.connection.SingleConnectionFactory - Established
> >> shared JMS Connection: ActiveMQConnection
> >> {id=ID:sberyoz-4729-1226923006536-1:1,clientId=null,started=false}
> >>
> >> // S.B : I added these printlns :
> >> Receieve timeout******** : 500000
> >> Receieve timeout******** : 500000
> >> //S.B This is a server side now
> >> 17-Nov-2008 11:56:48 org.apache.cxf.transport.jms.JMSDestination
> >> onMessage FINE: server received request:
> >> 17-Nov-2008 11:56:48 org.apache.cxf.transport.jms.JMSDestination
> >> onMessage FINE: The Request Message is [ [B@1465ca2]
> >>
> >> and this is basically, the println in an application code is not even
> >> hit.
> >>
> >> Now, same test with 2 threads succeeding :
> >>
> >> // S.B. : client side
> >>
> >> 14594 [pool-1-thread-2] INFO
> >> org.springframework.jms.connection.SingleConnectionFactory - Established
> >> shared JMS Connection: ActiveMQConnection
> >> {id=ID:sberyoz-4775-1226923455594-1:1,clientId=null,started=false}
> >> Receieve timeout******** : 500000
> >> Receieve timeout******** : 500000
> >> // S.B. The first thread has managed to get through
> >> 7-Nov-2008 12:04:17 org.apache.cxf.transport.jms.JMSDestination
> >> onMessage INFO: The Request Message is [ [B@148c02f]
> >> Getting continuation for Fred
> >> Suspending invocation for Fred
> >> Request message has been suspended
> >> ....
> >>
> >> // S.B the second one is lucky too
> >> 17-Nov-2008 12:04:17 org.apache.cxf.transport.jms.JMSDestination
> >> onMessage INFO: The Request Message is [ [B@115470e]
> >> Invocation for Fred has been suspended
> >>
> >> Can someone help me to figure why I'm seeing this transient behaviour ?
> >>
> >> here's a wsdl configuration (which I copied from the other test's wsdl)
> >> shared by all the consumer threads and by the provider endpoint :
> >>
> >> <service name="HelloContinuationService">
> >> <port binding="tns:HelloContinuationServiceSoapBinding"
> >> name="HelloContinuationPort"> <jms:clientConfig
> >> clientReceiveTimeout="500000" messageTimeToLive="500000"/>
> >> <jms:serverConfig messageTimeToLive="500000"/>
> >> <jms:address
> >> jndiConnectionFactoryName="ConnectionFactory"
> >> jndiDestinationName="dynamicQueues/test.jmstransport.text">
> >> <jms:JMSNamingProperty name="java.naming.factory.initial"
> >> value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/>
> >> <jms:JMSNamingProperty name="java.naming.provider.url"
> >> value="tcp://localhost:61500"/> </jms:address>
> >> <jms:server durableSubscriberName="CXF_subscriber"/>
> >> </port>
> >> </service>
> >>
> >> Thanks, Sergey
--
Daniel Kulp
dkulp@apache.org
http://dankulp.com/blog
Re: JMS queues and multiple consumer threads
Posted by Sergey Beryozkin <se...@progress.com>.
Now that I believe I understand the reason for these transient failuires, it was very easy to update the test to have the control
threads spawned by the actual server endpoint code.
In other words, my JMS continuations test has been originally written the same way the HTTP one has been done. Control threads
invoke on a service endpoint and wait till they're told that the main user threads have invoked on say helloService.sayHi() and have
called cont.suspend(), so what they do next is call helloService.resume() which results in a given suspended continuation be
resumed.
For JMS this model simply does not walk for me at the moment due to the fact that multiple consumer threads sharing the same client
proxy end up sharinh the same JMS connection too with the connection factory transiently allocating a number of threads capable of
utilizing that connection, as per the traces I posted earlier. I'm sure this can be solved somehow and I'd love to know how, but in
meantime I've updated the server endpoint code to do 'local' control threads, which works quite well...
Cheers, Sergey
----- Original Message -----
From: "Sergey Beryozkin" <se...@progress.com>
To: <us...@cxf.apache.org>
Sent: Monday, November 17, 2008 12:30 PM
Subject: Re: JMS queues and multiple consumer threads
> Looking at the traces again, I think the problem might be to do with the spring connection factory
>
> For ex, this a trace for a failing test with 2 threads involved
>
>>> 14000 [pool-1-thread-1] INFO org.springframework.jms.connection.SingleConnectionFactory - Established shared JMS Connection:
>
> and here's the trace for a green test :
>
>> 14594 [pool-1-thread-2] INFO org.springframework.jms.connection.SingleConnectionFactory - Established shared JMS Connection:
>
> So I guess what happens is that if in my test the thread which expects the notification from the other thread squeezes in first,
> then it actually is never awaken (it does hit the application code though), while the other one is waiting on the client side.
>
> So I guess the question is how to configure a client side connection factory's thread pool ?
>
> Thanks, Sergey
>
>
>> Hi
>>
>> I have a system JMS test which involves multiple consumer threads hitting a single provider endpoint which is a JMS queue and
>> expecting some response.
>>
>> If more than 1 consumer thread is involved than I can often see a server side being unable even to get a given invocation to the
>> endpoint application code.
>>
>> For ex, with 2 threads involved, here's a failed test :
>>
>> // S.B - this is happening on the consumer side I believe :
>>
>> 14000 [pool-1-thread-1] INFO org.springframework.jms.connection.SingleConnectionFactory - Established shared JMS Connection:
>> ActiveMQConnection {id=ID:sberyoz-4729-1226923006536-1:1,clientId=null,started=false}
>>
>> // S.B : I added these printlns :
>> Receieve timeout******** : 500000
>> Receieve timeout******** : 500000
>> //S.B This is a server side now
>> 17-Nov-2008 11:56:48 org.apache.cxf.transport.jms.JMSDestination onMessage
>> FINE: server received request:
>> 17-Nov-2008 11:56:48 org.apache.cxf.transport.jms.JMSDestination onMessage
>> FINE: The Request Message is [ [B@1465ca2]
>>
>> and this is basically, the println in an application code is not even hit.
>>
>> Now, same test with 2 threads succeeding :
>>
>> // S.B. : client side
>>
>> 14594 [pool-1-thread-2] INFO org.springframework.jms.connection.SingleConnectionFactory - Established shared JMS Connection:
>> ActiveMQConnection {id=ID:sberyoz-4775-1226923455594-1:1,clientId=null,started=false}
>> Receieve timeout******** : 500000
>> Receieve timeout******** : 500000
>> // S.B. The first thread has managed to get through
>> 7-Nov-2008 12:04:17 org.apache.cxf.transport.jms.JMSDestination onMessage
>> INFO: The Request Message is [ [B@148c02f]
>> Getting continuation for Fred
>> Suspending invocation for Fred
>> Request message has been suspended
>> ....
>>
>> // S.B the second one is lucky too
>> 17-Nov-2008 12:04:17 org.apache.cxf.transport.jms.JMSDestination onMessage
>> INFO: The Request Message is [ [B@115470e]
>> Invocation for Fred has been suspended
>>
>> Can someone help me to figure why I'm seeing this transient behaviour ?
>>
>> here's a wsdl configuration (which I copied from the other test's wsdl) shared by all the consumer threads and by the provider
>> endpoint :
>>
>> <service name="HelloContinuationService">
>> <port binding="tns:HelloContinuationServiceSoapBinding" name="HelloContinuationPort">
>> <jms:clientConfig clientReceiveTimeout="500000" messageTimeToLive="500000"/>
>> <jms:serverConfig messageTimeToLive="500000"/>
>> <jms:address
>> jndiConnectionFactoryName="ConnectionFactory"
>> jndiDestinationName="dynamicQueues/test.jmstransport.text">
>> <jms:JMSNamingProperty name="java.naming.factory.initial" value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/>
>> <jms:JMSNamingProperty name="java.naming.provider.url" value="tcp://localhost:61500"/>
>> </jms:address>
>> <jms:server durableSubscriberName="CXF_subscriber"/>
>> </port>
>> </service>
>>
>> Thanks, Sergey
>>
>>
>
>
>
Re: JMS queues and multiple consumer threads
Posted by Sergey Beryozkin <se...@progress.com>.
Looking at the traces again, I think the problem might be to do with the spring connection factory
For ex, this a trace for a failing test with 2 threads involved
>> 14000 [pool-1-thread-1] INFO org.springframework.jms.connection.SingleConnectionFactory - Established shared JMS Connection:
and here's the trace for a green test :
> 14594 [pool-1-thread-2] INFO org.springframework.jms.connection.SingleConnectionFactory - Established shared JMS Connection:
So I guess what happens is that if in my test the thread which expects the notification from the other thread squeezes in first,
then it actually is never awaken (it does hit the application code though), while the other one is waiting on the client side.
So I guess the question is how to configure a client side connection factory's thread pool ?
Thanks, Sergey
> Hi
>
> I have a system JMS test which involves multiple consumer threads hitting a single provider endpoint which is a JMS queue and
> expecting some response.
>
> If more than 1 consumer thread is involved than I can often see a server side being unable even to get a given invocation to the
> endpoint application code.
>
> For ex, with 2 threads involved, here's a failed test :
>
> // S.B - this is happening on the consumer side I believe :
>
> 14000 [pool-1-thread-1] INFO org.springframework.jms.connection.SingleConnectionFactory - Established shared JMS Connection:
> ActiveMQConnection {id=ID:sberyoz-4729-1226923006536-1:1,clientId=null,started=false}
>
> // S.B : I added these printlns :
> Receieve timeout******** : 500000
> Receieve timeout******** : 500000
> //S.B This is a server side now
> 17-Nov-2008 11:56:48 org.apache.cxf.transport.jms.JMSDestination onMessage
> FINE: server received request:
> 17-Nov-2008 11:56:48 org.apache.cxf.transport.jms.JMSDestination onMessage
> FINE: The Request Message is [ [B@1465ca2]
>
> and this is basically, the println in an application code is not even hit.
>
> Now, same test with 2 threads succeeding :
>
> // S.B. : client side
>
> 14594 [pool-1-thread-2] INFO org.springframework.jms.connection.SingleConnectionFactory - Established shared JMS Connection:
> ActiveMQConnection {id=ID:sberyoz-4775-1226923455594-1:1,clientId=null,started=false}
> Receieve timeout******** : 500000
> Receieve timeout******** : 500000
> // S.B. The first thread has managed to get through
> 7-Nov-2008 12:04:17 org.apache.cxf.transport.jms.JMSDestination onMessage
> INFO: The Request Message is [ [B@148c02f]
> Getting continuation for Fred
> Suspending invocation for Fred
> Request message has been suspended
> ....
>
> // S.B the second one is lucky too
> 17-Nov-2008 12:04:17 org.apache.cxf.transport.jms.JMSDestination onMessage
> INFO: The Request Message is [ [B@115470e]
> Invocation for Fred has been suspended
>
> Can someone help me to figure why I'm seeing this transient behaviour ?
>
> here's a wsdl configuration (which I copied from the other test's wsdl) shared by all the consumer threads and by the provider
> endpoint :
>
> <service name="HelloContinuationService">
> <port binding="tns:HelloContinuationServiceSoapBinding" name="HelloContinuationPort">
> <jms:clientConfig clientReceiveTimeout="500000" messageTimeToLive="500000"/>
> <jms:serverConfig messageTimeToLive="500000"/>
> <jms:address
> jndiConnectionFactoryName="ConnectionFactory"
> jndiDestinationName="dynamicQueues/test.jmstransport.text">
> <jms:JMSNamingProperty name="java.naming.factory.initial" value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/>
> <jms:JMSNamingProperty name="java.naming.provider.url" value="tcp://localhost:61500"/>
> </jms:address>
> <jms:server durableSubscriberName="CXF_subscriber"/>
> </port>
> </service>
>
> Thanks, Sergey
>
>