You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@geronimo.apache.org by vineetc <vi...@gmail.com> on 2007/06/20 19:49:56 UTC
Problems with AMQ RA under Geronimo MDB container
Hello All Geronimo Users and Developers,
I am trying to build a MDB based recv. application that will process
messages delivered by AMQ RA.
ENVIRONMENT
Windows XP
JDK 1.6.0_01
ActiveMQ 4.1.1 and the the RA provided by the dist.
Geronimo1.1.1
MySQL 5x
I have set the "maxSessions" to 29 and "maxMessagesPerSession" to 900 on the
ActivationSpec. for the MDB. These properties are made available by AMQ for
performance tuning. MaxSession corresponds to the number of AMQ
ServerSession's or MessageEndpoint (listeners) i believe and
"maxMessagesPerSession" is the number of messages kept in RAM.
Scenario:
I send a bunch of messages while the EAR is not running.
After the messages have been sent (say 1000 messages) I start the EAR
application.
The EAR application seems to hang and or deadlock after processing 29
messages.
The JEECA's connector Thread Pool is configured with a size of 30 threads
and the property "waitWhenBlocked" is set to "true".
This WorkManager thread pool is provided by Geronimo server.
After looking at the threads from JConsole i saw following stack trace(s)
for Connector Threads which are stuck.
The first thread from connector thread pool had following stack trace.
Name: ConnectorThreadPool 0
Stack trace:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar.await(CondVar.java:75)
edu.emory.mathcs.backport.java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:318)
org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:42)
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:75)
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1175)
org.apache.activemq.ActiveMQConnectionConsumer.<init>(ActiveMQConnectionConsumer.java:86)
org.apache.activemq.ActiveMQConnection.createConnectionConsumer(ActiveMQConnection.java:1085)
org.apache.activemq.ra.ActiveMQEndpointWorker$1.run(ActiveMQEndpointWorker.java:164)
- locked org.apache.activemq.ra.ActiveMQEndpointWorker$1@4b755
org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:291)
org.apache.geronimo.pool.ThreadPool$1.run(ThreadPool.java:172)
org.apache.geronimo.pool.ThreadPool$ContextClassLoaderRunnable.run(ThreadPool.java:289)
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
java.lang.Thread.run(Thread.java:619)
All of the rest 29 threads from the connector thread pool have the same
stack trace as below.
Name: ConnectorThreadPool 29
State: WAITING on
edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar@4dd658
Total blocked: 156 Total waited: 1
Stack trace:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar.await(CondVar.java:75)
edu.emory.mathcs.backport.java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:318)
org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:42)
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:75)
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1175)
org.apache.activemq.TransactionContext.setXid(TransactionContext.java:578)
org.apache.activemq.TransactionContext.end(TransactionContext.java:329)
org.apache.activemq.ra.LocalAndXATransaction.end(LocalAndXATransaction.java:90)
org.apache.geronimo.transaction.manager.WrapperNamedXAResource.end(WrapperNamedXAResource.java:51)
org.apache.geronimo.transaction.manager.TransactionImpl.delistResource(TransactionImpl.java:209)
- locked org.apache.geronimo.transaction.manager.TransactionImpl@1221045
org.apache.geronimo.transaction.context.InheritableTransactionContext.delistResource(InheritableTransactionContext.java:99)
org.openejb.mdb.EndpointHandler.teardownDelivery(EndpointHandler.java:360)
org.openejb.mdb.EndpointHandler.afterDelivery(EndpointHandler.java:256)
org.openejb.mdb.EndpointProxy.afterDelivery(EndpointProxy.java:72)
org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:126)
org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:65)
org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:216)
org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:751)
org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:291)
org.apache.geronimo.pool.ThreadPool$1.run(ThreadPool.java:172)
org.apache.geronimo.pool.ThreadPool$ContextClassLoaderRunnable.run(ThreadPool.java:289)
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
java.lang.Thread.run(Thread.java:619)
Any idea whats causing the problem?
Any help is appreciated.
Thanks,
Vineet
--
View this message in context: http://www.nabble.com/Problems-with-AMQ-RA-under-Geronimo-MDB-container-tf3954028s134.html#a11218765
Sent from the Apache Geronimo - Users mailing list archive at Nabble.com.
Re: Problems with AMQ RA under Geronimo MDB container
Posted by vineetc <vi...@gmail.com>.
Well i did a small test of 1000 messages and it worked.
Heres what i did.
I sent a 1000 messages without the EAR running.
I configured the ThreadPool on RA to have 60 threads instead of 30.
So effectively making it twice the number of "maxSessions" value under AMQ.
I did not factor in "maxMessagesPerSession" value which to me looks like is
used to maintain a buffer of messages in RAM.
I am not too sure if 900 x 2 or 900 x 2 x 30 is the right number
But with the change i made things worked perfectly.
I also observed from JConsole that the Thread Pool never created more than
33 Threads.
Which makes me feel like it just needs some extra threads and not
necessarily 2ice.
But then i could be wrong.
I want to do more extensive testing with different configurations.
Will post my results on this thread once done.
Thanks a bunch for the observation.
I was blocked on this one just like a THREAD...lol and really didnt have
much time at hand to get into the Geronimo or AMQ code.
Vineet
On Jun 20, 2007, at 2:22 PM, vineetc wrote:
>
> I can try that.
> BTW what makes you feel it needs twice the Threads as number of
> Sessions?
It looks to me as if all the threads came out of the pool and are
trying to do amq work and are blocked looking for an additional
thread. It's entirely possible we have misunderstood the effect of
the amq settings and it really needs 900 X 2 or 900 X 29 X 2
threads.... If you make the settings lower do messages start flowing?
I'm just guessing :-(
thanks
david jencks
--
View this message in context: http://www.nabble.com/Problems-with-AMQ-RA-under-Geronimo-MDB-container-tf3954028s134.html#a11219866
Sent from the Apache Geronimo - Users mailing list archive at Nabble.com.
Re: Problems with AMQ RA under Geronimo MDB container
Posted by David Jencks <da...@yahoo.com>.
On Jun 20, 2007, at 2:22 PM, vineetc wrote:
>
> I can try that.
> BTW what makes you feel it needs twice the Threads as number of
> Sessions?
It looks to me as if all the threads came out of the pool and are
trying to do amq work and are blocked looking for an additional
thread. It's entirely possible we have misunderstood the effect of
the amq settings and it really needs 900 X 2 or 900 X 29 X 2
threads.... If you make the settings lower do messages start flowing?
I'm just guessing :-(
thanks
david jencks
> Just curious to know.
>
> Will get back with the results soon...:)
>
> Thanks,
> Vineet
> --
> View this message in context: http://www.nabble.com/Problems-with-
> AMQ-RA-under-Geronimo-MDB-container-tf3954028s134.html#a11219383
> Sent from the Apache Geronimo - Users mailing list archive at
> Nabble.com.
>
Re: Problems with AMQ RA under Geronimo MDB container
Posted by vineetc <vi...@gmail.com>.
I can try that.
BTW what makes you feel it needs twice the Threads as number of Sessions?
Just curious to know.
Will get back with the results soon...:)
Thanks,
Vineet
--
View this message in context: http://www.nabble.com/Problems-with-AMQ-RA-under-Geronimo-MDB-container-tf3954028s134.html#a11219383
Sent from the Apache Geronimo - Users mailing list archive at Nabble.com.
Re: Problems with AMQ RA under Geronimo MDB container
Posted by David Jencks <da...@yahoo.com>.
What happens if the thread pool is say 3 X larger than the number of
message endpoints/sessions? The code looks a little bit like it
needs twice as many threads as message endpoints (???)
Do any messages get through?
just guessing...
thanks
david jencks
On Jun 20, 2007, at 1:49 PM, vineetc wrote:
>
> Hello All Geronimo Users and Developers,
>
> I am trying to build a MDB based recv. application that will process
> messages delivered by AMQ RA.
>
> ENVIRONMENT
> Windows XP
> JDK 1.6.0_01
> ActiveMQ 4.1.1 and the the RA provided by the dist.
> Geronimo1.1.1
> MySQL 5x
>
> I have set the "maxSessions" to 29 and "maxMessagesPerSession" to
> 900 on the
> ActivationSpec. for the MDB. These properties are made available by
> AMQ for
> performance tuning. MaxSession corresponds to the number of AMQ
> ServerSession's or MessageEndpoint (listeners) i believe and
> "maxMessagesPerSession" is the number of messages kept in RAM.
>
> Scenario:
> I send a bunch of messages while the EAR is not running.
> After the messages have been sent (say 1000 messages) I start the EAR
> application.
> The EAR application seems to hang and or deadlock after processing 29
> messages.
>
> The JEECA's connector Thread Pool is configured with a size of 30
> threads
> and the property "waitWhenBlocked" is set to "true".
> This WorkManager thread pool is provided by Geronimo server.
> After looking at the threads from JConsole i saw following stack
> trace(s)
> for Connector Threads which are stuck.
>
> The first thread from connector thread pool had following stack trace.
>
> Name: ConnectorThreadPool 0
> Stack trace:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar.await
> (CondVar.java:75)
> edu.emory.mathcs.backport.java.util.concurrent.ArrayBlockingQueue.take
> (ArrayBlockingQueue.java:318)
> org.apache.activemq.transport.FutureResponse.getResult
> (FutureResponse.java:42)
> org.apache.activemq.transport.ResponseCorrelator.request
> (ResponseCorrelator.java:75)
> org.apache.activemq.ActiveMQConnection.syncSendPacket
> (ActiveMQConnection.java:1175)
> org.apache.activemq.ActiveMQConnectionConsumer.<init>
> (ActiveMQConnectionConsumer.java:86)
> org.apache.activemq.ActiveMQConnection.createConnectionConsumer
> (ActiveMQConnection.java:1085)
> org.apache.activemq.ra.ActiveMQEndpointWorker$1.run
> (ActiveMQEndpointWorker.java:164)
> - locked org.apache.activemq.ra.ActiveMQEndpointWorker$1@4b755
> org.apache.geronimo.connector.work.WorkerContext.run
> (WorkerContext.java:291)
> org.apache.geronimo.pool.ThreadPool$1.run(ThreadPool.java:172)
> org.apache.geronimo.pool.ThreadPool$ContextClassLoaderRunnable.run
> (ThreadPool.java:289)
> EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
> Source)
> java.lang.Thread.run(Thread.java:619)
>
>
> All of the rest 29 threads from the connector thread pool have the
> same
> stack trace as below.
>
> Name: ConnectorThreadPool 29
> State: WAITING on
> edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar@4dd658
> Total blocked: 156 Total waited: 1
>
> Stack trace:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar.await
> (CondVar.java:75)
> edu.emory.mathcs.backport.java.util.concurrent.ArrayBlockingQueue.take
> (ArrayBlockingQueue.java:318)
> org.apache.activemq.transport.FutureResponse.getResult
> (FutureResponse.java:42)
> org.apache.activemq.transport.ResponseCorrelator.request
> (ResponseCorrelator.java:75)
> org.apache.activemq.ActiveMQConnection.syncSendPacket
> (ActiveMQConnection.java:1175)
> org.apache.activemq.TransactionContext.setXid
> (TransactionContext.java:578)
> org.apache.activemq.TransactionContext.end(TransactionContext.java:
> 329)
> org.apache.activemq.ra.LocalAndXATransaction.end
> (LocalAndXATransaction.java:90)
> org.apache.geronimo.transaction.manager.WrapperNamedXAResource.end
> (WrapperNamedXAResource.java:51)
> org.apache.geronimo.transaction.manager.TransactionImpl.delistResource
> (TransactionImpl.java:209)
> - locked
> org.apache.geronimo.transaction.manager.TransactionImpl@1221045
> org.apache.geronimo.transaction.context.InheritableTransactionContext.
> delistResource(InheritableTransactionContext.java:99)
> org.openejb.mdb.EndpointHandler.teardownDelivery
> (EndpointHandler.java:360)
> org.openejb.mdb.EndpointHandler.afterDelivery(EndpointHandler.java:
> 256)
> org.openejb.mdb.EndpointProxy.afterDelivery(EndpointProxy.java:72)
> org.apache.activemq.ra.MessageEndpointProxy
> $MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:126)
> org.apache.activemq.ra.MessageEndpointProxy.afterDelivery
> (MessageEndpointProxy.java:65)
> org.apache.activemq.ra.ServerSessionImpl.afterDelivery
> (ServerSessionImpl.java:216)
> org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:751)
> org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:
> 165)
> org.apache.geronimo.connector.work.WorkerContext.run
> (WorkerContext.java:291)
> org.apache.geronimo.pool.ThreadPool$1.run(ThreadPool.java:172)
> org.apache.geronimo.pool.ThreadPool$ContextClassLoaderRunnable.run
> (ThreadPool.java:289)
> EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
> Source)
> java.lang.Thread.run(Thread.java:619)
>
>
> Any idea whats causing the problem?
> Any help is appreciated.
>
> Thanks,
> Vineet
> --
> View this message in context: http://www.nabble.com/Problems-with-
> AMQ-RA-under-Geronimo-MDB-container-tf3954028s134.html#a11218765
> Sent from the Apache Geronimo - Users mailing list archive at
> Nabble.com.
>