You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by ashoknan <as...@chase.com> on 2009/03/18 20:31:47 UTC
ActiveMQ producer/consumer hangs when on of the consumer didn't
respond
We are using ActiveMQ 5.1 & Spring 2.5.4...
We are using ActiveMQ Embeeded broker
<bean id="jmsFactory"
class="org.apache.activemq.pool.PooledConnectionFactory"
destroy-method="stop">
<property name="connectionFactory">
<bean class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL">
<value>vm://localhost?broker.persistent=false</value>
</property>
</bean>
</property>
</bean>
<bean id="destination" class=" org.apache.activemq.command.ActiveMQQueue">
<constructor-arg ref="jmsQueueJndiName"></constructor-arg>
</bean>
<bean id="calendarEventContainer"
class="org.springframework.jms.listener.SimpleMessageListenerContainer" >
<property name="connectionFactory" ref="jmsFactory" />
<property name="destination" ref="destination" />
<property name="messageListener" ref="calendarEventProcessor" />
<property name="messageSelector">
<util:constant
static-field="com.jpmc.spex.utils.lookup.EventCategoryConstants.EVENT_CATEGORY_CALENDAR"/>
</property>
</bean>
<bean id="runtimeEventContainer"
class="org.springframework.jms.listener.SimpleMessageListenerContainer" >
<property name="connectionFactory" ref="jmsFactory" />
<property name="destination" ref="destination" />
<property name="messageListener" ref="runtimeEventProcessor" />
<property name="messageSelector">
<util:constant
static-field="com.jpmc.spex.utils.lookup.EventCategoryConstants.EVENT_CATEGORY_RUNTIME"/>
</property>
</bean>
<bean id="systemEventContainer"
class="org.springframework.jms.listener.SimpleMessageListenerContainer" >
<property name="connectionFactory" ref="jmsFactory" />
<property name="destination" ref="destination" />
<property name="messageListener" ref="systemEventProcessor" />
<property name="concurrentConsumers" value="5" />
<property name="messageSelector">
<util:constant
static-field="com.jpmc.spex.utils.lookup.EventCategoryConstants.EVENT_CATEGORY_SYSTEM"/>
</property>
</bean>
<bean id="genericCalendarEventContainer"
class="org.springframework.jms.listener.SimpleMessageListenerContainer" >
<property name="connectionFactory" ref="jmsFactory" />
<property name="destination" ref="destination" />
<property name="messageListener" ref="genericCalendarEventProcessor" />
<property name="messageSelector">
<util:constant
static-field="com.jpmc.spex.utils.lookup.EventCategoryConstants.EVENT_CATEGORY_GENERIC"/>
</property>
</bean>
We are sending the email from all these listeners after doing some
logic.when something happens in the SMTP side and if there is no response
from SMTP side and if the Listener is waiting then the entire
producer/consumer is halted.
My Question is why One Consumer blocking all other consumer /producer. (see
the stack trace which shows one of the Consumer is waiting for some response
from SMTP server).
ActiveMQ Session Task" - Thread t@2554
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.sun.mail.util.TraceInputStream.read(TraceInputStream.java:97)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
locked java.io.BufferedInputStream@13a80ea
at com.sun.mail.util.LineInputStream.readLine(LineInputStream.java:75)
at
com.sun.mail.smtp.SMTPTransport.readServerResponse(SMTPTransport.java:1440)
at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1260)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:370)
at javax.mail.Service.connect(Service.java:275)
at
org.springframework.mail.javamail.JavaMailSenderImpl.doSend(JavaMailSenderImpl.java:389)
at
org.springframework.mail.javamail.JavaMailSenderImpl.send(JavaMailSenderImpl.java:342)
at
org.springframework.mail.javamail.JavaMailSenderImpl.send(JavaMailSenderImpl.java:338)
at
com.jpmc.ibtech.spring.email.MimeEmailService.sendEmail(MimeEmailService.java:61)
at
com.jpmc.spex.background.events.processor.AbstractEventAction.sendEmail(AbstractEventAction.java:63)
at
com.jpmc.spex.background.events.processor.AbstractEventAction.execute(AbstractEventAction.java:79)
at
com.jpmc.spex.background.events.processor.system.SystemEventAction.execute(SystemEventAction.java:47)
at
com.jpmc.ibtech.spring.jms.EventProcessor.onMessage(EventProcessor.java:101)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:531)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:466)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:435)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:407)
at
org.springframework.jms.listener.SimpleMessageListenerContainer.processMessage(SimpleMessageListenerContainer.java:290)
at
org.springframework.jms.listener.SimpleMessageListenerContainer$2.onMessage(SimpleMessageListenerContainer.java:266)
at
org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:967)
locked java.lang.Object@1e58347
at
org.apache.activemq.ActiveMQSessionExecutor.dispatch(ActiveMQSessionExecutor.java:122)
at
org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:192)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
I was wondering why this is blocking other concurrent consumer for the same
selector.
locked java.util.concurrent.locks.ReentrantLock$NonfairSync@10dbec7
See the attached ThreadDump which shows Producer is waiting..
I saw that..I can enable ProduceFlowControl...I am just worried whether it
will start spilling to the disk and create more problems.
I am not sure whether setting SMTP timeout will avoid the consumer waiting
indefinitely.
http://www.nabble.com/file/p22586789/Thread%252BDump%252Bfor%252BActiveMQ_2.tdump.txt
Thread%2BDump%2Bfor%2BActiveMQ_2.tdump.txt
--
View this message in context: http://www.nabble.com/ActiveMQ-producer-consumer-hangs-when-on-of-the-consumer-didn%27t-respond-tp22586789p22586789.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ producer/consumer hangs when on of the consumer didn't
respond
Posted by ashoknan <as...@chase.com>.
Thanks Gary for your response..We have a prefetch of
10..jms.prefetchPolicy.all=10.
The thing which i am not sure ,why when one of the Consumer got the stuck in
the smtp response other consumer for the same selector is stuck.(I have 5
Conusmer for the same selector). Not only the consumer got stuck the entire
producer/consumer is stuck(Attachment has thread stack). See the queue stat
when the entire producer/consumer is stuck.
queueSize=20548
memoryPercentUsage=100
memoryUsagePortion=1.0
memoryLimit=67108864
enqueueCount=400720
dequeueCount=380172
inFlightCount=10
maxEnqueueTime=227302
minEnqueueTime=1
averageEnqueueTime=32840.18067399298
consumerCount=8
producerCount=0
isProducerFlowControl=true
Is it because that particular consumer is holding the shared connection?
Also i am thinking whether setting the SMTP timeout will avoid the thread
getting stuck forever and it may not block the connection.
Does it make sense?
Gary Tully wrote:
>
> Prefetch <http://activemq.apache.org/what-is-the-prefetch-limit-for.html>
> is
> the problem with pooling consumers. A consumers pre-fetched messages,
> when
> that consumer is in a pool, may stay there till the pool is closed and
> will
> not be available to other consumers.
>
> 2009/3/20 ashoknan <as...@chase.com>
>
>>
>> Am i not supposed to use PooledConnectionFactory for consuming
>> messages.See
>> the below Java Doc
>>
>> http://activemq.apache.org/maven/act...onFactory.html
>>
>>
>> A JMS provider which pools Connection, Session and MessageProducer
>> instances
>> so it can be used with tools like Spring's JmsTemplate. NOTE this
>> implementation is only intended for use when sending messages. It does
>> not
>> deal with pooling of consumers; for that look at a library like Jencks
>> such
>> as in this example
>> I am not sure whether this is outdated.
>>
>> Please let me know if anyone faced this issue before
>> --
>> View this message in context:
>> http://www.nabble.com/ActiveMQ-producer-consumer-hangs-when-on-of-the-consumer-didn%27t-respond-tp22586789p22614510.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> http://blog.garytully.com
>
> Open Source SOA
> http://FUSESource.com
>
>
--
View this message in context: http://www.nabble.com/ActiveMQ-producer-consumer-hangs-when-on-of-the-consumer-didn%27t-respond-tp22586789p22621769.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ producer/consumer hangs when on of the consumer didn't
respond
Posted by Gary Tully <ga...@gmail.com>.
Prefetch <http://activemq.apache.org/what-is-the-prefetch-limit-for.html> is
the problem with pooling consumers. A consumers pre-fetched messages, when
that consumer is in a pool, may stay there till the pool is closed and will
not be available to other consumers.
2009/3/20 ashoknan <as...@chase.com>
>
> Am i not supposed to use PooledConnectionFactory for consuming messages.See
> the below Java Doc
>
> http://activemq.apache.org/maven/act...onFactory.html
>
>
> A JMS provider which pools Connection, Session and MessageProducer
> instances
> so it can be used with tools like Spring's JmsTemplate. NOTE this
> implementation is only intended for use when sending messages. It does not
> deal with pooling of consumers; for that look at a library like Jencks such
> as in this example
> I am not sure whether this is outdated.
>
> Please let me know if anyone faced this issue before
> --
> View this message in context:
> http://www.nabble.com/ActiveMQ-producer-consumer-hangs-when-on-of-the-consumer-didn%27t-respond-tp22586789p22614510.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source SOA
http://FUSESource.com
Re: ActiveMQ producer/consumer hangs when on of the consumer didn't
respond
Posted by ashoknan <as...@chase.com>.
Am i not supposed to use PooledConnectionFactory for consuming messages.See
the below Java Doc
http://activemq.apache.org/maven/act...onFactory.html
A JMS provider which pools Connection, Session and MessageProducer instances
so it can be used with tools like Spring's JmsTemplate. NOTE this
implementation is only intended for use when sending messages. It does not
deal with pooling of consumers; for that look at a library like Jencks such
as in this example
I am not sure whether this is outdated.
Please let me know if anyone faced this issue before
--
View this message in context: http://www.nabble.com/ActiveMQ-producer-consumer-hangs-when-on-of-the-consumer-didn%27t-respond-tp22586789p22614510.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.