You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Rupesh <mr...@gmail.com> on 2014/01/02 15:20:48 UTC
ActiveMQ consumer count is set to "0"
Hi,We have been using ActiveMQ in production from almost 6 months. There are
four queues/topics in my production queue. We observed sometimes even if
consumer application is up and running, the consumer count for one of the
queue was visible as "0". Unfortunately, we had debug desabled and hence
have no much logs and it is difficult to debug as well. Could you provide
some insight on why this happens and is there any solution apart from
restarting the consumer application as doing this causes a downtime for
other users of the application?I appreciate your response in advance.Thanks
& Regards,Rupesh Sinha
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-consumer-count-is-set-to-0-tp4675930.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ consumer count is set to "0"
Posted by Tim Bain <tb...@alumni.duke.edu>.
I'm not sure what your use case is, so I can't say for sure whether
throwing away a potentially-large number of messages is a valid option for
you. But if it were me, I'd focus on figuring out why your broker can't
handle your required workload and on how you can adjust either the broker
or your use-case to make it work. Maybe the broker needs larger limits
(and the hardware to back them), maybe you need to publish your 40,000
messages to multiple brokers (effectively sharding your destinations so
that a single broker doesn't have to handle the full load), maybe you need
to publish fewer messages, etc. You seem to be taking for granted that 1)
you have to publish 40,000 messages, and 2) the broker's configuration is
the best possible one to accept them, and I'd question those assumptions
before I started looking at discarding messages.
Tim
On Fri, Oct 16, 2015 at 2:00 PM, dhananjay.patkar <
dhananjay.patkar@gmail.com> wrote:
> Setting broker to fail producer in case its out of resources, instead of
> making producer wait forever
>
> <systemUsage>
> <systemUsage sendFailIfNoSpaceAfterTimeout="3000">
> <memoryUsage>
> <.....
> </memoryUsage>
> </systemUsage>
> </systemUsage>
>
> Your views on this?
> Broker machine is very high config vm (56 G ARM out of which 70% for broker
> and has 8 cores )
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-consumer-count-is-set-to-0-tp4675930p4703077.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: ActiveMQ consumer count is set to "0"
Posted by "dhananjay.patkar" <dh...@gmail.com>.
Setting broker to fail producer in case its out of resources, instead of
making producer wait forever
<systemUsage>
<systemUsage sendFailIfNoSpaceAfterTimeout="3000">
<memoryUsage>
<.....
</memoryUsage>
</systemUsage>
</systemUsage>
Your views on this?
Broker machine is very high config vm (56 G ARM out of which 70% for broker
and has 8 cores )
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-consumer-count-is-set-to-0-tp4675930p4703077.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ consumer count is set to "0"
Posted by Tim Bain <tb...@alumni.duke.edu>.
It really sounds like you're simply overwhelming the broker with those
40,000 messages you're publishing. If the broker couldn't keep up with the
volume of messages you were sending (or worse, if you ran it into a limit
and the broker stopped accepting messages until something started consuming
them), that would be consistent with the behavior you're seeing. When this
is happening, is the broker or any of its destinations running into any
limits (including available space on disk if applicable)? I know you said
"broker resources are within limit", but does that apply to the queue to
which you're publishing as well? Is the broker burning all the CPU cores
on the machine on which it's running? Is the broker responsive to activity
on other destinations and via web console and JMX?
The fact that you lost messages while using async sends seems like the same
fundamental problem: the broker can't keep up with what you're asking it to
do, for one reason or another.
If you switch to sync sends (just for testing) and enable Producer Flow
Control, does the behavior change?
On Thu, Oct 15, 2015 at 7:13 AM, dhananjay.patkar <
dhananjay.patkar@gmail.com> wrote:
> Yes all the messages are getting published to the same broker.
>
> Producer window size was introduced to avoid message loss, previously we
> used to lose ~10-20% of messages as we use async send. Usually we get huge
> number of messages in short time (message burst).
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-consumer-count-is-set-to-0-tp4675930p4702994.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: ActiveMQ consumer count is set to "0"
Posted by "dhananjay.patkar" <dh...@gmail.com>.
Yes all the messages are getting published to the same broker.
Producer window size was introduced to avoid message loss, previously we
used to lose ~10-20% of messages as we use async send. Usually we get huge
number of messages in short time (message burst).
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-consumer-count-is-set-to-0-tp4675930p4702994.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ consumer count is set to "0"
Posted by Tim Bain <tb...@alumni.duke.edu>.
Step #2 looks like your problem, based on that thread dump. Are you
publishing back to the same broker, or another one? Whichever one it is,
is it just overloaded trying to accept 40000 messages all at once? Are
those messages actually getting to that broker, and at what rate?
Why is a producer window being used? To what end?
Tim
On Oct 15, 2015 6:54 AM, "dhananjay.patkar" <dh...@gmail.com>
wrote:
> No, the broker logs are clear and I dont see any warnings in broker logs.
> *I am worried as, even broker resources are within limit this waiting
> thread
> never returns causing this consumer to die*.
>
> My workflow is
>
> Queue -> Camel route jms consumer (uses connectionfactory without
> producer window size as its consumer)
> 1. Bean Processor to split the
> message into multiple (~40k) messages
> 2. Send ~40k messages using spring
> jmstemplate (uses connectionfactory with producer window)
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-consumer-count-is-set-to-0-tp4675930p4702980.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: ActiveMQ consumer count is set to "0"
Posted by "dhananjay.patkar" <dh...@gmail.com>.
No, the broker logs are clear and I dont see any warnings in broker logs.
*I am worried as, even broker resources are within limit this waiting thread
never returns causing this consumer to die*.
My workflow is
Queue -> Camel route jms consumer (uses connectionfactory without
producer window size as its consumer)
1. Bean Processor to split the
message into multiple (~40k) messages
2. Send ~40k messages using spring
jmstemplate (uses connectionfactory with producer window)
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-consumer-count-is-set-to-0-tp4675930p4702980.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ consumer count is set to "0"
Posted by Tim Bain <tb...@alumni.duke.edu>.
Um, that looks like Producer Flow Control kicking in, which would result in
warnings in the broker's logs. Is that what you're seeing?
On Oct 14, 2015 11:47 AM, "dhananjay.patkar" <dh...@gmail.com>
wrote:
> I was able to capture a thread dump on the jvm on which consumers are
> vanishing.
> The threads are blocked on producer window, waitForSpace().
>
>
> "Camel (CamelContext) thread #573 - JmsConsumer[RequestQueue]" daemon
> prio=6
> tid=0x0000000011ded000 nid=0x156c in Object.wait() [0x000000005b69d000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.activemq.usage.MemoryUsage.waitForSpace(MemoryUsage.java:67)
> - locked <0x00000006a0bfc058> (a java.lang.Object)
> at
>
> org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:271)
> at
>
> org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:212)
> at
> org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:79)
> - locked <0x00000006a0bfbec8> (a
> org.apache.activemq.ActiveMQMessageProducer)
> at
> org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:67)
> at
> org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:589)
> at
> org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:569)
> at
> org.springframework.jms.core.JmsTemplate$4.doInJms(JmsTemplate.java:546)
> at
> org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:466)
> at
> org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:543)
> at
>
> org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:620)
> at
>
> com.gehcit.haf.mu.messaging.impl.jms.JmsCalcJobMessageProducerImpl.sendMessage(JmsCalcJobMessageProducerImpl.java:125)
> at
>
> com.gehcit.haf.mu.service.impl.MeasureCalculationServiceImpl.queueCqmCalcJobRequestMessage(MeasureCalculationServiceImpl.java:557)
> at
>
> com.gehcit.haf.mu.service.impl.MeasureCalculationServiceImpl.calculateMeasureResultsMultiFromQueue(MeasureCalculationServiceImpl.java:312)
> at sun.reflect.GeneratedMethodAccessor508.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.camel.component.bean.MethodInfo.invoke(MethodInfo.java:341)
> at
> org.apache.camel.component.bean.MethodInfo$1.proceed(MethodInfo.java:238)
> at
>
> org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:166)
> at
>
> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:99)
> at
>
> org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:73)
> at
>
> org.apache.camel.impl.ProcessorEndpoint.onExchange(ProcessorEndpoint.java:101)
> at
>
> org.apache.camel.impl.ProcessorEndpoint$1.process(ProcessorEndpoint.java:71)
> at
>
> org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
> at
>
> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
> at
>
> org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:122)
> at
>
> org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:298)
> at
> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:117)
> at
>
> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
> at
>
> org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
> at
>
> org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
> at
>
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:73)
> at
>
> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
> at
>
> org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
> at
>
> org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
> at
>
> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91)
> at
>
> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
> at
>
> org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:334)
> at
>
> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:220)
> at
>
> org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
> at
>
> org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
> at
>
> org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:303)
> at
>
> org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
> at
>
> org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
> at
>
> org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:150)
> at
>
> org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:117)
> at
>
> org.apache.camel.processor.RouteInflightRepositoryProcessor.processNext(RouteInflightRepositoryProcessor.java:48)
> at
>
> org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
> at
>
> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
> at
>
> org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
> at
>
> org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
> at
>
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:73)
> at
>
> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:99)
> at
>
> org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:86)
> at
>
> org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:104)
> at
>
> org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:562)
> at
>
> org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:500)
> at
>
> org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:468)
> at
>
> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:326)
> at
>
> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:264)
> at
>
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1071)
> at
>
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1063)
> at
>
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:960)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
>
>
> This wait never finishes, even if memory usage returns to normal.
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-consumer-count-is-set-to-0-tp4675930p4702955.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: ActiveMQ consumer count is set to "0"
Posted by "dhananjay.patkar" <dh...@gmail.com>.
I was able to capture a thread dump on the jvm on which consumers are
vanishing.
The threads are blocked on producer window, waitForSpace().
"Camel (CamelContext) thread #573 - JmsConsumer[RequestQueue]" daemon prio=6
tid=0x0000000011ded000 nid=0x156c in Object.wait() [0x000000005b69d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.activemq.usage.MemoryUsage.waitForSpace(MemoryUsage.java:67)
- locked <0x00000006a0bfc058> (a java.lang.Object)
at
org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:271)
at
org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:212)
at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:79)
- locked <0x00000006a0bfbec8> (a
org.apache.activemq.ActiveMQMessageProducer)
at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:67)
at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:589)
at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:569)
at org.springframework.jms.core.JmsTemplate$4.doInJms(JmsTemplate.java:546)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:466)
at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:543)
at
org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:620)
at
com.gehcit.haf.mu.messaging.impl.jms.JmsCalcJobMessageProducerImpl.sendMessage(JmsCalcJobMessageProducerImpl.java:125)
at
com.gehcit.haf.mu.service.impl.MeasureCalculationServiceImpl.queueCqmCalcJobRequestMessage(MeasureCalculationServiceImpl.java:557)
at
com.gehcit.haf.mu.service.impl.MeasureCalculationServiceImpl.calculateMeasureResultsMultiFromQueue(MeasureCalculationServiceImpl.java:312)
at sun.reflect.GeneratedMethodAccessor508.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.camel.component.bean.MethodInfo.invoke(MethodInfo.java:341)
at
org.apache.camel.component.bean.MethodInfo$1.proceed(MethodInfo.java:238)
at
org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:166)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:99)
at
org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:73)
at
org.apache.camel.impl.ProcessorEndpoint.onExchange(ProcessorEndpoint.java:101)
at
org.apache.camel.impl.ProcessorEndpoint$1.process(ProcessorEndpoint.java:71)
at
org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:122)
at
org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:298)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:117)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:73)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:334)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:220)
at
org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:303)
at
org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:150)
at
org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:117)
at
org.apache.camel.processor.RouteInflightRepositoryProcessor.processNext(RouteInflightRepositoryProcessor.java:48)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)
at
org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:73)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:99)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:86)
at
org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:104)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:562)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:500)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:468)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:326)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:264)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1071)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1063)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:960)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
This wait never finishes, even if memory usage returns to normal.
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-consumer-count-is-set-to-0-tp4675930p4702955.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ consumer count is set to "0"
Posted by Tim Bain <tb...@alumni.duke.edu>.
So you're using ActiveMQ, Camel, and Spring, and the problem (whether it's
your error or a bug) could be in any of them. I've got to say, given that
this isn't clearly a problem with any one of those technologies, I'd post
this to StackOverflow and tag it with all three technologies and hope that
someone from the right technology who knows what might cause this sees it.
My instinct tells me this isn't an ActiveMQ issue (I'd suspect Camel if I
was forced to pick one since they're managing the customers, though if
consumer threads are hung that's probably a Spring problem), but it really
could be any of the three (and we're all probably going to suspect the
problem lies with the others; sorry, I know it's not fun when people say
"not our problem, check with the other guys").
For your other questions:
- If you're using asyncSend and the broker restarts, I believe that the
already-sent messages will be lost.
- I'd be very surprised if this had to do with maxInactivityInterval,
though anything's possible. But 5 minutes seems like a long time to wait
before declaring a connection dead...
- If the default recoveryInterval of 5 seconds isn't preventing the
disappearance of consumers, I'm not sure how a different value would change
that behavior.
- Does this only happen when you restart the broker? Or does it also
happen when all processes are up and connected?
Tim
On Fri, Oct 9, 2015 at 8:16 AM, dhananjay.patkar <dhananjay.patkar@gmail.com
> wrote:
> I am able to reproduce this issue by restarting the broker, when camel
> consumers are sending those millions of messages on the queue.
>
> I am planning to set recoveryInterval on the camel JmsConfiguration, please
> post your views on this.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-consumer-count-is-set-to-0-tp4675930p4702810.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: ActiveMQ consumer count is set to "0"
Posted by "dhananjay.patkar" <dh...@gmail.com>.
I am able to reproduce this issue by restarting the broker, when camel
consumers are sending those millions of messages on the queue.
I am planning to set recoveryInterval on the camel JmsConfiguration, please
post your views on this.
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-consumer-count-is-set-to-0-tp4675930p4702810.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ consumer count is set to "0"
Posted by "dhananjay.patkar" <dh...@gmail.com>.
We are using ActiveMQ 5.10 broker with following configuration
JVM heap 48G out of which 70% allocated for memoryUsage
70G storage and 1G for temp
we are using Apache camel 2.10.2 route to consume messages from queue. After
receiving message, we create ~40k smaller messages/ received message and
publish them onto another queue using asyncSend using spring's jmstemplate.
We have configured different connectionFactory for consumer and producer
which work on pooledconnection
As we generate millions of messages onto the queue, we have configured
producerWindowSize on the producerConnectionFactory
Camel Route
<route id="Request-Route" autoStartup="true">
<from uri="jmsQueue:queue:RequestQueue" />
<to uri="bean:service?method=divideAndPublishToQueue"/>
</route>
Problem: After few hours of running we generate ~10 million messages onto
the broker and consumers on the queue starts disappearing. Eventually there
are no consumers and our system comes to halt. The only work around we have
as of now is to restart the application
Question:
1. What will be the behavior of asyncSend producer with producerWindoeSize,
in case broker restarts?
2. We have maxInactiveInterval set to 300000 in the activemq.xml
3. When I inspect the logs I see the consumer starts publishing to the queue
using jmsTemplate and there are no logs which says it finished. There are
several such logs indicating consumer threads most probably in hung state.
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-consumer-count-is-set-to-0-tp4675930p4702798.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ consumer count is set to "0"
Posted by Tim Bain <tb...@alumni.duke.edu>.
To clarify, I mean that you should post the actual Camel routes you're
using.
On Oct 8, 2015 1:48 PM, "Tim Bain" <tb...@alumni.duke.edu> wrote:
> If you're hoping for help from anyone other than the OP, you'll want to
> provide the same information about your clients that Art asked the OP for.
> On Oct 8, 2015 8:35 AM, "dhananjay.patkar" <dh...@gmail.com>
> wrote:
>
>> We are experiencing same issue, with activemq 5.10 broker and consumers
>> are
>> created through camel routes.
>>
>> Are you able to conclude on your problem, please share the if you have
>> fixed
>> it.
>>
>> Thanks,
>> Dhananjay
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/ActiveMQ-consumer-count-is-set-to-0-tp4675930p4702754.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
Re: ActiveMQ consumer count is set to "0"
Posted by Tim Bain <tb...@alumni.duke.edu>.
If you're hoping for help from anyone other than the OP, you'll want to
provide the same information about your clients that Art asked the OP for.
On Oct 8, 2015 8:35 AM, "dhananjay.patkar" <dh...@gmail.com>
wrote:
> We are experiencing same issue, with activemq 5.10 broker and consumers are
> created through camel routes.
>
> Are you able to conclude on your problem, please share the if you have
> fixed
> it.
>
> Thanks,
> Dhananjay
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-consumer-count-is-set-to-0-tp4675930p4702754.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: ActiveMQ consumer count is set to "0"
Posted by "dhananjay.patkar" <dh...@gmail.com>.
We are experiencing same issue, with activemq 5.10 broker and consumers are
created through camel routes.
Are you able to conclude on your problem, please share the if you have fixed
it.
Thanks,
Dhananjay
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-consumer-count-is-set-to-0-tp4675930p4702754.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ consumer count is set to "0"
Posted by artnaseef <ar...@artnaseef.com>.
Typically the failover transport is used by applications which will make sure
clients reconnect once the broker restarts, and will survive across multiple
broker restarts.
Is the failover transport being used?
If you can post the connection factory configuration for the application,
that could help.
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-consumer-count-is-set-to-0-tp4675930p4676386.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ consumer count is set to "0"
Posted by Rupesh <mr...@gmail.com>.
Hi,
The consumer is created when the application is deployed. In my case, the
ActiveMQ went down due to DB connectivity issues and had to be restarted.
Once I restarted ActiveMQ, I observed that, out of 4 queues, 3 queues had
correct consumer count but one did not. I had to restart the application
server to reestablish the connection to the queue and then it all started
working. In some occasions, I have observed that none of the queue gets a
consumer count after restating the MQ and it becomes mandatory to restart
the application.
Please advise if I can avoid the application restart to resolve this issue.
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-consumer-count-is-set-to-0-tp4675930p4676362.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ consumer count is set to "0"
Posted by artnaseef <ar...@artnaseef.com>.
Do you have any logs for the consumer? Without knowing the internals of the
solution, there are a lot of possibilities.
For example, does the consumer get created immediately at application
startup time, or after some event occurs?
But first, besides the consumer count reading as 0, are there any other
indications of a problem?
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-consumer-count-is-set-to-0-tp4675930p4676014.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.