You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by drunkKnight <up...@gmail.com> on 2019/03/26 06:39:29 UTC
Disptach Queue filled up and consumer not consuming
Hello,
I have set up a project that uses ActiveMQ for messaging. My configurations
are the following:
Producer Side:
producerSession: transacted= false and Acknowledgment =
CLIENT_ACKNOWLEDGEMENT
Consumer Side:
setQueuePrefetch = 1;
Whenever there is an event like generate OTP for a phone number. the
producer generates a message with required fields, adding the phone number
as JMSXGroupId and posts it to the queue. After this, the producer gets
offline until a new event generation is required. There are multiple types
of events and our user base is around 15k.
Now, on the consumer side, I have 10 consumers that use receive() method
without any timeout mentioned. Now this message is received and converted to
the respective event and passed into a Service that consumes this message.
There could be three responses from the service:
1. The event is executed successfully and returns true. In this case, I
invoke the *consumer.acknowledge()* method.
2. The event execution is not successful and returns false. In this case, I
again invoke the *consumer.acknowledge() *method and post a delayed message
(AMQ_Scheduled_Delay as 1 hour) to the queue.
3. While executing the event, it threw an exception. Now this exception, if
a JMSException then I reestablish the consumer connections to the broker, if
not then, I explicitly *acknowledge* the message and post a delayed message
after 6 hours.
Now the issue I'm facing is that the dispatch queue gets filled up with
value as 2 for most of the consumers (We have around 25K events a day
normally). And out of these 10, only 2 or 3 consumers with dispatch queue
count as 0 are present. I could not figure out what is the issue. Could
anyone please help me as I need to restart the server twice a day so that
the consumers start consuming?
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Disptach Queue filled up and consumer not consuming
Posted by drunkKnight <up...@gmail.com>.
Hi Tim,
Thanks for replying!
My question is why the dispatch queue getting filled up even when I'm
explicitly acknowledging a message. I have 10 consumers that listen, consume
and do everything that they are intended for. Around midnight, when the
events posting count increases, the queue starts getting filled up and I
need to restart the server in the morning and then everything gets consumed.
*Regarding the message group link*, I'm not closing any consumer. All 10
consumers are alive throughout the period when the server is up.
*Regarding your statement*, "So I'd expect that when you're down to only 2-3
consumers, all of the message groups would be assigned to those consumers,
and they'd be getting messages dispatched to them", I checked and found out
that the most of the pending message's JMSXGroupId is same as the consumers
that had already consumed messages with same JMSXGroupId. So these messages
are not being forwarded to other consumers and wait to be delivered to those
consumers only.
*Regarding your statement*, "Did you tweak anything that would affect the
broker's ability to detect a disconnected consumer, such as TCP timeout
settings, the inactivity checking within ActiveMQ, etc.", the only change I
did in configuration file was to add schedulerSupport = "true" for broker.
Also for receiving a message, the consumers are using *receive()* method
without any parameter.
*Regarding your statement*And is there anything unusual in the broker or
consumer logs that might shed any light onto what's going on?", the last log
for the cons1 is "INFO [2019-03-27 05:56:17,448]
com.tinkerix.talkmore.core.ActiveMQConsumer: retrieving a new message from
queue for consumer ID:ip-XX-X-X-XXX-XXXXX-1553602691234-1:10:1:1". (Time is
in UTC and this reply posting time is 2019-03-27 06:45:06). After this log
statement, I have a call consumer.receive() method and that's where the
consumer is stuck. The broker is not directing any message to consumer as
the disptach queue is at 1. The attached images and the above analysis are
of the same period.
*Regarding the details of configuration: *
Apache ActiveMQ 5.15.8
OpenWire protocol
Not checking inactivity of consumers. Earlier, I used to put consumers on
sleep when I was using *receive(1000L)* method, but I discarded that and the
sleep code both.
I tested that if the connection is lost while receiving a message then
JMSException is thrown so I handle that and reestablish the connection for
that consumer. But this is not happening i.e. the connection failure (I
checked cloudwatch logs).
I had spawned 10 threads that set up the connection with the broker using
the same configuration.
I am using AmazonMQ as broker service support for ActiveMQ.
7-8 consumers are still connected. Please refer the attachments. I tried
setting the prefetch as ZERO and the attached images are with the prefetch
set as zero. On setting prefetch as zero, the dispatch queue is again
filling up with upper bound value 1.
To create a consumer, I have the following code:
ActiveMQConnectionFactory activeMQConnectionFactory =
activeMQ.getActiveMQConnectionFactory();
MessageConsumer consumer =
getConfiguredConsumerConnection(activeMQConnectionFactory, queueName);
The prefetch value is set in getConfiguredConsumerConnection method. It
looks like
getConfiguredConsumerConnection() {
ActiveMQPrefetchPolicy activeMQPrefetchPolicy = new
ActiveMQPrefetchPolicy();
activeMQPrefetchPolicy.setQueuePrefetch(0);
connectionFactory.setPrefetchPolicy(activeMQPrefetchPolicy);
//createConnection
//startConnection
consumerConnection.createSession(false,
Session.CLIENT_ACKNOWLEDGE); //session creation
//createConsumer and return MessageConsumer object;
}
For connection, I'm using AmazonMQ's OpenWire Endpoint.
ConsumerFilledUpAfterPrefetchSetTo0.jpg
<http://activemq.2283324.n4.nabble.com/file/t379351/ConsumerFilledUpAfterPrefetchSetTo0.jpg>
QueueSizeIncresingImage.jpg
<http://activemq.2283324.n4.nabble.com/file/t379351/QueueSizeIncresingImage.jpg>
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Disptach Queue filled up and consumer not consuming
Posted by drunkKnight <up...@gmail.com>.
Hello,
This issue is resolved. I checked the thread dump and found out that the
consumer threads were in WAITING state.
Just briefing my error here so that if anyone with similar case hit this,
then, they can do a broader-level of debugging at their own. We had a
SlackAppenderFactory registered with Logger, and so whenever the logging was
above DEBUG (logging levels: INFO/DEBUG/WARN/ERROR, we had the threshold set
to DEBUG), we were reporting it to SLACK. Since the user events were
increasing and we were on an unpaid version, there was a throttling
exception thrown by SLACK that we were not anticipating (This code was
written around 2 years back and we never hit it).
Apart from IOException, every other error/exception would again be routed to
slack and thus the LOGGER was keeping a lock and no consumer could access it
(since we have LOGGER statement in almost every method). This resulted in
consumer-threads in WAITING state.
I wanted to report this earlier but got too busy so posting the abstract.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Disptach Queue filled up and consumer not consuming
Posted by drunkKnight <up...@gmail.com>.
Hi Tim,
Thanks for replying!
My question is why the dispatch queue getting filled up even when I'm
explicitly acknowledging a message. I have 10 consumers that listen, consume
and do everything that they are intended for. Around midnight, when the
events posting count increases, the queue starts getting filled up and I
need to restart the server in the morning and then everything gets consumed.
Regarding the message group link, I'm not closing any consumer. All 10
consumers are alive throughout the period when the server is up.
Regarding your statement, "So I'd expect that when you're down to only 2-3
consumers, all of the message groups would be assigned to those consumers,
and they'd be getting messages dispatched to them", I checked and found out
that the most of the pending message's JMSXGroupId is same as the consumers
that had already consumed messages with same JMSXGroupId. So these messages
are not being forwarded to other consumers and wait to be delivered to those
consumers only.
Regarding your statement, "Did you tweak anything that would affect the
broker's ability to detect a disconnected consumer, such as TCP timeout
settings, the inactivity checking within ActiveMQ, etc.", the only change I
did in configuration file was to add schedulerSupport = "true" for broker.
Also for receiving a message, the consumers are using receive() method
without any parameter.
Regarding your statementAnd is there anything unusual in the broker or
consumer logs that might shed any light onto what's going on?", the last log
for the cons1 is "INFO [2019-03-27 05:56:17,448]
com.tinkerix.talkmore.core.ActiveMQConsumer: retrieving a new message from
queue for consumer ID:ip-XX-X-X-XXX-XXXXX-1553602691234-1:10:1:1". (Time is
in UTC and this reply posting time is 2019-03-27 06:45:06). After this log
statement, I have a call consumer.receive() method and that's where the
consumer is stuck. The broker is not directing any message to consumer as
the disptach queue is at 1. The attached images and the above analysis are
of the same period.
Regarding the details of configuration:
Apache ActiveMQ 5.15.8
OpenWire protocol
Not checking inactivity of consumers. Earlier, I used to put consumers on
sleep when I was using receive(1000L) method, but I discarded that and the
sleep code both.
I tested that if the connection is lost while receiving a message then
JMSException is thrown so I handle that and reestablish the connection for
that consumer. But this is not happening i.e. the connection failure (I
checked cloudwatch logs).
I had spawned 10 threads that set up the connection with the broker using
the same configuration.
I am using AmazonMQ as broker service support for ActiveMQ.
7-8 consumers are still connected. Please refer the attachments. I tried
setting the prefetch as ZERO and the attached images are with the prefetch
set as zero. On setting prefetch as zero, the dispatch queue is again
filling up with upper bound value 1.
To create a consumer, I have the following code:
ActiveMQConnectionFactory activeMQConnectionFactory =
activeMQ.getActiveMQConnectionFactory();
MessageConsumer consumer =
getConfiguredConsumerConnection(activeMQConnectionFactory, queueName);
The prefetch value is set in getConfiguredConsumerConnection method. It
looks like
getConfiguredConsumerConnection() {
ActiveMQPrefetchPolicy activeMQPrefetchPolicy = new
ActiveMQPrefetchPolicy();
activeMQPrefetchPolicy.setQueuePrefetch(0);
connectionFactory.setPrefetchPolicy(activeMQPrefetchPolicy);
//createConnection
//startConnection
consumerConnection.createSession(false,
Session.CLIENT_ACKNOWLEDGE); //session creation
//createConsumer and return MessageConsumer object;
}
For connection, I'm using AmazonMQ's OpenWire Endpoint.
QueueSizeIncresingImage.jpg
<http://activemq.2283324.n4.nabble.com/file/t379351/QueueSizeIncresingImage.jpg>
ConsumerFilledUpAfterPrefetchSetTo0.jpg
<http://activemq.2283324.n4.nabble.com/file/t379351/ConsumerFilledUpAfterPrefetchSetTo0.jpg>
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Disptach Queue filled up and consumer not consuming
Posted by drunkKnight <up...@gmail.com>.
Hi Tim,
Thanks for replying!
My question is why the dispatch queue getting filled up even when I'm
explicitly acknowledging a message. I have 10 consumers that listen, consume
and do everything that they are intended for. Around midnight, when the
events posting count increases, the queue starts getting filled up and I
need to restart the server in the morning and then everything gets consumed.
Regarding the message group link, I'm not closing any consumer. All 10
consumers are alive throughout the period when the server is up.
Regarding your statement, "So I'd expect that when you're down to only 2-3
consumers, all of the message groups would be assigned to those consumers,
and they'd be getting messages dispatched to them", I checked and found out
that the most of the pending message's JMSXGroupId is same as the consumers
that had already consumed messages with same JMSXGroupId. So these messages
are not being forwarded to other consumers and wait to be delivered to those
consumers only.
Regarding your statement, "Did you tweak anything that would affect the
broker's ability to detect a disconnected consumer, such as TCP timeout
settings, the inactivity checking within ActiveMQ, etc.", the only change I
did in configuration file was to add schedulerSupport = "true" for broker.
Also for receiving a message, the consumers are using receive() method
without any parameter.
Regarding your statementAnd is there anything unusual in the broker or
consumer logs that might shed any light onto what's going on?", the last log
for the cons1 is "INFO [2019-03-27 05:56:17,448]
com.tinkerix.talkmore.core.ActiveMQConsumer: retrieving a new message from
queue for consumer ID:ip-XX-X-X-XXX-XXXXX-1553602691234-1:10:1:1". (Time is
in UTC and this reply posting time is 2019-03-27 06:45:06). After this log
statement, I have a call consumer.receive() method and that's where the
consumer is stuck. The broker is not directing any message to consumer as
the disptach queue is at 1. The attached images and the above analysis are
of the same period.
Regarding the details of configuration:
Apache ActiveMQ 5.15.8
OpenWire protocol
Not checking inactivity of consumers. Earlier, I used to put consumers on
sleep when I was using receive(1000L) method, but I discarded that and the
sleep code both.
I tested that if the connection is lost while receiving a message then
JMSException is thrown so I handle that and reestablish the connection for
that consumer. But this is not happening i.e. the connection failure (I
checked cloudwatch logs).
I had spawned 10 threads that set up the connection with the broker using
the same configuration.
I am using AmazonMQ as broker service support for ActiveMQ.
7-8 consumers are still connected. Please refer the attachments. I tried
setting the prefetch as ZERO and the attached images are with the prefetch
set as zero. On setting prefetch as zero, the dispatch queue is again
filling up with upper bound value 1.
To create a consumer, I have the following code:
ActiveMQConnectionFactory activeMQConnectionFactory =
activeMQ.getActiveMQConnectionFactory();
MessageConsumer consumer =
getConfiguredConsumerConnection(activeMQConnectionFactory, queueName);
The prefetch value is set in getConfiguredConsumerConnection method. It
looks like
getConfiguredConsumerConnection() {
ActiveMQPrefetchPolicy activeMQPrefetchPolicy = new
ActiveMQPrefetchPolicy();
activeMQPrefetchPolicy.setQueuePrefetch(0);
connectionFactory.setPrefetchPolicy(activeMQPrefetchPolicy);
//createConnection
//startConnection
consumerConnection.createSession(false,
Session.CLIENT_ACKNOWLEDGE); //session creation
//createConsumer and return MessageConsumer object;
}
For connection, I'm using AmazonMQ's OpenWire Endpoint.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Disptach Queue filled up and consumer not consuming
Posted by Tim Bain <tb...@alumni.duke.edu>.
I think I followed your use case, but I'm not sure I fully understand your
explanation of the problem you're seeing.
So you have 10 consumers, but only 2-3 of them are connected? What's
happening with the other consumers, such that they stop being connected? Is
this the crux of your question, i.e. "Why do my consumers keep dying?"
I also think that I understand that when most of your consumers are
disconnected, the dispatch queue on the broker gets completely full of
messages not for the 2-3 consumers who are connected. Did I understand that
right? If so, this surprises me, since
http://activemq.apache.org/message-groups.html indicates that a new
consumer will be assigned to the message group when "the consumer closes
(or the client which created the consumer dies etc)." So I'd expect that
when you're down to only 2-3 consumers, all of the message groups would be
assigned to those consumers, and they'd be getting messages dispatched to
them. If that's not happening, then maybe the broker's failing to detect
that the consumer is disconnected? Did you tweak anything that would affect
the broker's ability to detect a disconnected consumer, such as TCP timeout
settings, the inactivity checking within ActiveMQ, etc.?
Can you tell us more about your configuration? What version of ActiveMQ,
what transport protocol are you using, how have you configured your
prefetch buffer sizes for your consumers, etc.? And is there anything
unusual in the broker or consumer logs that might shed any light onto
what's going on?
Tim
On Tue, Mar 26, 2019 at 12:39 AM drunkKnight <up...@gmail.com>
wrote:
> Hello,
> I have set up a project that uses ActiveMQ for messaging. My configurations
> are the following:
> Producer Side:
> producerSession: transacted= false and Acknowledgment =
> CLIENT_ACKNOWLEDGEMENT
>
> Consumer Side:
> setQueuePrefetch = 1;
>
> Whenever there is an event like generate OTP for a phone number. the
> producer generates a message with required fields, adding the phone number
> as JMSXGroupId and posts it to the queue. After this, the producer gets
> offline until a new event generation is required. There are multiple types
> of events and our user base is around 15k.
>
> Now, on the consumer side, I have 10 consumers that use receive() method
> without any timeout mentioned. Now this message is received and converted
> to
> the respective event and passed into a Service that consumes this message.
> There could be three responses from the service:
> 1. The event is executed successfully and returns true. In this case, I
> invoke the *consumer.acknowledge()* method.
> 2. The event execution is not successful and returns false. In this case, I
> again invoke the *consumer.acknowledge() *method and post a delayed message
> (AMQ_Scheduled_Delay as 1 hour) to the queue.
> 3. While executing the event, it threw an exception. Now this exception, if
> a JMSException then I reestablish the consumer connections to the broker,
> if
> not then, I explicitly *acknowledge* the message and post a delayed message
> after 6 hours.
>
> Now the issue I'm facing is that the dispatch queue gets filled up with
> value as 2 for most of the consumers (We have around 25K events a day
> normally). And out of these 10, only 2 or 3 consumers with dispatch queue
> count as 0 are present. I could not figure out what is the issue. Could
> anyone please help me as I need to restart the server twice a day so that
> the consumers start consuming?
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>