Re: Messages are stuck in ActiveMQ 5.11 and delivered for after more than 24 hours

[sravan <sr...@gmail.com>]

Unfortunately we are in a predicament where we have issues in reproducing the
problem in performance test environments as well as debugging. ActiveMQ INFO
logs does not contain much information to understand what's going on. When
we turn on DEBUG logging (in lower env) ActiveMQ is hanging just in few
minutes and never comes back. So far the only one clue we are relying on is,
while monitoring ActiveMQ consoles in our prod environment, we observed
messages stuck on network bridge for multiple hours. Also in our research,
we know that consumers are doing fine and the main issue is with ActiveMQ
which is delivering messages very late and we simply don't know the root
cause. When we worked with an ActiveMQ consultant a year ago, we were
advised to scale ActiveMQs  vertically  rather than scaling them
horizontally. i.e. Have 2 larger AMQ nodes rather than having 4 smaller
nodes in a cluster. We were told that this avoids many potential anomalies
with network brdige and shipping messages across the nodes etc...Does anyone
here have any suggestions with respect to how else we could debug or fix our
issue. Again just to refresh your minds, the main issue we’ve been
struggling with is...Our AMQs are delivering messages extremely late,
sometimes a day or even two days later. We do not see any JMS exceptions,
and no exceptions or issues at consumer's end. Any inputs are greatly
appreciated.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-are-stuck-in-ActiveMQ-5-11-and-delivered-for-after-more-than-24-hours-tp4727694p4728468.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Messages are stuck in ActiveMQ 5.11 and delivered for after more than 24 hours

[Tim Bain <tb...@alumni.duke.edu>]

I repeat my earlier suggestion that you should use a sampler on all brokers
to characterize where the time is being spent.

The fact that turning logging to debug causes the broker to experience a
slowdown almost immediately makes it seem like maybe there's an issue with
disk I/O or space, but that's just a guess and using a sampler will give
you something more concrete than a guess.

Also, you've characterized the problem as messages getting "stuck" in the
network connectors. When this happens, are individual messages truly stuck
(i.e. no messages are being passed), or is it simply that the rate they're
flowing out at os lower than the rate they're flowing in at (so there's a
net backup but individual messages are still being passed)? And when
messages are passed, do they arrive in order, or do they show up vastly
different from the order in which they were sent?

Tim

On Jul 11, 2017 4:29 PM, "sravan" <sr...@gmail.com> wrote:

> Unfortunately we are in a predicament where we have issues in reproducing
> the
> problem in performance test environments as well as debugging. ActiveMQ
> INFO
> logs does not contain much information to understand what's going on. When
> we turn on DEBUG logging (in lower env) ActiveMQ is hanging just in few
> minutes and never comes back. So far the only one clue we are relying on
> is,
> while monitoring ActiveMQ consoles in our prod environment, we observed
> messages stuck on network bridge for multiple hours. Also in our research,
> we know that consumers are doing fine and the main issue is with ActiveMQ
> which is delivering messages very late and we simply don't know the root
> cause. When we worked with an ActiveMQ consultant a year ago, we were
> advised to scale ActiveMQs  vertically  rather than scaling them
> horizontally. i.e. Have 2 larger AMQ nodes rather than having 4 smaller
> nodes in a cluster. We were told that this avoids many potential anomalies
> with network brdige and shipping messages across the nodes etc...Does
> anyone
> here have any suggestions with respect to how else we could debug or fix
> our
> issue. Again just to refresh your minds, the main issue we’ve been
> struggling with is...Our AMQs are delivering messages extremely late,
> sometimes a day or even two days later. We do not see any JMS exceptions,
> and no exceptions or issues at consumer's end. Any inputs are greatly
> appreciated.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Messages-are-stuck-in-ActiveMQ-5-11-and-
> delivered-for-after-more-than-24-hours-tp4727694p4728468.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>