Activemq-Artemis cpu increases endlessly in kubernetes

[Lavi-Feivel Blumberg <lb...@novus.com.INVALID>]

Hello,

Background:
We are currently running our application using a very old version of hornetq. As part of our migration to Kubernetes, we wanted to try to use activemq-artemis as a drop-in replacement for this old broker with some newer monitoring features. Our application is not very big in terms of the number of messages/consumers. We are usually only generating maybe tens of messages a second at peak and we have less than 50 consumers in our qa environment. Most of the time we actually aren’t even sending any messages.

Problem:
We have given artemis roughly the same cpu/memory we have on hornet but we are finding that over time the cpu just keeps increasing until it hits the limit we give it in Kubernetes. At that point the service continues to work but can be a bit spotty. However, we completely lose the web console as well as the jmx exporter so we have no visibility/monitoring.

Attached is the graph of our kubenetes metrics for the pod. Note the constantly increasing cpu and that we lose jmx exporter (heap) metrics once the cpu becomes too high.


Any help would be greatly appreciated.

Best,

Lavi-Feivel Blumberg
Novus

Re: Activemq-Artemis cpu increases endlessly in kubernetes

[aarner <aa...@gmail.com>]

While I haven't run artemis in kubernetes or another container, I have seen a
similar phenomenon in certain situations.  What can you tell us about the
average size of your messages, and can you post details about your
queue/topic configuration.  On occasion when I have artemis processing
either a large number (millions) or messages, or some few hundred very large
messages and also made changes to the memory configuration and/or
global-max-size, I've seen artemis run away with the CPU in one thread and
begin responding very poorly.  It seems to be especially acute if you start
an artemis instance with an unbounded global-max-size and less memory than
the size of your in-flight messages in the journal.  My experience is all on
windows VMs, using the journaling file system so may not be relevant, but
the situation you're describing did ring a bell.  Please post those config
details and I'd be glad to review them.

-a




--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: Activemq-Artemis cpu increases endlessly in kubernetes

[AntonR <an...@volvo.com>]

I think you missed the attached graph, but one thing to try out might be to
disable these new monitoring features just to rule out these as the reason
for increased CPU load over time. Just disable them and take some manual
measurements over time to see if the effect is still taking place or not.

Br,
Anton



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html