You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by s....@inform-technology.de on 2025/01/14 08:48:39 UTC
Artemis - Help understanding heap dump
Hello group,
currently I am analyzing an OOM crash of a productive broker (ActiveMQ Artemis 2.28). The broker ran under 64 bit OpenJDK 17.0.9+9 (Temurin) with 4GB of heap configured via -Xmx VM option and G1 garbage collector.
In the analyzer histogram I can see, that there are 5 big consumers of the VM’s heap. Four of those are very much the same:
1. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
2. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
3. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
4. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
The fifth seems to be closely related to these four:
19 instances of java.util.TreeMap$Entry occupy 777.531.504 (18,18 %) bytes.
Biggest instances:
* java.util.TreeMap$Entry @ 0x7584c1ce8 - 231.178.240 (5,41 %) bytes.
* java.util.TreeMap$Entry @ 0x7094185d8 - 220.148.272 (5,15 %) bytes.
* java.util.TreeMap$Entry @ 0x77c22a440 - 116.097.024 (2,71 %) bytes.
* java.util.TreeMap$Entry @ 0x796469d78 - 103.186.432 (2,41 %) bytes.
* java.util.TreeMap$Entry @ 0x7c8c1e078 - 53.215.232 (1,24 %) bytes.
Most of these instances are referenced from one instance of org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1 which occupies 1.282.832 (0,03 %) bytes.
The analyzer states
Common Path To the Accumulation Point:
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1 @ 0x79e40f078 Thread-1 (ActiveMQ-PageExecutor-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$9@4aeaadc1) <mat://object/0x79e40f078> Thread
After a restart of the broker after 6 hours the memory consumption is still at < 300 mb.
Can someone tell me what the cause of this OOM exception might be (excessive paging, fast reconnects, zombie sessions, …)?
Kind regards
Sebastian Götz
Re: AW: AW: Artemis - Help understanding heap dump
Posted by Clebert Suconic <cl...@apache.org>.
I suggest you keep a prefetch lower than max-read.. just to be on a safe side avoiding some false log.warn calls that would happen for too many credits issues from consumers.
On 2025/01/23 06:23:05 s.goetz@inform-technology.de wrote:
> Okay, I think I have understood the difference. As we tend to have very small transactions or immediate commits this will not make much of a difference in our use case.
> Last question:
> The example you showed demonstrates how to starve the client,because of the limit in max-read/prefetch. In a real world application this would be quite hard to figure out, I think. Is there some sort of logging if the server hits any of those limits or the chance to find this state via Artemis Management Web Application?
>
> -----Ursprüngliche Nachricht-----
> Von: Clebert Suconic <cl...@apache.org>
> Gesendet: Mittwoch, 22. Januar 2025 17:32
> An: users@activemq.apache.org
> Betreff: Re: AW: Artemis - Help understanding heap dump
>
> just to make my response a bit clearer: max-read will look into the pending acks, while prefetch will only look at what's currently in the queue.
>
>
> (that's a TL;DR (Too long, didn't read) of my previous messages).
>
> On 2025/01/22 16:29:21 Clebert Suconic wrote:
> > The logic is not complex.. the whole thing is controlled here. The source code will be a very good documentation in this case:
> >
> > https://github.com/apache/activemq-artemis/blob/1ba0b65babf298e4dc4795
> > 1aa6998b2d0ac02be6/artemis-server/src/main/java/org/apache/activemq/ar
> > temis/core/server/impl/QueueImpl.java#L3469-L3493
> >
> > On 2025/01/22 16:25:19 Clebert Suconic wrote:
> > > sure...
> > >
> > >
> > > I originally did not want to implement prefetch.. but my user had a
> > > case where they needed a soft limit and a hard limit
> > >
> > >
> > > At max-read, the system will stop fetching for data... period... no more messages will come.
> > >
> > >
> > > With Prefetch.. if you receive the messages on the consumer, it will still request for more messages, but if you don't ack them.. things will stop at max-read.. there will be a log.warn in the system as you would starve otherwise.
> > >
> > >
> > > Example:
> > >
> > >
> > > Say you configured max-read at 100, and then you have a consumer doing the following:
> > >
> > > Session session = connection.createSession(... whatever you do to
> > > create a transaction section); consumer = session.createConsumer
> > >
> > > for (int I = 0; I < 1000; I++) {
> > > consumer.receive(); // notice I'm blocking here forever if no messages are coming.
> > > }
> > > // The system will never reach here, but max-read=100 and there are 100 messages pending.
> > > session.commit();
> > >
> > > As you take messages out of the queue.. the system will try to keep 10 messages in memory...
> > >
> > >
> > > But as you don't ack these messages the system will block at max-read.
> > >
> > >
> > >
> > > Makes sense?
> > >
> > >
> > > Also there are the bytes versions of these metrics where I use the payload to manage the flow. The system will always reach to whatever happens first on the rules.
> > >
> > > On 2025/01/15 06:32:52 s.goetz@inform-technology.de wrote:
> > > > Hey Clebert,
> > > >
> > > > thank you for the tip about ARTEMIS-4447.
> > > > If I understand correctly, these settings will influence how many messages of a persistent page are loaded into memory.
> > > > This sounds quite likely. We have had similar issues in the past with many consumers that were unable to dispatch the messages to their clients leading to massive paging and OOM.
> > > > As we are still on 2.28 I will recommend the team to migrate to a more recent version and have a look into these address settings.
> > > > But after a first look into those settings, I cannot quite figure out the difference between max-read-page-messages and prefetch-page-messages. Both claim to somehow limit the amount of messages read into memory.
> > > >
> > > > Can you explain the difference?
> > > >
> > > > Many thanks
> > > >
> > > > Sebastian
> > > >
> > > > -----Ursprüngliche Nachricht-----
> > > > Von: Clebert Suconic <cl...@apache.org>
> > > > Gesendet: Dienstag, 14. Januar 2025 18:11
> > > > An: users@activemq.apache.org
> > > > Betreff: Re: Artemis - Help understanding heap dump
> > > >
> > > > as part of ARTEMIS-4447, I added parameters to address-settings for prefetching.
> > > >
> > > > Also there was some flow changes based on an issue I had with an user, as they had consumers prefetching for a long time issuing more messages out of paging into memory.
> > > >
> > > >
> > > > You can now configure prefetch and such conditions would just keep the server from running out of memory.
> > > >
> > > >
> > > > This is the most likely scenario why you're having OME issues as it's similar to what I have dealt with my user.
> > > >
> > > >
> > > >
> > > >
> > > > On 2025/01/14 08:48:39 s.goetz@inform-technology.de wrote:
> > > > > Hello group,
> > > > >
> > > > >
> > > > >
> > > > > currently I am analyzing an OOM crash of a productive broker (ActiveMQ Artemis 2.28). The broker ran under 64 bit OpenJDK 17.0.9+9 (Temurin) with 4GB of heap configured via -Xmx VM option and G1 garbage collector.
> > > > >
> > > > > In the analyzer histogram I can see, that there are 5 big consumers of the VM’s heap. Four of those are very much the same:
> > > > >
> > > > >
> > > > >
> > > > > 1. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> > > > > 2. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> > > > > 3. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> > > > > 4. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> > > > >
> > > > >
> > > > >
> > > > > The fifth seems to be closely related to these four:
> > > > >
> > > > > 19 instances of java.util.TreeMap$Entry occupy 777.531.504 (18,18 %) bytes.
> > > > >
> > > > > Biggest instances:
> > > > >
> > > > > * java.util.TreeMap$Entry @ 0x7584c1ce8 - 231.178.240 (5,41 %) bytes.
> > > > >
> > > > > * java.util.TreeMap$Entry @ 0x7094185d8 - 220.148.272 (5,15 %) bytes.
> > > > >
> > > > > * java.util.TreeMap$Entry @ 0x77c22a440 - 116.097.024 (2,71 %) bytes.
> > > > >
> > > > > * java.util.TreeMap$Entry @ 0x796469d78 - 103.186.432 (2,41 %) bytes.
> > > > >
> > > > > * java.util.TreeMap$Entry @ 0x7c8c1e078 - 53.215.232 (1,24 %) bytes.
> > > > >
> > > > > Most of these instances are referenced from one instance of org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1 which occupies 1.282.832 (0,03 %) bytes.
> > > > >
> > > > >
> > > > >
> > > > > The analyzer states
> > > > >
> > > > > Common Path To the Accumulation Point:
> > > > >
> > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1 @
> > > > > 0x79e40f078 Thread-1
> > > > > (ActiveMQ-PageExecutor-server-org.apache.activemq.artemis.core.server.
> > > > > impl.ActiveMQServerImpl$9@4aeaadc1) <mat://object/0x79e40f078>
> > > > > Thread
> > > > >
> > > > >
> > > > >
> > > > > After a restart of the broker after 6 hours the memory consumption is still at < 300 mb.
> > > > >
> > > > >
> > > > >
> > > > > Can someone tell me what the cause of this OOM exception might be (excessive paging, fast reconnects, zombie sessions, …)?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Kind regards
> > > > > Sebastian Götz
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > --- To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> > > > For additional commands, e-mail: users-help@activemq.apache.org
> > > > For further information, visit:
> > > > https://activemq.apache.org/contact
> > > >
> > > >
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > --- To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> > > > For additional commands, e-mail: users-help@activemq.apache.org
> > > > For further information, visit:
> > > > https://activemq.apache.org/contact
> > > >
> > > >
> > > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> > > For additional commands, e-mail: users-help@activemq.apache.org For
> > > further information, visit: https://activemq.apache.org/contact
> > >
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> > For additional commands, e-mail: users-help@activemq.apache.org For
> > further information, visit: https://activemq.apache.org/contact
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> For additional commands, e-mail: users-help@activemq.apache.org For further information, visit: https://activemq.apache.org/contact
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> For additional commands, e-mail: users-help@activemq.apache.org
> For further information, visit: https://activemq.apache.org/contact
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
For additional commands, e-mail: users-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
Re: AW: AW: Artemis - Help understanding heap dump
Posted by Clebert Suconic <cl...@apache.org>.
As I showed on the code, there will be a log.warn if max-read is reached.
You would need really a client consuming messages and not doing any acks to reach the limit.
You can find this on the web console by monitoring delivering and asked metrics.
One rule of thumb I have used is to keep about 1 page prefetch (10MB usually) or 100MB max read... Some (using the bytes metrics). although some of the users I deal with prefer to use max-messages and they will use their avg size of messages to have the max-message and prefetch-messages metrics.
but this all depends on the number of queues, number of messages, pending messages...etc.
BTW: on the example I gave you, I had a typo and I missed saying I was going to configure prefetch=10 and max-read=100
On 2025/01/23 06:23:05 s.goetz@inform-technology.de wrote:
> Okay, I think I have understood the difference. As we tend to have very small transactions or immediate commits this will not make much of a difference in our use case.
> Last question:
> The example you showed demonstrates how to starve the client,because of the limit in max-read/prefetch. In a real world application this would be quite hard to figure out, I think. Is there some sort of logging if the server hits any of those limits or the chance to find this state via Artemis Management Web Application?
>
> -----Ursprüngliche Nachricht-----
> Von: Clebert Suconic <cl...@apache.org>
> Gesendet: Mittwoch, 22. Januar 2025 17:32
> An: users@activemq.apache.org
> Betreff: Re: AW: Artemis - Help understanding heap dump
>
> just to make my response a bit clearer: max-read will look into the pending acks, while prefetch will only look at what's currently in the queue.
>
>
> (that's a TL;DR (Too long, didn't read) of my previous messages).
>
> On 2025/01/22 16:29:21 Clebert Suconic wrote:
> > The logic is not complex.. the whole thing is controlled here. The source code will be a very good documentation in this case:
> >
> > https://github.com/apache/activemq-artemis/blob/1ba0b65babf298e4dc4795
> > 1aa6998b2d0ac02be6/artemis-server/src/main/java/org/apache/activemq/ar
> > temis/core/server/impl/QueueImpl.java#L3469-L3493
> >
> > On 2025/01/22 16:25:19 Clebert Suconic wrote:
> > > sure...
> > >
> > >
> > > I originally did not want to implement prefetch.. but my user had a
> > > case where they needed a soft limit and a hard limit
> > >
> > >
> > > At max-read, the system will stop fetching for data... period... no more messages will come.
> > >
> > >
> > > With Prefetch.. if you receive the messages on the consumer, it will still request for more messages, but if you don't ack them.. things will stop at max-read.. there will be a log.warn in the system as you would starve otherwise.
> > >
> > >
> > > Example:
> > >
> > >
> > > Say you configured max-read at 100, and then you have a consumer doing the following:
> > >
> > > Session session = connection.createSession(... whatever you do to
> > > create a transaction section); consumer = session.createConsumer
> > >
> > > for (int I = 0; I < 1000; I++) {
> > > consumer.receive(); // notice I'm blocking here forever if no messages are coming.
> > > }
> > > // The system will never reach here, but max-read=100 and there are 100 messages pending.
> > > session.commit();
> > >
> > > As you take messages out of the queue.. the system will try to keep 10 messages in memory...
> > >
> > >
> > > But as you don't ack these messages the system will block at max-read.
> > >
> > >
> > >
> > > Makes sense?
> > >
> > >
> > > Also there are the bytes versions of these metrics where I use the payload to manage the flow. The system will always reach to whatever happens first on the rules.
> > >
> > > On 2025/01/15 06:32:52 s.goetz@inform-technology.de wrote:
> > > > Hey Clebert,
> > > >
> > > > thank you for the tip about ARTEMIS-4447.
> > > > If I understand correctly, these settings will influence how many messages of a persistent page are loaded into memory.
> > > > This sounds quite likely. We have had similar issues in the past with many consumers that were unable to dispatch the messages to their clients leading to massive paging and OOM.
> > > > As we are still on 2.28 I will recommend the team to migrate to a more recent version and have a look into these address settings.
> > > > But after a first look into those settings, I cannot quite figure out the difference between max-read-page-messages and prefetch-page-messages. Both claim to somehow limit the amount of messages read into memory.
> > > >
> > > > Can you explain the difference?
> > > >
> > > > Many thanks
> > > >
> > > > Sebastian
> > > >
> > > > -----Ursprüngliche Nachricht-----
> > > > Von: Clebert Suconic <cl...@apache.org>
> > > > Gesendet: Dienstag, 14. Januar 2025 18:11
> > > > An: users@activemq.apache.org
> > > > Betreff: Re: Artemis - Help understanding heap dump
> > > >
> > > > as part of ARTEMIS-4447, I added parameters to address-settings for prefetching.
> > > >
> > > > Also there was some flow changes based on an issue I had with an user, as they had consumers prefetching for a long time issuing more messages out of paging into memory.
> > > >
> > > >
> > > > You can now configure prefetch and such conditions would just keep the server from running out of memory.
> > > >
> > > >
> > > > This is the most likely scenario why you're having OME issues as it's similar to what I have dealt with my user.
> > > >
> > > >
> > > >
> > > >
> > > > On 2025/01/14 08:48:39 s.goetz@inform-technology.de wrote:
> > > > > Hello group,
> > > > >
> > > > >
> > > > >
> > > > > currently I am analyzing an OOM crash of a productive broker (ActiveMQ Artemis 2.28). The broker ran under 64 bit OpenJDK 17.0.9+9 (Temurin) with 4GB of heap configured via -Xmx VM option and G1 garbage collector.
> > > > >
> > > > > In the analyzer histogram I can see, that there are 5 big consumers of the VM’s heap. Four of those are very much the same:
> > > > >
> > > > >
> > > > >
> > > > > 1. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> > > > > 2. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> > > > > 3. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> > > > > 4. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> > > > >
> > > > >
> > > > >
> > > > > The fifth seems to be closely related to these four:
> > > > >
> > > > > 19 instances of java.util.TreeMap$Entry occupy 777.531.504 (18,18 %) bytes.
> > > > >
> > > > > Biggest instances:
> > > > >
> > > > > * java.util.TreeMap$Entry @ 0x7584c1ce8 - 231.178.240 (5,41 %) bytes.
> > > > >
> > > > > * java.util.TreeMap$Entry @ 0x7094185d8 - 220.148.272 (5,15 %) bytes.
> > > > >
> > > > > * java.util.TreeMap$Entry @ 0x77c22a440 - 116.097.024 (2,71 %) bytes.
> > > > >
> > > > > * java.util.TreeMap$Entry @ 0x796469d78 - 103.186.432 (2,41 %) bytes.
> > > > >
> > > > > * java.util.TreeMap$Entry @ 0x7c8c1e078 - 53.215.232 (1,24 %) bytes.
> > > > >
> > > > > Most of these instances are referenced from one instance of org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1 which occupies 1.282.832 (0,03 %) bytes.
> > > > >
> > > > >
> > > > >
> > > > > The analyzer states
> > > > >
> > > > > Common Path To the Accumulation Point:
> > > > >
> > > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1 @
> > > > > 0x79e40f078 Thread-1
> > > > > (ActiveMQ-PageExecutor-server-org.apache.activemq.artemis.core.server.
> > > > > impl.ActiveMQServerImpl$9@4aeaadc1) <mat://object/0x79e40f078>
> > > > > Thread
> > > > >
> > > > >
> > > > >
> > > > > After a restart of the broker after 6 hours the memory consumption is still at < 300 mb.
> > > > >
> > > > >
> > > > >
> > > > > Can someone tell me what the cause of this OOM exception might be (excessive paging, fast reconnects, zombie sessions, …)?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Kind regards
> > > > > Sebastian Götz
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > --- To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> > > > For additional commands, e-mail: users-help@activemq.apache.org
> > > > For further information, visit:
> > > > https://activemq.apache.org/contact
> > > >
> > > >
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > --- To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> > > > For additional commands, e-mail: users-help@activemq.apache.org
> > > > For further information, visit:
> > > > https://activemq.apache.org/contact
> > > >
> > > >
> > > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> > > For additional commands, e-mail: users-help@activemq.apache.org For
> > > further information, visit: https://activemq.apache.org/contact
> > >
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> > For additional commands, e-mail: users-help@activemq.apache.org For
> > further information, visit: https://activemq.apache.org/contact
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> For additional commands, e-mail: users-help@activemq.apache.org For further information, visit: https://activemq.apache.org/contact
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> For additional commands, e-mail: users-help@activemq.apache.org
> For further information, visit: https://activemq.apache.org/contact
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
For additional commands, e-mail: users-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
AW: AW: Artemis - Help understanding heap dump
Posted by s....@inform-technology.de.
Okay, I think I have understood the difference. As we tend to have very small transactions or immediate commits this will not make much of a difference in our use case.
Last question:
The example you showed demonstrates how to starve the client,because of the limit in max-read/prefetch. In a real world application this would be quite hard to figure out, I think. Is there some sort of logging if the server hits any of those limits or the chance to find this state via Artemis Management Web Application?
-----Ursprüngliche Nachricht-----
Von: Clebert Suconic <cl...@apache.org>
Gesendet: Mittwoch, 22. Januar 2025 17:32
An: users@activemq.apache.org
Betreff: Re: AW: Artemis - Help understanding heap dump
just to make my response a bit clearer: max-read will look into the pending acks, while prefetch will only look at what's currently in the queue.
(that's a TL;DR (Too long, didn't read) of my previous messages).
On 2025/01/22 16:29:21 Clebert Suconic wrote:
> The logic is not complex.. the whole thing is controlled here. The source code will be a very good documentation in this case:
>
> https://github.com/apache/activemq-artemis/blob/1ba0b65babf298e4dc4795
> 1aa6998b2d0ac02be6/artemis-server/src/main/java/org/apache/activemq/ar
> temis/core/server/impl/QueueImpl.java#L3469-L3493
>
> On 2025/01/22 16:25:19 Clebert Suconic wrote:
> > sure...
> >
> >
> > I originally did not want to implement prefetch.. but my user had a
> > case where they needed a soft limit and a hard limit
> >
> >
> > At max-read, the system will stop fetching for data... period... no more messages will come.
> >
> >
> > With Prefetch.. if you receive the messages on the consumer, it will still request for more messages, but if you don't ack them.. things will stop at max-read.. there will be a log.warn in the system as you would starve otherwise.
> >
> >
> > Example:
> >
> >
> > Say you configured max-read at 100, and then you have a consumer doing the following:
> >
> > Session session = connection.createSession(... whatever you do to
> > create a transaction section); consumer = session.createConsumer
> >
> > for (int I = 0; I < 1000; I++) {
> > consumer.receive(); // notice I'm blocking here forever if no messages are coming.
> > }
> > // The system will never reach here, but max-read=100 and there are 100 messages pending.
> > session.commit();
> >
> > As you take messages out of the queue.. the system will try to keep 10 messages in memory...
> >
> >
> > But as you don't ack these messages the system will block at max-read.
> >
> >
> >
> > Makes sense?
> >
> >
> > Also there are the bytes versions of these metrics where I use the payload to manage the flow. The system will always reach to whatever happens first on the rules.
> >
> > On 2025/01/15 06:32:52 s.goetz@inform-technology.de wrote:
> > > Hey Clebert,
> > >
> > > thank you for the tip about ARTEMIS-4447.
> > > If I understand correctly, these settings will influence how many messages of a persistent page are loaded into memory.
> > > This sounds quite likely. We have had similar issues in the past with many consumers that were unable to dispatch the messages to their clients leading to massive paging and OOM.
> > > As we are still on 2.28 I will recommend the team to migrate to a more recent version and have a look into these address settings.
> > > But after a first look into those settings, I cannot quite figure out the difference between max-read-page-messages and prefetch-page-messages. Both claim to somehow limit the amount of messages read into memory.
> > >
> > > Can you explain the difference?
> > >
> > > Many thanks
> > >
> > > Sebastian
> > >
> > > -----Ursprüngliche Nachricht-----
> > > Von: Clebert Suconic <cl...@apache.org>
> > > Gesendet: Dienstag, 14. Januar 2025 18:11
> > > An: users@activemq.apache.org
> > > Betreff: Re: Artemis - Help understanding heap dump
> > >
> > > as part of ARTEMIS-4447, I added parameters to address-settings for prefetching.
> > >
> > > Also there was some flow changes based on an issue I had with an user, as they had consumers prefetching for a long time issuing more messages out of paging into memory.
> > >
> > >
> > > You can now configure prefetch and such conditions would just keep the server from running out of memory.
> > >
> > >
> > > This is the most likely scenario why you're having OME issues as it's similar to what I have dealt with my user.
> > >
> > >
> > >
> > >
> > > On 2025/01/14 08:48:39 s.goetz@inform-technology.de wrote:
> > > > Hello group,
> > > >
> > > >
> > > >
> > > > currently I am analyzing an OOM crash of a productive broker (ActiveMQ Artemis 2.28). The broker ran under 64 bit OpenJDK 17.0.9+9 (Temurin) with 4GB of heap configured via -Xmx VM option and G1 garbage collector.
> > > >
> > > > In the analyzer histogram I can see, that there are 5 big consumers of the VM’s heap. Four of those are very much the same:
> > > >
> > > >
> > > >
> > > > 1. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> > > > 2. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> > > > 3. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> > > > 4. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> > > >
> > > >
> > > >
> > > > The fifth seems to be closely related to these four:
> > > >
> > > > 19 instances of java.util.TreeMap$Entry occupy 777.531.504 (18,18 %) bytes.
> > > >
> > > > Biggest instances:
> > > >
> > > > * java.util.TreeMap$Entry @ 0x7584c1ce8 - 231.178.240 (5,41 %) bytes.
> > > >
> > > > * java.util.TreeMap$Entry @ 0x7094185d8 - 220.148.272 (5,15 %) bytes.
> > > >
> > > > * java.util.TreeMap$Entry @ 0x77c22a440 - 116.097.024 (2,71 %) bytes.
> > > >
> > > > * java.util.TreeMap$Entry @ 0x796469d78 - 103.186.432 (2,41 %) bytes.
> > > >
> > > > * java.util.TreeMap$Entry @ 0x7c8c1e078 - 53.215.232 (1,24 %) bytes.
> > > >
> > > > Most of these instances are referenced from one instance of org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1 which occupies 1.282.832 (0,03 %) bytes.
> > > >
> > > >
> > > >
> > > > The analyzer states
> > > >
> > > > Common Path To the Accumulation Point:
> > > >
> > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1 @
> > > > 0x79e40f078 Thread-1
> > > > (ActiveMQ-PageExecutor-server-org.apache.activemq.artemis.core.server.
> > > > impl.ActiveMQServerImpl$9@4aeaadc1) <mat://object/0x79e40f078>
> > > > Thread
> > > >
> > > >
> > > >
> > > > After a restart of the broker after 6 hours the memory consumption is still at < 300 mb.
> > > >
> > > >
> > > >
> > > > Can someone tell me what the cause of this OOM exception might be (excessive paging, fast reconnects, zombie sessions, …)?
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Kind regards
> > > > Sebastian Götz
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > > ------------------------------------------------------------------
> > > --- To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> > > For additional commands, e-mail: users-help@activemq.apache.org
> > > For further information, visit:
> > > https://activemq.apache.org/contact
> > >
> > >
> > >
> > >
> > > ------------------------------------------------------------------
> > > --- To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> > > For additional commands, e-mail: users-help@activemq.apache.org
> > > For further information, visit:
> > > https://activemq.apache.org/contact
> > >
> > >
> > >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> > For additional commands, e-mail: users-help@activemq.apache.org For
> > further information, visit: https://activemq.apache.org/contact
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> For additional commands, e-mail: users-help@activemq.apache.org For
> further information, visit: https://activemq.apache.org/contact
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
For additional commands, e-mail: users-help@activemq.apache.org For further information, visit: https://activemq.apache.org/contact
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
For additional commands, e-mail: users-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
Re: AW: Artemis - Help understanding heap dump
Posted by Clebert Suconic <cl...@apache.org>.
just to make my response a bit clearer: max-read will look into the pending acks, while prefetch will only look at what's currently in the queue.
(that's a TL;DR (Too long, didn't read) of my previous messages).
On 2025/01/22 16:29:21 Clebert Suconic wrote:
> The logic is not complex.. the whole thing is controlled here. The source code will be a very good documentation in this case:
>
> https://github.com/apache/activemq-artemis/blob/1ba0b65babf298e4dc47951aa6998b2d0ac02be6/artemis-server/src/main/java/org/apache/activemq/artemis/core/server/impl/QueueImpl.java#L3469-L3493
>
> On 2025/01/22 16:25:19 Clebert Suconic wrote:
> > sure...
> >
> >
> > I originally did not want to implement prefetch.. but my user had a case where they needed a soft limit and a hard limit
> >
> >
> > At max-read, the system will stop fetching for data... period... no more messages will come.
> >
> >
> > With Prefetch.. if you receive the messages on the consumer, it will still request for more messages, but if you don't ack them.. things will stop at max-read.. there will be a log.warn in the system as you would starve otherwise.
> >
> >
> > Example:
> >
> >
> > Say you configured max-read at 100, and then you have a consumer doing the following:
> >
> > Session session = connection.createSession(... whatever you do to create a transaction section);
> > consumer = session.createConsumer
> >
> > for (int I = 0; I < 1000; I++) {
> > consumer.receive(); // notice I'm blocking here forever if no messages are coming.
> > }
> > // The system will never reach here, but max-read=100 and there are 100 messages pending.
> > session.commit();
> >
> > As you take messages out of the queue.. the system will try to keep 10 messages in memory...
> >
> >
> > But as you don't ack these messages the system will block at max-read.
> >
> >
> >
> > Makes sense?
> >
> >
> > Also there are the bytes versions of these metrics where I use the payload to manage the flow. The system will always reach to whatever happens first on the rules.
> >
> > On 2025/01/15 06:32:52 s.goetz@inform-technology.de wrote:
> > > Hey Clebert,
> > >
> > > thank you for the tip about ARTEMIS-4447.
> > > If I understand correctly, these settings will influence how many messages of a persistent page are loaded into memory.
> > > This sounds quite likely. We have had similar issues in the past with many consumers that were unable to dispatch the messages to their clients leading to massive paging and OOM.
> > > As we are still on 2.28 I will recommend the team to migrate to a more recent version and have a look into these address settings.
> > > But after a first look into those settings, I cannot quite figure out the difference between max-read-page-messages and prefetch-page-messages. Both claim to somehow limit the amount of messages read into memory.
> > >
> > > Can you explain the difference?
> > >
> > > Many thanks
> > >
> > > Sebastian
> > >
> > > -----Ursprüngliche Nachricht-----
> > > Von: Clebert Suconic <cl...@apache.org>
> > > Gesendet: Dienstag, 14. Januar 2025 18:11
> > > An: users@activemq.apache.org
> > > Betreff: Re: Artemis - Help understanding heap dump
> > >
> > > as part of ARTEMIS-4447, I added parameters to address-settings for prefetching.
> > >
> > > Also there was some flow changes based on an issue I had with an user, as they had consumers prefetching for a long time issuing more messages out of paging into memory.
> > >
> > >
> > > You can now configure prefetch and such conditions would just keep the server from running out of memory.
> > >
> > >
> > > This is the most likely scenario why you're having OME issues as it's similar to what I have dealt with my user.
> > >
> > >
> > >
> > >
> > > On 2025/01/14 08:48:39 s.goetz@inform-technology.de wrote:
> > > > Hello group,
> > > >
> > > >
> > > >
> > > > currently I am analyzing an OOM crash of a productive broker (ActiveMQ Artemis 2.28). The broker ran under 64 bit OpenJDK 17.0.9+9 (Temurin) with 4GB of heap configured via -Xmx VM option and G1 garbage collector.
> > > >
> > > > In the analyzer histogram I can see, that there are 5 big consumers of the VM’s heap. Four of those are very much the same:
> > > >
> > > >
> > > >
> > > > 1. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> > > > 2. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> > > > 3. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> > > > 4. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> > > >
> > > >
> > > >
> > > > The fifth seems to be closely related to these four:
> > > >
> > > > 19 instances of java.util.TreeMap$Entry occupy 777.531.504 (18,18 %) bytes.
> > > >
> > > > Biggest instances:
> > > >
> > > > * java.util.TreeMap$Entry @ 0x7584c1ce8 - 231.178.240 (5,41 %) bytes.
> > > >
> > > > * java.util.TreeMap$Entry @ 0x7094185d8 - 220.148.272 (5,15 %) bytes.
> > > >
> > > > * java.util.TreeMap$Entry @ 0x77c22a440 - 116.097.024 (2,71 %) bytes.
> > > >
> > > > * java.util.TreeMap$Entry @ 0x796469d78 - 103.186.432 (2,41 %) bytes.
> > > >
> > > > * java.util.TreeMap$Entry @ 0x7c8c1e078 - 53.215.232 (1,24 %) bytes.
> > > >
> > > > Most of these instances are referenced from one instance of org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1 which occupies 1.282.832 (0,03 %) bytes.
> > > >
> > > >
> > > >
> > > > The analyzer states
> > > >
> > > > Common Path To the Accumulation Point:
> > > >
> > > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1 @
> > > > 0x79e40f078 Thread-1
> > > > (ActiveMQ-PageExecutor-server-org.apache.activemq.artemis.core.server.
> > > > impl.ActiveMQServerImpl$9@4aeaadc1) <mat://object/0x79e40f078> Thread
> > > >
> > > >
> > > >
> > > > After a restart of the broker after 6 hours the memory consumption is still at < 300 mb.
> > > >
> > > >
> > > >
> > > > Can someone tell me what the cause of this OOM exception might be (excessive paging, fast reconnects, zombie sessions, …)?
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Kind regards
> > > > Sebastian Götz
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> > > For additional commands, e-mail: users-help@activemq.apache.org For further information, visit: https://activemq.apache.org/contact
> > >
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> > > For additional commands, e-mail: users-help@activemq.apache.org
> > > For further information, visit: https://activemq.apache.org/contact
> > >
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> > For additional commands, e-mail: users-help@activemq.apache.org
> > For further information, visit: https://activemq.apache.org/contact
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> For additional commands, e-mail: users-help@activemq.apache.org
> For further information, visit: https://activemq.apache.org/contact
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
For additional commands, e-mail: users-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
Re: AW: Artemis - Help understanding heap dump
Posted by Clebert Suconic <cl...@apache.org>.
The logic is not complex.. the whole thing is controlled here. The source code will be a very good documentation in this case:
https://github.com/apache/activemq-artemis/blob/1ba0b65babf298e4dc47951aa6998b2d0ac02be6/artemis-server/src/main/java/org/apache/activemq/artemis/core/server/impl/QueueImpl.java#L3469-L3493
On 2025/01/22 16:25:19 Clebert Suconic wrote:
> sure...
>
>
> I originally did not want to implement prefetch.. but my user had a case where they needed a soft limit and a hard limit
>
>
> At max-read, the system will stop fetching for data... period... no more messages will come.
>
>
> With Prefetch.. if you receive the messages on the consumer, it will still request for more messages, but if you don't ack them.. things will stop at max-read.. there will be a log.warn in the system as you would starve otherwise.
>
>
> Example:
>
>
> Say you configured max-read at 100, and then you have a consumer doing the following:
>
> Session session = connection.createSession(... whatever you do to create a transaction section);
> consumer = session.createConsumer
>
> for (int I = 0; I < 1000; I++) {
> consumer.receive(); // notice I'm blocking here forever if no messages are coming.
> }
> // The system will never reach here, but max-read=100 and there are 100 messages pending.
> session.commit();
>
> As you take messages out of the queue.. the system will try to keep 10 messages in memory...
>
>
> But as you don't ack these messages the system will block at max-read.
>
>
>
> Makes sense?
>
>
> Also there are the bytes versions of these metrics where I use the payload to manage the flow. The system will always reach to whatever happens first on the rules.
>
> On 2025/01/15 06:32:52 s.goetz@inform-technology.de wrote:
> > Hey Clebert,
> >
> > thank you for the tip about ARTEMIS-4447.
> > If I understand correctly, these settings will influence how many messages of a persistent page are loaded into memory.
> > This sounds quite likely. We have had similar issues in the past with many consumers that were unable to dispatch the messages to their clients leading to massive paging and OOM.
> > As we are still on 2.28 I will recommend the team to migrate to a more recent version and have a look into these address settings.
> > But after a first look into those settings, I cannot quite figure out the difference between max-read-page-messages and prefetch-page-messages. Both claim to somehow limit the amount of messages read into memory.
> >
> > Can you explain the difference?
> >
> > Many thanks
> >
> > Sebastian
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Clebert Suconic <cl...@apache.org>
> > Gesendet: Dienstag, 14. Januar 2025 18:11
> > An: users@activemq.apache.org
> > Betreff: Re: Artemis - Help understanding heap dump
> >
> > as part of ARTEMIS-4447, I added parameters to address-settings for prefetching.
> >
> > Also there was some flow changes based on an issue I had with an user, as they had consumers prefetching for a long time issuing more messages out of paging into memory.
> >
> >
> > You can now configure prefetch and such conditions would just keep the server from running out of memory.
> >
> >
> > This is the most likely scenario why you're having OME issues as it's similar to what I have dealt with my user.
> >
> >
> >
> >
> > On 2025/01/14 08:48:39 s.goetz@inform-technology.de wrote:
> > > Hello group,
> > >
> > >
> > >
> > > currently I am analyzing an OOM crash of a productive broker (ActiveMQ Artemis 2.28). The broker ran under 64 bit OpenJDK 17.0.9+9 (Temurin) with 4GB of heap configured via -Xmx VM option and G1 garbage collector.
> > >
> > > In the analyzer histogram I can see, that there are 5 big consumers of the VM’s heap. Four of those are very much the same:
> > >
> > >
> > >
> > > 1. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> > > 2. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> > > 3. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> > > 4. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> > >
> > >
> > >
> > > The fifth seems to be closely related to these four:
> > >
> > > 19 instances of java.util.TreeMap$Entry occupy 777.531.504 (18,18 %) bytes.
> > >
> > > Biggest instances:
> > >
> > > * java.util.TreeMap$Entry @ 0x7584c1ce8 - 231.178.240 (5,41 %) bytes.
> > >
> > > * java.util.TreeMap$Entry @ 0x7094185d8 - 220.148.272 (5,15 %) bytes.
> > >
> > > * java.util.TreeMap$Entry @ 0x77c22a440 - 116.097.024 (2,71 %) bytes.
> > >
> > > * java.util.TreeMap$Entry @ 0x796469d78 - 103.186.432 (2,41 %) bytes.
> > >
> > > * java.util.TreeMap$Entry @ 0x7c8c1e078 - 53.215.232 (1,24 %) bytes.
> > >
> > > Most of these instances are referenced from one instance of org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1 which occupies 1.282.832 (0,03 %) bytes.
> > >
> > >
> > >
> > > The analyzer states
> > >
> > > Common Path To the Accumulation Point:
> > >
> > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1 @
> > > 0x79e40f078 Thread-1
> > > (ActiveMQ-PageExecutor-server-org.apache.activemq.artemis.core.server.
> > > impl.ActiveMQServerImpl$9@4aeaadc1) <mat://object/0x79e40f078> Thread
> > >
> > >
> > >
> > > After a restart of the broker after 6 hours the memory consumption is still at < 300 mb.
> > >
> > >
> > >
> > > Can someone tell me what the cause of this OOM exception might be (excessive paging, fast reconnects, zombie sessions, …)?
> > >
> > >
> > >
> > >
> > >
> > > Kind regards
> > > Sebastian Götz
> > >
> > >
> > >
> > >
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> > For additional commands, e-mail: users-help@activemq.apache.org For further information, visit: https://activemq.apache.org/contact
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> > For additional commands, e-mail: users-help@activemq.apache.org
> > For further information, visit: https://activemq.apache.org/contact
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> For additional commands, e-mail: users-help@activemq.apache.org
> For further information, visit: https://activemq.apache.org/contact
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
For additional commands, e-mail: users-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
Re: AW: Artemis - Help understanding heap dump
Posted by Clebert Suconic <cl...@apache.org>.
sure...
I originally did not want to implement prefetch.. but my user had a case where they needed a soft limit and a hard limit
At max-read, the system will stop fetching for data... period... no more messages will come.
With Prefetch.. if you receive the messages on the consumer, it will still request for more messages, but if you don't ack them.. things will stop at max-read.. there will be a log.warn in the system as you would starve otherwise.
Example:
Say you configured max-read at 100, and then you have a consumer doing the following:
Session session = connection.createSession(... whatever you do to create a transaction section);
consumer = session.createConsumer
for (int I = 0; I < 1000; I++) {
consumer.receive(); // notice I'm blocking here forever if no messages are coming.
}
// The system will never reach here, but max-read=100 and there are 100 messages pending.
session.commit();
As you take messages out of the queue.. the system will try to keep 10 messages in memory...
But as you don't ack these messages the system will block at max-read.
Makes sense?
Also there are the bytes versions of these metrics where I use the payload to manage the flow. The system will always reach to whatever happens first on the rules.
On 2025/01/15 06:32:52 s.goetz@inform-technology.de wrote:
> Hey Clebert,
>
> thank you for the tip about ARTEMIS-4447.
> If I understand correctly, these settings will influence how many messages of a persistent page are loaded into memory.
> This sounds quite likely. We have had similar issues in the past with many consumers that were unable to dispatch the messages to their clients leading to massive paging and OOM.
> As we are still on 2.28 I will recommend the team to migrate to a more recent version and have a look into these address settings.
> But after a first look into those settings, I cannot quite figure out the difference between max-read-page-messages and prefetch-page-messages. Both claim to somehow limit the amount of messages read into memory.
>
> Can you explain the difference?
>
> Many thanks
>
> Sebastian
>
> -----Ursprüngliche Nachricht-----
> Von: Clebert Suconic <cl...@apache.org>
> Gesendet: Dienstag, 14. Januar 2025 18:11
> An: users@activemq.apache.org
> Betreff: Re: Artemis - Help understanding heap dump
>
> as part of ARTEMIS-4447, I added parameters to address-settings for prefetching.
>
> Also there was some flow changes based on an issue I had with an user, as they had consumers prefetching for a long time issuing more messages out of paging into memory.
>
>
> You can now configure prefetch and such conditions would just keep the server from running out of memory.
>
>
> This is the most likely scenario why you're having OME issues as it's similar to what I have dealt with my user.
>
>
>
>
> On 2025/01/14 08:48:39 s.goetz@inform-technology.de wrote:
> > Hello group,
> >
> >
> >
> > currently I am analyzing an OOM crash of a productive broker (ActiveMQ Artemis 2.28). The broker ran under 64 bit OpenJDK 17.0.9+9 (Temurin) with 4GB of heap configured via -Xmx VM option and G1 garbage collector.
> >
> > In the analyzer histogram I can see, that there are 5 big consumers of the VM’s heap. Four of those are very much the same:
> >
> >
> >
> > 1. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> > 2. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> > 3. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> > 4. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> >
> >
> >
> > The fifth seems to be closely related to these four:
> >
> > 19 instances of java.util.TreeMap$Entry occupy 777.531.504 (18,18 %) bytes.
> >
> > Biggest instances:
> >
> > * java.util.TreeMap$Entry @ 0x7584c1ce8 - 231.178.240 (5,41 %) bytes.
> >
> > * java.util.TreeMap$Entry @ 0x7094185d8 - 220.148.272 (5,15 %) bytes.
> >
> > * java.util.TreeMap$Entry @ 0x77c22a440 - 116.097.024 (2,71 %) bytes.
> >
> > * java.util.TreeMap$Entry @ 0x796469d78 - 103.186.432 (2,41 %) bytes.
> >
> > * java.util.TreeMap$Entry @ 0x7c8c1e078 - 53.215.232 (1,24 %) bytes.
> >
> > Most of these instances are referenced from one instance of org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1 which occupies 1.282.832 (0,03 %) bytes.
> >
> >
> >
> > The analyzer states
> >
> > Common Path To the Accumulation Point:
> >
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1 @
> > 0x79e40f078 Thread-1
> > (ActiveMQ-PageExecutor-server-org.apache.activemq.artemis.core.server.
> > impl.ActiveMQServerImpl$9@4aeaadc1) <mat://object/0x79e40f078> Thread
> >
> >
> >
> > After a restart of the broker after 6 hours the memory consumption is still at < 300 mb.
> >
> >
> >
> > Can someone tell me what the cause of this OOM exception might be (excessive paging, fast reconnects, zombie sessions, …)?
> >
> >
> >
> >
> >
> > Kind regards
> > Sebastian Götz
> >
> >
> >
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> For additional commands, e-mail: users-help@activemq.apache.org For further information, visit: https://activemq.apache.org/contact
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> For additional commands, e-mail: users-help@activemq.apache.org
> For further information, visit: https://activemq.apache.org/contact
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
For additional commands, e-mail: users-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
AW: Artemis - Help understanding heap dump
Posted by s....@inform-technology.de.
Hey Clebert,
thank you for the tip about ARTEMIS-4447.
If I understand correctly, these settings will influence how many messages of a persistent page are loaded into memory.
This sounds quite likely. We have had similar issues in the past with many consumers that were unable to dispatch the messages to their clients leading to massive paging and OOM.
As we are still on 2.28 I will recommend the team to migrate to a more recent version and have a look into these address settings.
But after a first look into those settings, I cannot quite figure out the difference between max-read-page-messages and prefetch-page-messages. Both claim to somehow limit the amount of messages read into memory.
Can you explain the difference?
Many thanks
Sebastian
-----Ursprüngliche Nachricht-----
Von: Clebert Suconic <cl...@apache.org>
Gesendet: Dienstag, 14. Januar 2025 18:11
An: users@activemq.apache.org
Betreff: Re: Artemis - Help understanding heap dump
as part of ARTEMIS-4447, I added parameters to address-settings for prefetching.
Also there was some flow changes based on an issue I had with an user, as they had consumers prefetching for a long time issuing more messages out of paging into memory.
You can now configure prefetch and such conditions would just keep the server from running out of memory.
This is the most likely scenario why you're having OME issues as it's similar to what I have dealt with my user.
On 2025/01/14 08:48:39 s.goetz@inform-technology.de wrote:
> Hello group,
>
>
>
> currently I am analyzing an OOM crash of a productive broker (ActiveMQ Artemis 2.28). The broker ran under 64 bit OpenJDK 17.0.9+9 (Temurin) with 4GB of heap configured via -Xmx VM option and G1 garbage collector.
>
> In the analyzer histogram I can see, that there are 5 big consumers of the VM’s heap. Four of those are very much the same:
>
>
>
> 1. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> 2. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> 3. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> 4. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
>
>
>
> The fifth seems to be closely related to these four:
>
> 19 instances of java.util.TreeMap$Entry occupy 777.531.504 (18,18 %) bytes.
>
> Biggest instances:
>
> * java.util.TreeMap$Entry @ 0x7584c1ce8 - 231.178.240 (5,41 %) bytes.
>
> * java.util.TreeMap$Entry @ 0x7094185d8 - 220.148.272 (5,15 %) bytes.
>
> * java.util.TreeMap$Entry @ 0x77c22a440 - 116.097.024 (2,71 %) bytes.
>
> * java.util.TreeMap$Entry @ 0x796469d78 - 103.186.432 (2,41 %) bytes.
>
> * java.util.TreeMap$Entry @ 0x7c8c1e078 - 53.215.232 (1,24 %) bytes.
>
> Most of these instances are referenced from one instance of org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1 which occupies 1.282.832 (0,03 %) bytes.
>
>
>
> The analyzer states
>
> Common Path To the Accumulation Point:
>
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1 @
> 0x79e40f078 Thread-1
> (ActiveMQ-PageExecutor-server-org.apache.activemq.artemis.core.server.
> impl.ActiveMQServerImpl$9@4aeaadc1) <mat://object/0x79e40f078> Thread
>
>
>
> After a restart of the broker after 6 hours the memory consumption is still at < 300 mb.
>
>
>
> Can someone tell me what the cause of this OOM exception might be (excessive paging, fast reconnects, zombie sessions, …)?
>
>
>
>
>
> Kind regards
> Sebastian Götz
>
>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
For additional commands, e-mail: users-help@activemq.apache.org For further information, visit: https://activemq.apache.org/contact
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
For additional commands, e-mail: users-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
Re: Artemis - Help understanding heap dump
Posted by Clebert Suconic <cl...@apache.org>.
as part of ARTEMIS-4447, I added parameters to address-settings for prefetching.
Also there was some flow changes based on an issue I had with an user, as they had consumers prefetching for a long time issuing more messages out of paging into memory.
You can now configure prefetch and such conditions would just keep the server from running out of memory.
This is the most likely scenario why you're having OME issues as it's similar to what I have dealt with my user.
On 2025/01/14 08:48:39 s.goetz@inform-technology.de wrote:
> Hello group,
>
>
>
> currently I am analyzing an OOM crash of a productive broker (ActiveMQ Artemis 2.28). The broker ran under 64 bit OpenJDK 17.0.9+9 (Temurin) with 4GB of heap configured via -Xmx VM option and G1 garbage collector.
>
> In the analyzer histogram I can see, that there are 5 big consumers of the VM’s heap. Four of those are very much the same:
>
>
>
> 1. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> 2. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> 3. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
> 4. One instance of org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl occupies 777.531.896 (18,18 %) bytes. The memory is accumulated in one instance of java.util.TreeMap$Entry which occupies 327.069.808 (7,65 %) bytes.
>
>
>
> The fifth seems to be closely related to these four:
>
> 19 instances of java.util.TreeMap$Entry occupy 777.531.504 (18,18 %) bytes.
>
> Biggest instances:
>
> * java.util.TreeMap$Entry @ 0x7584c1ce8 - 231.178.240 (5,41 %) bytes.
>
> * java.util.TreeMap$Entry @ 0x7094185d8 - 220.148.272 (5,15 %) bytes.
>
> * java.util.TreeMap$Entry @ 0x77c22a440 - 116.097.024 (2,71 %) bytes.
>
> * java.util.TreeMap$Entry @ 0x796469d78 - 103.186.432 (2,41 %) bytes.
>
> * java.util.TreeMap$Entry @ 0x7c8c1e078 - 53.215.232 (1,24 %) bytes.
>
> Most of these instances are referenced from one instance of org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1 which occupies 1.282.832 (0,03 %) bytes.
>
>
>
> The analyzer states
>
> Common Path To the Accumulation Point:
>
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1 @ 0x79e40f078 Thread-1 (ActiveMQ-PageExecutor-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$9@4aeaadc1) <mat://object/0x79e40f078> Thread
>
>
>
> After a restart of the broker after 6 hours the memory consumption is still at < 300 mb.
>
>
>
> Can someone tell me what the cause of this OOM exception might be (excessive paging, fast reconnects, zombie sessions, …)?
>
>
>
>
>
> Kind regards
> Sebastian Götz
>
>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
For additional commands, e-mail: users-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact