You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@artemis.apache.org by Shiv Kumar Dixit <sh...@it.eurofinseu.com> on 2026/01/14 12:44:21 UTC

K8s broker pod getting killed with OOM

We are hosting Artemis broker in Kubernetes using operator-based solution. We deploy the broker as statefulset with 2 or 4 replicas. We assign for e.g. 6 GB for heap and 9 GB for pod, 1.2 GB (1/5 of max heap) for global-max-size. All addresses normally use -1 for max-size-bytes but some less frequently used queues are defined with 100KB for max-size-bytes to allow early paging.

We have following observations:
1. As the broker pod starts, broker container immediately occupies 6 GB for max heap. It seems expected as both min and max heap are same.
2. Pod memory usage starts with 6+ GB and once we have pending messages, good producers and consumers connect to broker, invalid SSL attempts happen, broker GUI access happens etc. during normal broker operations - pod memory usage keeps increasing and now reaches 9 GB.
3. Once the pod hits limit of 9 GB, K8s kills the pod with OOMKilling event and restarts the pod. Here we don't see broker container getting killed with OOM rather pod is killed and restarted. It forces the broker to restart.
4. We have configured artemis.profile to capture memory dump in case of OOM of broker but it never happens. So, we are assuming broker process is not going out of memory, but pod is going out of memory due to increased non-heap usage.
5. Only way to recover here is to increase heap and pod memory limits from 6 GB and 9 GB to higher values and wait for next re-occurrence.

1. Is there any way to analyse what is going wrong with non-heap native memory usage?
2. If non-heap native memory is expected to increase to such extent due to pending messages, SSL errors etc.?
3. Is there any param we can use to restrict the non-heap native memory usage?
4. If netty which handles connection aspect of broker can create such memory consumption and cause OOM of pod?
5. Can we have any monitoring param that can hint that pod is potentially in danger of getting killed?

Thanks
Shiv

RE: K8s broker pod getting killed with OOM

Posted by Vilius Šumskas via users <us...@artemis.apache.org>.
Hi Kumar,

I tried to read all your questions. It’s really hard to understand the problem but, to add to what other already said, I would suggest the following things:

  1.  Try to reproduce the OOM issue under simple setup. Maybe even without Kubernetes involved.
  2.  Remove -Xms and -Xms JVM options and start using -XX:MaxRAMPercentage. It allows for automatic heap adjustment depending on memory available for JVM. This is especially important in containers where memory availability could change dynamically.
You didn’t provide information which JVM you are using. On HotSpot this parameter is set to 25% by default. On OpenJ9 it is set to 50%. For Artemis I would advice starting with 50%. If your memory configuration is on the higher end, then maybe go to 60 or 65%. Don’t go above 70%.
  3.  Another important parameter is -XX:MaxDirectMemorySize. Netty uses direct memory. Others may correct me if I’m wrong, but I assume most of the direct memory in Artemis is used by those connections.
By default -XX:MaxDirectMemorySize is set to the same _inferred_ value as -Xmx OR -XX:MaxRAMPercentage. If you set -XX:MaxRAMPercentage to lower values, you probably could ignore direct memory parameter for now. However, if you set -XX:MaxRAMPercentage to something like 60% and -XX:MaxDirectMemorySize is not set, and the container has, let’s say, 10GB of RAM available, maximum size of heap will be set to 6GB and the maximum size of direct memory will also be 6GB. 12 GB is obviously more than 10 GB. It basically means the more you increase heap percentage limit, probability to consume all the memory using heap and direct memory combination rises. In addition to that JVM uses additional memory for internal stuff. Sadly -XX:DirectMemorySize cannot be set to percentage of total memory available. For most stable result you should set -XX:MaxDirectMemorySize so that heap + direct memory size never goes above 80-85% of total memory. But before that, I would  measure what’s really happening in JVM first.
  4.  The interesting part is Netty. Depending on your JVM and its version, Netty uses different strategies for memory allocation. The most interesting parameter for you is -Dio.netty.maxDirectMemory https://github.com/netty/netty/blob/f80b70c75ed7dff27d7e74d2c18ca8a0724a1cc7/common/src/main/java/io/netty/util/internal/PlatformDependent.java#L156 Netty uses this parameter to determine maximum direct memory available for JVM, however it uses both, DirectByteBuffers, and Unsafe memory allocation strategies (again, depending on your JVM). This means you cannot see full memory usage produced by Netty using just JVM tools (like JMX for example). See this comment https://github.com/netty/netty/issues/11895#issuecomment-987739006 and whole thread for more information on that. Also, don’t miss this great explanation https://netty.io/wiki/java-24-and-sun.misc.unsafe.html#what-netty-uses-unsafe-for
  5.  You should be able to see what’s going on memory wise in Artemis and JVM (including all the memory used by the Netty https://artemis.apache.org/components/artemis/documentation/latest/metrics.html#netty-allocator) using Artemis Prometheus Metrics plugin https://github.com/rh-messaging/artemis-prometheus-metrics-plugin. This should show you a more clear picture what memory parameters and values you need to set.
  6.  Also, don’t forget that on Kubernetes JVM takes max memory available from “limits” parameter. Don’t ever set “limits” to larger value than that available on the smallest node.

And last but not the least. Unless you have very specific reason to do it I, personally, advice against running Artemis (or any broker for that matter) under Kubernetes. Most brokers use memory and CPU extensively anyway. One might see using operator as convenient configuration option, however, there are plenty of tools to do modern configurations under simple VMs. In summary, I don’t see how Kubernetes provide any value for such use case. It just complicates debugging and maintenance. You also don’t have a full power of Linux kernel in case of memory pressure, for example, swap is (mostly) not available under Kubernetes.

Hope that helps.

--
   Best Regards,
    Vilius

From: Shiv Kumar Dixit <sh...@it.eurofinseu.com>
Sent: Wednesday, January 14, 2026 2:44 PM
To: users@artemis.apache.org
Subject: K8s broker pod getting killed with OOM

We are hosting Artemis broker in Kubernetes using operator-based solution. We deploy the broker as statefulset with 2 or 4 replicas. We assign for e.g. 6 GB for heap and 9 GB for pod, 1.2 GB (1/5 of max heap) for global-max-size. All addresses normally use -1 for max-size-bytes but some less frequently used queues are defined with 100KB for max-size-bytes to allow early paging.

We have following observations:
1. As the broker pod starts, broker container immediately occupies 6 GB for max heap. It seems expected as both min and max heap are same.
2. Pod memory usage starts with 6+ GB and once we have pending messages, good producers and consumers connect to broker, invalid SSL attempts happen, broker GUI access happens etc. during normal broker operations - pod memory usage keeps increasing and now reaches 9 GB.
3. Once the pod hits limit of 9 GB, K8s kills the pod with OOMKilling event and restarts the pod. Here we don’t see broker container getting killed with OOM rather pod is killed and restarted. It forces the broker to restart.
4. We have configured artemis.profile to capture memory dump in case of OOM of broker but it never happens. So, we are assuming broker process is not going out of memory, but pod is going out of memory due to increased non-heap usage.
5. Only way to recover here is to increase heap and pod memory limits from 6 GB and 9 GB to higher values and wait for next re-occurrence.

1. Is there any way to analyse what is going wrong with non-heap native memory usage?
2. If non-heap native memory is expected to increase to such extent due to pending messages, SSL errors etc.?
3. Is there any param we can use to restrict the non-heap native memory usage?
4. If netty which handles connection aspect of broker can create such memory consumption and cause OOM of pod?
5. Can we have any monitoring param that can hint that pod is potentially in danger of getting killed?

Thanks
Shiv

RE: K8s broker pod getting killed with OOM

Posted by Vilius Šumskas via users <us...@artemis.apache.org>.
You are right, that in default configuration Netty could use more memory than that defined in JDK's -XX:MaxDirectMemorySize, however if you don't run anything else on the same JDK in container, the amount of that oversize should be minimal.
Even if overallocation is the case, I don't think -Dio.netty.maxDirectMemory=0 will help you either. Looking at the Netty's code I don't see anything what disabled unsafe memory allocation out of JDK's range when this parameter is 0. But maybe I miss something.

If you really really want to control maximum memory allocated by Netty, you should set it to > 0, or use io.netty.noPreferDirect=true and move everything to Heap (which I would not advice to do for performance reasons).

-- 
    Vilius

-----Original Message-----
From: Thorsten Meinl <th...@knime.com> 
Sent: Monday, January 19, 2026 12:33 PM
To: users@artemis.apache.org
Subject: Re: K8s broker pod getting killed with OOM

Hi,

For other containerized appplications that use netty, we have a start script that sets memory sizes as follows:

----
JAVA_MAX_MEM_RATIO=${JAVA_MAX_MEM_RATIO:-60}
JAVA_MAX_DIRECT_MEM_RATIO=${JAVA_MAX_DIRECT_MEM_RATIO:-10}

if [ -n "$CONTAINER_MEMORY_REQUESTS" ]; then
    JAVA_HEAP="-Xmx$((CONTAINER_MEMORY_REQUESTS * JAVA_MAX_MEM_RATIO /
100)) \
        -XX:MaxDirectMemorySize=$((CONTAINER_MEMORY_REQUESTS * JAVA_MAX_DIRECT_MEM_RATIO / 100)) \
        -Dio.netty.maxDirectMemory=0"
else
    JAVA_HEAP="-XX:MaxRAMPercentage=$JAVA_MAX_MEM_RATIO - Dio.netty.maxDirectMemory=33554432"
fi
----

Note that we use the memory _requests_ instead of the limits to size memory pools (if available). Also the -Dio.netty.maxDirectMemory=0 is important because otherwise netty might still allocate memory outside the direct memory.

I would assume that the same rationale can also be applied to Artemis.

Regards,

Thorsten

Am Montag, dem 19.01.2026 um 10:21 +0000 schrieb Vilius Šumskas via
users:
> -Xmx defines Heap memory. Direct memory is not part of the Heap.
> To expand a little more on a topic: by default, if you don't set any 
> memory parameters, JVM uses 25% of total memory assigned to pod for 
> Heap, and the same amount for Direct memory. As you can imagine this 
> is fine because the remaining 50% can be used by native JVM memory, 
> libraries and "OS stuff". The default configuration is the most stable 
> one. Now if you set -Xmx to 50% or more, but don't set - 
> XX:MaxDirectMemorySize, it means that both Heap and Direct memory can 
> now expand beyond 100% of Pod memory. Hence not so stable 
> configuration.
> So you should set -XX:MaxDirectMemorySize not to the percentage of - 
> Xmx, but so that the -XX:MaxDirectMemorySize + -Xmx is not above 80- 
> 85% of Pod limit. Of course, that means that some of that reserved 
> memory will always be "wasted" and free, but if you want the most 
> stable configuration, this is it.
> 
> How much percentage of that 80-85% sum should be for Direct Memory and 
> how much for Heap is another question. That depends on your load type. 
> Memory metrics should show you that during the time. Maybe you need 
> 50/50 for both. Or maybe you need 60/40 or 70/30.
> 
> Regarding Netty configuration, I'm not expert on the topic, but from 
> the information you've provided you should be fine with default 
> configuration. In any case, given a little bit convulated Netty 
> documentation be sure to read links to the source code I have 
> provided. Also, you could enable Netty debugging to see how much 
> memory it actually detects (it is written in debug log) and if you 
> need to change anything. For more advanced Netty questions you should 
> ask on their mailing list.
> 
> --
>    Best Regards,
>     Vilius
> 
> -----Original Message-----
> From: Shiv Kumar Dixit <sh...@it.eurofinseu.com>
> Sent: Monday, January 19, 2026 10:43 AM
> To: users@artemis.apache.org
> Cc: Vilius Šumskas <vi...@rivile.lt>
> Subject: RE: K8s broker pod getting killed with OOM
> 
> Hello Vilius,
> Thanks for your input.
> 
> 1. When we get set limit of pod, it is much lesser than the node's 
> memory capacity. So, there is no issue of over allocation resulting in 
> OOM.
> 2. Do we need set both -XX:MaxDirectMemorySize and - 
> Dio.netty.maxDirectMemory or Netty will derive - 
> Dio.netty.maxDirectMemory from -XX:MaxDirectMemorySize?
> 3. We are using OpenJDK 17 with HotSpot.
> 4. We will enable netty-allocator and see what data we can extract out 
> of it. Thanks for the hint.
> 5. Based on your input that "most stable result you should set - 
> XX:MaxDirectMemorySize so that heap + direct memory size never goes 
> above 80-85% of total memory", it appears if we set - 
> XX:MaxDirectMemorySize as 25% of -Xmx, it will be within limit of pod.
> 
> 
> Best Regards
> Shiv
> 
> -----Original Message-----
> From: Vilius Šumskas via users <us...@artemis.apache.org>
> Sent: 16 January 2026 06:36 AM
> To: users@artemis.apache.org
> Cc: Vilius Šumskas <vi...@rivile.lt>
> Subject: RE: K8s broker pod getting killed with OOM
> 
> 
> 
> Unverified Sender: The sender of this email has not been verified.
> Review the content of the message carefully and verify the identity of 
> the sender before acting on this email: replying, opening attachments 
> or clicking links.
> 
> 
> That's a little bit off-topic, but would you consider exchanging -Xmx 
> into -XX:MaxRAMPercentage for default configuration in 3.0 too? This 
> would universally improve default JVM memory settings under various 
> scenarios, without the need to change it every time more physical 
> memory is added. And if yes, should I create a ticket for this?
> 
> --
>     Vilius
> 
> -----Original Message-----
> From: Clebert Suconic <cl...@gmail.com>
> Sent: Thursday, January 15, 2026 12:36 AM
> To: users@artemis.apache.org
> Subject: Re: K8s broker pod getting killed with OOM
> 
> so, in summary, what I'm recommending you is:
> 
> use max-size-messages for all the queues.. for your large queues, use 
> something like 10MB and for your small queues 100K
> 
> also keep max-read-page-bytes in use... keep it at 20M
> 
> 
> 
> If I could change the past I would have a max-size on every address we 
> deploy, and having global-max-size for the upmost emergency case..
> it's something I'm looking to change into artemis 3.0 or 4.0. (I can't 
> change that into a minor version, as it could break certain cases...
> as some users that I know use heavy filtering and can't really rely on 
> paging).
> 
> 
> On Wed, Jan 14, 2026 at 5:31 PM Clebert Suconic 
> <cl...@gmail.com> wrote:
> > 
> > I would recommend against trusting global-max-size. and use max- 
> > size for all the addresses.
> > 
> > Also what is your reading attributes. I would recommending using the 
> > new prefetch values.
> > 
> > 
> > 
> > And also what operator are you using? arkmq? your own?
> > 
> > On Wed, Jan 14, 2026 at 7:44 AM Shiv Kumar Dixit 
> > <sh...@it.eurofinseu.com> wrote:
> > > 
> > > We are hosting Artemis broker in Kubernetes using operator-based 
> > > solution. We deploy the broker as statefulset with 2 or 4 
> > > replicas. We assign for e.g. 6 GB for heap and 9 GB for pod, 1.2 
> > > GB (1/5 of max heap) for global-max-size. All addresses normally 
> > > use -1 for max-size-bytes but some less frequently used queues are 
> > > defined with 100KB for max-size-bytes to allow early paging.
> > > 
> > > 
> > > 
> > > We have following observations:
> > > 
> > > 1. As the broker pod starts, broker container immediately occupies 
> > > 6 GB for max heap. It seems expected as both min and max heap are 
> > > same.
> > > 
> > > 2. Pod memory usage starts with 6+ GB and once we have pending 
> > > messages, good producers and consumers connect to broker, invalid 
> > > SSL attempts happen, broker GUI access happens etc. during normal 
> > > broker operations - pod memory usage keeps increasing and now 
> > > reaches 9 GB.
> > > 
> > > 3. Once the pod hits limit of 9 GB, K8s kills the pod with 
> > > OOMKilling event and restarts the pod. Here we don’t see broker 
> > > container getting killed with OOM rather pod is killed and 
> > > restarted. It forces the broker to restart.
> > > 
> > > 4. We have configured artemis.profile to capture memory dump in 
> > > case of OOM of broker but it never happens. So, we are assuming 
> > > broker process is not going out of memory, but pod is going out of 
> > > memory due to increased non-heap usage.
> > > 
> > > 5. Only way to recover here is to increase heap and pod memory 
> > > limits from 6 GB and 9 GB to higher values and wait for next re- 
> > > occurrence.
> > > 
> > > 
> > > 
> > > 1. Is there any way to analyse what is going wrong with non-heap 
> > > native memory usage?
> > > 
> > > 2. If non-heap native memory is expected to increase to such 
> > > extent due to pending messages, SSL errors etc.?
> > > 
> > > 3. Is there any param we can use to restrict the non-heap native 
> > > memory usage?
> > > 
> > > 4. If netty which handles connection aspect of broker can create 
> > > such memory consumption and cause OOM of pod?
> > > 
> > > 5. Can we have any monitoring param that can hint that pod is 
> > > potentially in danger of getting killed?
> > > 
> > > 
> > > 
> > > Thanks
> > > 
> > > Shiv
> > 
> > 
> > 
> > --
> > Clebert Suconic
> 
> 
> 
> --
> Clebert Suconic
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
> For additional commands, e-mail: users-help@artemis.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
> For additional commands, e-mail: users-help@artemis.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
> For additional commands, e-mail: users-help@artemis.apache.org

--
Dr.-Ing. Thorsten Meinl
KNIME AG
Talacker 50
8001 Zurich, Switzerland


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
For additional commands, e-mail: users-help@artemis.apache.org

Re: K8s broker pod getting killed with OOM

Posted by Thorsten Meinl <th...@knime.com>.
Hi,

For other containerized appplications that use netty, we have a start
script that sets memory sizes as follows:

----
JAVA_MAX_MEM_RATIO=${JAVA_MAX_MEM_RATIO:-60}
JAVA_MAX_DIRECT_MEM_RATIO=${JAVA_MAX_DIRECT_MEM_RATIO:-10}

if [ -n "$CONTAINER_MEMORY_REQUESTS" ]; then
    JAVA_HEAP="-Xmx$((CONTAINER_MEMORY_REQUESTS * JAVA_MAX_MEM_RATIO /
100)) \
        -XX:MaxDirectMemorySize=$((CONTAINER_MEMORY_REQUESTS *
JAVA_MAX_DIRECT_MEM_RATIO / 100)) \
        -Dio.netty.maxDirectMemory=0"
else
    JAVA_HEAP="-XX:MaxRAMPercentage=$JAVA_MAX_MEM_RATIO -
Dio.netty.maxDirectMemory=33554432"
fi
----

Note that we use the memory _requests_ instead of the limits to size
memory pools (if available). Also the -Dio.netty.maxDirectMemory=0 is
important because otherwise netty might still allocate memory outside
the direct memory.

I would assume that the same rationale can also be applied to Artemis.

Regards,

Thorsten

Am Montag, dem 19.01.2026 um 10:21 +0000 schrieb Vilius Šumskas via
users:
> -Xmx defines Heap memory. Direct memory is not part of the Heap.
> To expand a little more on a topic: by default, if you don't set any
> memory parameters, JVM uses 25% of total memory assigned to pod for
> Heap, and the same amount for Direct memory. As you can imagine this
> is fine because the remaining 50% can be used by native JVM memory,
> libraries and "OS stuff". The default configuration is the most
> stable one. Now if you set -Xmx to 50% or more, but don't set -
> XX:MaxDirectMemorySize, it means that both Heap and Direct memory can
> now expand beyond 100% of Pod memory. Hence not so stable
> configuration.
> So you should set -XX:MaxDirectMemorySize not to the percentage of -
> Xmx, but so that the -XX:MaxDirectMemorySize + -Xmx is not above 80-
> 85% of Pod limit. Of course, that means that some of that reserved
> memory will always be "wasted" and free, but if you want the most
> stable configuration, this is it.
> 
> How much percentage of that 80-85% sum should be for Direct Memory
> and how much for Heap is another question. That depends on your load
> type. Memory metrics should show you that during the time. Maybe you
> need 50/50 for both. Or maybe you need 60/40 or 70/30. 
> 
> Regarding Netty configuration, I'm not expert on the topic, but from
> the information you've provided you should be fine with default
> configuration. In any case, given a little bit convulated Netty
> documentation be sure to read links to the source code I have
> provided. Also, you could enable Netty debugging to see how much
> memory it actually detects (it is written in debug log) and if you
> need to change anything. For more advanced Netty questions you should
> ask on their mailing list.
> 
> -- 
>    Best Regards,
>     Vilius
> 
> -----Original Message-----
> From: Shiv Kumar Dixit <sh...@it.eurofinseu.com> 
> Sent: Monday, January 19, 2026 10:43 AM
> To: users@artemis.apache.org
> Cc: Vilius Šumskas <vi...@rivile.lt>
> Subject: RE: K8s broker pod getting killed with OOM
> 
> Hello Vilius,
> Thanks for your input.
> 
> 1. When we get set limit of pod, it is much lesser than the node's
> memory capacity. So, there is no issue of over allocation resulting
> in OOM.
> 2. Do we need set both -XX:MaxDirectMemorySize and -
> Dio.netty.maxDirectMemory or Netty will derive -
> Dio.netty.maxDirectMemory from -XX:MaxDirectMemorySize?
> 3. We are using OpenJDK 17 with HotSpot.
> 4. We will enable netty-allocator and see what data we can extract
> out of it. Thanks for the hint.
> 5. Based on your input that "most stable result you should set -
> XX:MaxDirectMemorySize so that heap + direct memory size never goes
> above 80-85% of total memory", it appears if we set -
> XX:MaxDirectMemorySize as 25% of -Xmx, it will be within limit of
> pod.
> 
> 
> Best Regards
> Shiv
> 
> -----Original Message-----
> From: Vilius Šumskas via users <us...@artemis.apache.org> 
> Sent: 16 January 2026 06:36 AM
> To: users@artemis.apache.org
> Cc: Vilius Šumskas <vi...@rivile.lt>
> Subject: RE: K8s broker pod getting killed with OOM
> 
> 
> 
> Unverified Sender: The sender of this email has not been verified.
> Review the content of the message carefully and verify the identity
> of the sender before acting on this email: replying, opening
> attachments or clicking links.
> 
> 
> That's a little bit off-topic, but would you consider exchanging -Xmx
> into -XX:MaxRAMPercentage for default configuration in 3.0 too? This
> would universally improve default JVM memory settings under various
> scenarios, without the need to change it every time more physical
> memory is added. And if yes, should I create a ticket for this?
> 
> --
>     Vilius
> 
> -----Original Message-----
> From: Clebert Suconic <cl...@gmail.com>
> Sent: Thursday, January 15, 2026 12:36 AM
> To: users@artemis.apache.org
> Subject: Re: K8s broker pod getting killed with OOM
> 
> so, in summary, what I'm recommending you is:
> 
> use max-size-messages for all the queues.. for your large queues, use
> something like 10MB and for your small queues 100K
> 
> also keep max-read-page-bytes in use... keep it at 20M
> 
> 
> 
> If I could change the past I would have a max-size on every address
> we deploy, and having global-max-size for the upmost emergency case..
> it's something I'm looking to change into artemis 3.0 or 4.0. (I
> can't change that into a minor version, as it could break certain
> cases...
> as some users that I know use heavy filtering and can't really rely
> on paging).
> 
> 
> On Wed, Jan 14, 2026 at 5:31 PM Clebert Suconic
> <cl...@gmail.com> wrote:
> > 
> > I would recommend against trusting global-max-size. and use max-
> > size 
> > for all the addresses.
> > 
> > Also what is your reading attributes. I would recommending using
> > the 
> > new prefetch values.
> > 
> > 
> > 
> > And also what operator are you using? arkmq? your own?
> > 
> > On Wed, Jan 14, 2026 at 7:44 AM Shiv Kumar Dixit 
> > <sh...@it.eurofinseu.com> wrote:
> > > 
> > > We are hosting Artemis broker in Kubernetes using operator-based
> > > solution. We deploy the broker as statefulset with 2 or 4
> > > replicas. We assign for e.g. 6 GB for heap and 9 GB for pod, 1.2
> > > GB (1/5 of max heap) for global-max-size. All addresses normally
> > > use -1 for max-size-bytes but some less frequently used queues
> > > are defined with 100KB for max-size-bytes to allow early paging.
> > > 
> > > 
> > > 
> > > We have following observations:
> > > 
> > > 1. As the broker pod starts, broker container immediately
> > > occupies 6 GB for max heap. It seems expected as both min and max
> > > heap are same.
> > > 
> > > 2. Pod memory usage starts with 6+ GB and once we have pending
> > > messages, good producers and consumers connect to broker, invalid
> > > SSL attempts happen, broker GUI access happens etc. during normal
> > > broker operations - pod memory usage keeps increasing and now
> > > reaches 9 GB.
> > > 
> > > 3. Once the pod hits limit of 9 GB, K8s kills the pod with
> > > OOMKilling event and restarts the pod. Here we don’t see broker
> > > container getting killed with OOM rather pod is killed and
> > > restarted. It forces the broker to restart.
> > > 
> > > 4. We have configured artemis.profile to capture memory dump in
> > > case of OOM of broker but it never happens. So, we are assuming
> > > broker process is not going out of memory, but pod is going out
> > > of memory due to increased non-heap usage.
> > > 
> > > 5. Only way to recover here is to increase heap and pod memory
> > > limits from 6 GB and 9 GB to higher values and wait for next re-
> > > occurrence.
> > > 
> > > 
> > > 
> > > 1. Is there any way to analyse what is going wrong with non-heap
> > > native memory usage?
> > > 
> > > 2. If non-heap native memory is expected to increase to such
> > > extent due to pending messages, SSL errors etc.?
> > > 
> > > 3. Is there any param we can use to restrict the non-heap native
> > > memory usage?
> > > 
> > > 4. If netty which handles connection aspect of broker can create
> > > such memory consumption and cause OOM of pod?
> > > 
> > > 5. Can we have any monitoring param that can hint that pod is
> > > potentially in danger of getting killed?
> > > 
> > > 
> > > 
> > > Thanks
> > > 
> > > Shiv
> > 
> > 
> > 
> > --
> > Clebert Suconic
> 
> 
> 
> --
> Clebert Suconic
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
> For additional commands, e-mail: users-help@artemis.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
> For additional commands, e-mail: users-help@artemis.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
> For additional commands, e-mail: users-help@artemis.apache.org

-- 
Dr.-Ing. Thorsten Meinl
KNIME AG
Talacker 50
8001 Zurich, Switzerland


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
For additional commands, e-mail: users-help@artemis.apache.org

RE: K8s broker pod getting killed with OOM

Posted by Vilius Šumskas via users <us...@artemis.apache.org>.
-Xmx defines Heap memory. Direct memory is not part of the Heap.
To expand a little more on a topic: by default, if you don't set any memory parameters, JVM uses 25% of total memory assigned to pod for Heap, and the same amount for Direct memory. As you can imagine this is fine because the remaining 50% can be used by native JVM memory, libraries and "OS stuff". The default configuration is the most stable one. Now if you set -Xmx to 50% or more, but don't set -XX:MaxDirectMemorySize, it means that both Heap and Direct memory can now expand beyond 100% of Pod memory. Hence not so stable configuration.
So you should set -XX:MaxDirectMemorySize not to the percentage of -Xmx, but so that the -XX:MaxDirectMemorySize + -Xmx is not above 80-85% of Pod limit. Of course, that means that some of that reserved memory will always be "wasted" and free, but if you want the most stable configuration, this is it.

How much percentage of that 80-85% sum should be for Direct Memory and how much for Heap is another question. That depends on your load type. Memory metrics should show you that during the time. Maybe you need 50/50 for both. Or maybe you need 60/40 or 70/30. 

Regarding Netty configuration, I'm not expert on the topic, but from the information you've provided you should be fine with default configuration. In any case, given a little bit convulated Netty documentation be sure to read links to the source code I have provided. Also, you could enable Netty debugging to see how much memory it actually detects (it is written in debug log) and if you need to change anything. For more advanced Netty questions you should ask on their mailing list.

-- 
   Best Regards,
    Vilius

-----Original Message-----
From: Shiv Kumar Dixit <sh...@it.eurofinseu.com> 
Sent: Monday, January 19, 2026 10:43 AM
To: users@artemis.apache.org
Cc: Vilius Šumskas <vi...@rivile.lt>
Subject: RE: K8s broker pod getting killed with OOM

Hello Vilius,
Thanks for your input.

1. When we get set limit of pod, it is much lesser than the node's memory capacity. So, there is no issue of over allocation resulting in OOM.
2. Do we need set both -XX:MaxDirectMemorySize and -Dio.netty.maxDirectMemory or Netty will derive -Dio.netty.maxDirectMemory from -XX:MaxDirectMemorySize?
3. We are using OpenJDK 17 with HotSpot.
4. We will enable netty-allocator and see what data we can extract out of it. Thanks for the hint.
5. Based on your input that "most stable result you should set -XX:MaxDirectMemorySize so that heap + direct memory size never goes above 80-85% of total memory", it appears if we set -XX:MaxDirectMemorySize as 25% of -Xmx, it will be within limit of pod.


Best Regards
Shiv

-----Original Message-----
From: Vilius Šumskas via users <us...@artemis.apache.org> 
Sent: 16 January 2026 06:36 AM
To: users@artemis.apache.org
Cc: Vilius Šumskas <vi...@rivile.lt>
Subject: RE: K8s broker pod getting killed with OOM



Unverified Sender: The sender of this email has not been verified. Review the content of the message carefully and verify the identity of the sender before acting on this email: replying, opening attachments or clicking links.


That's a little bit off-topic, but would you consider exchanging -Xmx into -XX:MaxRAMPercentage for default configuration in 3.0 too? This would universally improve default JVM memory settings under various scenarios, without the need to change it every time more physical memory is added. And if yes, should I create a ticket for this?

--
    Vilius

-----Original Message-----
From: Clebert Suconic <cl...@gmail.com>
Sent: Thursday, January 15, 2026 12:36 AM
To: users@artemis.apache.org
Subject: Re: K8s broker pod getting killed with OOM

so, in summary, what I'm recommending you is:

use max-size-messages for all the queues.. for your large queues, use something like 10MB and for your small queues 100K

also keep max-read-page-bytes in use... keep it at 20M



If I could change the past I would have a max-size on every address we deploy, and having global-max-size for the upmost emergency case..
it's something I'm looking to change into artemis 3.0 or 4.0. (I can't change that into a minor version, as it could break certain cases...
as some users that I know use heavy filtering and can't really rely on paging).


On Wed, Jan 14, 2026 at 5:31 PM Clebert Suconic <cl...@gmail.com> wrote:
>
> I would recommend against trusting global-max-size. and use max-size 
> for all the addresses.
>
> Also what is your reading attributes. I would recommending using the 
> new prefetch values.
>
>
>
> And also what operator are you using? arkmq? your own?
>
> On Wed, Jan 14, 2026 at 7:44 AM Shiv Kumar Dixit 
> <sh...@it.eurofinseu.com> wrote:
> >
> > We are hosting Artemis broker in Kubernetes using operator-based solution. We deploy the broker as statefulset with 2 or 4 replicas. We assign for e.g. 6 GB for heap and 9 GB for pod, 1.2 GB (1/5 of max heap) for global-max-size. All addresses normally use -1 for max-size-bytes but some less frequently used queues are defined with 100KB for max-size-bytes to allow early paging.
> >
> >
> >
> > We have following observations:
> >
> > 1. As the broker pod starts, broker container immediately occupies 6 GB for max heap. It seems expected as both min and max heap are same.
> >
> > 2. Pod memory usage starts with 6+ GB and once we have pending messages, good producers and consumers connect to broker, invalid SSL attempts happen, broker GUI access happens etc. during normal broker operations - pod memory usage keeps increasing and now reaches 9 GB.
> >
> > 3. Once the pod hits limit of 9 GB, K8s kills the pod with OOMKilling event and restarts the pod. Here we don’t see broker container getting killed with OOM rather pod is killed and restarted. It forces the broker to restart.
> >
> > 4. We have configured artemis.profile to capture memory dump in case of OOM of broker but it never happens. So, we are assuming broker process is not going out of memory, but pod is going out of memory due to increased non-heap usage.
> >
> > 5. Only way to recover here is to increase heap and pod memory limits from 6 GB and 9 GB to higher values and wait for next re-occurrence.
> >
> >
> >
> > 1. Is there any way to analyse what is going wrong with non-heap native memory usage?
> >
> > 2. If non-heap native memory is expected to increase to such extent due to pending messages, SSL errors etc.?
> >
> > 3. Is there any param we can use to restrict the non-heap native memory usage?
> >
> > 4. If netty which handles connection aspect of broker can create such memory consumption and cause OOM of pod?
> >
> > 5. Can we have any monitoring param that can hint that pod is potentially in danger of getting killed?
> >
> >
> >
> > Thanks
> >
> > Shiv
>
>
>
> --
> Clebert Suconic



--
Clebert Suconic

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
For additional commands, e-mail: users-help@artemis.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
For additional commands, e-mail: users-help@artemis.apache.org

RE: K8s broker pod getting killed with OOM

Posted by Shiv Kumar Dixit <sh...@it.eurofinseu.com>.
Hello Vilius,
Thanks for your input.

1. When we get set limit of pod, it is much lesser than the node's memory capacity. So, there is no issue of over allocation resulting in OOM.
2. Do we need set both -XX:MaxDirectMemorySize and -Dio.netty.maxDirectMemory or Netty will derive -Dio.netty.maxDirectMemory from -XX:MaxDirectMemorySize?
3. We are using OpenJDK 17 with HotSpot.
4. We will enable netty-allocator and see what data we can extract out of it. Thanks for the hint.
5. Based on your input that "most stable result you should set -XX:MaxDirectMemorySize so that heap + direct memory size never goes above 80-85% of total memory", it appears if we set -XX:MaxDirectMemorySize as 25% of -Xmx, it will be within limit of pod.


Best Regards
Shiv

-----Original Message-----
From: Vilius Šumskas via users <us...@artemis.apache.org> 
Sent: 16 January 2026 06:36 AM
To: users@artemis.apache.org
Cc: Vilius Šumskas <vi...@rivile.lt>
Subject: RE: K8s broker pod getting killed with OOM



Unverified Sender: The sender of this email has not been verified. Review the content of the message carefully and verify the identity of the sender before acting on this email: replying, opening attachments or clicking links.


That's a little bit off-topic, but would you consider exchanging -Xmx into -XX:MaxRAMPercentage for default configuration in 3.0 too? This would universally improve default JVM memory settings under various scenarios, without the need to change it every time more physical memory is added. And if yes, should I create a ticket for this?

--
    Vilius

-----Original Message-----
From: Clebert Suconic <cl...@gmail.com>
Sent: Thursday, January 15, 2026 12:36 AM
To: users@artemis.apache.org
Subject: Re: K8s broker pod getting killed with OOM

so, in summary, what I'm recommending you is:

use max-size-messages for all the queues.. for your large queues, use something like 10MB and for your small queues 100K

also keep max-read-page-bytes in use... keep it at 20M



If I could change the past I would have a max-size on every address we deploy, and having global-max-size for the upmost emergency case..
it's something I'm looking to change into artemis 3.0 or 4.0. (I can't change that into a minor version, as it could break certain cases...
as some users that I know use heavy filtering and can't really rely on paging).


On Wed, Jan 14, 2026 at 5:31 PM Clebert Suconic <cl...@gmail.com> wrote:
>
> I would recommend against trusting global-max-size. and use max-size 
> for all the addresses.
>
> Also what is your reading attributes. I would recommending using the 
> new prefetch values.
>
>
>
> And also what operator are you using? arkmq? your own?
>
> On Wed, Jan 14, 2026 at 7:44 AM Shiv Kumar Dixit 
> <sh...@it.eurofinseu.com> wrote:
> >
> > We are hosting Artemis broker in Kubernetes using operator-based solution. We deploy the broker as statefulset with 2 or 4 replicas. We assign for e.g. 6 GB for heap and 9 GB for pod, 1.2 GB (1/5 of max heap) for global-max-size. All addresses normally use -1 for max-size-bytes but some less frequently used queues are defined with 100KB for max-size-bytes to allow early paging.
> >
> >
> >
> > We have following observations:
> >
> > 1. As the broker pod starts, broker container immediately occupies 6 GB for max heap. It seems expected as both min and max heap are same.
> >
> > 2. Pod memory usage starts with 6+ GB and once we have pending messages, good producers and consumers connect to broker, invalid SSL attempts happen, broker GUI access happens etc. during normal broker operations - pod memory usage keeps increasing and now reaches 9 GB.
> >
> > 3. Once the pod hits limit of 9 GB, K8s kills the pod with OOMKilling event and restarts the pod. Here we don’t see broker container getting killed with OOM rather pod is killed and restarted. It forces the broker to restart.
> >
> > 4. We have configured artemis.profile to capture memory dump in case of OOM of broker but it never happens. So, we are assuming broker process is not going out of memory, but pod is going out of memory due to increased non-heap usage.
> >
> > 5. Only way to recover here is to increase heap and pod memory limits from 6 GB and 9 GB to higher values and wait for next re-occurrence.
> >
> >
> >
> > 1. Is there any way to analyse what is going wrong with non-heap native memory usage?
> >
> > 2. If non-heap native memory is expected to increase to such extent due to pending messages, SSL errors etc.?
> >
> > 3. Is there any param we can use to restrict the non-heap native memory usage?
> >
> > 4. If netty which handles connection aspect of broker can create such memory consumption and cause OOM of pod?
> >
> > 5. Can we have any monitoring param that can hint that pod is potentially in danger of getting killed?
> >
> >
> >
> > Thanks
> >
> > Shiv
>
>
>
> --
> Clebert Suconic



--
Clebert Suconic

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
For additional commands, e-mail: users-help@artemis.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
For additional commands, e-mail: users-help@artemis.apache.org

RE: K8s broker pod getting killed with OOM

Posted by Vilius Šumskas via users <us...@artemis.apache.org>.
That's a little bit off-topic, but would you consider exchanging -Xmx into -XX:MaxRAMPercentage for default configuration in 3.0 too? This would universally improve default JVM memory settings under various scenarios, without the need to change it every time more physical memory is added. And if yes, should I create a ticket for this?

-- 
    Vilius

-----Original Message-----
From: Clebert Suconic <cl...@gmail.com> 
Sent: Thursday, January 15, 2026 12:36 AM
To: users@artemis.apache.org
Subject: Re: K8s broker pod getting killed with OOM

so, in summary, what I'm recommending you is:

use max-size-messages for all the queues.. for your large queues, use something like 10MB and for your small queues 100K

also keep max-read-page-bytes in use... keep it at 20M



If I could change the past I would have a max-size on every address we deploy, and having global-max-size for the upmost emergency case..
it's something I'm looking to change into artemis 3.0 or 4.0. (I can't change that into a minor version, as it could break certain cases...
as some users that I know use heavy filtering and can't really rely on paging).


On Wed, Jan 14, 2026 at 5:31 PM Clebert Suconic <cl...@gmail.com> wrote:
>
> I would recommend against trusting global-max-size. and use max-size 
> for all the addresses.
>
> Also what is your reading attributes. I would recommending using the 
> new prefetch values.
>
>
>
> And also what operator are you using? arkmq? your own?
>
> On Wed, Jan 14, 2026 at 7:44 AM Shiv Kumar Dixit 
> <sh...@it.eurofinseu.com> wrote:
> >
> > We are hosting Artemis broker in Kubernetes using operator-based solution. We deploy the broker as statefulset with 2 or 4 replicas. We assign for e.g. 6 GB for heap and 9 GB for pod, 1.2 GB (1/5 of max heap) for global-max-size. All addresses normally use -1 for max-size-bytes but some less frequently used queues are defined with 100KB for max-size-bytes to allow early paging.
> >
> >
> >
> > We have following observations:
> >
> > 1. As the broker pod starts, broker container immediately occupies 6 GB for max heap. It seems expected as both min and max heap are same.
> >
> > 2. Pod memory usage starts with 6+ GB and once we have pending messages, good producers and consumers connect to broker, invalid SSL attempts happen, broker GUI access happens etc. during normal broker operations - pod memory usage keeps increasing and now reaches 9 GB.
> >
> > 3. Once the pod hits limit of 9 GB, K8s kills the pod with OOMKilling event and restarts the pod. Here we don’t see broker container getting killed with OOM rather pod is killed and restarted. It forces the broker to restart.
> >
> > 4. We have configured artemis.profile to capture memory dump in case of OOM of broker but it never happens. So, we are assuming broker process is not going out of memory, but pod is going out of memory due to increased non-heap usage.
> >
> > 5. Only way to recover here is to increase heap and pod memory limits from 6 GB and 9 GB to higher values and wait for next re-occurrence.
> >
> >
> >
> > 1. Is there any way to analyse what is going wrong with non-heap native memory usage?
> >
> > 2. If non-heap native memory is expected to increase to such extent due to pending messages, SSL errors etc.?
> >
> > 3. Is there any param we can use to restrict the non-heap native memory usage?
> >
> > 4. If netty which handles connection aspect of broker can create such memory consumption and cause OOM of pod?
> >
> > 5. Can we have any monitoring param that can hint that pod is potentially in danger of getting killed?
> >
> >
> >
> > Thanks
> >
> > Shiv
>
>
>
> --
> Clebert Suconic



--
Clebert Suconic

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
For additional commands, e-mail: users-help@artemis.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
For additional commands, e-mail: users-help@artemis.apache.org

Re: K8s broker pod getting killed with OOM

Posted by Clebert Suconic <cl...@gmail.com>.
Please look into implementing my suggestions.  Let me know how it goes and
we will figure out next steps?

Clebert Suconic


On Thu, Jan 15, 2026 at 11:08 AM Shiv Kumar Dixit <
shivkumar.dixit@it.eurofinseu.com> wrote:

> We are using open source one from ArkMQ
>
>
>
> Best Regards
>
> Shiv
>
>
>
> *From:* Clebert Suconic <cl...@gmail.com>
> *Sent:* 15 January 2026 08:26 PM
> *To:* users@artemis.apache.org
> *Subject:* Re: K8s broker pod getting killed with OOM
>
>
>
>
>
> *Unverified Sender: *The sender of this email has not been verified.
> Review the content of the message carefully and verify the identity of the
> sender before acting on this email: replying, opening attachments or
> clicking links.
>
>
>
> >>
> We are using Artemis 2.37.0 version in K8s and Artemis IO operator version
> is 1.2.5.
>
>
>
> Which one? the commercial version from Red Hat / openshift, or the
> opensource one from ArkMQ?
>
>
>
> On Thu, Jan 15, 2026 at 9:51 AM Clebert Suconic <cl...@gmail.com>
> wrote:
>
> Those points you described are the reason why I suggesting using
> max-address-size on the every destination...
>
>
>
>
>
> have max-size at say 20M for every destination (make 100K for small
> destinations if you like, but I think 20M for every destination is mostly
> okay, unless you have a lot of destinations).
>
>
>
>
>
> have max-read at 10M...
>
>
>
> This should then optimize your memory usage
>
>
>
> On Thu, Jan 15, 2026 at 6:53 AM Shiv Kumar Dixit <
> shivkumar.dixit@it.eurofinseu.com> wrote:
>
> Hello Arthur and Clebert
>
> When our broker pod starts, it first starts 2 init containers which
> terminate and release resources after completing the setup. So our pod
> basically runs 2 containers – one for vault and another for broker. We
> verified the memory and CPU usage of these init containers and main
> containers using top pod and it shows reasonable data.
>
>
>
> Yes we see Linux OOMKiller is invoked and we are trying to read its report
> to see any meaningful information.
>
>
>
> In the meanwhile, we have noticed below scenario is causing OOMKilling of
> broker container.
>
>    1. There are lot of pending messages on a given queue TEST along with
>    small pending messages on various other queues. Since we are using global
>    max size, some of messages are loaded in memory and rest are in paging
>    folder.
>
>
>
>    2. There are 3-4 consumers on TEST queue but they are very slow hence
>    pending message backlog is not cleared. We see below log in broker:
>
> AMQ224127: Message dispatch from paging is blocked. Address TEST/Queue
> TEST will not read any more messages from paging until pending messages are
> acknowledged. There are currently 5150 messages pending (20972400 bytes)
> with max reads at maxPageReadMessages(-1) and maxPageReadBytes(20971520).
> Either increase reading attributes at the address-settings or change your
> consumers to acknowledge more often.
>
>
>
>    3. We also see below log in broker:
>
> AMQ224108: Stopped paging on address ‘TEST’; size=62986496 bytes (96016
> messages); maxSize=-1 bytes (-1 messages); globalSize=430581015 bytes
> (158406 messages); globalMaxSize=4194304000 bytes (-1 messages);
>
>
>
>    4. If such blocked consumers and pending messages combination can
>    cause broker pod to go into OOM which is running with 30 GB of heap and 40
>    GB of pod memory?
>
>
>
>    5. Since consumers were not consuming messages on time and gave
>    consent to purge the messages, we tried to purge the message manually via
>    broker GUI. Sometimes it worked and more messages got loaded from pages to
>    broker memory but many times broker pod got OOM and restarted.
>
>
>
>    6. This cycle of successful purge or broker restart continued till all
>    messages from pages were loaded into memory and purged. Post cleanup there
>    was no broker restart.
>
>
>
>    7. If purging messages via broker GUI can cause OOM even though broker
>    pod is running with 30 GB of heap and 40 GB of pod memory?
>
>
>
>    8. What is the best way to optimize the broker configuration in such
>    cases where we will always have slow consumers and possibly lot of pending
>    messages in memory and paging folders?
>
>
>
> This impacted broker pod A has a network bridge with another independent
> broker pod B in a hub and spoke model which has very less connection and
> almost no pending messages. We also noticed that if broker pod A goes into
> OOM due to slow consumer and pending messages as described above, second
> broker pod B which is connected over network bridge with first broker pod A
> also goes into restart loop with OOM. Does restart of source pod A and
> disconnection-reconnection of small numbers of bridges can cause target
> broker pod B to restart? We have seen this side effect as well.
>
>
>
> We are using Artemis 2.37.0 version in K8s and Artemis IO operator version
> is 1.2.5.
>
>
>
> Best Regards
>
> Shiv
>
>
>
> *From:* Arthur Naseef <ar...@apache.org>
> *Sent:* 15 January 2026 06:40 AM
> *To:* users@artemis.apache.org
> *Subject:* Re: K8s broker pod getting killed with OOM
>
>
>
>
>
> *Unverified Sender: *The sender of this email has not been verified.
> Review the content of the message carefully and verify the identity of the
> sender before acting on this email: replying, opening attachments or
> clicking links.
>
>
>
> So 3100 connections is a large number, but that doesn't sound like a good
> reason for the broker pod to go OOM.  Also, getting up to 40gb, I would say
> the 50% rule of thumb may be too conservative (i.e. a higher percentage
> could be reasonable), which is contradicted by your outcome.  Are there
> other containers running in the same POD that might be taking up memory?
> Maybe sidecars?
>
>
>
> Unfortunately, I don't have a working kubernetes setup available right
> now.  If I did, I could poke around and try to give specific tips on
> checking the memory use of the POD.
>
>
>
> Do you know if the Linux OOM killer is getting invoked?  That would be
> reported by the kernel of the node on which the pod was executing.  If you
> can view that report, it includes a lot of useful information, including
> all of the processes involved and the amount of memory used by each.
>
>
>
> Art
>
>
>
>
>
> On Wed, Jan 14, 2026 at 3:52 PM Shiv Kumar Dixit <
> shivkumar.dixit@it.eurofinseu.com> wrote:
>
> Thanks Clebert and Arthur for inputs. I will try your suggestions and let
> you know how it goes.
>
> I have another observation based on issue happening in live. Based on
> input from Arthur, current setup is configured with 20 GB heap and 40 GB
> pod. As the pod started, we got 3100 connections to broker and within
> minutes the pod got OOMKilled. If there is any relation b/w number of
> connections on broker and pod going OOM?
>
> Best Regards
> Shiv
>
> -----Original Message-----
> From: Clebert Suconic <cl...@gmail.com>
> Sent: 15 January 2026 04:06 AM
> To: users@artemis.apache.org
> Subject: Re: K8s broker pod getting killed with OOM
>
>
>
> Unverified Sender: The sender of this email has not been verified. Review
> the content of the message carefully and verify the identity of the sender
> before acting on this email: replying, opening attachments or clicking
> links.
>
>
> so, in summary, what I'm recommending you is:
>
> use max-size-messages for all the queues.. for your large queues, use
> something like 10MB and for your small queues 100K
>
> also keep max-read-page-bytes in use... keep it at 20M
>
>
>
> If I could change the past I would have a max-size on every address we
> deploy, and having global-max-size for the upmost emergency case..
> it's something I'm looking to change into artemis 3.0 or 4.0. (I can't
> change that into a minor version, as it could break certain cases...
> as some users that I know use heavy filtering and can't really rely on
> paging).
>
>
> On Wed, Jan 14, 2026 at 5:31 PM Clebert Suconic <cl...@gmail.com>
> wrote:
> >
> > I would recommend against trusting global-max-size. and use max-size
> > for all the addresses.
> >
> > Also what is your reading attributes. I would recommending using the
> > new prefetch values.
> >
> >
> >
> > And also what operator are you using? arkmq? your own?
> >
> > On Wed, Jan 14, 2026 at 7:44 AM Shiv Kumar Dixit
> > <sh...@it.eurofinseu.com> wrote:
> > >
> > > We are hosting Artemis broker in Kubernetes using operator-based
> solution. We deploy the broker as statefulset with 2 or 4 replicas. We
> assign for e.g. 6 GB for heap and 9 GB for pod, 1.2 GB (1/5 of max heap)
> for global-max-size. All addresses normally use -1 for max-size-bytes but
> some less frequently used queues are defined with 100KB for max-size-bytes
> to allow early paging.
> > >
> > >
> > >
> > > We have following observations:
> > >
> > > 1. As the broker pod starts, broker container immediately occupies 6
> GB for max heap. It seems expected as both min and max heap are same.
> > >
> > > 2. Pod memory usage starts with 6+ GB and once we have pending
> messages, good producers and consumers connect to broker, invalid SSL
> attempts happen, broker GUI access happens etc. during normal broker
> operations - pod memory usage keeps increasing and now reaches 9 GB.
> > >
> > > 3. Once the pod hits limit of 9 GB, K8s kills the pod with OOMKilling
> event and restarts the pod. Here we don’t see broker container getting
> killed with OOM rather pod is killed and restarted. It forces the broker to
> restart.
> > >
> > > 4. We have configured artemis.profile to capture memory dump in case
> of OOM of broker but it never happens. So, we are assuming broker process
> is not going out of memory, but pod is going out of memory due to increased
> non-heap usage.
> > >
> > > 5. Only way to recover here is to increase heap and pod memory limits
> from 6 GB and 9 GB to higher values and wait for next re-occurrence.
> > >
> > >
> > >
> > > 1. Is there any way to analyse what is going wrong with non-heap
> native memory usage?
> > >
> > > 2. If non-heap native memory is expected to increase to such extent
> due to pending messages, SSL errors etc.?
> > >
> > > 3. Is there any param we can use to restrict the non-heap native
> memory usage?
> > >
> > > 4. If netty which handles connection aspect of broker can create such
> memory consumption and cause OOM of pod?
> > >
> > > 5. Can we have any monitoring param that can hint that pod is
> potentially in danger of getting killed?
> > >
> > >
> > >
> > > Thanks
> > >
> > > Shiv
> >
> >
> >
> > --
> > Clebert Suconic
>
>
>
> --
> Clebert Suconic
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
> For additional commands, e-mail: users-help@artemis.apache.org
>
>
>
>
> --
>
> Clebert Suconic
>
>
>
>
> --
>
> Clebert Suconic
>

RE: K8s broker pod getting killed with OOM

Posted by Shiv Kumar Dixit <sh...@it.eurofinseu.com>.
Hi Arthur/Thorsten,
I was exploring and analysing memory settings of broker container and came across few things. Some input from you guys might be useful.


  1.  As of now we have not explicitly defined -XX:MaxDirectMemorySize so it will be taking JVM default and can be a potential reason of OOM.
  2.  Also, we have not explicitly defined -Dio.netty.maxDirectMemory and as broker heavily uses netty, it could be another potential reason of OOM.
  3.  We will try to fix both based on recommendations.
  4.  For testing, I set both min and max heap to 6000 MB and container request limit to 9000 MB, I checked following.



     *   Native memory summary and it came as below. Current committed max heap shows 6144000KB or 6000 MB and total committed JVM memory (heap + non-heap) shows 6560172KB which is more than max heap.

Total: reserved=7349741KB, committed=6666361KB

       malloc: 106189KB #580494

       mmap:   reserved=7243552KB, committed=6560172KB

Java Heap (reserved=6144000KB, committed=6144000KB)

                    (mmap: reserved=6144000KB, committed=6144000KB)



     *   I checked top container stat and it came as below. If top command gives total container memory usage (heap + non heap + container’s internal need), why it is less than native memory’s committed value? Ideally JVM memory usage (heap + non heap) should be less than current container’s memory usage, right?

k -n artemis-demo top pod demo-broker-ss-0 --containers
POD                                       NAME                    CPU(cores)   MEMORY(bytes)
demo-broker-ss-0  demo-broker-container   18m          6010Mi
demo-broker-ss-0    vault-agent                            1m              35Mi



Here container total memory comes as 6010Mi (6301942KB) which is less than total committed memory value (6666361KB) from native memory step. If this is feasible?



     *   I checked JMX memory in Artemis broker GUI, it came as below. Committed heap 6291456000 matches with max heap defined i.e. 6000 MB. Non heap committed is 139657216. Total committed JVM memory usage appears to be 6431113216 or 6280384 KB. This also does not match to total native memory committed value of 6666361KB from native memory step. However it is less than container’s total memory usage of 6010Mi (6301942KB).



Heap memory usage -> { "init": 6291456000, "committed": 6291456000, "max": 6291456000, "used": 4526118136 }
Non heap memory usage -> { "init": 7667712, "committed": 139657216, "max": 1224736768, "used": 134421688 }


                    Shall we expect all 3 results to be in matching range?

Best Regards
Shiv

From: Arthur Naseef <ar...@apache.org>
Sent: 15 January 2026 09:59 PM
To: users@artemis.apache.org
Subject: Re: K8s broker pod getting killed with OOM




Unverified Sender: The sender of this email has not been verified. Review the content of the message carefully and verify the identity of the sender before acting on this email: replying, opening attachments or clicking links.

Shiv:

I recommend you lookup OOMKilled in Kubernetes.  I did that myself and found a page explaining how Kubernetes nodes and PODs are allocated.

From what you are describing, I think Artemis is working as-intended, and the JVM is working correctly.  Artemis cannot use more heap than the JVM allows.  And the Linux OOM Killer (in the Linux Kernel itself) doesn't care about heap versus other types of memory used by the JVM - it only cares about the total amount of memory used by the JVM process - together with the total system memory and memory allocated by all processes.

Clebert has some advice on your tuning and settings - I recommend you follow his advice.  But I doubt that will stop the OOMKiller problems.

So 2 things I'd like to share here:

  1.  In your description of the test queue flow and broker log messages around paging - it all looks normal, and those log messages do NOT indicate a blocked consumer (which you suggested was the case in point #4).
  2.  My suspicion is that your Kubernetes PODs are overallocating the memory on their NODE and then when they go to use more memory than the node actually has available, the OOM Killer kicks off. (That's the heart of the OOM Killer - overallocation of memory when processes request it initially).
For #2, I saw an article explaining that Kubernetes does allow overallocation of PODS on a NODE.  It appears that it assigns PODS to NODES based on the "memory request" - which is a minimum requested memory size for the POD.  However, PODS are allowed to expand up to their memory limit, which can then cause the actual NODE system memory limit to be exceeded.

A simple way to verify my assertions would be to create a process that performs a fixed memory allocation of the total POD size, and make sure the process actually writes to all of those pages, to force the O/S to allocate physical pages.  Then see if that simple process leads to the OOM Killer at various sizes.  Note that I don't know of a way to write such a program in Java, but it would be a fairly simple C program.

Another test you could perform - use a small broker (say 2gb heap) on a good size POD (say 8gb of memory request) and see if you can invoke the OOM Killer.

Cheers!

Art


On Thu, Jan 15, 2026 at 9:08 AM Shiv Kumar Dixit <sh...@it.eurofinseu.com>> wrote:
We are using open source one from ArkMQ

Best Regards
Shiv

From: Clebert Suconic <cl...@gmail.com>>
Sent: 15 January 2026 08:26 PM
To: users@artemis.apache.org<ma...@artemis.apache.org>
Subject: Re: K8s broker pod getting killed with OOM




Unverified Sender: The sender of this email has not been verified. Review the content of the message carefully and verify the identity of the sender before acting on this email: replying, opening attachments or clicking links.

>>
We are using Artemis 2.37.0 version in K8s and Artemis IO operator version is 1.2.5.

Which one? the commercial version from Red Hat / openshift, or the opensource one from ArkMQ?

On Thu, Jan 15, 2026 at 9:51 AM Clebert Suconic <cl...@gmail.com>> wrote:
Those points you described are the reason why I suggesting using max-address-size on the every destination...


have max-size at say 20M for every destination (make 100K for small destinations if you like, but I think 20M for every destination is mostly okay, unless you have a lot of destinations).


have max-read at 10M...

This should then optimize your memory usage

On Thu, Jan 15, 2026 at 6:53 AM Shiv Kumar Dixit <sh...@it.eurofinseu.com>> wrote:
Hello Arthur and Clebert
When our broker pod starts, it first starts 2 init containers which terminate and release resources after completing the setup. So our pod basically runs 2 containers – one for vault and another for broker. We verified the memory and CPU usage of these init containers and main containers using top pod and it shows reasonable data.

Yes we see Linux OOMKiller is invoked and we are trying to read its report to see any meaningful information.

In the meanwhile, we have noticed below scenario is causing OOMKilling of broker container.

  1.  There are lot of pending messages on a given queue TEST along with small pending messages on various other queues. Since we are using global max size, some of messages are loaded in memory and rest are in paging folder.



  1.  There are 3-4 consumers on TEST queue but they are very slow hence pending message backlog is not cleared. We see below log in broker:

AMQ224127: Message dispatch from paging is blocked. Address TEST/Queue TEST will not read any more messages from paging until pending messages are acknowledged. There are currently 5150 messages pending (20972400 bytes) with max reads at maxPageReadMessages(-1) and maxPageReadBytes(20971520). Either increase reading attributes at the address-settings or change your consumers to acknowledge more often.



  1.  We also see below log in broker:

AMQ224108: Stopped paging on address ‘TEST’; size=62986496 bytes (96016 messages); maxSize=-1 bytes (-1 messages); globalSize=430581015 bytes (158406 messages); globalMaxSize=4194304000 bytes (-1 messages);



  1.  If such blocked consumers and pending messages combination can cause broker pod to go into OOM which is running with 30 GB of heap and 40 GB of pod memory?



  1.  Since consumers were not consuming messages on time and gave consent to purge the messages, we tried to purge the message manually via broker GUI. Sometimes it worked and more messages got loaded from pages to broker memory but many times broker pod got OOM and restarted.


  1.  This cycle of successful purge or broker restart continued till all messages from pages were loaded into memory and purged. Post cleanup there was no broker restart.



  1.  If purging messages via broker GUI can cause OOM even though broker pod is running with 30 GB of heap and 40 GB of pod memory?



  1.  What is the best way to optimize the broker configuration in such cases where we will always have slow consumers and possibly lot of pending messages in memory and paging folders?


This impacted broker pod A has a network bridge with another independent broker pod B in a hub and spoke model which has very less connection and almost no pending messages. We also noticed that if broker pod A goes into OOM due to slow consumer and pending messages as described above, second broker pod B which is connected over network bridge with first broker pod A also goes into restart loop with OOM. Does restart of source pod A and disconnection-reconnection of small numbers of bridges can cause target broker pod B to restart? We have seen this side effect as well.

We are using Artemis 2.37.0 version in K8s and Artemis IO operator version is 1.2.5.

Best Regards
Shiv

From: Arthur Naseef <ar...@apache.org>>
Sent: 15 January 2026 06:40 AM
To: users@artemis.apache.org<ma...@artemis.apache.org>
Subject: Re: K8s broker pod getting killed with OOM




Unverified Sender: The sender of this email has not been verified. Review the content of the message carefully and verify the identity of the sender before acting on this email: replying, opening attachments or clicking links.

So 3100 connections is a large number, but that doesn't sound like a good reason for the broker pod to go OOM.  Also, getting up to 40gb, I would say the 50% rule of thumb may be too conservative (i.e. a higher percentage could be reasonable), which is contradicted by your outcome.  Are there other containers running in the same POD that might be taking up memory?  Maybe sidecars?

Unfortunately, I don't have a working kubernetes setup available right now.  If I did, I could poke around and try to give specific tips on checking the memory use of the POD.

Do you know if the Linux OOM killer is getting invoked?  That would be reported by the kernel of the node on which the pod was executing.  If you can view that report, it includes a lot of useful information, including all of the processes involved and the amount of memory used by each.

Art


On Wed, Jan 14, 2026 at 3:52 PM Shiv Kumar Dixit <sh...@it.eurofinseu.com>> wrote:
Thanks Clebert and Arthur for inputs. I will try your suggestions and let you know how it goes.

I have another observation based on issue happening in live. Based on input from Arthur, current setup is configured with 20 GB heap and 40 GB pod. As the pod started, we got 3100 connections to broker and within minutes the pod got OOMKilled. If there is any relation b/w number of connections on broker and pod going OOM?

Best Regards
Shiv

-----Original Message-----
From: Clebert Suconic <cl...@gmail.com>>
Sent: 15 January 2026 04:06 AM
To: users@artemis.apache.org<ma...@artemis.apache.org>
Subject: Re: K8s broker pod getting killed with OOM



Unverified Sender: The sender of this email has not been verified. Review the content of the message carefully and verify the identity of the sender before acting on this email: replying, opening attachments or clicking links.


so, in summary, what I'm recommending you is:

use max-size-messages for all the queues.. for your large queues, use something like 10MB and for your small queues 100K

also keep max-read-page-bytes in use... keep it at 20M



If I could change the past I would have a max-size on every address we deploy, and having global-max-size for the upmost emergency case..
it's something I'm looking to change into artemis 3.0 or 4.0. (I can't change that into a minor version, as it could break certain cases...
as some users that I know use heavy filtering and can't really rely on paging).


On Wed, Jan 14, 2026 at 5:31 PM Clebert Suconic <cl...@gmail.com>> wrote:
>
> I would recommend against trusting global-max-size. and use max-size
> for all the addresses.
>
> Also what is your reading attributes. I would recommending using the
> new prefetch values.
>
>
>
> And also what operator are you using? arkmq? your own?
>
> On Wed, Jan 14, 2026 at 7:44 AM Shiv Kumar Dixit
> <sh...@it.eurofinseu.com>> wrote:
> >
> > We are hosting Artemis broker in Kubernetes using operator-based solution. We deploy the broker as statefulset with 2 or 4 replicas. We assign for e.g. 6 GB for heap and 9 GB for pod, 1.2 GB (1/5 of max heap) for global-max-size. All addresses normally use -1 for max-size-bytes but some less frequently used queues are defined with 100KB for max-size-bytes to allow early paging.
> >
> >
> >
> > We have following observations:
> >
> > 1. As the broker pod starts, broker container immediately occupies 6 GB for max heap. It seems expected as both min and max heap are same.
> >
> > 2. Pod memory usage starts with 6+ GB and once we have pending messages, good producers and consumers connect to broker, invalid SSL attempts happen, broker GUI access happens etc. during normal broker operations - pod memory usage keeps increasing and now reaches 9 GB.
> >
> > 3. Once the pod hits limit of 9 GB, K8s kills the pod with OOMKilling event and restarts the pod. Here we don’t see broker container getting killed with OOM rather pod is killed and restarted. It forces the broker to restart.
> >
> > 4. We have configured artemis.profile to capture memory dump in case of OOM of broker but it never happens. So, we are assuming broker process is not going out of memory, but pod is going out of memory due to increased non-heap usage.
> >
> > 5. Only way to recover here is to increase heap and pod memory limits from 6 GB and 9 GB to higher values and wait for next re-occurrence.
> >
> >
> >
> > 1. Is there any way to analyse what is going wrong with non-heap native memory usage?
> >
> > 2. If non-heap native memory is expected to increase to such extent due to pending messages, SSL errors etc.?
> >
> > 3. Is there any param we can use to restrict the non-heap native memory usage?
> >
> > 4. If netty which handles connection aspect of broker can create such memory consumption and cause OOM of pod?
> >
> > 5. Can we have any monitoring param that can hint that pod is potentially in danger of getting killed?
> >
> >
> >
> > Thanks
> >
> > Shiv
>
>
>
> --
> Clebert Suconic



--
Clebert Suconic

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org<ma...@artemis.apache.org>
For additional commands, e-mail: users-help@artemis.apache.org<ma...@artemis.apache.org>


--
Clebert Suconic


--
Clebert Suconic

Re: K8s broker pod getting killed with OOM

Posted by Arthur Naseef <ar...@apache.org>.
Shiv:

I recommend you lookup OOMKilled in Kubernetes.  I did that myself and
found a page explaining how Kubernetes nodes and PODs are allocated.

From what you are describing, I think Artemis is working as-intended, and
the JVM is working correctly.  Artemis cannot use more heap than the JVM
allows.  And the Linux OOM Killer (in the Linux Kernel itself) doesn't care
about heap versus other types of memory used by the JVM - it only cares
about the total amount of memory used by the JVM process - together with
the total system memory and memory allocated by all processes.

Clebert has some advice on your tuning and settings - I recommend you
follow his advice.  But I doubt that will stop the OOMKiller problems.

So 2 things I'd like to share here:

   1. In your description of the test queue flow and broker log messages
   around paging - it all looks normal, and those log messages do NOT indicate
   a blocked consumer (which you suggested was the case in point #4).
   2. My suspicion is that your Kubernetes PODs are overallocating the
   memory on their NODE and then when they go to use more memory than the node
   actually has available, the OOM Killer kicks off. (That's the heart of the
   OOM Killer - overallocation of memory when processes request it initially).

For #2, I saw an article explaining that Kubernetes does allow
overallocation of PODS on a NODE.  It appears that it assigns PODS to NODES
based on the "memory request" - which is a minimum requested memory size
for the POD.  However, PODS are allowed to expand up to their memory limit,
which can then cause the actual NODE system memory limit to be exceeded.

A simple way to verify my assertions would be to create a process that
performs a fixed memory allocation of the total POD size, and make sure the
process actually writes to all of those pages, to force the O/S to allocate
physical pages.  Then see if that simple process leads to the OOM Killer at
various sizes.  Note that I don't know of a way to write such a program in
Java, but it would be a fairly simple C program.

Another test you could perform - use a small broker (say 2gb heap) on a
good size POD (say 8gb of memory request) and see if you can invoke the OOM
Killer.

Cheers!

Art


On Thu, Jan 15, 2026 at 9:08 AM Shiv Kumar Dixit <
shivkumar.dixit@it.eurofinseu.com> wrote:

> We are using open source one from ArkMQ
>
>
>
> Best Regards
>
> Shiv
>
>
>
> *From:* Clebert Suconic <cl...@gmail.com>
> *Sent:* 15 January 2026 08:26 PM
> *To:* users@artemis.apache.org
> *Subject:* Re: K8s broker pod getting killed with OOM
>
>
>
>
>
> *Unverified Sender: *The sender of this email has not been verified.
> Review the content of the message carefully and verify the identity of the
> sender before acting on this email: replying, opening attachments or
> clicking links.
>
>
>
> >>
> We are using Artemis 2.37.0 version in K8s and Artemis IO operator version
> is 1.2.5.
>
>
>
> Which one? the commercial version from Red Hat / openshift, or the
> opensource one from ArkMQ?
>
>
>
> On Thu, Jan 15, 2026 at 9:51 AM Clebert Suconic <cl...@gmail.com>
> wrote:
>
> Those points you described are the reason why I suggesting using
> max-address-size on the every destination...
>
>
>
>
>
> have max-size at say 20M for every destination (make 100K for small
> destinations if you like, but I think 20M for every destination is mostly
> okay, unless you have a lot of destinations).
>
>
>
>
>
> have max-read at 10M...
>
>
>
> This should then optimize your memory usage
>
>
>
> On Thu, Jan 15, 2026 at 6:53 AM Shiv Kumar Dixit <
> shivkumar.dixit@it.eurofinseu.com> wrote:
>
> Hello Arthur and Clebert
>
> When our broker pod starts, it first starts 2 init containers which
> terminate and release resources after completing the setup. So our pod
> basically runs 2 containers – one for vault and another for broker. We
> verified the memory and CPU usage of these init containers and main
> containers using top pod and it shows reasonable data.
>
>
>
> Yes we see Linux OOMKiller is invoked and we are trying to read its report
> to see any meaningful information.
>
>
>
> In the meanwhile, we have noticed below scenario is causing OOMKilling of
> broker container.
>
>    1. There are lot of pending messages on a given queue TEST along with
>    small pending messages on various other queues. Since we are using global
>    max size, some of messages are loaded in memory and rest are in paging
>    folder.
>
>
>
>    2. There are 3-4 consumers on TEST queue but they are very slow hence
>    pending message backlog is not cleared. We see below log in broker:
>
> AMQ224127: Message dispatch from paging is blocked. Address TEST/Queue
> TEST will not read any more messages from paging until pending messages are
> acknowledged. There are currently 5150 messages pending (20972400 bytes)
> with max reads at maxPageReadMessages(-1) and maxPageReadBytes(20971520).
> Either increase reading attributes at the address-settings or change your
> consumers to acknowledge more often.
>
>
>
>    3. We also see below log in broker:
>
> AMQ224108: Stopped paging on address ‘TEST’; size=62986496 bytes (96016
> messages); maxSize=-1 bytes (-1 messages); globalSize=430581015 bytes
> (158406 messages); globalMaxSize=4194304000 bytes (-1 messages);
>
>
>
>    4. If such blocked consumers and pending messages combination can
>    cause broker pod to go into OOM which is running with 30 GB of heap and 40
>    GB of pod memory?
>
>
>
>    5. Since consumers were not consuming messages on time and gave
>    consent to purge the messages, we tried to purge the message manually via
>    broker GUI. Sometimes it worked and more messages got loaded from pages to
>    broker memory but many times broker pod got OOM and restarted.
>
>
>
>    6. This cycle of successful purge or broker restart continued till all
>    messages from pages were loaded into memory and purged. Post cleanup there
>    was no broker restart.
>
>
>
>    7. If purging messages via broker GUI can cause OOM even though broker
>    pod is running with 30 GB of heap and 40 GB of pod memory?
>
>
>
>    8. What is the best way to optimize the broker configuration in such
>    cases where we will always have slow consumers and possibly lot of pending
>    messages in memory and paging folders?
>
>
>
> This impacted broker pod A has a network bridge with another independent
> broker pod B in a hub and spoke model which has very less connection and
> almost no pending messages. We also noticed that if broker pod A goes into
> OOM due to slow consumer and pending messages as described above, second
> broker pod B which is connected over network bridge with first broker pod A
> also goes into restart loop with OOM. Does restart of source pod A and
> disconnection-reconnection of small numbers of bridges can cause target
> broker pod B to restart? We have seen this side effect as well.
>
>
>
> We are using Artemis 2.37.0 version in K8s and Artemis IO operator version
> is 1.2.5.
>
>
>
> Best Regards
>
> Shiv
>
>
>
> *From:* Arthur Naseef <ar...@apache.org>
> *Sent:* 15 January 2026 06:40 AM
> *To:* users@artemis.apache.org
> *Subject:* Re: K8s broker pod getting killed with OOM
>
>
>
>
>
> *Unverified Sender: *The sender of this email has not been verified.
> Review the content of the message carefully and verify the identity of the
> sender before acting on this email: replying, opening attachments or
> clicking links.
>
>
>
> So 3100 connections is a large number, but that doesn't sound like a good
> reason for the broker pod to go OOM.  Also, getting up to 40gb, I would say
> the 50% rule of thumb may be too conservative (i.e. a higher percentage
> could be reasonable), which is contradicted by your outcome.  Are there
> other containers running in the same POD that might be taking up memory?
> Maybe sidecars?
>
>
>
> Unfortunately, I don't have a working kubernetes setup available right
> now.  If I did, I could poke around and try to give specific tips on
> checking the memory use of the POD.
>
>
>
> Do you know if the Linux OOM killer is getting invoked?  That would be
> reported by the kernel of the node on which the pod was executing.  If you
> can view that report, it includes a lot of useful information, including
> all of the processes involved and the amount of memory used by each.
>
>
>
> Art
>
>
>
>
>
> On Wed, Jan 14, 2026 at 3:52 PM Shiv Kumar Dixit <
> shivkumar.dixit@it.eurofinseu.com> wrote:
>
> Thanks Clebert and Arthur for inputs. I will try your suggestions and let
> you know how it goes.
>
> I have another observation based on issue happening in live. Based on
> input from Arthur, current setup is configured with 20 GB heap and 40 GB
> pod. As the pod started, we got 3100 connections to broker and within
> minutes the pod got OOMKilled. If there is any relation b/w number of
> connections on broker and pod going OOM?
>
> Best Regards
> Shiv
>
> -----Original Message-----
> From: Clebert Suconic <cl...@gmail.com>
> Sent: 15 January 2026 04:06 AM
> To: users@artemis.apache.org
> Subject: Re: K8s broker pod getting killed with OOM
>
>
>
> Unverified Sender: The sender of this email has not been verified. Review
> the content of the message carefully and verify the identity of the sender
> before acting on this email: replying, opening attachments or clicking
> links.
>
>
> so, in summary, what I'm recommending you is:
>
> use max-size-messages for all the queues.. for your large queues, use
> something like 10MB and for your small queues 100K
>
> also keep max-read-page-bytes in use... keep it at 20M
>
>
>
> If I could change the past I would have a max-size on every address we
> deploy, and having global-max-size for the upmost emergency case..
> it's something I'm looking to change into artemis 3.0 or 4.0. (I can't
> change that into a minor version, as it could break certain cases...
> as some users that I know use heavy filtering and can't really rely on
> paging).
>
>
> On Wed, Jan 14, 2026 at 5:31 PM Clebert Suconic <cl...@gmail.com>
> wrote:
> >
> > I would recommend against trusting global-max-size. and use max-size
> > for all the addresses.
> >
> > Also what is your reading attributes. I would recommending using the
> > new prefetch values.
> >
> >
> >
> > And also what operator are you using? arkmq? your own?
> >
> > On Wed, Jan 14, 2026 at 7:44 AM Shiv Kumar Dixit
> > <sh...@it.eurofinseu.com> wrote:
> > >
> > > We are hosting Artemis broker in Kubernetes using operator-based
> solution. We deploy the broker as statefulset with 2 or 4 replicas. We
> assign for e.g. 6 GB for heap and 9 GB for pod, 1.2 GB (1/5 of max heap)
> for global-max-size. All addresses normally use -1 for max-size-bytes but
> some less frequently used queues are defined with 100KB for max-size-bytes
> to allow early paging.
> > >
> > >
> > >
> > > We have following observations:
> > >
> > > 1. As the broker pod starts, broker container immediately occupies 6
> GB for max heap. It seems expected as both min and max heap are same.
> > >
> > > 2. Pod memory usage starts with 6+ GB and once we have pending
> messages, good producers and consumers connect to broker, invalid SSL
> attempts happen, broker GUI access happens etc. during normal broker
> operations - pod memory usage keeps increasing and now reaches 9 GB.
> > >
> > > 3. Once the pod hits limit of 9 GB, K8s kills the pod with OOMKilling
> event and restarts the pod. Here we don’t see broker container getting
> killed with OOM rather pod is killed and restarted. It forces the broker to
> restart.
> > >
> > > 4. We have configured artemis.profile to capture memory dump in case
> of OOM of broker but it never happens. So, we are assuming broker process
> is not going out of memory, but pod is going out of memory due to increased
> non-heap usage.
> > >
> > > 5. Only way to recover here is to increase heap and pod memory limits
> from 6 GB and 9 GB to higher values and wait for next re-occurrence.
> > >
> > >
> > >
> > > 1. Is there any way to analyse what is going wrong with non-heap
> native memory usage?
> > >
> > > 2. If non-heap native memory is expected to increase to such extent
> due to pending messages, SSL errors etc.?
> > >
> > > 3. Is there any param we can use to restrict the non-heap native
> memory usage?
> > >
> > > 4. If netty which handles connection aspect of broker can create such
> memory consumption and cause OOM of pod?
> > >
> > > 5. Can we have any monitoring param that can hint that pod is
> potentially in danger of getting killed?
> > >
> > >
> > >
> > > Thanks
> > >
> > > Shiv
> >
> >
> >
> > --
> > Clebert Suconic
>
>
>
> --
> Clebert Suconic
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
> For additional commands, e-mail: users-help@artemis.apache.org
>
>
>
>
> --
>
> Clebert Suconic
>
>
>
>
> --
>
> Clebert Suconic
>

RE: K8s broker pod getting killed with OOM

Posted by Shiv Kumar Dixit <sh...@it.eurofinseu.com>.
We are using open source one from ArkMQ

Best Regards
Shiv

From: Clebert Suconic <cl...@gmail.com>
Sent: 15 January 2026 08:26 PM
To: users@artemis.apache.org
Subject: Re: K8s broker pod getting killed with OOM




Unverified Sender: The sender of this email has not been verified. Review the content of the message carefully and verify the identity of the sender before acting on this email: replying, opening attachments or clicking links.

>>
We are using Artemis 2.37.0 version in K8s and Artemis IO operator version is 1.2.5.

Which one? the commercial version from Red Hat / openshift, or the opensource one from ArkMQ?

On Thu, Jan 15, 2026 at 9:51 AM Clebert Suconic <cl...@gmail.com>> wrote:
Those points you described are the reason why I suggesting using max-address-size on the every destination...


have max-size at say 20M for every destination (make 100K for small destinations if you like, but I think 20M for every destination is mostly okay, unless you have a lot of destinations).


have max-read at 10M...

This should then optimize your memory usage

On Thu, Jan 15, 2026 at 6:53 AM Shiv Kumar Dixit <sh...@it.eurofinseu.com>> wrote:
Hello Arthur and Clebert
When our broker pod starts, it first starts 2 init containers which terminate and release resources after completing the setup. So our pod basically runs 2 containers – one for vault and another for broker. We verified the memory and CPU usage of these init containers and main containers using top pod and it shows reasonable data.

Yes we see Linux OOMKiller is invoked and we are trying to read its report to see any meaningful information.

In the meanwhile, we have noticed below scenario is causing OOMKilling of broker container.

  1.  There are lot of pending messages on a given queue TEST along with small pending messages on various other queues. Since we are using global max size, some of messages are loaded in memory and rest are in paging folder.



  1.  There are 3-4 consumers on TEST queue but they are very slow hence pending message backlog is not cleared. We see below log in broker:

AMQ224127: Message dispatch from paging is blocked. Address TEST/Queue TEST will not read any more messages from paging until pending messages are acknowledged. There are currently 5150 messages pending (20972400 bytes) with max reads at maxPageReadMessages(-1) and maxPageReadBytes(20971520). Either increase reading attributes at the address-settings or change your consumers to acknowledge more often.



  1.  We also see below log in broker:

AMQ224108: Stopped paging on address ‘TEST’; size=62986496 bytes (96016 messages); maxSize=-1 bytes (-1 messages); globalSize=430581015 bytes (158406 messages); globalMaxSize=4194304000 bytes (-1 messages);



  1.  If such blocked consumers and pending messages combination can cause broker pod to go into OOM which is running with 30 GB of heap and 40 GB of pod memory?



  1.  Since consumers were not consuming messages on time and gave consent to purge the messages, we tried to purge the message manually via broker GUI. Sometimes it worked and more messages got loaded from pages to broker memory but many times broker pod got OOM and restarted.


  1.  This cycle of successful purge or broker restart continued till all messages from pages were loaded into memory and purged. Post cleanup there was no broker restart.



  1.  If purging messages via broker GUI can cause OOM even though broker pod is running with 30 GB of heap and 40 GB of pod memory?



  1.  What is the best way to optimize the broker configuration in such cases where we will always have slow consumers and possibly lot of pending messages in memory and paging folders?


This impacted broker pod A has a network bridge with another independent broker pod B in a hub and spoke model which has very less connection and almost no pending messages. We also noticed that if broker pod A goes into OOM due to slow consumer and pending messages as described above, second broker pod B which is connected over network bridge with first broker pod A also goes into restart loop with OOM. Does restart of source pod A and disconnection-reconnection of small numbers of bridges can cause target broker pod B to restart? We have seen this side effect as well.

We are using Artemis 2.37.0 version in K8s and Artemis IO operator version is 1.2.5.

Best Regards
Shiv

From: Arthur Naseef <ar...@apache.org>>
Sent: 15 January 2026 06:40 AM
To: users@artemis.apache.org<ma...@artemis.apache.org>
Subject: Re: K8s broker pod getting killed with OOM




Unverified Sender: The sender of this email has not been verified. Review the content of the message carefully and verify the identity of the sender before acting on this email: replying, opening attachments or clicking links.

So 3100 connections is a large number, but that doesn't sound like a good reason for the broker pod to go OOM.  Also, getting up to 40gb, I would say the 50% rule of thumb may be too conservative (i.e. a higher percentage could be reasonable), which is contradicted by your outcome.  Are there other containers running in the same POD that might be taking up memory?  Maybe sidecars?

Unfortunately, I don't have a working kubernetes setup available right now.  If I did, I could poke around and try to give specific tips on checking the memory use of the POD.

Do you know if the Linux OOM killer is getting invoked?  That would be reported by the kernel of the node on which the pod was executing.  If you can view that report, it includes a lot of useful information, including all of the processes involved and the amount of memory used by each.

Art


On Wed, Jan 14, 2026 at 3:52 PM Shiv Kumar Dixit <sh...@it.eurofinseu.com>> wrote:
Thanks Clebert and Arthur for inputs. I will try your suggestions and let you know how it goes.

I have another observation based on issue happening in live. Based on input from Arthur, current setup is configured with 20 GB heap and 40 GB pod. As the pod started, we got 3100 connections to broker and within minutes the pod got OOMKilled. If there is any relation b/w number of connections on broker and pod going OOM?

Best Regards
Shiv

-----Original Message-----
From: Clebert Suconic <cl...@gmail.com>>
Sent: 15 January 2026 04:06 AM
To: users@artemis.apache.org<ma...@artemis.apache.org>
Subject: Re: K8s broker pod getting killed with OOM



Unverified Sender: The sender of this email has not been verified. Review the content of the message carefully and verify the identity of the sender before acting on this email: replying, opening attachments or clicking links.


so, in summary, what I'm recommending you is:

use max-size-messages for all the queues.. for your large queues, use something like 10MB and for your small queues 100K

also keep max-read-page-bytes in use... keep it at 20M



If I could change the past I would have a max-size on every address we deploy, and having global-max-size for the upmost emergency case..
it's something I'm looking to change into artemis 3.0 or 4.0. (I can't change that into a minor version, as it could break certain cases...
as some users that I know use heavy filtering and can't really rely on paging).


On Wed, Jan 14, 2026 at 5:31 PM Clebert Suconic <cl...@gmail.com>> wrote:
>
> I would recommend against trusting global-max-size. and use max-size
> for all the addresses.
>
> Also what is your reading attributes. I would recommending using the
> new prefetch values.
>
>
>
> And also what operator are you using? arkmq? your own?
>
> On Wed, Jan 14, 2026 at 7:44 AM Shiv Kumar Dixit
> <sh...@it.eurofinseu.com>> wrote:
> >
> > We are hosting Artemis broker in Kubernetes using operator-based solution. We deploy the broker as statefulset with 2 or 4 replicas. We assign for e.g. 6 GB for heap and 9 GB for pod, 1.2 GB (1/5 of max heap) for global-max-size. All addresses normally use -1 for max-size-bytes but some less frequently used queues are defined with 100KB for max-size-bytes to allow early paging.
> >
> >
> >
> > We have following observations:
> >
> > 1. As the broker pod starts, broker container immediately occupies 6 GB for max heap. It seems expected as both min and max heap are same.
> >
> > 2. Pod memory usage starts with 6+ GB and once we have pending messages, good producers and consumers connect to broker, invalid SSL attempts happen, broker GUI access happens etc. during normal broker operations - pod memory usage keeps increasing and now reaches 9 GB.
> >
> > 3. Once the pod hits limit of 9 GB, K8s kills the pod with OOMKilling event and restarts the pod. Here we don’t see broker container getting killed with OOM rather pod is killed and restarted. It forces the broker to restart.
> >
> > 4. We have configured artemis.profile to capture memory dump in case of OOM of broker but it never happens. So, we are assuming broker process is not going out of memory, but pod is going out of memory due to increased non-heap usage.
> >
> > 5. Only way to recover here is to increase heap and pod memory limits from 6 GB and 9 GB to higher values and wait for next re-occurrence.
> >
> >
> >
> > 1. Is there any way to analyse what is going wrong with non-heap native memory usage?
> >
> > 2. If non-heap native memory is expected to increase to such extent due to pending messages, SSL errors etc.?
> >
> > 3. Is there any param we can use to restrict the non-heap native memory usage?
> >
> > 4. If netty which handles connection aspect of broker can create such memory consumption and cause OOM of pod?
> >
> > 5. Can we have any monitoring param that can hint that pod is potentially in danger of getting killed?
> >
> >
> >
> > Thanks
> >
> > Shiv
>
>
>
> --
> Clebert Suconic



--
Clebert Suconic

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org<ma...@artemis.apache.org>
For additional commands, e-mail: users-help@artemis.apache.org<ma...@artemis.apache.org>


--
Clebert Suconic


--
Clebert Suconic

Re: K8s broker pod getting killed with OOM

Posted by Clebert Suconic <cl...@gmail.com>.
>>
We are using Artemis 2.37.0 version in K8s and Artemis IO operator version
is 1.2.5.

Which one? the commercial version from Red Hat / openshift, or the
opensource one from ArkMQ?

On Thu, Jan 15, 2026 at 9:51 AM Clebert Suconic <cl...@gmail.com>
wrote:

> Those points you described are the reason why I suggesting using
> max-address-size on the every destination...
>
>
> have max-size at say 20M for every destination (make 100K for small
> destinations if you like, but I think 20M for every destination is mostly
> okay, unless you have a lot of destinations).
>
>
> have max-read at 10M...
>
> This should then optimize your memory usage
>
> On Thu, Jan 15, 2026 at 6:53 AM Shiv Kumar Dixit <
> shivkumar.dixit@it.eurofinseu.com> wrote:
>
>> Hello Arthur and Clebert
>>
>> When our broker pod starts, it first starts 2 init containers which
>> terminate and release resources after completing the setup. So our pod
>> basically runs 2 containers – one for vault and another for broker. We
>> verified the memory and CPU usage of these init containers and main
>> containers using top pod and it shows reasonable data.
>>
>>
>>
>> Yes we see Linux OOMKiller is invoked and we are trying to read its
>> report to see any meaningful information.
>>
>>
>>
>> In the meanwhile, we have noticed below scenario is causing OOMKilling of
>> broker container.
>>
>>    1. There are lot of pending messages on a given queue TEST along with
>>    small pending messages on various other queues. Since we are using global
>>    max size, some of messages are loaded in memory and rest are in paging
>>    folder.
>>
>>
>>
>>    2. There are 3-4 consumers on TEST queue but they are very slow hence
>>    pending message backlog is not cleared. We see below log in broker:
>>
>> AMQ224127: Message dispatch from paging is blocked. Address TEST/Queue
>> TEST will not read any more messages from paging until pending messages are
>> acknowledged. There are currently 5150 messages pending (20972400 bytes)
>> with max reads at maxPageReadMessages(-1) and maxPageReadBytes(20971520).
>> Either increase reading attributes at the address-settings or change your
>> consumers to acknowledge more often.
>>
>>
>>
>>    3. We also see below log in broker:
>>
>> AMQ224108: Stopped paging on address ‘TEST’; size=62986496 bytes (96016
>> messages); maxSize=-1 bytes (-1 messages); globalSize=430581015 bytes
>> (158406 messages); globalMaxSize=4194304000 bytes (-1 messages);
>>
>>
>>
>>    4. If such blocked consumers and pending messages combination can
>>    cause broker pod to go into OOM which is running with 30 GB of heap and 40
>>    GB of pod memory?
>>
>>
>>
>>    5. Since consumers were not consuming messages on time and gave
>>    consent to purge the messages, we tried to purge the message manually via
>>    broker GUI. Sometimes it worked and more messages got loaded from pages to
>>    broker memory but many times broker pod got OOM and restarted.
>>
>>
>>
>>    6. This cycle of successful purge or broker restart continued till
>>    all messages from pages were loaded into memory and purged. Post cleanup
>>    there was no broker restart.
>>
>>
>>
>>    7. If purging messages via broker GUI can cause OOM even though
>>    broker pod is running with 30 GB of heap and 40 GB of pod memory?
>>
>>
>>
>>    8. What is the best way to optimize the broker configuration in such
>>    cases where we will always have slow consumers and possibly lot of pending
>>    messages in memory and paging folders?
>>
>>
>>
>> This impacted broker pod A has a network bridge with another independent
>> broker pod B in a hub and spoke model which has very less connection and
>> almost no pending messages. We also noticed that if broker pod A goes into
>> OOM due to slow consumer and pending messages as described above, second
>> broker pod B which is connected over network bridge with first broker pod A
>> also goes into restart loop with OOM. Does restart of source pod A and
>> disconnection-reconnection of small numbers of bridges can cause target
>> broker pod B to restart? We have seen this side effect as well.
>>
>>
>>
>> We are using Artemis 2.37.0 version in K8s and Artemis IO operator
>> version is 1.2.5.
>>
>>
>>
>> Best Regards
>>
>> Shiv
>>
>>
>>
>> *From:* Arthur Naseef <ar...@apache.org>
>> *Sent:* 15 January 2026 06:40 AM
>> *To:* users@artemis.apache.org
>> *Subject:* Re: K8s broker pod getting killed with OOM
>>
>>
>>
>>
>>
>> *Unverified Sender: *The sender of this email has not been verified.
>> Review the content of the message carefully and verify the identity of the
>> sender before acting on this email: replying, opening attachments or
>> clicking links.
>>
>>
>>
>> So 3100 connections is a large number, but that doesn't sound like a good
>> reason for the broker pod to go OOM.  Also, getting up to 40gb, I would say
>> the 50% rule of thumb may be too conservative (i.e. a higher percentage
>> could be reasonable), which is contradicted by your outcome.  Are there
>> other containers running in the same POD that might be taking up memory?
>> Maybe sidecars?
>>
>>
>>
>> Unfortunately, I don't have a working kubernetes setup available right
>> now.  If I did, I could poke around and try to give specific tips on
>> checking the memory use of the POD.
>>
>>
>>
>> Do you know if the Linux OOM killer is getting invoked?  That would be
>> reported by the kernel of the node on which the pod was executing.  If you
>> can view that report, it includes a lot of useful information, including
>> all of the processes involved and the amount of memory used by each.
>>
>>
>>
>> Art
>>
>>
>>
>>
>>
>> On Wed, Jan 14, 2026 at 3:52 PM Shiv Kumar Dixit <
>> shivkumar.dixit@it.eurofinseu.com> wrote:
>>
>> Thanks Clebert and Arthur for inputs. I will try your suggestions and let
>> you know how it goes.
>>
>> I have another observation based on issue happening in live. Based on
>> input from Arthur, current setup is configured with 20 GB heap and 40 GB
>> pod. As the pod started, we got 3100 connections to broker and within
>> minutes the pod got OOMKilled. If there is any relation b/w number of
>> connections on broker and pod going OOM?
>>
>> Best Regards
>> Shiv
>>
>> -----Original Message-----
>> From: Clebert Suconic <cl...@gmail.com>
>> Sent: 15 January 2026 04:06 AM
>> To: users@artemis.apache.org
>> Subject: Re: K8s broker pod getting killed with OOM
>>
>>
>>
>> Unverified Sender: The sender of this email has not been verified. Review
>> the content of the message carefully and verify the identity of the sender
>> before acting on this email: replying, opening attachments or clicking
>> links.
>>
>>
>> so, in summary, what I'm recommending you is:
>>
>> use max-size-messages for all the queues.. for your large queues, use
>> something like 10MB and for your small queues 100K
>>
>> also keep max-read-page-bytes in use... keep it at 20M
>>
>>
>>
>> If I could change the past I would have a max-size on every address we
>> deploy, and having global-max-size for the upmost emergency case..
>> it's something I'm looking to change into artemis 3.0 or 4.0. (I can't
>> change that into a minor version, as it could break certain cases...
>> as some users that I know use heavy filtering and can't really rely on
>> paging).
>>
>>
>> On Wed, Jan 14, 2026 at 5:31 PM Clebert Suconic <
>> clebert.suconic@gmail.com> wrote:
>> >
>> > I would recommend against trusting global-max-size. and use max-size
>> > for all the addresses.
>> >
>> > Also what is your reading attributes. I would recommending using the
>> > new prefetch values.
>> >
>> >
>> >
>> > And also what operator are you using? arkmq? your own?
>> >
>> > On Wed, Jan 14, 2026 at 7:44 AM Shiv Kumar Dixit
>> > <sh...@it.eurofinseu.com> wrote:
>> > >
>> > > We are hosting Artemis broker in Kubernetes using operator-based
>> solution. We deploy the broker as statefulset with 2 or 4 replicas. We
>> assign for e.g. 6 GB for heap and 9 GB for pod, 1.2 GB (1/5 of max heap)
>> for global-max-size. All addresses normally use -1 for max-size-bytes but
>> some less frequently used queues are defined with 100KB for max-size-bytes
>> to allow early paging.
>> > >
>> > >
>> > >
>> > > We have following observations:
>> > >
>> > > 1. As the broker pod starts, broker container immediately occupies 6
>> GB for max heap. It seems expected as both min and max heap are same.
>> > >
>> > > 2. Pod memory usage starts with 6+ GB and once we have pending
>> messages, good producers and consumers connect to broker, invalid SSL
>> attempts happen, broker GUI access happens etc. during normal broker
>> operations - pod memory usage keeps increasing and now reaches 9 GB.
>> > >
>> > > 3. Once the pod hits limit of 9 GB, K8s kills the pod with OOMKilling
>> event and restarts the pod. Here we don’t see broker container getting
>> killed with OOM rather pod is killed and restarted. It forces the broker to
>> restart.
>> > >
>> > > 4. We have configured artemis.profile to capture memory dump in case
>> of OOM of broker but it never happens. So, we are assuming broker process
>> is not going out of memory, but pod is going out of memory due to increased
>> non-heap usage.
>> > >
>> > > 5. Only way to recover here is to increase heap and pod memory limits
>> from 6 GB and 9 GB to higher values and wait for next re-occurrence.
>> > >
>> > >
>> > >
>> > > 1. Is there any way to analyse what is going wrong with non-heap
>> native memory usage?
>> > >
>> > > 2. If non-heap native memory is expected to increase to such extent
>> due to pending messages, SSL errors etc.?
>> > >
>> > > 3. Is there any param we can use to restrict the non-heap native
>> memory usage?
>> > >
>> > > 4. If netty which handles connection aspect of broker can create such
>> memory consumption and cause OOM of pod?
>> > >
>> > > 5. Can we have any monitoring param that can hint that pod is
>> potentially in danger of getting killed?
>> > >
>> > >
>> > >
>> > > Thanks
>> > >
>> > > Shiv
>> >
>> >
>> >
>> > --
>> > Clebert Suconic
>>
>>
>>
>> --
>> Clebert Suconic
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
>> For additional commands, e-mail: users-help@artemis.apache.org
>>
>>
>
> --
> Clebert Suconic
>


-- 
Clebert Suconic

Re: K8s broker pod getting killed with OOM

Posted by Clebert Suconic <cl...@gmail.com>.
Those points you described are the reason why I suggesting using
max-address-size on the every destination...


have max-size at say 20M for every destination (make 100K for small
destinations if you like, but I think 20M for every destination is mostly
okay, unless you have a lot of destinations).


have max-read at 10M...

This should then optimize your memory usage

On Thu, Jan 15, 2026 at 6:53 AM Shiv Kumar Dixit <
shivkumar.dixit@it.eurofinseu.com> wrote:

> Hello Arthur and Clebert
>
> When our broker pod starts, it first starts 2 init containers which
> terminate and release resources after completing the setup. So our pod
> basically runs 2 containers – one for vault and another for broker. We
> verified the memory and CPU usage of these init containers and main
> containers using top pod and it shows reasonable data.
>
>
>
> Yes we see Linux OOMKiller is invoked and we are trying to read its report
> to see any meaningful information.
>
>
>
> In the meanwhile, we have noticed below scenario is causing OOMKilling of
> broker container.
>
>    1. There are lot of pending messages on a given queue TEST along with
>    small pending messages on various other queues. Since we are using global
>    max size, some of messages are loaded in memory and rest are in paging
>    folder.
>
>
>
>    2. There are 3-4 consumers on TEST queue but they are very slow hence
>    pending message backlog is not cleared. We see below log in broker:
>
> AMQ224127: Message dispatch from paging is blocked. Address TEST/Queue
> TEST will not read any more messages from paging until pending messages are
> acknowledged. There are currently 5150 messages pending (20972400 bytes)
> with max reads at maxPageReadMessages(-1) and maxPageReadBytes(20971520).
> Either increase reading attributes at the address-settings or change your
> consumers to acknowledge more often.
>
>
>
>    3. We also see below log in broker:
>
> AMQ224108: Stopped paging on address ‘TEST’; size=62986496 bytes (96016
> messages); maxSize=-1 bytes (-1 messages); globalSize=430581015 bytes
> (158406 messages); globalMaxSize=4194304000 bytes (-1 messages);
>
>
>
>    4. If such blocked consumers and pending messages combination can
>    cause broker pod to go into OOM which is running with 30 GB of heap and 40
>    GB of pod memory?
>
>
>
>    5. Since consumers were not consuming messages on time and gave
>    consent to purge the messages, we tried to purge the message manually via
>    broker GUI. Sometimes it worked and more messages got loaded from pages to
>    broker memory but many times broker pod got OOM and restarted.
>
>
>
>    6. This cycle of successful purge or broker restart continued till all
>    messages from pages were loaded into memory and purged. Post cleanup there
>    was no broker restart.
>
>
>
>    7. If purging messages via broker GUI can cause OOM even though broker
>    pod is running with 30 GB of heap and 40 GB of pod memory?
>
>
>
>    8. What is the best way to optimize the broker configuration in such
>    cases where we will always have slow consumers and possibly lot of pending
>    messages in memory and paging folders?
>
>
>
> This impacted broker pod A has a network bridge with another independent
> broker pod B in a hub and spoke model which has very less connection and
> almost no pending messages. We also noticed that if broker pod A goes into
> OOM due to slow consumer and pending messages as described above, second
> broker pod B which is connected over network bridge with first broker pod A
> also goes into restart loop with OOM. Does restart of source pod A and
> disconnection-reconnection of small numbers of bridges can cause target
> broker pod B to restart? We have seen this side effect as well.
>
>
>
> We are using Artemis 2.37.0 version in K8s and Artemis IO operator version
> is 1.2.5.
>
>
>
> Best Regards
>
> Shiv
>
>
>
> *From:* Arthur Naseef <ar...@apache.org>
> *Sent:* 15 January 2026 06:40 AM
> *To:* users@artemis.apache.org
> *Subject:* Re: K8s broker pod getting killed with OOM
>
>
>
>
>
> *Unverified Sender: *The sender of this email has not been verified.
> Review the content of the message carefully and verify the identity of the
> sender before acting on this email: replying, opening attachments or
> clicking links.
>
>
>
> So 3100 connections is a large number, but that doesn't sound like a good
> reason for the broker pod to go OOM.  Also, getting up to 40gb, I would say
> the 50% rule of thumb may be too conservative (i.e. a higher percentage
> could be reasonable), which is contradicted by your outcome.  Are there
> other containers running in the same POD that might be taking up memory?
> Maybe sidecars?
>
>
>
> Unfortunately, I don't have a working kubernetes setup available right
> now.  If I did, I could poke around and try to give specific tips on
> checking the memory use of the POD.
>
>
>
> Do you know if the Linux OOM killer is getting invoked?  That would be
> reported by the kernel of the node on which the pod was executing.  If you
> can view that report, it includes a lot of useful information, including
> all of the processes involved and the amount of memory used by each.
>
>
>
> Art
>
>
>
>
>
> On Wed, Jan 14, 2026 at 3:52 PM Shiv Kumar Dixit <
> shivkumar.dixit@it.eurofinseu.com> wrote:
>
> Thanks Clebert and Arthur for inputs. I will try your suggestions and let
> you know how it goes.
>
> I have another observation based on issue happening in live. Based on
> input from Arthur, current setup is configured with 20 GB heap and 40 GB
> pod. As the pod started, we got 3100 connections to broker and within
> minutes the pod got OOMKilled. If there is any relation b/w number of
> connections on broker and pod going OOM?
>
> Best Regards
> Shiv
>
> -----Original Message-----
> From: Clebert Suconic <cl...@gmail.com>
> Sent: 15 January 2026 04:06 AM
> To: users@artemis.apache.org
> Subject: Re: K8s broker pod getting killed with OOM
>
>
>
> Unverified Sender: The sender of this email has not been verified. Review
> the content of the message carefully and verify the identity of the sender
> before acting on this email: replying, opening attachments or clicking
> links.
>
>
> so, in summary, what I'm recommending you is:
>
> use max-size-messages for all the queues.. for your large queues, use
> something like 10MB and for your small queues 100K
>
> also keep max-read-page-bytes in use... keep it at 20M
>
>
>
> If I could change the past I would have a max-size on every address we
> deploy, and having global-max-size for the upmost emergency case..
> it's something I'm looking to change into artemis 3.0 or 4.0. (I can't
> change that into a minor version, as it could break certain cases...
> as some users that I know use heavy filtering and can't really rely on
> paging).
>
>
> On Wed, Jan 14, 2026 at 5:31 PM Clebert Suconic <cl...@gmail.com>
> wrote:
> >
> > I would recommend against trusting global-max-size. and use max-size
> > for all the addresses.
> >
> > Also what is your reading attributes. I would recommending using the
> > new prefetch values.
> >
> >
> >
> > And also what operator are you using? arkmq? your own?
> >
> > On Wed, Jan 14, 2026 at 7:44 AM Shiv Kumar Dixit
> > <sh...@it.eurofinseu.com> wrote:
> > >
> > > We are hosting Artemis broker in Kubernetes using operator-based
> solution. We deploy the broker as statefulset with 2 or 4 replicas. We
> assign for e.g. 6 GB for heap and 9 GB for pod, 1.2 GB (1/5 of max heap)
> for global-max-size. All addresses normally use -1 for max-size-bytes but
> some less frequently used queues are defined with 100KB for max-size-bytes
> to allow early paging.
> > >
> > >
> > >
> > > We have following observations:
> > >
> > > 1. As the broker pod starts, broker container immediately occupies 6
> GB for max heap. It seems expected as both min and max heap are same.
> > >
> > > 2. Pod memory usage starts with 6+ GB and once we have pending
> messages, good producers and consumers connect to broker, invalid SSL
> attempts happen, broker GUI access happens etc. during normal broker
> operations - pod memory usage keeps increasing and now reaches 9 GB.
> > >
> > > 3. Once the pod hits limit of 9 GB, K8s kills the pod with OOMKilling
> event and restarts the pod. Here we don’t see broker container getting
> killed with OOM rather pod is killed and restarted. It forces the broker to
> restart.
> > >
> > > 4. We have configured artemis.profile to capture memory dump in case
> of OOM of broker but it never happens. So, we are assuming broker process
> is not going out of memory, but pod is going out of memory due to increased
> non-heap usage.
> > >
> > > 5. Only way to recover here is to increase heap and pod memory limits
> from 6 GB and 9 GB to higher values and wait for next re-occurrence.
> > >
> > >
> > >
> > > 1. Is there any way to analyse what is going wrong with non-heap
> native memory usage?
> > >
> > > 2. If non-heap native memory is expected to increase to such extent
> due to pending messages, SSL errors etc.?
> > >
> > > 3. Is there any param we can use to restrict the non-heap native
> memory usage?
> > >
> > > 4. If netty which handles connection aspect of broker can create such
> memory consumption and cause OOM of pod?
> > >
> > > 5. Can we have any monitoring param that can hint that pod is
> potentially in danger of getting killed?
> > >
> > >
> > >
> > > Thanks
> > >
> > > Shiv
> >
> >
> >
> > --
> > Clebert Suconic
>
>
>
> --
> Clebert Suconic
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
> For additional commands, e-mail: users-help@artemis.apache.org
>
>

-- 
Clebert Suconic

RE: K8s broker pod getting killed with OOM

Posted by Shiv Kumar Dixit <sh...@it.eurofinseu.com>.
Hello Arthur and Clebert
When our broker pod starts, it first starts 2 init containers which terminate and release resources after completing the setup. So our pod basically runs 2 containers – one for vault and another for broker. We verified the memory and CPU usage of these init containers and main containers using top pod and it shows reasonable data.

Yes we see Linux OOMKiller is invoked and we are trying to read its report to see any meaningful information.

In the meanwhile, we have noticed below scenario is causing OOMKilling of broker container.

  1.  There are lot of pending messages on a given queue TEST along with small pending messages on various other queues. Since we are using global max size, some of messages are loaded in memory and rest are in paging folder.



  1.  There are 3-4 consumers on TEST queue but they are very slow hence pending message backlog is not cleared. We see below log in broker:

AMQ224127: Message dispatch from paging is blocked. Address TEST/Queue TEST will not read any more messages from paging until pending messages are acknowledged. There are currently 5150 messages pending (20972400 bytes) with max reads at maxPageReadMessages(-1) and maxPageReadBytes(20971520). Either increase reading attributes at the address-settings or change your consumers to acknowledge more often.



  1.  We also see below log in broker:

AMQ224108: Stopped paging on address ‘TEST’; size=62986496 bytes (96016 messages); maxSize=-1 bytes (-1 messages); globalSize=430581015 bytes (158406 messages); globalMaxSize=4194304000 bytes (-1 messages);



  1.  If such blocked consumers and pending messages combination can cause broker pod to go into OOM which is running with 30 GB of heap and 40 GB of pod memory?



  1.  Since consumers were not consuming messages on time and gave consent to purge the messages, we tried to purge the message manually via broker GUI. Sometimes it worked and more messages got loaded from pages to broker memory but many times broker pod got OOM and restarted.


  1.  This cycle of successful purge or broker restart continued till all messages from pages were loaded into memory and purged. Post cleanup there was no broker restart.



  1.  If purging messages via broker GUI can cause OOM even though broker pod is running with 30 GB of heap and 40 GB of pod memory?



  1.  What is the best way to optimize the broker configuration in such cases where we will always have slow consumers and possibly lot of pending messages in memory and paging folders?


This impacted broker pod A has a network bridge with another independent broker pod B in a hub and spoke model which has very less connection and almost no pending messages. We also noticed that if broker pod A goes into OOM due to slow consumer and pending messages as described above, second broker pod B which is connected over network bridge with first broker pod A also goes into restart loop with OOM. Does restart of source pod A and disconnection-reconnection of small numbers of bridges can cause target broker pod B to restart? We have seen this side effect as well.

We are using Artemis 2.37.0 version in K8s and Artemis IO operator version is 1.2.5.

Best Regards
Shiv

From: Arthur Naseef <ar...@apache.org>
Sent: 15 January 2026 06:40 AM
To: users@artemis.apache.org
Subject: Re: K8s broker pod getting killed with OOM




Unverified Sender: The sender of this email has not been verified. Review the content of the message carefully and verify the identity of the sender before acting on this email: replying, opening attachments or clicking links.

So 3100 connections is a large number, but that doesn't sound like a good reason for the broker pod to go OOM.  Also, getting up to 40gb, I would say the 50% rule of thumb may be too conservative (i.e. a higher percentage could be reasonable), which is contradicted by your outcome.  Are there other containers running in the same POD that might be taking up memory?  Maybe sidecars?

Unfortunately, I don't have a working kubernetes setup available right now.  If I did, I could poke around and try to give specific tips on checking the memory use of the POD.

Do you know if the Linux OOM killer is getting invoked?  That would be reported by the kernel of the node on which the pod was executing.  If you can view that report, it includes a lot of useful information, including all of the processes involved and the amount of memory used by each.

Art


On Wed, Jan 14, 2026 at 3:52 PM Shiv Kumar Dixit <sh...@it.eurofinseu.com>> wrote:
Thanks Clebert and Arthur for inputs. I will try your suggestions and let you know how it goes.

I have another observation based on issue happening in live. Based on input from Arthur, current setup is configured with 20 GB heap and 40 GB pod. As the pod started, we got 3100 connections to broker and within minutes the pod got OOMKilled. If there is any relation b/w number of connections on broker and pod going OOM?

Best Regards
Shiv

-----Original Message-----
From: Clebert Suconic <cl...@gmail.com>>
Sent: 15 January 2026 04:06 AM
To: users@artemis.apache.org<ma...@artemis.apache.org>
Subject: Re: K8s broker pod getting killed with OOM



Unverified Sender: The sender of this email has not been verified. Review the content of the message carefully and verify the identity of the sender before acting on this email: replying, opening attachments or clicking links.


so, in summary, what I'm recommending you is:

use max-size-messages for all the queues.. for your large queues, use something like 10MB and for your small queues 100K

also keep max-read-page-bytes in use... keep it at 20M



If I could change the past I would have a max-size on every address we deploy, and having global-max-size for the upmost emergency case..
it's something I'm looking to change into artemis 3.0 or 4.0. (I can't change that into a minor version, as it could break certain cases...
as some users that I know use heavy filtering and can't really rely on paging).


On Wed, Jan 14, 2026 at 5:31 PM Clebert Suconic <cl...@gmail.com>> wrote:
>
> I would recommend against trusting global-max-size. and use max-size
> for all the addresses.
>
> Also what is your reading attributes. I would recommending using the
> new prefetch values.
>
>
>
> And also what operator are you using? arkmq? your own?
>
> On Wed, Jan 14, 2026 at 7:44 AM Shiv Kumar Dixit
> <sh...@it.eurofinseu.com>> wrote:
> >
> > We are hosting Artemis broker in Kubernetes using operator-based solution. We deploy the broker as statefulset with 2 or 4 replicas. We assign for e.g. 6 GB for heap and 9 GB for pod, 1.2 GB (1/5 of max heap) for global-max-size. All addresses normally use -1 for max-size-bytes but some less frequently used queues are defined with 100KB for max-size-bytes to allow early paging.
> >
> >
> >
> > We have following observations:
> >
> > 1. As the broker pod starts, broker container immediately occupies 6 GB for max heap. It seems expected as both min and max heap are same.
> >
> > 2. Pod memory usage starts with 6+ GB and once we have pending messages, good producers and consumers connect to broker, invalid SSL attempts happen, broker GUI access happens etc. during normal broker operations - pod memory usage keeps increasing and now reaches 9 GB.
> >
> > 3. Once the pod hits limit of 9 GB, K8s kills the pod with OOMKilling event and restarts the pod. Here we don’t see broker container getting killed with OOM rather pod is killed and restarted. It forces the broker to restart.
> >
> > 4. We have configured artemis.profile to capture memory dump in case of OOM of broker but it never happens. So, we are assuming broker process is not going out of memory, but pod is going out of memory due to increased non-heap usage.
> >
> > 5. Only way to recover here is to increase heap and pod memory limits from 6 GB and 9 GB to higher values and wait for next re-occurrence.
> >
> >
> >
> > 1. Is there any way to analyse what is going wrong with non-heap native memory usage?
> >
> > 2. If non-heap native memory is expected to increase to such extent due to pending messages, SSL errors etc.?
> >
> > 3. Is there any param we can use to restrict the non-heap native memory usage?
> >
> > 4. If netty which handles connection aspect of broker can create such memory consumption and cause OOM of pod?
> >
> > 5. Can we have any monitoring param that can hint that pod is potentially in danger of getting killed?
> >
> >
> >
> > Thanks
> >
> > Shiv
>
>
>
> --
> Clebert Suconic



--
Clebert Suconic

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org<ma...@artemis.apache.org>
For additional commands, e-mail: users-help@artemis.apache.org<ma...@artemis.apache.org>

Re: K8s broker pod getting killed with OOM

Posted by Arthur Naseef <ar...@apache.org>.
So 3100 connections is a large number, but that doesn't sound like a good
reason for the broker pod to go OOM.  Also, getting up to 40gb, I would say
the 50% rule of thumb may be too conservative (i.e. a higher percentage
could be reasonable), which is contradicted by your outcome.  Are there
other containers running in the same POD that might be taking up memory?
Maybe sidecars?

Unfortunately, I don't have a working kubernetes setup available right
now.  If I did, I could poke around and try to give specific tips on
checking the memory use of the POD.

Do you know if the Linux OOM killer is getting invoked?  That would be
reported by the kernel of the node on which the pod was executing.  If you
can view that report, it includes a lot of useful information, including
all of the processes involved and the amount of memory used by each.

Art


On Wed, Jan 14, 2026 at 3:52 PM Shiv Kumar Dixit <
shivkumar.dixit@it.eurofinseu.com> wrote:

> Thanks Clebert and Arthur for inputs. I will try your suggestions and let
> you know how it goes.
>
> I have another observation based on issue happening in live. Based on
> input from Arthur, current setup is configured with 20 GB heap and 40 GB
> pod. As the pod started, we got 3100 connections to broker and within
> minutes the pod got OOMKilled. If there is any relation b/w number of
> connections on broker and pod going OOM?
>
> Best Regards
> Shiv
>
> -----Original Message-----
> From: Clebert Suconic <cl...@gmail.com>
> Sent: 15 January 2026 04:06 AM
> To: users@artemis.apache.org
> Subject: Re: K8s broker pod getting killed with OOM
>
>
>
> Unverified Sender: The sender of this email has not been verified. Review
> the content of the message carefully and verify the identity of the sender
> before acting on this email: replying, opening attachments or clicking
> links.
>
>
> so, in summary, what I'm recommending you is:
>
> use max-size-messages for all the queues.. for your large queues, use
> something like 10MB and for your small queues 100K
>
> also keep max-read-page-bytes in use... keep it at 20M
>
>
>
> If I could change the past I would have a max-size on every address we
> deploy, and having global-max-size for the upmost emergency case..
> it's something I'm looking to change into artemis 3.0 or 4.0. (I can't
> change that into a minor version, as it could break certain cases...
> as some users that I know use heavy filtering and can't really rely on
> paging).
>
>
> On Wed, Jan 14, 2026 at 5:31 PM Clebert Suconic <cl...@gmail.com>
> wrote:
> >
> > I would recommend against trusting global-max-size. and use max-size
> > for all the addresses.
> >
> > Also what is your reading attributes. I would recommending using the
> > new prefetch values.
> >
> >
> >
> > And also what operator are you using? arkmq? your own?
> >
> > On Wed, Jan 14, 2026 at 7:44 AM Shiv Kumar Dixit
> > <sh...@it.eurofinseu.com> wrote:
> > >
> > > We are hosting Artemis broker in Kubernetes using operator-based
> solution. We deploy the broker as statefulset with 2 or 4 replicas. We
> assign for e.g. 6 GB for heap and 9 GB for pod, 1.2 GB (1/5 of max heap)
> for global-max-size. All addresses normally use -1 for max-size-bytes but
> some less frequently used queues are defined with 100KB for max-size-bytes
> to allow early paging.
> > >
> > >
> > >
> > > We have following observations:
> > >
> > > 1. As the broker pod starts, broker container immediately occupies 6
> GB for max heap. It seems expected as both min and max heap are same.
> > >
> > > 2. Pod memory usage starts with 6+ GB and once we have pending
> messages, good producers and consumers connect to broker, invalid SSL
> attempts happen, broker GUI access happens etc. during normal broker
> operations - pod memory usage keeps increasing and now reaches 9 GB.
> > >
> > > 3. Once the pod hits limit of 9 GB, K8s kills the pod with OOMKilling
> event and restarts the pod. Here we don’t see broker container getting
> killed with OOM rather pod is killed and restarted. It forces the broker to
> restart.
> > >
> > > 4. We have configured artemis.profile to capture memory dump in case
> of OOM of broker but it never happens. So, we are assuming broker process
> is not going out of memory, but pod is going out of memory due to increased
> non-heap usage.
> > >
> > > 5. Only way to recover here is to increase heap and pod memory limits
> from 6 GB and 9 GB to higher values and wait for next re-occurrence.
> > >
> > >
> > >
> > > 1. Is there any way to analyse what is going wrong with non-heap
> native memory usage?
> > >
> > > 2. If non-heap native memory is expected to increase to such extent
> due to pending messages, SSL errors etc.?
> > >
> > > 3. Is there any param we can use to restrict the non-heap native
> memory usage?
> > >
> > > 4. If netty which handles connection aspect of broker can create such
> memory consumption and cause OOM of pod?
> > >
> > > 5. Can we have any monitoring param that can hint that pod is
> potentially in danger of getting killed?
> > >
> > >
> > >
> > > Thanks
> > >
> > > Shiv
> >
> >
> >
> > --
> > Clebert Suconic
>
>
>
> --
> Clebert Suconic
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
> For additional commands, e-mail: users-help@artemis.apache.org
>
>

RE: K8s broker pod getting killed with OOM

Posted by Shiv Kumar Dixit <sh...@it.eurofinseu.com>.
Thanks Clebert and Arthur for inputs. I will try your suggestions and let you know how it goes.

I have another observation based on issue happening in live. Based on input from Arthur, current setup is configured with 20 GB heap and 40 GB pod. As the pod started, we got 3100 connections to broker and within minutes the pod got OOMKilled. If there is any relation b/w number of connections on broker and pod going OOM?

Best Regards
Shiv

-----Original Message-----
From: Clebert Suconic <cl...@gmail.com> 
Sent: 15 January 2026 04:06 AM
To: users@artemis.apache.org
Subject: Re: K8s broker pod getting killed with OOM



Unverified Sender: The sender of this email has not been verified. Review the content of the message carefully and verify the identity of the sender before acting on this email: replying, opening attachments or clicking links.


so, in summary, what I'm recommending you is:

use max-size-messages for all the queues.. for your large queues, use something like 10MB and for your small queues 100K

also keep max-read-page-bytes in use... keep it at 20M



If I could change the past I would have a max-size on every address we deploy, and having global-max-size for the upmost emergency case..
it's something I'm looking to change into artemis 3.0 or 4.0. (I can't change that into a minor version, as it could break certain cases...
as some users that I know use heavy filtering and can't really rely on paging).


On Wed, Jan 14, 2026 at 5:31 PM Clebert Suconic <cl...@gmail.com> wrote:
>
> I would recommend against trusting global-max-size. and use max-size 
> for all the addresses.
>
> Also what is your reading attributes. I would recommending using the 
> new prefetch values.
>
>
>
> And also what operator are you using? arkmq? your own?
>
> On Wed, Jan 14, 2026 at 7:44 AM Shiv Kumar Dixit 
> <sh...@it.eurofinseu.com> wrote:
> >
> > We are hosting Artemis broker in Kubernetes using operator-based solution. We deploy the broker as statefulset with 2 or 4 replicas. We assign for e.g. 6 GB for heap and 9 GB for pod, 1.2 GB (1/5 of max heap) for global-max-size. All addresses normally use -1 for max-size-bytes but some less frequently used queues are defined with 100KB for max-size-bytes to allow early paging.
> >
> >
> >
> > We have following observations:
> >
> > 1. As the broker pod starts, broker container immediately occupies 6 GB for max heap. It seems expected as both min and max heap are same.
> >
> > 2. Pod memory usage starts with 6+ GB and once we have pending messages, good producers and consumers connect to broker, invalid SSL attempts happen, broker GUI access happens etc. during normal broker operations - pod memory usage keeps increasing and now reaches 9 GB.
> >
> > 3. Once the pod hits limit of 9 GB, K8s kills the pod with OOMKilling event and restarts the pod. Here we don’t see broker container getting killed with OOM rather pod is killed and restarted. It forces the broker to restart.
> >
> > 4. We have configured artemis.profile to capture memory dump in case of OOM of broker but it never happens. So, we are assuming broker process is not going out of memory, but pod is going out of memory due to increased non-heap usage.
> >
> > 5. Only way to recover here is to increase heap and pod memory limits from 6 GB and 9 GB to higher values and wait for next re-occurrence.
> >
> >
> >
> > 1. Is there any way to analyse what is going wrong with non-heap native memory usage?
> >
> > 2. If non-heap native memory is expected to increase to such extent due to pending messages, SSL errors etc.?
> >
> > 3. Is there any param we can use to restrict the non-heap native memory usage?
> >
> > 4. If netty which handles connection aspect of broker can create such memory consumption and cause OOM of pod?
> >
> > 5. Can we have any monitoring param that can hint that pod is potentially in danger of getting killed?
> >
> >
> >
> > Thanks
> >
> > Shiv
>
>
>
> --
> Clebert Suconic



--
Clebert Suconic

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
For additional commands, e-mail: users-help@artemis.apache.org

Re: K8s broker pod getting killed with OOM

Posted by Clebert Suconic <cl...@gmail.com>.
so, in summary, what I'm recommending you is:

use max-size-messages for all the queues.. for your large queues, use
something like 10MB and for your small queues 100K

also keep max-read-page-bytes in use... keep it at 20M



If I could change the past I would have a max-size on every address we
deploy, and having global-max-size for the upmost emergency case..
it's something I'm looking to change into artemis 3.0 or 4.0. (I can't
change that into a minor version, as it could break certain cases...
as some users that I know use heavy filtering and can't really rely on
paging).


On Wed, Jan 14, 2026 at 5:31 PM Clebert Suconic
<cl...@gmail.com> wrote:
>
> I would recommend against trusting global-max-size. and use max-size
> for all the addresses.
>
> Also what is your reading attributes. I would recommending using the
> new prefetch values.
>
>
>
> And also what operator are you using? arkmq? your own?
>
> On Wed, Jan 14, 2026 at 7:44 AM Shiv Kumar Dixit
> <sh...@it.eurofinseu.com> wrote:
> >
> > We are hosting Artemis broker in Kubernetes using operator-based solution. We deploy the broker as statefulset with 2 or 4 replicas. We assign for e.g. 6 GB for heap and 9 GB for pod, 1.2 GB (1/5 of max heap) for global-max-size. All addresses normally use -1 for max-size-bytes but some less frequently used queues are defined with 100KB for max-size-bytes to allow early paging.
> >
> >
> >
> > We have following observations:
> >
> > 1. As the broker pod starts, broker container immediately occupies 6 GB for max heap. It seems expected as both min and max heap are same.
> >
> > 2. Pod memory usage starts with 6+ GB and once we have pending messages, good producers and consumers connect to broker, invalid SSL attempts happen, broker GUI access happens etc. during normal broker operations - pod memory usage keeps increasing and now reaches 9 GB.
> >
> > 3. Once the pod hits limit of 9 GB, K8s kills the pod with OOMKilling event and restarts the pod. Here we don’t see broker container getting killed with OOM rather pod is killed and restarted. It forces the broker to restart.
> >
> > 4. We have configured artemis.profile to capture memory dump in case of OOM of broker but it never happens. So, we are assuming broker process is not going out of memory, but pod is going out of memory due to increased non-heap usage.
> >
> > 5. Only way to recover here is to increase heap and pod memory limits from 6 GB and 9 GB to higher values and wait for next re-occurrence.
> >
> >
> >
> > 1. Is there any way to analyse what is going wrong with non-heap native memory usage?
> >
> > 2. If non-heap native memory is expected to increase to such extent due to pending messages, SSL errors etc.?
> >
> > 3. Is there any param we can use to restrict the non-heap native memory usage?
> >
> > 4. If netty which handles connection aspect of broker can create such memory consumption and cause OOM of pod?
> >
> > 5. Can we have any monitoring param that can hint that pod is potentially in danger of getting killed?
> >
> >
> >
> > Thanks
> >
> > Shiv
>
>
>
> --
> Clebert Suconic



-- 
Clebert Suconic

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
For additional commands, e-mail: users-help@artemis.apache.org

Re: K8s broker pod getting killed with OOM

Posted by Clebert Suconic <cl...@gmail.com>.
I would recommend against trusting global-max-size. and use max-size
for all the addresses.

Also what is your reading attributes. I would recommending using the
new prefetch values.



And also what operator are you using? arkmq? your own?

On Wed, Jan 14, 2026 at 7:44 AM Shiv Kumar Dixit
<sh...@it.eurofinseu.com> wrote:
>
> We are hosting Artemis broker in Kubernetes using operator-based solution. We deploy the broker as statefulset with 2 or 4 replicas. We assign for e.g. 6 GB for heap and 9 GB for pod, 1.2 GB (1/5 of max heap) for global-max-size. All addresses normally use -1 for max-size-bytes but some less frequently used queues are defined with 100KB for max-size-bytes to allow early paging.
>
>
>
> We have following observations:
>
> 1. As the broker pod starts, broker container immediately occupies 6 GB for max heap. It seems expected as both min and max heap are same.
>
> 2. Pod memory usage starts with 6+ GB and once we have pending messages, good producers and consumers connect to broker, invalid SSL attempts happen, broker GUI access happens etc. during normal broker operations - pod memory usage keeps increasing and now reaches 9 GB.
>
> 3. Once the pod hits limit of 9 GB, K8s kills the pod with OOMKilling event and restarts the pod. Here we don’t see broker container getting killed with OOM rather pod is killed and restarted. It forces the broker to restart.
>
> 4. We have configured artemis.profile to capture memory dump in case of OOM of broker but it never happens. So, we are assuming broker process is not going out of memory, but pod is going out of memory due to increased non-heap usage.
>
> 5. Only way to recover here is to increase heap and pod memory limits from 6 GB and 9 GB to higher values and wait for next re-occurrence.
>
>
>
> 1. Is there any way to analyse what is going wrong with non-heap native memory usage?
>
> 2. If non-heap native memory is expected to increase to such extent due to pending messages, SSL errors etc.?
>
> 3. Is there any param we can use to restrict the non-heap native memory usage?
>
> 4. If netty which handles connection aspect of broker can create such memory consumption and cause OOM of pod?
>
> 5. Can we have any monitoring param that can hint that pod is potentially in danger of getting killed?
>
>
>
> Thanks
>
> Shiv



-- 
Clebert Suconic

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@artemis.apache.org
For additional commands, e-mail: users-help@artemis.apache.org

Re: K8s broker pod getting killed with OOM

Posted by Arthur Naseef <ar...@apache.org>.
Based on your description, it sounds like the pod may be too restrictive in
its total system memory.  The general rule-of-thumb I like to use is that
the broker's max heap size should be no more than 50% of the total system
memory available.

So for a POD with 9GB of memory, that would mean a max heap of 4.5GB.

As for measuring the total system memory, there are ways to do so, although
kubernetes complicates the picture a good amount.  You would need to look
at all of the processes running inside of the pod and the memory allocated
by each, and understanding the difference between RSS, and other categories
of process memory.  And also note that part of the reason for the 50% rule
is that the JVM itself uses a lot more memory than heap, so a JVM with a
6GB maximum heap size is going to use a lot more than 6GB of system memory.

One way you may be able to get more insight into the POD's memory usage is
to use the "kubectl exec" command to access the POD and use ps and other
tools to check memory usage.

BTW, even though the JVM allocates a 6GB heap initially, that doesn't
necessarily mean the O/S has assigned physical pages to all of that heap
(Java could force that allocation, but I don't know that it ever does).  If
that's correct, you could be simply running into the case that once the
actual heap use grows close to its limit (which java heaps are designed to
do normally all the time - hence the sawtooth memory-usage graphs), the
system physical page allocations increase, eventually hitting the POD
limit.  So it is entirely possible that the problem is just sizing of the
JVM and its heap.  One way you could test this hypothesis of heap usage and
physical pages would be to intentionally over-allocate a heap - for example
using -Xmx12g on a POD with a limit of 9gb of memory, and see if the broker
and POD fail quickly.

Hope this helps.

Art


On Wed, Jan 14, 2026 at 5:44 AM Shiv Kumar Dixit <
shivkumar.dixit@it.eurofinseu.com> wrote:

> We are hosting Artemis broker in Kubernetes using operator-based solution.
> We deploy the broker as statefulset with 2 or 4 replicas. We assign for
> e.g. 6 GB for heap and 9 GB for pod, 1.2 GB (1/5 of max heap) for
> global-max-size. All addresses normally use -1 for max-size-bytes but some
> less frequently used queues are defined with 100KB for max-size-bytes to
> allow early paging.
>
>
>
> We have following observations:
>
> 1. As the broker pod starts, broker container immediately occupies 6 GB
> for max heap. It seems expected as both min and max heap are same.
>
> 2. Pod memory usage starts with 6+ GB and once we have pending messages,
> good producers and consumers connect to broker, invalid SSL attempts
> happen, broker GUI access happens etc. during normal broker operations -
> pod memory usage keeps increasing and now reaches 9 GB.
>
> 3. Once the pod hits limit of 9 GB, K8s kills the pod with OOMKilling
> event and restarts the pod. Here we don’t see broker container getting
> killed with OOM rather pod is killed and restarted. It forces the broker to
> restart.
>
> 4. We have configured artemis.profile to capture memory dump in case of
> OOM of broker but it never happens. So, we are assuming broker process is
> not going out of memory, but pod is going out of memory due to increased
> non-heap usage.
>
> 5. Only way to recover here is to increase heap and pod memory limits from
> 6 GB and 9 GB to higher values and wait for next re-occurrence.
>
>
>
> 1. Is there any way to analyse what is going wrong with non-heap native
> memory usage?
>
> 2. If non-heap native memory is expected to increase to such extent due to
> pending messages, SSL errors etc.?
>
> 3. Is there any param we can use to restrict the non-heap native memory
> usage?
>
> 4. If netty which handles connection aspect of broker can create such
> memory consumption and cause OOM of pod?
>
> 5. Can we have any monitoring param that can hint that pod is potentially
> in danger of getting killed?
>
>
>
> Thanks
>
> Shiv
>