You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by "Joel Ferreira (Nokia) via user" <us...@ignite.apache.org> on 2026/01/19 10:21:29 UTC
Memory issue on kubernetes
Hi,
In our project, we implemented an in-memory cache, based on Apache Ignite 3.1, running on Kubernetes. The Ignite cluster is configured with 3 replicas.
We are noticing that each of the 3 pods has a memory consumption that progressively increases until it reaches the configured limit, at which point a pod restart occurs.
For your reference, we are using the following configuration:
*
StatefulSet memory limit: 2Gi
*
3 replicas
*
JVM_MAX_MEM = JVM_MIN_MEM: 1Gi
*
In-memory table created with:
*
Zone with high availability
*
Profile using "aimem" engine
We also created a cronjob that keeps the number of entries in cache within a limit of 6000, but this has no effect on the memory stability of the pods.
In fact, we even tried clearing all entries from the cache table and did not observe any memory release from the pods. On the contrary, consumption continued to rise slowly.
Is there any known memory issue that could explain this behavior? Or could there be some setting we missed?
Thanks in advance.
Kind regards,
Joel Ferreira
Re: Memory issue on kubernetes
Posted by Pavel Tupitsyn <pt...@apache.org>.
maxSizeBytes is "256M or 20% of physical memory", try setting it explicitly.
Overall I think 2G is just too low for Ignite to work efficiently. You
have 400M offheap, 1G heap, some JVM and OS overhead, it is too tight.
I would start with at least 8G. Production use cases are usually 32G+ per node.
[1] https://ignite.apache.org/docs/ignite3/latest/administrators-guide/storage/engines/aimem#profile-configuration
On Mon, Jan 19, 2026 at 5:09 PM Joel Ferreira (Nokia) via user
<us...@ignite.apache.org> wrote:
>
> "Please share aimem configuration - have you set aimem.maxSizeBytes?"
>
> We did not set "maxSizeBytes", delegating it to the default value, which is "268435456" bytes (256M).
> Here is our aimem config:
> storage {
> profiles = [
> {
> name: pmfeProfile,
> engine: aimem
> }
> ]
> }
Re: Memory issue on kubernetes
Posted by "Joel Ferreira (Nokia) via user" <us...@ignite.apache.org>.
"Please share aimem configuration - have you set aimem.maxSizeBytes?"
We did not set "maxSizeBytes", delegating it to the default value, which is "268435456" bytes (256M).
Here is our aimem config:
storage {
profiles = [
{
name: pmfeProfile,
engine: aimem
}
]
}
Re: Memory issue on kubernetes
Posted by Pavel Tupitsyn <pt...@apache.org>.
> "aimem" engine
Please share aimem configuration - have you set aimem.maxSizeBytes?
On Mon, Jan 19, 2026 at 2:09 PM Joel Ferreira (Nokia) via user
<us...@ignite.apache.org> wrote:
>
> "Hi, are there any exceptions in the node log that lead to the pod restart?"
>
> Hi.
> Kubernetes restarts the pod when the 2Gi memory limit is reached - we see an OOMKill error with exit code 137.
> We don't see any exceptions in Ignite node logs.
>
Re: Memory issue on kubernetes
Posted by "Joel Ferreira (Nokia) via user" <us...@ignite.apache.org>.
"Hi, are there any exceptions in the node log that lead to the pod restart?"
Hi.
Kubernetes restarts the pod when the 2Gi memory limit is reached - we see an OOMKill error with exit code 137.
We don't see any exceptions in Ignite node logs.
Re: Memory issue on kubernetes
Posted by Pavel Tupitsyn <pt...@apache.org>.
Hi, are there any exceptions in the node log that lead to the pod restart?
On Mon, Jan 19, 2026 at 12:22 PM Joel Ferreira (Nokia) via user
<us...@ignite.apache.org> wrote:
>
> Hi,
>
> In our project, we implemented an in-memory cache, based on Apache Ignite 3.1, running on Kubernetes. The Ignite cluster is configured with 3 replicas.
> We are noticing that each of the 3 pods has a memory consumption that progressively increases until it reaches the configured limit, at which point a pod restart occurs.
>
> For your reference, we are using the following configuration:
>
> StatefulSet memory limit: 2Gi
> 3 replicas
> JVM_MAX_MEM = JVM_MIN_MEM: 1Gi
> In-memory table created with:
>
> Zone with high availability
> Profile using "aimem" engine
>
>
> We also created a cronjob that keeps the number of entries in cache within a limit of 6000, but this has no effect on the memory stability of the pods.
> In fact, we even tried clearing all entries from the cache table and did not observe any memory release from the pods. On the contrary, consumption continued to rise slowly.
> Is there any known memory issue that could explain this behavior? Or could there be some setting we missed?
>
> Thanks in advance.
> Kind regards,
> Joel Ferreira
Re: Memory issue on kubernetes
Posted by Aleksandr Polovtsev <al...@gmail.com>.
Hello, Ignite 3 uses the MVCC approach for transaction isolation, which
means that multiple versions of data (including removed data) is stored for
a configured period of time, see [1]. I can suggest changing the
"dataAvailabilityTimeMillis" property, see [2].
[1]
https://ignite.apache.org/docs/ignite3/latest/administrators-guide/storage/data-partitions#version-storage
[2]
https://ignite.apache.org/docs/ignite3/latest/administrators-guide/config/cluster-config#garbage-collection-configuration
On Mon, Jan 19, 2026 at 12:22 PM Joel Ferreira (Nokia) via user <
user@ignite.apache.org> wrote:
> Hi,
>
> In our project, we implemented an in-memory cache, based on Apache Ignite
> 3.1, running on Kubernetes. The Ignite cluster is configured with 3
> replicas.
> We are noticing that each of the 3 pods has a memory consumption that
> progressively increases until it reaches the configured limit, at which
> point a pod restart occurs.
>
> For your reference, we are using the following configuration:
>
> - StatefulSet memory limit: 2Gi
> - 3 replicas
> - JVM_MAX_MEM = JVM_MIN_MEM: 1Gi
> - In-memory table created with:
> - Zone with high availability
> - Profile using "aimem" engine
>
>
> We also created a cronjob that keeps the number of entries in cache within
> a limit of 6000, but this has no effect on the memory stability of the pods.
> In fact, we even tried clearing all entries from the cache table and did
> not observe any memory release from the pods. On the contrary, consumption
> continued to rise slowly.
> Is there any known memory issue that could explain this behavior? Or could
> there be some setting we missed?
>
> Thanks in advance.
> Kind regards,
> Joel Ferreira
>
--
With regards,
Aleksandr Polovtsev