You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Vilius Šumskas <vi...@rivile.lt> on 2025/01/01 18:16:22 UTC
backup not activated after OOM on primary
Hi,
we had an incident where our applications sent too much traffic to Artemis broker and the broker got Java Heap Out Of Memory errors. I’m trying to understand why backup broker never became primary after this happened.
We run Artemis static cluster with two nodes (primary and backup) under Shared Storage. Configuration is pretty straightforward:
<ha-policy>
<shared-store>
<primary>
<failover-on-shutdown>true</failover-on-shutdown>
</primary>
</shared-store>
</ha-policy>
<ha-policy>
<shared-store>
<backup>
<failover-on-shutdown>true</failover-on-shutdown>
</backup>
</shared-store>
</ha-policy>
Backup always becomes primary if we reboot primary during maintenance. We also tested our HA configuration with various other tests, like disabling network connections, killing storage mount point, etc, so I’m positive configuration should be correct.
Primary logs during that time: https://p.defau.lt/?cNVdPPEMN2qM8XbLZomKdQ
Backup logs during that time: https://p.defau.lt/?nJvfQKdc4rrUGC9lL7_JCA
OOM have happened at ~2:51 and the primary was in this state until I have restarted it ~13:25.
Any pointers are much appreciated!
--
Best Regards,
Vilius Šumskas
Rivile
IT manager
RE: backup not activated after OOM on primary
Posted by Vilius Šumskas <vi...@rivile.lt>.
Thank you! This is exactly what I wanted to know.
--
Vilius
-----Original Message-----
From: Justin Bertram <jb...@apache.org>
Sent: Monday, January 6, 2025 9:07 PM
To: users@activemq.apache.org
Subject: Re: backup not activated after OOM on primary
I'm not aware of any broker-specific dangers of adding
+ExitOnOutOfMemoryError.
It's worth noting that the broker is written knowing that at any point the JVM or OS could crash or there could be a hardware failure of some kind.
This is, in part, why transactions are implemented a particular way, why we only acknowledge durable messages once the date has been flushed to disk, etc. In short, there _should_ never be any message/journal corruption. If there was then it would be considered a bug and it would be fixed.
Justin
On Thu, Jan 2, 2025 at 12:21 AM Vilius Šumskas <vi...@rivile.lt>
wrote:
> I will rephrase my question. Are the any exact dangers by adding
> +ExitOnOutOfMemoryError? For example, message/journal corruption if
> +JVM
> shuts down abruptly because of OOME in one broker part of the broker,
> but another part is still working?
>
> --
> Vilius
>
> -----Original Message-----
> From: Justin Bertram <jb...@apache.org>
> Sent: Thursday, January 2, 2025 3:42 AM
> To: users@activemq.apache.org
> Subject: Re: backup not activated after OOM on primary
>
> The caveat with adding +ExitOnOutOfMemoryError is that the JVM will
> now exit when an OOME occurs now rather than simply carrying on. Keep
> in mind that an OOME isn't necessarily a death sentence in and of
> itself. It is technically possible (although unlikely) for the broker to recover.
>
> I believe it's not in the default artemis.profile because it was
> essentially brand new when Artemis 2.0 was released, and it hasn't
> been possible to add it and change the default behavior in a minor release.
>
>
> Justin
>
> On Wed, Jan 1, 2025 at 4:32 PM Vilius Šumskas
> <vi...@rivile.lt>
> wrote:
>
> > Are there any caveats adding +ExitOnOutOfMemoryError? Just wondering
> > why it's not in the default JAVA_ARGS in "artemis.profile".
> >
> > --
> > Vilius
> >
> > -----Original Message-----
> > From: Justin Bertram <jb...@apache.org>
> > Sent: Wednesday, January 1, 2025 11:01 PM
> > To: users@activemq.apache.org
> > Subject: Re: backup not activated after OOM on primary
> >
> > I think what you're seeing is expected. An OOME usually isn't enough
> > to trigger the broker to fail completely and trigger a failover. As
> > you can see, the broker continued to run after the OOME which means
> > it was still holding the lock on the shared journal (preventing the
> > backup from activating). If you want to ensure the broker fails over
> > in this situation you should pass this to the JVM:
> >
> > -XX:+ExitOnOutOfMemoryError
> >
> > This will ensure the JVM stops when an OOME occurs which will then
> > allow the backup to activate.
> >
> > It might also be worth passing this as well:
> >
> > -XX:+HeapDumpOnOutOfMemoryError
> >
> > This will allow you to do some post-mortem analysis and see exactly
> > why the OOME occurred.
> >
> >
> > Justin
> >
> > On Wed, Jan 1, 2025 at 12:17 PM Vilius Šumskas
> > <vi...@rivile.lt>
> > wrote:
> >
> > > Hi,
> > >
> > > we had an incident where our applications sent too much traffic to
> > > Artemis broker and the broker got Java Heap Out Of Memory errors.
> > > I’m trying to understand why backup broker never became primary
> > > after this
> > happened.
> > > We run Artemis static cluster with two nodes (primary and backup)
> > > under Shared Storage. Configuration is pretty straightforward:
> > > <ha-policy>
> > > <shared-store>
> > > <primary>
> > > <failover-on-shutdown>true</failover-on-shutdown>
> > > </primary>
> > > </shared-store>
> > > </ha-policy>
> > >
> > > <ha-policy>
> > > <shared-store>
> > > <backup>
> > > <failover-on-shutdown>true</failover-on-shutdown>
> > > </backup>
> > > </shared-store>
> > > </ha-policy>
> > >
> > > Backup always becomes primary if we reboot primary during maintenance.
> > > We also tested our HA configuration with various other tests, like
> > > disabling network connections, killing storage mount point, etc,
> > > so I’m positive configuration should be correct.
> > >
> > > Primary logs during that time:
> > > https://p.defau.lt/?cNVdPPEMN2qM8XbLZomKdQ
> > > Backup logs during that time:
> > > https://p.defau.lt/?nJvfQKdc4rrUGC9lL7_JCA
> > >
> > > OOM have happened at ~2:51 and the primary was in this state until
> > > I have restarted it ~13:25.
> > >
> > > Any pointers are much appreciated!
> > >
> > > --
> > > Best Regards,
> > >
> > > Vilius Šumskas
> > > Rivile
> > > IT manager
> > >
> > >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> > For additional commands, e-mail: users-help@activemq.apache.org For
> > further information, visit: https://activemq.apache.org/contact
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> For additional commands, e-mail: users-help@activemq.apache.org For
> further information, visit: https://activemq.apache.org/contact
>
>
Re: backup not activated after OOM on primary
Posted by Justin Bertram <jb...@apache.org>.
I'm not aware of any broker-specific dangers of adding
+ExitOnOutOfMemoryError.
It's worth noting that the broker is written knowing that at any point the
JVM or OS could crash or there could be a hardware failure of some kind.
This is, in part, why transactions are implemented a particular way, why we
only acknowledge durable messages once the date has been flushed to disk,
etc. In short, there _should_ never be any message/journal corruption. If
there was then it would be considered a bug and it would be fixed.
Justin
On Thu, Jan 2, 2025 at 12:21 AM Vilius Šumskas <vi...@rivile.lt>
wrote:
> I will rephrase my question. Are the any exact dangers by adding
> +ExitOnOutOfMemoryError? For example, message/journal corruption if JVM
> shuts down abruptly because of OOME in one broker part of the broker, but
> another part is still working?
>
> --
> Vilius
>
> -----Original Message-----
> From: Justin Bertram <jb...@apache.org>
> Sent: Thursday, January 2, 2025 3:42 AM
> To: users@activemq.apache.org
> Subject: Re: backup not activated after OOM on primary
>
> The caveat with adding +ExitOnOutOfMemoryError is that the JVM will now
> exit when an OOME occurs now rather than simply carrying on. Keep in mind
> that an OOME isn't necessarily a death sentence in and of itself. It is
> technically possible (although unlikely) for the broker to recover.
>
> I believe it's not in the default artemis.profile because it was
> essentially brand new when Artemis 2.0 was released, and it hasn't been
> possible to add it and change the default behavior in a minor release.
>
>
> Justin
>
> On Wed, Jan 1, 2025 at 4:32 PM Vilius Šumskas <vi...@rivile.lt>
> wrote:
>
> > Are there any caveats adding +ExitOnOutOfMemoryError? Just wondering
> > why it's not in the default JAVA_ARGS in "artemis.profile".
> >
> > --
> > Vilius
> >
> > -----Original Message-----
> > From: Justin Bertram <jb...@apache.org>
> > Sent: Wednesday, January 1, 2025 11:01 PM
> > To: users@activemq.apache.org
> > Subject: Re: backup not activated after OOM on primary
> >
> > I think what you're seeing is expected. An OOME usually isn't enough
> > to trigger the broker to fail completely and trigger a failover. As
> > you can see, the broker continued to run after the OOME which means it
> > was still holding the lock on the shared journal (preventing the
> > backup from activating). If you want to ensure the broker fails over
> > in this situation you should pass this to the JVM:
> >
> > -XX:+ExitOnOutOfMemoryError
> >
> > This will ensure the JVM stops when an OOME occurs which will then
> > allow the backup to activate.
> >
> > It might also be worth passing this as well:
> >
> > -XX:+HeapDumpOnOutOfMemoryError
> >
> > This will allow you to do some post-mortem analysis and see exactly
> > why the OOME occurred.
> >
> >
> > Justin
> >
> > On Wed, Jan 1, 2025 at 12:17 PM Vilius Šumskas
> > <vi...@rivile.lt>
> > wrote:
> >
> > > Hi,
> > >
> > > we had an incident where our applications sent too much traffic to
> > > Artemis broker and the broker got Java Heap Out Of Memory errors.
> > > I’m trying to understand why backup broker never became primary
> > > after this
> > happened.
> > > We run Artemis static cluster with two nodes (primary and backup)
> > > under Shared Storage. Configuration is pretty straightforward:
> > > <ha-policy>
> > > <shared-store>
> > > <primary>
> > > <failover-on-shutdown>true</failover-on-shutdown>
> > > </primary>
> > > </shared-store>
> > > </ha-policy>
> > >
> > > <ha-policy>
> > > <shared-store>
> > > <backup>
> > > <failover-on-shutdown>true</failover-on-shutdown>
> > > </backup>
> > > </shared-store>
> > > </ha-policy>
> > >
> > > Backup always becomes primary if we reboot primary during maintenance.
> > > We also tested our HA configuration with various other tests, like
> > > disabling network connections, killing storage mount point, etc, so
> > > I’m positive configuration should be correct.
> > >
> > > Primary logs during that time:
> > > https://p.defau.lt/?cNVdPPEMN2qM8XbLZomKdQ
> > > Backup logs during that time:
> > > https://p.defau.lt/?nJvfQKdc4rrUGC9lL7_JCA
> > >
> > > OOM have happened at ~2:51 and the primary was in this state until I
> > > have restarted it ~13:25.
> > >
> > > Any pointers are much appreciated!
> > >
> > > --
> > > Best Regards,
> > >
> > > Vilius Šumskas
> > > Rivile
> > > IT manager
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> > For additional commands, e-mail: users-help@activemq.apache.org For
> > further information, visit: https://activemq.apache.org/contact
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> For additional commands, e-mail: users-help@activemq.apache.org
> For further information, visit: https://activemq.apache.org/contact
>
>
RE: backup not activated after OOM on primary
Posted by Vilius Šumskas <vi...@rivile.lt>.
I will rephrase my question. Are the any exact dangers by adding +ExitOnOutOfMemoryError? For example, message/journal corruption if JVM shuts down abruptly because of OOME in one broker part of the broker, but another part is still working?
--
Vilius
-----Original Message-----
From: Justin Bertram <jb...@apache.org>
Sent: Thursday, January 2, 2025 3:42 AM
To: users@activemq.apache.org
Subject: Re: backup not activated after OOM on primary
The caveat with adding +ExitOnOutOfMemoryError is that the JVM will now exit when an OOME occurs now rather than simply carrying on. Keep in mind that an OOME isn't necessarily a death sentence in and of itself. It is technically possible (although unlikely) for the broker to recover.
I believe it's not in the default artemis.profile because it was essentially brand new when Artemis 2.0 was released, and it hasn't been possible to add it and change the default behavior in a minor release.
Justin
On Wed, Jan 1, 2025 at 4:32 PM Vilius Šumskas <vi...@rivile.lt>
wrote:
> Are there any caveats adding +ExitOnOutOfMemoryError? Just wondering
> why it's not in the default JAVA_ARGS in "artemis.profile".
>
> --
> Vilius
>
> -----Original Message-----
> From: Justin Bertram <jb...@apache.org>
> Sent: Wednesday, January 1, 2025 11:01 PM
> To: users@activemq.apache.org
> Subject: Re: backup not activated after OOM on primary
>
> I think what you're seeing is expected. An OOME usually isn't enough
> to trigger the broker to fail completely and trigger a failover. As
> you can see, the broker continued to run after the OOME which means it
> was still holding the lock on the shared journal (preventing the
> backup from activating). If you want to ensure the broker fails over
> in this situation you should pass this to the JVM:
>
> -XX:+ExitOnOutOfMemoryError
>
> This will ensure the JVM stops when an OOME occurs which will then
> allow the backup to activate.
>
> It might also be worth passing this as well:
>
> -XX:+HeapDumpOnOutOfMemoryError
>
> This will allow you to do some post-mortem analysis and see exactly
> why the OOME occurred.
>
>
> Justin
>
> On Wed, Jan 1, 2025 at 12:17 PM Vilius Šumskas
> <vi...@rivile.lt>
> wrote:
>
> > Hi,
> >
> > we had an incident where our applications sent too much traffic to
> > Artemis broker and the broker got Java Heap Out Of Memory errors.
> > I’m trying to understand why backup broker never became primary
> > after this
> happened.
> > We run Artemis static cluster with two nodes (primary and backup)
> > under Shared Storage. Configuration is pretty straightforward:
> > <ha-policy>
> > <shared-store>
> > <primary>
> > <failover-on-shutdown>true</failover-on-shutdown>
> > </primary>
> > </shared-store>
> > </ha-policy>
> >
> > <ha-policy>
> > <shared-store>
> > <backup>
> > <failover-on-shutdown>true</failover-on-shutdown>
> > </backup>
> > </shared-store>
> > </ha-policy>
> >
> > Backup always becomes primary if we reboot primary during maintenance.
> > We also tested our HA configuration with various other tests, like
> > disabling network connections, killing storage mount point, etc, so
> > I’m positive configuration should be correct.
> >
> > Primary logs during that time:
> > https://p.defau.lt/?cNVdPPEMN2qM8XbLZomKdQ
> > Backup logs during that time:
> > https://p.defau.lt/?nJvfQKdc4rrUGC9lL7_JCA
> >
> > OOM have happened at ~2:51 and the primary was in this state until I
> > have restarted it ~13:25.
> >
> > Any pointers are much appreciated!
> >
> > --
> > Best Regards,
> >
> > Vilius Šumskas
> > Rivile
> > IT manager
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> For additional commands, e-mail: users-help@activemq.apache.org For
> further information, visit: https://activemq.apache.org/contact
>
>
Re: backup not activated after OOM on primary
Posted by Justin Bertram <jb...@apache.org>.
The caveat with adding +ExitOnOutOfMemoryError is that the JVM will now
exit when an OOME occurs now rather than simply carrying on. Keep in mind
that an OOME isn't necessarily a death sentence in and of itself. It is
technically possible (although unlikely) for the broker to recover.
I believe it's not in the default artemis.profile because it was
essentially brand new when Artemis 2.0 was released, and it hasn't been
possible to add it and change the default behavior in a minor release.
Justin
On Wed, Jan 1, 2025 at 4:32 PM Vilius Šumskas <vi...@rivile.lt>
wrote:
> Are there any caveats adding +ExitOnOutOfMemoryError? Just wondering why
> it's not in the default JAVA_ARGS in "artemis.profile".
>
> --
> Vilius
>
> -----Original Message-----
> From: Justin Bertram <jb...@apache.org>
> Sent: Wednesday, January 1, 2025 11:01 PM
> To: users@activemq.apache.org
> Subject: Re: backup not activated after OOM on primary
>
> I think what you're seeing is expected. An OOME usually isn't enough to
> trigger the broker to fail completely and trigger a failover. As you can
> see, the broker continued to run after the OOME which means it was still
> holding the lock on the shared journal (preventing the backup from
> activating). If you want to ensure the broker fails over in this situation
> you should pass this to the JVM:
>
> -XX:+ExitOnOutOfMemoryError
>
> This will ensure the JVM stops when an OOME occurs which will then allow
> the backup to activate.
>
> It might also be worth passing this as well:
>
> -XX:+HeapDumpOnOutOfMemoryError
>
> This will allow you to do some post-mortem analysis and see exactly why
> the OOME occurred.
>
>
> Justin
>
> On Wed, Jan 1, 2025 at 12:17 PM Vilius Šumskas <vi...@rivile.lt>
> wrote:
>
> > Hi,
> >
> > we had an incident where our applications sent too much traffic to
> > Artemis broker and the broker got Java Heap Out Of Memory errors. I’m
> > trying to understand why backup broker never became primary after this
> happened.
> > We run Artemis static cluster with two nodes (primary and backup)
> > under Shared Storage. Configuration is pretty straightforward:
> > <ha-policy>
> > <shared-store>
> > <primary>
> > <failover-on-shutdown>true</failover-on-shutdown>
> > </primary>
> > </shared-store>
> > </ha-policy>
> >
> > <ha-policy>
> > <shared-store>
> > <backup>
> > <failover-on-shutdown>true</failover-on-shutdown>
> > </backup>
> > </shared-store>
> > </ha-policy>
> >
> > Backup always becomes primary if we reboot primary during maintenance.
> > We also tested our HA configuration with various other tests, like
> > disabling network connections, killing storage mount point, etc, so
> > I’m positive configuration should be correct.
> >
> > Primary logs during that time:
> > https://p.defau.lt/?cNVdPPEMN2qM8XbLZomKdQ
> > Backup logs during that time:
> > https://p.defau.lt/?nJvfQKdc4rrUGC9lL7_JCA
> >
> > OOM have happened at ~2:51 and the primary was in this state until I
> > have restarted it ~13:25.
> >
> > Any pointers are much appreciated!
> >
> > --
> > Best Regards,
> >
> > Vilius Šumskas
> > Rivile
> > IT manager
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> For additional commands, e-mail: users-help@activemq.apache.org
> For further information, visit: https://activemq.apache.org/contact
>
>
RE: backup not activated after OOM on primary
Posted by Vilius Šumskas <vi...@rivile.lt>.
Are there any caveats adding +ExitOnOutOfMemoryError? Just wondering why it's not in the default JAVA_ARGS in "artemis.profile".
--
Vilius
-----Original Message-----
From: Justin Bertram <jb...@apache.org>
Sent: Wednesday, January 1, 2025 11:01 PM
To: users@activemq.apache.org
Subject: Re: backup not activated after OOM on primary
I think what you're seeing is expected. An OOME usually isn't enough to trigger the broker to fail completely and trigger a failover. As you can see, the broker continued to run after the OOME which means it was still holding the lock on the shared journal (preventing the backup from activating). If you want to ensure the broker fails over in this situation you should pass this to the JVM:
-XX:+ExitOnOutOfMemoryError
This will ensure the JVM stops when an OOME occurs which will then allow the backup to activate.
It might also be worth passing this as well:
-XX:+HeapDumpOnOutOfMemoryError
This will allow you to do some post-mortem analysis and see exactly why the OOME occurred.
Justin
On Wed, Jan 1, 2025 at 12:17 PM Vilius Šumskas <vi...@rivile.lt>
wrote:
> Hi,
>
> we had an incident where our applications sent too much traffic to
> Artemis broker and the broker got Java Heap Out Of Memory errors. I’m
> trying to understand why backup broker never became primary after this happened.
> We run Artemis static cluster with two nodes (primary and backup)
> under Shared Storage. Configuration is pretty straightforward:
> <ha-policy>
> <shared-store>
> <primary>
> <failover-on-shutdown>true</failover-on-shutdown>
> </primary>
> </shared-store>
> </ha-policy>
>
> <ha-policy>
> <shared-store>
> <backup>
> <failover-on-shutdown>true</failover-on-shutdown>
> </backup>
> </shared-store>
> </ha-policy>
>
> Backup always becomes primary if we reboot primary during maintenance.
> We also tested our HA configuration with various other tests, like
> disabling network connections, killing storage mount point, etc, so
> I’m positive configuration should be correct.
>
> Primary logs during that time:
> https://p.defau.lt/?cNVdPPEMN2qM8XbLZomKdQ
> Backup logs during that time:
> https://p.defau.lt/?nJvfQKdc4rrUGC9lL7_JCA
>
> OOM have happened at ~2:51 and the primary was in this state until I
> have restarted it ~13:25.
>
> Any pointers are much appreciated!
>
> --
> Best Regards,
>
> Vilius Šumskas
> Rivile
> IT manager
>
>
Re: backup not activated after OOM on primary
Posted by Justin Bertram <jb...@apache.org>.
I think what you're seeing is expected. An OOME usually isn't enough to
trigger the broker to fail completely and trigger a failover. As you can
see, the broker continued to run after the OOME which means it was still
holding the lock on the shared journal (preventing the backup from
activating). If you want to ensure the broker fails over in this situation
you should pass this to the JVM:
-XX:+ExitOnOutOfMemoryError
This will ensure the JVM stops when an OOME occurs which will then allow
the backup to activate.
It might also be worth passing this as well:
-XX:+HeapDumpOnOutOfMemoryError
This will allow you to do some post-mortem analysis and see exactly why the
OOME occurred.
Justin
On Wed, Jan 1, 2025 at 12:17 PM Vilius Šumskas <vi...@rivile.lt>
wrote:
> Hi,
>
> we had an incident where our applications sent too much traffic to Artemis
> broker and the broker got Java Heap Out Of Memory errors. I’m trying to
> understand why backup broker never became primary after this happened.
> We run Artemis static cluster with two nodes (primary and backup) under
> Shared Storage. Configuration is pretty straightforward:
> <ha-policy>
> <shared-store>
> <primary>
> <failover-on-shutdown>true</failover-on-shutdown>
> </primary>
> </shared-store>
> </ha-policy>
>
> <ha-policy>
> <shared-store>
> <backup>
> <failover-on-shutdown>true</failover-on-shutdown>
> </backup>
> </shared-store>
> </ha-policy>
>
> Backup always becomes primary if we reboot primary during maintenance. We
> also tested our HA configuration with various other tests, like disabling
> network connections, killing storage mount point, etc, so I’m positive
> configuration should be correct.
>
> Primary logs during that time: https://p.defau.lt/?cNVdPPEMN2qM8XbLZomKdQ
> Backup logs during that time: https://p.defau.lt/?nJvfQKdc4rrUGC9lL7_JCA
>
> OOM have happened at ~2:51 and the primary was in this state until I have
> restarted it ~13:25.
>
> Any pointers are much appreciated!
>
> --
> Best Regards,
>
> Vilius Šumskas
> Rivile
> IT manager
>
>