You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by "Gunawan, Rahman (GSFC-SEWP)[Halvik Corp]" <ra...@nasa.gov.INVALID> on 2025/10/15 18:05:48 UTC
Primary server is partially down and never recover.
When primary server network card is disabled and isolate from the network, artemis was partially shut down and never recovered when network card enabled again. Is there a configuration to bring back the server when it's connected back to network?
The message in the log was:
2025-10-15 12:22:22,264 INFO [org.apache.activemq.artemis.core.server] AMQ221029: Stopped bridge $.artemis.internal.sf.ha-cluster.e3f804cf-a908-11f0-bd59-00505698dec6
2025-10-15 12:22:22,264 INFO [org.apache.activemq.artemis.core.server] AMQ221029: Stopped bridge $.artemis.internal.sf.ha-cluster.6b5d640d-a9c1-11f0-ba37-00505698573f
2025-10-15 12:22:32,412 WARN [org.apache.activemq.artemis.core.server] AMQ222002: Timed out waiting for pool to terminate org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@6dfcbe93[Shutting<mailto:org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@6dfcbe93[Shutting> down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 4393]. Interrupting all its threads!
2025-10-15 12:22:32,412 INFO [org.apache.activemq.artemis] AMQ241005: Stopping embedded web server
java.lang.InterruptedException
at java.base/java.lang.Object.wait(Native Method)
at java.base/java.lang.Thread.join(Thread.java:1313)
at java.base/java.lang.Thread.join(Thread.java:1381)
at org.apache.activemq.artemis.core.server.impl.SharedNothingPrimaryActivation$ReplicationFailureListener.lambda$handleClose$1(SharedNothingPrimaryActivation.java:272)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:120)
2025-10-15 12:22:32,427 INFO [io.hawt.HawtioContextListener] Destroying Hawtio services
2025-10-15 12:22:32,431 INFO [org.apache.activemq.artemis] AMQ241006: Stopped embedded web server
2025-10-15 12:22:32,431 INFO [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.42.0 [7ffb0f2e-a9c0-11f0-8e3d-00505698dec6] stopped, uptime 15 minutes
[cid:image001.png@01DC3DDC.D6A836C0]
Rahman Gunawan
SITSS Software Developer
NASA SEWP / SITSS
HALVIK Corporation
7601 Ora Glen Dr, Greenbelt, MD 20770
C: 240.444.6003
O: 301.614.5257
rahman.gunawan@nasa.gov<ma...@nasa.gov>
RE: Primary server is partially down and never recover.
Posted by Vilius Šumskas <vi...@rivile.lt.INVALID>.
Hi,
I don‘t believe there is an easy way for Artemis (or any network software for that matter) to somehow stop listening on the TCP network interface and then start listening, when network interface is available again. How you even know if that’s the same interface?
The proper way would be to monitor if Artemis service is alive and, if it is dead (be it network issue or anything else, like OOM for example), inform the required people. Of course, you can try to automate automatic restart with let’s say systemd unit file, but how do you know why the Artemis service was stopped? Maybe it’s storage issue and restarting it would produce more harm.
Hope that helps.
--
Vilius
From: Gunawan, Rahman (GSFC-SEWP)[Halvik Corp] <ra...@nasa.gov.INVALID>
Sent: Wednesday, October 15, 2025 9:06 PM
To: 'users@activemq.apache.org' <us...@activemq.apache.org>
Subject: Primary server is partially down and never recover.
When primary server network card is disabled and isolate from the network, artemis was partially shut down and never recovered when network card enabled again. Is there a configuration to bring back the server when it’s connected back to network?
The message in the log was:
2025-10-15 12:22:22,264 INFO [org.apache.activemq.artemis.core.server] AMQ221029: Stopped bridge $.artemis.internal.sf.ha-cluster.e3f804cf-a908-11f0-bd59-00505698dec6
2025-10-15 12:22:22,264 INFO [org.apache.activemq.artemis.core.server] AMQ221029: Stopped bridge $.artemis.internal.sf.ha-cluster.6b5d640d-a9c1-11f0-ba37-00505698573f
2025-10-15 12:22:32,412 WARN [org.apache.activemq.artemis.core.server] AMQ222002: Timed out waiting for pool to terminate org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@6dfcbe93[Shutting<mailto:org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@6dfcbe93[Shutting> down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 4393]. Interrupting all its threads!
2025-10-15 12:22:32,412 INFO [org.apache.activemq.artemis] AMQ241005: Stopping embedded web server
java.lang.InterruptedException
at java.base/java.lang.Object.wait(Native Method)
at java.base/java.lang.Thread.join(Thread.java:1313)
at java.base/java.lang.Thread.join(Thread.java:1381)
at org.apache.activemq.artemis.core.server.impl.SharedNothingPrimaryActivation$ReplicationFailureListener.lambda$handleClose$1(SharedNothingPrimaryActivation.java:272)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:120)
2025-10-15 12:22:32,427 INFO [io.hawt.HawtioContextListener] Destroying Hawtio services
2025-10-15 12:22:32,431 INFO [org.apache.activemq.artemis] AMQ241006: Stopped embedded web server
2025-10-15 12:22:32,431 INFO [org.apache.activemq.artemis.core.server] AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.42.0 [7ffb0f2e-a9c0-11f0-8e3d-00505698dec6] stopped, uptime 15 minutes
[cid:image001.png@01DC3DDC.D6A836C0]
Rahman Gunawan
SITSS Software Developer
NASA SEWP / SITSS
HALVIK Corporation
7601 Ora Glen Dr, Greenbelt, MD 20770
C: 240.444.6003
O: 301.614.5257
rahman.gunawan@nasa.gov<ma...@nasa.gov>
Re: [EXTERNAL] [BULK] Re: Primary server is partially down and never recover.
Posted by Justin Bertram <jb...@apache.org>.
> Yes, it's not created dump file.
I wouldn't expect the broker to create a thread dump on its own in this
situation. It's customary for an administrator to trigger the JVM to create
a thread dump (e.g. using the jstack command).
> It's still running but in zombie state.
This is why I would like to see a thread dump. It will show exactly what is
still running and preventing the JVM from terminating completely.
> Can't do anything, web console doesn't load.
You should still be able to generate a thread dump despite the web console
not working. Do you think you could provide a thread dump from the zombie
broker?
> Every time the network card disabled for over 5 minutes, this problem
occurred.
What platform is the broker running on? How exactly are you disabling the
network card?
Justin
On Mon, Oct 27, 2025 at 2:00 PM Gunawan, Rahman (GSFC-SEWP)[Halvik Corp]
<ra...@nasa.gov.invalid> wrote:
> Answers below each question.
>
> -----Original Message-----
> From: Justin Bertram <jb...@apache.org>
> Sent: Wednesday, October 22, 2025 3:33 PM
> To: users@activemq.apache.org
> Subject: Re: [EXTERNAL] [BULK] Re: Primary server is partially down and
> never recover.
>
> CAUTION: This email originated from outside of NASA. Please take care
> when clicking links or opening attachments. Use the "Report Message"
> button to report suspicious messages to the NASA SOC.
>
>
>
>
> > ...the artemis UI indicated that it's not fully running...
>
> What exactly does this mean? Are you talking about the web console that
> ships with Artemis? What specifically indicated that the broker was not
> fully running? Did you happen to collect any thread dumps from the broker
> when it was in this state?
> RG: Yes, it's not created dump file. It's still running but in zombie
> state. Can't do anything, web console doesn't load.
>
> The log you pasted indicates that something failed during shutdown so
> maybe that's why the broker is in a kind of zombie state. Can you reproduce
> this?
> If you can provide a way to reproduce this I can investigate further.
> RG: Every time the network card disabled for over 5 minutes, this problem
> occurred.
>
>
> Justin
>
> On Wed, Oct 22, 2025 at 7:55 AM Gunawan, Rahman (GSFC-SEWP)[Halvik Corp] <
> rahman.gunawan@nasa.gov.invalid> wrote:
>
> > Unix command ps -ef|grep artemis indicated that the process is running
> > but the artemis UI indicated that it's not fully running and the
> > backup server is running although the allow-failback = true.
> >
> > Rahman Gunawan
> >
> > -----Original Message-----
> > From: Justin Bertram <jb...@apache.org>
> > Sent: Tuesday, October 21, 2025 5:17 PM
> > To: users@activemq.apache.org
> > Subject: [EXTERNAL] [BULK] Re: Primary server is partially down and
> > never recover.
> >
> > CAUTION: This email originated from outside of NASA. Please take care
> > when clicking links or opening attachments. Use the "Report Message"
> > button to report suspicious messages to the NASA SOC.
> >
> >
> >
> >
> > > When primary server network card is disabled and isolate from the
> > network, artemis was partially shut down...
> >
> > Can you elaborate on what you mean by "partially shut down"? Is the
> > log message you shared from this partial shutdown? If so, in what
> > sense was it partial?
> >
> >
> > Justin
> >
> > On Wed, Oct 15, 2025 at 1:07 PM Gunawan, Rahman (GSFC-SEWP)[Halvik
> > Corp] < rahman.gunawan@nasa.gov.invalid> wrote:
> >
> > > When primary server network card is disabled and isolate from the
> > > network, artemis was partially shut down and never recovered when
> > > network card enabled again. Is there a configuration to bring back
> > > the server when it’s connected back to network?
> > >
> > > The message in the log was:
> > >
> > > 2025-10-15 12:22:22,264 INFO
> > > [org.apache.activemq.artemis.core.server]
> > > AMQ221029: Stopped bridge
> > > $.artemis.internal.sf.ha-cluster.e3f804cf-a908-11f0-bd59-00505698dec
> > > 6
> > >
> > > 2025-10-15 12:22:22,264 INFO
> > > [org.apache.activemq.artemis.core.server]
> > > AMQ221029: Stopped bridge
> > > $.artemis.internal.sf.ha-cluster.6b5d640d-a9c1-11f0-ba37-00505698573
> > > f
> > >
> > > 2025-10-15 12:22:32,412 WARN
> > > [org.apache.activemq.artemis.core.server]
> > > AMQ222002: Timed out waiting for pool to terminate
> > > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@6dfcbe9
> > > 3[ Shutting down, pool size = 2, active threads = 2, queued tasks =
> > > 0, completed tasks = 4393]. Interrupting all its threads!
> > >
> > > 2025-10-15 12:22:32,412 INFO [org.apache.activemq.artemis] AMQ241005:
> > > Stopping embedded web server
> > >
> > > java.lang.InterruptedException
> > >
> > > at java.base/java.lang.Object.wait(Native Method)
> > >
> > > at java.base/java.lang.Thread.join(Thread.java:1313)
> > >
> > > at java.base/java.lang.Thread.join(Thread.java:1381)
> > >
> > > at
> > > org.apache.activemq.artemis.core.server.impl.SharedNothingPrimaryAct
> > > iv
> > > ation$ReplicationFailureListener.lambda$handleClose$1(SharedNothingP
> > > ri
> > > maryActivation.java:272)
> > >
> > > at
> > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPo
> > > ol
> > > Executor.java:1136)
> > >
> > > at
> > > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadP
> > > oo
> > > lExecutor.java:635)
> > >
> > > at
> > > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(Active
> > > MQ
> > > ThreadFactory.java:120)
> > >
> > > 2025-10-15 12:22:32,427 INFO [io.hawt.HawtioContextListener]
> > > Destroying Hawtio services
> > >
> > > 2025-10-15 12:22:32,431 INFO [org.apache.activemq.artemis] AMQ241006:
> > > Stopped embedded web server
> > >
> > > 2025-10-15 12:22:32,431 INFO
> > > [org.apache.activemq.artemis.core.server]
> > > AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.42.0
> > > [7ffb0f2e-a9c0-11f0-8e3d-00505698dec6] stopped, uptime 15 minutes
> > >
> > >
> > >
> > > *Rahman Gunawan*
> > >
> > > *SITSS Software Developer*
> > >
> > > NASA SEWP / SITSS
> > >
> > > HALV*I*K Corporation
> > >
> > > 7601 Ora Glen Dr, Greenbelt, MD 20770
> > >
> > > C: 240.444.6003
> > > O: 301.614.5257
> > >
> > > rahman.gunawan@nasa.gov
> > >
> > >
> > >
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> > For additional commands, e-mail: users-help@activemq.apache.org For
> > further information, visit:
> > https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Facti
> > vemq.apache.org%2Fcontact&data=05%7C02%7Crahman.gunawan%40nasa.gov%7C5
> > 7ee7f902eee42fbd57a08de11a1f12c%7C7005d45845be48ae8140d43da96dd17b%7C0
> > %7C0%7C638967584391846913%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRy
> > dWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%
> > 3D%7C0%7C%7C%7C&sdata=sPvHmQybzWkxwLjOlShGihzp%2F90H1tDCMg1lMFlyWms%3D
> > &reserved=0
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> For additional commands, e-mail: users-help@activemq.apache.org
> For further information, visit: https://activemq.apache.org/contact
>
>
RE: [EXTERNAL] [BULK] Re: Primary server is partially down and never recover.
Posted by "Gunawan, Rahman (GSFC-SEWP)[Halvik Corp]" <ra...@nasa.gov.INVALID>.
Answers below each question.
-----Original Message-----
From: Justin Bertram <jb...@apache.org>
Sent: Wednesday, October 22, 2025 3:33 PM
To: users@activemq.apache.org
Subject: Re: [EXTERNAL] [BULK] Re: Primary server is partially down and never recover.
CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC.
> ...the artemis UI indicated that it's not fully running...
What exactly does this mean? Are you talking about the web console that ships with Artemis? What specifically indicated that the broker was not fully running? Did you happen to collect any thread dumps from the broker when it was in this state?
RG: Yes, it's not created dump file. It's still running but in zombie state. Can't do anything, web console doesn't load.
The log you pasted indicates that something failed during shutdown so maybe that's why the broker is in a kind of zombie state. Can you reproduce this?
If you can provide a way to reproduce this I can investigate further.
RG: Every time the network card disabled for over 5 minutes, this problem occurred.
Justin
On Wed, Oct 22, 2025 at 7:55 AM Gunawan, Rahman (GSFC-SEWP)[Halvik Corp] <ra...@nasa.gov.invalid> wrote:
> Unix command ps -ef|grep artemis indicated that the process is running
> but the artemis UI indicated that it's not fully running and the
> backup server is running although the allow-failback = true.
>
> Rahman Gunawan
>
> -----Original Message-----
> From: Justin Bertram <jb...@apache.org>
> Sent: Tuesday, October 21, 2025 5:17 PM
> To: users@activemq.apache.org
> Subject: [EXTERNAL] [BULK] Re: Primary server is partially down and
> never recover.
>
> CAUTION: This email originated from outside of NASA. Please take care
> when clicking links or opening attachments. Use the "Report Message"
> button to report suspicious messages to the NASA SOC.
>
>
>
>
> > When primary server network card is disabled and isolate from the
> network, artemis was partially shut down...
>
> Can you elaborate on what you mean by "partially shut down"? Is the
> log message you shared from this partial shutdown? If so, in what
> sense was it partial?
>
>
> Justin
>
> On Wed, Oct 15, 2025 at 1:07 PM Gunawan, Rahman (GSFC-SEWP)[Halvik
> Corp] < rahman.gunawan@nasa.gov.invalid> wrote:
>
> > When primary server network card is disabled and isolate from the
> > network, artemis was partially shut down and never recovered when
> > network card enabled again. Is there a configuration to bring back
> > the server when it’s connected back to network?
> >
> > The message in the log was:
> >
> > 2025-10-15 12:22:22,264 INFO
> > [org.apache.activemq.artemis.core.server]
> > AMQ221029: Stopped bridge
> > $.artemis.internal.sf.ha-cluster.e3f804cf-a908-11f0-bd59-00505698dec
> > 6
> >
> > 2025-10-15 12:22:22,264 INFO
> > [org.apache.activemq.artemis.core.server]
> > AMQ221029: Stopped bridge
> > $.artemis.internal.sf.ha-cluster.6b5d640d-a9c1-11f0-ba37-00505698573
> > f
> >
> > 2025-10-15 12:22:32,412 WARN
> > [org.apache.activemq.artemis.core.server]
> > AMQ222002: Timed out waiting for pool to terminate
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@6dfcbe9
> > 3[ Shutting down, pool size = 2, active threads = 2, queued tasks =
> > 0, completed tasks = 4393]. Interrupting all its threads!
> >
> > 2025-10-15 12:22:32,412 INFO [org.apache.activemq.artemis] AMQ241005:
> > Stopping embedded web server
> >
> > java.lang.InterruptedException
> >
> > at java.base/java.lang.Object.wait(Native Method)
> >
> > at java.base/java.lang.Thread.join(Thread.java:1313)
> >
> > at java.base/java.lang.Thread.join(Thread.java:1381)
> >
> > at
> > org.apache.activemq.artemis.core.server.impl.SharedNothingPrimaryAct
> > iv
> > ation$ReplicationFailureListener.lambda$handleClose$1(SharedNothingP
> > ri
> > maryActivation.java:272)
> >
> > at
> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPo
> > ol
> > Executor.java:1136)
> >
> > at
> > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadP
> > oo
> > lExecutor.java:635)
> >
> > at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(Active
> > MQ
> > ThreadFactory.java:120)
> >
> > 2025-10-15 12:22:32,427 INFO [io.hawt.HawtioContextListener]
> > Destroying Hawtio services
> >
> > 2025-10-15 12:22:32,431 INFO [org.apache.activemq.artemis] AMQ241006:
> > Stopped embedded web server
> >
> > 2025-10-15 12:22:32,431 INFO
> > [org.apache.activemq.artemis.core.server]
> > AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.42.0
> > [7ffb0f2e-a9c0-11f0-8e3d-00505698dec6] stopped, uptime 15 minutes
> >
> >
> >
> > *Rahman Gunawan*
> >
> > *SITSS Software Developer*
> >
> > NASA SEWP / SITSS
> >
> > HALV*I*K Corporation
> >
> > 7601 Ora Glen Dr, Greenbelt, MD 20770
> >
> > C: 240.444.6003
> > O: 301.614.5257
> >
> > rahman.gunawan@nasa.gov
> >
> >
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> For additional commands, e-mail: users-help@activemq.apache.org For
> further information, visit:
> https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Facti
> vemq.apache.org%2Fcontact&data=05%7C02%7Crahman.gunawan%40nasa.gov%7C5
> 7ee7f902eee42fbd57a08de11a1f12c%7C7005d45845be48ae8140d43da96dd17b%7C0
> %7C0%7C638967584391846913%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRy
> dWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%
> 3D%7C0%7C%7C%7C&sdata=sPvHmQybzWkxwLjOlShGihzp%2F90H1tDCMg1lMFlyWms%3D
> &reserved=0
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
For additional commands, e-mail: users-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
Re: [EXTERNAL] [BULK] Re: Primary server is partially down and never recover.
Posted by Justin Bertram <jb...@apache.org>.
> ...the artemis UI indicated that it's not fully running...
What exactly does this mean? Are you talking about the web console that
ships with Artemis? What specifically indicated that the broker was not
fully running? Did you happen to collect any thread dumps from the broker
when it was in this state?
The log you pasted indicates that something failed during shutdown so maybe
that's why the broker is in a kind of zombie state. Can you reproduce this?
If you can provide a way to reproduce this I can investigate further.
Justin
On Wed, Oct 22, 2025 at 7:55 AM Gunawan, Rahman (GSFC-SEWP)[Halvik Corp]
<ra...@nasa.gov.invalid> wrote:
> Unix command ps -ef|grep artemis indicated that the process is running but
> the artemis UI indicated that it's not fully running and the backup server
> is running although the allow-failback = true.
>
> Rahman Gunawan
>
> -----Original Message-----
> From: Justin Bertram <jb...@apache.org>
> Sent: Tuesday, October 21, 2025 5:17 PM
> To: users@activemq.apache.org
> Subject: [EXTERNAL] [BULK] Re: Primary server is partially down and never
> recover.
>
> CAUTION: This email originated from outside of NASA. Please take care
> when clicking links or opening attachments. Use the "Report Message"
> button to report suspicious messages to the NASA SOC.
>
>
>
>
> > When primary server network card is disabled and isolate from the
> network, artemis was partially shut down...
>
> Can you elaborate on what you mean by "partially shut down"? Is the log
> message you shared from this partial shutdown? If so, in what sense was it
> partial?
>
>
> Justin
>
> On Wed, Oct 15, 2025 at 1:07 PM Gunawan, Rahman (GSFC-SEWP)[Halvik Corp] <
> rahman.gunawan@nasa.gov.invalid> wrote:
>
> > When primary server network card is disabled and isolate from the
> > network, artemis was partially shut down and never recovered when
> > network card enabled again. Is there a configuration to bring back
> > the server when it’s connected back to network?
> >
> > The message in the log was:
> >
> > 2025-10-15 12:22:22,264 INFO
> > [org.apache.activemq.artemis.core.server]
> > AMQ221029: Stopped bridge
> > $.artemis.internal.sf.ha-cluster.e3f804cf-a908-11f0-bd59-00505698dec6
> >
> > 2025-10-15 12:22:22,264 INFO
> > [org.apache.activemq.artemis.core.server]
> > AMQ221029: Stopped bridge
> > $.artemis.internal.sf.ha-cluster.6b5d640d-a9c1-11f0-ba37-00505698573f
> >
> > 2025-10-15 12:22:32,412 WARN
> > [org.apache.activemq.artemis.core.server]
> > AMQ222002: Timed out waiting for pool to terminate
> > org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@6dfcbe93[
> > Shutting down, pool size = 2, active threads = 2, queued tasks = 0,
> > completed tasks = 4393]. Interrupting all its threads!
> >
> > 2025-10-15 12:22:32,412 INFO [org.apache.activemq.artemis] AMQ241005:
> > Stopping embedded web server
> >
> > java.lang.InterruptedException
> >
> > at java.base/java.lang.Object.wait(Native Method)
> >
> > at java.base/java.lang.Thread.join(Thread.java:1313)
> >
> > at java.base/java.lang.Thread.join(Thread.java:1381)
> >
> > at
> > org.apache.activemq.artemis.core.server.impl.SharedNothingPrimaryActiv
> > ation$ReplicationFailureListener.lambda$handleClose$1(SharedNothingPri
> > maryActivation.java:272)
> >
> > at
> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> > Executor.java:1136)
> >
> > at
> > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> > lExecutor.java:635)
> >
> > at
> > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQ
> > ThreadFactory.java:120)
> >
> > 2025-10-15 12:22:32,427 INFO [io.hawt.HawtioContextListener]
> > Destroying Hawtio services
> >
> > 2025-10-15 12:22:32,431 INFO [org.apache.activemq.artemis] AMQ241006:
> > Stopped embedded web server
> >
> > 2025-10-15 12:22:32,431 INFO
> > [org.apache.activemq.artemis.core.server]
> > AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.42.0
> > [7ffb0f2e-a9c0-11f0-8e3d-00505698dec6] stopped, uptime 15 minutes
> >
> >
> >
> > *Rahman Gunawan*
> >
> > *SITSS Software Developer*
> >
> > NASA SEWP / SITSS
> >
> > HALV*I*K Corporation
> >
> > 7601 Ora Glen Dr, Greenbelt, MD 20770
> >
> > C: 240.444.6003
> > O: 301.614.5257
> >
> > rahman.gunawan@nasa.gov
> >
> >
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@activemq.apache.org
> For additional commands, e-mail: users-help@activemq.apache.org
> For further information, visit: https://activemq.apache.org/contact
>
>
RE: [EXTERNAL] [BULK] Re: Primary server is partially down and never recover.
Posted by "Gunawan, Rahman (GSFC-SEWP)[Halvik Corp]" <ra...@nasa.gov.INVALID>.
Unix command ps -ef|grep artemis indicated that the process is running but the artemis UI indicated that it's not fully running and the backup server is running although the allow-failback = true.
Rahman Gunawan
-----Original Message-----
From: Justin Bertram <jb...@apache.org>
Sent: Tuesday, October 21, 2025 5:17 PM
To: users@activemq.apache.org
Subject: [EXTERNAL] [BULK] Re: Primary server is partially down and never recover.
CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC.
> When primary server network card is disabled and isolate from the
network, artemis was partially shut down...
Can you elaborate on what you mean by "partially shut down"? Is the log message you shared from this partial shutdown? If so, in what sense was it partial?
Justin
On Wed, Oct 15, 2025 at 1:07 PM Gunawan, Rahman (GSFC-SEWP)[Halvik Corp] <ra...@nasa.gov.invalid> wrote:
> When primary server network card is disabled and isolate from the
> network, artemis was partially shut down and never recovered when
> network card enabled again. Is there a configuration to bring back
> the server when it’s connected back to network?
>
> The message in the log was:
>
> 2025-10-15 12:22:22,264 INFO
> [org.apache.activemq.artemis.core.server]
> AMQ221029: Stopped bridge
> $.artemis.internal.sf.ha-cluster.e3f804cf-a908-11f0-bd59-00505698dec6
>
> 2025-10-15 12:22:22,264 INFO
> [org.apache.activemq.artemis.core.server]
> AMQ221029: Stopped bridge
> $.artemis.internal.sf.ha-cluster.6b5d640d-a9c1-11f0-ba37-00505698573f
>
> 2025-10-15 12:22:32,412 WARN
> [org.apache.activemq.artemis.core.server]
> AMQ222002: Timed out waiting for pool to terminate
> org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@6dfcbe93[
> Shutting down, pool size = 2, active threads = 2, queued tasks = 0,
> completed tasks = 4393]. Interrupting all its threads!
>
> 2025-10-15 12:22:32,412 INFO [org.apache.activemq.artemis] AMQ241005:
> Stopping embedded web server
>
> java.lang.InterruptedException
>
> at java.base/java.lang.Object.wait(Native Method)
>
> at java.base/java.lang.Thread.join(Thread.java:1313)
>
> at java.base/java.lang.Thread.join(Thread.java:1381)
>
> at
> org.apache.activemq.artemis.core.server.impl.SharedNothingPrimaryActiv
> ation$ReplicationFailureListener.lambda$handleClose$1(SharedNothingPri
> maryActivation.java:272)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1136)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:635)
>
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQ
> ThreadFactory.java:120)
>
> 2025-10-15 12:22:32,427 INFO [io.hawt.HawtioContextListener]
> Destroying Hawtio services
>
> 2025-10-15 12:22:32,431 INFO [org.apache.activemq.artemis] AMQ241006:
> Stopped embedded web server
>
> 2025-10-15 12:22:32,431 INFO
> [org.apache.activemq.artemis.core.server]
> AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.42.0
> [7ffb0f2e-a9c0-11f0-8e3d-00505698dec6] stopped, uptime 15 minutes
>
>
>
> *Rahman Gunawan*
>
> *SITSS Software Developer*
>
> NASA SEWP / SITSS
>
> HALV*I*K Corporation
>
> 7601 Ora Glen Dr, Greenbelt, MD 20770
>
> C: 240.444.6003
> O: 301.614.5257
>
> rahman.gunawan@nasa.gov
>
>
>
>
>
Re: Primary server is partially down and never recover.
Posted by Justin Bertram <jb...@apache.org>.
> When primary server network card is disabled and isolate from the
network, artemis was partially shut down...
Can you elaborate on what you mean by "partially shut down"? Is the log
message you shared from this partial shutdown? If so, in what sense was it
partial?
Justin
On Wed, Oct 15, 2025 at 1:07 PM Gunawan, Rahman (GSFC-SEWP)[Halvik Corp]
<ra...@nasa.gov.invalid> wrote:
> When primary server network card is disabled and isolate from the network,
> artemis was partially shut down and never recovered when network card
> enabled again. Is there a configuration to bring back the server when it’s
> connected back to network?
>
> The message in the log was:
>
> 2025-10-15 12:22:22,264 INFO [org.apache.activemq.artemis.core.server]
> AMQ221029: Stopped bridge
> $.artemis.internal.sf.ha-cluster.e3f804cf-a908-11f0-bd59-00505698dec6
>
> 2025-10-15 12:22:22,264 INFO [org.apache.activemq.artemis.core.server]
> AMQ221029: Stopped bridge
> $.artemis.internal.sf.ha-cluster.6b5d640d-a9c1-11f0-ba37-00505698573f
>
> 2025-10-15 12:22:32,412 WARN [org.apache.activemq.artemis.core.server]
> AMQ222002: Timed out waiting for pool to terminate
> org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@6dfcbe93[Shutting
> down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks
> = 4393]. Interrupting all its threads!
>
> 2025-10-15 12:22:32,412 INFO [org.apache.activemq.artemis] AMQ241005:
> Stopping embedded web server
>
> java.lang.InterruptedException
>
> at java.base/java.lang.Object.wait(Native Method)
>
> at java.base/java.lang.Thread.join(Thread.java:1313)
>
> at java.base/java.lang.Thread.join(Thread.java:1381)
>
> at
> org.apache.activemq.artemis.core.server.impl.SharedNothingPrimaryActivation$ReplicationFailureListener.lambda$handleClose$1(SharedNothingPrimaryActivation.java:272)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:120)
>
> 2025-10-15 12:22:32,427 INFO [io.hawt.HawtioContextListener] Destroying
> Hawtio services
>
> 2025-10-15 12:22:32,431 INFO [org.apache.activemq.artemis] AMQ241006:
> Stopped embedded web server
>
> 2025-10-15 12:22:32,431 INFO [org.apache.activemq.artemis.core.server]
> AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.42.0
> [7ffb0f2e-a9c0-11f0-8e3d-00505698dec6] stopped, uptime 15 minutes
>
>
>
> *Rahman Gunawan*
>
> *SITSS Software Developer*
>
> NASA SEWP / SITSS
>
> HALV*I*K Corporation
>
> 7601 Ora Glen Dr, Greenbelt, MD 20770
>
> C: 240.444.6003
> O: 301.614.5257
>
> rahman.gunawan@nasa.gov
>
>
>
>
>