You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Michael Brennock <mb...@gmail.com> on 2022/10/05 17:00:19 UTC
Re: Newbie Looking for Help with a Crashing Artemis Server
Thanks for the tip! I confess it took me much longer than I hoped to get a
word with our sysadmim because they were crazy busy. they did give me some
promising information though!
The sysadmim affirmed that any process that runs in a terminal, like
Artemis for example, will be shut down when the terminal is session is shut
down. I ran a test last night after following their instructions: run
Artemis like this to get it to run in the background:
./Artemis run $
Admittedly, 16 hours is twice as many hours as 8, but it still just shut
down with the same lack of apparently useful information from artemis.log,
from audit.log, or from the application that is using Artemis as a message
broker.
I have two questions:
1. Is there any way to make sure a Java process runs indefinitely on a unix
(AIX) environment that I should know about?
2. Is it possible to automatically restart Artemis from within a shell
script, or some such? I'd consider embedding Artemis into the application
itself to control Artemis, but my marching orders are to find a way to get
the application to interface with a standalone message queue if possible.
I appreciate your help so far everyone!
Michael
On Tue, Sep 20, 2022, 10:51 AM Justin Bertram <jb...@apache.org> wrote:
> Based on the logging it looks to me like the broker was simply stopped
> administratively. I definitely think the first step is checking with the
> sysadmins.
>
>
> Justin
>
> On Tue, Sep 20, 2022 at 12:23 PM Michael Brennock <mb...@gmail.com>
> wrote:
>
> > The artemis.log files show the server going offline at random times
> between
> > 12:00am and 2:00am on different nights, after being online somewhere
> > between 8 and 13 hours.
> >
> > Here's part of the stack trace:
> >
> > 2022-09-20 01:07:09,359 INFO [io.hawt.web.auth.AuthenticationFilter]
> > Destroying hawtio authentication filter
> > 2022-09-20 01:07:09,369 INFO [io.hawt.HawtioContextListener] Destroying
> > hawtio services
> > 2022-09-20 01:07:09,472 INFO
> > [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed
> > artemis-plugin plugin
> > 2022-09-20 01:07:09,524 INFO
> > [org.apache.activemq.hawtio.branding.PluginCont
> > 2022-09-20 01:07:09,591 INFO [org.apache.activemq.artemis.core.server]
> > AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.19.1
> > [6bb50282-27d7-11ed-af01-6ab9759765a1] stopped, uptime 13 hours 58
> minutes
> >
> > For more background, I have a java application that is successfully
> > creating and using queues. As far as I can tell, the application isn't
> > posting any unusual messages to the queues. The application isn't posting
> > any messages within hours of the artemis server going offline.
> >
> > (As I write this, I realize I should contact the sysadmins to see if they
> > have any maintenance programs running at that hour)
> >
> > In the meantime, do you guys have any thoughts on where I should look? I
> > should also clarify that this application previously used embedded
> > activemq, but we are evaluating artemis as a replacement MQ. Someone
> > realized they could swap the activemq broker for artemis, and I'm
> following
> > their work, I wonder if they or I overlooked something when we took this
> > shortcut?
> >
> > Thanks for your help,
> >
> > Michael
> >
> >
> > On Mon, Sep 19, 2022 at 11:18 AM Justin Bertram <jb...@apache.org>
> > wrote:
> >
> > > Ekta, this sounds like an issue worth discussing. However, it doesn't
> > seem
> > > directly related to Michael's question. I recommend you start your own
> > > thread about your issue rather than using this thread. Thanks!
> > >
> > >
> > > Justin
> > >
> > > On Mon, Sep 19, 2022 at 12:53 PM Ekta Awasthi
> > > <Ek...@officedepot.com.invalid> wrote:
> > >
> > > > Hello,
> > > >
> > > > My apache artemis server running on 2.22.0 is also crashing randomly,
> > > upon
> > > > checking it turns out that the OS kernel is killing it with below
> > error.
> > > > To my understanding the process with highest consuming memory should
> be
> > > > getting killed but this is NOT the case when I am checking metrics
> > from a
> > > > metric tool. The memory and all are well under where it should be,
> but
> > I
> > > am
> > > > following up with my OS team to see if they can point me in the right
> > > > direction or something is off with the version itself...?
> > > >
> > > > Out of memory: Kill process 21457 (java) score 852 or sacrifice child
> > > >
> > > > Can you also try to run below command and see if you are facing the
> > same.
> > > > Thanks
> > > >
> > > > dmesg | grep "your_process_ID"
> > > >
> > > > Thanks
> > > > Ekta
> > > >
> > > > ________________________________
> > > > From: Michael Brennock <mb...@gmail.com>
> > > > Sent: Monday, September 19, 2022 1:38 PM
> > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > Subject: Newbie Looking for Help with a Crashing Artemis Server
> > > >
> > > > [CAUTION: EXTERNAL SENDER]
> > > >
> > > >
> > > > Good day,
> > > >
> > > > I was hoping to ask some of the more experienced Apache Artemis why
> my
> > > > server keeps crashing after a few hours. I'm running Artemis 2.19.1
> on
> > > the
> > > > default settings on an IBM AIX box. I can provide a copy of the
> > > stacktrace,
> > > > but it doesn't have much information. I'm also trying to be careful
> not
> > > to
> > > > overshare confidential information from work.
> > > >
> > > > Thanks for your help,
> > > >
> > > > Michael Brennock
> > > >
> > > > CONFIDENTIALITY NOTICE: The information contained in this email and
> > > > attached document(s) may contain confidential information that is
> > > intended
> > > > only for the addressee(s). If you are not the intended recipient, you
> > are
> > > > hereby advised that any disclosure, copying, distribution or the
> taking
> > > of
> > > > any action in reliance upon the information is prohibited. If you
> have
> > > > received this email in error, please immediately notify the sender
> and
> > > > delete it from your system.
> > > >
> > >
> >
>
Re: Newbie Looking for Help with a Crashing Artemis Server
Posted by Michael Brennock <mb...@gmail.com>.
Ah, thanks for the clarification! I will try using artmeis-service start
like you suggested. I will also talk with the sysadmim about the second
method you suggested.
To clarify, I have an existing Java application that was starting apache
ActiveMQ as an embedded process. I was asked to replace ActiveMQ with
Artemis. Before we can perform an in-place upgrade to Artemis, the
stakeholders want us to run a performance test with a standalone Artemis
server connected to the application, to see if it might be worth the
effort. I was able to disable the code that started the embedded message
queue and redirect it to the standalone Artemis server.
On Wed, Oct 5, 2022, 10:40 AM Justin Bertram <jb...@apache.org> wrote:
> > Is there any way to make sure a Java process runs indefinitely on a unix
> (AIX) environment that I should know about?
>
> You can use the artemis-service script which will automatically start the
> broker in the background and it should run until it is shutdown via the
> same script. This script is mentioned when you first create the broker
> instance, e.g.:
>
> You can now start the broker by executing:
>
> "/path/to/broker/bin/artemis" run
>
> Or you can run the broker in the background using:
>
> "/path/to/broker/bin/artemis-service" start
>
> > Is it possible to automatically restart Artemis from within a shell
> script, or some such?
>
> Unix variants have an init system [1] of some kind (e.g. systemd) through
> which you configure such functionality. I bet your system admin could help
> you with the specifics here. Such details are really beyond the scope of
> this list. You definitely shouldn't need to write your own application and
> embed ActiveMQ Artemis into it to get this functionality.
>
>
> Justin
>
> [1] https://en.wikipedia.org/wiki/Init
>
> On Wed, Oct 5, 2022 at 12:01 PM Michael Brennock <mb...@gmail.com>
> wrote:
>
> > Thanks for the tip! I confess it took me much longer than I hoped to get
> a
> > word with our sysadmim because they were crazy busy. they did give me
> some
> > promising information though!
> >
> > The sysadmim affirmed that any process that runs in a terminal, like
> > Artemis for example, will be shut down when the terminal is session is
> shut
> > down. I ran a test last night after following their instructions: run
> > Artemis like this to get it to run in the background:
> >
> > ./Artemis run $
> >
> > Admittedly, 16 hours is twice as many hours as 8, but it still just shut
> > down with the same lack of apparently useful information from
> artemis.log,
> > from audit.log, or from the application that is using Artemis as a
> message
> > broker.
> >
> > I have two questions:
> >
> > 1. Is there any way to make sure a Java process runs indefinitely on a
> unix
> > (AIX) environment that I should know about?
> > 2. Is it possible to automatically restart Artemis from within a shell
> > script, or some such? I'd consider embedding Artemis into the application
> > itself to control Artemis, but my marching orders are to find a way to
> get
> > the application to interface with a standalone message queue if possible.
> >
> > I appreciate your help so far everyone!
> >
> > Michael
> >
> > On Tue, Sep 20, 2022, 10:51 AM Justin Bertram <jb...@apache.org>
> wrote:
> >
> > > Based on the logging it looks to me like the broker was simply stopped
> > > administratively. I definitely think the first step is checking with
> the
> > > sysadmins.
> > >
> > >
> > > Justin
> > >
> > > On Tue, Sep 20, 2022 at 12:23 PM Michael Brennock <mbrennock@gmail.com
> >
> > > wrote:
> > >
> > > > The artemis.log files show the server going offline at random times
> > > between
> > > > 12:00am and 2:00am on different nights, after being online somewhere
> > > > between 8 and 13 hours.
> > > >
> > > > Here's part of the stack trace:
> > > >
> > > > 2022-09-20 01:07:09,359 INFO [io.hawt.web.auth.AuthenticationFilter]
> > > > Destroying hawtio authentication filter
> > > > 2022-09-20 01:07:09,369 INFO [io.hawt.HawtioContextListener]
> > Destroying
> > > > hawtio services
> > > > 2022-09-20 01:07:09,472 INFO
> > > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed
> > > > artemis-plugin plugin
> > > > 2022-09-20 01:07:09,524 INFO
> > > > [org.apache.activemq.hawtio.branding.PluginCont
> > > > 2022-09-20 01:07:09,591 INFO
> [org.apache.activemq.artemis.core.server]
> > > > AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.19.1
> > > > [6bb50282-27d7-11ed-af01-6ab9759765a1] stopped, uptime 13 hours 58
> > > minutes
> > > >
> > > > For more background, I have a java application that is successfully
> > > > creating and using queues. As far as I can tell, the application
> isn't
> > > > posting any unusual messages to the queues. The application isn't
> > posting
> > > > any messages within hours of the artemis server going offline.
> > > >
> > > > (As I write this, I realize I should contact the sysadmins to see if
> > they
> > > > have any maintenance programs running at that hour)
> > > >
> > > > In the meantime, do you guys have any thoughts on where I should
> look?
> > I
> > > > should also clarify that this application previously used embedded
> > > > activemq, but we are evaluating artemis as a replacement MQ. Someone
> > > > realized they could swap the activemq broker for artemis, and I'm
> > > following
> > > > their work, I wonder if they or I overlooked something when we took
> > this
> > > > shortcut?
> > > >
> > > > Thanks for your help,
> > > >
> > > > Michael
> > > >
> > > >
> > > > On Mon, Sep 19, 2022 at 11:18 AM Justin Bertram <jbertram@apache.org
> >
> > > > wrote:
> > > >
> > > > > Ekta, this sounds like an issue worth discussing. However, it
> doesn't
> > > > seem
> > > > > directly related to Michael's question. I recommend you start your
> > own
> > > > > thread about your issue rather than using this thread. Thanks!
> > > > >
> > > > >
> > > > > Justin
> > > > >
> > > > > On Mon, Sep 19, 2022 at 12:53 PM Ekta Awasthi
> > > > > <Ek...@officedepot.com.invalid> wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > My apache artemis server running on 2.22.0 is also crashing
> > randomly,
> > > > > upon
> > > > > > checking it turns out that the OS kernel is killing it with below
> > > > error.
> > > > > > To my understanding the process with highest consuming memory
> > should
> > > be
> > > > > > getting killed but this is NOT the case when I am checking
> metrics
> > > > from a
> > > > > > metric tool. The memory and all are well under where it should
> be,
> > > but
> > > > I
> > > > > am
> > > > > > following up with my OS team to see if they can point me in the
> > right
> > > > > > direction or something is off with the version itself...?
> > > > > >
> > > > > > Out of memory: Kill process 21457 (java) score 852 or sacrifice
> > child
> > > > > >
> > > > > > Can you also try to run below command and see if you are facing
> the
> > > > same.
> > > > > > Thanks
> > > > > >
> > > > > > dmesg | grep "your_process_ID"
> > > > > >
> > > > > > Thanks
> > > > > > Ekta
> > > > > >
> > > > > > ________________________________
> > > > > > From: Michael Brennock <mb...@gmail.com>
> > > > > > Sent: Monday, September 19, 2022 1:38 PM
> > > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > > Subject: Newbie Looking for Help with a Crashing Artemis Server
> > > > > >
> > > > > > [CAUTION: EXTERNAL SENDER]
> > > > > >
> > > > > >
> > > > > > Good day,
> > > > > >
> > > > > > I was hoping to ask some of the more experienced Apache Artemis
> why
> > > my
> > > > > > server keeps crashing after a few hours. I'm running Artemis
> 2.19.1
> > > on
> > > > > the
> > > > > > default settings on an IBM AIX box. I can provide a copy of the
> > > > > stacktrace,
> > > > > > but it doesn't have much information. I'm also trying to be
> careful
> > > not
> > > > > to
> > > > > > overshare confidential information from work.
> > > > > >
> > > > > > Thanks for your help,
> > > > > >
> > > > > > Michael Brennock
> > > > > >
> > > > > > CONFIDENTIALITY NOTICE: The information contained in this email
> and
> > > > > > attached document(s) may contain confidential information that is
> > > > > intended
> > > > > > only for the addressee(s). If you are not the intended recipient,
> > you
> > > > are
> > > > > > hereby advised that any disclosure, copying, distribution or the
> > > taking
> > > > > of
> > > > > > any action in reliance upon the information is prohibited. If you
> > > have
> > > > > > received this email in error, please immediately notify the
> sender
> > > and
> > > > > > delete it from your system.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Newbie Looking for Help with a Crashing Artemis Server
Posted by Justin Bertram <jb...@apache.org>.
> Is there any way to make sure a Java process runs indefinitely on a unix
(AIX) environment that I should know about?
You can use the artemis-service script which will automatically start the
broker in the background and it should run until it is shutdown via the
same script. This script is mentioned when you first create the broker
instance, e.g.:
You can now start the broker by executing:
"/path/to/broker/bin/artemis" run
Or you can run the broker in the background using:
"/path/to/broker/bin/artemis-service" start
> Is it possible to automatically restart Artemis from within a shell
script, or some such?
Unix variants have an init system [1] of some kind (e.g. systemd) through
which you configure such functionality. I bet your system admin could help
you with the specifics here. Such details are really beyond the scope of
this list. You definitely shouldn't need to write your own application and
embed ActiveMQ Artemis into it to get this functionality.
Justin
[1] https://en.wikipedia.org/wiki/Init
On Wed, Oct 5, 2022 at 12:01 PM Michael Brennock <mb...@gmail.com>
wrote:
> Thanks for the tip! I confess it took me much longer than I hoped to get a
> word with our sysadmim because they were crazy busy. they did give me some
> promising information though!
>
> The sysadmim affirmed that any process that runs in a terminal, like
> Artemis for example, will be shut down when the terminal is session is shut
> down. I ran a test last night after following their instructions: run
> Artemis like this to get it to run in the background:
>
> ./Artemis run $
>
> Admittedly, 16 hours is twice as many hours as 8, but it still just shut
> down with the same lack of apparently useful information from artemis.log,
> from audit.log, or from the application that is using Artemis as a message
> broker.
>
> I have two questions:
>
> 1. Is there any way to make sure a Java process runs indefinitely on a unix
> (AIX) environment that I should know about?
> 2. Is it possible to automatically restart Artemis from within a shell
> script, or some such? I'd consider embedding Artemis into the application
> itself to control Artemis, but my marching orders are to find a way to get
> the application to interface with a standalone message queue if possible.
>
> I appreciate your help so far everyone!
>
> Michael
>
> On Tue, Sep 20, 2022, 10:51 AM Justin Bertram <jb...@apache.org> wrote:
>
> > Based on the logging it looks to me like the broker was simply stopped
> > administratively. I definitely think the first step is checking with the
> > sysadmins.
> >
> >
> > Justin
> >
> > On Tue, Sep 20, 2022 at 12:23 PM Michael Brennock <mb...@gmail.com>
> > wrote:
> >
> > > The artemis.log files show the server going offline at random times
> > between
> > > 12:00am and 2:00am on different nights, after being online somewhere
> > > between 8 and 13 hours.
> > >
> > > Here's part of the stack trace:
> > >
> > > 2022-09-20 01:07:09,359 INFO [io.hawt.web.auth.AuthenticationFilter]
> > > Destroying hawtio authentication filter
> > > 2022-09-20 01:07:09,369 INFO [io.hawt.HawtioContextListener]
> Destroying
> > > hawtio services
> > > 2022-09-20 01:07:09,472 INFO
> > > [org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed
> > > artemis-plugin plugin
> > > 2022-09-20 01:07:09,524 INFO
> > > [org.apache.activemq.hawtio.branding.PluginCont
> > > 2022-09-20 01:07:09,591 INFO [org.apache.activemq.artemis.core.server]
> > > AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.19.1
> > > [6bb50282-27d7-11ed-af01-6ab9759765a1] stopped, uptime 13 hours 58
> > minutes
> > >
> > > For more background, I have a java application that is successfully
> > > creating and using queues. As far as I can tell, the application isn't
> > > posting any unusual messages to the queues. The application isn't
> posting
> > > any messages within hours of the artemis server going offline.
> > >
> > > (As I write this, I realize I should contact the sysadmins to see if
> they
> > > have any maintenance programs running at that hour)
> > >
> > > In the meantime, do you guys have any thoughts on where I should look?
> I
> > > should also clarify that this application previously used embedded
> > > activemq, but we are evaluating artemis as a replacement MQ. Someone
> > > realized they could swap the activemq broker for artemis, and I'm
> > following
> > > their work, I wonder if they or I overlooked something when we took
> this
> > > shortcut?
> > >
> > > Thanks for your help,
> > >
> > > Michael
> > >
> > >
> > > On Mon, Sep 19, 2022 at 11:18 AM Justin Bertram <jb...@apache.org>
> > > wrote:
> > >
> > > > Ekta, this sounds like an issue worth discussing. However, it doesn't
> > > seem
> > > > directly related to Michael's question. I recommend you start your
> own
> > > > thread about your issue rather than using this thread. Thanks!
> > > >
> > > >
> > > > Justin
> > > >
> > > > On Mon, Sep 19, 2022 at 12:53 PM Ekta Awasthi
> > > > <Ek...@officedepot.com.invalid> wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > My apache artemis server running on 2.22.0 is also crashing
> randomly,
> > > > upon
> > > > > checking it turns out that the OS kernel is killing it with below
> > > error.
> > > > > To my understanding the process with highest consuming memory
> should
> > be
> > > > > getting killed but this is NOT the case when I am checking metrics
> > > from a
> > > > > metric tool. The memory and all are well under where it should be,
> > but
> > > I
> > > > am
> > > > > following up with my OS team to see if they can point me in the
> right
> > > > > direction or something is off with the version itself...?
> > > > >
> > > > > Out of memory: Kill process 21457 (java) score 852 or sacrifice
> child
> > > > >
> > > > > Can you also try to run below command and see if you are facing the
> > > same.
> > > > > Thanks
> > > > >
> > > > > dmesg | grep "your_process_ID"
> > > > >
> > > > > Thanks
> > > > > Ekta
> > > > >
> > > > > ________________________________
> > > > > From: Michael Brennock <mb...@gmail.com>
> > > > > Sent: Monday, September 19, 2022 1:38 PM
> > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > Subject: Newbie Looking for Help with a Crashing Artemis Server
> > > > >
> > > > > [CAUTION: EXTERNAL SENDER]
> > > > >
> > > > >
> > > > > Good day,
> > > > >
> > > > > I was hoping to ask some of the more experienced Apache Artemis why
> > my
> > > > > server keeps crashing after a few hours. I'm running Artemis 2.19.1
> > on
> > > > the
> > > > > default settings on an IBM AIX box. I can provide a copy of the
> > > > stacktrace,
> > > > > but it doesn't have much information. I'm also trying to be careful
> > not
> > > > to
> > > > > overshare confidential information from work.
> > > > >
> > > > > Thanks for your help,
> > > > >
> > > > > Michael Brennock
> > > > >
> > > > > CONFIDENTIALITY NOTICE: The information contained in this email and
> > > > > attached document(s) may contain confidential information that is
> > > > intended
> > > > > only for the addressee(s). If you are not the intended recipient,
> you
> > > are
> > > > > hereby advised that any disclosure, copying, distribution or the
> > taking
> > > > of
> > > > > any action in reliance upon the information is prohibited. If you
> > have
> > > > > received this email in error, please immediately notify the sender
> > and
> > > > > delete it from your system.
> > > > >
> > > >
> > >
> >
>