You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by "Thomas, Patrick R" <Pa...@questdiagnostics.com> on 2022/09/26 18:37:48 UTC
ActiveMQ Strange Delays (old version)
Hello,
I am new to this mailing list. I built at application years ago using WildFly 10 and the ActiveMQ that is embedded. Recently, on extremely rare occasions, one of our queues slow down. My log files of my application show that messages are delayed by anywhere from 5 to 30 minutes. The queue is on the same server running under the same WildFly instance. One web app is sending another web app a simple message that a file has been received. We're using Spring to create and read the messages. There is very little code on our part, and it has not changed since the beginning.
To add to the mystery, we have 2 other queues that seem unaffected. One queue is virtually the same message being sent back to the previous web app. The third queue is a message being sent from the middle app to another app, and those messages are much larger but are never delayed.
Usually a reboot clears the issue, but today I did a regular reboot and the delay started happening right away, although a smaller delay of only a few minutes. Does anyone have any idea how I can troubleshoot this? Could there be something wrong with my queue, like corruption? Is there any log I can check to be sure the issue is with the queue itself? Any help would be greatly appreciated. It may be a bug that was fixed long ago. I won't be able to update this application any time soon.
Thank you,
Patrick
______________________________________________________________________
The contents of this message, together with any attachments, are intended only for the use of the person(s) to which they are addressed and may contain confidential and/or privileged information. Further, any medical information herein is confidential and protected by law. It is unlawful for unauthorized persons to use, review, copy, disclose, or disseminate confidential medical information. If you are not the intended recipient, immediately advise the sender and delete this message and any attachments. Any distribution, or copying of this message, or any attachment, is prohibited.
Re: ActiveMQ Strange Delays (old version)
Posted by Justin Bertram <jb...@apache.org>.
Please provide a thread dump when you are eventually able to acquire one.
Justin
On Thu, Nov 17, 2022 at 6:50 AM Thomas, Patrick R <
Patrick.R.Thomas@questdiagnostics.com> wrote:
> Both senders and consumers are running in the same WildFly instance. It
> was built to run on different servers, but we never went that far. The
> queues basically send simple messages that say "here is a new file" and "I
> am done with the file". The message of "here is a new file" is the one that
> randomly slows down. The "I am done with the file" message has no issues.
> There is a third queue that gets much larger messages, and it has no issues
> either.
>
> Thank you,
>
> Patrick R. Thomas
>
> -----Original Message-----
> From: Justin Bertram <jb...@apache.org>
> Sent: Wednesday, November 16, 2022 3:56 PM
> To: users@activemq.apache.org
> Subject: Re: ActiveMQ Strange Delays (old version)
>
> CAUTION! This email originated outside of Quest Diagnostics. DO NOT click
> links or open attachments unless you recognize the sender and know the
> content is safe. Please report suspicious emails to:
> phish@questdiagnostics.com
>
>
> The diagnostic data is consistent with a slow or stuck consumer. However,
> without a thread dump from the consumer it's impossible to say for sure.
>
> Is the consumer running in WildFly or on a remote machine?
>
>
> Justin
>
> On Wed, Nov 9, 2022 at 4:00 PM Thomas, Patrick R <
> Patrick.R.Thomas@questdiagnostics.com> wrote:
>
> > This problem is occurring again today after more than a month without
> > issues. I ran the request again, and this time I am seeing the
> > delivering-count rise and fall. It's not completely stuck. It's oddly
> > slow to process very small messages. If I check during normal times,
> > the number is usually 0 or 1.
> >
> > I haven't figured out how to get the thread dump yet. I'm having
> > difficulty figuring out how to connect it to my WildFly instance.
> >
> > {
> > "outcome" => "success",
> > "result" => {
> > "consumer-count" => 1,
> > "dead-letter-address" => "jms.queue.DLQ",
> > "delivering-count" => 17,
> > "durable" => true,
> > "entries" => ["java:jboss/exported/jms/queue/MyQueue"],
> > "expiry-address" => "jms.queue.ExpiryQueue",
> > "legacy-entries" => undefined,
> > "message-count" => 17L,
> > "messages-added" => 17481L,
> > "paused" => false,
> > "queue-address" => "jms.queue.MyQueue",
> > "scheduled-count" => 0L,
> > "selector" => undefined,
> > "temporary" => false
> > }
> > }
> >
> > Thank you,
> >
> > Patrick R. Thomas
> > Software Engineer
> >
> > -----Original Message-----
> > From: Justin Bertram <jb...@apache.org>
> > Sent: Monday, September 26, 2022 4:27 PM
> > To: users@activemq.apache.org
> > Subject: Re: ActiveMQ Strange Delays (old version)
> >
> > CAUTION! This email originated outside of Quest Diagnostics. DO NOT
> > click links or open attachments unless you recognize the sender and
> > know the content is safe. Please report suspicious emails to:
> > phish@questdiagnostics.com
> >
> >
> > You may need to use the WildFly CLI to get all the information you need.
> > I'm not terribly familiar with the WildFly admin console. For example,
> > if you execute this command in the WildFly CLI:
> >
> >
> >
> > /subsystem=messaging-activemq/server=default/jms-queue=myQueue:read-re
> > source(include-runtime=true)
> >
> > You'll get something like this:
> >
> > {
> > "outcome" => "success",
> > "result" => {
> > "consumer-count" => 0,
> > "dead-letter-address" => "jms.queue.myQueue",
> > "delivering-count" => 0,
> > "durable" => true,
> > "entries" => ["java:/jms/queue/myQueue"],
> > "expiry-address" => "jms.queue.ExpiryQueue",
> > "legacy-entries" => undefined,
> > "message-count" => 0L,
> > "messages-added" => 0L,
> > "paused" => false,
> > "queue-address" => "jms.queue.myQueue",
> > "scheduled-count" => 0L,
> > "selector" => undefined,
> > "temporary" => false
> > }
> > }
> >
> > That will give you all the metrics I mentioned previously. The
> > "delivery-count" is a key metric here so it's important to get that if
> > you can.
> >
> > Regarding the metrics you did provide, it looks like there's nothing
> > stuck at that point since there are no messages on the queue. It's
> > important to capture the metrics when the problem actually happens.
> >
> > Regarding the thread dump, check out this info [1]. Again, you need to
> > get a thread dump from the consumer when the problem actually happens.
> >
> > I recommend you set up scripts to gather this data at regular
> > intervals so that the next time the problem arises you can go back and
> > look at what was happening on the server and the client leading up to
> the issue.
> >
> >
> > Justin
> >
> > [1]
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> > baeldung.com%2Fjava-thread-dump&data=05%7C01%7CPatrick.R.Thomas%40
> > questdiagnostics.com%7Cd9d96c16fbaf4ccf209008dac81510d4%7Cb68c6481b22b
> > 46b38c4c0024bb9b9b1f%7C1%7C0%7C638042290056052255%7CUnknown%7CTWFpbGZs
> > b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
> > %7C3000%7C%7C%7C&sdata=QKnZizer23F08TvL4%2BM6zkp6HAHoDV%2F5dRRYfSp
> > K0cc%3D&reserved=0
> >
> > On Mon, Sep 26, 2022 at 2:40 PM Thomas, Patrick R <
> > Patrick.R.Thomas@questdiagnostics.com> wrote:
> >
> > > Thank you for the speedy response. Yes, it is very difficult to
> > > upgrade systems in my particular area for a variety of reasons. I
> > > would not have expected issues with the messaging system to be the
> > > area that caused me problems. Our system only sends a few thousand
> > > messages per day. That seems like a tiny amount compared to what
> > ActiveMQ can handle.
> > >
> > > I am checking this through WildFly's admin console, and this is what
> > > I can
> > > see:
> > >
> > > Consumer Count: 1
> > > Message Count: 0
> > > Messages Added: 1061 (going up gradually)
> > > Scheduled Count: 0
> > >
> > > I'm not sure how to check the thread dump. I've never found any
> > > errors in any of my application's or WildFly's logs.
> > >
> > > Thank you,
> > >
> > > Patrick R. Thomas
> > >
> > > -----Original Message-----
> > > From: Justin Bertram <jb...@apache.org>
> > > Sent: Monday, September 26, 2022 3:20 PM
> > > To: users@activemq.apache.org
> > > Subject: Re: ActiveMQ Strange Delays (old version)
> > >
> > > CAUTION! This email originated outside of Quest Diagnostics. DO NOT
> > > click links or open attachments unless you recognize the sender and
> > > know the content is safe. Please report suspicious emails to:
> > > phish@questdiagnostics.com
> > >
> > >
> > > You weren't kidding about the problem being on an "old version."
> > > WildFly
> > > 10 was released in 2016 and shipped with ActiveMQ Artemis 1.1.0.
> > > There's been almost 50 releases of ActiveMQ Artemis and around 30
> > > releases of WildFly in the last 6 years since then.
> > >
> > > Things I would check:
> > > - On the queue:
> > > - Number of consumers
> > > - Number of messages
> > > - Number of messages in delivery (this indicates how many
> > > messages have been dispatched to consumers but have not yet been
> > > acknowledged)
> > > - On the consumer:
> > > - Thread dump to see if the consumer is stuck for any reason
> > >
> > >
> > > Justin
> > >
> > > On Mon, Sep 26, 2022 at 1:38 PM Thomas, Patrick R <
> > > Patrick.R.Thomas@questdiagnostics.com> wrote:
> > >
> > > > Hello,
> > > >
> > > > I am new to this mailing list. I built at application years ago
> > > > using WildFly 10 and the ActiveMQ that is embedded. Recently, on
> > > > extremely rare occasions, one of our queues slow down. My log
> > > > files of my application show that messages are delayed by anywhere
> > > > from 5 to 30 minutes. The queue is on the same server running
> > > > under the same WildFly instance. One web app is sending another
> > > > web app a simple
> > > message that a file has been received.
> > > > We're using Spring to create and read the messages. There is very
> > > > little code on our part, and it has not changed since the beginning.
> > > >
> > > > To add to the mystery, we have 2 other queues that seem unaffected.
> > > > One queue is virtually the same message being sent back to the
> > > > previous web app. The third queue is a message being sent from the
> > > > middle app to another app, and those messages are much larger but
> > > > are
> > > never delayed.
> > > >
> > > > Usually a reboot clears the issue, but today I did a regular
> > > > reboot and the delay started happening right away, although a
> > > > smaller delay of only a few minutes. Does anyone have any idea how
> > > > I can troubleshoot this? Could there be something wrong with my
> > > > queue, like corruption? Is there any log I can check to be sure
> > > > the issue is with the queue itself? Any help would be greatly
> > > > appreciated. It may be a bug that was fixed long ago. I won't be
> > > > able to update this application
> > > any time soon.
> > > >
> > > > Thank you,
> > > >
> > > > Patrick
> > > >
> > > > __________________________________________________________________
> > > > __ __ The contents of this message, together with any attachments,
> > > > are intended only for the use of the person(s) to which they are
> > > > addressed and may contain confidential and/or privileged
> > > > information. Further, any medical information herein is
> > > > confidential
> > and protected by law.
> > > > It is unlawful for unauthorized persons to use, review, copy,
> > > > disclose, or disseminate confidential medical information. If you
> > > > are not the intended recipient, immediately advise the sender and
> > > > delete
> > > this message and any attachments.
> > > > Any distribution, or copying of this message, or any attachment,
> > > > is prohibited.
> > > >
> > > >
> > >
> > > ____________________________________________________________________
> > > __ The contents of this message, together with any attachments, are
> > > intended only for the use of the person(s) to which they are
> > > addressed and may contain confidential and/or privileged
> > > information. Further, any medical information herein is confidential
> and protected by law.
> > > It is unlawful for unauthorized persons to use, review, copy,
> > > disclose, or disseminate confidential medical information. If you
> > > are not the intended recipient, immediately advise the sender and
> > > delete
> > this message and any attachments.
> > > Any distribution, or copying of this message, or any attachment, is
> > > prohibited.
> > >
> >
> > ______________________________________________________________________
> > The contents of this message, together with any attachments, are
> > intended only for the use of the person(s) to which they are addressed
> > and may contain confidential and/or privileged information. Further,
> > any medical information herein is confidential and protected by law.
> > It is unlawful for unauthorized persons to use, review, copy,
> > disclose, or disseminate confidential medical information. If you are
> > not the intended recipient, immediately advise the sender and delete
> this message and any attachments.
> > Any distribution, or copying of this message, or any attachment, is
> > prohibited.
> >
> >
>
> ______________________________________________________________________
> The contents of this message, together with any attachments, are intended
> only for the use of the person(s) to which they are addressed and may
> contain confidential and/or privileged information. Further, any medical
> information herein is confidential and protected by law. It is unlawful for
> unauthorized persons to use, review, copy, disclose, or disseminate
> confidential medical information. If you are not the intended recipient,
> immediately advise the sender and delete this message and any attachments.
> Any distribution, or copying of this message, or any attachment, is
> prohibited.
>
>
RE: ActiveMQ Strange Delays (old version)
Posted by "Thomas, Patrick R" <Pa...@questdiagnostics.com>.
Both senders and consumers are running in the same WildFly instance. It was built to run on different servers, but we never went that far. The queues basically send simple messages that say "here is a new file" and "I am done with the file". The message of "here is a new file" is the one that randomly slows down. The "I am done with the file" message has no issues. There is a third queue that gets much larger messages, and it has no issues either.
Thank you,
Patrick R. Thomas
-----Original Message-----
From: Justin Bertram <jb...@apache.org>
Sent: Wednesday, November 16, 2022 3:56 PM
To: users@activemq.apache.org
Subject: Re: ActiveMQ Strange Delays (old version)
CAUTION! This email originated outside of Quest Diagnostics. DO NOT click links or open attachments unless you recognize the sender and know the content is safe. Please report suspicious emails to: phish@questdiagnostics.com
The diagnostic data is consistent with a slow or stuck consumer. However, without a thread dump from the consumer it's impossible to say for sure.
Is the consumer running in WildFly or on a remote machine?
Justin
On Wed, Nov 9, 2022 at 4:00 PM Thomas, Patrick R < Patrick.R.Thomas@questdiagnostics.com> wrote:
> This problem is occurring again today after more than a month without
> issues. I ran the request again, and this time I am seeing the
> delivering-count rise and fall. It's not completely stuck. It's oddly
> slow to process very small messages. If I check during normal times,
> the number is usually 0 or 1.
>
> I haven't figured out how to get the thread dump yet. I'm having
> difficulty figuring out how to connect it to my WildFly instance.
>
> {
> "outcome" => "success",
> "result" => {
> "consumer-count" => 1,
> "dead-letter-address" => "jms.queue.DLQ",
> "delivering-count" => 17,
> "durable" => true,
> "entries" => ["java:jboss/exported/jms/queue/MyQueue"],
> "expiry-address" => "jms.queue.ExpiryQueue",
> "legacy-entries" => undefined,
> "message-count" => 17L,
> "messages-added" => 17481L,
> "paused" => false,
> "queue-address" => "jms.queue.MyQueue",
> "scheduled-count" => 0L,
> "selector" => undefined,
> "temporary" => false
> }
> }
>
> Thank you,
>
> Patrick R. Thomas
> Software Engineer
>
> -----Original Message-----
> From: Justin Bertram <jb...@apache.org>
> Sent: Monday, September 26, 2022 4:27 PM
> To: users@activemq.apache.org
> Subject: Re: ActiveMQ Strange Delays (old version)
>
> CAUTION! This email originated outside of Quest Diagnostics. DO NOT
> click links or open attachments unless you recognize the sender and
> know the content is safe. Please report suspicious emails to:
> phish@questdiagnostics.com
>
>
> You may need to use the WildFly CLI to get all the information you need.
> I'm not terribly familiar with the WildFly admin console. For example,
> if you execute this command in the WildFly CLI:
>
>
>
> /subsystem=messaging-activemq/server=default/jms-queue=myQueue:read-re
> source(include-runtime=true)
>
> You'll get something like this:
>
> {
> "outcome" => "success",
> "result" => {
> "consumer-count" => 0,
> "dead-letter-address" => "jms.queue.myQueue",
> "delivering-count" => 0,
> "durable" => true,
> "entries" => ["java:/jms/queue/myQueue"],
> "expiry-address" => "jms.queue.ExpiryQueue",
> "legacy-entries" => undefined,
> "message-count" => 0L,
> "messages-added" => 0L,
> "paused" => false,
> "queue-address" => "jms.queue.myQueue",
> "scheduled-count" => 0L,
> "selector" => undefined,
> "temporary" => false
> }
> }
>
> That will give you all the metrics I mentioned previously. The
> "delivery-count" is a key metric here so it's important to get that if
> you can.
>
> Regarding the metrics you did provide, it looks like there's nothing
> stuck at that point since there are no messages on the queue. It's
> important to capture the metrics when the problem actually happens.
>
> Regarding the thread dump, check out this info [1]. Again, you need to
> get a thread dump from the consumer when the problem actually happens.
>
> I recommend you set up scripts to gather this data at regular
> intervals so that the next time the problem arises you can go back and
> look at what was happening on the server and the client leading up to the issue.
>
>
> Justin
>
> [1]
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> baeldung.com%2Fjava-thread-dump&data=05%7C01%7CPatrick.R.Thomas%40
> questdiagnostics.com%7Cd9d96c16fbaf4ccf209008dac81510d4%7Cb68c6481b22b
> 46b38c4c0024bb9b9b1f%7C1%7C0%7C638042290056052255%7CUnknown%7CTWFpbGZs
> b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
> %7C3000%7C%7C%7C&sdata=QKnZizer23F08TvL4%2BM6zkp6HAHoDV%2F5dRRYfSp
> K0cc%3D&reserved=0
>
> On Mon, Sep 26, 2022 at 2:40 PM Thomas, Patrick R <
> Patrick.R.Thomas@questdiagnostics.com> wrote:
>
> > Thank you for the speedy response. Yes, it is very difficult to
> > upgrade systems in my particular area for a variety of reasons. I
> > would not have expected issues with the messaging system to be the
> > area that caused me problems. Our system only sends a few thousand
> > messages per day. That seems like a tiny amount compared to what
> ActiveMQ can handle.
> >
> > I am checking this through WildFly's admin console, and this is what
> > I can
> > see:
> >
> > Consumer Count: 1
> > Message Count: 0
> > Messages Added: 1061 (going up gradually)
> > Scheduled Count: 0
> >
> > I'm not sure how to check the thread dump. I've never found any
> > errors in any of my application's or WildFly's logs.
> >
> > Thank you,
> >
> > Patrick R. Thomas
> >
> > -----Original Message-----
> > From: Justin Bertram <jb...@apache.org>
> > Sent: Monday, September 26, 2022 3:20 PM
> > To: users@activemq.apache.org
> > Subject: Re: ActiveMQ Strange Delays (old version)
> >
> > CAUTION! This email originated outside of Quest Diagnostics. DO NOT
> > click links or open attachments unless you recognize the sender and
> > know the content is safe. Please report suspicious emails to:
> > phish@questdiagnostics.com
> >
> >
> > You weren't kidding about the problem being on an "old version."
> > WildFly
> > 10 was released in 2016 and shipped with ActiveMQ Artemis 1.1.0.
> > There's been almost 50 releases of ActiveMQ Artemis and around 30
> > releases of WildFly in the last 6 years since then.
> >
> > Things I would check:
> > - On the queue:
> > - Number of consumers
> > - Number of messages
> > - Number of messages in delivery (this indicates how many
> > messages have been dispatched to consumers but have not yet been
> > acknowledged)
> > - On the consumer:
> > - Thread dump to see if the consumer is stuck for any reason
> >
> >
> > Justin
> >
> > On Mon, Sep 26, 2022 at 1:38 PM Thomas, Patrick R <
> > Patrick.R.Thomas@questdiagnostics.com> wrote:
> >
> > > Hello,
> > >
> > > I am new to this mailing list. I built at application years ago
> > > using WildFly 10 and the ActiveMQ that is embedded. Recently, on
> > > extremely rare occasions, one of our queues slow down. My log
> > > files of my application show that messages are delayed by anywhere
> > > from 5 to 30 minutes. The queue is on the same server running
> > > under the same WildFly instance. One web app is sending another
> > > web app a simple
> > message that a file has been received.
> > > We're using Spring to create and read the messages. There is very
> > > little code on our part, and it has not changed since the beginning.
> > >
> > > To add to the mystery, we have 2 other queues that seem unaffected.
> > > One queue is virtually the same message being sent back to the
> > > previous web app. The third queue is a message being sent from the
> > > middle app to another app, and those messages are much larger but
> > > are
> > never delayed.
> > >
> > > Usually a reboot clears the issue, but today I did a regular
> > > reboot and the delay started happening right away, although a
> > > smaller delay of only a few minutes. Does anyone have any idea how
> > > I can troubleshoot this? Could there be something wrong with my
> > > queue, like corruption? Is there any log I can check to be sure
> > > the issue is with the queue itself? Any help would be greatly
> > > appreciated. It may be a bug that was fixed long ago. I won't be
> > > able to update this application
> > any time soon.
> > >
> > > Thank you,
> > >
> > > Patrick
> > >
> > > __________________________________________________________________
> > > __ __ The contents of this message, together with any attachments,
> > > are intended only for the use of the person(s) to which they are
> > > addressed and may contain confidential and/or privileged
> > > information. Further, any medical information herein is
> > > confidential
> and protected by law.
> > > It is unlawful for unauthorized persons to use, review, copy,
> > > disclose, or disseminate confidential medical information. If you
> > > are not the intended recipient, immediately advise the sender and
> > > delete
> > this message and any attachments.
> > > Any distribution, or copying of this message, or any attachment,
> > > is prohibited.
> > >
> > >
> >
> > ____________________________________________________________________
> > __ The contents of this message, together with any attachments, are
> > intended only for the use of the person(s) to which they are
> > addressed and may contain confidential and/or privileged
> > information. Further, any medical information herein is confidential and protected by law.
> > It is unlawful for unauthorized persons to use, review, copy,
> > disclose, or disseminate confidential medical information. If you
> > are not the intended recipient, immediately advise the sender and
> > delete
> this message and any attachments.
> > Any distribution, or copying of this message, or any attachment, is
> > prohibited.
> >
>
> ______________________________________________________________________
> The contents of this message, together with any attachments, are
> intended only for the use of the person(s) to which they are addressed
> and may contain confidential and/or privileged information. Further,
> any medical information herein is confidential and protected by law.
> It is unlawful for unauthorized persons to use, review, copy,
> disclose, or disseminate confidential medical information. If you are
> not the intended recipient, immediately advise the sender and delete this message and any attachments.
> Any distribution, or copying of this message, or any attachment, is
> prohibited.
>
>
______________________________________________________________________
The contents of this message, together with any attachments, are intended only for the use of the person(s) to which they are addressed and may contain confidential and/or privileged information. Further, any medical information herein is confidential and protected by law. It is unlawful for unauthorized persons to use, review, copy, disclose, or disseminate confidential medical information. If you are not the intended recipient, immediately advise the sender and delete this message and any attachments. Any distribution, or copying of this message, or any attachment, is prohibited.
Re: ActiveMQ Strange Delays (old version)
Posted by Justin Bertram <jb...@apache.org>.
The diagnostic data is consistent with a slow or stuck consumer. However,
without a thread dump from the consumer it's impossible to say for sure.
Is the consumer running in WildFly or on a remote machine?
Justin
On Wed, Nov 9, 2022 at 4:00 PM Thomas, Patrick R <
Patrick.R.Thomas@questdiagnostics.com> wrote:
> This problem is occurring again today after more than a month without
> issues. I ran the request again, and this time I am seeing the
> delivering-count rise and fall. It's not completely stuck. It's oddly slow
> to process very small messages. If I check during normal times, the number
> is usually 0 or 1.
>
> I haven't figured out how to get the thread dump yet. I'm having
> difficulty figuring out how to connect it to my WildFly instance.
>
> {
> "outcome" => "success",
> "result" => {
> "consumer-count" => 1,
> "dead-letter-address" => "jms.queue.DLQ",
> "delivering-count" => 17,
> "durable" => true,
> "entries" => ["java:jboss/exported/jms/queue/MyQueue"],
> "expiry-address" => "jms.queue.ExpiryQueue",
> "legacy-entries" => undefined,
> "message-count" => 17L,
> "messages-added" => 17481L,
> "paused" => false,
> "queue-address" => "jms.queue.MyQueue",
> "scheduled-count" => 0L,
> "selector" => undefined,
> "temporary" => false
> }
> }
>
> Thank you,
>
> Patrick R. Thomas
> Software Engineer
>
> Quest Diagnostics | Action from Insight | 14225 Newbrook Drive |
> Chantilly, VA 20151 USA | phone 703.802.6900 x67351 | fax 703.802.7107
> | Patrick.R.Thomas@QuestDiagnostics.com | QuestDiagnostics.com
>
> -----Original Message-----
> From: Justin Bertram <jb...@apache.org>
> Sent: Monday, September 26, 2022 4:27 PM
> To: users@activemq.apache.org
> Subject: Re: ActiveMQ Strange Delays (old version)
>
> CAUTION! This email originated outside of Quest Diagnostics. DO NOT click
> links or open attachments unless you recognize the sender and know the
> content is safe. Please report suspicious emails to:
> phish@questdiagnostics.com
>
>
> You may need to use the WildFly CLI to get all the information you need.
> I'm not terribly familiar with the WildFly admin console. For example, if
> you execute this command in the WildFly CLI:
>
>
>
> /subsystem=messaging-activemq/server=default/jms-queue=myQueue:read-resource(include-runtime=true)
>
> You'll get something like this:
>
> {
> "outcome" => "success",
> "result" => {
> "consumer-count" => 0,
> "dead-letter-address" => "jms.queue.myQueue",
> "delivering-count" => 0,
> "durable" => true,
> "entries" => ["java:/jms/queue/myQueue"],
> "expiry-address" => "jms.queue.ExpiryQueue",
> "legacy-entries" => undefined,
> "message-count" => 0L,
> "messages-added" => 0L,
> "paused" => false,
> "queue-address" => "jms.queue.myQueue",
> "scheduled-count" => 0L,
> "selector" => undefined,
> "temporary" => false
> }
> }
>
> That will give you all the metrics I mentioned previously. The
> "delivery-count" is a key metric here so it's important to get that if you
> can.
>
> Regarding the metrics you did provide, it looks like there's nothing stuck
> at that point since there are no messages on the queue. It's important to
> capture the metrics when the problem actually happens.
>
> Regarding the thread dump, check out this info [1]. Again, you need to get
> a thread dump from the consumer when the problem actually happens.
>
> I recommend you set up scripts to gather this data at regular intervals so
> that the next time the problem arises you can go back and look at what was
> happening on the server and the client leading up to the issue.
>
>
> Justin
>
> [1]
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.baeldung.com%2Fjava-thread-dump&data=05%7C01%7CPatrick.R.Thomas%40questdiagnostics.com%7Ce1e7c55cc6fa4fc0e47708da9ffd7eb2%7Cb68c6481b22b46b38c4c0024bb9b9b1f%7C1%7C0%7C637998208343868967%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=DQPdPE3e7bRb9sVWx92U2oidj1bv4X4lm8OxN0BhlWs%3D&reserved=0
>
> On Mon, Sep 26, 2022 at 2:40 PM Thomas, Patrick R <
> Patrick.R.Thomas@questdiagnostics.com> wrote:
>
> > Thank you for the speedy response. Yes, it is very difficult to
> > upgrade systems in my particular area for a variety of reasons. I
> > would not have expected issues with the messaging system to be the
> > area that caused me problems. Our system only sends a few thousand
> > messages per day. That seems like a tiny amount compared to what
> ActiveMQ can handle.
> >
> > I am checking this through WildFly's admin console, and this is what I
> > can
> > see:
> >
> > Consumer Count: 1
> > Message Count: 0
> > Messages Added: 1061 (going up gradually)
> > Scheduled Count: 0
> >
> > I'm not sure how to check the thread dump. I've never found any errors
> > in any of my application's or WildFly's logs.
> >
> > Thank you,
> >
> > Patrick R. Thomas
> >
> > -----Original Message-----
> > From: Justin Bertram <jb...@apache.org>
> > Sent: Monday, September 26, 2022 3:20 PM
> > To: users@activemq.apache.org
> > Subject: Re: ActiveMQ Strange Delays (old version)
> >
> > CAUTION! This email originated outside of Quest Diagnostics. DO NOT
> > click links or open attachments unless you recognize the sender and
> > know the content is safe. Please report suspicious emails to:
> > phish@questdiagnostics.com
> >
> >
> > You weren't kidding about the problem being on an "old version."
> > WildFly
> > 10 was released in 2016 and shipped with ActiveMQ Artemis 1.1.0.
> > There's been almost 50 releases of ActiveMQ Artemis and around 30
> > releases of WildFly in the last 6 years since then.
> >
> > Things I would check:
> > - On the queue:
> > - Number of consumers
> > - Number of messages
> > - Number of messages in delivery (this indicates how many messages
> > have been dispatched to consumers but have not yet been acknowledged)
> > - On the consumer:
> > - Thread dump to see if the consumer is stuck for any reason
> >
> >
> > Justin
> >
> > On Mon, Sep 26, 2022 at 1:38 PM Thomas, Patrick R <
> > Patrick.R.Thomas@questdiagnostics.com> wrote:
> >
> > > Hello,
> > >
> > > I am new to this mailing list. I built at application years ago
> > > using WildFly 10 and the ActiveMQ that is embedded. Recently, on
> > > extremely rare occasions, one of our queues slow down. My log files
> > > of my application show that messages are delayed by anywhere from 5
> > > to 30 minutes. The queue is on the same server running under the
> > > same WildFly instance. One web app is sending another web app a
> > > simple
> > message that a file has been received.
> > > We're using Spring to create and read the messages. There is very
> > > little code on our part, and it has not changed since the beginning.
> > >
> > > To add to the mystery, we have 2 other queues that seem unaffected.
> > > One queue is virtually the same message being sent back to the
> > > previous web app. The third queue is a message being sent from the
> > > middle app to another app, and those messages are much larger but
> > > are
> > never delayed.
> > >
> > > Usually a reboot clears the issue, but today I did a regular reboot
> > > and the delay started happening right away, although a smaller delay
> > > of only a few minutes. Does anyone have any idea how I can
> > > troubleshoot this? Could there be something wrong with my queue,
> > > like corruption? Is there any log I can check to be sure the issue
> > > is with the queue itself? Any help would be greatly appreciated. It
> > > may be a bug that was fixed long ago. I won't be able to update this
> > > application
> > any time soon.
> > >
> > > Thank you,
> > >
> > > Patrick
> > >
> > > ____________________________________________________________________
> > > __ The contents of this message, together with any attachments, are
> > > intended only for the use of the person(s) to which they are
> > > addressed and may contain confidential and/or privileged
> > > information. Further, any medical information herein is confidential
> and protected by law.
> > > It is unlawful for unauthorized persons to use, review, copy,
> > > disclose, or disseminate confidential medical information. If you
> > > are not the intended recipient, immediately advise the sender and
> > > delete
> > this message and any attachments.
> > > Any distribution, or copying of this message, or any attachment, is
> > > prohibited.
> > >
> > >
> >
> > ______________________________________________________________________
> > The contents of this message, together with any attachments, are
> > intended only for the use of the person(s) to which they are addressed
> > and may contain confidential and/or privileged information. Further,
> > any medical information herein is confidential and protected by law.
> > It is unlawful for unauthorized persons to use, review, copy,
> > disclose, or disseminate confidential medical information. If you are
> > not the intended recipient, immediately advise the sender and delete
> this message and any attachments.
> > Any distribution, or copying of this message, or any attachment, is
> > prohibited.
> >
>
> ______________________________________________________________________
> The contents of this message, together with any attachments, are intended
> only for the use of the person(s) to which they are addressed and may
> contain confidential and/or privileged information. Further, any medical
> information herein is confidential and protected by law. It is unlawful for
> unauthorized persons to use, review, copy, disclose, or disseminate
> confidential medical information. If you are not the intended recipient,
> immediately advise the sender and delete this message and any attachments.
> Any distribution, or copying of this message, or any attachment, is
> prohibited.
>
>
RE: ActiveMQ Strange Delays (old version)
Posted by "Thomas, Patrick R" <Pa...@questdiagnostics.com>.
This problem is occurring again today after more than a month without issues. I ran the request again, and this time I am seeing the delivering-count rise and fall. It's not completely stuck. It's oddly slow to process very small messages. If I check during normal times, the number is usually 0 or 1.
I haven't figured out how to get the thread dump yet. I'm having difficulty figuring out how to connect it to my WildFly instance.
{
"outcome" => "success",
"result" => {
"consumer-count" => 1,
"dead-letter-address" => "jms.queue.DLQ",
"delivering-count" => 17,
"durable" => true,
"entries" => ["java:jboss/exported/jms/queue/MyQueue"],
"expiry-address" => "jms.queue.ExpiryQueue",
"legacy-entries" => undefined,
"message-count" => 17L,
"messages-added" => 17481L,
"paused" => false,
"queue-address" => "jms.queue.MyQueue",
"scheduled-count" => 0L,
"selector" => undefined,
"temporary" => false
}
}
Thank you,
Patrick R. Thomas
Software Engineer
Quest Diagnostics | Action from Insight | 14225 Newbrook Drive | Chantilly, VA 20151 USA | phone 703.802.6900 x67351 | fax 703.802.7107 | Patrick.R.Thomas@QuestDiagnostics.com | QuestDiagnostics.com
-----Original Message-----
From: Justin Bertram <jb...@apache.org>
Sent: Monday, September 26, 2022 4:27 PM
To: users@activemq.apache.org
Subject: Re: ActiveMQ Strange Delays (old version)
CAUTION! This email originated outside of Quest Diagnostics. DO NOT click links or open attachments unless you recognize the sender and know the content is safe. Please report suspicious emails to: phish@questdiagnostics.com
You may need to use the WildFly CLI to get all the information you need.
I'm not terribly familiar with the WildFly admin console. For example, if you execute this command in the WildFly CLI:
/subsystem=messaging-activemq/server=default/jms-queue=myQueue:read-resource(include-runtime=true)
You'll get something like this:
{
"outcome" => "success",
"result" => {
"consumer-count" => 0,
"dead-letter-address" => "jms.queue.myQueue",
"delivering-count" => 0,
"durable" => true,
"entries" => ["java:/jms/queue/myQueue"],
"expiry-address" => "jms.queue.ExpiryQueue",
"legacy-entries" => undefined,
"message-count" => 0L,
"messages-added" => 0L,
"paused" => false,
"queue-address" => "jms.queue.myQueue",
"scheduled-count" => 0L,
"selector" => undefined,
"temporary" => false
}
}
That will give you all the metrics I mentioned previously. The "delivery-count" is a key metric here so it's important to get that if you can.
Regarding the metrics you did provide, it looks like there's nothing stuck at that point since there are no messages on the queue. It's important to capture the metrics when the problem actually happens.
Regarding the thread dump, check out this info [1]. Again, you need to get a thread dump from the consumer when the problem actually happens.
I recommend you set up scripts to gather this data at regular intervals so that the next time the problem arises you can go back and look at what was happening on the server and the client leading up to the issue.
Justin
[1] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.baeldung.com%2Fjava-thread-dump&data=05%7C01%7CPatrick.R.Thomas%40questdiagnostics.com%7Ce1e7c55cc6fa4fc0e47708da9ffd7eb2%7Cb68c6481b22b46b38c4c0024bb9b9b1f%7C1%7C0%7C637998208343868967%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=DQPdPE3e7bRb9sVWx92U2oidj1bv4X4lm8OxN0BhlWs%3D&reserved=0
On Mon, Sep 26, 2022 at 2:40 PM Thomas, Patrick R < Patrick.R.Thomas@questdiagnostics.com> wrote:
> Thank you for the speedy response. Yes, it is very difficult to
> upgrade systems in my particular area for a variety of reasons. I
> would not have expected issues with the messaging system to be the
> area that caused me problems. Our system only sends a few thousand
> messages per day. That seems like a tiny amount compared to what ActiveMQ can handle.
>
> I am checking this through WildFly's admin console, and this is what I
> can
> see:
>
> Consumer Count: 1
> Message Count: 0
> Messages Added: 1061 (going up gradually)
> Scheduled Count: 0
>
> I'm not sure how to check the thread dump. I've never found any errors
> in any of my application's or WildFly's logs.
>
> Thank you,
>
> Patrick R. Thomas
>
> -----Original Message-----
> From: Justin Bertram <jb...@apache.org>
> Sent: Monday, September 26, 2022 3:20 PM
> To: users@activemq.apache.org
> Subject: Re: ActiveMQ Strange Delays (old version)
>
> CAUTION! This email originated outside of Quest Diagnostics. DO NOT
> click links or open attachments unless you recognize the sender and
> know the content is safe. Please report suspicious emails to:
> phish@questdiagnostics.com
>
>
> You weren't kidding about the problem being on an "old version."
> WildFly
> 10 was released in 2016 and shipped with ActiveMQ Artemis 1.1.0.
> There's been almost 50 releases of ActiveMQ Artemis and around 30
> releases of WildFly in the last 6 years since then.
>
> Things I would check:
> - On the queue:
> - Number of consumers
> - Number of messages
> - Number of messages in delivery (this indicates how many messages
> have been dispatched to consumers but have not yet been acknowledged)
> - On the consumer:
> - Thread dump to see if the consumer is stuck for any reason
>
>
> Justin
>
> On Mon, Sep 26, 2022 at 1:38 PM Thomas, Patrick R <
> Patrick.R.Thomas@questdiagnostics.com> wrote:
>
> > Hello,
> >
> > I am new to this mailing list. I built at application years ago
> > using WildFly 10 and the ActiveMQ that is embedded. Recently, on
> > extremely rare occasions, one of our queues slow down. My log files
> > of my application show that messages are delayed by anywhere from 5
> > to 30 minutes. The queue is on the same server running under the
> > same WildFly instance. One web app is sending another web app a
> > simple
> message that a file has been received.
> > We're using Spring to create and read the messages. There is very
> > little code on our part, and it has not changed since the beginning.
> >
> > To add to the mystery, we have 2 other queues that seem unaffected.
> > One queue is virtually the same message being sent back to the
> > previous web app. The third queue is a message being sent from the
> > middle app to another app, and those messages are much larger but
> > are
> never delayed.
> >
> > Usually a reboot clears the issue, but today I did a regular reboot
> > and the delay started happening right away, although a smaller delay
> > of only a few minutes. Does anyone have any idea how I can
> > troubleshoot this? Could there be something wrong with my queue,
> > like corruption? Is there any log I can check to be sure the issue
> > is with the queue itself? Any help would be greatly appreciated. It
> > may be a bug that was fixed long ago. I won't be able to update this
> > application
> any time soon.
> >
> > Thank you,
> >
> > Patrick
> >
> > ____________________________________________________________________
> > __ The contents of this message, together with any attachments, are
> > intended only for the use of the person(s) to which they are
> > addressed and may contain confidential and/or privileged
> > information. Further, any medical information herein is confidential and protected by law.
> > It is unlawful for unauthorized persons to use, review, copy,
> > disclose, or disseminate confidential medical information. If you
> > are not the intended recipient, immediately advise the sender and
> > delete
> this message and any attachments.
> > Any distribution, or copying of this message, or any attachment, is
> > prohibited.
> >
> >
>
> ______________________________________________________________________
> The contents of this message, together with any attachments, are
> intended only for the use of the person(s) to which they are addressed
> and may contain confidential and/or privileged information. Further,
> any medical information herein is confidential and protected by law.
> It is unlawful for unauthorized persons to use, review, copy,
> disclose, or disseminate confidential medical information. If you are
> not the intended recipient, immediately advise the sender and delete this message and any attachments.
> Any distribution, or copying of this message, or any attachment, is
> prohibited.
>
______________________________________________________________________
The contents of this message, together with any attachments, are intended only for the use of the person(s) to which they are addressed and may contain confidential and/or privileged information. Further, any medical information herein is confidential and protected by law. It is unlawful for unauthorized persons to use, review, copy, disclose, or disseminate confidential medical information. If you are not the intended recipient, immediately advise the sender and delete this message and any attachments. Any distribution, or copying of this message, or any attachment, is prohibited.
RE: ActiveMQ Strange Delays (old version)
Posted by "Thomas, Patrick R" <Pa...@questdiagnostics.com>.
Here is the information from the command. Today's problem seems to have resolved itself. The first time it happened, the delay was 30 minutes. I had to reboot right away because of the backlog it was creating. I didn't get another report of it happening for at least a year and a half. It is possible that smaller delays happen without my knowledge. It's a 24/7 application, and users may just wait to see if resolves itself if they are running off hours. The last report I got was a month or so ago, so it may be happening more frequently. I am simultaneously pursuing other possibilities including issues with the VM itself.
I will keep these tips handy for the next time it happens. I really appreciate your help.
{
"outcome" => "success",
"result" => {
"consumer-count" => 1,
"dead-letter-address" => "jms.queue.DLQ",
"delivering-count" => 0,
"durable" => true,
"entries" => ["java:jboss/exported/jms/queue/MyQueue"],
"expiry-address" => "jms.queue.ExpiryQueue",
"legacy-entries" => undefined,
"message-count" => 0L,
"messages-added" => 1211L,
"paused" => false,
"queue-address" => "jms.queue.MyQueue",
"scheduled-count" => 0L,
"selector" => undefined,
"temporary" => false
}
}
Thank you,
Patrick
-----Original Message-----
From: Justin Bertram <jb...@apache.org>
Sent: Monday, September 26, 2022 4:27 PM
To: users@activemq.apache.org
Subject: Re: ActiveMQ Strange Delays (old version)
CAUTION! This email originated outside of Quest Diagnostics. DO NOT click links or open attachments unless you recognize the sender and know the content is safe. Please report suspicious emails to: phish@questdiagnostics.com
You may need to use the WildFly CLI to get all the information you need.
I'm not terribly familiar with the WildFly admin console. For example, if you execute this command in the WildFly CLI:
/subsystem=messaging-activemq/server=default/jms-queue=myQueue:read-resource(include-runtime=true)
You'll get something like this:
{
"outcome" => "success",
"result" => {
"consumer-count" => 0,
"dead-letter-address" => "jms.queue.myQueue",
"delivering-count" => 0,
"durable" => true,
"entries" => ["java:/jms/queue/myQueue"],
"expiry-address" => "jms.queue.ExpiryQueue",
"legacy-entries" => undefined,
"message-count" => 0L,
"messages-added" => 0L,
"paused" => false,
"queue-address" => "jms.queue.myQueue",
"scheduled-count" => 0L,
"selector" => undefined,
"temporary" => false
}
}
That will give you all the metrics I mentioned previously. The "delivery-count" is a key metric here so it's important to get that if you can.
Regarding the metrics you did provide, it looks like there's nothing stuck at that point since there are no messages on the queue. It's important to capture the metrics when the problem actually happens.
Regarding the thread dump, check out this info [1]. Again, you need to get a thread dump from the consumer when the problem actually happens.
I recommend you set up scripts to gather this data at regular intervals so that the next time the problem arises you can go back and look at what was happening on the server and the client leading up to the issue.
Justin
[1] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.baeldung.com%2Fjava-thread-dump&data=05%7C01%7CPatrick.R.Thomas%40questdiagnostics.com%7Ce1e7c55cc6fa4fc0e47708da9ffd7eb2%7Cb68c6481b22b46b38c4c0024bb9b9b1f%7C1%7C0%7C637998208343868967%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=DQPdPE3e7bRb9sVWx92U2oidj1bv4X4lm8OxN0BhlWs%3D&reserved=0
On Mon, Sep 26, 2022 at 2:40 PM Thomas, Patrick R < Patrick.R.Thomas@questdiagnostics.com> wrote:
> Thank you for the speedy response. Yes, it is very difficult to
> upgrade systems in my particular area for a variety of reasons. I
> would not have expected issues with the messaging system to be the
> area that caused me problems. Our system only sends a few thousand
> messages per day. That seems like a tiny amount compared to what ActiveMQ can handle.
>
> I am checking this through WildFly's admin console, and this is what I
> can
> see:
>
> Consumer Count: 1
> Message Count: 0
> Messages Added: 1061 (going up gradually)
> Scheduled Count: 0
>
> I'm not sure how to check the thread dump. I've never found any errors
> in any of my application's or WildFly's logs.
>
> Thank you,
>
> Patrick R. Thomas
>
> -----Original Message-----
> From: Justin Bertram <jb...@apache.org>
> Sent: Monday, September 26, 2022 3:20 PM
> To: users@activemq.apache.org
> Subject: Re: ActiveMQ Strange Delays (old version)
>
> CAUTION! This email originated outside of Quest Diagnostics. DO NOT
> click links or open attachments unless you recognize the sender and
> know the content is safe. Please report suspicious emails to:
> phish@questdiagnostics.com
>
>
> You weren't kidding about the problem being on an "old version."
> WildFly
> 10 was released in 2016 and shipped with ActiveMQ Artemis 1.1.0.
> There's been almost 50 releases of ActiveMQ Artemis and around 30
> releases of WildFly in the last 6 years since then.
>
> Things I would check:
> - On the queue:
> - Number of consumers
> - Number of messages
> - Number of messages in delivery (this indicates how many messages
> have been dispatched to consumers but have not yet been acknowledged)
> - On the consumer:
> - Thread dump to see if the consumer is stuck for any reason
>
>
> Justin
>
> On Mon, Sep 26, 2022 at 1:38 PM Thomas, Patrick R <
> Patrick.R.Thomas@questdiagnostics.com> wrote:
>
> > Hello,
> >
> > I am new to this mailing list. I built at application years ago
> > using WildFly 10 and the ActiveMQ that is embedded. Recently, on
> > extremely rare occasions, one of our queues slow down. My log files
> > of my application show that messages are delayed by anywhere from 5
> > to 30 minutes. The queue is on the same server running under the
> > same WildFly instance. One web app is sending another web app a
> > simple
> message that a file has been received.
> > We're using Spring to create and read the messages. There is very
> > little code on our part, and it has not changed since the beginning.
> >
> > To add to the mystery, we have 2 other queues that seem unaffected.
> > One queue is virtually the same message being sent back to the
> > previous web app. The third queue is a message being sent from the
> > middle app to another app, and those messages are much larger but
> > are
> never delayed.
> >
> > Usually a reboot clears the issue, but today I did a regular reboot
> > and the delay started happening right away, although a smaller delay
> > of only a few minutes. Does anyone have any idea how I can
> > troubleshoot this? Could there be something wrong with my queue,
> > like corruption? Is there any log I can check to be sure the issue
> > is with the queue itself? Any help would be greatly appreciated. It
> > may be a bug that was fixed long ago. I won't be able to update this
> > application
> any time soon.
> >
> > Thank you,
> >
> > Patrick
> >
> > ____________________________________________________________________
> > __ The contents of this message, together with any attachments, are
> > intended only for the use of the person(s) to which they are
> > addressed and may contain confidential and/or privileged
> > information. Further, any medical information herein is confidential and protected by law.
> > It is unlawful for unauthorized persons to use, review, copy,
> > disclose, or disseminate confidential medical information. If you
> > are not the intended recipient, immediately advise the sender and
> > delete
> this message and any attachments.
> > Any distribution, or copying of this message, or any attachment, is
> > prohibited.
> >
> >
>
> ______________________________________________________________________
> The contents of this message, together with any attachments, are
> intended only for the use of the person(s) to which they are addressed
> and may contain confidential and/or privileged information. Further,
> any medical information herein is confidential and protected by law.
> It is unlawful for unauthorized persons to use, review, copy,
> disclose, or disseminate confidential medical information. If you are
> not the intended recipient, immediately advise the sender and delete this message and any attachments.
> Any distribution, or copying of this message, or any attachment, is
> prohibited.
>
______________________________________________________________________
The contents of this message, together with any attachments, are intended only for the use of the person(s) to which they are addressed and may contain confidential and/or privileged information. Further, any medical information herein is confidential and protected by law. It is unlawful for unauthorized persons to use, review, copy, disclose, or disseminate confidential medical information. If you are not the intended recipient, immediately advise the sender and delete this message and any attachments. Any distribution, or copying of this message, or any attachment, is prohibited.
Re: ActiveMQ Strange Delays (old version)
Posted by Justin Bertram <jb...@apache.org>.
You may need to use the WildFly CLI to get all the information you need.
I'm not terribly familiar with the WildFly admin console. For example, if
you execute this command in the WildFly CLI:
/subsystem=messaging-activemq/server=default/jms-queue=myQueue:read-resource(include-runtime=true)
You'll get something like this:
{
"outcome" => "success",
"result" => {
"consumer-count" => 0,
"dead-letter-address" => "jms.queue.myQueue",
"delivering-count" => 0,
"durable" => true,
"entries" => ["java:/jms/queue/myQueue"],
"expiry-address" => "jms.queue.ExpiryQueue",
"legacy-entries" => undefined,
"message-count" => 0L,
"messages-added" => 0L,
"paused" => false,
"queue-address" => "jms.queue.myQueue",
"scheduled-count" => 0L,
"selector" => undefined,
"temporary" => false
}
}
That will give you all the metrics I mentioned previously. The
"delivery-count" is a key metric here so it's important to get that if you
can.
Regarding the metrics you did provide, it looks like there's nothing stuck
at that point since there are no messages on the queue. It's important to
capture the metrics when the problem actually happens.
Regarding the thread dump, check out this info [1]. Again, you need to get
a thread dump from the consumer when the problem actually happens.
I recommend you set up scripts to gather this data at regular intervals so
that the next time the problem arises you can go back and look at what was
happening on the server and the client leading up to the issue.
Justin
[1] https://www.baeldung.com/java-thread-dump
On Mon, Sep 26, 2022 at 2:40 PM Thomas, Patrick R <
Patrick.R.Thomas@questdiagnostics.com> wrote:
> Thank you for the speedy response. Yes, it is very difficult to upgrade
> systems in my particular area for a variety of reasons. I would not have
> expected issues with the messaging system to be the area that caused me
> problems. Our system only sends a few thousand messages per day. That seems
> like a tiny amount compared to what ActiveMQ can handle.
>
> I am checking this through WildFly's admin console, and this is what I can
> see:
>
> Consumer Count: 1
> Message Count: 0
> Messages Added: 1061 (going up gradually)
> Scheduled Count: 0
>
> I'm not sure how to check the thread dump. I've never found any errors in
> any of my application's or WildFly's logs.
>
> Thank you,
>
> Patrick R. Thomas
>
> -----Original Message-----
> From: Justin Bertram <jb...@apache.org>
> Sent: Monday, September 26, 2022 3:20 PM
> To: users@activemq.apache.org
> Subject: Re: ActiveMQ Strange Delays (old version)
>
> CAUTION! This email originated outside of Quest Diagnostics. DO NOT click
> links or open attachments unless you recognize the sender and know the
> content is safe. Please report suspicious emails to:
> phish@questdiagnostics.com
>
>
> You weren't kidding about the problem being on an "old version." WildFly
> 10 was released in 2016 and shipped with ActiveMQ Artemis 1.1.0. There's
> been almost 50 releases of ActiveMQ Artemis and around 30 releases of
> WildFly in the last 6 years since then.
>
> Things I would check:
> - On the queue:
> - Number of consumers
> - Number of messages
> - Number of messages in delivery (this indicates how many messages
> have been dispatched to consumers but have not yet been acknowledged)
> - On the consumer:
> - Thread dump to see if the consumer is stuck for any reason
>
>
> Justin
>
> On Mon, Sep 26, 2022 at 1:38 PM Thomas, Patrick R <
> Patrick.R.Thomas@questdiagnostics.com> wrote:
>
> > Hello,
> >
> > I am new to this mailing list. I built at application years ago using
> > WildFly 10 and the ActiveMQ that is embedded. Recently, on extremely
> > rare occasions, one of our queues slow down. My log files of my
> > application show that messages are delayed by anywhere from 5 to 30
> > minutes. The queue is on the same server running under the same
> > WildFly instance. One web app is sending another web app a simple
> message that a file has been received.
> > We're using Spring to create and read the messages. There is very
> > little code on our part, and it has not changed since the beginning.
> >
> > To add to the mystery, we have 2 other queues that seem unaffected.
> > One queue is virtually the same message being sent back to the
> > previous web app. The third queue is a message being sent from the
> > middle app to another app, and those messages are much larger but are
> never delayed.
> >
> > Usually a reboot clears the issue, but today I did a regular reboot
> > and the delay started happening right away, although a smaller delay
> > of only a few minutes. Does anyone have any idea how I can
> > troubleshoot this? Could there be something wrong with my queue, like
> > corruption? Is there any log I can check to be sure the issue is with
> > the queue itself? Any help would be greatly appreciated. It may be a
> > bug that was fixed long ago. I won't be able to update this application
> any time soon.
> >
> > Thank you,
> >
> > Patrick
> >
> > ______________________________________________________________________
> > The contents of this message, together with any attachments, are
> > intended only for the use of the person(s) to which they are addressed
> > and may contain confidential and/or privileged information. Further,
> > any medical information herein is confidential and protected by law.
> > It is unlawful for unauthorized persons to use, review, copy,
> > disclose, or disseminate confidential medical information. If you are
> > not the intended recipient, immediately advise the sender and delete
> this message and any attachments.
> > Any distribution, or copying of this message, or any attachment, is
> > prohibited.
> >
> >
>
> ______________________________________________________________________
> The contents of this message, together with any attachments, are intended
> only for the use of the person(s) to which they are addressed and may
> contain confidential and/or privileged information. Further, any medical
> information herein is confidential and protected by law. It is unlawful for
> unauthorized persons to use, review, copy, disclose, or disseminate
> confidential medical information. If you are not the intended recipient,
> immediately advise the sender and delete this message and any attachments.
> Any distribution, or copying of this message, or any attachment, is
> prohibited.
>
RE: ActiveMQ Strange Delays (old version)
Posted by "Thomas, Patrick R" <Pa...@questdiagnostics.com>.
Thank you for the speedy response. Yes, it is very difficult to upgrade systems in my particular area for a variety of reasons. I would not have expected issues with the messaging system to be the area that caused me problems. Our system only sends a few thousand messages per day. That seems like a tiny amount compared to what ActiveMQ can handle.
I am checking this through WildFly's admin console, and this is what I can see:
Consumer Count: 1
Message Count: 0
Messages Added: 1061 (going up gradually)
Scheduled Count: 0
I'm not sure how to check the thread dump. I've never found any errors in any of my application's or WildFly's logs.
Thank you,
Patrick R. Thomas
-----Original Message-----
From: Justin Bertram <jb...@apache.org>
Sent: Monday, September 26, 2022 3:20 PM
To: users@activemq.apache.org
Subject: Re: ActiveMQ Strange Delays (old version)
CAUTION! This email originated outside of Quest Diagnostics. DO NOT click links or open attachments unless you recognize the sender and know the content is safe. Please report suspicious emails to: phish@questdiagnostics.com
You weren't kidding about the problem being on an "old version." WildFly 10 was released in 2016 and shipped with ActiveMQ Artemis 1.1.0. There's been almost 50 releases of ActiveMQ Artemis and around 30 releases of WildFly in the last 6 years since then.
Things I would check:
- On the queue:
- Number of consumers
- Number of messages
- Number of messages in delivery (this indicates how many messages have been dispatched to consumers but have not yet been acknowledged)
- On the consumer:
- Thread dump to see if the consumer is stuck for any reason
Justin
On Mon, Sep 26, 2022 at 1:38 PM Thomas, Patrick R < Patrick.R.Thomas@questdiagnostics.com> wrote:
> Hello,
>
> I am new to this mailing list. I built at application years ago using
> WildFly 10 and the ActiveMQ that is embedded. Recently, on extremely
> rare occasions, one of our queues slow down. My log files of my
> application show that messages are delayed by anywhere from 5 to 30
> minutes. The queue is on the same server running under the same
> WildFly instance. One web app is sending another web app a simple message that a file has been received.
> We're using Spring to create and read the messages. There is very
> little code on our part, and it has not changed since the beginning.
>
> To add to the mystery, we have 2 other queues that seem unaffected.
> One queue is virtually the same message being sent back to the
> previous web app. The third queue is a message being sent from the
> middle app to another app, and those messages are much larger but are never delayed.
>
> Usually a reboot clears the issue, but today I did a regular reboot
> and the delay started happening right away, although a smaller delay
> of only a few minutes. Does anyone have any idea how I can
> troubleshoot this? Could there be something wrong with my queue, like
> corruption? Is there any log I can check to be sure the issue is with
> the queue itself? Any help would be greatly appreciated. It may be a
> bug that was fixed long ago. I won't be able to update this application any time soon.
>
> Thank you,
>
> Patrick
>
> ______________________________________________________________________
> The contents of this message, together with any attachments, are
> intended only for the use of the person(s) to which they are addressed
> and may contain confidential and/or privileged information. Further,
> any medical information herein is confidential and protected by law.
> It is unlawful for unauthorized persons to use, review, copy,
> disclose, or disseminate confidential medical information. If you are
> not the intended recipient, immediately advise the sender and delete this message and any attachments.
> Any distribution, or copying of this message, or any attachment, is
> prohibited.
>
>
______________________________________________________________________
The contents of this message, together with any attachments, are intended only for the use of the person(s) to which they are addressed and may contain confidential and/or privileged information. Further, any medical information herein is confidential and protected by law. It is unlawful for unauthorized persons to use, review, copy, disclose, or disseminate confidential medical information. If you are not the intended recipient, immediately advise the sender and delete this message and any attachments. Any distribution, or copying of this message, or any attachment, is prohibited.
Re: ActiveMQ Strange Delays (old version)
Posted by Justin Bertram <jb...@apache.org>.
You weren't kidding about the problem being on an "old version." WildFly 10
was released in 2016 and shipped with ActiveMQ Artemis 1.1.0. There's been
almost 50 releases of ActiveMQ Artemis and around 30 releases of WildFly in
the last 6 years since then.
Things I would check:
- On the queue:
- Number of consumers
- Number of messages
- Number of messages in delivery (this indicates how many messages have
been dispatched to consumers but have not yet been acknowledged)
- On the consumer:
- Thread dump to see if the consumer is stuck for any reason
Justin
On Mon, Sep 26, 2022 at 1:38 PM Thomas, Patrick R <
Patrick.R.Thomas@questdiagnostics.com> wrote:
> Hello,
>
> I am new to this mailing list. I built at application years ago using
> WildFly 10 and the ActiveMQ that is embedded. Recently, on extremely rare
> occasions, one of our queues slow down. My log files of my application show
> that messages are delayed by anywhere from 5 to 30 minutes. The queue is on
> the same server running under the same WildFly instance. One web app is
> sending another web app a simple message that a file has been received.
> We're using Spring to create and read the messages. There is very little
> code on our part, and it has not changed since the beginning.
>
> To add to the mystery, we have 2 other queues that seem unaffected. One
> queue is virtually the same message being sent back to the previous web
> app. The third queue is a message being sent from the middle app to another
> app, and those messages are much larger but are never delayed.
>
> Usually a reboot clears the issue, but today I did a regular reboot and
> the delay started happening right away, although a smaller delay of only a
> few minutes. Does anyone have any idea how I can troubleshoot this? Could
> there be something wrong with my queue, like corruption? Is there any log I
> can check to be sure the issue is with the queue itself? Any help would be
> greatly appreciated. It may be a bug that was fixed long ago. I won't be
> able to update this application any time soon.
>
> Thank you,
>
> Patrick
>
> ______________________________________________________________________
> The contents of this message, together with any attachments, are intended
> only for the use of the person(s) to which they are addressed and may
> contain confidential and/or privileged information. Further, any medical
> information herein is confidential and protected by law. It is unlawful for
> unauthorized persons to use, review, copy, disclose, or disseminate
> confidential medical information. If you are not the intended recipient,
> immediately advise the sender and delete this message and any attachments.
> Any distribution, or copying of this message, or any attachment, is
> prohibited.
>
>