Consumers in Wildfly stops consuming with no visible reason while there are plenty of messages left

["Pachod, Joseph (GE Healthcare)" <Jo...@ge.com>]

Hi

Every few weeks/months, consumers in Wildfly stop consuming and the messages stack up in the broker. We have to restart Wildfly for the consumers to restart consuming.

Logs show no error/warning, both for the broker and Wildfly, no suspicious system event is present and memory consumption is all right.
When doing a thread dump, we have noticed that there is no more thread named ActiveMQ-client-global-threads in Wildfly.
Spring Boot consumers are doing fine.
So we’ve very few information on why consumption stops suddenly while there are plenty of messages.

The only idea we have so far is to activate debug log for Artemis on both the broker and Wildfly, zipping and keeping it to look at it next time it fails.

FYI, we’re deploying our application on premises, on windows servers, on many customers hardware. The issue appeared about 10 times.

We’re currently running on Java 8, Artemis 1.5.5 in a Spring Boot app and Wildfly is in version 14. We’ll start to upgrade our applications to Artemis 2.6 in February.

Thanks in advance

Kind regards,
Joseph PACHOD
Senior Sofware Architect, DoseWatch
GE Healthcare

RE: EXT: Re: Consumers in Wildfly stops consuming with no visible reason while there are plenty of messages left

["Pachod, Joseph (GE Healthcare)" <Jo...@ge.com>]

Thanks for this reply Justin.

Up to now we didn't manage to reproduce it internally, despite our load testing and the like... 

We didn't expect a crazy answer from the mailing list, we tried "in case someone knew of something". And you already give one point for going directly at 2.10 instead of 2.6, so not too bad!

Thanks again

-----Original Message-----
From: Justin Bertram <jb...@apache.org> 
Sent: Friday, January 3, 2020 5:15 PM
To: users@activemq.apache.org
Subject: EXT: Re: Consumers in Wildfly stops consuming with no visible reason while there are plenty of messages left

I recommend you reproduce this on the latest release of ActiveMQ Artemis (i.e. 2.10.1 at this point). The 1.5.5 release is a couple of years old at this point at hundreds (if not thousands) of issues have been resolved since then. It's highly unlikely you're going to get any community support for such an old release. Even 2.6.0 is pretty old at this point. There's been 10 releases since then and over 1,400 commits to the code-base.


Justin


On Fri, Jan 3, 2020 at 8:43 AM Pachod, Joseph (GE Healthcare) < Joseph.Pachod@ge.com> wrote:

> Hi
>
> Every few weeks/months, consumers in Wildfly stop consuming and the 
> messages stack up in the broker. We have to restart Wildfly for the 
> consumers to restart consuming.
>
> Logs show no error/warning, both for the broker and Wildfly, no 
> suspicious system event is present and memory consumption is all right.
> When doing a thread dump, we have noticed that there is no more thread 
> named ActiveMQ-client-global-threads in Wildfly.
> Spring Boot consumers are doing fine.
> So we’ve very few information on why consumption stops suddenly while 
> there are plenty of messages.
>
> The only idea we have so far is to activate debug log for Artemis on 
> both the broker and Wildfly, zipping and keeping it to look at it next 
> time it fails.
>
> FYI, we’re deploying our application on premises, on windows servers, 
> on many customers hardware. The issue appeared about 10 times.
>
> We’re currently running on Java 8, Artemis 1.5.5 in a Spring Boot app 
> and Wildfly is in version 14. We’ll start to upgrade our applications 
> to Artemis 2.6 in February.
>
> Thanks in advance
>
> Kind regards,
> Joseph PACHOD
> Senior Sofware Architect, DoseWatch
> GE Healthcare
>
>

Re: Consumers in Wildfly stops consuming with no visible reason while there are plenty of messages left

[Justin Bertram <jb...@apache.org>]

I recommend you reproduce this on the latest release of ActiveMQ Artemis
(i.e. 2.10.1 at this point). The 1.5.5 release is a couple of years old at
this point at hundreds (if not thousands) of issues have been resolved
since then. It's highly unlikely you're going to get any community support
for such an old release. Even 2.6.0 is pretty old at this point. There's
been 10 releases since then and over 1,400 commits to the code-base.


Justin


On Fri, Jan 3, 2020 at 8:43 AM Pachod, Joseph (GE Healthcare) <
Joseph.Pachod@ge.com> wrote:

> Hi
>
> Every few weeks/months, consumers in Wildfly stop consuming and the
> messages stack up in the broker. We have to restart Wildfly for the
> consumers to restart consuming.
>
> Logs show no error/warning, both for the broker and Wildfly, no suspicious
> system event is present and memory consumption is all right.
> When doing a thread dump, we have noticed that there is no more thread
> named ActiveMQ-client-global-threads in Wildfly.
> Spring Boot consumers are doing fine.
> So we’ve very few information on why consumption stops suddenly while
> there are plenty of messages.
>
> The only idea we have so far is to activate debug log for Artemis on both
> the broker and Wildfly, zipping and keeping it to look at it next time it
> fails.
>
> FYI, we’re deploying our application on premises, on windows servers, on
> many customers hardware. The issue appeared about 10 times.
>
> We’re currently running on Java 8, Artemis 1.5.5 in a Spring Boot app and
> Wildfly is in version 14. We’ll start to upgrade our applications to
> Artemis 2.6 in February.
>
> Thanks in advance
>
> Kind regards,
> Joseph PACHOD
> Senior Sofware Architect, DoseWatch
> GE Healthcare
>
>