[jira] [Commented] (ARTEMIS-3505) Activemq Broker Keeps Crashing

["Justin Bertram (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARTEMIS-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17423404#comment-17423404 ] 

Justin Bertram commented on ARTEMIS-3505:
-----------------------------------------

bq. After reviewing our case, would you still recommend us to try LOG in lower environment and wait & see as we can't reproduce this.

At this point I'm not confident enough to make any strong recommendations. I've never seen a broker expire on so many critical paths before. Each of these needs to be reviewed individually. They each may have an independent cause, but the fact that so many are occurring in one of your environments and not in the other(s) indicates that there may be a root cause for all of them. In any case, without thread dumps it's impossible to tell whether or not you are hitting real deadlocks or just temporary issues.

If you choose to use {{LOG}} then I recommend you monitor your broker closely to ensure it is operating as you expect. If you need guidance on how to set up such monitoring please follow up on the ActiveMQ users mailing list.

In any case, it would be useful to gather regular thread dumps (e.g. every few minutes) until the root cause for this problem is identified.

bq. In case the server is running in the background with some deadlocks, what are the chances it would clear own its own and if so, the time estimate?

A true deadlock will _never_ clear up on its own. That's why the the default {{broker.xml}} uses {{HALT}}.

> Activemq Broker Keeps Crashing
> ------------------------------
>
>                 Key: ARTEMIS-3505
>                 URL: https://issues.apache.org/jira/browse/ARTEMIS-3505
>             Project: ActiveMQ Artemis
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 2.18.0
>         Environment: DEV/UAT/PROD
>            Reporter: Ekta
>            Priority: Critical
>         Attachments: samplebroker.xml, threadDump.txt
>
>
> Hello All,
>   
>  We have noticed the same problem which we reported earlier with 2.17 and were told that it would be fixed in 2.18 version. We have recently moved all our environments to 2.18 version and can see the problem still exists across all of our env's.
>   
>  We have below architecture in respect to activemq master/slave setup.  
> {noformat}
> producer/consumer --> Apache QPID (1.14) --> Artemis 2.18 (master/slave)
> {noformat}
> Basically, we see our master and slave brokers going down abruptly with below log. I have also attached the thread dump for analysis to see if anyone can spot anything, for sure we can see it is to do with some concurrent deadlocks. Please go through the attached logs and suggest any feedback, if any. 
> The log that is causing the issue is highlighted below, as soon as the broker prints this, it prints The Critical Analyzer detected slow paths on the broker. *and therefore*, AMQ224079: The process for the virtual machine will be killed. 
>  2021-09-29 10:37:43,327 WARN  [org.apache.activemq.artemis.utils.critical.CriticalMeasure] Component org.apache.activemq.artemis.core.io.buffer.TimedBuffer is expired on path 4
> It has been happening quite frequently now and we need to come to bottom of this. 
>   
>  Appreciate everyone's effort on this. [^threadDump.txt]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)