[jira] [Comment Edited] (ARTEMIS-3505) Activemq Broker Keeps Crashing

["Ekta (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARTEMIS-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17422848#comment-17422848 ] 

Ekta edited comment on ARTEMIS-3505 at 9/30/21, 3:28 PM:
---------------------------------------------------------

Hello Gary, 

I can certainly provide you the broker.xml. Below is what I see in my broker.xml as far as critical analyzer setting goes.

<!-- should the broker detect dead locks and other issues -->
 <critical-analyzer>true</critical-analyzer>

<critical-analyzer-timeout>120000</critical-analyzer-timeout>

<critical-analyzer-check-period>60000</critical-analyzer-check-period>

<critical-analyzer-policy>HALT</critical-analyzer-policy>

Unfortunately, I won't be able to turn on trace logs on org.apache.activemq.artemis.utils.critical component as we are not able to reproduce this issue. It just randomly happens. 

Regarding what file system is in play, we have a centos ec2 instance with that we have a disk mounted to ec2 instance for additional space like 100gb. 

Thanks. 


was (Author: ekta-awasthi):
Hello Gary, 

I can certainly provide you the broker.xml. Below is what I see in my broker.xml as far as critical analyzer setting goes.

<!-- should the broker detect dead locks and other issues -->
 <critical-analyzer>true</critical-analyzer>

<critical-analyzer-timeout>120000</critical-analyzer-timeout>

<critical-analyzer-check-period>60000</critical-analyzer-check-period>

<critical-analyzer-policy>HALT</critical-analyzer-policy>

Unfortunately, I won't be able to turn on trace logs on org.apache.activemq.artemis.utils.critical component as we are not able to reproduce this issue. It just randomly happens. 

Regarding what file system is in play, we have a centos ec2 instance on top of that we have a disk mounted to ec2 for additional space like 100gb. 

Thanks. 

> Activemq Broker Keeps Crashing
> ------------------------------
>
>                 Key: ARTEMIS-3505
>                 URL: https://issues.apache.org/jira/browse/ARTEMIS-3505
>             Project: ActiveMQ Artemis
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 2.18.0
>         Environment: DEV/UAT/PROD
>            Reporter: Ekta
>            Priority: Critical
>         Attachments: threadDump.txt
>
>
> Hello All,
>   
>  We have noticed the same problem which we reported earlier with 2.17 and were told that it would be fixed in 2.18 version. We have recently moved all our environments to 2.18 version and can see the problem still exists across all of our env's.
>   
>  We have below architecture in respect to activemq master/slave setup.  
> {noformat}
> producer/consumer --> Apache QPID (1.14) --> Artemis 2.18 (master/slave)
> {noformat}
> Basically, we see our master and slave brokers going down abruptly with below log. I have also attached the thread dump for analysis to see if anyone can spot anything, for sure we can see it is to do with some concurrent deadlocks. Please go through the attached logs and suggest any feedback, if any. 
> The log that is causing the issue is highlighted below, as soon as the broker prints this, it prints The Critical Analyzer detected slow paths on the broker. *and therefore*, AMQ224079: The process for the virtual machine will be killed. 
>  2021-09-29 10:37:43,327 WARN  [org.apache.activemq.artemis.utils.critical.CriticalMeasure] Component org.apache.activemq.artemis.core.io.buffer.TimedBuffer is expired on path 4
> It has been happening quite frequently now and we need to come to bottom of this. 
>   
>  Appreciate everyone's effort on this. [^threadDump.txt]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)