[jira] [Commented] (ARTEMIS-2173) Artemis lose messages on broker restarts

["Justin Bertram (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARTEMIS-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013634#comment-17013634 ] 

Justin Bertram commented on ARTEMIS-2173:
-----------------------------------------

I've been circling back on a bunch of Jiras since the beginning of the year, and I invested some time here to see if I could reproduce the issue you observed using [~erik.wramner]'s [JmsTools project|https://github.com/erik-wramner/JmsTools]. In short, I couldn't. Everything looks to be working fine. In other words, *there's no message loss with the default configuration of ActiveMQ Artemis*. I simplified the test greatly by removing Ansible and AWS completely. Here's how I conducted my test:

h4. *High level setup*

* Producer, consumer, & broker running on a single machine.
* ActiveMQ Artemis 2.10.1
* JmsTools 1.10
* Ubuntu Linux

h4. *Steps to reproduce*

* Open a terminal and execute the following to setup the environment:
    ** {{mkdir ~/ARTEMIS-2173}}
    ** {{cd ~/ARTEMIS-2173}}
    ** {{mkdir logs}}
    ** {{mkdir broker}}
    ** {{wget https://github.com/erik-wramner/JmsTools/archive/v1.10.zip}}
    ** {{unzip v1.10.zip}}
    ** {{cd JmsTools-1.10}}
    ** {{mvn package}}
    ** {{cp shaded-jars/ArtemisJmsConsumer.jar ../}}
    ** {{cp shaded-jars/ArtemisJmsProducer.jar ../}}
    ** {{cp shaded-jars/LogAnalyzer.jar ../logs}}
    ** {{cd ../broker}}
    ** {{wget https://www-us.apache.org/dist/activemq/activemq-artemis/2.10.1/apache-artemis-2.10.1-bin.zip}}
    ** {{unzip apache-artemis-2.10.1-bin.zip}}
    ** {{./apache-artemis-2.10.1/bin/artemis create ./instance --user myUser --password myPass --allow-anonymous}}
* Open a terminal and execute the following to run the broker (which will run for 1 minute and restart endlessly)
    ** {{cd ~/ARTEMIS-2173/broker/instance/bin}}
    ** {{printf '%s\n' '#!/bin/sh' '' 'while :; do' '  ./artemis-service start' '  sleep 1m' '  ./artemis-service stop' 'done' > cycle.sh}}
    ** {{chmod +x cycle.sh}}
    ** {{./cycle.sh}}
* Open a terminal and execute the following to run the producer
    ** {{cd ~/ARTEMIS-2173}}
    ** {{java -jar ArtemisJmsProducer.jar -url tcp://127.0.0.1:61616 -log logs/ -t 5 -duration 120 -id -type TEXT -queue MyQueue}}
* Open a terminal and execute the following to run the consumer
    ** {{cd ~/ARTEMIS-2173}}
    ** {{java -jar ArtemisJmsConsumer.jar -url tcp://127.0.0.1:61616 -log logs/ -t 5 -timeout 45000 -verify -drain -queue MyQueue}}
* When the producer and consumer both finish open a terminal and execute the following to analyze the logs:
    ** {{cd ~/ARTEMIS-2173/logs}}
    ** {{java -Xmx16384m -jar LogAnalyzer.jar -o ./report.html .}}

h4. *Results*

The report only indicates that there are in-doubt messages. This is what I would expect given the nature of failures during a transaction commit operation (i.e. you don't know if the failure happened before the broker actually committed or not). It reports 0 lost, duplicate, ghost, alien, & undead messages.

h4. *Analysis*

It's impossible for me to say exactly why these results differ from your own. However, the two most likely causes are that the versions of both JmsTools and ActiveMQ Artemis are different. Also, our testing procedures were slightly different in that you ran multiple iterations of a short test and I ran a single long test (although I ran several short tests to make sure everything was working properly and there were never any lost messages).

h4. *Notes*

* I used {{120}} for the producer's {{duration}} (i.e. two hours) which may seem excessive, but I wanted to give the test plenty of opportunity to fails (i.e. lose messages).
* Due to the long run-time there were 10 log files that were each around 110MB. Running the log analyzer on these files took *over 2 hours*. Also, I kept hitting OutOfMemoryErrors so I eventually gave the analyzer 16GB of heap (I have 32 available).

> Artemis lose messages on broker restarts
> ----------------------------------------
>
>                 Key: ARTEMIS-2173
>                 URL: https://issues.apache.org/jira/browse/ARTEMIS-2173
>             Project: ActiveMQ Artemis
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 2.6.3
>            Reporter: Veaceslav Doina
>            Priority: Critical
>
> Hello,
> We recently performed Artemis reliability test in order to verify if it can guarantee 0 messages loss. Results show, that if the broker is restarted in the process of producing/consuming we lose messages.
> Testing environment: 
> AWS
> Hardware: 
> |Broker|m5.large|2vCPU / 8GiB|
> |Producer/Consumer|t3.large|2vCPU / 8GiB|
> Software:
> OS: CentOS Linux release 7.5.1804 
> Java: Oracle JDK SE 8 Update 192
> Broker: Artemis 2.6.3
> Client: [JmsTools|https://github.com/erik-wramner/JmsTools]
> Default Artemis configuration was used with only modifications in artemis.profile and jolokia-access.xml files.
> The result and project with all required data for issue reproducing may be found on GitHub: [https://github.com/veaceslavdoina/messages-brokers-testing]
> Discussion on the Mailing list: [http://activemq.2283324.n4.nabble.com/ActiveMQ-and-Artemis-reliability-Messages-lost-tt4744881.html]
>  
> Thank you!
>  
> Slava.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)