ActiveMQ 5.10 looses message acknowledgements.

[sbarlabanov <co...@sergiy-barlabanov.de>]

Hi,

we have the following problem with ActiveMQ:
we have some consumer code running in an app server and acknowledging
messages manually. ActiveMQ broker is running outside the app server as a
separate process. Everything is ok until we restart the broker. After we
restart it resends several (a few) messages acknowledged hours ago (goes
back to 12-14 hours). There are not many: yesterday we had 3 from many
thousands at that day. These 3 messages were consumed at different times of
the day.
KahaDB is running with enableJournalDiskSyncs=false. We also observed that
each time such a message was consumed and acknowledged  KahaDB rotated the
data file. So my suspicios is that there is somewhere a leak in KahaDB logic
when using async writes and rotating the data files. So a KahaDB remove
command is not persisted and after ActiveMQ restart, the broker rereads
commands from the files and resends the corresponding messages.
Is it a feasible explanation? Does it make sense? Maybe there is already
some JIRA from this area?
I did not find any clues during my initial analysis of ActiveMQ source code.
Any help appreciated.
Currenty we are planing to switch on journal syncs with the hope that this
will work around this problem.

Best regards,
Sergiy



--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-5-10-looses-message-acknowledgements-tp4690306.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ 5.10 looses message acknowledgements.

[sbarlabanov <co...@sergiy-barlabanov.de>]

I created a test case and prepared a patch and attached everything to the
JIRA ticket.
But the fix has quite important consequences. See the commend of Tim in
https://issues.apache.org/jira/browse/AMQ-5542





--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-5-10-looses-message-acknowledgements-tp4690306p4690535.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ 5.10 looses message acknowledgements.

[artnaseef <ar...@artnaseef.com>]

That's a great looking find.  Can you put together a small test case that
reproduces the problem and submit a patch to fix it (preferably via a github
pull request)?



--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-5-10-looses-message-acknowledgements-tp4690306p4690488.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ 5.10 looses message acknowledgements.

[sbarlabanov <co...@sergiy-barlabanov.de>]

I think we found the problem: seems to be a bug in MessageDatabase:

https://issues.apache.org/jira/browse/AMQ-5542




--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-5-10-looses-message-acknowledgements-tp4690306p4690487.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ 5.10 looses message acknowledgements.

[sbarlabanov <co...@sergiy-barlabanov.de>]

Tonight we got new experience with the problem:
before we restart ActiveMQ we looked at the messages in a queue and we saw
that the size of the queue was 0.
After having restarted ActiveMQ we looked at the messages in the queue again
and there were 4 messages from today morning. All 4 messages correspond to
the timestamps of KahaDB file rotations: we see those files on the file
system and we see that the last modification times of those files are the
same time as the timestamps of the messages.
After that we removed those messages manually from ActiveMQ via JMX Console
and there were away and were not delivered at that run.
But when we restarted ActiveMQ the second time, those 4 messages were again
in the queue! So Remove via JMX did not removed them from the storage. Only
after our consumer consumed and acknowledged those 4 messages, they
disappeared completely and did not come with the next restart.




--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-5-10-looses-message-acknowledgements-tp4690306p4690308.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.