You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Rick Parker (Jira)" <ji...@apache.org> on 2024/09/04 14:11:00 UTC
[jira] [Comment Edited] (ARTEMIS-4928) SendAcknowledgementHandler not getting called
[ https://issues.apache.org/jira/browse/ARTEMIS-4928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17879248#comment-17879248 ]
Rick Parker edited comment on ARTEMIS-4928 at 9/4/24 2:10 PM:
--------------------------------------------------------------
An update from our end. We were wondering how best to proceed with this issue, and because I was never fond of our decision to run 2 brokers in the same JVM, I ran an experiment to look at consolidating down to a single broker. That proved to be even easier than I thought, and so we have been running tests with a single broker in the JVM. We did not reproduce the exact problem above (at least not yet), but I think we have reproduced something similar. The symptoms were the same in that our application locked up.
Looking into the heap dump (early days), I can again see one {{OperationContextImpl}} with some entries.
!image-2024-09-04-15-10-04-610.png!
was (Author: parkri):
An update from our end. We were wondering how best to proceed with this issue, and because I was never fond of our decision to run 2 brokers in the same JVM, I ran an experiment to look at consolidating down to a single broker. That proved to be even easier than I thought, and so we have been running tests with a single broker in the JVM. We did not reproduce the exact problem above (at least not yet), but I think we have reproduced something similar. The symptoms were the same in that our application locked up.
Looking into the heap dump (early days), I can again see one {{OperationContextImpl}} with some entries.
!image-2024-09-04-15-09-31-595.png!
> SendAcknowledgementHandler not getting called
> ---------------------------------------------
>
> Key: ARTEMIS-4928
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4928
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
> Affects Versions: 2.32.0
> Environment: The environment is Linux based, with Azul Java 17. I can update with more precise details if needed.
> Artemis version is 2.32.0. However, Artemis broker and the application (and thus client producer) are in the same JVM with socket transports.
> We do not see any exceptions in our logs.
>
> Reporter: Rick Parker
> Priority: Critical
> Attachments: image-2024-07-17-13-41-55-900.png, image-2024-07-17-13-49-53-962.png, image-2024-08-12-17-44-03-698.png, image-2024-08-12-17-45-55-812.png, image-2024-08-12-18-36-35-508.png, image-2024-08-12-18-37-10-723.png, image-2024-08-12-18-37-44-279.png, image-2024-09-04-15-09-31-595.png, image-2024-09-04-15-10-04-610.png
>
>
> We have been using ArtemisMQ since 2016, and recently upgrading from 2.19.1 on JDK8 to 2.32.0 on JDK17. We occasionally experience what looks like a failure to acknowledge the sending of a message by a (CORE) producer, since doing that upgrade, and it brings our application to a halt.
> When I say occasionally, we have a nightly performance test of our application that sends about 20-30 million messages from the one producer. This failure to acknowledge the send so far has happened twice in the space of about a month, which means it is happening approximately every 250-400 million messages or perhaps more. This also means we don't currently have a self contained reproduction of the problem. We are starting to think about how we might reproduce it more frequently, if possible, since we have now seen it twice and have gained a tiny bit more understanding.
> The symptom is a failure to be called back from the send, and inspecting a heap dump I _think_ confirms that the producer is sitting on a send - but I am not an expert on the internal workings of Artemis and many apologies in advance if I either mislead or point fingers inappropriately.
> We will try upgrading to the latest 2.35.0 (as at time of writing) to see if it goes away - the fixed issues don't immediately shout out that it might be solved however.
> The API from which we do not get called back is:
> {{org.apache.activemq.artemis.api.core.client.ClientProducer.}}
> {{send(SimpleString address, Message message, SendAcknowledgementHandler handler)}}
> Can a misbehaving handler/callback somehow cause this? e.g. what happens if it throws an exception? (which we are not seeing bubble up anywhere, but haven not ruled it out)
> I have a screenshot of what looks like an interesting part of the heap dump - the {{{}ChannelImpl{}}}. To my eyes the {{firstStoredCommandID}} value looks out of sync with the content ({{{}correlationID{}}} of message) of the {{resendCache}} which is lagging behind for some reason. 8,815,497 is the message that has not had the handler called. But like I say, I'm looking at all this for the first time with little understanding.
> !image-2024-07-17-13-41-55-900.png!
> It also looks like the same message is still present in the broker data structures / heap dump, along with 8,815,495
> !image-2024-07-17-13-49-53-962.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@activemq.apache.org
For additional commands, e-mail: issues-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact