You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@activemq.apache.org by "mattrpav (via GitHub)" <gi...@apache.org> on 2026/09/19 12:23:16 UTC
[PR] [#2583] Duplicate message publish (during failover) lead to incorrect queue size metric [activemq]
mattrpav opened a new pull request, #2584:
URL: https://github.com/apache/activemq/pull/2584
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For additional commands, e-mail: gitbox-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
Re: [PR] [#2583] Duplicate message publish during failover leads to incorrect queue size metric [activemq]
Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on code in PR #2584:
URL: https://github.com/apache/activemq/pull/2584#discussion_r4077534738
##########
activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/KahaDBStore.java:
##########
@@ -526,6 +526,28 @@ public void removeAsyncMessage(ConnectionContext context, MessageAck ack) throws
@Override
public void addMessage(final ConnectionContext context, final Message message) throws IOException {
+ // A synchronous add (no future assigned yet) can race a pending
+ // asynchronous add of the same message id (concurrentStoreAndDispatch):
+ // the async task still holds the journal/index update while its message
+ // has already been counted and dispatched from the cursor cache. If the
+ // sync add wins the index it is assigned a real sequence, evades the
+ // duplicate suppression in the cursor add path, and inflates the
+ // destination statistics. Reject it here as the duplicate it is; the
+ // -1 marker is handled by the existing suppression in
+ // Queue.doPendingCursorAdditions.
+ if (isConcurrentStoreAndDispatchQueues()
Review Comment:
Is it possible for multiple async adds to also race during failover?
It looks like the code just calls put() on the asyncTaskMap and never checks if an existing entry is in the map.
If we are adding this check here, don't we also need to add the same check [here](https://github.com/mattrpav/activemq/blob/e3d559b7936d68e923f37926724bbdf890d69392/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/KahaDBStore.java#L321) and [here](https://github.com/mattrpav/activemq/blob/e3d559b7936d68e923f37926724bbdf890d69392/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/KahaDBStore.java#L335) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For additional commands, e-mail: gitbox-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
Re: [PR] [#2583] Duplicate message publish during failover leads to incorrect queue size metric [activemq]
Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on code in PR #2584:
URL: https://github.com/apache/activemq/pull/2584#discussion_r4077534738
##########
activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/KahaDBStore.java:
##########
@@ -526,6 +526,28 @@ public void removeAsyncMessage(ConnectionContext context, MessageAck ack) throws
@Override
public void addMessage(final ConnectionContext context, final Message message) throws IOException {
+ // A synchronous add (no future assigned yet) can race a pending
+ // asynchronous add of the same message id (concurrentStoreAndDispatch):
+ // the async task still holds the journal/index update while its message
+ // has already been counted and dispatched from the cursor cache. If the
+ // sync add wins the index it is assigned a real sequence, evades the
+ // duplicate suppression in the cursor add path, and inflates the
+ // destination statistics. Reject it here as the duplicate it is; the
+ // -1 marker is handled by the existing suppression in
+ // Queue.doPendingCursorAdditions.
+ if (isConcurrentStoreAndDispatchQueues()
Review Comment:
Is it possible for multiple async adds to happen and cause a race condition during failover?
It looks like the code just calls put() on the asyncTaskMap and never checks if an existing entry is in the map.
If we are adding this check here, don't we also need to add the same check [here](https://github.com/mattrpav/activemq/blob/e3d559b7936d68e923f37926724bbdf890d69392/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/KahaDBStore.java#L321) and [here](https://github.com/mattrpav/activemq/blob/e3d559b7936d68e923f37926724bbdf890d69392/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/KahaDBStore.java#L335) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For additional commands, e-mail: gitbox-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
Re: [PR] [#2583] Duplicate message publish during failover leads to incorrect queue size metric [activemq]
Posted by "mattrpav (via GitHub)" <gi...@apache.org>.
mattrpav merged PR #2584:
URL: https://github.com/apache/activemq/pull/2584
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For additional commands, e-mail: gitbox-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
Re: [PR] [#2583] Duplicate message publish during failover leads to incorrect queue size metric [activemq]
Posted by "mattrpav (via GitHub)" <gi...@apache.org>.
mattrpav commented on code in PR #2584:
URL: https://github.com/apache/activemq/pull/2584#discussion_r4083175576
##########
activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/KahaDBStore.java:
##########
@@ -526,6 +526,28 @@ public void removeAsyncMessage(ConnectionContext context, MessageAck ack) throws
@Override
public void addMessage(final ConnectionContext context, final Message message) throws IOException {
+ // A synchronous add (no future assigned yet) can race a pending
+ // asynchronous add of the same message id (concurrentStoreAndDispatch):
+ // the async task still holds the journal/index update while its message
+ // has already been counted and dispatched from the cursor cache. If the
+ // sync add wins the index it is assigned a real sequence, evades the
+ // duplicate suppression in the cursor add path, and inflates the
+ // destination statistics. Reject it here as the duplicate it is; the
+ // -1 marker is handled by the existing suppression in
+ // Queue.doPendingCursorAdditions.
+ if (isConcurrentStoreAndDispatchQueues()
Review Comment:
@cshannon L544 checks for previous entry.
Good call on adding checks in the other two spots, I'll look at those call spots
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For additional commands, e-mail: gitbox-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
Re: [PR] [#2583] Duplicate message publish during failover leads to incorrect queue size metric [activemq]
Posted by "mattrpav (via GitHub)" <gi...@apache.org>.
mattrpav commented on code in PR #2584:
URL: https://github.com/apache/activemq/pull/2584#discussion_r4094856288
##########
activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/KahaDBStore.java:
##########
@@ -526,6 +526,28 @@ public void removeAsyncMessage(ConnectionContext context, MessageAck ack) throws
@Override
public void addMessage(final ConnectionContext context, final Message message) throws IOException {
+ // A synchronous add (no future assigned yet) can race a pending
+ // asynchronous add of the same message id (concurrentStoreAndDispatch):
+ // the async task still holds the journal/index update while its message
+ // has already been counted and dispatched from the cursor cache. If the
+ // sync add wins the index it is assigned a real sequence, evades the
+ // duplicate suppression in the cursor add path, and inflates the
+ // destination statistics. Reject it here as the duplicate it is; the
+ // -1 marker is handled by the existing suppression in
+ // Queue.doPendingCursorAdditions.
+ if (isConcurrentStoreAndDispatchQueues()
Review Comment:
@cshannon I don't think we need to add checks in those other two locations, b/c the cursor audit catches dupe add attempts.
The core scenario in this fix is the async v sync race condition where the cache can get full and it converts the failover send to sync.
The cursor audit catches async v async races already. New test pushed up with parameterized useCache (T|F) to validate both straight-line paths as well as the async-to-sync path already tested on this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For additional commands, e-mail: gitbox-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
Re: [PR] [#2583] Duplicate message publish during failover leads to incorrect queue size metric [activemq]
Posted by "mattrpav (via GitHub)" <gi...@apache.org>.
mattrpav commented on code in PR #2584:
URL: https://github.com/apache/activemq/pull/2584#discussion_r4083175576
##########
activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/KahaDBStore.java:
##########
@@ -526,6 +526,28 @@ public void removeAsyncMessage(ConnectionContext context, MessageAck ack) throws
@Override
public void addMessage(final ConnectionContext context, final Message message) throws IOException {
+ // A synchronous add (no future assigned yet) can race a pending
+ // asynchronous add of the same message id (concurrentStoreAndDispatch):
+ // the async task still holds the journal/index update while its message
+ // has already been counted and dispatched from the cursor cache. If the
+ // sync add wins the index it is assigned a real sequence, evades the
+ // duplicate suppression in the cursor add path, and inflates the
+ // destination statistics. Reject it here as the duplicate it is; the
+ // -1 marker is handled by the existing suppression in
+ // Queue.doPendingCursorAdditions.
+ if (isConcurrentStoreAndDispatchQueues()
Review Comment:
@cshannon L544 checks for previous entry.
Good call on adding checks in the other two spots, I'll look at those call spots.
edit: Those call sites are already under lock, so I believe it is as easy as checking for presence and then logging the duplicate add attempt and swallowing vs throwing an exception.
I think the idempotent behavior is a valid approach in this case. Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For additional commands, e-mail: gitbox-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
Re: [PR] [#2583] Duplicate message publish during failover leads to incorrect queue size metric [activemq]
Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on code in PR #2584:
URL: https://github.com/apache/activemq/pull/2584#discussion_r4122370513
##########
activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/KahaDBStore.java:
##########
@@ -526,6 +526,28 @@ public void removeAsyncMessage(ConnectionContext context, MessageAck ack) throws
@Override
public void addMessage(final ConnectionContext context, final Message message) throws IOException {
+ // A synchronous add (no future assigned yet) can race a pending
+ // asynchronous add of the same message id (concurrentStoreAndDispatch):
+ // the async task still holds the journal/index update while its message
+ // has already been counted and dispatched from the cursor cache. If the
+ // sync add wins the index it is assigned a real sequence, evades the
+ // duplicate suppression in the cursor add path, and inflates the
+ // destination statistics. Reject it here as the duplicate it is; the
+ // -1 marker is handled by the existing suppression in
+ // Queue.doPendingCursorAdditions.
+ if (isConcurrentStoreAndDispatchQueues()
Review Comment:
That's good to know but it might be worth hardening/improving the logic maybe in a future PR. The cursor audit cache limited to a certain size so I'm not sure what happens If we exceeded the cache limit (or how likely that even is, ideally the cache is sized for your use case). Either way, we should likely be using putIfAbsent() semantics and not put() because it's an unexpected state if the key already existed. But like I said, maybe something to address later because the producer audit currently guards it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For additional commands, e-mail: gitbox-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
Re: [PR] [#2583] Duplicate message publish during failover leads to incorrect queue size metric [activemq]
Posted by "mattrpav (via GitHub)" <gi...@apache.org>.
mattrpav commented on PR #2584:
URL: https://github.com/apache/activemq/pull/2584#issuecomment-5996867811
Since tx producers send async by default, I added an additional test that confirms transacted send would receive the failure and signal to producer to resend the entire transaction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@activemq.apache.org
For additional commands, e-mail: gitbox-help@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact