You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Lionel Cons (Jira)" <ji...@apache.org> on 2022/08/16 11:59:00 UTC
[jira] [Commented] (AMQ-9051) Ghost connections in the broker holding messages
[ https://issues.apache.org/jira/browse/AMQ-9051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17580282#comment-17580282 ]
Lionel Cons commented on AMQ-9051:
----------------------------------
FWIW, I restarted another broker showing the same problem (for the same queue). The broker logged:
{code}
2022-08-16T13:40:41.128+0200 [WrapperSimpleAppMain] INFO BrokerService - Apache ActiveMQ 5.16.5 (role_broker.acme.org, role_broker.acme.org) is starting
2022-08-16T13:40:41.683+0200 [JobScheduler:JMS] WARN MessageDatabase - Duplicate message add attempt rejected. Destination: QUEUE://queue.name, Message id: ID:broker.acme.org-3787-1659612872255-3:165023:-1:1:1
2022-08-16T13:40:41.803+0200 [JobScheduler:JMS] WARN Queue - queue://queue.name, subscriptions=0, memory=0%, size=113, pending=0 messageStore indicated duplicate add attempt for ID:broker.acme.org-3787-1659612872255-3:165023:-1:1:1, suppressing duplicate dispatch
2022-08-16T13:40:43.358+0200 [JobScheduler:JMS] WARN MessageDatabase - Duplicate message add attempt rejected. Destination: QUEUE://queue.name, Message id: ID:broker.acme.org-3787-1659612872255-3:165021:-1:1:1
2022-08-16T13:40:43.358+0200 [JobScheduler:JMS] WARN Queue - queue://queue.name, subscriptions=0, memory=0%, size=223, pending=0 messageStore indicated duplicate add attempt for ID:broker.acme.org-3787-1659612872255-3:165021:-1:1:1, suppressing duplicate dispatch
{code}
> Ghost connections in the broker holding messages
> ------------------------------------------------
>
> Key: AMQ-9051
> URL: https://issues.apache.org/jira/browse/AMQ-9051
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.16.4, 5.16.5
> Environment: CentOS 7 and Java 8.
> Reporter: Lionel Cons
> Priority: Major
>
> ActiveMQ can have "ghost connections". These are connections effectively closed (as confirmed by the operating system) but still appearing in ActiveMQ.
> Here is an example.
> Subscription object as reported by JMX:
> {code:java}
> "org.apache.activemq:brokerName=role_broker.acme.org,clientId=ID_broker.acme.org-21914-1645625556236-3_3056814,consumerId=ID_broker.acme.org-21914-1645625556236-3_3056814_-1_1,destinationName=queue.name,destinationType=Queue,endpoint=Consumer,type=Broker" : {
> "Active" : true,
> "ClientId" : "ID:broker.acme.org-21914-1645625556236-3:3056814",
> "Connection" : {
> "objectName" : "org.apache.activemq:brokerName=role_broker.acme.org,connectionName=ID_broker.acme.org-21914-1645625556236-3_3056814,connectionViewType=clientId,connector=clientConnectors,connectorName=stomp,type=Broker"
> },
> "ConnectionId" : "ID:broker.acme.org-21914-1645625556236-3:3056814",
> "ConsumedCount" : 0,
> "DequeueCounter" : 496,
> "DestinationName" : "queue.name",
> "DestinationQueue" : true,
> "DestinationTemporary" : false,
> "DestinationTopic" : false,
> "DispatchAsync" : true,
> "DispatchedCounter" : 548,
> "DispatchedQueueSize" : 52,
> "Durable" : false,
> "EnqueueCounter" : 549,
> "Exclusive" : false,
> "MaximumPendingMessageLimit" : 0,
> "MessageCountAwaitingAcknowledge" : 52,
> "Network" : false,
> "NoLocal" : false,
> "PendingQueueSize" : 1,
> "PrefetchSize" : 563,
> "Priority" : 0,
> "Retroactive" : false,
> "Selector" : null,
> "SessionId" : -1,
> "SlowConsumer" : false,
> "SubscriptionId" : 1,
> "SubscriptionName" : null,
> "UserName" : null
> },
> {code}
> Connection object as reported by JMX:
> {code:java}
> "org.apache.activemq:brokerName=role_broker.acme.org,connectionName=ID_broker.acme.org-21914-1645625556236-3_3056814,connectionViewType=clientId,connector=clientConnectors,connectorName=stomp,type=Broker" : {
> "Active" : true,
> "ActiveTransactionCount" : 0,
> "Blocked" : false,
> "ClientId" : "ID:broker.acme.org-21914-1645625556236-3:3056814",
> "Connected" : true,
> "Consumers" : [
> {
> "objectName" : "org.apache.activemq:brokerName=role_broker.acme.org,clientId=ID_broker.acme.org-21914-1645625556236-3_3056814,consumerId=ID_broker.acme.org-21914-1645625556236-3_3056814_-1_1,destinationName=queue.name,destinationType=Queue,endpoint=Consumer,type=Broker"
> }
> ],
> "DispatchQueueSize" : 0,
> "OldestActiveTransactionDuration" : null,
> "Producers" : [],
> "RemoteAddress" : "tcp://192.168.27.159:44730",
> "Slow" : false,
> "UserName" : null
> },
> {code}
> Yet system tools like {{lsof}} and {{netstat}} show no matching peer ({{{}192.168.27.159:44730{}}}). The peer itself shows no connections to the broker.
> These ghost connections sometimes have pending messages that are therefore stuck on the broker.
> The connections are using STOMP, with heart-beats enabled.
> These ghost connections are not transient. They are not cleaned by the broker (although they should wrt heart-beats).
> This happens on both 5.16.4 and 5.16.5. Other ActiveMQ versions have not been tested.
> We haven't found a way to reproduce the problem. It just happens for one messaging application, after some time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)