You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Lionel Cons (Jira)" <ji...@apache.org> on 2022/08/16 11:50:00 UTC
[jira] [Updated] (AMQ-9051) Ghost connections in the broker holding messages
[ https://issues.apache.org/jira/browse/AMQ-9051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lionel Cons updated AMQ-9051:
-----------------------------
Description:
ActiveMQ can have "ghost connections". These are connections effectively closed (as confirmed by the operating system) but still appearing in ActiveMQ.
Here is an example.
Subscription object as reported by JMX:
{code:java}
"org.apache.activemq:brokerName=role_broker.acme.org,clientId=ID_broker.acme.org-21914-1645625556236-3_3056814,consumerId=ID_broker.acme.org-21914-1645625556236-3_3056814_-1_1,destinationName=queue.name,destinationType=Queue,endpoint=Consumer,type=Broker" : {
"Active" : true,
"ClientId" : "ID:broker.acme.org-21914-1645625556236-3:3056814",
"Connection" : {
"objectName" : "org.apache.activemq:brokerName=role_broker.acme.org,connectionName=ID_broker.acme.org-21914-1645625556236-3_3056814,connectionViewType=clientId,connector=clientConnectors,connectorName=stomp,type=Broker"
},
"ConnectionId" : "ID:broker.acme.org-21914-1645625556236-3:3056814",
"ConsumedCount" : 0,
"DequeueCounter" : 496,
"DestinationName" : "queue.name",
"DestinationQueue" : true,
"DestinationTemporary" : false,
"DestinationTopic" : false,
"DispatchAsync" : true,
"DispatchedCounter" : 548,
"DispatchedQueueSize" : 52,
"Durable" : false,
"EnqueueCounter" : 549,
"Exclusive" : false,
"MaximumPendingMessageLimit" : 0,
"MessageCountAwaitingAcknowledge" : 52,
"Network" : false,
"NoLocal" : false,
"PendingQueueSize" : 1,
"PrefetchSize" : 563,
"Priority" : 0,
"Retroactive" : false,
"Selector" : null,
"SessionId" : -1,
"SlowConsumer" : false,
"SubscriptionId" : 1,
"SubscriptionName" : null,
"UserName" : null
},
{code}
Connection object as reported by JMX:
{code:java}
"org.apache.activemq:brokerName=role_broker.acme.org,connectionName=ID_broker.acme.org-21914-1645625556236-3_3056814,connectionViewType=clientId,connector=clientConnectors,connectorName=stomp,type=Broker" : {
"Active" : true,
"ActiveTransactionCount" : 0,
"Blocked" : false,
"ClientId" : "ID:broker.acme.org-21914-1645625556236-3:3056814",
"Connected" : true,
"Consumers" : [
{
"objectName" : "org.apache.activemq:brokerName=role_broker.acme.org,clientId=ID_broker.acme.org-21914-1645625556236-3_3056814,consumerId=ID_broker.acme.org-21914-1645625556236-3_3056814_-1_1,destinationName=queue.name,destinationType=Queue,endpoint=Consumer,type=Broker"
}
],
"DispatchQueueSize" : 0,
"OldestActiveTransactionDuration" : null,
"Producers" : [],
"RemoteAddress" : "tcp://192.168.27.159:44730",
"Slow" : false,
"UserName" : null
},
{code}
Yet system tools like {{lsof}} and {{netstat}} show no matching peer ({{{}192.168.27.159:44730{}}}). The peer itself shows no connections to the broker.
These ghost connections sometimes have pending messages that are therefore stuck on the broker.
The connections are using STOMP, with heart-beats enabled.
These ghost connections are not transient. They are not cleaned by the broker (although they should wrt heart-beats).
This happens on both 5.16.4 and 5.16.5. Other ActiveMQ versions have not been tested.
We haven't found a way to reproduce the problem. It just happens for one messaging application, after some time.
was:
ActiveMQ can have "ghost connections". These are connections effectively closed (as confirmed by the operating system) but still appearing in ActiveMQ.
Here is an example.
Subscription object as reported by JMX:
{code:java}
"org.apache.activemq:brokerName=agileinf_broker.acme.org,clientId=ID_broker.acme.org-21914-1645625556236-3_3056814,consumerId=ID_broker.acme.org-21914-1645625556236-3_3056814_-1_1,destinationName=queue.name,destinationType=Queue,endpoint=Consumer,type=Broker" : {
"Active" : true,
"ClientId" : "ID:broker.acme.org-21914-1645625556236-3:3056814",
"Connection" : {
"objectName" : "org.apache.activemq:brokerName=agileinf_broker.acme.org,connectionName=ID_broker.acme.org-21914-1645625556236-3_3056814,connectionViewType=clientId,connector=clientConnectors,connectorName=stomp,type=Broker"
},
"ConnectionId" : "ID:broker.acme.org-21914-1645625556236-3:3056814",
"ConsumedCount" : 0,
"DequeueCounter" : 496,
"DestinationName" : "queue.name",
"DestinationQueue" : true,
"DestinationTemporary" : false,
"DestinationTopic" : false,
"DispatchAsync" : true,
"DispatchedCounter" : 548,
"DispatchedQueueSize" : 52,
"Durable" : false,
"EnqueueCounter" : 549,
"Exclusive" : false,
"MaximumPendingMessageLimit" : 0,
"MessageCountAwaitingAcknowledge" : 52,
"Network" : false,
"NoLocal" : false,
"PendingQueueSize" : 1,
"PrefetchSize" : 563,
"Priority" : 0,
"Retroactive" : false,
"Selector" : null,
"SessionId" : -1,
"SlowConsumer" : false,
"SubscriptionId" : 1,
"SubscriptionName" : null,
"UserName" : null
},
{code}
Connection object as reported by JMX:
{code:java}
"org.apache.activemq:brokerName=agileinf_broker.acme.org,connectionName=ID_broker.acme.org-21914-1645625556236-3_3056814,connectionViewType=clientId,connector=clientConnectors,connectorName=stomp,type=Broker" : {
"Active" : true,
"ActiveTransactionCount" : 0,
"Blocked" : false,
"ClientId" : "ID:broker.acme.org-21914-1645625556236-3:3056814",
"Connected" : true,
"Consumers" : [
{
"objectName" : "org.apache.activemq:brokerName=agileinf_broker.acme.org,clientId=ID_broker.acme.org-21914-1645625556236-3_3056814,consumerId=ID_broker.acme.org-21914-1645625556236-3_3056814_-1_1,destinationName=queue.name,destinationType=Queue,endpoint=Consumer,type=Broker"
}
],
"DispatchQueueSize" : 0,
"OldestActiveTransactionDuration" : null,
"Producers" : [],
"RemoteAddress" : "tcp://192.168.27.159:44730",
"Slow" : false,
"UserName" : null
},
{code}
Yet system tools like {{lsof}} and {{netstat}} show no matching peer ({{{}192.168.27.159:44730{}}}). The peer itself shows no connections to the broker.
These ghost connections sometimes have pending messages that are therefore stuck on the broker.
The connections are using STOMP, with heart-beats enabled.
These ghost connections are not transient. They are not cleaned by the broker (although they should wrt heart-beats).
This happens on both 5.16.4 and 5.16.5. Other ActiveMQ versions have not been tested.
We haven't found a way to reproduce the problem. It just happens for one messaging application, after some time.
> Ghost connections in the broker holding messages
> ------------------------------------------------
>
> Key: AMQ-9051
> URL: https://issues.apache.org/jira/browse/AMQ-9051
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.16.4, 5.16.5
> Environment: CentOS 7 and Java 8.
> Reporter: Lionel Cons
> Priority: Major
>
> ActiveMQ can have "ghost connections". These are connections effectively closed (as confirmed by the operating system) but still appearing in ActiveMQ.
> Here is an example.
> Subscription object as reported by JMX:
> {code:java}
> "org.apache.activemq:brokerName=role_broker.acme.org,clientId=ID_broker.acme.org-21914-1645625556236-3_3056814,consumerId=ID_broker.acme.org-21914-1645625556236-3_3056814_-1_1,destinationName=queue.name,destinationType=Queue,endpoint=Consumer,type=Broker" : {
> "Active" : true,
> "ClientId" : "ID:broker.acme.org-21914-1645625556236-3:3056814",
> "Connection" : {
> "objectName" : "org.apache.activemq:brokerName=role_broker.acme.org,connectionName=ID_broker.acme.org-21914-1645625556236-3_3056814,connectionViewType=clientId,connector=clientConnectors,connectorName=stomp,type=Broker"
> },
> "ConnectionId" : "ID:broker.acme.org-21914-1645625556236-3:3056814",
> "ConsumedCount" : 0,
> "DequeueCounter" : 496,
> "DestinationName" : "queue.name",
> "DestinationQueue" : true,
> "DestinationTemporary" : false,
> "DestinationTopic" : false,
> "DispatchAsync" : true,
> "DispatchedCounter" : 548,
> "DispatchedQueueSize" : 52,
> "Durable" : false,
> "EnqueueCounter" : 549,
> "Exclusive" : false,
> "MaximumPendingMessageLimit" : 0,
> "MessageCountAwaitingAcknowledge" : 52,
> "Network" : false,
> "NoLocal" : false,
> "PendingQueueSize" : 1,
> "PrefetchSize" : 563,
> "Priority" : 0,
> "Retroactive" : false,
> "Selector" : null,
> "SessionId" : -1,
> "SlowConsumer" : false,
> "SubscriptionId" : 1,
> "SubscriptionName" : null,
> "UserName" : null
> },
> {code}
> Connection object as reported by JMX:
> {code:java}
> "org.apache.activemq:brokerName=role_broker.acme.org,connectionName=ID_broker.acme.org-21914-1645625556236-3_3056814,connectionViewType=clientId,connector=clientConnectors,connectorName=stomp,type=Broker" : {
> "Active" : true,
> "ActiveTransactionCount" : 0,
> "Blocked" : false,
> "ClientId" : "ID:broker.acme.org-21914-1645625556236-3:3056814",
> "Connected" : true,
> "Consumers" : [
> {
> "objectName" : "org.apache.activemq:brokerName=role_broker.acme.org,clientId=ID_broker.acme.org-21914-1645625556236-3_3056814,consumerId=ID_broker.acme.org-21914-1645625556236-3_3056814_-1_1,destinationName=queue.name,destinationType=Queue,endpoint=Consumer,type=Broker"
> }
> ],
> "DispatchQueueSize" : 0,
> "OldestActiveTransactionDuration" : null,
> "Producers" : [],
> "RemoteAddress" : "tcp://192.168.27.159:44730",
> "Slow" : false,
> "UserName" : null
> },
> {code}
> Yet system tools like {{lsof}} and {{netstat}} show no matching peer ({{{}192.168.27.159:44730{}}}). The peer itself shows no connections to the broker.
> These ghost connections sometimes have pending messages that are therefore stuck on the broker.
> The connections are using STOMP, with heart-beats enabled.
> These ghost connections are not transient. They are not cleaned by the broker (although they should wrt heart-beats).
> This happens on both 5.16.4 and 5.16.5. Other ActiveMQ versions have not been tested.
> We haven't found a way to reproduce the problem. It just happens for one messaging application, after some time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)