You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Carl Allain (JIRA)" <ji...@apache.org> on 2011/08/12 21:30:27 UTC
[jira] [Created] (AMQ-3448) Zombie ActiveMQ Connection Dispatcher
threads - seems to consume all process File Descriptors (FD leak)
Zombie ActiveMQ Connection Dispatcher threads - seems to consume all process File Descriptors (FD leak)
-------------------------------------------------------------------------------------------------------
Key: AMQ-3448
URL: https://issues.apache.org/jira/browse/AMQ-3448
Project: ActiveMQ
Issue Type: Bug
Affects Versions: 5.3.2
Environment: Active MQ 5.3.2
java version "1.6.0_23"
Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
Distributor ID: CentOS
Description: CentOS release 5.5 (Final)
Release: 5.5
Codename: Final
Reporter: Carl Allain
Priority: Critical
Somehow linked to https://issues.apache.org/jira/browse/AMQ-3286 which was closed. I am opening here as I cannot reopen the old bug and I hope that with the information I provide here, someone will be able to have some insight of the possible cause and a fix.
I don't know how to reproduce with a test case, but I have found 800+ of such "ActiveMQ Connection Dispatcher" threads. I also noted that the number of FDs for the process keeps increasing and after a few days, we have a "too many files opened" when going beyond the 1024 limit. Most of those FDs (hundreds) do look like:
java 6700 lexo-ext 904u sock 0,5 1741176200 can't identify protocol
There is not much activity on the system and that is still enough to get the problem.
When we reach the limit of FDs, we get tons of stack traces in the logs, filling up the hard disk...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3448) Zombie ActiveMQ Connection Dispatcher
threads - seems to consume all process File Descriptors (FD leak)
Posted by "Timothy Bish (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084352#comment-13084352 ]
Timothy Bish commented on AMQ-3448:
-----------------------------------
Have you tested with a newer version, 5.3.2 is a bit old, I'd recommend you try 5.5 or the latest 5.6-SNAPSHOT build.
> Zombie ActiveMQ Connection Dispatcher threads - seems to consume all process File Descriptors (FD leak)
> -------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3448
> URL: https://issues.apache.org/jira/browse/AMQ-3448
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Environment: Active MQ 5.3.2
> java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
> LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
> Distributor ID: CentOS
> Description: CentOS release 5.5 (Final)
> Release: 5.5
> Codename: Final
> Reporter: Carl Allain
> Priority: Critical
>
> Somehow linked to https://issues.apache.org/jira/browse/AMQ-3286 which was closed. I am opening here as I cannot reopen the old bug and I hope that with the information I provide here, someone will be able to have some insight of the possible cause and a fix.
> I don't know how to reproduce with a test case, but I have found 800+ of such "ActiveMQ Connection Dispatcher" threads. I also noted that the number of FDs for the process keeps increasing and after a few days, we have a "too many files opened" when going beyond the 1024 limit. Most of those FDs (hundreds) do look like:
> java 6700 lexo-ext 904u sock 0,5 1741176200 can't identify protocol
> There is not much activity on the system and that is still enough to get the problem.
> When we reach the limit of FDs, we get tons of stack traces in the logs, filling up the hard disk...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3448) Zombie ActiveMQ Connection Dispatcher
threads - seems to consume all process File Descriptors (FD leak)
Posted by "Carl Allain (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126632#comment-13126632 ]
Carl Allain commented on AMQ-3448:
----------------------------------
One short-term solution for us could be to keep the 5.3.2 clients and use the 5.5 server. Would that version difference between the client and the server be ok or are they incompatible???
> Zombie ActiveMQ Connection Dispatcher threads - seems to consume all process File Descriptors (FD leak)
> -------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3448
> URL: https://issues.apache.org/jira/browse/AMQ-3448
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Environment: Active MQ 5.3.2
> java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
> LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
> Distributor ID: CentOS
> Description: CentOS release 5.5 (Final)
> Release: 5.5
> Codename: Final
> Reporter: Carl Allain
> Priority: Critical
>
> Somehow linked to https://issues.apache.org/jira/browse/AMQ-3286 which was closed. I am opening here as I cannot reopen the old bug and I hope that with the information I provide here, someone will be able to have some insight of the possible cause and a fix.
> I don't know how to reproduce with a test case, but I have found 800+ of such "ActiveMQ Connection Dispatcher" threads. I also noted that the number of FDs for the process keeps increasing and after a few days, we have a "too many files opened" when going beyond the 1024 limit. Most of those FDs (hundreds) do look like:
> java 6700 lexo-ext 904u sock 0,5 1741176200 can't identify protocol
> There is not much activity on the system and that is still enough to get the problem.
> When we reach the limit of FDs, we get tons of stack traces in the logs, filling up the hard disk...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3448) Zombie ActiveMQ Connection Dispatcher
threads - seems to consume all process File Descriptors (FD leak)
Posted by "Timothy Bish (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119632#comment-13119632 ]
Timothy Bish commented on AMQ-3448:
-----------------------------------
Without a test case or some way of reproducing this its going to hard for anyone to help. I suggest trying a newer broker version like 5.5 or trying the latest 5.6-SNAPSHOT to see if this issue has already been fixed.
> Zombie ActiveMQ Connection Dispatcher threads - seems to consume all process File Descriptors (FD leak)
> -------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3448
> URL: https://issues.apache.org/jira/browse/AMQ-3448
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Environment: Active MQ 5.3.2
> java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
> LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
> Distributor ID: CentOS
> Description: CentOS release 5.5 (Final)
> Release: 5.5
> Codename: Final
> Reporter: Carl Allain
> Priority: Critical
>
> Somehow linked to https://issues.apache.org/jira/browse/AMQ-3286 which was closed. I am opening here as I cannot reopen the old bug and I hope that with the information I provide here, someone will be able to have some insight of the possible cause and a fix.
> I don't know how to reproduce with a test case, but I have found 800+ of such "ActiveMQ Connection Dispatcher" threads. I also noted that the number of FDs for the process keeps increasing and after a few days, we have a "too many files opened" when going beyond the 1024 limit. Most of those FDs (hundreds) do look like:
> java 6700 lexo-ext 904u sock 0,5 1741176200 can't identify protocol
> There is not much activity on the system and that is still enough to get the problem.
> When we reach the limit of FDs, we get tons of stack traces in the logs, filling up the hard disk...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-3448) Zombie ActiveMQ Connection Dispatcher
threads - seems to consume all process File Descriptors (FD leak)
Posted by "Carl Allain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Allain updated AMQ-3448:
-----------------------------
Comment: was deleted
(was: De retour lundi le 29 aout. En cas d'urgence, veuillez communiquer avec Patrick Li. Merci.
)
> Zombie ActiveMQ Connection Dispatcher threads - seems to consume all process File Descriptors (FD leak)
> -------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3448
> URL: https://issues.apache.org/jira/browse/AMQ-3448
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Environment: Active MQ 5.3.2
> java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
> LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
> Distributor ID: CentOS
> Description: CentOS release 5.5 (Final)
> Release: 5.5
> Codename: Final
> Reporter: Carl Allain
> Priority: Critical
>
> Somehow linked to https://issues.apache.org/jira/browse/AMQ-3286 which was closed. I am opening here as I cannot reopen the old bug and I hope that with the information I provide here, someone will be able to have some insight of the possible cause and a fix.
> I don't know how to reproduce with a test case, but I have found 800+ of such "ActiveMQ Connection Dispatcher" threads. I also noted that the number of FDs for the process keeps increasing and after a few days, we have a "too many files opened" when going beyond the 1024 limit. Most of those FDs (hundreds) do look like:
> java 6700 lexo-ext 904u sock 0,5 1741176200 can't identify protocol
> There is not much activity on the system and that is still enough to get the problem.
> When we reach the limit of FDs, we get tons of stack traces in the logs, filling up the hard disk...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3448) Zombie ActiveMQ Connection Dispatcher
threads - seems to consume all process File Descriptors (FD leak)
Posted by "Carl Allain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084324#comment-13084324 ]
Carl Allain commented on AMQ-3448:
----------------------------------
It seems to me like there is a race condition making some ActiveMQ Connection Dispatcher threads hang (waiting for something that never occurs), which keep an FD open forever. Connections are made normally at the other end (the AMQ client does not show ERRORs but a few (one in two days)
2011-08-10 10:41:00,334 WARN [org.apache.activemq.transport.failover.FailoverTransport] (InactivityMonitor Async Task: java.util.co
ncurrent.ThreadPoolExecutor$Worker@1fdd357) Transport failed to tcp://host:port , attempting to automatically reconnect
due to: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too long: host/ip:port
> Zombie ActiveMQ Connection Dispatcher threads - seems to consume all process File Descriptors (FD leak)
> -------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3448
> URL: https://issues.apache.org/jira/browse/AMQ-3448
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Environment: Active MQ 5.3.2
> java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
> LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
> Distributor ID: CentOS
> Description: CentOS release 5.5 (Final)
> Release: 5.5
> Codename: Final
> Reporter: Carl Allain
> Priority: Critical
>
> Somehow linked to https://issues.apache.org/jira/browse/AMQ-3286 which was closed. I am opening here as I cannot reopen the old bug and I hope that with the information I provide here, someone will be able to have some insight of the possible cause and a fix.
> I don't know how to reproduce with a test case, but I have found 800+ of such "ActiveMQ Connection Dispatcher" threads. I also noted that the number of FDs for the process keeps increasing and after a few days, we have a "too many files opened" when going beyond the 1024 limit. Most of those FDs (hundreds) do look like:
> java 6700 lexo-ext 904u sock 0,5 1741176200 can't identify protocol
> There is not much activity on the system and that is still enough to get the problem.
> When we reach the limit of FDs, we get tons of stack traces in the logs, filling up the hard disk...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3448) Zombie ActiveMQ Connection Dispatcher
threads - seems to consume all process File Descriptors (FD leak)
Posted by "Carl Allain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13092882#comment-13092882 ]
Carl Allain commented on AMQ-3448:
----------------------------------
Hi, no we have not tested with latest version as we cannot upgrade quickly to latest versions (production setup).
> Zombie ActiveMQ Connection Dispatcher threads - seems to consume all process File Descriptors (FD leak)
> -------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3448
> URL: https://issues.apache.org/jira/browse/AMQ-3448
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Environment: Active MQ 5.3.2
> java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
> LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
> Distributor ID: CentOS
> Description: CentOS release 5.5 (Final)
> Release: 5.5
> Codename: Final
> Reporter: Carl Allain
> Priority: Critical
>
> Somehow linked to https://issues.apache.org/jira/browse/AMQ-3286 which was closed. I am opening here as I cannot reopen the old bug and I hope that with the information I provide here, someone will be able to have some insight of the possible cause and a fix.
> I don't know how to reproduce with a test case, but I have found 800+ of such "ActiveMQ Connection Dispatcher" threads. I also noted that the number of FDs for the process keeps increasing and after a few days, we have a "too many files opened" when going beyond the 1024 limit. Most of those FDs (hundreds) do look like:
> java 6700 lexo-ext 904u sock 0,5 1741176200 can't identify protocol
> There is not much activity on the system and that is still enough to get the problem.
> When we reach the limit of FDs, we get tons of stack traces in the logs, filling up the hard disk...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3448) Zombie ActiveMQ Connection Dispatcher
threads - seems to consume all process File Descriptors (FD leak)
Posted by "Carl Allain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084317#comment-13084317 ]
Carl Allain commented on AMQ-3448:
----------------------------------
Here is a dump of all activemq live objects in the JVM:
jmap -histo:live 6700 | grep activemq
1: 1640 105894928 [Lorg.apache.activemq.command.DataStructure;
26: 818 124336 org.apache.activemq.broker.jmx.ManagedTransportConnection
28: 818 111248 org.apache.activemq.transport.tcp.TcpTransport
33: 839 93968 org.apache.activemq.thread.DedicatedTaskRunner$1
35: 1816 87168 org.apache.activemq.management.CountStatisticImpl
36: 818 85072 org.apache.activemq.transport.InactivityMonitor
39: 1747 69880 org.apache.activemq.command.ProducerId
43: 1941 62112 org.apache.activemq.command.SessionId
46: 1722 55104 org.apache.activemq.command.LocalTransactionId
49: 820 45920 org.apache.activemq.openwire.OpenWireFormat
50: 818 45808 org.apache.activemq.broker.region.ConnectionStatistics
51: 2776 44416 org.apache.activemq.command.ConnectionId
53: 1046 41840 org.apache.activemq.command.ConsumerId
57: 818 39264 org.apache.activemq.transport.WireFormatNegotiator
71: 831 26592 org.apache.activemq.command.WireFormatInfo
75: 818 26176 org.apache.activemq.transport.tcp.TcpTransport$2
76: 818 26176 org.apache.activemq.transport.tcp.TcpBufferedOutputStream
83: 839 20136 org.apache.activemq.thread.DedicatedTaskRunner
85: 822 19728 org.apache.activemq.util.ByteSequence
86: 821 19704 org.apache.activemq.util.DataByteArrayInputStream
88: 818 19632 org.apache.activemq.transport.InactivityMonitor$1
89: 818 19632 org.apache.activemq.transport.InactivityMonitor$2
90: 818 19632 org.apache.activemq.transport.MutexTransport
104: 821 13136 org.apache.activemq.util.DataByteArrayOutputStream
108: 818 13088 org.apache.activemq.broker.TransportConnection$1
109: 818 13088 org.apache.activemq.broker.jmx.ConnectionView
110: 818 13088 org.apache.activemq.broker.SingleTransportConnectionStateRegister
132: 160 5120 org.apache.activemq.util.BitArrayBin
133: 317 5072 org.apache.activemq.broker.jmx.AnnotatedMBean
141: 1 4112 [Lorg.apache.activemq.command.ConsumerId;
144: 160 3840 org.apache.activemq.util.BitArray
158: 86 2752 org.apache.activemq.filter.DestinationMapNode
168: 60 2400 org.apache.activemq.command.ActiveMQTopic
174: 24 2112 org.apache.activemq.command.ConsumerInfo
179: 22 1936 org.apache.activemq.broker.region.DestinationStatistics
182: 14 1904 org.apache.activemq.broker.region.Topic
186: 22 1760 org.apache.activemq.management.TimeStatisticImpl
187: 31 1736 org.apache.activemq.util.LRUCache
190: 40 1600 org.apache.activemq.thread.SchedulerTimerTask
191: 20 1600 org.apache.activemq.broker.region.cursors.FilePendingMessageCursor
192: 22 1584 org.apache.activemq.usage.MemoryUsage
198: 26 1456 org.apache.activemq.command.ConnectionInfo
201: 22 1408 org.apache.activemq.usage.TempUsage
202: 22 1408 org.apache.activemq.usage.StoreUsage
203: 7 1400 org.apache.activemq.broker.region.Queue
215: 11 1232 org.apache.activemq.broker.region.QueueSubscription
223: 13 1144 org.apache.activemq.broker.region.TopicSubscription
227: 23 1104 org.apache.activemq.management.PollCountStatisticImpl
229: 22 1056 org.apache.activemq.usage.SystemUsage
231: 66 1056 org.apache.activemq.usage.DefaultUsageCapacity
235: 1 1040 [Lorg.apache.activemq.openwire.DataStreamMarshaller;
240: 13 936 org.apache.activemq.broker.ConnectionContext
249: 22 880 org.apache.activemq.command.ActiveMQQueue
252: 26 832 org.apache.activemq.command.SessionInfo
259: 7 784 org.apache.activemq.broker.region.Queue$FlowControlTimeoutTask
265: 31 744 org.apache.activemq.ActiveMQMessageAudit
282: 26 624 org.apache.activemq.state.SessionState
287: 13 624 org.apache.activemq.broker.TransportConnectionState
292: 37 592 org.apache.activemq.filter.SimpleDestinationFilter
301: 11 528 org.apache.activemq.broker.region.cursors.VMPendingMessageCursor
303: 7 504 org.apache.activemq.broker.region.cursors.StoreQueueCursor
304: 7 504 org.apache.activemq.broker.region.cursors.QueueStorePrefetch
329: 24 384 org.apache.activemq.state.ConsumerState
336: 7 336 org.apache.activemq.command.DestinationInfo
338: 7 336 org.apache.activemq.store.kahadb.MessageDatabase$StoredDestination
341: 14 336 [Lorg.apache.activemq.command.ActiveMQDestination;
342: 14 336 org.apache.activemq.thread.Valve
348: 13 312 [Lorg.apache.activemq.filter.DestinationFilter;
350: 13 312 org.apache.activemq.filter.MessageEvaluationContext
374: 1 248 org.apache.activemq.xbean.XBeanBrokerService
390: 7 224 org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore
391: 14 224 org.apache.activemq.broker.jmx.TopicView
394: 14 224 org.apache.activemq.broker.region.Topic$1
400: 13 208 org.apache.activemq.filter.CompositeDestinationFilter
401: 13 208 org.apache.activemq.filter.NoLocalExpression
402: 13 208 org.apache.activemq.broker.region.policy.OldestMessageEvictionStrategy
406: 13 208 org.apache.activemq.broker.jmx.TopicSubscriptionView
409: 5 200 org.apache.activemq.command.MessageId
415: 6 192 org.apache.activemq.broker.ConsumerBrokerExchange
420: 8 192 org.apache.activemq.store.kahadb.data.KahaEntryType
432: 1 176 org.apache.activemq.command.ActiveMQMessage
435: 11 176 org.apache.activemq.broker.jmx.SubscriptionView
438: 7 168 org.apache.activemq.store.kahadb.data.KahaDestination
451: 2 160 org.apache.activemq.kaha.impl.index.IndexItem
457: 1 152 org.apache.activemq.broker.region.policy.PolicyEntry
459: 1 144 org.apache.activemq.broker.jmx.ManagedRegionBroker
469: 1 136 org.apache.activemq.transport.tcp.TcpTransportServer
494: 1 120 org.apache.activemq.store.kahadb.KahaDBStore
497: 1 120 org.apache.activemq.ActiveMQConnectionFactory
508: 7 112 org.apache.activemq.broker.jmx.QueueView
509: 1 112 org.apache.activemq.console.command.StartCommand$1
514: 1 112 org.apache.activemq.store.kahadb.MessageDatabase$3
518: 7 112 org.apache.activemq.broker.region.Queue$1
519: 7 112 org.apache.activemq.broker.region.Queue$2
521: 7 112 org.apache.activemq.broker.region.Queue$3
523: 7 112 org.apache.activemq.broker.region.QueueDispatchSelector
524: 14 112 org.apache.activemq.broker.region.policy.SimpleDispatchPolicy
527: 1 112 org.apache.activemq.broker.BrokerService$4
528: 14 112 org.apache.activemq.broker.region.policy.NoSubscriptionRecoveryPolicy
535: 2 96 org.apache.activemq.command.ProducerInfo
557: 4 96 org.apache.activemq.store.kahadb.data.KahaDestination$DestinationType
573: 1 88 org.apache.activemq.web.AjaxServlet
582: 1 80 org.apache.activemq.web.MessageServlet
584: 5 80 org.apache.activemq.filter.DestinationMap
587: 1 80 org.apache.activemq.broker.jmx.ManagedTransportConnector
594: 1 80 org.apache.activemq.broker.jmx.ManagedTempQueueRegion
599: 1 80 org.apache.activemq.broker.jmx.ManagedTempTopicRegion
613: 1 72 org.apache.activemq.kaha.impl.index.IndexManager
616: 1 72 org.apache.activemq.kaha.impl.KahaStore
617: 1 72 org.apache.activemq.broker.jmx.ManagedTopicRegion
636: 2 64 org.apache.activemq.thread.TaskRunnerFactory
638: 1 64 org.apache.activemq.web.handler.BindingBeanNameUrlHandlerMapping
642: 1 64 org.apache.activemq.broker.jmx.ManagementContext
653: 2 64 org.apache.activemq.kaha.impl.IndexRootContainer
660: 1 64 org.apache.activemq.broker.region.ConnectorStatistics
667: 1 64 org.apache.activemq.command.BrokerInfo
671: 2 64 org.apache.activemq.broker.ProducerBrokerExchange
674: 1 56 org.apache.activemq.broker.jmx.ManagedQueueRegion
685: 1 56 org.apache.activemq.kaha.impl.data.DataManagerImpl
689: 7 56 org.apache.activemq.broker.region.policy.RoundRobinDispatchPolicy
703: 3 48 org.apache.activemq.util.FactoryFinder
707: 2 48 org.apache.activemq.console.command.ShellCommand
730: 2 48 org.apache.activemq.util.IdGenerator
733: 1 48 [Lorg.apache.activemq.store.kahadb.data.KahaEntryType;
760: 3 48 org.apache.activemq.util.LongSequenceGenerator
761: 1 48 org.apache.activemq.management.JMSStatsImpl
772: 1 40 org.apache.activemq.RedeliveryPolicy
778: 1 40 org.apache.activemq.openwire.OpenWireFormatFactory
788: 1 40 org.apache.activemq.advisory.AdvisoryBroker
821: 1 40 org.apache.activemq.ActiveMQPrefetchPolicy
822: 2 32 org.apache.activemq.state.ProducerState
830: 1 32 org.apache.activemq.blob.BlobTransferPolicy
854: 1 32 org.apache.activemq.console.command.StartCommand
871: 1 32 [Lorg.apache.activemq.store.kahadb.data.KahaDestination$DestinationType;
882: 1 32 org.apache.activemq.store.kahadb.MessageDatabase$Metadata
902: 1 32 org.apache.activemq.console.Main
911: 1 24 org.apache.activemq.web.filter.ApplicationContextFilter
915: 1 24 org.apache.activemq.web.filter.ApplicationContextFilter$1
924: 1 24 org.apache.activemq.web.SessionPool
926: 1 24 org.apache.activemq.console.command.StartCommand$2
927: 1 24 org.apache.activemq.kaha.impl.index.StoreIndexReader
928: 1 24 org.apache.activemq.kaha.impl.index.StoreIndexWriter
929: 1 24 org.apache.activemq.broker.region.DestinationFactoryImpl
933: 1 24 org.apache.activemq.security.SecurityContext$1
949: 1 24 org.apache.activemq.broker.region.policy.PolicyMap
950: 1 24 org.apache.activemq.broker.region.virtual.VirtualTopic
957: 1 24 org.apache.activemq.broker.region.policy.IndividualDeadLetterStrategy
979: 1 24 org.apache.activemq.broker.BrokerService$3
983: 1 24 org.apache.activemq.util.DefaultIOExceptionHandler
988: 1 24 org.apache.activemq.broker.jmx.BrokerView
989: 1 24 org.apache.activemq.broker.TransactionBroker
992: 1 16 org.apache.activemq.store.kahadb.KahaDBStore$1
994: 1 16 org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller
1001: 1 16 org.apache.activemq.broker.BrokerRegistry
1019: 1 16 org.apache.activemq.util.LogWriterFinder
1026: 1 16 org.apache.activemq.broker.CompositeDestinationBroker
1036: 1 16 org.apache.activemq.broker.region.group.MessageGroupHashBucket
1038: 1 16 org.apache.activemq.util.FactoryFinder$StandaloneObjectFactory
1041: 1 16 org.apache.activemq.console.CommandContext
1046: 1 16 org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter
1053: 1 16 org.apache.activemq.broker.region.virtual.VirtualDestinationInterceptor
1056: 1 16 org.apache.activemq.console.formatter.CommandShellOutputFormatter
1061: 1 16 [Lorg.apache.activemq.broker.region.DestinationInterceptor;
1062: 1 16 [Lorg.apache.activemq.broker.region.virtual.VirtualDestination;
1065: 1 16 org.apache.activemq.broker.region.policy.SharedDeadLetterStrategy
1067: 1 16 org.apache.activemq.broker.region.CompositeDestinationInterceptor
1088: 1 16 org.apache.activemq.kaha.CommandMarshaller
1107: 1 16 org.apache.activemq.thread.Scheduler
1109: 1 16 org.apache.activemq.broker.jmx.ConnectorView
1114: 1 16 [Lorg.apache.activemq.command.BrokerInfo;
1120: 1 16 org.apache.activemq.broker.TransportConnector$1
1123: 1 16 org.apache.activemq.transport.tcp.TcpTransportServer$1
1124: 1 16 org.apache.activemq.broker.region.group.MessageGroupHashBucketFactory
1127: 1 16 org.apache.activemq.store.kahadb.MessageDatabase$MetadataMarshaller
1128: 1 16 org.apache.activemq.xbean.XBeanBrokerService$1
1134: 1 16 org.apache.activemq.command.BrokerId
1156: 1 16 org.apache.activemq.broker.region.NullMessageReference
1162: 1 8 org.apache.activemq.openwire.v5.ConnectionIdMarshaller
1167: 1 8 org.apache.activemq.openwire.v5.LocalTransactionIdMarshaller
1171: 1 8 org.apache.activemq.kaha.impl.index.RedoStoreIndexItem$1
1174: 1 8 org.apache.activemq.openwire.v5.ConnectionInfoMarshaller
1179: 1 8 org.apache.activemq.openwire.v5.MessageAckMarshaller
1186: 1 8 org.apache.activemq.web.SessionFilter
1191: 1 8 org.apache.activemq.openwire.v5.ConsumerControlMarshaller
1196: 1 8 org.apache.activemq.openwire.v5.MessageDispatchMarshaller
1200: 1 8 org.apache.activemq.openwire.v5.ConsumerIdMarshaller
1204: 1 8 org.apache.activemq.ActiveMQConnectionMetaData
1208: 1 8 org.apache.activemq.openwire.v5.MessageDispatchNotificationMarshaller
1216: 1 8 org.apache.activemq.openwire.v5.MessageIdMarshaller
1219: 1 8 org.apache.activemq.openwire.v5.ConsumerInfoMarshaller
1223: 1 8 org.apache.activemq.openwire.v5.MessagePullMarshaller
1226: 1 8 org.apache.activemq.openwire.v5.ControlCommandMarshaller
1228: 1 8 org.apache.activemq.transport.InactivityMonitor$5
1229: 1 8 org.apache.activemq.openwire.v5.ActiveMQMessageMarshaller
1230: 1 8 org.apache.activemq.broker.region.group.EmptyMessageGroupSet
1233: 1 8 org.apache.activemq.openwire.v5.ActiveMQBlobMessageMarshaller
1235: 1 8 org.apache.activemq.openwire.v5.NetworkBridgeFilterMarshaller
1244: 1 8 org.apache.activemq.openwire.v5.ResponseMarshaller
1247: 1 8 org.apache.activemq.openwire.v5.DataArrayResponseMarshaller
1248: 1 8 org.apache.activemq.openwire.v5.ProducerAckMarshaller
1249: 1 8 org.apache.activemq.transport.tcp.TcpTransportFactory
1251: 1 8 org.apache.activemq.openwire.v5.ActiveMQBytesMessageMarshaller
1253: 1 8 org.apache.activemq.usage.Usage$3
1262: 1 8 org.apache.activemq.openwire.v5.DataResponseMarshaller
1263: 1 8 org.apache.activemq.openwire.v5.ProducerIdMarshaller
1265: 1 8 org.apache.activemq.openwire.v5.ActiveMQMapMessageMarshaller
1275: 1 8 org.apache.activemq.openwire.v5.ActiveMQObjectMessageMarshaller
1277: 1 8 org.apache.activemq.openwire.v5.DestinationInfoMarshaller
1278: 1 8 org.apache.activemq.openwire.v5.ProducerInfoMarshaller
1279: 1 8 org.apache.activemq.store.kahadb.MessageDatabase$MessageKeysMarshaller
1280: 1 8 org.apache.activemq.kaha.BytesMarshaller
1283: 1 8 org.apache.activemq.kaha.ObjectMarshaller
1285: 1 8 org.apache.activemq.openwire.v5.DiscoveryEventMarshaller
1288: 1 8 org.apache.activemq.openwire.v5.RemoveInfoMarshaller
1293: 1 8 org.apache.activemq.openwire.v5.ActiveMQQueueMarshaller
1294: 1 8 org.apache.activemq.kaha.StringMarshaller
1295: 1 8 org.apache.activemq.command.ActiveMQMessage$1
1297: 1 8 org.apache.activemq.openwire.v5.ExceptionResponseMarshaller
1300: 1 8 org.apache.activemq.command.ActiveMQMessage$2
1303: 1 8 org.apache.activemq.openwire.v5.RemoveSubscriptionInfoMarshaller
1306: 1 8 org.apache.activemq.openwire.v5.ActiveMQStreamMessageMarshaller
1307: 1 8 org.apache.activemq.command.ActiveMQMessage$3
1310: 1 8 org.apache.activemq.kaha.MessageIdMarshaller
1311: 1 8 org.apache.activemq.web.WebConsoleStarter
1312: 1 8 org.apache.activemq.openwire.v5.FlushCommandMarshaller
1314: 1 8 org.apache.activemq.command.ActiveMQMessage$4
1316: 1 8 org.apache.activemq.openwire.v5.ReplayCommandMarshaller
1318: 1 8 org.apache.activemq.command.ActiveMQMessage$5
1319: 1 8 org.apache.activemq.openwire.v5.IntegerResponseMarshaller
1321: 1 8 org.apache.activemq.openwire.v5.ActiveMQTempQueueMarshaller
1324: 1 8 org.apache.activemq.command.ActiveMQMessage$6
1325: 1 8 org.apache.activemq.openwire.v5.SessionIdMarshaller
1327: 1 8 org.apache.activemq.command.ActiveMQMessage$7
1329: 1 8 org.apache.activemq.openwire.v5.ActiveMQTempTopicMarshaller
1330: 1 8 org.apache.activemq.openwire.v5.JournalQueueAckMarshaller
1332: 1 8 org.apache.activemq.broker.region.Queue$4
1337: 1 8 org.apache.activemq.command.ActiveMQMessage$8
1339: 1 8 org.apache.activemq.openwire.v5.SessionInfoMarshaller
1342: 1 8 org.apache.activemq.openwire.v5.ActiveMQTextMessageMarshaller
1343: 1 8 org.apache.activemq.command.ActiveMQMessage$9
1346: 1 8 org.apache.activemq.openwire.v5.JournalTopicAckMarshaller
1347: 1 8 org.apache.activemq.openwire.v5.ShutdownInfoMarshaller
1349: 1 8 org.apache.activemq.command.ActiveMQMessage$10
1350: 1 8 org.apache.activemq.openwire.v5.ActiveMQTopicMarshaller
1352: 1 8 org.apache.activemq.command.ActiveMQMessage$11
1357: 1 8 org.apache.activemq.openwire.v5.JournalTraceMarshaller
1359: 1 8 org.apache.activemq.openwire.v5.SubscriptionInfoMarshaller
1360: 1 8 org.apache.activemq.openwire.v5.BrokerIdMarshaller
1367: 1 8 org.apache.activemq.openwire.v5.JournalTransactionMarshaller
1368: 1 8 org.apache.activemq.ActiveMQConnectionFactory$1
1369: 1 8 org.apache.activemq.openwire.v5.TransactionInfoMarshaller
1371: 1 8 org.apache.activemq.openwire.v5.BrokerInfoMarshaller
1372: 1 8 org.apache.activemq.openwire.v5.KeepAliveInfoMarshaller
1374: 1 8 org.apache.activemq.openwire.v5.WireFormatInfoMarshaller
1378: 1 8 org.apache.activemq.transport.tcp.TcpTransport$3
1379: 1 8 org.apache.activemq.openwire.v5.ConnectionControlMarshaller
1381: 1 8 org.apache.activemq.openwire.v5.XATransactionIdMarshaller
1385: 1 8 org.apache.activemq.openwire.v5.PartialCommandMarshaller
1387: 1 8 org.apache.activemq.openwire.v5.LastPartialCommandMarshaller
1390: 1 8 org.apache.activemq.openwire.v5.ConnectionErrorMarshaller
1392: 1 8 org.apache.activemq.store.kahadb.MessageDatabase$LocationMarshaller
/usr/sbin/lsof -p 6700 | grep "can't identify" | wc -l
797
jstack 6700 | grep "nid=" | grep "ActiveMQ Connection Dispatcher" | wc -l
818
I can't see why I should have 818 ActiveMQ Connection Dispatcher threads running with their corresponding objects in the JVM. Could they be stuck there?
"ActiveMQ Connection Dispatcher: /172.30.210.39:38631" daemon prio=10 tid=0x09ef8c00 nid=0x1b12 in Object.wait() [0xce3c7000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
■waiting on <0xd47f5080> (a java.lang.Object)
at java.lang.Object.wait(Object.java:485)
at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
■locked <0xd47f5080> (a java.lang.Object)
at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
"ActiveMQ Connection Dispatcher: /172.30.210.39:38632" daemon prio=10 tid=0x09efa400 nid=0x1b15 in Object.wait() [0xce4ba000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
■waiting on <0xd480c968> (a java.lang.Object)
at java.lang.Object.wait(Object.java:485)
at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
■locked <0xd480c968> (a java.lang.Object)
at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> Zombie ActiveMQ Connection Dispatcher threads - seems to consume all process File Descriptors (FD leak)
> -------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3448
> URL: https://issues.apache.org/jira/browse/AMQ-3448
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Environment: Active MQ 5.3.2
> java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
> LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
> Distributor ID: CentOS
> Description: CentOS release 5.5 (Final)
> Release: 5.5
> Codename: Final
> Reporter: Carl Allain
> Priority: Critical
>
> Somehow linked to https://issues.apache.org/jira/browse/AMQ-3286 which was closed. I am opening here as I cannot reopen the old bug and I hope that with the information I provide here, someone will be able to have some insight of the possible cause and a fix.
> I don't know how to reproduce with a test case, but I have found 800+ of such "ActiveMQ Connection Dispatcher" threads. I also noted that the number of FDs for the process keeps increasing and after a few days, we have a "too many files opened" when going beyond the 1024 limit. Most of those FDs (hundreds) do look like:
> java 6700 lexo-ext 904u sock 0,5 1741176200 can't identify protocol
> There is not much activity on the system and that is still enough to get the problem.
> When we reach the limit of FDs, we get tons of stack traces in the logs, filling up the hard disk...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3448) Zombie ActiveMQ Connection Dispatcher
threads - seems to consume all process File Descriptors (FD leak)
Posted by "Carl Allain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084318#comment-13084318 ]
Carl Allain commented on AMQ-3448:
----------------------------------
To contrast, I have only 15 ActiveMQ Transport threads like this one:
~/activemq/apache-activemq-5.3.2/logs$ jstack 6700 | grep "nid=" | grep "ActiveMQ Transport" | wc -l
15
> Zombie ActiveMQ Connection Dispatcher threads - seems to consume all process File Descriptors (FD leak)
> -------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3448
> URL: https://issues.apache.org/jira/browse/AMQ-3448
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Environment: Active MQ 5.3.2
> java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
> LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
> Distributor ID: CentOS
> Description: CentOS release 5.5 (Final)
> Release: 5.5
> Codename: Final
> Reporter: Carl Allain
> Priority: Critical
>
> Somehow linked to https://issues.apache.org/jira/browse/AMQ-3286 which was closed. I am opening here as I cannot reopen the old bug and I hope that with the information I provide here, someone will be able to have some insight of the possible cause and a fix.
> I don't know how to reproduce with a test case, but I have found 800+ of such "ActiveMQ Connection Dispatcher" threads. I also noted that the number of FDs for the process keeps increasing and after a few days, we have a "too many files opened" when going beyond the 1024 limit. Most of those FDs (hundreds) do look like:
> java 6700 lexo-ext 904u sock 0,5 1741176200 can't identify protocol
> There is not much activity on the system and that is still enough to get the problem.
> When we reach the limit of FDs, we get tons of stack traces in the logs, filling up the hard disk...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3448) Zombie ActiveMQ Connection Dispatcher
threads - seems to consume all process File Descriptors (FD leak)
Posted by "Carl Allain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13093810#comment-13093810 ]
Carl Allain commented on AMQ-3448:
----------------------------------
One thing important to note, is that among the various connections made against this AMQ server, one of them is subject to a firewall and I suspect that the firewall closes unused connections after a while, but that AMQ does not handle well the closing of the connection by the firewall. What do you think of that possibility?
> Zombie ActiveMQ Connection Dispatcher threads - seems to consume all process File Descriptors (FD leak)
> -------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3448
> URL: https://issues.apache.org/jira/browse/AMQ-3448
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Environment: Active MQ 5.3.2
> java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
> LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
> Distributor ID: CentOS
> Description: CentOS release 5.5 (Final)
> Release: 5.5
> Codename: Final
> Reporter: Carl Allain
> Priority: Critical
>
> Somehow linked to https://issues.apache.org/jira/browse/AMQ-3286 which was closed. I am opening here as I cannot reopen the old bug and I hope that with the information I provide here, someone will be able to have some insight of the possible cause and a fix.
> I don't know how to reproduce with a test case, but I have found 800+ of such "ActiveMQ Connection Dispatcher" threads. I also noted that the number of FDs for the process keeps increasing and after a few days, we have a "too many files opened" when going beyond the 1024 limit. Most of those FDs (hundreds) do look like:
> java 6700 lexo-ext 904u sock 0,5 1741176200 can't identify protocol
> There is not much activity on the system and that is still enough to get the problem.
> When we reach the limit of FDs, we get tons of stack traces in the logs, filling up the hard disk...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3448) Zombie ActiveMQ Connection Dispatcher
threads - seems to consume all process File Descriptors (FD leak)
Posted by "mikab PENG (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126412#comment-13126412 ]
mikab PENG commented on AMQ-3448:
---------------------------------
sorry, the below code is test code :
public static void main(String[] argv) throws Exception {
for (int i = 0; i < 1000; i++) {
String url = "tcp://localhost:61616?wireFormat.maxInactivityDuration=0";
ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory(url);
ActiveMQConnection connection = (ActiveMQConnection) factory.createConnection();
System.err.println(connection);
connection.start();
}
System.exit(0);
}
I tried the last version(5.5.0) and didn't reproduce this error.
when I run these code on 4.1.1, each connection will create a zombie thread.
> Zombie ActiveMQ Connection Dispatcher threads - seems to consume all process File Descriptors (FD leak)
> -------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3448
> URL: https://issues.apache.org/jira/browse/AMQ-3448
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Environment: Active MQ 5.3.2
> java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
> LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
> Distributor ID: CentOS
> Description: CentOS release 5.5 (Final)
> Release: 5.5
> Codename: Final
> Reporter: Carl Allain
> Priority: Critical
>
> Somehow linked to https://issues.apache.org/jira/browse/AMQ-3286 which was closed. I am opening here as I cannot reopen the old bug and I hope that with the information I provide here, someone will be able to have some insight of the possible cause and a fix.
> I don't know how to reproduce with a test case, but I have found 800+ of such "ActiveMQ Connection Dispatcher" threads. I also noted that the number of FDs for the process keeps increasing and after a few days, we have a "too many files opened" when going beyond the 1024 limit. Most of those FDs (hundreds) do look like:
> java 6700 lexo-ext 904u sock 0,5 1741176200 can't identify protocol
> There is not much activity on the system and that is still enough to get the problem.
> When we reach the limit of FDs, we get tons of stack traces in the logs, filling up the hard disk...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3448) Zombie ActiveMQ Connection Dispatcher
threads - seems to consume all process File Descriptors (FD leak)
Posted by "Carl Allain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084357#comment-13084357 ]
Carl Allain commented on AMQ-3448:
----------------------------------
De retour lundi le 29 aout. En cas d'urgence, veuillez communiquer avec Patrick Li. Merci.
> Zombie ActiveMQ Connection Dispatcher threads - seems to consume all process File Descriptors (FD leak)
> -------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3448
> URL: https://issues.apache.org/jira/browse/AMQ-3448
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Environment: Active MQ 5.3.2
> java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
> LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
> Distributor ID: CentOS
> Description: CentOS release 5.5 (Final)
> Release: 5.5
> Codename: Final
> Reporter: Carl Allain
> Priority: Critical
>
> Somehow linked to https://issues.apache.org/jira/browse/AMQ-3286 which was closed. I am opening here as I cannot reopen the old bug and I hope that with the information I provide here, someone will be able to have some insight of the possible cause and a fix.
> I don't know how to reproduce with a test case, but I have found 800+ of such "ActiveMQ Connection Dispatcher" threads. I also noted that the number of FDs for the process keeps increasing and after a few days, we have a "too many files opened" when going beyond the 1024 limit. Most of those FDs (hundreds) do look like:
> java 6700 lexo-ext 904u sock 0,5 1741176200 can't identify protocol
> There is not much activity on the system and that is still enough to get the problem.
> When we reach the limit of FDs, we get tons of stack traces in the logs, filling up the hard disk...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (AMQ-3448) Zombie ActiveMQ Connection Dispatcher
threads - seems to consume all process File Descriptors (FD leak)
Posted by "Timothy Bish (Closed) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Timothy Bish closed AMQ-3448.
-----------------------------
Resolution: Cannot Reproduce
Unable to reproduce this, recommend trying a newer release to see if this occurs anymore.
> Zombie ActiveMQ Connection Dispatcher threads - seems to consume all process File Descriptors (FD leak)
> -------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3448
> URL: https://issues.apache.org/jira/browse/AMQ-3448
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Environment: Active MQ 5.3.2
> java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
> LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
> Distributor ID: CentOS
> Description: CentOS release 5.5 (Final)
> Release: 5.5
> Codename: Final
> Reporter: Carl Allain
> Priority: Critical
>
> Somehow linked to https://issues.apache.org/jira/browse/AMQ-3286 which was closed. I am opening here as I cannot reopen the old bug and I hope that with the information I provide here, someone will be able to have some insight of the possible cause and a fix.
> I don't know how to reproduce with a test case, but I have found 800+ of such "ActiveMQ Connection Dispatcher" threads. I also noted that the number of FDs for the process keeps increasing and after a few days, we have a "too many files opened" when going beyond the 1024 limit. Most of those FDs (hundreds) do look like:
> java 6700 lexo-ext 904u sock 0,5 1741176200 can't identify protocol
> There is not much activity on the system and that is still enough to get the problem.
> When we reach the limit of FDs, we get tons of stack traces in the logs, filling up the hard disk...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3448) Zombie ActiveMQ Connection Dispatcher
threads - seems to consume all process File Descriptors (FD leak)
Posted by "mikab PENG (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13108411#comment-13108411 ]
mikab PENG commented on AMQ-3448:
---------------------------------
I found this issue too.
I use a very old version(4.1.1) of ActiveMQ server, and use a very old version of NMS client to connect to the MQ server. I use the default configuration of the ActiveMQ Release. I found in the default configuration , ActiveMQ create a dedicated thread to dipatch message, these thread are the the threads you talking about. in my system, if the client connection doesn't provide the MaxInactivityDuration parameter or the value of this parameter less than 0 , the monitor thread will not started, so if the socket closed before we call the connection's close method, the dispatch thread will not close.
the error is occour in the method onException of class org.apache.activemq.transport.InactivityMonitor in 4.1.1. I have saw the newwest code, this code have been move to org.apache.activemq.transport.InactivityMonitor.AbstractInactivityMonitor, I don't think it have been fixed.
you can disable the asynchronous dispatch in the #{ACTIVEMQ_HOME}/conf/activemq.xml, or don't use dedicated thread to dipatch message in the cmd line, set the system property org.apache.activemq.UseDedicatedTaskRunner to false in the #{ACTIVEMQ_HOME}/bin/activemq.
i hope this is useful for you.
> Zombie ActiveMQ Connection Dispatcher threads - seems to consume all process File Descriptors (FD leak)
> -------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3448
> URL: https://issues.apache.org/jira/browse/AMQ-3448
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Environment: Active MQ 5.3.2
> java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
> LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
> Distributor ID: CentOS
> Description: CentOS release 5.5 (Final)
> Release: 5.5
> Codename: Final
> Reporter: Carl Allain
> Priority: Critical
>
> Somehow linked to https://issues.apache.org/jira/browse/AMQ-3286 which was closed. I am opening here as I cannot reopen the old bug and I hope that with the information I provide here, someone will be able to have some insight of the possible cause and a fix.
> I don't know how to reproduce with a test case, but I have found 800+ of such "ActiveMQ Connection Dispatcher" threads. I also noted that the number of FDs for the process keeps increasing and after a few days, we have a "too many files opened" when going beyond the 1024 limit. Most of those FDs (hundreds) do look like:
> java 6700 lexo-ext 904u sock 0,5 1741176200 can't identify protocol
> There is not much activity on the system and that is still enough to get the problem.
> When we reach the limit of FDs, we get tons of stack traces in the logs, filling up the hard disk...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (AMQ-3448) Zombie ActiveMQ Connection
Dispatcher threads - seems to consume all process File Descriptors (FD
leak)
Posted by "mikab PENG (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13108411#comment-13108411 ]
mikab PENG edited comment on AMQ-3448 at 9/20/11 7:12 AM:
----------------------------------------------------------
I found this issue too.
I use a very old version(4.1.1) of ActiveMQ server, and use a very old version of NMS client to connect to the MQ server. I use the default configuration of the ActiveMQ Release. I found in the default configuration , ActiveMQ create a dedicated thread to dipatch message, these thread are the the threads you talking about. in my system, if the client connection doesn't provide the MaxInactivityDuration parameter or the value of this parameter less than 0 , the monitor thread will not started, so if the socket closed before we call the connection's close method, the dispatch thread will not close.
the error is occour in the method onException of class org.apache.activemq.transport.InactivityMonitor in 4.1.1. I have saw the newwest code, this code have been move to org.apache.activemq.transport.AbstractInactivityMonitor, I don't think it have been fixed.
you can disable the asynchronous dispatch in the #{ACTIVEMQ_HOME}/conf/activemq.xml, or don't use dedicated thread to dipatch message in the cmd line, set the system property org.apache.activemq.UseDedicatedTaskRunner to false in the #{ACTIVEMQ_HOME}/bin/activemq.
i hope this is useful for you.
was (Author: mikab):
I found this issue too.
I use a very old version(4.1.1) of ActiveMQ server, and use a very old version of NMS client to connect to the MQ server. I use the default configuration of the ActiveMQ Release. I found in the default configuration , ActiveMQ create a dedicated thread to dipatch message, these thread are the the threads you talking about. in my system, if the client connection doesn't provide the MaxInactivityDuration parameter or the value of this parameter less than 0 , the monitor thread will not started, so if the socket closed before we call the connection's close method, the dispatch thread will not close.
the error is occour in the method onException of class org.apache.activemq.transport.InactivityMonitor in 4.1.1. I have saw the newwest code, this code have been move to org.apache.activemq.transport.InactivityMonitor.AbstractInactivityMonitor, I don't think it have been fixed.
you can disable the asynchronous dispatch in the #{ACTIVEMQ_HOME}/conf/activemq.xml, or don't use dedicated thread to dipatch message in the cmd line, set the system property org.apache.activemq.UseDedicatedTaskRunner to false in the #{ACTIVEMQ_HOME}/bin/activemq.
i hope this is useful for you.
> Zombie ActiveMQ Connection Dispatcher threads - seems to consume all process File Descriptors (FD leak)
> -------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3448
> URL: https://issues.apache.org/jira/browse/AMQ-3448
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Environment: Active MQ 5.3.2
> java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
> LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
> Distributor ID: CentOS
> Description: CentOS release 5.5 (Final)
> Release: 5.5
> Codename: Final
> Reporter: Carl Allain
> Priority: Critical
>
> Somehow linked to https://issues.apache.org/jira/browse/AMQ-3286 which was closed. I am opening here as I cannot reopen the old bug and I hope that with the information I provide here, someone will be able to have some insight of the possible cause and a fix.
> I don't know how to reproduce with a test case, but I have found 800+ of such "ActiveMQ Connection Dispatcher" threads. I also noted that the number of FDs for the process keeps increasing and after a few days, we have a "too many files opened" when going beyond the 1024 limit. Most of those FDs (hundreds) do look like:
> java 6700 lexo-ext 904u sock 0,5 1741176200 can't identify protocol
> There is not much activity on the system and that is still enough to get the problem.
> When we reach the limit of FDs, we get tons of stack traces in the logs, filling up the hard disk...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3448) Zombie ActiveMQ Connection Dispatcher
threads - seems to consume all process File Descriptors (FD leak)
Posted by "Carl Allain (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13125861#comment-13125861 ]
Carl Allain commented on AMQ-3448:
----------------------------------
Hi, one of the problems we face, is that AMQ 5.4 and 5.5 do use Spring 3 and all our client apps are Spring 2.5.6 based so upgrading to the latest and greatest is not that trivial.
> Zombie ActiveMQ Connection Dispatcher threads - seems to consume all process File Descriptors (FD leak)
> -------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3448
> URL: https://issues.apache.org/jira/browse/AMQ-3448
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Environment: Active MQ 5.3.2
> java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
> LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
> Distributor ID: CentOS
> Description: CentOS release 5.5 (Final)
> Release: 5.5
> Codename: Final
> Reporter: Carl Allain
> Priority: Critical
>
> Somehow linked to https://issues.apache.org/jira/browse/AMQ-3286 which was closed. I am opening here as I cannot reopen the old bug and I hope that with the information I provide here, someone will be able to have some insight of the possible cause and a fix.
> I don't know how to reproduce with a test case, but I have found 800+ of such "ActiveMQ Connection Dispatcher" threads. I also noted that the number of FDs for the process keeps increasing and after a few days, we have a "too many files opened" when going beyond the 1024 limit. Most of those FDs (hundreds) do look like:
> java 6700 lexo-ext 904u sock 0,5 1741176200 can't identify protocol
> There is not much activity on the system and that is still enough to get the problem.
> When we reach the limit of FDs, we get tons of stack traces in the logs, filling up the hard disk...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3448) Zombie ActiveMQ Connection Dispatcher
threads - seems to consume all process File Descriptors (FD leak)
Posted by "Carl Allain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100431#comment-13100431 ]
Carl Allain commented on AMQ-3448:
----------------------------------
More info. I found that the IP addresses that were seen by AMQ had no hostname information available. Could it be the source of the problem, in relation with errors like these ones:
WARN | Failed to register MBean: org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=openwire,ViewType=address,Name=/192.168.235.7_55843
WARN | Failed to register MBean: org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=openwire,ViewType=address,Name=/192.168.235.7_65303
that I have in the logs.
I mean, could an impossible hostname resolution end up with our FD/thread leak problem somehow?
> Zombie ActiveMQ Connection Dispatcher threads - seems to consume all process File Descriptors (FD leak)
> -------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3448
> URL: https://issues.apache.org/jira/browse/AMQ-3448
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Environment: Active MQ 5.3.2
> java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
> LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
> Distributor ID: CentOS
> Description: CentOS release 5.5 (Final)
> Release: 5.5
> Codename: Final
> Reporter: Carl Allain
> Priority: Critical
>
> Somehow linked to https://issues.apache.org/jira/browse/AMQ-3286 which was closed. I am opening here as I cannot reopen the old bug and I hope that with the information I provide here, someone will be able to have some insight of the possible cause and a fix.
> I don't know how to reproduce with a test case, but I have found 800+ of such "ActiveMQ Connection Dispatcher" threads. I also noted that the number of FDs for the process keeps increasing and after a few days, we have a "too many files opened" when going beyond the 1024 limit. Most of those FDs (hundreds) do look like:
> java 6700 lexo-ext 904u sock 0,5 1741176200 can't identify protocol
> There is not much activity on the system and that is still enough to get the problem.
> When we reach the limit of FDs, we get tons of stack traces in the logs, filling up the hard disk...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3448) Zombie ActiveMQ Connection Dispatcher
threads - seems to consume all process File Descriptors (FD leak)
Posted by "Carl Allain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084316#comment-13084316 ]
Carl Allain commented on AMQ-3448:
----------------------------------
In the logs, I also found messages like this (I don't know if there is a correlation between that message, the number of threads and the number of FDs being consumed):
WARN | Failed to register MBean: org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=openwire,ViewType=address,Name=/192.168.235.8_63370
It does NOT seemed to me that as soon as a new log entry like that one was showing up, a new FD was consumed, but again, this could be valuable information.
> Zombie ActiveMQ Connection Dispatcher threads - seems to consume all process File Descriptors (FD leak)
> -------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3448
> URL: https://issues.apache.org/jira/browse/AMQ-3448
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Environment: Active MQ 5.3.2
> java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
> LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
> Distributor ID: CentOS
> Description: CentOS release 5.5 (Final)
> Release: 5.5
> Codename: Final
> Reporter: Carl Allain
> Priority: Critical
>
> Somehow linked to https://issues.apache.org/jira/browse/AMQ-3286 which was closed. I am opening here as I cannot reopen the old bug and I hope that with the information I provide here, someone will be able to have some insight of the possible cause and a fix.
> I don't know how to reproduce with a test case, but I have found 800+ of such "ActiveMQ Connection Dispatcher" threads. I also noted that the number of FDs for the process keeps increasing and after a few days, we have a "too many files opened" when going beyond the 1024 limit. Most of those FDs (hundreds) do look like:
> java 6700 lexo-ext 904u sock 0,5 1741176200 can't identify protocol
> There is not much activity on the system and that is still enough to get the problem.
> When we reach the limit of FDs, we get tons of stack traces in the logs, filling up the hard disk...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3448) Zombie ActiveMQ Connection Dispatcher
threads - seems to consume all process File Descriptors (FD leak)
Posted by "Carl Allain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100358#comment-13100358 ]
Carl Allain commented on AMQ-3448:
----------------------------------
Seems too like the host from where the leaking threads/connections come from is creating a connection
everysecond or so. Could it be the problem? From the dev team, I obtained the "spring" config:
<!-- ActiveMQ connection factory -->
<bean id="activemqConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="failover${JmsQueueManager.brokerUrl})?startupMaxReconnectAttempts=1"/>
<property name="redeliveryPolicy" ref="redeliveryPolicy"/>
<property name="prefetchPolicy" ref="activemqPrefetchPolicy"/>
<property name="userName" value="${JmsQueueManager.userName}"/>
<property name="password" value="${JmsQueueManager.password.encrypted}"/>
</bean>
<bean id="redeliveryPolicy" class="org.apache.activemq.RedeliveryPolicy">
<property name="maximumRedeliveries" value="0"/>
</bean>
<bean id="activemqPrefetchPolicy" class="org.apache.activemq.ActiveMQPrefetchPolicy">
<property name="queuePrefetch" value="1"/>
</bean>
<bean id="pooledConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory" destroy-method="stop">
<property name="connectionFactory" ref="activemqConnectionFactory"/>
</bean>
<!-- Message listener for the ticketing queue -->
<bean id="tktMessageListener" class="com.xyz.rex.broker.jms.TktMessageListener"/>
<bean id="transactionManager" class="org.springframework.transaction.jta.JtaTransactionManager"/>
<!-- Container for the ticketing queue -->
<bean id="tktJmsContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="connectionFactory" ref="pooledConnectionFactory"/>
<property name="destinationName" value="${JmsQueueManager.ticketingQueue.name}"/>
<property name="messageListener" ref="tktMessageListener"/>
<property name="recoveryInterval" value="${JmsQueueManager.connectionRetryInterval}"/>
<property name="transactionManager" ref="transactionManager"/>
</bean>
> Zombie ActiveMQ Connection Dispatcher threads - seems to consume all process File Descriptors (FD leak)
> -------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3448
> URL: https://issues.apache.org/jira/browse/AMQ-3448
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Environment: Active MQ 5.3.2
> java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
> LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
> Distributor ID: CentOS
> Description: CentOS release 5.5 (Final)
> Release: 5.5
> Codename: Final
> Reporter: Carl Allain
> Priority: Critical
>
> Somehow linked to https://issues.apache.org/jira/browse/AMQ-3286 which was closed. I am opening here as I cannot reopen the old bug and I hope that with the information I provide here, someone will be able to have some insight of the possible cause and a fix.
> I don't know how to reproduce with a test case, but I have found 800+ of such "ActiveMQ Connection Dispatcher" threads. I also noted that the number of FDs for the process keeps increasing and after a few days, we have a "too many files opened" when going beyond the 1024 limit. Most of those FDs (hundreds) do look like:
> java 6700 lexo-ext 904u sock 0,5 1741176200 can't identify protocol
> There is not much activity on the system and that is still enough to get the problem.
> When we reach the limit of FDs, we get tons of stack traces in the logs, filling up the hard disk...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3448) Zombie ActiveMQ Connection Dispatcher
threads - seems to consume all process File Descriptors (FD leak)
Posted by "Carl Allain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084319#comment-13084319 ]
Carl Allain commented on AMQ-3448:
----------------------------------
It seems every time a new ActiveMQ Connection Dispatcher thread become "zombie", that a file descriptor is held for an indefinite time, leading to the consumption of all FDs eventually (or all RAM/Permgen available as ...)
> Zombie ActiveMQ Connection Dispatcher threads - seems to consume all process File Descriptors (FD leak)
> -------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3448
> URL: https://issues.apache.org/jira/browse/AMQ-3448
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Environment: Active MQ 5.3.2
> java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
> LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
> Distributor ID: CentOS
> Description: CentOS release 5.5 (Final)
> Release: 5.5
> Codename: Final
> Reporter: Carl Allain
> Priority: Critical
>
> Somehow linked to https://issues.apache.org/jira/browse/AMQ-3286 which was closed. I am opening here as I cannot reopen the old bug and I hope that with the information I provide here, someone will be able to have some insight of the possible cause and a fix.
> I don't know how to reproduce with a test case, but I have found 800+ of such "ActiveMQ Connection Dispatcher" threads. I also noted that the number of FDs for the process keeps increasing and after a few days, we have a "too many files opened" when going beyond the 1024 limit. Most of those FDs (hundreds) do look like:
> java 6700 lexo-ext 904u sock 0,5 1741176200 can't identify protocol
> There is not much activity on the system and that is still enough to get the problem.
> When we reach the limit of FDs, we get tons of stack traces in the logs, filling up the hard disk...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3448) Zombie ActiveMQ Connection Dispatcher
threads - seems to consume all process File Descriptors (FD leak)
Posted by "Carl Allain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084325#comment-13084325 ]
Carl Allain commented on AMQ-3448:
----------------------------------
Also, we see all FDs consumed with lsof, but none of them shows up with netstat -an
> Zombie ActiveMQ Connection Dispatcher threads - seems to consume all process File Descriptors (FD leak)
> -------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3448
> URL: https://issues.apache.org/jira/browse/AMQ-3448
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Environment: Active MQ 5.3.2
> java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
> LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
> Distributor ID: CentOS
> Description: CentOS release 5.5 (Final)
> Release: 5.5
> Codename: Final
> Reporter: Carl Allain
> Priority: Critical
>
> Somehow linked to https://issues.apache.org/jira/browse/AMQ-3286 which was closed. I am opening here as I cannot reopen the old bug and I hope that with the information I provide here, someone will be able to have some insight of the possible cause and a fix.
> I don't know how to reproduce with a test case, but I have found 800+ of such "ActiveMQ Connection Dispatcher" threads. I also noted that the number of FDs for the process keeps increasing and after a few days, we have a "too many files opened" when going beyond the 1024 limit. Most of those FDs (hundreds) do look like:
> java 6700 lexo-ext 904u sock 0,5 1741176200 can't identify protocol
> There is not much activity on the system and that is still enough to get the problem.
> When we reach the limit of FDs, we get tons of stack traces in the logs, filling up the hard disk...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3448) Zombie ActiveMQ Connection Dispatcher
threads - seems to consume all process File Descriptors (FD leak)
Posted by "Carl Allain (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126633#comment-13126633 ]
Carl Allain commented on AMQ-3448:
----------------------------------
Is there a client vs server compatibility matrix chart somewhere that is available for such questions (that I guess are frequent)?
> Zombie ActiveMQ Connection Dispatcher threads - seems to consume all process File Descriptors (FD leak)
> -------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3448
> URL: https://issues.apache.org/jira/browse/AMQ-3448
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.2
> Environment: Active MQ 5.3.2
> java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
> LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
> Distributor ID: CentOS
> Description: CentOS release 5.5 (Final)
> Release: 5.5
> Codename: Final
> Reporter: Carl Allain
> Priority: Critical
>
> Somehow linked to https://issues.apache.org/jira/browse/AMQ-3286 which was closed. I am opening here as I cannot reopen the old bug and I hope that with the information I provide here, someone will be able to have some insight of the possible cause and a fix.
> I don't know how to reproduce with a test case, but I have found 800+ of such "ActiveMQ Connection Dispatcher" threads. I also noted that the number of FDs for the process keeps increasing and after a few days, we have a "too many files opened" when going beyond the 1024 limit. Most of those FDs (hundreds) do look like:
> java 6700 lexo-ext 904u sock 0,5 1741176200 can't identify protocol
> There is not much activity on the system and that is still enough to get the problem.
> When we reach the limit of FDs, we get tons of stack traces in the logs, filling up the hard disk...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira