You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Simon-Pierre LeBel (JIRA)" <ji...@apache.org> on 2011/04/19 14:08:05 UTC
[jira] [Created] (AMQ-3286) ActiveMQ Connection Dispatcher thread
ActiveMQ Connection Dispatcher thread
-------------------------------------
Key: AMQ-3286
URL: https://issues.apache.org/jira/browse/AMQ-3286
Project: ActiveMQ
Issue Type: Bug
Affects Versions: 5.3.2, 5.3.0
Environment: == Java ==
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)
== ActiveMQ ==
- Versions: 5.3.0 and 5.3.2
- Pure Master/Slave
- Openwire enabled
- FlowControl enabled
- KahaDB
Reporter: Simon-Pierre LeBel
I'm having some issues with ActiveMQ Connection Dispatcher threads. Whenever I get an InactivityIOException the thread goes into the Waiting state and a new thread is started.
== activemq.log ==
2011-04-18 10:15:31,244 | ERROR | Could not accept connection : org.apache.activemq.transport.InactivityIOException: Channel was inactive for too long: /127.0.0.1:35401 | org.apache.activemq.broker.TransportConnector | ActiveMQ Task
== jstack ==
"ActiveMQ Connection Dispatcher: /127.0.0.1:35401" daemon prio=10
tid=0x00007f10e440f800 nid=0x63e9 in Object.wait() [0x00007f10ce578000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
- locked <0x00007f113bbd4618> (a java.lang.Object)
at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
After running ActiveMQ for a couple of weeks there are more than 3k waiting threads doing nothing but wasting memory. If i disable the inactivity monitor the problem goes away (wireFormat.maxInactivityDuration=0).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3286) ActiveMQ Connection Dispatcher thread
Posted by "Carl Allain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084326#comment-13084326 ]
Carl Allain commented on AMQ-3286:
----------------------------------
I opened https://issues.apache.org/jira/browse/AMQ-3448 since I cannnot see how to reopen this bug.
> ActiveMQ Connection Dispatcher thread
> -------------------------------------
>
> Key: AMQ-3286
> URL: https://issues.apache.org/jira/browse/AMQ-3286
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.0, 5.3.2
> Environment: == Java ==
> java version "1.6.0_20"
> Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)
> == ActiveMQ ==
> - Versions: 5.3.0 and 5.3.2
> - Pure Master/Slave
> - Openwire enabled
> - FlowControl enabled
> - KahaDB
> Reporter: Simon-Pierre LeBel
>
> I'm having some issues with ActiveMQ Connection Dispatcher threads. Whenever I get an InactivityIOException the thread goes into the Waiting state and a new thread is started.
> == activemq.log ==
> 2011-04-18 10:15:31,244 | ERROR | Could not accept connection : org.apache.activemq.transport.InactivityIOException: Channel was inactive for too long: /127.0.0.1:35401 | org.apache.activemq.broker.TransportConnector | ActiveMQ Task
> == jstack ==
> "ActiveMQ Connection Dispatcher: /127.0.0.1:35401" daemon prio=10
> tid=0x00007f10e440f800 nid=0x63e9 in Object.wait() [0x00007f10ce578000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
> - locked <0x00007f113bbd4618> (a java.lang.Object)
> at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> After running ActiveMQ for a couple of weeks there are more than 3k waiting threads doing nothing but wasting memory. If i disable the inactivity monitor the problem goes away (wireFormat.maxInactivityDuration=0).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3286) ActiveMQ Connection Dispatcher thread
Posted by "Carl Allain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082596#comment-13082596 ]
Carl Allain commented on AMQ-3286:
----------------------------------
To contrast, I have only 15 ActiveMQ Transport threads like this one:
lexo-ext@cert1-lexo-util:~/activemq/apache-activemq-5.3.2/logs$ jstack 6700 | grep "nid=" | grep "ActiveMQ Transport" | wc -l
15
> ActiveMQ Connection Dispatcher thread
> -------------------------------------
>
> Key: AMQ-3286
> URL: https://issues.apache.org/jira/browse/AMQ-3286
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.0, 5.3.2
> Environment: == Java ==
> java version "1.6.0_20"
> Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)
> == ActiveMQ ==
> - Versions: 5.3.0 and 5.3.2
> - Pure Master/Slave
> - Openwire enabled
> - FlowControl enabled
> - KahaDB
> Reporter: Simon-Pierre LeBel
>
> I'm having some issues with ActiveMQ Connection Dispatcher threads. Whenever I get an InactivityIOException the thread goes into the Waiting state and a new thread is started.
> == activemq.log ==
> 2011-04-18 10:15:31,244 | ERROR | Could not accept connection : org.apache.activemq.transport.InactivityIOException: Channel was inactive for too long: /127.0.0.1:35401 | org.apache.activemq.broker.TransportConnector | ActiveMQ Task
> == jstack ==
> "ActiveMQ Connection Dispatcher: /127.0.0.1:35401" daemon prio=10
> tid=0x00007f10e440f800 nid=0x63e9 in Object.wait() [0x00007f10ce578000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
> - locked <0x00007f113bbd4618> (a java.lang.Object)
> at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> After running ActiveMQ for a couple of weeks there are more than 3k waiting threads doing nothing but wasting memory. If i disable the inactivity monitor the problem goes away (wireFormat.maxInactivityDuration=0).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3286) ActiveMQ Connection Dispatcher thread
Posted by "Carl Allain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082585#comment-13082585 ]
Carl Allain commented on AMQ-3286:
----------------------------------
In the logs, I also found messages like this (I don't know if there is a correlation between that message, the number of threads and the number of FDs being consumed):
WARN | Failed to register MBean: org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=openwire,ViewType=address,Name=/192.168.235.8_63370
> ActiveMQ Connection Dispatcher thread
> -------------------------------------
>
> Key: AMQ-3286
> URL: https://issues.apache.org/jira/browse/AMQ-3286
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.0, 5.3.2
> Environment: == Java ==
> java version "1.6.0_20"
> Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)
> == ActiveMQ ==
> - Versions: 5.3.0 and 5.3.2
> - Pure Master/Slave
> - Openwire enabled
> - FlowControl enabled
> - KahaDB
> Reporter: Simon-Pierre LeBel
>
> I'm having some issues with ActiveMQ Connection Dispatcher threads. Whenever I get an InactivityIOException the thread goes into the Waiting state and a new thread is started.
> == activemq.log ==
> 2011-04-18 10:15:31,244 | ERROR | Could not accept connection : org.apache.activemq.transport.InactivityIOException: Channel was inactive for too long: /127.0.0.1:35401 | org.apache.activemq.broker.TransportConnector | ActiveMQ Task
> == jstack ==
> "ActiveMQ Connection Dispatcher: /127.0.0.1:35401" daemon prio=10
> tid=0x00007f10e440f800 nid=0x63e9 in Object.wait() [0x00007f10ce578000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
> - locked <0x00007f113bbd4618> (a java.lang.Object)
> at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> After running ActiveMQ for a couple of weeks there are more than 3k waiting threads doing nothing but wasting memory. If i disable the inactivity monitor the problem goes away (wireFormat.maxInactivityDuration=0).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3286) ActiveMQ Connection Dispatcher thread
Posted by "Carl Allain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084312#comment-13084312 ]
Carl Allain commented on AMQ-3286:
----------------------------------
It seems every time a new ActiveMQ Connection Dispatcher thread become "zombie", that a file descriptor is held for an indefinite time, leading to the consumption of all FDs eventually (or all RAM/Permgen available as ...)
> ActiveMQ Connection Dispatcher thread
> -------------------------------------
>
> Key: AMQ-3286
> URL: https://issues.apache.org/jira/browse/AMQ-3286
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.0, 5.3.2
> Environment: == Java ==
> java version "1.6.0_20"
> Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)
> == ActiveMQ ==
> - Versions: 5.3.0 and 5.3.2
> - Pure Master/Slave
> - Openwire enabled
> - FlowControl enabled
> - KahaDB
> Reporter: Simon-Pierre LeBel
>
> I'm having some issues with ActiveMQ Connection Dispatcher threads. Whenever I get an InactivityIOException the thread goes into the Waiting state and a new thread is started.
> == activemq.log ==
> 2011-04-18 10:15:31,244 | ERROR | Could not accept connection : org.apache.activemq.transport.InactivityIOException: Channel was inactive for too long: /127.0.0.1:35401 | org.apache.activemq.broker.TransportConnector | ActiveMQ Task
> == jstack ==
> "ActiveMQ Connection Dispatcher: /127.0.0.1:35401" daemon prio=10
> tid=0x00007f10e440f800 nid=0x63e9 in Object.wait() [0x00007f10ce578000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
> - locked <0x00007f113bbd4618> (a java.lang.Object)
> at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> After running ActiveMQ for a couple of weeks there are more than 3k waiting threads doing nothing but wasting memory. If i disable the inactivity monitor the problem goes away (wireFormat.maxInactivityDuration=0).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3286) ActiveMQ Connection Dispatcher thread
Posted by "Carl Allain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082594#comment-13082594 ]
Carl Allain commented on AMQ-3286:
----------------------------------
Here is a dump of all activemq live objects in the JVM:
jmap -histo:live 6700 | grep activemq
1: 1640 105894928 [Lorg.apache.activemq.command.DataStructure;
26: 818 124336 org.apache.activemq.broker.jmx.ManagedTransportConnection
28: 818 111248 org.apache.activemq.transport.tcp.TcpTransport
33: 839 93968 org.apache.activemq.thread.DedicatedTaskRunner$1
35: 1816 87168 org.apache.activemq.management.CountStatisticImpl
36: 818 85072 org.apache.activemq.transport.InactivityMonitor
39: 1747 69880 org.apache.activemq.command.ProducerId
43: 1941 62112 org.apache.activemq.command.SessionId
46: 1722 55104 org.apache.activemq.command.LocalTransactionId
49: 820 45920 org.apache.activemq.openwire.OpenWireFormat
50: 818 45808 org.apache.activemq.broker.region.ConnectionStatistics
51: 2776 44416 org.apache.activemq.command.ConnectionId
53: 1046 41840 org.apache.activemq.command.ConsumerId
57: 818 39264 org.apache.activemq.transport.WireFormatNegotiator
71: 831 26592 org.apache.activemq.command.WireFormatInfo
75: 818 26176 org.apache.activemq.transport.tcp.TcpTransport$2
76: 818 26176 org.apache.activemq.transport.tcp.TcpBufferedOutputStream
83: 839 20136 org.apache.activemq.thread.DedicatedTaskRunner
85: 822 19728 org.apache.activemq.util.ByteSequence
86: 821 19704 org.apache.activemq.util.DataByteArrayInputStream
88: 818 19632 org.apache.activemq.transport.InactivityMonitor$1
89: 818 19632 org.apache.activemq.transport.InactivityMonitor$2
90: 818 19632 org.apache.activemq.transport.MutexTransport
104: 821 13136 org.apache.activemq.util.DataByteArrayOutputStream
108: 818 13088 org.apache.activemq.broker.TransportConnection$1
109: 818 13088 org.apache.activemq.broker.jmx.ConnectionView
110: 818 13088 org.apache.activemq.broker.SingleTransportConnectionStateRegister
132: 160 5120 org.apache.activemq.util.BitArrayBin
133: 317 5072 org.apache.activemq.broker.jmx.AnnotatedMBean
141: 1 4112 [Lorg.apache.activemq.command.ConsumerId;
144: 160 3840 org.apache.activemq.util.BitArray
158: 86 2752 org.apache.activemq.filter.DestinationMapNode
168: 60 2400 org.apache.activemq.command.ActiveMQTopic
174: 24 2112 org.apache.activemq.command.ConsumerInfo
179: 22 1936 org.apache.activemq.broker.region.DestinationStatistics
182: 14 1904 org.apache.activemq.broker.region.Topic
186: 22 1760 org.apache.activemq.management.TimeStatisticImpl
187: 31 1736 org.apache.activemq.util.LRUCache
190: 40 1600 org.apache.activemq.thread.SchedulerTimerTask
191: 20 1600 org.apache.activemq.broker.region.cursors.FilePendingMessageCursor
192: 22 1584 org.apache.activemq.usage.MemoryUsage
198: 26 1456 org.apache.activemq.command.ConnectionInfo
201: 22 1408 org.apache.activemq.usage.TempUsage
202: 22 1408 org.apache.activemq.usage.StoreUsage
203: 7 1400 org.apache.activemq.broker.region.Queue
215: 11 1232 org.apache.activemq.broker.region.QueueSubscription
223: 13 1144 org.apache.activemq.broker.region.TopicSubscription
227: 23 1104 org.apache.activemq.management.PollCountStatisticImpl
229: 22 1056 org.apache.activemq.usage.SystemUsage
231: 66 1056 org.apache.activemq.usage.DefaultUsageCapacity
235: 1 1040 [Lorg.apache.activemq.openwire.DataStreamMarshaller;
240: 13 936 org.apache.activemq.broker.ConnectionContext
249: 22 880 org.apache.activemq.command.ActiveMQQueue
252: 26 832 org.apache.activemq.command.SessionInfo
259: 7 784 org.apache.activemq.broker.region.Queue$FlowControlTimeoutTask
265: 31 744 org.apache.activemq.ActiveMQMessageAudit
282: 26 624 org.apache.activemq.state.SessionState
287: 13 624 org.apache.activemq.broker.TransportConnectionState
292: 37 592 org.apache.activemq.filter.SimpleDestinationFilter
301: 11 528 org.apache.activemq.broker.region.cursors.VMPendingMessageCursor
303: 7 504 org.apache.activemq.broker.region.cursors.StoreQueueCursor
304: 7 504 org.apache.activemq.broker.region.cursors.QueueStorePrefetch
329: 24 384 org.apache.activemq.state.ConsumerState
336: 7 336 org.apache.activemq.command.DestinationInfo
338: 7 336 org.apache.activemq.store.kahadb.MessageDatabase$StoredDestination
341: 14 336 [Lorg.apache.activemq.command.ActiveMQDestination;
342: 14 336 org.apache.activemq.thread.Valve
348: 13 312 [Lorg.apache.activemq.filter.DestinationFilter;
350: 13 312 org.apache.activemq.filter.MessageEvaluationContext
374: 1 248 org.apache.activemq.xbean.XBeanBrokerService
390: 7 224 org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore
391: 14 224 org.apache.activemq.broker.jmx.TopicView
394: 14 224 org.apache.activemq.broker.region.Topic$1
400: 13 208 org.apache.activemq.filter.CompositeDestinationFilter
401: 13 208 org.apache.activemq.filter.NoLocalExpression
402: 13 208 org.apache.activemq.broker.region.policy.OldestMessageEvictionStrategy
406: 13 208 org.apache.activemq.broker.jmx.TopicSubscriptionView
409: 5 200 org.apache.activemq.command.MessageId
415: 6 192 org.apache.activemq.broker.ConsumerBrokerExchange
420: 8 192 org.apache.activemq.store.kahadb.data.KahaEntryType
432: 1 176 org.apache.activemq.command.ActiveMQMessage
435: 11 176 org.apache.activemq.broker.jmx.SubscriptionView
438: 7 168 org.apache.activemq.store.kahadb.data.KahaDestination
451: 2 160 org.apache.activemq.kaha.impl.index.IndexItem
457: 1 152 org.apache.activemq.broker.region.policy.PolicyEntry
459: 1 144 org.apache.activemq.broker.jmx.ManagedRegionBroker
469: 1 136 org.apache.activemq.transport.tcp.TcpTransportServer
494: 1 120 org.apache.activemq.store.kahadb.KahaDBStore
497: 1 120 org.apache.activemq.ActiveMQConnectionFactory
508: 7 112 org.apache.activemq.broker.jmx.QueueView
509: 1 112 org.apache.activemq.console.command.StartCommand$1
514: 1 112 org.apache.activemq.store.kahadb.MessageDatabase$3
518: 7 112 org.apache.activemq.broker.region.Queue$1
519: 7 112 org.apache.activemq.broker.region.Queue$2
521: 7 112 org.apache.activemq.broker.region.Queue$3
523: 7 112 org.apache.activemq.broker.region.QueueDispatchSelector
524: 14 112 org.apache.activemq.broker.region.policy.SimpleDispatchPolicy
527: 1 112 org.apache.activemq.broker.BrokerService$4
528: 14 112 org.apache.activemq.broker.region.policy.NoSubscriptionRecoveryPolicy
535: 2 96 org.apache.activemq.command.ProducerInfo
557: 4 96 org.apache.activemq.store.kahadb.data.KahaDestination$DestinationType
573: 1 88 org.apache.activemq.web.AjaxServlet
582: 1 80 org.apache.activemq.web.MessageServlet
584: 5 80 org.apache.activemq.filter.DestinationMap
587: 1 80 org.apache.activemq.broker.jmx.ManagedTransportConnector
594: 1 80 org.apache.activemq.broker.jmx.ManagedTempQueueRegion
599: 1 80 org.apache.activemq.broker.jmx.ManagedTempTopicRegion
613: 1 72 org.apache.activemq.kaha.impl.index.IndexManager
616: 1 72 org.apache.activemq.kaha.impl.KahaStore
617: 1 72 org.apache.activemq.broker.jmx.ManagedTopicRegion
636: 2 64 org.apache.activemq.thread.TaskRunnerFactory
638: 1 64 org.apache.activemq.web.handler.BindingBeanNameUrlHandlerMapping
642: 1 64 org.apache.activemq.broker.jmx.ManagementContext
653: 2 64 org.apache.activemq.kaha.impl.IndexRootContainer
660: 1 64 org.apache.activemq.broker.region.ConnectorStatistics
667: 1 64 org.apache.activemq.command.BrokerInfo
671: 2 64 org.apache.activemq.broker.ProducerBrokerExchange
674: 1 56 org.apache.activemq.broker.jmx.ManagedQueueRegion
685: 1 56 org.apache.activemq.kaha.impl.data.DataManagerImpl
689: 7 56 org.apache.activemq.broker.region.policy.RoundRobinDispatchPolicy
703: 3 48 org.apache.activemq.util.FactoryFinder
707: 2 48 org.apache.activemq.console.command.ShellCommand
730: 2 48 org.apache.activemq.util.IdGenerator
733: 1 48 [Lorg.apache.activemq.store.kahadb.data.KahaEntryType;
760: 3 48 org.apache.activemq.util.LongSequenceGenerator
761: 1 48 org.apache.activemq.management.JMSStatsImpl
772: 1 40 org.apache.activemq.RedeliveryPolicy
778: 1 40 org.apache.activemq.openwire.OpenWireFormatFactory
788: 1 40 org.apache.activemq.advisory.AdvisoryBroker
821: 1 40 org.apache.activemq.ActiveMQPrefetchPolicy
822: 2 32 org.apache.activemq.state.ProducerState
830: 1 32 org.apache.activemq.blob.BlobTransferPolicy
854: 1 32 org.apache.activemq.console.command.StartCommand
871: 1 32 [Lorg.apache.activemq.store.kahadb.data.KahaDestination$DestinationType;
882: 1 32 org.apache.activemq.store.kahadb.MessageDatabase$Metadata
902: 1 32 org.apache.activemq.console.Main
911: 1 24 org.apache.activemq.web.filter.ApplicationContextFilter
915: 1 24 org.apache.activemq.web.filter.ApplicationContextFilter$1
924: 1 24 org.apache.activemq.web.SessionPool
926: 1 24 org.apache.activemq.console.command.StartCommand$2
927: 1 24 org.apache.activemq.kaha.impl.index.StoreIndexReader
928: 1 24 org.apache.activemq.kaha.impl.index.StoreIndexWriter
929: 1 24 org.apache.activemq.broker.region.DestinationFactoryImpl
933: 1 24 org.apache.activemq.security.SecurityContext$1
949: 1 24 org.apache.activemq.broker.region.policy.PolicyMap
950: 1 24 org.apache.activemq.broker.region.virtual.VirtualTopic
957: 1 24 org.apache.activemq.broker.region.policy.IndividualDeadLetterStrategy
979: 1 24 org.apache.activemq.broker.BrokerService$3
983: 1 24 org.apache.activemq.util.DefaultIOExceptionHandler
988: 1 24 org.apache.activemq.broker.jmx.BrokerView
989: 1 24 org.apache.activemq.broker.TransactionBroker
992: 1 16 org.apache.activemq.store.kahadb.KahaDBStore$1
994: 1 16 org.apache.activemq.store.kahadb.MessageDatabase$StoredDestinationMarshaller
1001: 1 16 org.apache.activemq.broker.BrokerRegistry
1019: 1 16 org.apache.activemq.util.LogWriterFinder
1026: 1 16 org.apache.activemq.broker.CompositeDestinationBroker
1036: 1 16 org.apache.activemq.broker.region.group.MessageGroupHashBucket
1038: 1 16 org.apache.activemq.util.FactoryFinder$StandaloneObjectFactory
1041: 1 16 org.apache.activemq.console.CommandContext
1046: 1 16 org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter
1053: 1 16 org.apache.activemq.broker.region.virtual.VirtualDestinationInterceptor
1056: 1 16 org.apache.activemq.console.formatter.CommandShellOutputFormatter
1061: 1 16 [Lorg.apache.activemq.broker.region.DestinationInterceptor;
1062: 1 16 [Lorg.apache.activemq.broker.region.virtual.VirtualDestination;
1065: 1 16 org.apache.activemq.broker.region.policy.SharedDeadLetterStrategy
1067: 1 16 org.apache.activemq.broker.region.CompositeDestinationInterceptor
1088: 1 16 org.apache.activemq.kaha.CommandMarshaller
1107: 1 16 org.apache.activemq.thread.Scheduler
1109: 1 16 org.apache.activemq.broker.jmx.ConnectorView
1114: 1 16 [Lorg.apache.activemq.command.BrokerInfo;
1120: 1 16 org.apache.activemq.broker.TransportConnector$1
1123: 1 16 org.apache.activemq.transport.tcp.TcpTransportServer$1
1124: 1 16 org.apache.activemq.broker.region.group.MessageGroupHashBucketFactory
1127: 1 16 org.apache.activemq.store.kahadb.MessageDatabase$MetadataMarshaller
1128: 1 16 org.apache.activemq.xbean.XBeanBrokerService$1
1134: 1 16 org.apache.activemq.command.BrokerId
1156: 1 16 org.apache.activemq.broker.region.NullMessageReference
1162: 1 8 org.apache.activemq.openwire.v5.ConnectionIdMarshaller
1167: 1 8 org.apache.activemq.openwire.v5.LocalTransactionIdMarshaller
1171: 1 8 org.apache.activemq.kaha.impl.index.RedoStoreIndexItem$1
1174: 1 8 org.apache.activemq.openwire.v5.ConnectionInfoMarshaller
1179: 1 8 org.apache.activemq.openwire.v5.MessageAckMarshaller
1186: 1 8 org.apache.activemq.web.SessionFilter
1191: 1 8 org.apache.activemq.openwire.v5.ConsumerControlMarshaller
1196: 1 8 org.apache.activemq.openwire.v5.MessageDispatchMarshaller
1200: 1 8 org.apache.activemq.openwire.v5.ConsumerIdMarshaller
1204: 1 8 org.apache.activemq.ActiveMQConnectionMetaData
1208: 1 8 org.apache.activemq.openwire.v5.MessageDispatchNotificationMarshaller
1216: 1 8 org.apache.activemq.openwire.v5.MessageIdMarshaller
1219: 1 8 org.apache.activemq.openwire.v5.ConsumerInfoMarshaller
1223: 1 8 org.apache.activemq.openwire.v5.MessagePullMarshaller
1226: 1 8 org.apache.activemq.openwire.v5.ControlCommandMarshaller
1228: 1 8 org.apache.activemq.transport.InactivityMonitor$5
1229: 1 8 org.apache.activemq.openwire.v5.ActiveMQMessageMarshaller
1230: 1 8 org.apache.activemq.broker.region.group.EmptyMessageGroupSet
1233: 1 8 org.apache.activemq.openwire.v5.ActiveMQBlobMessageMarshaller
1235: 1 8 org.apache.activemq.openwire.v5.NetworkBridgeFilterMarshaller
1244: 1 8 org.apache.activemq.openwire.v5.ResponseMarshaller
1247: 1 8 org.apache.activemq.openwire.v5.DataArrayResponseMarshaller
1248: 1 8 org.apache.activemq.openwire.v5.ProducerAckMarshaller
1249: 1 8 org.apache.activemq.transport.tcp.TcpTransportFactory
1251: 1 8 org.apache.activemq.openwire.v5.ActiveMQBytesMessageMarshaller
1253: 1 8 org.apache.activemq.usage.Usage$3
1262: 1 8 org.apache.activemq.openwire.v5.DataResponseMarshaller
1263: 1 8 org.apache.activemq.openwire.v5.ProducerIdMarshaller
1265: 1 8 org.apache.activemq.openwire.v5.ActiveMQMapMessageMarshaller
1275: 1 8 org.apache.activemq.openwire.v5.ActiveMQObjectMessageMarshaller
1277: 1 8 org.apache.activemq.openwire.v5.DestinationInfoMarshaller
1278: 1 8 org.apache.activemq.openwire.v5.ProducerInfoMarshaller
1279: 1 8 org.apache.activemq.store.kahadb.MessageDatabase$MessageKeysMarshaller
1280: 1 8 org.apache.activemq.kaha.BytesMarshaller
1283: 1 8 org.apache.activemq.kaha.ObjectMarshaller
1285: 1 8 org.apache.activemq.openwire.v5.DiscoveryEventMarshaller
1288: 1 8 org.apache.activemq.openwire.v5.RemoveInfoMarshaller
1293: 1 8 org.apache.activemq.openwire.v5.ActiveMQQueueMarshaller
1294: 1 8 org.apache.activemq.kaha.StringMarshaller
1295: 1 8 org.apache.activemq.command.ActiveMQMessage$1
1297: 1 8 org.apache.activemq.openwire.v5.ExceptionResponseMarshaller
1300: 1 8 org.apache.activemq.command.ActiveMQMessage$2
1303: 1 8 org.apache.activemq.openwire.v5.RemoveSubscriptionInfoMarshaller
1306: 1 8 org.apache.activemq.openwire.v5.ActiveMQStreamMessageMarshaller
1307: 1 8 org.apache.activemq.command.ActiveMQMessage$3
1310: 1 8 org.apache.activemq.kaha.MessageIdMarshaller
1311: 1 8 org.apache.activemq.web.WebConsoleStarter
1312: 1 8 org.apache.activemq.openwire.v5.FlushCommandMarshaller
1314: 1 8 org.apache.activemq.command.ActiveMQMessage$4
1316: 1 8 org.apache.activemq.openwire.v5.ReplayCommandMarshaller
1318: 1 8 org.apache.activemq.command.ActiveMQMessage$5
1319: 1 8 org.apache.activemq.openwire.v5.IntegerResponseMarshaller
1321: 1 8 org.apache.activemq.openwire.v5.ActiveMQTempQueueMarshaller
1324: 1 8 org.apache.activemq.command.ActiveMQMessage$6
1325: 1 8 org.apache.activemq.openwire.v5.SessionIdMarshaller
1327: 1 8 org.apache.activemq.command.ActiveMQMessage$7
1329: 1 8 org.apache.activemq.openwire.v5.ActiveMQTempTopicMarshaller
1330: 1 8 org.apache.activemq.openwire.v5.JournalQueueAckMarshaller
1332: 1 8 org.apache.activemq.broker.region.Queue$4
1337: 1 8 org.apache.activemq.command.ActiveMQMessage$8
1339: 1 8 org.apache.activemq.openwire.v5.SessionInfoMarshaller
1342: 1 8 org.apache.activemq.openwire.v5.ActiveMQTextMessageMarshaller
1343: 1 8 org.apache.activemq.command.ActiveMQMessage$9
1346: 1 8 org.apache.activemq.openwire.v5.JournalTopicAckMarshaller
1347: 1 8 org.apache.activemq.openwire.v5.ShutdownInfoMarshaller
1349: 1 8 org.apache.activemq.command.ActiveMQMessage$10
1350: 1 8 org.apache.activemq.openwire.v5.ActiveMQTopicMarshaller
1352: 1 8 org.apache.activemq.command.ActiveMQMessage$11
1357: 1 8 org.apache.activemq.openwire.v5.JournalTraceMarshaller
1359: 1 8 org.apache.activemq.openwire.v5.SubscriptionInfoMarshaller
1360: 1 8 org.apache.activemq.openwire.v5.BrokerIdMarshaller
1367: 1 8 org.apache.activemq.openwire.v5.JournalTransactionMarshaller
1368: 1 8 org.apache.activemq.ActiveMQConnectionFactory$1
1369: 1 8 org.apache.activemq.openwire.v5.TransactionInfoMarshaller
1371: 1 8 org.apache.activemq.openwire.v5.BrokerInfoMarshaller
1372: 1 8 org.apache.activemq.openwire.v5.KeepAliveInfoMarshaller
1374: 1 8 org.apache.activemq.openwire.v5.WireFormatInfoMarshaller
1378: 1 8 org.apache.activemq.transport.tcp.TcpTransport$3
1379: 1 8 org.apache.activemq.openwire.v5.ConnectionControlMarshaller
1381: 1 8 org.apache.activemq.openwire.v5.XATransactionIdMarshaller
1385: 1 8 org.apache.activemq.openwire.v5.PartialCommandMarshaller
1387: 1 8 org.apache.activemq.openwire.v5.LastPartialCommandMarshaller
1390: 1 8 org.apache.activemq.openwire.v5.ConnectionErrorMarshaller
1392: 1 8 org.apache.activemq.store.kahadb.MessageDatabase$LocationMarshaller
/usr/sbin/lsof -p 6700 | grep "can't identify" | wc -l
797
jstack 6700 | grep "nid=" | grep "ActiveMQ Connection Dispatcher" | wc -l
818
I can't see why I should have 818 ActiveMQ Connection Dispatcher threads running with their corresponding objects in the JVM. Could they be stuck there?
"ActiveMQ Connection Dispatcher: /172.30.210.39:38631" daemon prio=10 tid=0x09ef8c00 nid=0x1b12 in Object.wait() [0xce3c7000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xd47f5080> (a java.lang.Object)
at java.lang.Object.wait(Object.java:485)
at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
- locked <0xd47f5080> (a java.lang.Object)
at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
"ActiveMQ Connection Dispatcher: /172.30.210.39:38632" daemon prio=10 tid=0x09efa400 nid=0x1b15 in Object.wait() [0xce4ba000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xd480c968> (a java.lang.Object)
at java.lang.Object.wait(Object.java:485)
at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
- locked <0xd480c968> (a java.lang.Object)
at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> ActiveMQ Connection Dispatcher thread
> -------------------------------------
>
> Key: AMQ-3286
> URL: https://issues.apache.org/jira/browse/AMQ-3286
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.0, 5.3.2
> Environment: == Java ==
> java version "1.6.0_20"
> Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)
> == ActiveMQ ==
> - Versions: 5.3.0 and 5.3.2
> - Pure Master/Slave
> - Openwire enabled
> - FlowControl enabled
> - KahaDB
> Reporter: Simon-Pierre LeBel
>
> I'm having some issues with ActiveMQ Connection Dispatcher threads. Whenever I get an InactivityIOException the thread goes into the Waiting state and a new thread is started.
> == activemq.log ==
> 2011-04-18 10:15:31,244 | ERROR | Could not accept connection : org.apache.activemq.transport.InactivityIOException: Channel was inactive for too long: /127.0.0.1:35401 | org.apache.activemq.broker.TransportConnector | ActiveMQ Task
> == jstack ==
> "ActiveMQ Connection Dispatcher: /127.0.0.1:35401" daemon prio=10
> tid=0x00007f10e440f800 nid=0x63e9 in Object.wait() [0x00007f10ce578000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
> - locked <0x00007f113bbd4618> (a java.lang.Object)
> at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> After running ActiveMQ for a couple of weeks there are more than 3k waiting threads doing nothing but wasting memory. If i disable the inactivity monitor the problem goes away (wireFormat.maxInactivityDuration=0).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (AMQ-3286) ActiveMQ Connection Dispatcher thread
Posted by "Timothy Bish (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Timothy Bish closed AMQ-3286.
-----------------------------
Resolution: Cannot Reproduce
Tried a couple different tests but couldn't reproduce this issue, if you have a test case that can demonstrate the issue please reopen and attach it.
> ActiveMQ Connection Dispatcher thread
> -------------------------------------
>
> Key: AMQ-3286
> URL: https://issues.apache.org/jira/browse/AMQ-3286
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.0, 5.3.2
> Environment: == Java ==
> java version "1.6.0_20"
> Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)
> == ActiveMQ ==
> - Versions: 5.3.0 and 5.3.2
> - Pure Master/Slave
> - Openwire enabled
> - FlowControl enabled
> - KahaDB
> Reporter: Simon-Pierre LeBel
>
> I'm having some issues with ActiveMQ Connection Dispatcher threads. Whenever I get an InactivityIOException the thread goes into the Waiting state and a new thread is started.
> == activemq.log ==
> 2011-04-18 10:15:31,244 | ERROR | Could not accept connection : org.apache.activemq.transport.InactivityIOException: Channel was inactive for too long: /127.0.0.1:35401 | org.apache.activemq.broker.TransportConnector | ActiveMQ Task
> == jstack ==
> "ActiveMQ Connection Dispatcher: /127.0.0.1:35401" daemon prio=10
> tid=0x00007f10e440f800 nid=0x63e9 in Object.wait() [0x00007f10ce578000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
> - locked <0x00007f113bbd4618> (a java.lang.Object)
> at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> After running ActiveMQ for a couple of weeks there are more than 3k waiting threads doing nothing but wasting memory. If i disable the inactivity monitor the problem goes away (wireFormat.maxInactivityDuration=0).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3286) ActiveMQ Connection Dispatcher thread
Posted by "Carl Allain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082562#comment-13082562 ]
Carl Allain commented on AMQ-3286:
----------------------------------
Active MQ 5.3.2
java version "1.6.0_23"
Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
Java HotSpot(TM) Server VM (build 19.0-b09, mixed mode)
LSB Version: :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
Distributor ID: CentOS
Description: CentOS release 5.5 (Final)
Release: 5.5
Codename: Final
I don't know how to reproduce with a test case, but I have found 800+ of such "ActiveMQ Connection Dispatcher" threads. I also noted that the number of FDs for the process keeps increasing and after a few days, we have a "too many files opened" when going beyond the 1024 limit. Most of those FDs (hundreds) do look like:
java 6700 lexo-ext 904u sock 0,5 1741176200 can't identify protocol
There is not much activity on the system and that is still enough to get the problem.
When we reach the limit of FDs, we get tons of stack traces in the logs, filling up the hard disk...
This is a pain.
> ActiveMQ Connection Dispatcher thread
> -------------------------------------
>
> Key: AMQ-3286
> URL: https://issues.apache.org/jira/browse/AMQ-3286
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.3.0, 5.3.2
> Environment: == Java ==
> java version "1.6.0_20"
> Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)
> == ActiveMQ ==
> - Versions: 5.3.0 and 5.3.2
> - Pure Master/Slave
> - Openwire enabled
> - FlowControl enabled
> - KahaDB
> Reporter: Simon-Pierre LeBel
>
> I'm having some issues with ActiveMQ Connection Dispatcher threads. Whenever I get an InactivityIOException the thread goes into the Waiting state and a new thread is started.
> == activemq.log ==
> 2011-04-18 10:15:31,244 | ERROR | Could not accept connection : org.apache.activemq.transport.InactivityIOException: Channel was inactive for too long: /127.0.0.1:35401 | org.apache.activemq.broker.TransportConnector | ActiveMQ Task
> == jstack ==
> "ActiveMQ Connection Dispatcher: /127.0.0.1:35401" daemon prio=10
> tid=0x00007f10e440f800 nid=0x63e9 in Object.wait() [0x00007f10ce578000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
> - locked <0x00007f113bbd4618> (a java.lang.Object)
> at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> After running ActiveMQ for a couple of weeks there are more than 3k waiting threads doing nothing but wasting memory. If i disable the inactivity monitor the problem goes away (wireFormat.maxInactivityDuration=0).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira