You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Shobhana <sh...@quickride.in> on 2017/11/13 09:59:27 UTC
Messages stuck in broker
Hi,
I have a network of brokers with just 2 brokers in the network. Relevant
configurations are shown in attached files :
On Node-1 :
activemq-PrimaryBorker.xml
<http://activemq.2283324.n4.nabble.com/file/t377415/activemq-PrimaryBorker.xml>
On Node-2 :
activemq-SecondaryBorker.xml
<http://activemq.2283324.n4.nabble.com/file/t377415/activemq-SecondaryBorker.xml>
When a topic subscriber is connected to Node-1 and publisher is connected to
Node-2, sometimes I see that messages are stuck in the broker and not given
to subscriber. Statistics on Node-1 show that consumer is connected :
{"timestamp":1510559053,"status":200,"request":{"mbean":"org.apache.activemq:brokerName=PrimaryBroker,destinationName=topicname,destinationType=Topic,type=Broker","attribute":"ConsumerCount","type":"read"},"value":1}
I can also see some messages are enqueued :
{"timestamp":1510559073,"status":200,"request":{"mbean":"org.apache.activemq:brokerName=PrimaryBroker,destinationName=topicname,destinationType=Topic,type=Broker","attribute":"EnqueueCount","type":"read"},"value":12}
However, dequeue count is 0 :
{"timestamp":1510559078,"status":200,"request":{"mbean":"org.apache.activemq:brokerName=PrimaryBroker,destinationName=topicname,destinationType=Topic,type=Broker","attribute":"DequeueCount","type":"read"},"value":0}
I had used same configuration in my test environment before rolling into
production and in my test env, my subscriber would always receive the
messages sent to any of the brokers. The main difference between my test env
and production env is that the production env has a high no of connections
(several tens of thousands) and topics (again several tens of thousands)!
Any hints on why messages are not delivered? How can I analyse further?
Thanks & Regards,
Shobhana
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Messages stuck in broker
Posted by Tim Bain <tb...@alumni.duke.edu>.
The inactivity monitor is supposed to keep the connection from timing out,
but clearly that's not happening. You haven't done anything to
intentionally disable its keepalive feature, have you? I don't see anything
in your config files, but I want to make sure.
Tim
On Nov 13, 2017 3:54 AM, "Shobhana" <sh...@quickride.in> wrote:
> I see a bunch of these messages in the logs of Node-2 :
>
> 2017-11-13 15:30:06,506 | INFO | Establishing network connection from
> vm://SecondaryBroker?create=false&async=false to tcp://x.x.x.x:61616 |
> org.apache.activemq.network.DiscoveryNetworkConnector | ActiveMQ Task-4
> 2017-11-13 15:37:49,129 | INFO | ActiveMQ.Advisory.MasterBroker Inactive
> for longer than 1800000 ms - removing ... |
> org.apache.activemq.broker.region.Topic | ActiveMQ Broker[SecondaryBroker]
> Scheduler
> 2017-11-13 15:38:37,095 | WARN | Network connection between
> vm://SecondaryBroker#16 and tcp://x.x.x.x/x.x.x.x:61616@46009 shutdown due
> to a remote error: java.util.concurrent.TimeoutException |
> org.apache.activemq.network.DemandForwardingBridgeSupport |
> triggerStartAsyncNetworkBridgeCreation:
> remoteBroker=tcp://x.x.x.x/x.x.x.x:61616@46009, localBroker=
> vm://SecondaryBroker#16
> 2017-11-13 15:38:37,101 | INFO | Connector vm://SecondaryBroker stopped |
> org.apache.activemq.broker.TransportConnector | ActiveMQ
> BrokerService[SecondaryBroker] Task-21
> 2017-11-13 15:38:37,102 | INFO | SecondaryBroker bridge to Unknown stopped
> | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
> BrokerService[SecondaryBroker] Task-21
> 2017-11-13 15:38:38,101 | INFO | Establishing network connection from
> vm://SecondaryBroker?create=false&async=false to tcp://x.x.x.x:61616 |
> org.apache.activemq.network.DiscoveryNetworkConnector | ActiveMQ Task-5
> 2017-11-13 15:38:38,102 | INFO | Connector vm://SecondaryBroker started |
> org.apache.activemq.broker.TransportConnector | ActiveMQ Task-5
> 2017-11-13 15:39:08,169 | WARN | Network connection between
> vm://SecondaryBroker#20 and tcp://x.x.x.x/x.x.x.x:61616@46010 shutdown due
> to a remote error: java.util.concurrent.TimeoutException |
> org.apache.activemq.network.DemandForwardingBridgeSupport |
> triggerStartAsyncNetworkBridgeCreation:
> remoteBroker=tcp://x.x.x.x/x.x.x.x:61616@46010, localBroker=
> vm://SecondaryBroker#20
> 2017-11-13 15:39:08,171 | INFO | Connector vm://SecondaryBroker stopped |
> org.apache.activemq.broker.TransportConnector | ActiveMQ
> BrokerService[SecondaryBroker] Task-25
> 2017-11-13 15:39:08,171 | INFO | SecondaryBroker bridge to Unknown stopped
> | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
> BrokerService[SecondaryBroker] Task-25
>
> I could only understand second statement in the logs above. How do I
> indicate to the broker that network connector should not be timed out? Can
> I
> increase the maxInactivityDuration of the transport connector to a large
> value in both brokers?
>
> What is the reason for other logs? I couldn't see anything relevant in the
> logs of Node-1 during this time.
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>
Re: Messages stuck in broker
Posted by Shobhana <sh...@quickride.in>.
I see a bunch of these messages in the logs of Node-2 :
2017-11-13 15:30:06,506 | INFO | Establishing network connection from
vm://SecondaryBroker?create=false&async=false to tcp://x.x.x.x:61616 |
org.apache.activemq.network.DiscoveryNetworkConnector | ActiveMQ Task-4
2017-11-13 15:37:49,129 | INFO | ActiveMQ.Advisory.MasterBroker Inactive
for longer than 1800000 ms - removing ... |
org.apache.activemq.broker.region.Topic | ActiveMQ Broker[SecondaryBroker]
Scheduler
2017-11-13 15:38:37,095 | WARN | Network connection between
vm://SecondaryBroker#16 and tcp://x.x.x.x/x.x.x.x:61616@46009 shutdown due
to a remote error: java.util.concurrent.TimeoutException |
org.apache.activemq.network.DemandForwardingBridgeSupport |
triggerStartAsyncNetworkBridgeCreation:
remoteBroker=tcp://x.x.x.x/x.x.x.x:61616@46009, localBroker=
vm://SecondaryBroker#16
2017-11-13 15:38:37,101 | INFO | Connector vm://SecondaryBroker stopped |
org.apache.activemq.broker.TransportConnector | ActiveMQ
BrokerService[SecondaryBroker] Task-21
2017-11-13 15:38:37,102 | INFO | SecondaryBroker bridge to Unknown stopped
| org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
BrokerService[SecondaryBroker] Task-21
2017-11-13 15:38:38,101 | INFO | Establishing network connection from
vm://SecondaryBroker?create=false&async=false to tcp://x.x.x.x:61616 |
org.apache.activemq.network.DiscoveryNetworkConnector | ActiveMQ Task-5
2017-11-13 15:38:38,102 | INFO | Connector vm://SecondaryBroker started |
org.apache.activemq.broker.TransportConnector | ActiveMQ Task-5
2017-11-13 15:39:08,169 | WARN | Network connection between
vm://SecondaryBroker#20 and tcp://x.x.x.x/x.x.x.x:61616@46010 shutdown due
to a remote error: java.util.concurrent.TimeoutException |
org.apache.activemq.network.DemandForwardingBridgeSupport |
triggerStartAsyncNetworkBridgeCreation:
remoteBroker=tcp://x.x.x.x/x.x.x.x:61616@46010, localBroker=
vm://SecondaryBroker#20
2017-11-13 15:39:08,171 | INFO | Connector vm://SecondaryBroker stopped |
org.apache.activemq.broker.TransportConnector | ActiveMQ
BrokerService[SecondaryBroker] Task-25
2017-11-13 15:39:08,171 | INFO | SecondaryBroker bridge to Unknown stopped
| org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
BrokerService[SecondaryBroker] Task-25
I could only understand second statement in the logs above. How do I
indicate to the broker that network connector should not be timed out? Can I
increase the maxInactivityDuration of the transport connector to a large
value in both brokers?
What is the reason for other logs? I couldn't see anything relevant in the
logs of Node-1 during this time.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Messages stuck in broker
Posted by Tim Bain <tb...@alumni.duke.edu>.
I believe that no messages are being passed because you haven't enabled any
destinations to be sent across the network bridge. I see you're excluding
some, but you're not including any so the exclusion has no effect.
Tim
On Nov 13, 2017 2:59 AM, "Shobhana" <sh...@quickride.in> wrote:
> Hi,
>
> I have a network of brokers with just 2 brokers in the network. Relevant
> configurations are shown in attached files :
>
> On Node-1 :
> activemq-PrimaryBorker.xml
> <http://activemq.2283324.n4.nabble.com/file/t377415/
> activemq-PrimaryBorker.xml>
>
> On Node-2 :
> activemq-SecondaryBorker.xml
> <http://activemq.2283324.n4.nabble.com/file/t377415/
> activemq-SecondaryBorker.xml>
>
> When a topic subscriber is connected to Node-1 and publisher is connected
> to
> Node-2, sometimes I see that messages are stuck in the broker and not given
> to subscriber. Statistics on Node-1 show that consumer is connected :
> {"timestamp":1510559053,"status":200,"request":{"mbean"
> :"org.apache.activemq:brokerName=PrimaryBroker,destinationName=topicname,
> destinationType=Topic,type=Broker","attribute":"
> ConsumerCount","type":"read"},"value":1}
>
> I can also see some messages are enqueued :
> {"timestamp":1510559073,"status":200,"request":{"mbean"
> :"org.apache.activemq:brokerName=PrimaryBroker,destinationName=topicname,
> destinationType=Topic,type=Broker","attribute":"
> EnqueueCount","type":"read"},"value":12}
>
> However, dequeue count is 0 :
> {"timestamp":1510559078,"status":200,"request":{"mbean"
> :"org.apache.activemq:brokerName=PrimaryBroker,destinationName=topicname,
> destinationType=Topic,type=Broker","attribute":"
> DequeueCount","type":"read"},"value":0}
>
> I had used same configuration in my test environment before rolling into
> production and in my test env, my subscriber would always receive the
> messages sent to any of the brokers. The main difference between my test
> env
> and production env is that the production env has a high no of connections
> (several tens of thousands) and topics (again several tens of thousands)!
>
> Any hints on why messages are not delivered? How can I analyse further?
>
> Thanks & Regards,
> Shobhana
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>