You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by oholiab <oh...@gmail.com> on 2014/01/17 22:50:03 UTC

TCP send and receive buffers filling preventing openwire communication

Problem:
We're experiencing a problem whereby openwire communication completely locks
up and the tcp send and receive buffers fill up preventing any further
communication between brokers, whilst the stomp client connectors continue
to function perfectly.

Below is an example of this happening between our hub broker (B) and another
broker (C)

$ netstat -a | grep :61616
tcp6       0      0 [::]:61616              [::]:*                  LISTEN
tcp6       0      0 B:34326 A:61616 ESTABLISHED
tcp6  116166 142456 B:39099 C:61616 ESTABLISHED
tcp6  115169 156860 B:39100 C:61616 ESTABLISHED
tcp6       0      0 B:34328 A:61616 ESTABLISHED

Topology:
I've reduced our mesh down to 3 nodes, A <-> B <-> C where the connectors
are dual openwire connections, using failover:(static:(nio://...)), one with
topics being pushed down it and the other with queues, both in duplex. All
networkConnectors are initiated from B

Usage summary:
We are primarily using the mesh for MCollective
(http://puppetlabs.com/mcollective) which only utilises the STOMP transport
connectors directly, servicing approximately 600 nodes on a roughly equal
distribution across the three brokers.

We also have a team using virtual topics primarily via C (that will
multiplex to queues in the presence of consumers on the relevant
destinations - usually 2 to any one virtual queue), but with a couple of
consumers on B. We are still trying to gauge their level of usage in some
meaningful metrics, but my understanding is that the volume of produced
messages is not high.

Configuration:

E.g. networkConnectors from B to A

      <networkConnector
        name="B-A-topics"
        uri="static:(failover:(nio://A:61616))"
        userName="amq"
        password="secret"
        duplex="true"
        decreaseNetworkConsumerPriority="true"
        networkTTL="4"
        dynamicOnly="true">
        <excludedDestinations>
          <queue physicalName=">" />
        </excludedDestinations>
      </networkConnector>
      <networkConnector
        name="B-A-queues"
        uri="static:(failover:(nio://A:61616))"
        userName="amq"
        password="secret"
        duplex="true"
        decreaseNetworkConsumerPriority="true"
        networkTTL="4"
        dynamicOnly="true"
        conduitSubscriptions="false">
        <excludedDestinations>
          <topic physicalName=">" />
        </excludedDestinations>
      </networkConnector>

Other information:

Version 5.9.0
Ubuntu 10.04
java version "1.6.0_35"
Java(TM) SE Runtime Environment (build 1.6.0_35-b10)
Java HotSpot(TM) 64-Bit Server VM (build 20.10-b01, mixed mode)


Any help or advice is very much appreciated



--
View this message in context: http://activemq.2283324.n4.nabble.com/TCP-send-and-receive-buffers-filling-preventing-openwire-communication-tp4676493.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: TCP send and receive buffers filling preventing openwire communication

Posted by Gary Tully <ga...@gmail.com>.
thread dumps of the brokers may help, for some reason a topic send is
blocking when being forwarded, and there is no discarding, so there is
an eventual push back on the producer.
A thread dump of the receiving broker, Broker A, may point to what it is doing.

On 17 January 2014 22:03, oholiab <oh...@gmail.com> wrote:
> Sorry, forgot to add logfile stuff...
>
> These hangups are often accompanied or preceded by log entries like:
>
> 2014-01-16 17:45:02,816 | WARN  | TopicSubscription:
> consumer=B->A-35971-1389919240382-4:3:1:1, destinations=23, dispatched=1000,
> delivered=0, matched=1001, discarded=0: has twice its prefetch limit
> pending, without an ack; it appears to be slow |
> org.apache.activemq.broker.region.TopicSubscription | ActiveMQ NIO Worker 94
>
> But the network connectors always continue to appear open in jconsole
>
> I'm suspecting this has something to do with the buffers filling up and not
> having space for an ack to be sent before the message can be consumed but
> it's only a theory
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/TCP-send-and-receive-buffers-filling-preventing-openwire-communication-tp4676493p4676495.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.



-- 
http://redhat.com
http://blog.garytully.com

Re: TCP send and receive buffers filling preventing openwire communication

Posted by oholiab <oh...@gmail.com>.
Sorry, forgot to add logfile stuff...

These hangups are often accompanied or preceded by log entries like:

2014-01-16 17:45:02,816 | WARN  | TopicSubscription:
consumer=B->A-35971-1389919240382-4:3:1:1, destinations=23, dispatched=1000,
delivered=0, matched=1001, discarded=0: has twice its prefetch limit
pending, without an ack; it appears to be slow |
org.apache.activemq.broker.region.TopicSubscription | ActiveMQ NIO Worker 94

But the network connectors always continue to appear open in jconsole

I'm suspecting this has something to do with the buffers filling up and not
having space for an ack to be sent before the message can be consumed but
it's only a theory



--
View this message in context: http://activemq.2283324.n4.nabble.com/TCP-send-and-receive-buffers-filling-preventing-openwire-communication-tp4676493p4676495.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.