durable subscribers causing message flow to halt

["michael.hart" <mi...@arcticwolf.com>]

We are running ActiveMQ 5.10.0 with a network of brokers. For one topic
called 'results' we have one subscriber running on a single broker 'X' and
many (9 in staging, 24 in production and growing) brokers (let's call them
sensors) connected in with full duplex connections, each broker with a
producer connected and sending messages to that topic. Consumers and
producers are all connected using stomp. brokers are connected using
openwire. 

There are other queues and topics on the brokers, but let's focus on just
this one.

Until yesterday, we were using non-durable consumers, and non-persistent
messaging. Yesterday we enabled durable consumers by setting the client-id
on the CONNECT header, and setting activemq.subscriptionName on the
SUBSCRIBE header. At first things seem to work, but very quickly the brokers
on the sensors stopped forwarding messages and went into producer flow
control. 

Under normal runtime, when I run "/opt/activemq/bin/activemq-admin query
-QTopic=sensor-results" I should see one network connector section per
sensor broker. However once we enabled durable subscribers, only a few of
the sensor brokers would show up, even though they were actually connected
(tcp connection was active). No amount of restarts of anything fixed it. The
number of sensor brokers connected/subscribed was transient, between 2 and 8
would be connected, usually around 4.

Then it got worse. We rolled back our code change and went back to
non-durable subscribers, but things did not get better. We had
offlineDurableSubscriberTimeout and offlineDurableSubscriberTaskSchedule set
to one minute, but the durable subscritions did not clear out. We eventually
shutdown the entire network of brokers, deleted the kahadb directory on
every broker, and restarted everything.

It appears to us that the subscription from single broker 'X' did not make
it to the sensor brokers, or only did sometimes, but then refused to clear
out. We don't know how to further debug this, and would appreciate any help
and suggestions.

thanks
mike



--
View this message in context: http://activemq.2283324.n4.nabble.com/durable-subscribers-causing-message-flow-to-halt-tp4685694.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: durable subscribers causing message flow to halt

["michael.hart" <mi...@arcticwolf.com>]

Thanks Tim. There is no consistency, it appears to be somewhat random. 

I can't attach a debugger unfortunately, I'll have to setup a test network
of sorts to try to reproduce it. :(

mike



--
View this message in context: http://activemq.2283324.n4.nabble.com/durable-subscribers-causing-message-flow-to-halt-tp4685694p4685697.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: durable subscribers causing message flow to halt

[Tim Bain <tb...@alumni.duke.edu>]

Is there any consistency in which sensor brokers succeed and which fail, or
is it random every time?

If you're able to download the source code for your version of ActiveMQ and
attach a debugger to one of your sensor brokers, you could set breakpoints
and watch the subscription request from broker X be processed.
Topic.addSubscription() would be a good place for your first breakpoint
when you're failing to add the subscription, and TopicRegion.doCleanup()
would be a good place for a breakpoint when you're failing to close offline
durable subscribers.

Tim

On Thu, Sep 18, 2014 at 8:42 AM, michael.hart <mi...@arcticwolf.com>
wrote:

> We are running ActiveMQ 5.10.0 with a network of brokers. For one topic
> called 'results' we have one subscriber running on a single broker 'X' and
> many (9 in staging, 24 in production and growing) brokers (let's call them
> sensors) connected in with full duplex connections, each broker with a
> producer connected and sending messages to that topic. Consumers and
> producers are all connected using stomp. brokers are connected using
> openwire.
>
> There are other queues and topics on the brokers, but let's focus on just
> this one.
>
> Until yesterday, we were using non-durable consumers, and non-persistent
> messaging. Yesterday we enabled durable consumers by setting the client-id
> on the CONNECT header, and setting activemq.subscriptionName on the
> SUBSCRIBE header. At first things seem to work, but very quickly the
> brokers
> on the sensors stopped forwarding messages and went into producer flow
> control.
>
> Under normal runtime, when I run "/opt/activemq/bin/activemq-admin query
> -QTopic=sensor-results" I should see one network connector section per
> sensor broker. However once we enabled durable subscribers, only a few of
> the sensor brokers would show up, even though they were actually connected
> (tcp connection was active). No amount of restarts of anything fixed it.
> The
> number of sensor brokers connected/subscribed was transient, between 2 and
> 8
> would be connected, usually around 4.
>
> Then it got worse. We rolled back our code change and went back to
> non-durable subscribers, but things did not get better. We had
> offlineDurableSubscriberTimeout and offlineDurableSubscriberTaskSchedule
> set
> to one minute, but the durable subscritions did not clear out. We
> eventually
> shutdown the entire network of brokers, deleted the kahadb directory on
> every broker, and restarted everything.
>
> It appears to us that the subscription from single broker 'X' did not make
> it to the sensor brokers, or only did sometimes, but then refused to clear
> out. We don't know how to further debug this, and would appreciate any help
> and suggestions.
>
> thanks
> mike
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/durable-subscribers-causing-message-flow-to-halt-tp4685694.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>