Re: Master/Slave: Message Hoarding

[Les Hazlewood <lh...@apache.org>]

bump.  Could someone take a peek at this again and see if they have any
ideas?

(we tried prefetch size = 0 on the client connection URL, and that didn't
change anything)

Thanks!

Les

On Mon, Jul 22, 2013 at 7:10 PM, stormtrooper <he...@stormpath.com> wrote:

> Hello all,
>
> Using ActiveMQ 5.8.0 we are seeing a message hoarding issue that only crops
> up when we use a Master/Slave configuration.
>
> A single producer sends persistent, never expiring messages to two discrete
> Queues in the Broker in spurts. Each queue has a single polling consumer
> (so
> we when we look in the Admin Web Console we see two consumers attached to
> each queue: One is our consumer, the other is automatically created for the
> Slave), polling for max X messages every Y seconds, no selectors.
>
> When we have just a plain (not a master) instance of the broker +
> jdbc/mySQL, everything works as expected: messages are sent to the queues
> and delivered in order, 100% throughput.
>
> When we have the Master/Slave setup using JDBC
> (http://activemq.apache.org/jdbc-master-slave.html) and mySQL, every spurt
> of messages (well under the max # messages/poll) results in only some being
> delivered while others are hoarded in the database.
> Example: First spurt sends 10 messages, only 7 get delivered to the
> consumer. The other 3 are stored in the database (ACTIVEMQ_MSGS table).
> When
> the next spurt comes, let's say 13 messages, only 8 of those get delivered
> and 5 of them are stored in the database; so now the database has 8 total
> hoarded messages. Every successive spurt results in only partial sets
> delivered and more messages being hoarded.
> This is happening with both queues.
>
> Some possible points of interest:
> 1) The messages selected for hoarding appear random. They are not
> necessarily consecutive, they are not necessarily the first messages, nor
> necessarily last messages of each spurt.
> 2) If we restart the consumers, nothing happens; The hoarded messages
> remain
> in the DB, undelivered.
> 3) If we restart the broker, the hoarded messages ARE delivered (as
> expected
> since they are persistent and never expire, which ostensibly is the point
> of
> those options).
> 4) We get the same hoarding results whether our producer/consumer connects
> to the master broker directly using tcp, or using the failover protocol
> with
> both master and slave listed, or using the failover protocol with both
> master and slave listed and the randomize=false param. (This is not
> particularly surprising, just included for completeness).
>
> Our activemq.xml configuration: http://pastebin.com/Y87hZuh5
>
> We've been trying every permutation of configuration we can think of to see
> where the culprit may lay (even swapping in KahaDB for mySQL => whole other
> can of worms, but then we saw it wasn't supported anyway:
> http://activemq.apache.org/kahadb-master-slave.html, a story for another
> day).
>
> Does anyone have any suggestions/pointers of where we should look next? How
> to get us back on track?
> It would be greatly appreciated,
> -h
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Master-Slave-Message-Hoarding-tp4669591.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: Master/Slave: Message Hoarding

[Christian Posta <ch...@gmail.com>]

Thanks for taking the time to post your findings!



On Thu, Aug 29, 2013 at 6:05 PM, stormtrooper <he...@stormpath.com> wrote:

> Hello,
>
> Well, we've figured this out so I'll record it here for posterity.
>
> Our master/slave setup was incorrectly configured.
>
> Originally our architecture was going to have two clusters that talked to
> each other. In our phased development approach we scaled back to work with
> just one initially. However, the Network Connector config node was
> inadvertently/misunderstood and left in:
>
> <networkConnector
> uri="masterslave:(tcp://localhost:61616,tcp://[slavehost]:61616)"/>
>
> What caused us to zero in on this was that every time we connected to one
> of
> our queues, in the Admin Console we would see our consumers in the "Active
> Consumers" page for the queue, but ALSO an additional consumer:
> "NC_[our_master_brokerName]_inbound_[our_master_brokerName]"
> This consumer would also disappear when our consumers disconnected from the
> queue.
>
> Behavior would manifest as such:
> For a queue:
> * Enqueue is always correct, it's the total number sent.
> * When the consumers are done consuming what they will be sent, the
> inflights are 0.
> * The hoarded difference always corresponds to the # of rows in the
> ACTIVEMQ_MSGS table that won't budge.
> * Dequeue is also always correct, it's the total number received by the
> consumers BUT that INCLUDES the NC consumers.
>
> Example: We send 150 messages to the queue.
> * Our client consumer: Enqueue/Dequeue/Dispatch are all 50 each.
> * The NC consumer: Enqueue/Dequeue/Dispatch are all 50 each.
> * Pending for queue 50 = 50 rows in the ACTIVEMQ_MSGS db
>
> Our client and the NC consumer would alternate consuming messages until
> 100,
> and then the last contiguous block of 50 would be hoarded.
>
> Oddly enough, a couple of times we left the cluster up, but inactive, all
> night. And when we returned to run tests, there would be no creation of NC
> consumers and no hoarding. Everything worked fine.
> Our tip off was that in this scenario, failover from the master to slave
> worked fine, so it was clear that the NC was not needed within a single
> master/slave cluster. [Not sure why this is?]
>
> But without fail, if we restarted the cluster and started using it
> relatively soon, we would see the NC consumers and the hoarding.
>
> Whoosh.
> Hopefully this might help someone else save some time,
> -h
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Master-Slave-Message-Hoarding-tp4669591p4670940.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

Re: Master/Slave: Message Hoarding

[stormtrooper <he...@stormpath.com>]

Hello,

Well, we've figured this out so I'll record it here for posterity.

Our master/slave setup was incorrectly configured. 

Originally our architecture was going to have two clusters that talked to
each other. In our phased development approach we scaled back to work with
just one initially. However, the Network Connector config node was
inadvertently/misunderstood and left in:

<networkConnector
uri="masterslave:(tcp://localhost:61616,tcp://[slavehost]:61616)"/>

What caused us to zero in on this was that every time we connected to one of
our queues, in the Admin Console we would see our consumers in the "Active
Consumers" page for the queue, but ALSO an additional consumer:
"NC_[our_master_brokerName]_inbound_[our_master_brokerName]"
This consumer would also disappear when our consumers disconnected from the
queue.

Behavior would manifest as such:
For a queue:
* Enqueue is always correct, it's the total number sent. 
* When the consumers are done consuming what they will be sent, the
inflights are 0. 
* The hoarded difference always corresponds to the # of rows in the
ACTIVEMQ_MSGS table that won't budge.
* Dequeue is also always correct, it's the total number received by the
consumers BUT that INCLUDES the NC consumers.

Example: We send 150 messages to the queue. 
* Our client consumer: Enqueue/Dequeue/Dispatch are all 50 each. 
* The NC consumer: Enqueue/Dequeue/Dispatch are all 50 each.
* Pending for queue 50 = 50 rows in the ACTIVEMQ_MSGS db

Our client and the NC consumer would alternate consuming messages until 100,
and then the last contiguous block of 50 would be hoarded.

Oddly enough, a couple of times we left the cluster up, but inactive, all
night. And when we returned to run tests, there would be no creation of NC
consumers and no hoarding. Everything worked fine.
Our tip off was that in this scenario, failover from the master to slave
worked fine, so it was clear that the NC was not needed within a single
master/slave cluster. [Not sure why this is?]

But without fail, if we restarted the cluster and started using it
relatively soon, we would see the NC consumers and the hoarding.

Whoosh.
Hopefully this might help someone else save some time,
-h




--
View this message in context: http://activemq.2283324.n4.nabble.com/Master-Slave-Message-Hoarding-tp4669591p4670940.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Master/Slave: Message Hoarding

[Christian Posta <ch...@gmail.com>]

That's quite odd.
Can you reproduce this on KahaDB? ie, kahadb is supported for shared master
slave (http://activemq.apache.org/shared-file-system-master-slave.html) and
it might be easier to debug (without the dependency on the DB) if you can
recreate with KahaDB...

What do the broker stats say as far as dispatched, inflight, enqueue,
dequeue?




On Wed, Aug 28, 2013 at 2:10 PM, Les Hazlewood <lh...@apache.org>wrote:

> bump.  Could someone take a peek at this again and see if they have any
> ideas?
>
> (we tried prefetch size = 0 on the client connection URL, and that didn't
> change anything)
>
> Thanks!
>
> Les
>
> On Mon, Jul 22, 2013 at 7:10 PM, stormtrooper <he...@stormpath.com>
> wrote:
>
> > Hello all,
> >
> > Using ActiveMQ 5.8.0 we are seeing a message hoarding issue that only
> crops
> > up when we use a Master/Slave configuration.
> >
> > A single producer sends persistent, never expiring messages to two
> discrete
> > Queues in the Broker in spurts. Each queue has a single polling consumer
> > (so
> > we when we look in the Admin Web Console we see two consumers attached to
> > each queue: One is our consumer, the other is automatically created for
> the
> > Slave), polling for max X messages every Y seconds, no selectors.
> >
> > When we have just a plain (not a master) instance of the broker +
> > jdbc/mySQL, everything works as expected: messages are sent to the queues
> > and delivered in order, 100% throughput.
> >
> > When we have the Master/Slave setup using JDBC
> > (http://activemq.apache.org/jdbc-master-slave.html) and mySQL, every
> spurt
> > of messages (well under the max # messages/poll) results in only some
> being
> > delivered while others are hoarded in the database.
> > Example: First spurt sends 10 messages, only 7 get delivered to the
> > consumer. The other 3 are stored in the database (ACTIVEMQ_MSGS table).
> > When
> > the next spurt comes, let's say 13 messages, only 8 of those get
> delivered
> > and 5 of them are stored in the database; so now the database has 8 total
> > hoarded messages. Every successive spurt results in only partial sets
> > delivered and more messages being hoarded.
> > This is happening with both queues.
> >
> > Some possible points of interest:
> > 1) The messages selected for hoarding appear random. They are not
> > necessarily consecutive, they are not necessarily the first messages, nor
> > necessarily last messages of each spurt.
> > 2) If we restart the consumers, nothing happens; The hoarded messages
> > remain
> > in the DB, undelivered.
> > 3) If we restart the broker, the hoarded messages ARE delivered (as
> > expected
> > since they are persistent and never expire, which ostensibly is the point
> > of
> > those options).
> > 4) We get the same hoarding results whether our producer/consumer
> connects
> > to the master broker directly using tcp, or using the failover protocol
> > with
> > both master and slave listed, or using the failover protocol with both
> > master and slave listed and the randomize=false param. (This is not
> > particularly surprising, just included for completeness).
> >
> > Our activemq.xml configuration: http://pastebin.com/Y87hZuh5
> >
> > We've been trying every permutation of configuration we can think of to
> see
> > where the culprit may lay (even swapping in KahaDB for mySQL => whole
> other
> > can of worms, but then we saw it wasn't supported anyway:
> > http://activemq.apache.org/kahadb-master-slave.html, a story for another
> > day).
> >
> > Does anyone have any suggestions/pointers of where we should look next?
> How
> > to get us back on track?
> > It would be greatly appreciated,
> > -h
> >
> >
> >
> >
> > --
> > View this message in context:
> >
> http://activemq.2283324.n4.nabble.com/Master-Slave-Message-Hoarding-tp4669591.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>



-- 
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta