You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by dcheckoway <dc...@gmail.com> on 2011/04/15 08:13:53 UTC
Upstream throttling and backlog
I'm looking for help "decoupling" producers from consumers. In a nutshell,
this is my setup:
producer --> embedded broker
embedded broker --> (camel route) --> external broker
external broker --> consumer
As of right now, if I shut down the consumer, the queue in the external
broker grows -- that's fully expected. But what's unexpected is that
everything upstream slows down, and I end up with a queue backlog in the
embedded broker. I have producerFlowControl="false" across the board. I'm
having trouble understanding why this upstream throttling and backlog is
happening.
My goal is...if the consumer is the only piece that's down (or slow), I
still want messages to flow at full speed all the way to the external
broker, where they're welcome to queue up. I don't want anything producer
flow controlled...ever.
Is this possible with ActiveMQ?
-----
Dan Checkoway
dcheckoway gmail com
--
View this message in context: http://activemq.2283324.n4.nabble.com/Upstream-throttling-and-backlog-tp3451330p3451330.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Upstream throttling and backlog
Posted by dcheckoway <dc...@gmail.com>.
Thanks for the reply, Gary. We're not doing anything specific
persistence-wise with the messages themselves. Broker-wise, persistence is
enabled on the external broker, but persistence is disabled on the embedded
broker.
I believe this problem occurs regardless of persistence being
enabled/disabled, but I may be missing something.
On Thu, Aug 25, 2011 at 5:22 AM, Gary Tully [via ActiveMQ] <
ml-node+3767760-1228269095-140184@n4.nabble.com> wrote:
> @dan,
> Are you using persistent messages?
>
> If yes, typically dispatching is faster than persisting messages and
> when the consumers are fast we can by pass storing and syncing to disk
> if the consumer acks before the disk write has started. As a result,
> through put can be very high.
>
> When we persist, the producer typically waits for the disk sync. Using
> alwaysSendAsync=true avoids that.
>
> You should use the default store cursor, which will just stop caching
> messages when its memory limit is reached.
>
> Note, that producer flow control false does not stop memory limits, it
> just moves the wait to the broker rather than the producer (client) so
> watch out for log messages that indicate a producer is waiting for
> space.
>
> If you are using non persistent messages, ensure that the temp store
> limit is not being reached.
>
> On 15 April 2011 07:13, dcheckoway <[hidden email]<http://user/SendEmail.jtp?type=node&node=3767760&i=0>>
> wrote:
>
> > I'm looking for help "decoupling" producers from consumers. In a
> nutshell,
> > this is my setup:
> >
> > producer --> embedded broker
> >
> > embedded broker --> (camel route) --> external broker
> >
> > external broker --> consumer
> >
> > As of right now, if I shut down the consumer, the queue in the external
> > broker grows -- that's fully expected. But what's unexpected is that
> > everything upstream slows down, and I end up with a queue backlog in the
> > embedded broker. I have producerFlowControl="false" across the board.
> I'm
> > having trouble understanding why this upstream throttling and backlog is
> > happening.
> >
> > My goal is...if the consumer is the only piece that's down (or slow), I
> > still want messages to flow at full speed all the way to the external
> > broker, where they're welcome to queue up. I don't want anything
> producer
> > flow controlled...ever.
> >
> > Is this possible with ActiveMQ?
> >
> > -----
> > Dan Checkoway
> > dcheckoway gmail com
> > --
> > View this message in context:
> http://activemq.2283324.n4.nabble.com/Upstream-throttling-and-backlog-tp3451330p3451330.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
>
>
> --
> http://fusesource.com
> http://blog.garytully.com
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://activemq.2283324.n4.nabble.com/Upstream-throttling-and-backlog-tp3451330p3767760.html
> To unsubscribe from Upstream throttling and backlog, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3451330&code=ZGNoZWNrb3dheUBnbWFpbC5jb218MzQ1MTMzMHw3MDc4NzEwMTU=>.
>
>
-----
Dan Checkoway
dcheckoway gmail com
--
View this message in context: http://activemq.2283324.n4.nabble.com/Upstream-throttling-and-backlog-tp3451330p3768393.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Upstream throttling and backlog
Posted by praveen27 <pr...@gmail.com>.
Thanks gary,
In our case we are using kaha-db here is the exact config in broker.xml file
<persistenceAdapter>
<kahaDB directory="${activemq.base}/data/kahadb"
indexWriteBatchSize="10000"
indexCacheSize="100000"
enableIndexWriteAsync="true"
journalMaxFileLength="128mb"
enableJournalDiskSyncs="false"
checkpointInterval="1000"
maxAsyncJobs="50000" />
</persistenceAdapter>
We would have expected a drop in throughput may be 30-40%, but not 90%. The
file system that the kaha db directory is pointing to is fairly beefy raid10
local disks.
What we are looking for is a system that achieves high throughput rates
especially under backlog situations. Appreciate if you can provide some
inputs on this.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Upstream-throttling-and-backlog-tp3451330p3770096.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Upstream throttling and backlog
Posted by Gary Tully <ga...@gmail.com>.
@dan,
Are you using persistent messages?
If yes, typically dispatching is faster than persisting messages and
when the consumers are fast we can by pass storing and syncing to disk
if the consumer acks before the disk write has started. As a result,
through put can be very high.
When we persist, the producer typically waits for the disk sync. Using
alwaysSendAsync=true avoids that.
You should use the default store cursor, which will just stop caching
messages when its memory limit is reached.
Note, that producer flow control false does not stop memory limits, it
just moves the wait to the broker rather than the producer (client) so
watch out for log messages that indicate a producer is waiting for
space.
If you are using non persistent messages, ensure that the temp store
limit is not being reached.
On 15 April 2011 07:13, dcheckoway <dc...@gmail.com> wrote:
> I'm looking for help "decoupling" producers from consumers. In a nutshell,
> this is my setup:
>
> producer --> embedded broker
>
> embedded broker --> (camel route) --> external broker
>
> external broker --> consumer
>
> As of right now, if I shut down the consumer, the queue in the external
> broker grows -- that's fully expected. But what's unexpected is that
> everything upstream slows down, and I end up with a queue backlog in the
> embedded broker. I have producerFlowControl="false" across the board. I'm
> having trouble understanding why this upstream throttling and backlog is
> happening.
>
> My goal is...if the consumer is the only piece that's down (or slow), I
> still want messages to flow at full speed all the way to the external
> broker, where they're welcome to queue up. I don't want anything producer
> flow controlled...ever.
>
> Is this possible with ActiveMQ?
>
> -----
> Dan Checkoway
> dcheckoway gmail com
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Upstream-throttling-and-backlog-tp3451330p3451330.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
http://fusesource.com
http://blog.garytully.com
Re: Upstream throttling and backlog
Posted by praveen27 <pr...@gmail.com>.
I have a similar similar setup (flow control turned off)
producer ==> EmbeddedConsumer ==> consumer. Producers producing messages to
kaha persisted topics
In the test setup we have
10 producers per virtual-topic with 10 topics. There are 10 consumers for
each virtual topic
i.e., total of 100 producers, 100 consumers & 10 topics.
*Issue 1: *
When both producer and consumers are up and running we get a very good
throughput from the setup. When consumers don't consume the messages the
producer throughput *drops by 90%*
*Issue 2:*
We see ~750 threads in the system. 100 threads for producers, 100 threads
for consumers is what we can account for. Not able understand why there are
~550 threads used by ActiveMQ. Even if we account 1 thread each for producer
& consumer we account for 200 off of 550.
Appreciate we can get a response explain this backlog issue and high
resource usage.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Upstream-throttling-and-backlog-tp3451330p3767280.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.