Message queue is filled up.

[Kuppe <ku...@360t.com>]

I am working on a use case where i am distributing fast moving market data
from a many different market data sources to many clients each receiving a
client-specific stream of market data from each source. Typically the
clients are connected to my broker over the internet, therefore can be
considered to be slow consumers.

The objective of the configuration is to provide high throughput on the
server side where registration/deregistration take place to the different
sources from a single registration coordinator, completely decoupled
delivery of updates from the sources to the specific topics of consumers,
and to only deliver to the consumers the latest market data updates in the
case that slow consumers cannot handle the number of updates delivered by
the sources. The configuration should also be able to handle a large number
of clients.

I am using ActiveMQ 4.0.1.

I use a queue for registration/deregistration of market data and topics to
distribute the market data updates, although there are usually only a small
number of consumers connected to each topic as the market data updates are
in fact receiver specific.

I have configured all slow consumers with a prefetch limit of 100 and have
configured a policy entry for the market data update topic with an
oldestMessageEvictionStrateygy and a prefetchRatePendingMessageLimitStrategy
with a multiplier of 1.

I have not enabled persistence as the market data updates are highly
volatile therefore it is more important that the latency of each market
update is low rather than being made persistent. (I think this is perhaps
the wrong way to think of persistence in this case, but i will leave that to
you to explain...)

The problem is that i constantly receive the message "Message queue is
filled up". Unfortunately i do not get any further information from the log
message that tells me where to look in the source for further information,
and i cannot find any text in the source that relates to the log message. So
i ask you how i can a) resolve this problem? and b) further improve my
current configuration to realize my stated objectives?

If you need further information regarding the configuration please let me
know.
-- 
View this message in context: http://www.nabble.com/Message-queue-is-filled-up.-tf1898400.html#a5193299
Sent from the ActiveMQ - User forum at Nabble.com.

Re: Message queue is filled up.

[Hiram Chirino <hi...@hiramchirino.com>]

On 7/6/06, James Strachan <ja...@gmail.com> wrote:
>
> On 7/7/06, Kuppe <ku...@360t.com> wrote:
> >
> > My sincere apologies, it seems that the "Message queue is filled up"
> error
> > message is not coming from ActiveMQ at all. It is coming from my own
> > framework that is not able to process the number of messages being
> delivered
> > by ActiveMQ:)
>
> LOL! :) Thanks for letting us know - phew, there's not a bug in our
> error message reporting :)
>
>
> > Due to limitations in our previous messaging solution, I have an
> incoming
> > and outgoing queue for receiving and sending messages implemented in my
> > framework. As i am now using async dispatch and async sending, these
> queues
> > are somehow redundant. At the same time, the throughput of my framework
> also
> > seems to be somewhat limited with all the context switching and
> processing.
>
> :)
>
> > Accordingly I would like to reduce this complexity by removing my
> framework
> > queues and rely entirely on the async dispatch/send that is implemented
> in
> > ActiveMQ. Is this an approach that you would recommend?
>
> Definitely. The sync v async configuration is something you may wish
> to change over time (on a per connection/producer/consumer basis).
> e.g. for broker dispatching to consumers, doing it synchronous is
> often a bit faster as it reduces context switching, though it
> increases the chance of a slow consumer blocking other consumers for a
> little while. Leaving all this stuff to the messaging system makes it
> easier for you to tweak things (often via a URI) and remove some of
> the layers in your code.
>
>
> > At the same time, my
> > framework does offer me control as to how many threads are processing
> the
> > messages in these two receive and send queues. If I am to replace my
> > framework queues with the async functionality of ActiveMQ, i would like
> to
> > know the answer to the following two related questions;
> >
> > Q1: When using async send, is this doing the message sending in another
> > thread context?
>
> Yes



Actually no.  Async send only a single thread is used and it block only
until the message is serialized and put on the wire.  Disabling Async send
cause the send to block until the message is serialized on the wire and send
ack is received from the broker.


>  If so, is there any configuration options for limiting the
> > number of threads processing the sending of messages? What is the
> default
> > algorithm for the number of sending threads - one per session,
> producer???
>
> We use thread pools for pretty much everything now to minimise the
> number of actual threads used; we've done quite a bit of tuning in
> that regard in 4.x. (In 3.x there tended to be lots of threads created
> in a transport, session etc).


And the use of thread pools vs. using dedicated threads can be
enabled/disabled using a System property.  On the broker, out of the box we
set set it not to use ThreadPools since thread pools can add a little more
context switching overhead than if you use dedicated threads at the expense
of using more memory due to more active thread stacks.  That system property
is set like: -Dorg.apache.activemq.UseDedicatedTaskRunner=true


Whether you use sync or async in the JMS client there is typically
> (for tcp:// and vm:// at least) a thread sending and a thread
> receiving messages. All these threads are doing is streaming messages
> onto/off of a socket so there is no real reason to pool them (e.g.
> having 2 threads writing would generally be slower).
>
> Async send just means the thread doing the producer.send() doesn't
> block waiting for a response from the broker; so there's no real
> difference from the number of threads used etc; it just removes the
> latency in the send() thread.
>
>
> > Q2: In line with Q1, if i am using async dispatch, is this doing the
> message
> > dispatching in another thread context?
>
> Yes. e.g. the default in non durable topics is that the thread
> processing incoming messages from a producer will dispatch the message
> to each consumers socket. With async dispatch a thread pool is used to
> configure this.


agreed.

> If so, is there any configuration
> > options for limiting the number of threads processing the dispatching of
> > messages?
>
> There could well be - not sure off the top of my head :)


Not at the current time.

> What is the default algorithm for the number of dispatchin threads
> > - one per session, consumer???
>
> This is a broker side thing so its purely to do with how many
> consumers there are; each async dispatch consumer is a separate task
> which is added to the thread pool.


Actually on the broker side there is 1 thread per JMS connection doing the
async dispatch.

Its on the client side that all of a sessions's messages are
> dispatched to consumers in same thread (but since we use a thread pool
> if you have many sessions we may not use that many threads).



Yep. on the client side each session gets a dedicated thread too.  But these
are thread pooled by default.  But there is an option to disable
asyncSessionDispatch so that the connection thread dispatches directly to
the consumer.

--
>
> James
> -------
> http://radio.weblogs.com/0112098/
>



-- 
Regards,
Hiram

Blog: http://hiramchirino.com

Re: Message queue is filled up.

[James Strachan <ja...@gmail.com>]

On 7/7/06, Kuppe <ku...@360t.com> wrote:
>
> My sincere apologies, it seems that the "Message queue is filled up" error
> message is not coming from ActiveMQ at all. It is coming from my own
> framework that is not able to process the number of messages being delivered
> by ActiveMQ:)

LOL! :) Thanks for letting us know - phew, there's not a bug in our
error message reporting :)


> Due to limitations in our previous messaging solution, I have an incoming
> and outgoing queue for receiving and sending messages implemented in my
> framework. As i am now using async dispatch and async sending, these queues
> are somehow redundant. At the same time, the throughput of my framework also
> seems to be somewhat limited with all the context switching and processing.

:)

> Accordingly I would like to reduce this complexity by removing my framework
> queues and rely entirely on the async dispatch/send that is implemented in
> ActiveMQ. Is this an approach that you would recommend?

Definitely. The sync v async configuration is something you may wish
to change over time (on a per connection/producer/consumer basis).
e.g. for broker dispatching to consumers, doing it synchronous is
often a bit faster as it reduces context switching, though it
increases the chance of a slow consumer blocking other consumers for a
little while. Leaving all this stuff to the messaging system makes it
easier for you to tweak things (often via a URI) and remove some of
the layers in your code.


> At the same time, my
> framework does offer me control as to how many threads are processing the
> messages in these two receive and send queues. If I am to replace my
> framework queues with the async functionality of ActiveMQ, i would like to
> know the answer to the following two related questions;
>
> Q1: When using async send, is this doing the message sending in another
> thread context?

Yes

>  If so, is there any configuration options for limiting the
> number of threads processing the sending of messages? What is the default
> algorithm for the number of sending threads - one per session, producer???

We use thread pools for pretty much everything now to minimise the
number of actual threads used; we've done quite a bit of tuning in
that regard in 4.x. (In 3.x there tended to be lots of threads created
in a transport, session etc).

Whether you use sync or async in the JMS client there is typically
(for tcp:// and vm:// at least) a thread sending and a thread
receiving messages. All these threads are doing is streaming messages
onto/off of a socket so there is no real reason to pool them (e.g.
having 2 threads writing would generally be slower).

Async send just means the thread doing the producer.send() doesn't
block waiting for a response from the broker; so there's no real
difference from the number of threads used etc; it just removes the
latency in the send() thread.


> Q2: In line with Q1, if i am using async dispatch, is this doing the message
> dispatching in another thread context?

Yes. e.g. the default in non durable topics is that the thread
processing incoming messages from a producer will dispatch the message
to each consumers socket. With async dispatch a thread pool is used to
configure this.


> If so, is there any configuration
> options for limiting the number of threads processing the dispatching of
> messages?

There could well be - not sure off the top of my head :)

> What is the default algorithm for the number of dispatchin threads
> - one per session, consumer???

This is a broker side thing so its purely to do with how many
consumers there are; each async dispatch consumer is a separate task
which is added to the thread pool.

Its on the client side that all of a sessions's messages are
dispatched to consumers in same thread (but since we use a thread pool
if you have many sessions we may not use that many threads).
-- 

James
-------
http://radio.weblogs.com/0112098/

Re: Message queue is filled up.

[Kuppe <ku...@360t.com>]

My sincere apologies, it seems that the "Message queue is filled up" error
message is not coming from ActiveMQ at all. It is coming from my own
framework that is not able to process the number of messages being delivered
by ActiveMQ:)

Due to limitations in our previous messaging solution, I have an incoming
and outgoing queue for receiving and sending messages implemented in my
framework. As i am now using async dispatch and async sending, these queues
are somehow redundant. At the same time, the throughput of my framework also
seems to be somewhat limited with all the context switching and processing. 

Accordingly I would like to reduce this complexity by removing my framework
queues and rely entirely on the async dispatch/send that is implemented in
ActiveMQ. Is this an approach that you would recommend? At the same time, my
framework does offer me control as to how many threads are processing the
messages in these two receive and send queues. If I am to replace my
framework queues with the async functionality of ActiveMQ, i would like to
know the answer to the following two related questions;

Q1: When using async send, is this doing the message sending in another
thread context? If so, is there any configuration options for limiting the
number of threads processing the sending of messages? What is the default
algorithm for the number of sending threads - one per session, producer???

Q2: In line with Q1, if i am using async dispatch, is this doing the message
dispatching in another thread context? If so, is there any configuration
options for limiting the number of threads processing the dispatching of
messages? What is the default algorithm for the number of dispatchin threads
- one per session, consumer???

Thanks in advance.
-- 
View this message in context: http://www.nabble.com/Message-queue-is-filled-up.-tf1898400.html#a5209682
Sent from the ActiveMQ - User forum at Nabble.com.

Re: Message queue is filled up.

[Kuppe <ku...@360t.com>]

Ok, let me investigate the queue sizes and come back to you... Perhaps i am
overlooking something else... I do have a jmx console configured so i should
be able to find this out by analyzing the destinations created on the
broker. At the same time, it would of course be much easier if the warning
message did provide some information regarding the queue that is in fact
filling up...

I am assuming the the configuration that you refer to for the usage manager
is in the element

    <memoryManager>  
        <usageManager id="memory-manager" limit="1048576"/>
    </memoryManager>

Is this correct? Just out of curiosity, what is the default value that is
causing my queue size error?

At least you give me confidence that the general setup is the right one for
the job:)

Thanks again!
-- 
View this message in context: http://www.nabble.com/Message-queue-is-filled-up.-tf1898400.html#a5195061
Sent from the ActiveMQ - User forum at Nabble.com.

Re: Message queue is filled up.

[James Strachan <ja...@gmail.com>]

On 7/6/06, Kuppe <ku...@360t.com> wrote:
>
> James, thanks for the positive feedback.
>
> Firstly, i think the question as to which queue is actually filling up is
> really the one that is confusing me the most. From your reply below it seems
> that you are suggesting that it is the registration/deregistration queue
> that is full. I was expecting in fact that it would have been the
> "subscription queue" buffer for the client - if there is one - inside the
> broker.

I don't think so; given the prefetch and slow consumer settings you
are using it shouldn't be the topic dispatching thats exhausting RAM


>  It is only on the topic that there is really high volumes of
> messages that could even hit the 300 mark which is being reported...
>
> Secondly, i have no concern whatsoever to use persistent messages for the
> registration/deregistration. The question is whether this is where the limit
> is reached...
>
> Q1: how can i determine which queue is actually filling up?

Use JMX and watch 'em grow?
http://incubator.apache.org/activemq/jmx.html


> How many queues
> are there - one per topic, one per subcriber per topic, one per client
> connection?

There are no queues by default - its your JMS client which creates
them by sending messages to them, so you'd have to look at your JMS
client - I've no idea.


> Q2: how do i configure the RAM UsageManager? What values should I use to
> configure the RAM UsageManager?

Try 100Mb or so of RAM?

Details of how to set the usage manager here...
http://www.activemq.org/site/xml-configuration.html

BTW there is every chance that it is in fact the non-durable topics
are using up most of the available RAM buffer for caching messages &
just a lousy error message is talking about a queue being full :)


> Should this simply be calculated as
> bytesPerMessage * prefetchLimit * (pendingMessageLimitMultiplier + 1) *
> numberOfClients? How exacltly does the UsageManager actually work?

If we're talking queues, so its basically just the amount of RAM that
can be used up by messages. So bytesPerMessage * maximumQueueSize
roughly. But for topics it'd be something like that (though the
chances are that one message is shared by many consumers)


> Q3: do you confirm that i should definitely NOT be using persistent messages
> for the market data updates?

Non-persistent topics work great for things like market data, so I'd
recommend using non-persistent topics.


> How well with this scale?

Very well.


> Must i allocate huge
> amounts of memory to the broker to manage a large number of subscibers?

No not really. With the prefetch & slow consumer handling settings you
mentioned previously, the broker shouldn't use up that much RAM for
the topics. Though I'm still a bit unsure what queues are being used.

-- 

James
-------
http://radio.weblogs.com/0112098/

Re: Message queue is filled up.

[Kuppe <ku...@360t.com>]

James, thanks for the positive feedback.

Firstly, i think the question as to which queue is actually filling up is
really the one that is confusing me the most. From your reply below it seems
that you are suggesting that it is the registration/deregistration queue
that is full. I was expecting in fact that it would have been the
"subscription queue" buffer for the client - if there is one - inside the
broker. It is only on the topic that there is really high volumes of
messages that could even hit the 300 mark which is being reported...

Secondly, i have no concern whatsoever to use persistent messages for the
registration/deregistration. The question is whether this is where the limit
is reached...

Q1: how can i determine which queue is actually filling up? How many queues
are there - one per topic, one per subcriber per topic, one per client
connection?

Q2: how do i configure the RAM UsageManager? What values should I use to
configure the RAM UsageManager? Should this simply be calculated as
bytesPerMessage * prefetchLimit * (pendingMessageLimitMultiplier + 1) *
numberOfClients? How exacltly does the UsageManager actually work?

Q3: do you confirm that i should definitely NOT be using persistent messages
for the market data updates? How well with this scale? Must i allocate huge
amounts of memory to the broker to manage a large number of subscibers?

Again, thanks for the continued support!!!



-- 
View this message in context: http://www.nabble.com/Message-queue-is-filled-up.-tf1898400.html#a5194586
Sent from the ActiveMQ - User forum at Nabble.com.

Re: Message queue is filled up.

[James Strachan <ja...@gmail.com>]

The configuration you're using sounds perfect for the topic
configuraiton of market data. The error you are getting sounds like
its related to the queue filling up. For non-persistent queues your
only option is to increase the RAM usage manager for the queue to be
able to deal with you rmaximum  peak.

Just out of interest what is the purpose of the
registration/deregistration queue and why is it filling up? One quick
workaround is to just make it persistent - or use a faster consumer.

On 7/6/06, Kuppe <ku...@360t.com> wrote:
>
> I am working on a use case where i am distributing fast moving market data
> from a many different market data sources to many clients each receiving a
> client-specific stream of market data from each source. Typically the
> clients are connected to my broker over the internet, therefore can be
> considered to be slow consumers.
>
> The objective of the configuration is to provide high throughput on the
> server side where registration/deregistration take place to the different
> sources from a single registration coordinator, completely decoupled
> delivery of updates from the sources to the specific topics of consumers,
> and to only deliver to the consumers the latest market data updates in the
> case that slow consumers cannot handle the number of updates delivered by
> the sources. The configuration should also be able to handle a large number
> of clients.
>
> I am using ActiveMQ 4.0.1.
>
> I use a queue for registration/deregistration of market data and topics to
> distribute the market data updates, although there are usually only a small
> number of consumers connected to each topic as the market data updates are
> in fact receiver specific.
>
> I have configured all slow consumers with a prefetch limit of 100 and have
> configured a policy entry for the market data update topic with an
> oldestMessageEvictionStrateygy and a prefetchRatePendingMessageLimitStrategy
> with a multiplier of 1.
>
> I have not enabled persistence as the market data updates are highly
> volatile therefore it is more important that the latency of each market
> update is low rather than being made persistent. (I think this is perhaps
> the wrong way to think of persistence in this case, but i will leave that to
> you to explain...)
>
> The problem is that i constantly receive the message "Message queue is
> filled up". Unfortunately i do not get any further information from the log
> message that tells me where to look in the source for further information,
> and i cannot find any text in the source that relates to the log message. So
> i ask you how i can a) resolve this problem? and b) further improve my
> current configuration to realize my stated objectives?
>
> If you need further information regarding the configuration please let me
> know.
> --
> View this message in context: http://www.nabble.com/Message-queue-is-filled-up.-tf1898400.html#a5193299
> Sent from the ActiveMQ - User forum at Nabble.com.
>
>


-- 

James
-------
http://radio.weblogs.com/0112098/