You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by vdevadhar <de...@yahoo.com> on 2011/07/01 01:05:27 UTC
What is the resource footprint of a queue on broker and client
sides?
Dear Active MQ users group,
I am benchmarking the maximum number of queues that activemq can support;
With default config I can
go up to 1000 queues;
For more than 1000, I need to add in the -
xbean:conf/activemq-scalability.xml to the command line for
broker startup;
I also needed to change the Xmx in activemq file to 4096M like shown below.
ACTIVEMQ_OPTS_MEMORY="-Xms256M -Xmx4096M"
Also, I had to add -Dorg.apache.activemq.UseDedicatedTaskRunner=false
to ACTIVEMQ_OPTS to get the client, server from running out of memory and
ability to create threads;
Now, I am able to create 10,000 + queues; Around 11108, I think I see memory
pressures; At this point,
I am seeing
Destination destination = session.createQueue(queueName);
and/or
MessageConsumer consumer = session.createConsumer(destination);
consumer.setMessageListener(amqHandler);
starts to seriously slow down;
Instead of further increasing JVM memory and further going up the number of
queues and iterating, I
wanted to find out the resource footprint required by a queue. What are the
number of threads,
memory, file descriptors, disk space that are needed when a new queue is
added to active mq?
Assume empty or almost empty queues. I want this information for both client
and broker side.
Assume the above mentioned scalable configs and suggest any other tweaks if
necessary.
Use case
We have 200 or so different job types that come in from 5000 or so
customers; The more queues we
can create, the more natural the system behavior since as we put multiple
job types or customers on the
same queue, we create sequencing between unrelated things. I know million
queues don't make a whole lot
of sense, but I am trying to find a big number that can be scalable.
Thanks
Vijay
--
View this message in context: http://activemq.2283324.n4.nabble.com/What-is-the-resource-footprint-of-a-queue-on-broker-and-client-sides-tp3637053p3637053.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: What is the resource footprint of a queue on broker and client sides?
Posted by Torsten Mielke <to...@fusesource.com>.
Can't get you all the number you ask for but
- since you already set UseDedicatedTaskRunner=false there is no extra thread created for each queue. Instead a thread pool is used
- without that setting each queue used its own thread. So memory overhead is that of a new thread plus surely some more bytes (don't know how much)
- file descriptor wise it should not make a diff as all queues are persisted through one instance of KahaDB (or whatever configured persistence adapter).
- disk space: Each newly created queue is persisted to disk. I don't see how that information alone (no msgs on queue) would take more than a couple of bytes. Its probably just one entry in the KahaDB journal file
There is a ActiveMQ tuning guide [1] available at www.fusesource.com that might provide additional tuning tips for your scenario.
However there will be a limit as to how many queues you can have on one broker.
Perhaps look into distributing the load across multiple independent brokers.
[1] http://fusesource.com/docs/broker/5.5/tuning/index.html
Torsten Mielke
torsten@fusesource.com
tmielke@blogspot.com
On Sep 16, 2011, at 12:52 AM, vdevadhar wrote:
> Dear Active MQ Users,
>
> Re-posting this question due to lack of reply; We are seeing 420K memory
> used per queue in the broker; Is that expected? This information is vital in
> our decision making; If any of you know, please respond.
>
> Thanks
> Vijay
>
> ------------------------------------------------------------------------------
>
> Dear Active MQ users group,
>
> I am benchmarking the maximum number of queues that activemq can support;
> With default config I can
> go up to 1000 queues;
>
> For more than 1000, I need to add in the -
> xbean:conf/activemq-scalability.xml to the command line for
> broker startup;
>
> I also needed to change the Xmx in activemq file to 4096M like shown below.
> ACTIVEMQ_OPTS_MEMORY="-Xms256M -Xmx4096M"
>
> Also, I had to add -Dorg.apache.activemq.UseDedicatedTaskRunner=false
> to ACTIVEMQ_OPTS to get the client, server from running out of memory and
> ability to create threads;
>
> Now, I am able to create 10,000 + queues; Around 11108, I think I see memory
> pressures; At this point,
> I am seeing
> Destination destination = session.createQueue(queueName);
>
> and/or
>
> MessageConsumer consumer = session.createConsumer(destination);
> consumer.setMessageListener(amqHandler);
>
> starts to seriously slow down;
>
> Instead of further increasing JVM memory and further going up the number of
> queues and iterating, I
> wanted to find out the resource footprint required by a queue. What are the
> number of threads,
> memory, file descriptors, disk space that are needed when a new queue is
> added to active mq?
> Assume empty or almost empty queues. I want this information for both client
> and broker side.
> Assume the above mentioned scalable configs and suggest any other tweaks if
> necessary.
>
> Use case
>
> We have 200 or so different job types that come in from 5000 or so
> customers; The more queues we
> can create, the more natural the system behavior since as we put multiple
> job types or customers on the
> same queue, we create sequencing between unrelated things. I know million
> queues don't make a whole lot
> of sense, but I am trying to find a big number that can be scalable.
>
> Thanks
> Vijay
Re: What is the resource footprint of a queue on broker and client
sides?
Posted by vdevadhar <de...@yahoo.com>.
Dear Active MQ Users,
Re-posting this question due to lack of reply; We are seeing 420K memory
used per queue in the broker; Is that expected? This information is vital in
our decision making; If any of you know, please respond.
Thanks
Vijay
------------------------------------------------------------------------------
Dear Active MQ users group,
I am benchmarking the maximum number of queues that activemq can support;
With default config I can
go up to 1000 queues;
For more than 1000, I need to add in the -
xbean:conf/activemq-scalability.xml to the command line for
broker startup;
I also needed to change the Xmx in activemq file to 4096M like shown below.
ACTIVEMQ_OPTS_MEMORY="-Xms256M -Xmx4096M"
Also, I had to add -Dorg.apache.activemq.UseDedicatedTaskRunner=false
to ACTIVEMQ_OPTS to get the client, server from running out of memory and
ability to create threads;
Now, I am able to create 10,000 + queues; Around 11108, I think I see memory
pressures; At this point,
I am seeing
Destination destination = session.createQueue(queueName);
and/or
MessageConsumer consumer = session.createConsumer(destination);
consumer.setMessageListener(amqHandler);
starts to seriously slow down;
Instead of further increasing JVM memory and further going up the number of
queues and iterating, I
wanted to find out the resource footprint required by a queue. What are the
number of threads,
memory, file descriptors, disk space that are needed when a new queue is
added to active mq?
Assume empty or almost empty queues. I want this information for both client
and broker side.
Assume the above mentioned scalable configs and suggest any other tweaks if
necessary.
Use case
We have 200 or so different job types that come in from 5000 or so
customers; The more queues we
can create, the more natural the system behavior since as we put multiple
job types or customers on the
same queue, we create sequencing between unrelated things. I know million
queues don't make a whole lot
of sense, but I am trying to find a big number that can be scalable.
Thanks
Vijay
----------------------------------------------------------------------------------------------
--
View this message in context: http://activemq.2283324.n4.nabble.com/What-is-the-resource-footprint-of-a-queue-on-broker-and-client-sides-tp3637053p3816983.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.