You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Nominal <no...@yahoo.com> on 2009/06/10 18:35:00 UTC
Qpid C++ Broker memory use
I've been doing some testing against the latest 0.5 release of qpid and have some questions on how the broker manages queues and the memory for them. I've searched the archives but couldn't find any answers. At this point I have narrowed my tests to only the c++ examples to eliminate any errors that may be in code that I am writing.
I am using the simple direct examples for the cpp broker. The qpidd.conf is empty so all the defaults are being used. When I start up qpid the memory use is as follows
Every 2.0s: ps -eLF | grep qpid | grep -v grep Wed Jun 10 11:48:59 2009
qpidd 29192 1 29192 0 5 20104 2868 0 11:47 ? 00:00:00 /usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd
qpidd 29192 1 29193 0 5 20104 2868 0 11:47 ? 00:00:00 /usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd
qpidd 29192 1 29194 0 5 20104 2868 0 11:47 ? 00:00:00 /usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd
qpidd 29192 1 29195 0 5 20104 2868 0 11:47 ? 00:00:00 /usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd
qpidd 29192 1 29197 0 5 20104 2868 0 11:47 ? 00:00:00 /usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd
The resident memory is 2868 from a clean start
I run ./declare_queues
Then I produce 50k messages on message_queue with: ./direct_producer localhost 5672 50000
resident size for each of the threads goes up to 53800
qpidd 29192 1 29192 0 5 32642 53800 0 11:47 ? 00:00:01 /usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd
then I run the ./listener to consume the messages
qpidd 29192 1 29192 0 5 49051 54256 0 11:47 ? 00:00:02 /usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd
the resident memory size goes up again.
The only way I can free this memory is to restart the daemon. Is this how qpid manages memory and is it something I should not be concerned with? How does qpid determine when it has used enough memory to avoid swapping? Is this something that is configured qpid.max_count qpid.max_size arguments? The way I am expecting it to work in very lay terms is:
1. put message on queue, take up memory needed for message.
2. consume message successfully, free memory used for message.
The way it appears to be working is:
1. put message on queue, take up memory needed for message.
2. consume message successfully, never free used memory.
Any info on this would be greatly appreciated, I would really like to move forward with using qpid for a messaging bus and have been very pleased with it overall, I'm just confused at what I'm observing and if its normal or not.
Thanks
Etrik
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Qpid C++ Broker memory use
Posted by Alan Conway <ac...@redhat.com>.
Nominal wrote:
> I've been doing some testing against the latest 0.5 release of qpid and have some questions on how the broker manages queues and the memory for them. I've searched the archives but couldn't find any answers. At this point I have narrowed my tests to only the c++ examples to eliminate any errors that may be in code that I am writing.
>
> I am using the simple direct examples for the cpp broker. The qpidd.conf is empty so all the defaults are being used. When I start up qpid the memory use is as follows
>
> Every 2.0s: ps -eLF | grep qpid | grep -v grep Wed Jun 10 11:48:59 2009
>
> qpidd 29192 1 29192 0 5 20104 2868 0 11:47 ? 00:00:00 /usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd
> qpidd 29192 1 29193 0 5 20104 2868 0 11:47 ? 00:00:00 /usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd
> qpidd 29192 1 29194 0 5 20104 2868 0 11:47 ? 00:00:00 /usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd
> qpidd 29192 1 29195 0 5 20104 2868 0 11:47 ? 00:00:00 /usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd
> qpidd 29192 1 29197 0 5 20104 2868 0 11:47 ? 00:00:00 /usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd
>
>
> The resident memory is 2868 from a clean start
>
> I run ./declare_queues
>
> Then I produce 50k messages on message_queue with: ./direct_producer localhost 5672 50000
>
> resident size for each of the threads goes up to 53800
>
> qpidd 29192 1 29192 0 5 32642 53800 0 11:47 ? 00:00:01 /usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd
>
> then I run the ./listener to consume the messages
>
> qpidd 29192 1 29192 0 5 49051 54256 0 11:47 ? 00:00:02 /usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd
>
> the resident memory size goes up again.
>
> The only way I can free this memory is to restart the daemon. Is this how qpid manages memory and is it something I should not be concerned with? How does qpid determine when it has used enough memory to avoid swapping? Is this something that is configured qpid.max_count qpid.max_size arguments? The way I am expecting it to work in very lay terms is:
>
> 1. put message on queue, take up memory needed for message.
> 2. consume message successfully, free memory used for message.
>
> The way it appears to be working is:
>
> 1. put message on queue, take up memory needed for message.
> 2. consume message successfully, never free used memory.
>
> Any info on this would be greatly appreciated, I would really like to move forward with using qpid for a messaging bus and have been very pleased with it overall, I'm just confused at what I'm observing and if its normal or not.
>
qpidd is currently using the standard C++ new/delete to manage memory, so memory
freed by qpidd is not necessarily returned to the OS immediately, particularly
if the heap is fragmented. I'm not surprised at your results, but I haven't done
much heap profiling to date so I can't say much about when you would start to
see memory being released back to the OS.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org