You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Jeff Armstrong <ja...@avvasi.com> on 2011/08/26 21:08:23 UTC
qpidd using approx 10x memory
When filling up some queues, I noticed that the qpidd process was using about 10x the memory than was actually being used by the queues. To do a simple test on this I wrote a simple client that sends a bunch of messages that get stored in the queue, then compared the total byte depth with the memory used by qpidd.
After sending 1 million messages, the byte count (using qpid-stat -q) showed 105MB, but checking the memory usage I saw 1.4GB (checked with htop and pmap). The messages were approximately 100 bytes large, so the byte count from qpid-stat makes sense. What doesn't make sense is that the total memory usage is over 10x as big. I checked the memory usage of qpidd before connecting with the client to send the messages and it was around 200MB, so there is still a discrepancy of about 1.1GB. These results also match with the original problem with an application I'm working on where qpidd used up all my memory (16GB) even though the queue sizes added up to about 1GB.
I'm using version 0.10. Is this a known issue? I couldn't find it in the issue tracker. Any workarounds?
Here is how I created the exchange/queues:
$ ./qpid-config add exchange direct qvue
$ ./qpid-config add queue testQueue
$ ./qpid-config bind qvue testQueue testBind
Here is the client code I used:
#include <iostream>
#include <qpid/client/Message.h>
#include <qpid/client/Connection.h>
#include <qpid/client/Session.h>
using namespace std;
using namespace qpid::client;
int main(int argc, char** argv) {
Connection connection;
connection.open("127.0.0.1");
Session session = connection.newSession();
int count = 0;
while(true) {
Message msg;
msg.setData("Hello World! 0123456789 abcdefghijklmnopqrstuvwxyz weeeeeeeeeeeeeooooooooooooooooooooooooooooooooooooooo!");
msg.getDeliveryProperties().setRoutingKey("testBind");
session.messageTransfer(qpid::client::arg::content=msg, qpid::client::arg::destination="testQueue");
count++;
if (count % 1000 == 0) {
cout << count << " messages sent" << endl;
}
}
return 0;
}
Thanks,
Jeff
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: qpidd using approx 10x memory
Posted by Alan Conway <ac...@redhat.com>.
On 09/16/2011 03:11 PM, Jeff Armstrong wrote:
> I tried increasing the message size to 1K and found that the overhead per message was indeed the same. So this leads me to the next questions: why is the overhead so large per message, and is there a way to reduce this? From what I can tell from the client side of things, each message has a bunch of bool/int fields and a few small strings (routing key, exchange name, etc.), but nothing that would put it up over 1K.
I don't think anyones done a detailed analysis of qpid's per-message memory
overhead. It would be interesting to run a tool like valgrind on the broker
and see where the memory is going. There is probably room for improvement.
>
> ________________________________________
> From: Kim van der Riet [kim.vdriet@redhat.com]
> Sent: Friday, September 09, 2011 2:34 PM
> To: users@qpid.apache.org
> Subject: RE: qpidd using approx 10x memory
>
> On Fri, 2011-09-09 at 12:19 -0400, Jeff Armstrong wrote:
>> So I just tried my test again after compiling the broker with the following code added and still saw the same issue of 10x memory consumption.
>>
>> mallopt(M_ARENA_TEST, 0);
>> mallopt(M_ARENA_MAX, 1);
>>
>> ________________________________________
>> From: Jeff Armstrong [jarmstrong@avvasi.com]
>> Sent: Friday, September 09, 2011 11:40 AM
>> To: users@qpid.apache.org
>> Subject: RE: qpidd using approx 10x memory
>>
>> The sample code I provided only publishes messages - there is no consuming going on, so this doesn't follow the pattern you are suggesting. If there is no freeing happening because there is no consumer, it seems like there must be some other cause of this.
>> Also, in my application, there are two connections publishing and 2+ connections consuming and this shows the same problem.
>>
>
> I re-read your original mail, and I think I misunderstood your issue,
> sorry.
>
> My guess is that the per-message overhead for each message is much
> larger than the message itself in your case - 100 bytes is a rather
> small message. I have always assumed a 1-2KiB memory overhead per
> message for headers and other bookkeeping purposes.
>
> One way to check this is to increase the size of the message only to,
> say, 1KiB or 10KiB, and check if the overhead remains constant.
>
> Perhaps others can comment?
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: qpidd using approx 10x memory
Posted by Jeff Armstrong <ja...@avvasi.com>.
I tried increasing the message size to 1K and found that the overhead per message was indeed the same. So this leads me to the next questions: why is the overhead so large per message, and is there a way to reduce this? From what I can tell from the client side of things, each message has a bunch of bool/int fields and a few small strings (routing key, exchange name, etc.), but nothing that would put it up over 1K.
________________________________________
From: Kim van der Riet [kim.vdriet@redhat.com]
Sent: Friday, September 09, 2011 2:34 PM
To: users@qpid.apache.org
Subject: RE: qpidd using approx 10x memory
On Fri, 2011-09-09 at 12:19 -0400, Jeff Armstrong wrote:
> So I just tried my test again after compiling the broker with the following code added and still saw the same issue of 10x memory consumption.
>
> mallopt(M_ARENA_TEST, 0);
> mallopt(M_ARENA_MAX, 1);
>
> ________________________________________
> From: Jeff Armstrong [jarmstrong@avvasi.com]
> Sent: Friday, September 09, 2011 11:40 AM
> To: users@qpid.apache.org
> Subject: RE: qpidd using approx 10x memory
>
> The sample code I provided only publishes messages - there is no consuming going on, so this doesn't follow the pattern you are suggesting. If there is no freeing happening because there is no consumer, it seems like there must be some other cause of this.
> Also, in my application, there are two connections publishing and 2+ connections consuming and this shows the same problem.
>
I re-read your original mail, and I think I misunderstood your issue,
sorry.
My guess is that the per-message overhead for each message is much
larger than the message itself in your case - 100 bytes is a rather
small message. I have always assumed a 1-2KiB memory overhead per
message for headers and other bookkeeping purposes.
One way to check this is to increase the size of the message only to,
say, 1KiB or 10KiB, and check if the overhead remains constant.
Perhaps others can comment?
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: qpidd using approx 10x memory
Posted by Kim van der Riet <ki...@redhat.com>.
On Fri, 2011-09-09 at 12:19 -0400, Jeff Armstrong wrote:
> So I just tried my test again after compiling the broker with the following code added and still saw the same issue of 10x memory consumption.
>
> mallopt(M_ARENA_TEST, 0);
> mallopt(M_ARENA_MAX, 1);
>
> ________________________________________
> From: Jeff Armstrong [jarmstrong@avvasi.com]
> Sent: Friday, September 09, 2011 11:40 AM
> To: users@qpid.apache.org
> Subject: RE: qpidd using approx 10x memory
>
> The sample code I provided only publishes messages - there is no consuming going on, so this doesn't follow the pattern you are suggesting. If there is no freeing happening because there is no consumer, it seems like there must be some other cause of this.
> Also, in my application, there are two connections publishing and 2+ connections consuming and this shows the same problem.
>
I re-read your original mail, and I think I misunderstood your issue,
sorry.
My guess is that the per-message overhead for each message is much
larger than the message itself in your case - 100 bytes is a rather
small message. I have always assumed a 1-2KiB memory overhead per
message for headers and other bookkeeping purposes.
One way to check this is to increase the size of the message only to,
say, 1KiB or 10KiB, and check if the overhead remains constant.
Perhaps others can comment?
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: qpidd using approx 10x memory
Posted by Jeff Armstrong <ja...@avvasi.com>.
So I just tried my test again after compiling the broker with the following code added and still saw the same issue of 10x memory consumption.
mallopt(M_ARENA_TEST, 0);
mallopt(M_ARENA_MAX, 1);
________________________________________
From: Jeff Armstrong [jarmstrong@avvasi.com]
Sent: Friday, September 09, 2011 11:40 AM
To: users@qpid.apache.org
Subject: RE: qpidd using approx 10x memory
The sample code I provided only publishes messages - there is no consuming going on, so this doesn't follow the pattern you are suggesting. If there is no freeing happening because there is no consumer, it seems like there must be some other cause of this.
Also, in my application, there are two connections publishing and 2+ connections consuming and this shows the same problem.
________________________________________
From: Kim van der Riet [kim.vdriet@redhat.com]
Sent: Friday, September 09, 2011 7:15 AM
To: users@qpid.apache.org
Subject: RE: qpidd using approx 10x memory
On Thu, 2011-09-08 at 22:14 -0400, Ohme, Gregory (GE Healthcare) wrote:
> The pooling is a feature of glibc's malloc and the pools are referred to
> as arenas. Arenas exist to ease thread contention in threaded programs.
> The down size is that memory fragmentation can increase with the number
> of arenas that get created. In a heavily threaded program the number of
> arenas will typically equal the number of hardware cores that are
> available in a system. You can disable the arenas in newer versions of
> glibc via malloc options but it depends on if the distro supports the
> behavior since it is an experimental flag for malloc. Also note that
> malloc doesn't want to give memory back to the system right away you can
> control this behavior via malloc options as well.
>
> http://www.gnu.org/s/libc/manual/html_node/Malloc-Tunable-Parameters.htm
> l#Malloc-Tunable-Parameters
>
> Here are some glibc bug reports on the matter
>
> http://sourceware.org/bugzilla/show_bug.cgi?id=11044
> http://sourceware.org/bugzilla/show_bug.cgi?id=11261
>
> Another downsize to note about arena's are that they only allocate using
> memmap and never using the more_core hook. This breaks libraries that
> implement the more_core hook, such as libhugetlbfs..
>
> Regards,
> Greg
>
I can confirm that this problem exists for patterns where one connection
on the broker is responsible for publishing and another for consuming.
If I understand correctly, the broker assigns worker threads to
connections to limit the locking on sockets, but the result of this is
that one thread is constantly mallocing memory while another is freeing
it. This results in the pooling of freed memory on a thread which does
not reuse it.
In Fedora/RHEL, this behavior can be controlled by environment variables
MALLOC_ARENA_TEST and MALLOC_ARENA_MAX. Setting MALLOC_ARENA_TEST to 0
and MALLOC_ARENA_MAX to 1 should revert the malloc/free behavior to
something similar to before this was introduced. I have googled these
and almost all refs come back to RHEL/Fedora and their derivatives so I
am uncertain to what extent they are supported outside these distros,
but it is worth a try. Otherwise, as is suggested by Ulrich Drepper in
the bugzilla links above, use mallopt().
I have confirmed that using these settings with a broker usage pattern
of repeatedly publishing on one connection then consuming on another
large blocks of messages (which has the effect of driving the free
memory down to zero and into swapping) permanently averted this problem.
Clearly using these parameters should be considered as tuning - there is
no solution which would fit all situations. If you have a usage pattern
in which connections consume as much as they publish, then this
situation would not occur and would probably realize a small performance
gain from the per-thread free memory pooling.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: qpidd using approx 10x memory
Posted by Jeff Armstrong <ja...@avvasi.com>.
The sample code I provided only publishes messages - there is no consuming going on, so this doesn't follow the pattern you are suggesting. If there is no freeing happening because there is no consumer, it seems like there must be some other cause of this.
Also, in my application, there are two connections publishing and 2+ connections consuming and this shows the same problem.
________________________________________
From: Kim van der Riet [kim.vdriet@redhat.com]
Sent: Friday, September 09, 2011 7:15 AM
To: users@qpid.apache.org
Subject: RE: qpidd using approx 10x memory
On Thu, 2011-09-08 at 22:14 -0400, Ohme, Gregory (GE Healthcare) wrote:
> The pooling is a feature of glibc's malloc and the pools are referred to
> as arenas. Arenas exist to ease thread contention in threaded programs.
> The down size is that memory fragmentation can increase with the number
> of arenas that get created. In a heavily threaded program the number of
> arenas will typically equal the number of hardware cores that are
> available in a system. You can disable the arenas in newer versions of
> glibc via malloc options but it depends on if the distro supports the
> behavior since it is an experimental flag for malloc. Also note that
> malloc doesn't want to give memory back to the system right away you can
> control this behavior via malloc options as well.
>
> http://www.gnu.org/s/libc/manual/html_node/Malloc-Tunable-Parameters.htm
> l#Malloc-Tunable-Parameters
>
> Here are some glibc bug reports on the matter
>
> http://sourceware.org/bugzilla/show_bug.cgi?id=11044
> http://sourceware.org/bugzilla/show_bug.cgi?id=11261
>
> Another downsize to note about arena's are that they only allocate using
> memmap and never using the more_core hook. This breaks libraries that
> implement the more_core hook, such as libhugetlbfs..
>
> Regards,
> Greg
>
I can confirm that this problem exists for patterns where one connection
on the broker is responsible for publishing and another for consuming.
If I understand correctly, the broker assigns worker threads to
connections to limit the locking on sockets, but the result of this is
that one thread is constantly mallocing memory while another is freeing
it. This results in the pooling of freed memory on a thread which does
not reuse it.
In Fedora/RHEL, this behavior can be controlled by environment variables
MALLOC_ARENA_TEST and MALLOC_ARENA_MAX. Setting MALLOC_ARENA_TEST to 0
and MALLOC_ARENA_MAX to 1 should revert the malloc/free behavior to
something similar to before this was introduced. I have googled these
and almost all refs come back to RHEL/Fedora and their derivatives so I
am uncertain to what extent they are supported outside these distros,
but it is worth a try. Otherwise, as is suggested by Ulrich Drepper in
the bugzilla links above, use mallopt().
I have confirmed that using these settings with a broker usage pattern
of repeatedly publishing on one connection then consuming on another
large blocks of messages (which has the effect of driving the free
memory down to zero and into swapping) permanently averted this problem.
Clearly using these parameters should be considered as tuning - there is
no solution which would fit all situations. If you have a usage pattern
in which connections consume as much as they publish, then this
situation would not occur and would probably realize a small performance
gain from the per-thread free memory pooling.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: qpidd using approx 10x memory
Posted by Kim van der Riet <ki...@redhat.com>.
On Thu, 2011-09-08 at 22:14 -0400, Ohme, Gregory (GE Healthcare) wrote:
> The pooling is a feature of glibc's malloc and the pools are referred to
> as arenas. Arenas exist to ease thread contention in threaded programs.
> The down size is that memory fragmentation can increase with the number
> of arenas that get created. In a heavily threaded program the number of
> arenas will typically equal the number of hardware cores that are
> available in a system. You can disable the arenas in newer versions of
> glibc via malloc options but it depends on if the distro supports the
> behavior since it is an experimental flag for malloc. Also note that
> malloc doesn't want to give memory back to the system right away you can
> control this behavior via malloc options as well.
>
> http://www.gnu.org/s/libc/manual/html_node/Malloc-Tunable-Parameters.htm
> l#Malloc-Tunable-Parameters
>
> Here are some glibc bug reports on the matter
>
> http://sourceware.org/bugzilla/show_bug.cgi?id=11044
> http://sourceware.org/bugzilla/show_bug.cgi?id=11261
>
> Another downsize to note about arena's are that they only allocate using
> memmap and never using the more_core hook. This breaks libraries that
> implement the more_core hook, such as libhugetlbfs..
>
> Regards,
> Greg
>
I can confirm that this problem exists for patterns where one connection
on the broker is responsible for publishing and another for consuming.
If I understand correctly, the broker assigns worker threads to
connections to limit the locking on sockets, but the result of this is
that one thread is constantly mallocing memory while another is freeing
it. This results in the pooling of freed memory on a thread which does
not reuse it.
In Fedora/RHEL, this behavior can be controlled by environment variables
MALLOC_ARENA_TEST and MALLOC_ARENA_MAX. Setting MALLOC_ARENA_TEST to 0
and MALLOC_ARENA_MAX to 1 should revert the malloc/free behavior to
something similar to before this was introduced. I have googled these
and almost all refs come back to RHEL/Fedora and their derivatives so I
am uncertain to what extent they are supported outside these distros,
but it is worth a try. Otherwise, as is suggested by Ulrich Drepper in
the bugzilla links above, use mallopt().
I have confirmed that using these settings with a broker usage pattern
of repeatedly publishing on one connection then consuming on another
large blocks of messages (which has the effect of driving the free
memory down to zero and into swapping) permanently averted this problem.
Clearly using these parameters should be considered as tuning - there is
no solution which would fit all situations. If you have a usage pattern
in which connections consume as much as they publish, then this
situation would not occur and would probably realize a small performance
gain from the per-thread free memory pooling.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: qpidd using approx 10x memory
Posted by "Ohme, Gregory (GE Healthcare)" <Gr...@ge.com>.
The pooling is a feature of glibc's malloc and the pools are referred to
as arenas. Arenas exist to ease thread contention in threaded programs.
The down size is that memory fragmentation can increase with the number
of arenas that get created. In a heavily threaded program the number of
arenas will typically equal the number of hardware cores that are
available in a system. You can disable the arenas in newer versions of
glibc via malloc options but it depends on if the distro supports the
behavior since it is an experimental flag for malloc. Also note that
malloc doesn't want to give memory back to the system right away you can
control this behavior via malloc options as well.
http://www.gnu.org/s/libc/manual/html_node/Malloc-Tunable-Parameters.htm
l#Malloc-Tunable-Parameters
Here are some glibc bug reports on the matter
http://sourceware.org/bugzilla/show_bug.cgi?id=11044
http://sourceware.org/bugzilla/show_bug.cgi?id=11261
Another downsize to note about arena's are that they only allocate using
memmap and never using the more_core hook. This breaks libraries that
implement the more_core hook, such as libhugetlbfs..
Regards,
Greg
-----Original Message-----
From: Carl Trieloff [mailto:cctrieloff@redhat.com]
Sent: Thursday, September 08, 2011 3:56 PM
To: users@qpid.apache.org
Subject: Re: qpidd using approx 10x memory
It's OS level, let me try google it for you tonight
Carl.
On 09/08/2011 04:44 PM, Jeff Armstrong wrote:
> Using gcc version 4.4.3.
> I couldn't find anything about thread pooling for Ubuntu. Are you sure
that this is done at the OS level and not at the application level?
Also, could this cause memory bloat relative to the number of messages
if it is threads that are pooled? This seems unlikely to me.
>
> Jeff
>
> ________________________________________
> From: Carl Trieloff [cctrieloff@redhat.com]
> Sent: Thursday, September 08, 2011 4:34 PM
> To: users@qpid.apache.org
> Subject: Re: qpidd using approx 10x memory
>
> what is the gcc version? or do you know if memory thread pooling is
> enabled in 10.04
>
> turning that off should resolve the memory bloat. On Fedora / RHEL it
> is an env var, should be able to google it for Ubuntu
>
> Carl.
>
>
> On 09/08/2011 01:30 PM, Jeff Armstrong wrote:
>> Ubuntu 10.04
>> ________________________________________
>> From: Carl Trieloff [cctrieloff@redhat.com]
>> Sent: Thursday, September 08, 2011 12:25 PM
>> To: users@qpid.apache.org
>> Subject: Re: qpidd using approx 10x memory
>>
>> What OS version?
>>
>> Carl.
>>
>>
>> On 09/08/2011 11:53 AM, Jeff Armstrong wrote:
>>> Does anyone have thoughts on this? Should this be submitted in the
issue tracker?
>>> ________________________________________
>>> From: Jeff Armstrong [jarmstrong@avvasi.com]
>>> Sent: Friday, August 26, 2011 3:08 PM
>>> To: users@qpid.apache.org
>>> Subject: qpidd using approx 10x memory
>>>
>>> When filling up some queues, I noticed that the qpidd process was
using about 10x the memory than was actually being used by the queues.
To do a simple test on this I wrote a simple client that sends a bunch
of messages that get stored in the queue, then compared the total byte
depth with the memory used by qpidd.
>>>
>>> After sending 1 million messages, the byte count (using qpid-stat
-q) showed 105MB, but checking the memory usage I saw 1.4GB (checked
with htop and pmap). The messages were approximately 100 bytes large, so
the byte count from qpid-stat makes sense. What doesn't make sense is
that the total memory usage is over 10x as big. I checked the memory
usage of qpidd before connecting with the client to send the messages
and it was around 200MB, so there is still a discrepancy of about 1.1GB.
These results also match with the original problem with an application
I'm working on where qpidd used up all my memory (16GB) even though the
queue sizes added up to about 1GB.
>>>
>>> I'm using version 0.10. Is this a known issue? I couldn't find it in
the issue tracker. Any workarounds?
>>>
>>>
>>> Here is how I created the exchange/queues:
>>> $ ./qpid-config add exchange direct qvue $ ./qpid-config add queue
>>> testQueue $ ./qpid-config bind qvue testQueue testBind
>>>
>>>
>>> Here is the client code I used:
>>> #include <iostream>
>>> #include <qpid/client/Message.h>
>>> #include <qpid/client/Connection.h>
>>> #include <qpid/client/Session.h>
>>>
>>> using namespace std;
>>> using namespace qpid::client;
>>>
>>> int main(int argc, char** argv) {
>>> Connection connection;
>>> connection.open("127.0.0.1");
>>> Session session = connection.newSession();
>>>
>>> int count = 0;
>>> while(true) {
>>> Message msg;
>>> msg.setData("Hello World! 0123456789
abcdefghijklmnopqrstuvwxyz
weeeeeeeeeeeeeooooooooooooooooooooooooooooooooooooooo!");
>>> msg.getDeliveryProperties().setRoutingKey("testBind");
>>> session.messageTransfer(qpid::client::arg::content=msg,
qpid::client::arg::destination="testQueue");
>>> count++;
>>> if (count % 1000 == 0) {
>>> cout << count << " messages sent" << endl;
>>> }
>>> }
>>> return 0;
>>> }
>>>
>>>
>>> Thanks,
>>> Jeff
>>> --------------------------------------------------------------------
>>> - Apache Qpid - AMQP Messaging Implementation
>>> Project: http://qpid.apache.org
>>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>>
>>>
>>> --------------------------------------------------------------------
>>> - Apache Qpid - AMQP Messaging Implementation
>>> Project: http://qpid.apache.org
>>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: qpidd using approx 10x memory
Posted by Carl Trieloff <cc...@redhat.com>.
It's OS level, let me try google it for you tonight
Carl.
On 09/08/2011 04:44 PM, Jeff Armstrong wrote:
> Using gcc version 4.4.3.
> I couldn't find anything about thread pooling for Ubuntu. Are you sure that this is done at the OS level and not at the application level? Also, could this cause memory bloat relative to the number of messages if it is threads that are pooled? This seems unlikely to me.
>
> Jeff
>
> ________________________________________
> From: Carl Trieloff [cctrieloff@redhat.com]
> Sent: Thursday, September 08, 2011 4:34 PM
> To: users@qpid.apache.org
> Subject: Re: qpidd using approx 10x memory
>
> what is the gcc version? or do you know if memory thread pooling is
> enabled in 10.04
>
> turning that off should resolve the memory bloat. On Fedora / RHEL it is
> an env var, should be able to google it for Ubuntu
>
> Carl.
>
>
> On 09/08/2011 01:30 PM, Jeff Armstrong wrote:
>> Ubuntu 10.04
>> ________________________________________
>> From: Carl Trieloff [cctrieloff@redhat.com]
>> Sent: Thursday, September 08, 2011 12:25 PM
>> To: users@qpid.apache.org
>> Subject: Re: qpidd using approx 10x memory
>>
>> What OS version?
>>
>> Carl.
>>
>>
>> On 09/08/2011 11:53 AM, Jeff Armstrong wrote:
>>> Does anyone have thoughts on this? Should this be submitted in the issue tracker?
>>> ________________________________________
>>> From: Jeff Armstrong [jarmstrong@avvasi.com]
>>> Sent: Friday, August 26, 2011 3:08 PM
>>> To: users@qpid.apache.org
>>> Subject: qpidd using approx 10x memory
>>>
>>> When filling up some queues, I noticed that the qpidd process was using about 10x the memory than was actually being used by the queues. To do a simple test on this I wrote a simple client that sends a bunch of messages that get stored in the queue, then compared the total byte depth with the memory used by qpidd.
>>>
>>> After sending 1 million messages, the byte count (using qpid-stat -q) showed 105MB, but checking the memory usage I saw 1.4GB (checked with htop and pmap). The messages were approximately 100 bytes large, so the byte count from qpid-stat makes sense. What doesn't make sense is that the total memory usage is over 10x as big. I checked the memory usage of qpidd before connecting with the client to send the messages and it was around 200MB, so there is still a discrepancy of about 1.1GB. These results also match with the original problem with an application I'm working on where qpidd used up all my memory (16GB) even though the queue sizes added up to about 1GB.
>>>
>>> I'm using version 0.10. Is this a known issue? I couldn't find it in the issue tracker. Any workarounds?
>>>
>>>
>>> Here is how I created the exchange/queues:
>>> $ ./qpid-config add exchange direct qvue
>>> $ ./qpid-config add queue testQueue
>>> $ ./qpid-config bind qvue testQueue testBind
>>>
>>>
>>> Here is the client code I used:
>>> #include <iostream>
>>> #include <qpid/client/Message.h>
>>> #include <qpid/client/Connection.h>
>>> #include <qpid/client/Session.h>
>>>
>>> using namespace std;
>>> using namespace qpid::client;
>>>
>>> int main(int argc, char** argv) {
>>> Connection connection;
>>> connection.open("127.0.0.1");
>>> Session session = connection.newSession();
>>>
>>> int count = 0;
>>> while(true) {
>>> Message msg;
>>> msg.setData("Hello World! 0123456789 abcdefghijklmnopqrstuvwxyz weeeeeeeeeeeeeooooooooooooooooooooooooooooooooooooooo!");
>>> msg.getDeliveryProperties().setRoutingKey("testBind");
>>> session.messageTransfer(qpid::client::arg::content=msg, qpid::client::arg::destination="testQueue");
>>> count++;
>>> if (count % 1000 == 0) {
>>> cout << count << " messages sent" << endl;
>>> }
>>> }
>>> return 0;
>>> }
>>>
>>>
>>> Thanks,
>>> Jeff
>>> ---------------------------------------------------------------------
>>> Apache Qpid - AMQP Messaging Implementation
>>> Project: http://qpid.apache.org
>>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>>
>>>
>>> ---------------------------------------------------------------------
>>> Apache Qpid - AMQP Messaging Implementation
>>> Project: http://qpid.apache.org
>>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: qpidd using approx 10x memory
Posted by Jeff Armstrong <ja...@avvasi.com>.
Using gcc version 4.4.3.
I couldn't find anything about thread pooling for Ubuntu. Are you sure that this is done at the OS level and not at the application level? Also, could this cause memory bloat relative to the number of messages if it is threads that are pooled? This seems unlikely to me.
Jeff
________________________________________
From: Carl Trieloff [cctrieloff@redhat.com]
Sent: Thursday, September 08, 2011 4:34 PM
To: users@qpid.apache.org
Subject: Re: qpidd using approx 10x memory
what is the gcc version? or do you know if memory thread pooling is
enabled in 10.04
turning that off should resolve the memory bloat. On Fedora / RHEL it is
an env var, should be able to google it for Ubuntu
Carl.
On 09/08/2011 01:30 PM, Jeff Armstrong wrote:
> Ubuntu 10.04
> ________________________________________
> From: Carl Trieloff [cctrieloff@redhat.com]
> Sent: Thursday, September 08, 2011 12:25 PM
> To: users@qpid.apache.org
> Subject: Re: qpidd using approx 10x memory
>
> What OS version?
>
> Carl.
>
>
> On 09/08/2011 11:53 AM, Jeff Armstrong wrote:
>> Does anyone have thoughts on this? Should this be submitted in the issue tracker?
>> ________________________________________
>> From: Jeff Armstrong [jarmstrong@avvasi.com]
>> Sent: Friday, August 26, 2011 3:08 PM
>> To: users@qpid.apache.org
>> Subject: qpidd using approx 10x memory
>>
>> When filling up some queues, I noticed that the qpidd process was using about 10x the memory than was actually being used by the queues. To do a simple test on this I wrote a simple client that sends a bunch of messages that get stored in the queue, then compared the total byte depth with the memory used by qpidd.
>>
>> After sending 1 million messages, the byte count (using qpid-stat -q) showed 105MB, but checking the memory usage I saw 1.4GB (checked with htop and pmap). The messages were approximately 100 bytes large, so the byte count from qpid-stat makes sense. What doesn't make sense is that the total memory usage is over 10x as big. I checked the memory usage of qpidd before connecting with the client to send the messages and it was around 200MB, so there is still a discrepancy of about 1.1GB. These results also match with the original problem with an application I'm working on where qpidd used up all my memory (16GB) even though the queue sizes added up to about 1GB.
>>
>> I'm using version 0.10. Is this a known issue? I couldn't find it in the issue tracker. Any workarounds?
>>
>>
>> Here is how I created the exchange/queues:
>> $ ./qpid-config add exchange direct qvue
>> $ ./qpid-config add queue testQueue
>> $ ./qpid-config bind qvue testQueue testBind
>>
>>
>> Here is the client code I used:
>> #include <iostream>
>> #include <qpid/client/Message.h>
>> #include <qpid/client/Connection.h>
>> #include <qpid/client/Session.h>
>>
>> using namespace std;
>> using namespace qpid::client;
>>
>> int main(int argc, char** argv) {
>> Connection connection;
>> connection.open("127.0.0.1");
>> Session session = connection.newSession();
>>
>> int count = 0;
>> while(true) {
>> Message msg;
>> msg.setData("Hello World! 0123456789 abcdefghijklmnopqrstuvwxyz weeeeeeeeeeeeeooooooooooooooooooooooooooooooooooooooo!");
>> msg.getDeliveryProperties().setRoutingKey("testBind");
>> session.messageTransfer(qpid::client::arg::content=msg, qpid::client::arg::destination="testQueue");
>> count++;
>> if (count % 1000 == 0) {
>> cout << count << " messages sent" << endl;
>> }
>> }
>> return 0;
>> }
>>
>>
>> Thanks,
>> Jeff
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: qpidd using approx 10x memory
Posted by Carl Trieloff <cc...@redhat.com>.
what is the gcc version? or do you know if memory thread pooling is
enabled in 10.04
turning that off should resolve the memory bloat. On Fedora / RHEL it is
an env var, should be able to google it for Ubuntu
Carl.
On 09/08/2011 01:30 PM, Jeff Armstrong wrote:
> Ubuntu 10.04
> ________________________________________
> From: Carl Trieloff [cctrieloff@redhat.com]
> Sent: Thursday, September 08, 2011 12:25 PM
> To: users@qpid.apache.org
> Subject: Re: qpidd using approx 10x memory
>
> What OS version?
>
> Carl.
>
>
> On 09/08/2011 11:53 AM, Jeff Armstrong wrote:
>> Does anyone have thoughts on this? Should this be submitted in the issue tracker?
>> ________________________________________
>> From: Jeff Armstrong [jarmstrong@avvasi.com]
>> Sent: Friday, August 26, 2011 3:08 PM
>> To: users@qpid.apache.org
>> Subject: qpidd using approx 10x memory
>>
>> When filling up some queues, I noticed that the qpidd process was using about 10x the memory than was actually being used by the queues. To do a simple test on this I wrote a simple client that sends a bunch of messages that get stored in the queue, then compared the total byte depth with the memory used by qpidd.
>>
>> After sending 1 million messages, the byte count (using qpid-stat -q) showed 105MB, but checking the memory usage I saw 1.4GB (checked with htop and pmap). The messages were approximately 100 bytes large, so the byte count from qpid-stat makes sense. What doesn't make sense is that the total memory usage is over 10x as big. I checked the memory usage of qpidd before connecting with the client to send the messages and it was around 200MB, so there is still a discrepancy of about 1.1GB. These results also match with the original problem with an application I'm working on where qpidd used up all my memory (16GB) even though the queue sizes added up to about 1GB.
>>
>> I'm using version 0.10. Is this a known issue? I couldn't find it in the issue tracker. Any workarounds?
>>
>>
>> Here is how I created the exchange/queues:
>> $ ./qpid-config add exchange direct qvue
>> $ ./qpid-config add queue testQueue
>> $ ./qpid-config bind qvue testQueue testBind
>>
>>
>> Here is the client code I used:
>> #include <iostream>
>> #include <qpid/client/Message.h>
>> #include <qpid/client/Connection.h>
>> #include <qpid/client/Session.h>
>>
>> using namespace std;
>> using namespace qpid::client;
>>
>> int main(int argc, char** argv) {
>> Connection connection;
>> connection.open("127.0.0.1");
>> Session session = connection.newSession();
>>
>> int count = 0;
>> while(true) {
>> Message msg;
>> msg.setData("Hello World! 0123456789 abcdefghijklmnopqrstuvwxyz weeeeeeeeeeeeeooooooooooooooooooooooooooooooooooooooo!");
>> msg.getDeliveryProperties().setRoutingKey("testBind");
>> session.messageTransfer(qpid::client::arg::content=msg, qpid::client::arg::destination="testQueue");
>> count++;
>> if (count % 1000 == 0) {
>> cout << count << " messages sent" << endl;
>> }
>> }
>> return 0;
>> }
>>
>>
>> Thanks,
>> Jeff
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: qpidd using approx 10x memory
Posted by Jeff Armstrong <ja...@avvasi.com>.
Ubuntu 10.04
________________________________________
From: Carl Trieloff [cctrieloff@redhat.com]
Sent: Thursday, September 08, 2011 12:25 PM
To: users@qpid.apache.org
Subject: Re: qpidd using approx 10x memory
What OS version?
Carl.
On 09/08/2011 11:53 AM, Jeff Armstrong wrote:
> Does anyone have thoughts on this? Should this be submitted in the issue tracker?
> ________________________________________
> From: Jeff Armstrong [jarmstrong@avvasi.com]
> Sent: Friday, August 26, 2011 3:08 PM
> To: users@qpid.apache.org
> Subject: qpidd using approx 10x memory
>
> When filling up some queues, I noticed that the qpidd process was using about 10x the memory than was actually being used by the queues. To do a simple test on this I wrote a simple client that sends a bunch of messages that get stored in the queue, then compared the total byte depth with the memory used by qpidd.
>
> After sending 1 million messages, the byte count (using qpid-stat -q) showed 105MB, but checking the memory usage I saw 1.4GB (checked with htop and pmap). The messages were approximately 100 bytes large, so the byte count from qpid-stat makes sense. What doesn't make sense is that the total memory usage is over 10x as big. I checked the memory usage of qpidd before connecting with the client to send the messages and it was around 200MB, so there is still a discrepancy of about 1.1GB. These results also match with the original problem with an application I'm working on where qpidd used up all my memory (16GB) even though the queue sizes added up to about 1GB.
>
> I'm using version 0.10. Is this a known issue? I couldn't find it in the issue tracker. Any workarounds?
>
>
> Here is how I created the exchange/queues:
> $ ./qpid-config add exchange direct qvue
> $ ./qpid-config add queue testQueue
> $ ./qpid-config bind qvue testQueue testBind
>
>
> Here is the client code I used:
> #include <iostream>
> #include <qpid/client/Message.h>
> #include <qpid/client/Connection.h>
> #include <qpid/client/Session.h>
>
> using namespace std;
> using namespace qpid::client;
>
> int main(int argc, char** argv) {
> Connection connection;
> connection.open("127.0.0.1");
> Session session = connection.newSession();
>
> int count = 0;
> while(true) {
> Message msg;
> msg.setData("Hello World! 0123456789 abcdefghijklmnopqrstuvwxyz weeeeeeeeeeeeeooooooooooooooooooooooooooooooooooooooo!");
> msg.getDeliveryProperties().setRoutingKey("testBind");
> session.messageTransfer(qpid::client::arg::content=msg, qpid::client::arg::destination="testQueue");
> count++;
> if (count % 1000 == 0) {
> cout << count << " messages sent" << endl;
> }
> }
> return 0;
> }
>
>
> Thanks,
> Jeff
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: qpidd using approx 10x memory
Posted by Carl Trieloff <cc...@redhat.com>.
What OS version?
Carl.
On 09/08/2011 11:53 AM, Jeff Armstrong wrote:
> Does anyone have thoughts on this? Should this be submitted in the issue tracker?
> ________________________________________
> From: Jeff Armstrong [jarmstrong@avvasi.com]
> Sent: Friday, August 26, 2011 3:08 PM
> To: users@qpid.apache.org
> Subject: qpidd using approx 10x memory
>
> When filling up some queues, I noticed that the qpidd process was using about 10x the memory than was actually being used by the queues. To do a simple test on this I wrote a simple client that sends a bunch of messages that get stored in the queue, then compared the total byte depth with the memory used by qpidd.
>
> After sending 1 million messages, the byte count (using qpid-stat -q) showed 105MB, but checking the memory usage I saw 1.4GB (checked with htop and pmap). The messages were approximately 100 bytes large, so the byte count from qpid-stat makes sense. What doesn't make sense is that the total memory usage is over 10x as big. I checked the memory usage of qpidd before connecting with the client to send the messages and it was around 200MB, so there is still a discrepancy of about 1.1GB. These results also match with the original problem with an application I'm working on where qpidd used up all my memory (16GB) even though the queue sizes added up to about 1GB.
>
> I'm using version 0.10. Is this a known issue? I couldn't find it in the issue tracker. Any workarounds?
>
>
> Here is how I created the exchange/queues:
> $ ./qpid-config add exchange direct qvue
> $ ./qpid-config add queue testQueue
> $ ./qpid-config bind qvue testQueue testBind
>
>
> Here is the client code I used:
> #include <iostream>
> #include <qpid/client/Message.h>
> #include <qpid/client/Connection.h>
> #include <qpid/client/Session.h>
>
> using namespace std;
> using namespace qpid::client;
>
> int main(int argc, char** argv) {
> Connection connection;
> connection.open("127.0.0.1");
> Session session = connection.newSession();
>
> int count = 0;
> while(true) {
> Message msg;
> msg.setData("Hello World! 0123456789 abcdefghijklmnopqrstuvwxyz weeeeeeeeeeeeeooooooooooooooooooooooooooooooooooooooo!");
> msg.getDeliveryProperties().setRoutingKey("testBind");
> session.messageTransfer(qpid::client::arg::content=msg, qpid::client::arg::destination="testQueue");
> count++;
> if (count % 1000 == 0) {
> cout << count << " messages sent" << endl;
> }
> }
> return 0;
> }
>
>
> Thanks,
> Jeff
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: qpidd using approx 10x memory
Posted by Jeff Armstrong <ja...@avvasi.com>.
Does anyone have thoughts on this? Should this be submitted in the issue tracker?
________________________________________
From: Jeff Armstrong [jarmstrong@avvasi.com]
Sent: Friday, August 26, 2011 3:08 PM
To: users@qpid.apache.org
Subject: qpidd using approx 10x memory
When filling up some queues, I noticed that the qpidd process was using about 10x the memory than was actually being used by the queues. To do a simple test on this I wrote a simple client that sends a bunch of messages that get stored in the queue, then compared the total byte depth with the memory used by qpidd.
After sending 1 million messages, the byte count (using qpid-stat -q) showed 105MB, but checking the memory usage I saw 1.4GB (checked with htop and pmap). The messages were approximately 100 bytes large, so the byte count from qpid-stat makes sense. What doesn't make sense is that the total memory usage is over 10x as big. I checked the memory usage of qpidd before connecting with the client to send the messages and it was around 200MB, so there is still a discrepancy of about 1.1GB. These results also match with the original problem with an application I'm working on where qpidd used up all my memory (16GB) even though the queue sizes added up to about 1GB.
I'm using version 0.10. Is this a known issue? I couldn't find it in the issue tracker. Any workarounds?
Here is how I created the exchange/queues:
$ ./qpid-config add exchange direct qvue
$ ./qpid-config add queue testQueue
$ ./qpid-config bind qvue testQueue testBind
Here is the client code I used:
#include <iostream>
#include <qpid/client/Message.h>
#include <qpid/client/Connection.h>
#include <qpid/client/Session.h>
using namespace std;
using namespace qpid::client;
int main(int argc, char** argv) {
Connection connection;
connection.open("127.0.0.1");
Session session = connection.newSession();
int count = 0;
while(true) {
Message msg;
msg.setData("Hello World! 0123456789 abcdefghijklmnopqrstuvwxyz weeeeeeeeeeeeeooooooooooooooooooooooooooooooooooooooo!");
msg.getDeliveryProperties().setRoutingKey("testBind");
session.messageTransfer(qpid::client::arg::content=msg, qpid::client::arg::destination="testQueue");
count++;
if (count % 1000 == 0) {
cout << count << " messages sent" << endl;
}
}
return 0;
}
Thanks,
Jeff
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org