You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by jbelch <ja...@verizon.net> on 2018/02/27 14:49:46 UTC
qpidd C++ broker memory leak on Redhat 6.8?
I have used the qpidd C++ broker on several projects over the years. I
started using 0.7 about 6 years ago, used 0.20 about 4 years ago, and I am
currently using 0.34. I have noticed a memory leak when running on the
system. qpidd starts out at about 10mb resident memory and the resident
memory usage increases about 30mb each day. After 75 days, we are using
resident memory in the 3gb range. We have a mix of durable queues and
topics, probably about 20 total. When I use the qpid-stat script, there
don't appear to be any queues or topics with messages hanging around. When
I run valgrind, it gives me a report that states there are no memory leaks.
Does anyone have any idea?
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpidd C++ broker memory leak on Redhat 6.8?
Posted by Virgilio Fornazin <vi...@gmail.com>.
I've used RHEL MRG (Qpid) in the last 8 years, and what I could say is that
qpidd C++ broker never releases back memory to SO.
I've found that the queue code use std::vector in some paths, and need to
call vector.shrink_to_fit() because std::vector (and some other C++ STL
container) never releases back memory, only when destructed. I didn't
verified if applying a patch that call shrink_to_fit when the queue is
empty could help, but is a good point to start checking.
On Tue, Feb 27, 2018 at 7:12 PM, CLIVE <cl...@ckjltd.co.uk> wrote:
> Hi,
>
> I have had a similar problem with a production QPID system, running Cento
> 6.8 for the past 14 months.
>
> I tried the good advice that Ted gave, using the malloc tuning environment
> variables, but unfortunately
> this made no difference to our environment. We would get about 5 days out
> of the system until the broker
> used up all 64G of memory and got killed by the kernels OOM process.
>
> I even tried using the gperftools tcmalloc library, but again this has
> made no difference to the memory consumption.
>
> Even moving up from version 0.34-1.36 didn't seem to make any difference.
> Then about 8 weeks ago, out of the blue, the
> problem just went away. The QPID broker is now staying solid at 5% memory
> usage. We have not changed our usage pattern
> in any way, still the same number of consumers and producers, but our
> message flow rate has increased.
>
> I have tried everything to replicate this issue, but to no avail.
> Interestingly the problem only affects our production
> system, all or other environments are fine. This leads me to believe it is
> somehow flow related, which I know sounds
> a bit stupid, but we had been running in production for about a year
> before the memory problem arose (at a point in time when the
> flow rates increased).
>
> Then again 8 weeks ago the system encountered another increased step
> change in message rates and the memory problem has now gone away (for the
> time being!)
>
> I don't know how flow rate would affect the memory in this way, hopefully
> someone else might have some ideas.
>
> The other area I looked at was message size (we have a wide range of
> message sizes in our system (1k-10M), but I could never replicate the issue.
>
> Sorry I cannot to be of more help, but if you do uncover the problem I
> would be very interested to hear about it.
>
> Clive
>
>
> On 27/02/2018 20:36, jbelch wrote:
>
>> Thanks. I will try it and let you know if it works.
>>
>>
>>
>> --
>> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936
>> .html
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: qpidd C++ broker memory leak on Redhat 6.8?
Posted by CLIVE <cl...@ckjltd.co.uk>.
Hi,
I have had a similar problem with a production QPID system, running
Cento 6.8 for the past 14 months.
I tried the good advice that Ted gave, using the malloc tuning
environment variables, but unfortunately
this made no difference to our environment. We would get about 5 days
out of the system until the broker
used up all 64G of memory and got killed by the kernels OOM process.
I even tried using the gperftools tcmalloc library, but again this has
made no difference to the memory consumption.
Even moving up from version 0.34-1.36 didn't seem to make any
difference. Then about 8 weeks ago, out of the blue, the
problem just went away. The QPID broker is now staying solid at 5%
memory usage. We have not changed our usage pattern
in any way, still the same number of consumers and producers, but our
message flow rate has increased.
I have tried everything to replicate this issue, but to no avail.
Interestingly the problem only affects our production
system, all or other environments are fine. This leads me to believe it
is somehow flow related, which I know sounds
a bit stupid, but we had been running in production for about a year
before the memory problem arose (at a point in time when the
flow rates increased).
Then again 8 weeks ago the system encountered another increased step
change in message rates and the memory problem has now gone away (for
the time being!)
I don't know how flow rate would affect the memory in this way,
hopefully someone else might have some ideas.
The other area I looked at was message size (we have a wide range of
message sizes in our system (1k-10M), but I could never replicate the issue.
Sorry I cannot to be of more help, but if you do uncover the problem I
would be very interested to hear about it.
Clive
On 27/02/2018 20:36, jbelch wrote:
> Thanks. I will try it and let you know if it works.
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpidd C++ broker memory leak on Redhat 6.8?
Posted by jbelch <ja...@verizon.net>.
Unfortunately, we only have 6.8 available. I will see if there is anyway to
get a RHEL7 machine setup.
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpidd C++ broker memory leak on Redhat 6.8?
Posted by Ted Ross <tr...@redhat.com>.
The problem that I'm familiar with is specific to RHEL6. I'm not
aware of any available patch for this issue (if that's even what you
are experiencing). Do you have the ability to test using RHEL7?
On Mon, Mar 5, 2018 at 3:09 PM, jbelch <ja...@verizon.net> wrote:
> Any other thoughts? It doesn't seem to be an issue on earlier versions of
> glibc. Is that correct? If so, is there an OS patch or something? We may
> have to abandon the qpid broker and go to something else if we can't figure
> this out.
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpidd C++ broker memory leak on Redhat 6.8?
Posted by jbelch <ja...@verizon.net>.
Any other thoughts? It doesn't seem to be an issue on earlier versions of
glibc. Is that correct? If so, is there an OS patch or something? We may
have to abandon the qpid broker and go to something else if we can't figure
this out.
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpidd C++ broker memory leak on Redhat 6.8?
Posted by jbelch <ja...@verizon.net>.
I ran overnight and the leak still appears to be present after setting the
MALLOC_ARENA_MAX environment variable.
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpidd C++ broker memory leak on Redhat 6.8?
Posted by jbelch <ja...@verizon.net>.
Thanks. I will try it and let you know if it works.
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpidd C++ broker memory leak on Redhat 6.8?
Posted by Ted Ross <tr...@redhat.com>.
This is an _old_ post that addresses a similar issue and proposes a
solution. I believe this was specific to RHEL 6.
http://qpid.2158936.n2.nabble.com/qpidd-using-approx-10x-memory-tp6730073p6775634.html
-Ted
On Tue, Feb 27, 2018 at 9:49 AM, jbelch <ja...@verizon.net> wrote:
> I have used the qpidd C++ broker on several projects over the years. I
> started using 0.7 about 6 years ago, used 0.20 about 4 years ago, and I am
> currently using 0.34. I have noticed a memory leak when running on the
> system. qpidd starts out at about 10mb resident memory and the resident
> memory usage increases about 30mb each day. After 75 days, we are using
> resident memory in the 3gb range. We have a mix of durable queues and
> topics, probably about 20 total. When I use the qpid-stat script, there
> don't appear to be any queues or topics with messages hanging around. When
> I run valgrind, it gives me a report that states there are no memory leaks.
> Does anyone have any idea?
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org