QPID C++ Broker MALLOC_ARENA_MAX

[CLIVE <cl...@ckjltd.co.uk>]

Hi,

Been a while since I last posted anything to the QPID newsgroup, mainly 
due to the excellent reliability of the QPID C++ broker, keep up the 
good work.

But I am seeing a strange issue at a clients site that i thought I would 
share with the community.

A client is running a QPID C++  Broker (version 0.32) on a CENTOS 6.7 
virtualized platform (8 CPU, 32 cores, and 64G RAM) and is experiencing 
memory exhaustion problems. Over the course of 5-30 days the broker 
resident memory steadily climbs up until it exhausts the available 
memory and gets killed by the kernels OOM. The memory pattern follows 
that of a form of memory leak, but I've never seen this kind of behavior 
before from a QPID C++ broker, and looking on JIRA that doesn't seem to 
be any known memory leak issues.

The broker is running 10 threads, currently supporting 134 long lived 
connections, from a range of JAVA JMS (Apache Camel), C++ and Python 
Clients, with 25 user defined exchanges and about 100 durable ring 
queues All messages are transient.  About 20GBytes of data is pushed 
through the broker each data ranging from small little messages of 1K, 
to messages of around 100K.

As the broker memory consumption climbs, a 'qpid-stat -g' gives a steady 
state queue depth of about 125,000 messages totaling 660M-1Gbytes of 
memory. So its not a queue depth issue.

Interestingly when I run pmap -x <qpid pid> I see lots and lots of 
64MBytes allocations (about 400) with 300 additional allocations of just 
under 64MBytes.

Some searching on the web has turned up a potential candidate for the 
memory consumption issue, associated with the design change that was 
made to the glibc malloc implementation in glibc 2.10+ which introduced 
memory arenas to reduce memory contention in multi-threaded processes. 
The malloc implementation uses some basic math to work out how much 
total memory is allocated to a process, no of cores * sizeof(long) * 
64Mb. So for our 64 bit system that would give 32*8*64Mb = 16G.

Apparently other products have had similar memory issues when they moved 
to RHEL 6 (CENTOS 6), from RHEL 5, as the newer OS used glibc 2.12. The 
use of the MALLOC_ARENA_MAX environment variable seems to be away of 
reducing the memory allocated to the process with a suggested value of 4.

Just wondered if any one else in the community had experienced a similar 
kind of broker memory issue, and what advice, if any could be supplied 
to localize the problem and stop the broker chewing through 64G of RAM.

Any help/advice gratefully appreciated.

Clive




---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org