High memory utilization with Java version of Qpid 7.0.6

[Brian O'Shea <bo...@salesforce.com>]

Hello Qpid Community,

I am seeing some strange memory utilization patterns when running the Java
implementation of Qpid, version 7.0.6 on Linux (Ubuntu 16.04).  I ran some
tests for the application that I am working on that uses Qpid, which
enqueues about 50 messages.  I see this very high memory usage by Qpid while
running these tests (see attached image).

My Qpid memory properties are set to the following:

# Java heap settings
qpid.java.min.memory     = 3072m
qpid.java.max.memory     = 3072m

qpid.java.direct.memory  = 9216m
qpid.flow_to_disk_threshold = 6606029

Several of our tests fail, but if I re-run them all, different ones will
pass and fail, so I suspect it is related to when the memory utilization is
high.  In production, we enqueue many more messages (millions per day), so I
am concerned that with this version of Qpid and/or with this combination of
memory settings, it may not scale.  Can you help me compute the optimal
memory settings for this application?  Let me know if you need to know
anything else about the way that we will be using Qpid.

Thanks,
Brian




--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: High memory utilization with Java version of Qpid 7.0.6

[Keith W <ke...@gmail.com>]

Hello

There is general advice [1] for tuning memory sizes and formulae to
help you size direct and heap memory appropriately for your use-case.

As you'll read, Broker-J keeps a message's payload and headers in
direct memory for performance reasons.  If direct memory comes under
pressure, it has an algorithm to evict the cached payload and headers
and write them to disk (if transient, persistent ones already are).
The use of direct memory was new with 6.0.0. In the heap, the Broker
maintains a linked list representing the entire queue.   This is a
light weight structure with pointers to the message.  The Broker
currently has no strategy to page out chunks of this linked list, so
it is possible just to exhaust heap by having a long queue.   This
aspect was exactly the same in 0.32 and below.   The Broker's defence
against this situation is producer flow control.  The controls for
this have become richer in the newer versions.

I am not certain that the graph that you share in itself actually
indicates a problem.   Might this not just be a normal saw-tooth GC
pattern?  Are you actually going OOME (heap).   There is heap garbage
created as messages traverse the Broker, so the JVM will collect this
when it sees fit.  You would have seen a similar pattern with 0.32 and
before.

Hope this helps

[1] https://qpid.apache.org/releases/qpid-broker-j-7.0.6/book/Java-Broker-Runtime-Memory.html
On Mon, 15 Oct 2018 at 21:01, Brian O'Shea <bo...@salesforce.com> wrote:
>
> Hello Qpid Community,
>
> I am seeing some strange memory utilization patterns when running the Java
> implementation of Qpid, version 7.0.6 on Linux (Ubuntu 16.04).  I ran some
> tests for the application that I am working on that uses Qpid, which
> enqueues about 50 messages.  I see this very high memory usage by Qpid while
> running these tests (see attached image).
>
> My Qpid memory properties are set to the following:
>
> # Java heap settings
> qpid.java.min.memory     = 3072m
> qpid.java.max.memory     = 3072m
>
> qpid.java.direct.memory  = 9216m
> qpid.flow_to_disk_threshold = 6606029
>
> Several of our tests fail, but if I re-run them all, different ones will
> pass and fail, so I suspect it is related to when the memory utilization is
> high.  In production, we enqueue many more messages (millions per day), so I
> am concerned that with this version of Qpid and/or with this combination of
> memory settings, it may not scale.  Can you help me compute the optimal
> memory settings for this application?  Let me know if you need to know
> anything else about the way that we will be using Qpid.
>
> Thanks,
> Brian
>
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org