[jira] [Commented] (QPID-7753) Sparsely occupied message buffers may lead to java.lang.OutOfMemoryError: Direct buffer memory

["Rob Godfrey (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/QPID-7753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988875#comment-15988875 ] 

Rob Godfrey commented on QPID-7753:
-----------------------------------

The commit https://git1-us-west.apache.org/repos/asf?p=qpid-broker-j.git;a=commit;h=36bab8f3 (which as of writing this comment does not seem to have automatically been registered against this JIRA) is not thread safe I think.

QpidByteBuffer.reallocateIfNecessary(...) takes a collection of byte buffers and returns a new collection where some buffers may have been replaced.  Buffers from the input collection may have been disposed of.

This would only be safe if the caller of QpidByteBuffer.reallocateIfNecessary does not expose a reference to or otherwise access the original collection (or the buffers within it) while QpidByteBuffer.reallocateIfNecessary is being called.  This can be avoided if the caller strictly observes the above implied contract, though violation of this contract would be easy to miss.  In the commit referenced I think, at the very least, the contract is not obeyed in AbstractSection, there may be other places but I have not checked carefully

> Sparsely occupied message buffers may lead to java.lang.OutOfMemoryError: Direct buffer memory
> ----------------------------------------------------------------------------------------------
>
>                 Key: QPID-7753
>                 URL: https://issues.apache.org/jira/browse/QPID-7753
>             Project: Qpid
>          Issue Type: Bug
>          Components: Java Broker
>    Affects Versions: qpid-java-6.0, qpid-java-6.1
>            Reporter: Keith Wall
>            Assignee: Keith Wall
>
> Some Broker usage patterns can lead to the Broker failing with a "java.lang.OutOfMemoryError: Direct buffer memory" error.
> For the condition to manifest a producing application needs to use a single connection for the production of messages. Some messages need to be consumed quickly whilst others remain on the Broker. This pattern might result from:
> # the consuming application using message selectors to selectively consume some messages whilst leaving others on the Broker.
> # the use of 'out of order' queue types (priority/sorted etc) where lower priority items are left of the Broker.
> # the producing application routing messages to multiple queues and the consuming application draining some queues but not others.
> The problem arises owing to the buffering strategy under the IO layer.
> {{NonBlockingConnection}} allocates a {{netInputBuffer}} from pooled direct memory of size 256K.  This buffer is used for all network reads until the space remaining in the buffer is less than the amount required to complete the AMQP frame that is currently being read, in which case a new netInputBuffer is allocated. The protocol layers identify the message payload/message headers as part of AMQP frame parsing and create a byte-buffer *views* onto the original input byte buffer.  This byte buffer is retained by the store until the message is consumed.   In the usage pattern described above, a single long lived message amongst a stream of shorted lived messages causes an entire 256K buffer chunk to be retained.  Qpid cannot dispose or reuse the buffer until it is entirely unoccupied.
> The flow to disk feature is designed to prevent excessive direct memory use by flushing messages to disk if thresholds are breached.  Flow to disk does not help the scenario described above because it considers the total payloads of live messages.  Its algorithm does not consider direct memory occupancy.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org