[jira] Updated: (QPID-32) Change message handling in broker so that only a minimal dataset must be kept in memory

["Robert Greig (JIRA)" <qp...@incubator.apache.org>]

     [ https://issues.apache.org/jira/browse/QPID-32?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Greig updated QPID-32:
-----------------------------

    Summary: Change message handling in broker so that only a minimal dataset must be kept in memory  (was: Amend persistence algorithm to introduce paging to physical memory)

More meaningful title

> Change message handling in broker so that only a minimal dataset must be kept in memory
> ---------------------------------------------------------------------------------------
>
>                 Key: QPID-32
>                 URL: https://issues.apache.org/jira/browse/QPID-32
>             Project: Qpid
>          Issue Type: Improvement
>          Components: Java Broker
>            Reporter: Marnie McCormack
>         Assigned To: Robert Greig
>             Fix For: M2
>
>
> Currently all messages, including the headers and body, are held in memory in the broker. This means that even with persistent queues, all messages must be able to fit in the heap. Clearly that is a severe limitation. 
> This effectively means that there is an upper limit for the number of messages a broker can handle, bound by the heap size available and defined by the size of the messages in transit.
> As we know that there are users who wish to combine high volume throughput with large message size (100Mb) we can expect that this limitation will be constraining fairly quickly, given finite available memory. We have had requests for the broker to be able to handle 5 million messages at any one time, with message size being frmo small (a few kb) to large (hundreds of Mb).
> We need to introduce a paging algorithm into the broker such that it can page messages in & out of memory as appropriate during transit. The broker needs to be enhanced so that messages do not need to be held in memory at all times. Ideally the redesign of this aspect will support flow-to-disk for non-persistent messages when configurable thresholds are reached.
> This may also affect transactional messages, persistent or not, as they are also held in memory during transit and thus are affected by the same constraint.
> The design of the solution for memory management is critical for the performance of the broker and should be published and reviewed, as well as having defined performance benchmarks to meet during testing. We need to be very confident that we're putting in a solution which is quick and efficient.
> We may also want to consider the configuration options available for memory management - allowing our users to tune deployed brokers in a way appropriate for their application profile i.e. if the app is sending large messages infrequently then the fastest solution could potentially differ from the appropriate model for an application sending mainly small messages and occasional very large messages.
> Also need to consider the impact of priority/qos settings on messages when designing the solution. 
> The assignee for this work should expand the requirements in more detail once analysis is complete, prior to completion of development.
> Please add/amend the detail here if you have hands on experience of the existing persistence code. Thanks.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.