[jira] Commented: (QPID-949) Implement Flow To Disk

["Aidan Skinner (JIRA)" <qp...@incubator.apache.org>]

    [ https://issues.apache.org/jira/browse/QPID-949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664565#action_12664565 ] 

Aidan Skinner commented on QPID-949:
------------------------------------

There are several changes necessary to implement this. Message payload storage should be seperated from metadata storage. Persistent messages should be written to disk immediately as they are recieved.

When memory usages reaches a threshold, in-memory persistant message payloads could be GC'd from (ideally the back of) queues until pressure has been relieved. These would be reloaded from disk as they were sent. The existing weak-ref stuff would work here. 

If this dump did not release sufficent memory transient messages would be written to the disk, again ideally from the back of queues and then reloaded. 

Internally, messages would hold a reference to a file in a shallow, wide hierarchy of files which would contain the message payload. These files would be deleted when the messages are sent or expired.

On startup, after the metadata had been read to restore the state of the broker the filesystem would be scanned and files which were not currently referenced would be deleted. This would clean up any transient messages that had been written to disk before the broker stopped running but not yet deleted. 


> Implement Flow To Disk
> ----------------------
>
>                 Key: QPID-949
>                 URL: https://issues.apache.org/jira/browse/QPID-949
>             Project: Qpid
>          Issue Type: New Feature
>          Components: Java Broker
>    Affects Versions: M2.1
>            Reporter: Marnie McCormack
>            Assignee: Rob Godfrey
>             Fix For: M5
>
>
> Currently, the Java Broker can do one of two things with a message it has to deliver: 
> 1. Keep transient messages in memory until delivered
> 2. Write persistent messages to a message store (like BDB) and keep in memory until delivery complete
> This means that the broker is not able to avoid OoM exceptions i.e. send enough messages to the broker, especially if your consumers are not active, and you could bring the broker done once it explodes its available heap.
> RG to add more details and design proposal here please :-)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.