You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "Kim van der Riet (JIRA)" <qp...@incubator.apache.org> on 2010/04/15 17:42:48 UTC
[jira] Created: (QPID-2509) Remove message staging from C++ broker
Remove message staging from C++ broker
--------------------------------------
Key: QPID-2509
URL: https://issues.apache.org/jira/browse/QPID-2509
Project: Qpid
Issue Type: Improvement
Components: C++ Broker
Reporter: Kim van der Riet
Assignee: Kim van der Riet
Staging was originally implemented so that a message with content that exceeds available memory can be handled. The message is sent to disk in parts as it arrives, and is never fully resident in memory. However, the current implementation of staging is problematic:
1. The current architecture requires the entire message to be loaded into memory before it can be delivered. This renders the staging facility superfluous. In fact, it can lead to a problem where it is possible to enqueue messages much larger than the available memory, but never be able to dequeue it again.
2. The current implementation uses a BDB database to save the message. This is done under a transaction to ensure atomicity of the operation. This results in two issues:
a) BDB is very slow for large messages, and gets exponentially slower as the message size grows.
b) The database leaves behind a large number of log files that are not currently managed and tend to accumulate.
The staging feature is still a valuable one, and the issue of reinstating staging should be revisited when the fundamental problem (1. above) has been corrected. The use of BDB is an implementation issue and can easily be changed for something more suitable.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
[jira] Resolved: (QPID-2509) Remove message staging from C++ broker
Posted by "Kim van der Riet (JIRA)" <qp...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/QPID-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kim van der Riet resolved QPID-2509.
------------------------------------
Resolution: Fixed
The broker option for setting the staging threshold was removed, as was the staging code and the related staging tests.
> Remove message staging from C++ broker
> --------------------------------------
>
> Key: QPID-2509
> URL: https://issues.apache.org/jira/browse/QPID-2509
> Project: Qpid
> Issue Type: Improvement
> Components: C++ Broker
> Reporter: Kim van der Riet
> Assignee: Kim van der Riet
>
> Staging was originally implemented so that a message with content that exceeds available memory can be handled. The message is sent to disk in parts as it arrives, and is never fully resident in memory. However, the current implementation of staging is problematic:
> 1. The current architecture requires the entire message to be loaded into memory before it can be delivered. This renders the staging facility superfluous. In fact, it can lead to a problem where it is possible to enqueue messages much larger than the available memory, but never be able to dequeue it again.
> 2. The current implementation uses a BDB database to save the message. This is done under a transaction to ensure atomicity of the operation. This results in two issues:
> a) BDB is very slow for large messages, and gets exponentially slower as the message size grows.
> b) The database leaves behind a large number of log files that are not currently managed and tend to accumulate.
> The staging feature is still a valuable one, and the issue of reinstating staging should be revisited when the fundamental problem (1. above) has been corrected. The use of BDB is an implementation issue and can easily be changed for something more suitable.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
[jira] Commented: (QPID-2509) Remove message staging from C++
broker
Posted by "Kim van der Riet (JIRA)" <qp...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/QPID-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857369#action_12857369 ]
Kim van der Riet commented on QPID-2509:
----------------------------------------
Fixed in r.934463.
> Remove message staging from C++ broker
> --------------------------------------
>
> Key: QPID-2509
> URL: https://issues.apache.org/jira/browse/QPID-2509
> Project: Qpid
> Issue Type: Improvement
> Components: C++ Broker
> Reporter: Kim van der Riet
> Assignee: Kim van der Riet
>
> Staging was originally implemented so that a message with content that exceeds available memory can be handled. The message is sent to disk in parts as it arrives, and is never fully resident in memory. However, the current implementation of staging is problematic:
> 1. The current architecture requires the entire message to be loaded into memory before it can be delivered. This renders the staging facility superfluous. In fact, it can lead to a problem where it is possible to enqueue messages much larger than the available memory, but never be able to dequeue it again.
> 2. The current implementation uses a BDB database to save the message. This is done under a transaction to ensure atomicity of the operation. This results in two issues:
> a) BDB is very slow for large messages, and gets exponentially slower as the message size grows.
> b) The database leaves behind a large number of log files that are not currently managed and tend to accumulate.
> The staging feature is still a valuable one, and the issue of reinstating staging should be revisited when the fundamental problem (1. above) has been corrected. The use of BDB is an implementation issue and can easily be changed for something more suitable.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org