You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Cullen Davis <cu...@commitent.com> on 2009/11/20 21:50:25 UTC
Request to Add autoDelete Parameter to qpid-route command
Synopsis:
Dynamic federation does not guarantee delivery of messages. Setting autoDelete=false on the “transit-queue” during dynamic federation queue creation would eliminate the issue.
Summary:
Our solution utilizes Qpid 0.5 C++ brokers and clients running on Red Hat Enterprise Linux 5.4. We are a government backed project that must deploy onto networks that are low bandwidth, high latency, and intermittently disconnected. Guaranteed delivery of the messages is a fundamental requirement. Our solution depends on brokers using direct exchanges and being dynamically federated. Using dynamic federation, when network connectivity is lost for more than seven minutes, all messages awaiting federation are deleted. At re-connect, the federation routes are rebuilt but the unsent messages are not restored. (Originally Stated in users mailing list on Thu, 05 Nov, 16:55)
Change Request:
Allow Dynamic Federation to create “transit-queues” with autoDelete=False. We would like to request a new option, autoDelete, on the qpid-route command. In our case, we would run
qpid-route --durable --autodelete=false dynamic add brokerB brokerA fed.direct
============
This would be used to create exchange bindings with transit-queues having the autoDelete property set to false.
Details:
We federate our brokers as follows:
qpid-route --durable dynamic add brokerB brokerA fed.direct
The dynamic option causes the qpid-route command to create a new “transit-queue”, with the name "bridge-queue" at brokerA. The bridge-queue serves as the means to move messages from brokerA to brokerB. The new queue had queue properties of durable=False, exclusive=True and autoDelete=True.
If the network connection is broken for more than seven minutes, the session on brokerB subscribing to the "bridge-queue" on brokerA is considered to have ended. The loss of the session subscriber triggers the the autoDelete of the bridge-queue. The result is that all messages awaiting federation via the bridge queue are deleted (NOT persisted at all).
Ted Ross suggested a work around. He suggested we create a “queue” route where the destination broker subscribes to an existing queue on the source broker. This should cause the inter-broker route to use message acknowledgement in such a way that recovery will be clean.
We encountered the following issues with the queue route:
1) “Out-of-order” message delivery was experienced
2) The first 1000KB of message data (in our case 10 messages) were not delivered until the broker process was restarted. This sounds crazy but we replicated the case multiple times.
3) Our solution loses the ability to do dynamic binding.
Our customer needs to have the in-order, guaranteed delivery of messages on low quality networks.
How should we proceed with this request?
Cullen Davis
CommIT Enterprises, Inc
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Request to Add autoDelete Parameter to qpid-route command
Posted by Gordon Sim <gs...@redhat.com>.
Apologies for delayed response, I was away on holiday.
On 11/24/2009 04:54 PM, Cullen Davis wrote:
>
>> I'd argue that if you are specifying durable that implies you don't want
>> auto deletion (durable is really only of any value in recovering from
>> loss of in-memory state and in that case you would have lost the session
>> and triggered auto-delete anyway). (The attached trivial patch would do
>> this I believe, though perhaps it unnecessarily couples the durability
>> of the bridge meta-data with the durability of the queue used in which
>> case a 'reliable' option might be preferred).
>
> Thank you for the reply Gordon - you really gave me a lot to think about. I realized that I still don't fully understand the intent of --durable.
>
> Our requirement for the federated brokers are:
> - Federated messages have a finite lifetime.
> - A federated connection must provide for a route that survives a broker restart.
> - All federated messages pending a dequeue must persist across a broker restart.
> - All federated messages pending a dequeue must persist across an extended network outage.
>
> We used the --durable option on the qpid-route command in so that the route would survive broker restarts. Is this inconsistent with our requirement?
No, that is what it is intended for. My patch just used it as
additionally implying that the queue used in that case should also be
durable (and it may perhaps be better to instead have a separate
parameter to the bridge controlling the queues durability).
> We set the persistent property on the messages to provide for the queue state to be journaled so that messages survive restarts.
> I expect the patch to allow the bridge-queue from being deleted. (the patch code should be in a test environment shortly)
Did that patch work for you as expected?
>> Having a timeout for a queue such that if there is no subscriber for x
>> seconds then the queue gets deleted might be a useful addition. That
>> would allow some protection over the case where a failed link never
>> comes back (or doesn't come back fast enough to avoid a critical build
>> up of messages).
>
> To provide for the case where a link never comes back up, I would expect the application to provide a system configuration such that the qpid journal files would be capable of storing a typical “pending connect” queue load (based on message frequency and time-to-live).
Yes the ttl on the messages will help control the size of the queue when
disconnected for longer periods.
> To go one step further, could the “bridge-queue” be created such that it could have a a policy type of RING?
It's certainly possible. You could do it already for a static route. For
a dynamic route there would need to be some way of specifying the
'policy' for the queue.
> In this way, the onus of message capacity and availability in disconnected environments is placed with the application owners configuration.
>
> Your thoughts would be greatly appreciated.
>
> Cullen J. Davis
> CommIT Enterprises, Inc.
>
> ________________________________________
> From: Gordon Sim [gsim@redhat.com]
> Sent: Monday, November 23, 2009 7:16 AM
> To: users@qpid.apache.org
> Subject: Re: Request to Add autoDelete Parameter to qpid-route command
>
> On 11/20/2009 08:50 PM, Cullen Davis wrote:
>> Synopsis:
>> Dynamic federation does not guarantee delivery of messages. Setting autoDelete=false on the “transit-queue” during dynamic federation queue creation would eliminate the issue.
>>
>> Summary:
>> Our solution utilizes Qpid 0.5 C++ brokers and clients running on Red Hat Enterprise Linux 5.4. We are a government backed project that must deploy onto networks that are low bandwidth, high latency, and intermittently disconnected. Guaranteed delivery of the messages is a fundamental requirement. Our solution depends on brokers using direct exchanges and being dynamically federated. Using dynamic federation, when network connectivity is lost for more than seven minutes, all messages awaiting federation are deleted. At re-connect, the federation routes are rebuilt but the unsent messages are not restored. (Originally Stated in users mailing list on Thu, 05 Nov, 16:55)
>>
>> Change Request:
>> Allow Dynamic Federation to create “transit-queues” with autoDelete=False. We would like to request a new option, autoDelete, on the qpid-route command. In our case, we would run
>> qpid-route --durable --autodelete=false dynamic add brokerB brokerA fed.direct
>> ============
>> This would be used to create exchange bindings with transit-queues having the autoDelete property set to false.
>
> I'd argue that if you are specifying durable that implies you don't want
> auto deletion (durable is really only of any value in recovering from
> loss of in-memory state and in that case you would have lost the session
> and triggered auto-delete anyway). (The attached trivial patch would do
> this I believe, though perhaps it unnecessarily couples the durability
> of the bridge meta-data with the durability of the queue used in which
> case a 'reliable' option might be prefered).
>
> Having a timeout for a queue such that if there is no subscriber for x
> seconds then the queue gets deleted might be a useful addition. That
> would allow some protection over the case where a failed link never
> comes back (or doesn't come back fast enough to avoid a critical build
> up of messages).
>
>> Details:
>> We federate our brokers as follows:
>> qpid-route --durable dynamic add brokerB brokerA fed.direct
>>
>> The dynamic option causes the qpid-route command to create a new “transit-queue”, with the name "bridge-queue" at brokerA. The bridge-queue serves as the means to move messages from brokerA to brokerB. The new queue had queue properties of durable=False, exclusive=True and autoDelete=True.
>>
>> If the network connection is broken for more than seven minutes, the session on brokerB subscribing to the "bridge-queue" on brokerA is considered to have ended. The loss of the session subscriber triggers the the autoDelete of the bridge-queue. The result is that all messages awaiting federation via the bridge queue are deleted (NOT persisted at all).
>>
>> Ted Ross suggested a work around. He suggested we create a “queue” route where the destination broker subscribes to an existing queue on the source broker. This should cause the inter-broker route to use message acknowledgement in such a way that recovery will be clean.
>>
>> We encountered the following issues with the queue route:
>> 1) “Out-of-order” message delivery was experienced
>> 2) The first 1000KB of message data (in our case 10 messages) were not delivered until the broker process was restarted. This sounds crazy but we replicated the case multiple times.
>
> That sounds like a session is not being correctly cleaned up perhaps and
> there are some unacked messages in a locked state. Would you be willing
> to raise a Jira for this and list the steps taken to reproduce?
>
>> 3) Our solution loses the ability to do dynamic binding.
>>
>> Our customer needs to have the in-order, guaranteed delivery of messages on low quality networks.
>>
>> How should we proceed with this request?
>>
>>
>> Cullen Davis
>> CommIT Enterprises, Inc
>>
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: Request to Add autoDelete Parameter to qpid-route command
Posted by Cullen Davis <cu...@commitent.com>.
> I'd argue that if you are specifying durable that implies you don't want
> auto deletion (durable is really only of any value in recovering from
> loss of in-memory state and in that case you would have lost the session
> and triggered auto-delete anyway). (The attached trivial patch would do
> this I believe, though perhaps it unnecessarily couples the durability
> of the bridge meta-data with the durability of the queue used in which
> case a 'reliable' option might be preferred).
Thank you for the reply Gordon - you really gave me a lot to think about. I realized that I still don't fully understand the intent of --durable.
Our requirement for the federated brokers are:
- Federated messages have a finite lifetime.
- A federated connection must provide for a route that survives a broker restart.
- All federated messages pending a dequeue must persist across a broker restart.
- All federated messages pending a dequeue must persist across an extended network outage.
We used the --durable option on the qpid-route command in so that the route would survive broker restarts. Is this inconsistent with our requirement?
We set the persistent property on the messages to provide for the queue state to be journaled so that messages survive restarts.
I expect the patch to allow the bridge-queue from being deleted. (the patch code should be in a test environment shortly)
> Having a timeout for a queue such that if there is no subscriber for x
> seconds then the queue gets deleted might be a useful addition. That
> would allow some protection over the case where a failed link never
> comes back (or doesn't come back fast enough to avoid a critical build
> up of messages).
To provide for the case where a link never comes back up, I would expect the application to provide a system configuration such that the qpid journal files would be capable of storing a typical “pending connect” queue load (based on message frequency and time-to-live). To go one step further, could the “bridge-queue” be created such that it could have a a policy type of RING? In this way, the onus of message capacity and availability in disconnected environments is placed with the application owners configuration.
Your thoughts would be greatly appreciated.
Cullen J. Davis
CommIT Enterprises, Inc.
________________________________________
From: Gordon Sim [gsim@redhat.com]
Sent: Monday, November 23, 2009 7:16 AM
To: users@qpid.apache.org
Subject: Re: Request to Add autoDelete Parameter to qpid-route command
On 11/20/2009 08:50 PM, Cullen Davis wrote:
> Synopsis:
> Dynamic federation does not guarantee delivery of messages. Setting autoDelete=false on the “transit-queue” during dynamic federation queue creation would eliminate the issue.
>
> Summary:
> Our solution utilizes Qpid 0.5 C++ brokers and clients running on Red Hat Enterprise Linux 5.4. We are a government backed project that must deploy onto networks that are low bandwidth, high latency, and intermittently disconnected. Guaranteed delivery of the messages is a fundamental requirement. Our solution depends on brokers using direct exchanges and being dynamically federated. Using dynamic federation, when network connectivity is lost for more than seven minutes, all messages awaiting federation are deleted. At re-connect, the federation routes are rebuilt but the unsent messages are not restored. (Originally Stated in users mailing list on Thu, 05 Nov, 16:55)
>
> Change Request:
> Allow Dynamic Federation to create “transit-queues” with autoDelete=False. We would like to request a new option, autoDelete, on the qpid-route command. In our case, we would run
> qpid-route --durable --autodelete=false dynamic add brokerB brokerA fed.direct
> ============
> This would be used to create exchange bindings with transit-queues having the autoDelete property set to false.
I'd argue that if you are specifying durable that implies you don't want
auto deletion (durable is really only of any value in recovering from
loss of in-memory state and in that case you would have lost the session
and triggered auto-delete anyway). (The attached trivial patch would do
this I believe, though perhaps it unnecessarily couples the durability
of the bridge meta-data with the durability of the queue used in which
case a 'reliable' option might be prefered).
Having a timeout for a queue such that if there is no subscriber for x
seconds then the queue gets deleted might be a useful addition. That
would allow some protection over the case where a failed link never
comes back (or doesn't come back fast enough to avoid a critical build
up of messages).
> Details:
> We federate our brokers as follows:
> qpid-route --durable dynamic add brokerB brokerA fed.direct
>
> The dynamic option causes the qpid-route command to create a new “transit-queue”, with the name "bridge-queue" at brokerA. The bridge-queue serves as the means to move messages from brokerA to brokerB. The new queue had queue properties of durable=False, exclusive=True and autoDelete=True.
>
> If the network connection is broken for more than seven minutes, the session on brokerB subscribing to the "bridge-queue" on brokerA is considered to have ended. The loss of the session subscriber triggers the the autoDelete of the bridge-queue. The result is that all messages awaiting federation via the bridge queue are deleted (NOT persisted at all).
>
> Ted Ross suggested a work around. He suggested we create a “queue” route where the destination broker subscribes to an existing queue on the source broker. This should cause the inter-broker route to use message acknowledgement in such a way that recovery will be clean.
>
> We encountered the following issues with the queue route:
> 1) “Out-of-order” message delivery was experienced
> 2) The first 1000KB of message data (in our case 10 messages) were not delivered until the broker process was restarted. This sounds crazy but we replicated the case multiple times.
That sounds like a session is not being correctly cleaned up perhaps and
there are some unacked messages in a locked state. Would you be willing
to raise a Jira for this and list the steps taken to reproduce?
> 3) Our solution loses the ability to do dynamic binding.
>
> Our customer needs to have the in-order, guaranteed delivery of messages on low quality networks.
>
> How should we proceed with this request?
>
>
> Cullen Davis
> CommIT Enterprises, Inc
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Request to Add autoDelete Parameter to qpid-route command
Posted by Gordon Sim <gs...@redhat.com>.
On 11/20/2009 08:50 PM, Cullen Davis wrote:
> Synopsis:
> Dynamic federation does not guarantee delivery of messages. Setting autoDelete=false on the �transit-queue� during dynamic federation queue creation would eliminate the issue.
>
> Summary:
> Our solution utilizes Qpid 0.5 C++ brokers and clients running on Red Hat Enterprise Linux 5.4. We are a government backed project that must deploy onto networks that are low bandwidth, high latency, and intermittently disconnected. Guaranteed delivery of the messages is a fundamental requirement. Our solution depends on brokers using direct exchanges and being dynamically federated. Using dynamic federation, when network connectivity is lost for more than seven minutes, all messages awaiting federation are deleted. At re-connect, the federation routes are rebuilt but the unsent messages are not restored. (Originally Stated in users mailing list on Thu, 05 Nov, 16:55)
>
> Change Request:
> Allow Dynamic Federation to create �transit-queues� with autoDelete=False. We would like to request a new option, autoDelete, on the qpid-route command. In our case, we would run
> qpid-route --durable --autodelete=false dynamic add brokerB brokerA fed.direct
> ============
> This would be used to create exchange bindings with transit-queues having the autoDelete property set to false.
I'd argue that if you are specifying durable that implies you don't want
auto deletion (durable is really only of any value in recovering from
loss of in-memory state and in that case you would have lost the session
and triggered auto-delete anyway). (The attached trivial patch would do
this I believe, though perhaps it unnecessarily couples the durability
of the bridge meta-data with the durability of the queue used in which
case a 'reliable' option might be prefered).
Having a timeout for a queue such that if there is no subscriber for x
seconds then the queue gets deleted might be a useful addition. That
would allow some protection over the case where a failed link never
comes back (or doesn't come back fast enough to avoid a critical build
up of messages).
> Details:
> We federate our brokers as follows:
> qpid-route --durable dynamic add brokerB brokerA fed.direct
>
> The dynamic option causes the qpid-route command to create a new �transit-queue�, with the name "bridge-queue" at brokerA. The bridge-queue serves as the means to move messages from brokerA to brokerB. The new queue had queue properties of durable=False, exclusive=True and autoDelete=True.
>
> If the network connection is broken for more than seven minutes, the session on brokerB subscribing to the "bridge-queue" on brokerA is considered to have ended. The loss of the session subscriber triggers the the autoDelete of the bridge-queue. The result is that all messages awaiting federation via the bridge queue are deleted (NOT persisted at all).
>
> Ted Ross suggested a work around. He suggested we create a �queue� route where the destination broker subscribes to an existing queue on the source broker. This should cause the inter-broker route to use message acknowledgement in such a way that recovery will be clean.
>
> We encountered the following issues with the queue route:
> 1) �Out-of-order� message delivery was experienced
> 2) The first 1000KB of message data (in our case 10 messages) were not delivered until the broker process was restarted. This sounds crazy but we replicated the case multiple times.
That sounds like a session is not being correctly cleaned up perhaps and
there are some unacked messages in a locked state. Would you be willing
to raise a Jira for this and list the steps taken to reproduce?
> 3) Our solution loses the ability to do dynamic binding.
>
> Our customer needs to have the in-order, guaranteed delivery of messages on low quality networks.
>
> How should we proceed with this request?
>
>
> Cullen Davis
> CommIT Enterprises, Inc
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>