You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by Andrew Kennedy <an...@gmail.com> on 2011/10/26 21:26:14 UTC
Wide-area clustering
Hi.
I'm trying to set up clustering with the C++ broker, but I'd like to do this on Amazon EC2 or similar cloud hosts. The problem with this is that I understand the current clustering (corosync/open AIS) requires multicast. This isn't available on EC2 or similar, so I'd like to find out if there are any alternatives or if I've missed some setting in the configuration?
I just need to run a failover pair of brokers, sharing state for many queues/topics, which are located in different data-centers or clouds separated by a reasonably large distance (blast-zone/disaster area for BCP) for my application.
I read about Red Hat clustering in the docs, but it didn't seem relevant for this use case. So, is there any alternative resilience mechanism available, that would allow me to run a failover pair of brokers wide-area? Or, is corosync the only one available, still? ? I understand the C++ broker has a plugin that implements the resilience, so is there an API that could be used to add different resilience mechanisms myself?
Thanks,
Andrew.
--
-- andrew d kennedy ? PHONE_MISSING : http://grkvlt.blogspot.com/ ;
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Wide-area clustering
Posted by Gordon Sim <gs...@redhat.com>.
On 10/28/2011 03:39 PM, Andrew Kennedy wrote:
> On 28 Oct 2011, at 15h19, Gordon Sim wrote:
>> On 10/28/2011 03:17 PM, Rob Godfrey wrote:
>>> On 28 October 2011 15:12, Gordon Sim<gs...@redhat.com> wrote:
>>>
>>>> On 10/26/2011 08:38 PM, Carl Trieloff wrote:
>>>>
>>>>> Use queue state replication and client failover via the address or
>>>>> failover exchange.
>>>>>
>>>>
>>>> A few words of warning... as it stands there are a few operational issues
>>>> with that 'feature' you should be aware of:
>>>>
>>>> * if links are down replication events can back up very fast and there is
>>>> no way to manage this
>>>> * resuming replication is impossible if any events are lost
>>>> * its a little cumbersome to configure
>
> Good to know.
>
>>>> [I've been experimenting with a much nicer design proposed by Ted. I have a
>>>> working patch but its unlikely to get committed anytime soon]
>>>>
>>>>
>>> Is this something that would also be useful to implement in the Java
>>> Broker...
>>
>> Yes
>
> +1, definitely.
>
>>> is there a public design document for it?
>>
>> No, not yet. As I say its an experimental patch only at this stage, primarily because I was intrigued by the idea. I'm only mentioning it here as it fixes the issues with the existing queue replication very effectively.
>
> I'd be very interested in looking at this, if you could supply more details about your design, Gordon.
The core idea is to use a browsing subscription as the means to
replicate the message content. This requires that browsers see all
messages, including those that are acquired.
In addition you need to send dequeue events. I do this on the same
subscription using messages with routing key 'dequeue_event' with an
encoded AMQP 0-10 sequence set as their body. The messages being
dequeued are identified by a sequence number assigned by the queue.
Allowing multiple dequeues to be combined in a single message helps
prevent build up of messages for a slow consumer (or a consumer trying
their best when the queue is heavily loaded!).
This approach means that when the replicating subscription is not
active, there is no build up of state for replication. When
resubscribing, the subscription can include their current sequence
number (i.e. the last one assigned) and the sequence number of the
oldest message. This allows the source queue to generate any necessary
dequeue events to keep the two in sync and start the browsing at the
correct position.
(Contrast with the current solution where there is a steadily growing
queue of events that are not coalesced in anyway and whose deletion
tends to render it impossible to resume replication).
The completion of published messages is delayed until either any
replicating subscriptions have themselves confirmed receipt or until the
message is dequeued. Likewise completions of accepts covering messages
delivered from that queue should be delayed until any replicating
subscriptions have confirmed receipt of the dequeue notifications (my
prototype doesn't yet do this, accepts complete immediately).
This gives synchronous replication as far as the client is concerned,
but doesn't delay delivery of messages until replicated so has little
impact on latency.
My prototype uses a quick-and-dirty hack to configure this. A special
pattern in the exchange name (starts with 'qpid.replicator-') on a
federation route identifies the route as being created for queue
replication (which causes the necessary arguments to be set on the
subscribe). Though the implementation (and tooling) would be done
differently in a real solution, I believe it would be as simple to setup
(which is a fair bit simpler than at present). A nice bonus is that you
don't have to signal the desire to replicate when first creating the
source queue.
Happy to send the patch if you want to play around with it.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Wide-area clustering
Posted by Andrew Kennedy <an...@gmail.com>.
On 28 Oct 2011, at 15h19, Gordon Sim wrote:
> On 10/28/2011 03:17 PM, Rob Godfrey wrote:
>> On 28 October 2011 15:12, Gordon Sim<gs...@redhat.com> wrote:
>>
>>> On 10/26/2011 08:38 PM, Carl Trieloff wrote:
>>>
>>>> Use queue state replication and client failover via the address or
>>>> failover exchange.
>>>>
>>>
>>> A few words of warning... as it stands there are a few operational issues
>>> with that 'feature' you should be aware of:
>>>
>>> * if links are down replication events can back up very fast and there is
>>> no way to manage this
>>> * resuming replication is impossible if any events are lost
>>> * its a little cumbersome to configure
Good to know.
>>> [I've been experimenting with a much nicer design proposed by Ted. I have a
>>> working patch but its unlikely to get committed anytime soon]
>>>
>>>
>> Is this something that would also be useful to implement in the Java
>> Broker...
>
> Yes
+1, definitely.
>> is there a public design document for it?
>
> No, not yet. As I say its an experimental patch only at this stage, primarily because I was intrigued by the idea. I'm only mentioning it here as it fixes the issues with the existing queue replication very effectively.
I'd be very interested in looking at this, if you could supply more details about your design, Gordon.
Cheers,
Andrew.
--
-- andrew d kennedy ? +447582293255 : http://grkvlt.blogspot.com/ ;
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Wide-area clustering
Posted by Gordon Sim <gs...@redhat.com>.
On 10/28/2011 03:17 PM, Rob Godfrey wrote:
> On 28 October 2011 15:12, Gordon Sim<gs...@redhat.com> wrote:
>
>> On 10/26/2011 08:38 PM, Carl Trieloff wrote:
>>
>>> Use queue state replication and client failover via the address or
>>> failover exchange.
>>>
>>
>> A few words of warning... as it stands there are a few operational issues
>> with that 'feature' you should be aware of:
>>
>> * if links are down replication events can back up very fast and there is
>> no way to manage this
>> * resuming replication is impossible if any events are lost
>> * its a little cumbersome to configure
>>
>> [I've been experimenting with a much nicer design proposed by Ted. I have a
>> working patch but its unlikely to get committed anytime soon]
>>
>>
> Is this something that would also be useful to implement in the Java
> Broker...
Yes
> is there a public design document for it?
No, not yet. As I say its an experimental patch only at this stage,
primarily because I was intrigued by the idea. I'm only mentioning it
here as it fixes the issues with the existing queue replication very
effectively.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Wide-area clustering
Posted by Rob Godfrey <ro...@gmail.com>.
On 28 October 2011 15:12, Gordon Sim <gs...@redhat.com> wrote:
> On 10/26/2011 08:38 PM, Carl Trieloff wrote:
>
>> Use queue state replication and client failover via the address or
>> failover exchange.
>>
>
> A few words of warning... as it stands there are a few operational issues
> with that 'feature' you should be aware of:
>
> * if links are down replication events can back up very fast and there is
> no way to manage this
> * resuming replication is impossible if any events are lost
> * its a little cumbersome to configure
>
> [I've been experimenting with a much nicer design proposed by Ted. I have a
> working patch but its unlikely to get committed anytime soon]
>
>
Is this something that would also be useful to implement in the Java
Broker... is there a public design document for it?
cheers,
Rob
>
> ------------------------------**------------------------------**---------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.**apache.org<de...@qpid.apache.org>
>
>
Re: Wide-area clustering
Posted by Gordon Sim <gs...@redhat.com>.
On 10/26/2011 08:38 PM, Carl Trieloff wrote:
> Use queue state replication and client failover via the address or
> failover exchange.
A few words of warning... as it stands there are a few operational
issues with that 'feature' you should be aware of:
* if links are down replication events can back up very fast and there
is no way to manage this
* resuming replication is impossible if any events are lost
* its a little cumbersome to configure
[I've been experimenting with a much nicer design proposed by Ted. I
have a working patch but its unlikely to get committed anytime soon]
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Wide-area clustering
Posted by Carl Trieloff <cc...@redhat.com>.
On 10/26/2011 06:28 PM, Andrew Kennedy wrote:
> Ah, thanks Carl.
>
> I just skimmed over that bit of the documentation [1] when I was reading up on clustering, but it's obvious now ;) I'll give it a try - it looks like I only need the replication plugin libraries, not corosync or anything.
>
> Reading the documentation, if I set 'qpid.queue_event_generation' to '1' when declaring a queue, I assume I can use the Java client to create replicated queues?
>
> [1] http://qpid.apache.org/books/trunk/AMQP-Messaging-Broker-CPP-Book/html/ch01s07.html
>
I have not done it from Java, but it should work. Just check it is set
correctly with qpid-tool
Carl.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Wide-area clustering
Posted by Andrew Kennedy <an...@gmail.com>.
On 26 Oct 2011, at 20h38, Carl Trieloff wrote:
> Use queue state replication and client failover via the address or failover exchange.
>
> Carl.
Ah, thanks Carl.
I just skimmed over that bit of the documentation [1] when I was reading up on clustering, but it's obvious now ;) I'll give it a try - it looks like I only need the replication plugin libraries, not corosync or anything.
Reading the documentation, if I set 'qpid.queue_event_generation' to '1' when declaring a queue, I assume I can use the Java client to create replicated queues?
[1] http://qpid.apache.org/books/trunk/AMQP-Messaging-Broker-CPP-Book/html/ch01s07.html
> On 10/26/2011 03:26 PM, Andrew Kennedy wrote:
>>
>> I'm trying to set up [...] failover pair of brokers, sharing state for many queues/topics, [...]
Andrew.
--
-- andrew d kennedy ? PHONE_MISSING : http://grkvlt.blogspot.com/ ;
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Wide-area clustering
Posted by Carl Trieloff <cc...@redhat.com>.
Use queue state replication and client failover via the address or
failover exchange.
Carl.
On 10/26/2011 03:26 PM, Andrew Kennedy wrote:
> Hi.
>
> I'm trying to set up clustering with the C++ broker, but I'd like to do this on Amazon EC2 or similar cloud hosts. The problem with this is that I understand the current clustering (corosync/open AIS) requires multicast. This isn't available on EC2 or similar, so I'd like to find out if there are any alternatives or if I've missed some setting in the configuration?
>
> I just need to run a failover pair of brokers, sharing state for many queues/topics, which are located in different data-centers or clouds separated by a reasonably large distance (blast-zone/disaster area for BCP) for my application.
>
> I read about Red Hat clustering in the docs, but it didn't seem relevant for this use case. So, is there any alternative resilience mechanism available, that would allow me to run a failover pair of brokers wide-area? Or, is corosync the only one available, still? ? I understand the C++ broker has a plugin that implements the resilience, so is there an API that could be used to add different resilience mechanisms myself?
>
> Thanks,
> Andrew.
> --
> -- andrew d kennedy ? PHONE_MISSING : http://grkvlt.blogspot.com/ ;
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org