Re: clustering

[John O'Hara <jo...@gmail.com>]

There's a whole lot of conversation here.

Above all else, we need to know what we're trying to achieve.
The current Qpid/J can achieve 200,000 transient 256 byte messages per
second on a 4-way machine over Gb ethernet.
For persistent messages, you will always be constrained by the performance
of the machine.

There are 3 broad families of connecting brokers:
1) Highly stateful - failover pairs (keep it simple)
2) Transient fanout scaling - lots of brokers sharing the load of
propagating messages to subscribers.  A bit like how routers do multicast.
But then, could we not build AMQP into the routers.....
3) Forwarding between independent systems - the notion of remote queues, or
going through firewalls.

Then there is how long do we hold onto messages for, etc.

I would love to see people draw up and document the scenarios which their
proposals work in.  There are a lot of email conversations where the people
involve are all coming from different assumptions.  We need to get to a
common frame.

I'm off back to drawing some diagrams for this :-)
John

On 22/12/06, lichtner <li...@bway.net> wrote:
>
>
> On Fri, 22 Dec 2006, Tim Fox wrote:
>
> > > But I am proposing that all queues be available on all nodes. So all
> the
> > > nodes do need to see all the messages.
> >
> > That's only true if you want to maintain exact copies of the queues on
> > each node. If you want to replicate for high availability that might be
> > useful.
>
> But up to a certain number of nodes the number of clients increases.
>
> As I said, you can put 25,000 1400-byte messages through a queue in this
> manner. The limiting factor on the number of clients is the latency,
> because since each node can only process, say, C clients, in order to add
> clients you have to add more nodes. But I think that with 3-4 nodes you
> are already going to have an award-winning jms server.
>
> If you want to scale beyond then you can use the multiple-ring protocol.
>
> > However, if you're intention is to be able to scale your system by
> > adding nodes to the cluster then maintaining exact replicas is not going
> > to help, since as we've both tacitly agreed each node would receive the
> > traffic from all other nodes, so you make no gains in adding extra nodes
> > to the cluster. (I.e. clustering is pointless)
>
> As explained above, by adding nodes you get to service a greater number of
> clients. I hope it's clear. But the price is that you increase the latency
> of all messages, a little bit. You just have to figure out how much is too
> much.
>
> But the implementation is so much easier than anything else that it should
> be good enough for a first implementation.
>
> > This boils down to the fact that the work in adding/removing an entry
> > from a strict FIFO queue is not inherently parallelizable (see Amdahl's
> > law).
>
> But the cost of adding to the queue is a marginal cost in the complete use
> case of adding to a queue. A machine has to do a context switch, read the
> incoming command from the client, and then add the message to the queue.
>
> > So how do you solve this problem? One way (and this is how several of
> > the popular messaging systems do it) do it is to relax the strict
> > FIFOness of the queue.
>
> You don't need to do that.
>
> > For many messaging applications/flavours, e.g. JMS, strict FIFO is not
> > necessary, it is only necessary to maintain order of messages produced
> > from a session. Therefore you can implement the "logical" clustered
> > queue, as a set of quasi-independent partial queues - one on each node
> > of the cluster. When you are attached at a particular node and sending a
> > message to the clustered queue, the particular partial queue it ends up
> > in is determined by a routing policy - typically you would always favour
> > the local queue to avoid extra network traffic. Consumers can be
> > attached at any node and consume from the partial queue. Then you
> > implement a "message redistribution" algorithm on top of this which
> > pulls messages from one partial quuee to another to avoid starvation of
> > consumers on a particular partial queue, or to spread load due to
> > consumers faster or slower on different nodes.
> >
> > Theoretically this has linear scalability.
>
> Is it easy to implement correctly?
>

Re: clustering

[lichtner <li...@bway.net>]

On Mon, 8 Jan 2007, Alan Conway wrote:

> On Mon, 2007-01-08 at 12:57 +0000, John O'Hara wrote:
> > There's a whole lot of conversation here.
> >
> > Above all else, we need to know what we're trying to achieve.
> > The current Qpid/J can achieve 200,000 transient 256 byte messages per
> > second on a 4-way machine over Gb ethernet.
> > For persistent messages, you will always be constrained by the performance
> > of the machine.

So what are you getting for persistent messages?

> > There are 3 broad families of connecting brokers:
> > 1) Highly stateful - failover pairs (keep it simple)
> > 2) Transient fanout scaling - lots of brokers sharing the load of
> > propagating messages to subscribers.  A bit like how routers do multicast.
> > But then, could we not build AMQP into the routers.....
> > 3) Forwarding between independent systems - the notion of remote queues, or
> > going through firewalls.

There is a fourth one. You can do active/active clustering running the
brokers in lock step. With this method two clients can use the same queue
simultaneously even though they are connecting to different brokers. You
can probably hit 130,000 255-byte persistent messages per second.

It's not as much as 200,000, but they are persistent.

Re: clustering

[Alan Conway <ac...@redhat.com>]

On Mon, 2007-01-08 at 12:57 +0000, John O'Hara wrote:
> There's a whole lot of conversation here.
> 
> Above all else, we need to know what we're trying to achieve.
> The current Qpid/J can achieve 200,000 transient 256 byte messages per
> second on a 4-way machine over Gb ethernet.
> For persistent messages, you will always be constrained by the performance
> of the machine.
> 
> There are 3 broad families of connecting brokers:
> 1) Highly stateful - failover pairs (keep it simple)
> 2) Transient fanout scaling - lots of brokers sharing the load of
> propagating messages to subscribers.  A bit like how routers do multicast.
> But then, could we not build AMQP into the routers.....
> 3) Forwarding between independent systems - the notion of remote queues, or
> going through firewalls.
> 
> Then there is how long do we hold onto messages for, etc.
> 
> I would love to see people draw up and document the scenarios which their
> proposals work in.  There are a lot of email conversations where the people
> involve are all coming from different assumptions.  We need to get to a
> common frame.

I will be looking at Qpid clustering in the near future and I would
dearly love to see some concrete use cases that are independent of any
design proposal. I'd like to start a qpid wiki page to gather links to
existing use case info and wiki pages for use cases we draw up ourselves
- anyone have material to contribute?

There are some initial design ideas for the next phase of qpid
clustering at
http://cwiki.apache.org/qpid/clustering-design-notes-2.html 
They're very unripe so comments are welcome, but more welcome would be
use cases that are not dependent on assumptions about the solution.

Cheers,
Alan.