clustering

[lichtner <li...@bway.net>]

Does this project need any contributions regarding building symmetric
clustering which includes high availability?

If the server is going to be in Java, I believe that EVS4J would provide a
solid foundation. Essentially the idea is that all operations which affect
the state of the server should be broadcast to all cluster nodes before
being applied to the server's state, but each node would service a
different set of client connections.

As the author of EVS4J, I know of at least one app server vendor which has
built it clustering code on EVS4J and has achieved 25,000 messages per
second (about 1400 bytes per message.) The throughput per queue would
therefore be about 25,000/s, (regardless of the size of the cluster.)
However, the latency would increase linearly with the number of nodes
(i.e. with the number of client connections to be serviced.)

Re: clustering

[lichtner <li...@bway.net>]

On Fri, 22 Dec 2006, Carl Trieloff wrote:

> Yes but I wouldn't, we can use AMQP all our traffic main traffic and as
> we are a AMQP impl and should, totem is only useful for us for EVT, CLM,
> CPG, CMAN and DLM on top of totem for clustering. Take a look at AIS as
> it also uses totem and how they add these on top of it. JGroups was just
> another example of providing some of these on top of totem.

Indeed, that is the standard approach to clustering.

However, because a) totem has such high performance and b) it has total
ordering, it makes it feasible to build fully replicated state machines,
and with minimal effort.

Re: clustering

[lichtner <li...@bway.net>]

On Mon, 8 Jan 2007, Alan Conway wrote:

> On Mon, 2007-01-08 at 12:57 +0000, John O'Hara wrote:
> > There's a whole lot of conversation here.
> >
> > Above all else, we need to know what we're trying to achieve.
> > The current Qpid/J can achieve 200,000 transient 256 byte messages per
> > second on a 4-way machine over Gb ethernet.
> > For persistent messages, you will always be constrained by the performance
> > of the machine.

So what are you getting for persistent messages?

> > There are 3 broad families of connecting brokers:
> > 1) Highly stateful - failover pairs (keep it simple)
> > 2) Transient fanout scaling - lots of brokers sharing the load of
> > propagating messages to subscribers.  A bit like how routers do multicast.
> > But then, could we not build AMQP into the routers.....
> > 3) Forwarding between independent systems - the notion of remote queues, or
> > going through firewalls.

There is a fourth one. You can do active/active clustering running the
brokers in lock step. With this method two clients can use the same queue
simultaneously even though they are connecting to different brokers. You
can probably hit 130,000 255-byte persistent messages per second.

It's not as much as 200,000, but they are persistent.

Re: clustering

[Alan Conway <ac...@redhat.com>]

On Mon, 2007-01-08 at 12:57 +0000, John O'Hara wrote:
> There's a whole lot of conversation here.
> 
> Above all else, we need to know what we're trying to achieve.
> The current Qpid/J can achieve 200,000 transient 256 byte messages per
> second on a 4-way machine over Gb ethernet.
> For persistent messages, you will always be constrained by the performance
> of the machine.
> 
> There are 3 broad families of connecting brokers:
> 1) Highly stateful - failover pairs (keep it simple)
> 2) Transient fanout scaling - lots of brokers sharing the load of
> propagating messages to subscribers.  A bit like how routers do multicast.
> But then, could we not build AMQP into the routers.....
> 3) Forwarding between independent systems - the notion of remote queues, or
> going through firewalls.
> 
> Then there is how long do we hold onto messages for, etc.
> 
> I would love to see people draw up and document the scenarios which their
> proposals work in.  There are a lot of email conversations where the people
> involve are all coming from different assumptions.  We need to get to a
> common frame.

I will be looking at Qpid clustering in the near future and I would
dearly love to see some concrete use cases that are independent of any
design proposal. I'd like to start a qpid wiki page to gather links to
existing use case info and wiki pages for use cases we draw up ourselves
- anyone have material to contribute?

There are some initial design ideas for the next phase of qpid
clustering at
http://cwiki.apache.org/qpid/clustering-design-notes-2.html 
They're very unripe so comments are welcome, but more welcome would be
use cases that are not dependent on assumptions about the solution.

Cheers,
Alan.

Re: clustering

[John O'Hara <jo...@gmail.com>]

There's a whole lot of conversation here.

Above all else, we need to know what we're trying to achieve.
The current Qpid/J can achieve 200,000 transient 256 byte messages per
second on a 4-way machine over Gb ethernet.
For persistent messages, you will always be constrained by the performance
of the machine.

There are 3 broad families of connecting brokers:
1) Highly stateful - failover pairs (keep it simple)
2) Transient fanout scaling - lots of brokers sharing the load of
propagating messages to subscribers.  A bit like how routers do multicast.
But then, could we not build AMQP into the routers.....
3) Forwarding between independent systems - the notion of remote queues, or
going through firewalls.

Then there is how long do we hold onto messages for, etc.

I would love to see people draw up and document the scenarios which their
proposals work in.  There are a lot of email conversations where the people
involve are all coming from different assumptions.  We need to get to a
common frame.

I'm off back to drawing some diagrams for this :-)
John

On 22/12/06, lichtner <li...@bway.net> wrote:
>
>
> On Fri, 22 Dec 2006, Tim Fox wrote:
>
> > > But I am proposing that all queues be available on all nodes. So all
> the
> > > nodes do need to see all the messages.
> >
> > That's only true if you want to maintain exact copies of the queues on
> > each node. If you want to replicate for high availability that might be
> > useful.
>
> But up to a certain number of nodes the number of clients increases.
>
> As I said, you can put 25,000 1400-byte messages through a queue in this
> manner. The limiting factor on the number of clients is the latency,
> because since each node can only process, say, C clients, in order to add
> clients you have to add more nodes. But I think that with 3-4 nodes you
> are already going to have an award-winning jms server.
>
> If you want to scale beyond then you can use the multiple-ring protocol.
>
> > However, if you're intention is to be able to scale your system by
> > adding nodes to the cluster then maintaining exact replicas is not going
> > to help, since as we've both tacitly agreed each node would receive the
> > traffic from all other nodes, so you make no gains in adding extra nodes
> > to the cluster. (I.e. clustering is pointless)
>
> As explained above, by adding nodes you get to service a greater number of
> clients. I hope it's clear. But the price is that you increase the latency
> of all messages, a little bit. You just have to figure out how much is too
> much.
>
> But the implementation is so much easier than anything else that it should
> be good enough for a first implementation.
>
> > This boils down to the fact that the work in adding/removing an entry
> > from a strict FIFO queue is not inherently parallelizable (see Amdahl's
> > law).
>
> But the cost of adding to the queue is a marginal cost in the complete use
> case of adding to a queue. A machine has to do a context switch, read the
> incoming command from the client, and then add the message to the queue.
>
> > So how do you solve this problem? One way (and this is how several of
> > the popular messaging systems do it) do it is to relax the strict
> > FIFOness of the queue.
>
> You don't need to do that.
>
> > For many messaging applications/flavours, e.g. JMS, strict FIFO is not
> > necessary, it is only necessary to maintain order of messages produced
> > from a session. Therefore you can implement the "logical" clustered
> > queue, as a set of quasi-independent partial queues - one on each node
> > of the cluster. When you are attached at a particular node and sending a
> > message to the clustered queue, the particular partial queue it ends up
> > in is determined by a routing policy - typically you would always favour
> > the local queue to avoid extra network traffic. Consumers can be
> > attached at any node and consume from the partial queue. Then you
> > implement a "message redistribution" algorithm on top of this which
> > pulls messages from one partial quuee to another to avoid starvation of
> > consumers on a particular partial queue, or to spread load due to
> > consumers faster or slower on different nodes.
> >
> > Theoretically this has linear scalability.
>
> Is it easy to implement correctly?
>

Re: clustering

[lichtner <li...@bway.net>]

On Fri, 22 Dec 2006, Tim Fox wrote:

> > But I am proposing that all queues be available on all nodes. So all the
> > nodes do need to see all the messages.
>
> That's only true if you want to maintain exact copies of the queues on
> each node. If you want to replicate for high availability that might be
> useful.

But up to a certain number of nodes the number of clients increases.

As I said, you can put 25,000 1400-byte messages through a queue in this
manner. The limiting factor on the number of clients is the latency,
because since each node can only process, say, C clients, in order to add
clients you have to add more nodes. But I think that with 3-4 nodes you
are already going to have an award-winning jms server.

If you want to scale beyond then you can use the multiple-ring protocol.

> However, if you're intention is to be able to scale your system by
> adding nodes to the cluster then maintaining exact replicas is not going
> to help, since as we've both tacitly agreed each node would receive the
> traffic from all other nodes, so you make no gains in adding extra nodes
> to the cluster. (I.e. clustering is pointless)

As explained above, by adding nodes you get to service a greater number of
clients. I hope it's clear. But the price is that you increase the latency
of all messages, a little bit. You just have to figure out how much is too
much.

But the implementation is so much easier than anything else that it should
be good enough for a first implementation.

> This boils down to the fact that the work in adding/removing an entry
> from a strict FIFO queue is not inherently parallelizable (see Amdahl's
> law).

But the cost of adding to the queue is a marginal cost in the complete use
case of adding to a queue. A machine has to do a context switch, read the
incoming command from the client, and then add the message to the queue.

> So how do you solve this problem? One way (and this is how several of
> the popular messaging systems do it) do it is to relax the strict
> FIFOness of the queue.

You don't need to do that.

> For many messaging applications/flavours, e.g. JMS, strict FIFO is not
> necessary, it is only necessary to maintain order of messages produced
> from a session. Therefore you can implement the "logical" clustered
> queue, as a set of quasi-independent partial queues - one on each node
> of the cluster. When you are attached at a particular node and sending a
> message to the clustered queue, the particular partial queue it ends up
> in is determined by a routing policy - typically you would always favour
> the local queue to avoid extra network traffic. Consumers can be
> attached at any node and consume from the partial queue. Then you
> implement a "message redistribution" algorithm on top of this which
> pulls messages from one partial quuee to another to avoid starvation of
> consumers on a particular partial queue, or to spread load due to
> consumers faster or slower on different nodes.
>
> Theoretically this has linear scalability.

Is it easy to implement correctly?

Re: clustering

[Tim Fox <ti...@jboss.com>]

lichtner wrote:
> On Fri, 22 Dec 2006, Tim Fox wrote:
> 
>> Personally I would never use a total ordering protocol (e.g. totem) to
>> implement clustering for messaging.
>>
>> Naive clustering approaches often try to replicate the queue state
>> machine across different nodes, and require some kind of total ordering
>> to ensure the states are consistent.
>>
>> You'll find this approach doesn't scale, since every replica in the
>> cluster has to receive all updates from every other node. Therefore the
>> amount of load on a particular node increases linearly with the number
>> of nodes in the cluster.
> 
> But I am proposing that all queues be available on all nodes. So all the
> nodes do need to see all the messages.

That's only true if you want to maintain exact copies of the queues on 
each node. If you want to replicate for high availability that might be 
useful.

However, if you're intention is to be able to scale your system by 
adding nodes to the cluster then maintaining exact replicas is not going 
to help, since as we've both tacitly agreed each node would receive the 
traffic from all other nodes, so you make no gains in adding extra nodes 
to the cluster. (I.e. clustering is pointless)

This boils down to the fact that the work in adding/removing an entry 
from a strict FIFO queue is not inherently parallelizable (see Amdahl's 
law).

So how do you solve this problem? One way (and this is how several of 
the popular messaging systems do it) do it is to relax the strict 
FIFOness of the queue.

For many messaging applications/flavours, e.g. JMS, strict FIFO is not 
necessary, it is only necessary to maintain order of messages produced 
from a session. Therefore you can implement the "logical" clustered 
queue, as a set of quasi-independent partial queues - one on each node 
of the cluster. When you are attached at a particular node and sending a 
message to the clustered queue, the particular partial queue it ends up 
in is determined by a routing policy - typically you would always favour 
the local queue to avoid extra network traffic. Consumers can be 
attached at any node and consume from the partial queue. Then you 
implement a "message redistribution" algorithm on top of this which 
pulls messages from one partial quuee to another to avoid starvation of 
consumers on a particular partial queue, or to spread load due to 
consumers faster or slower on different nodes.

Theoretically this has linear scalability.






> 

-- 
Tim Fox
JBoss Messaging Technical Lead
JBoss - a division of Red Hat
T: +44 2088006768
M: +44 7957983205
E: tim.fox@jboss.com tim.fox@redhat.com

Re: clustering

[lichtner <li...@bway.net>]

On Fri, 22 Dec 2006, Tim Fox wrote:

> Personally I would never use a total ordering protocol (e.g. totem) to
> implement clustering for messaging.
>
> Naive clustering approaches often try to replicate the queue state
> machine across different nodes, and require some kind of total ordering
> to ensure the states are consistent.
>
> You'll find this approach doesn't scale, since every replica in the
> cluster has to receive all updates from every other node. Therefore the
> amount of load on a particular node increases linearly with the number
> of nodes in the cluster.

But I am proposing that all queues be available on all nodes. So all the
nodes do need to see all the messages.

Re: clustering

[lichtner <li...@bway.net>]

On Fri, 22 Dec 2006, Carl Trieloff wrote:

> >> Sure, I can add to the wiki in that regard then once back from vacation
> >
> > Sounds good, thanks. My concern is that 1) we seem to have a number of
> > people involved with ideas on how to implement this, and 2) those
> > ideas seem to have quite a range to them. If we can get some notes in
> > the wiki about the various possibilities and their pros and cons,
> > we'll be in really good shape.
> >
> > --steve
> >
> >
>
>
> Yes, more write up never hurts. Most of it just flows from the 0-9 spec
> but the other pieces are worth getting onto the wiki.

The wiki explicitly mentions the concept of a "failover partner". This
means that you have already decided to use primary-backup.

What I am suggesting is something radically different.

Re: clustering

[Steve Vinoski <vi...@iona.com>]

On Dec 22, 2006, at 1:45 PM, Carl Trieloff wrote:

> Steve Vinoski wrote:
>> Sounds good, thanks. My concern is that 1) we seem to have a  
>> number of people involved with ideas on how to implement this, and  
>> 2) those ideas seem to have quite a range to them. If we can get  
>> some notes in the wiki about the various possibilities and their  
>> pros and cons, we'll be in really good shape.
>
> Yes, more write up never hurts. Most of it just flows from the 0-9  
> spec but the other pieces are worth getting onto the wiki.

Well, what doesn't flow from 0-9 is the choice of underlying Java  
clustering implementation packages. A few have already been mentioned  
in this thread, and other possibilities may exist as well. I'd like  
to see those enumerated with their pros and cons so that we can  
choose which one(s) appear best to use.

--steve

Re: clustering

[Carl Trieloff <cc...@redhat.com>]

Steve Vinoski wrote:
>
> On Dec 22, 2006, at 1:33 PM, Carl Trieloff wrote:
>
>> Steve Vinoski wrote:
>>>
>>> On Dec 22, 2006, at 1:24 PM, Carl Trieloff wrote:
>>>
>>>> Steve Vinoski wrote:
>>>>>
>>>>> On Dec 22, 2006, at 1:16 PM, Carl Trieloff wrote:
>>>>>
>>>>>> Tim Fox wrote:
>>>>>>> Personally I would never use a total ordering protocol (e.g. 
>>>>>>> totem) to implement clustering for messaging.
>>>>>>>
>>>>>>> Naive clustering approaches often try to replicate the queue 
>>>>>>> state machine across different nodes, and require some kind of 
>>>>>>> total ordering to ensure the states are consistent.
>>>>>>>
>>>>>>> You'll find this approach doesn't scale, since every replica in 
>>>>>>> the cluster has to receive all updates from every other node. 
>>>>>>> Therefore the amount of load on a particular node increases 
>>>>>>> linearly with the number of nodes in the cluster.
>>>>>>
>>>>>> yes, exactly - thus totem is only useful for the cluster 
>>>>>> management, not the data replication. That is why I asked about 
>>>>>> those pieces and not data replication.
>>>>>
>>>>> Given that people obviously have opinions on how we should 
>>>>> implement clustering, it would be nice to get them captured in the 
>>>>> wiki. Any takers?
>>>>>
>>>>> --steve
>>>>
>>>> Alan has already done so.
>>>
>>> I don't mean the design note, I mean implementation.
>>>
>>> --steve
>>>
>>
>>
>> Sure, I can add to the wiki in that regard then once back from vacation
>
> Sounds good, thanks. My concern is that 1) we seem to have a number of 
> people involved with ideas on how to implement this, and 2) those 
> ideas seem to have quite a range to them. If we can get some notes in 
> the wiki about the various possibilities and their pros and cons, 
> we'll be in really good shape.
>
> --steve
>
>


Yes, more write up never hurts. Most of it just flows from the 0-9 spec 
but the other pieces are worth getting onto the wiki.

Regards.

Re: clustering

[Steve Vinoski <vi...@iona.com>]

On Dec 22, 2006, at 1:33 PM, Carl Trieloff wrote:

> Steve Vinoski wrote:
>>
>> On Dec 22, 2006, at 1:24 PM, Carl Trieloff wrote:
>>
>>> Steve Vinoski wrote:
>>>>
>>>> On Dec 22, 2006, at 1:16 PM, Carl Trieloff wrote:
>>>>
>>>>> Tim Fox wrote:
>>>>>> Personally I would never use a total ordering protocol (e.g.  
>>>>>> totem) to implement clustering for messaging.
>>>>>>
>>>>>> Naive clustering approaches often try to replicate the queue  
>>>>>> state machine across different nodes, and require some kind of  
>>>>>> total ordering to ensure the states are consistent.
>>>>>>
>>>>>> You'll find this approach doesn't scale, since every replica  
>>>>>> in the cluster has to receive all updates from every other  
>>>>>> node. Therefore the amount of load on a particular node  
>>>>>> increases linearly with the number of nodes in the cluster.
>>>>>
>>>>> yes, exactly - thus totem is only useful for the cluster  
>>>>> management, not the data replication. That is why I asked about  
>>>>> those pieces and not data replication.
>>>>
>>>> Given that people obviously have opinions on how we should  
>>>> implement clustering, it would be nice to get them captured in  
>>>> the wiki. Any takers?
>>>>
>>>> --steve
>>>
>>> Alan has already done so.
>>
>> I don't mean the design note, I mean implementation.
>>
>> --steve
>>
>
>
> Sure, I can add to the wiki in that regard then once back from  
> vacation

Sounds good, thanks. My concern is that 1) we seem to have a number  
of people involved with ideas on how to implement this, and 2) those  
ideas seem to have quite a range to them. If we can get some notes in  
the wiki about the various possibilities and their pros and cons,  
we'll be in really good shape.

--steve

Re: clustering

[Carl Trieloff <cc...@redhat.com>]

Steve Vinoski wrote:
>
> On Dec 22, 2006, at 1:24 PM, Carl Trieloff wrote:
>
>> Steve Vinoski wrote:
>>>
>>> On Dec 22, 2006, at 1:16 PM, Carl Trieloff wrote:
>>>
>>>> Tim Fox wrote:
>>>>> Personally I would never use a total ordering protocol (e.g. 
>>>>> totem) to implement clustering for messaging.
>>>>>
>>>>> Naive clustering approaches often try to replicate the queue state 
>>>>> machine across different nodes, and require some kind of total 
>>>>> ordering to ensure the states are consistent.
>>>>>
>>>>> You'll find this approach doesn't scale, since every replica in 
>>>>> the cluster has to receive all updates from every other node. 
>>>>> Therefore the amount of load on a particular node increases 
>>>>> linearly with the number of nodes in the cluster.
>>>>
>>>> yes, exactly - thus totem is only useful for the cluster 
>>>> management, not the data replication. That is why I asked about 
>>>> those pieces and not data replication.
>>>
>>> Given that people obviously have opinions on how we should implement 
>>> clustering, it would be nice to get them captured in the wiki. Any 
>>> takers?
>>>
>>> --steve
>>
>> Alan has already done so.
>
> I don't mean the design note, I mean implementation.
>
> --steve
>


Sure, I can add to the wiki in that regard then once back from vacation

Merry Christmas
Carl.

Re: clustering

[Steve Vinoski <vi...@iona.com>]

On Dec 22, 2006, at 1:24 PM, Carl Trieloff wrote:

> Steve Vinoski wrote:
>>
>> On Dec 22, 2006, at 1:16 PM, Carl Trieloff wrote:
>>
>>> Tim Fox wrote:
>>>> Personally I would never use a total ordering protocol (e.g.  
>>>> totem) to implement clustering for messaging.
>>>>
>>>> Naive clustering approaches often try to replicate the queue  
>>>> state machine across different nodes, and require some kind of  
>>>> total ordering to ensure the states are consistent.
>>>>
>>>> You'll find this approach doesn't scale, since every replica in  
>>>> the cluster has to receive all updates from every other node.  
>>>> Therefore the amount of load on a particular node increases  
>>>> linearly with the number of nodes in the cluster.
>>>
>>> yes, exactly - thus totem is only useful for the cluster  
>>> management, not the data replication. That is why I asked about  
>>> those pieces and not data replication.
>>
>> Given that people obviously have opinions on how we should  
>> implement clustering, it would be nice to get them captured in the  
>> wiki. Any takers?
>>
>> --steve
>
> Alan has already done so.

I don't mean the design note, I mean implementation.

--steve

Re: clustering

[Carl Trieloff <cc...@redhat.com>]

Steve Vinoski wrote:
>
> On Dec 22, 2006, at 1:16 PM, Carl Trieloff wrote:
>
>> Tim Fox wrote:
>>> Personally I would never use a total ordering protocol (e.g. totem) 
>>> to implement clustering for messaging.
>>>
>>> Naive clustering approaches often try to replicate the queue state 
>>> machine across different nodes, and require some kind of total 
>>> ordering to ensure the states are consistent.
>>>
>>> You'll find this approach doesn't scale, since every replica in the 
>>> cluster has to receive all updates from every other node. Therefore 
>>> the amount of load on a particular node increases linearly with the 
>>> number of nodes in the cluster.
>>
>> yes, exactly - thus totem is only useful for the cluster management, 
>> not the data replication. That is why I asked about those pieces and 
>> not data replication.
>
> Given that people obviously have opinions on how we should implement 
> clustering, it would be nice to get them captured in the wiki. Any 
> takers?
>
> --steve

Alan has already done so.

Re: clustering

[Steve Vinoski <vi...@iona.com>]

On Dec 22, 2006, at 1:16 PM, Carl Trieloff wrote:

> Tim Fox wrote:
>> Personally I would never use a total ordering protocol (e.g.  
>> totem) to implement clustering for messaging.
>>
>> Naive clustering approaches often try to replicate the queue state  
>> machine across different nodes, and require some kind of total  
>> ordering to ensure the states are consistent.
>>
>> You'll find this approach doesn't scale, since every replica in  
>> the cluster has to receive all updates from every other node.  
>> Therefore the amount of load on a particular node increases  
>> linearly with the number of nodes in the cluster.
>
> yes, exactly - thus totem is only useful for the cluster  
> management, not the data replication. That is why I asked about  
> those pieces and not data replication.

Given that people obviously have opinions on how we should implement  
clustering, it would be nice to get them captured in the wiki. Any  
takers?

--steve

Re: clustering

[Carl Trieloff <cc...@redhat.com>]

Tim Fox wrote:
> Personally I would never use a total ordering protocol (e.g. totem) to 
> implement clustering for messaging.
>
> Naive clustering approaches often try to replicate the queue state 
> machine across different nodes, and require some kind of total 
> ordering to ensure the states are consistent.
>
> You'll find this approach doesn't scale, since every replica in the 
> cluster has to receive all updates from every other node. Therefore 
> the amount of load on a particular node increases linearly with the 
> number of nodes in the cluster.

yes, exactly - thus totem is only useful for the cluster management, not 
the data replication. That is why I asked about those pieces and not 
data replication.

Regards
Carl.

Re: clustering

[Tim Fox <ti...@jboss.com>]

Personally I would never use a total ordering protocol (e.g. totem) to 
implement clustering for messaging.

Naive clustering approaches often try to replicate the queue state 
machine across different nodes, and require some kind of total ordering 
to ensure the states are consistent.

You'll find this approach doesn't scale, since every replica in the 
cluster has to receive all updates from every other node. Therefore the 
amount of load on a particular node increases linearly with the number 
of nodes in the cluster.

BTW we use JGroups for clustering, but don't use total ordering. JGroups 
is a common API to a number of different protocols.

Carl Trieloff wrote:
> lichtner wrote:
>> On Fri, 22 Dec 2006, Carl Trieloff wrote:
>>
>>  
>>>> It's just an in-process implementation of the totem protocol, so it 
>>>> does
>>>> not have checkpointing, special event messages, or lock management. 
>>>> That
>>>> is not part of totem.
>>>>
>>>>
>>>>       
>>> ok, have you tried to integrate with JGroups then?
>>>     
>>
>> I am not sure what you mean there. Totem is a completely self-contained
>> protocol. It doesn't need any other layer. You can build an electronic
>> stock exchange or an air traffic control system just using totem.
>>   
> 
> 
> Yes but I wouldn't, we can use AMQP all our traffic main traffic and as 
> we are a AMQP impl and should, totem is only useful for us for EVT, CLM, 
> CPG, CMAN and DLM on top of totem for clustering. Take a look at AIS as 
> it also uses totem and how they add these on top of it. JGroups was just 
> another example of providing some of these on top of totem.
> 
> Carl.
> 
> 

-- 
Tim Fox
JBoss Messaging Technical Lead
JBoss - a division of Red Hat
T: +44 2088006768
M: +44 7957983205
E: tim.fox@jboss.com tim.fox@redhat.com

Re: clustering

[Carl Trieloff <cc...@redhat.com>]

lichtner wrote:
> On Fri, 22 Dec 2006, Carl Trieloff wrote:
>
>   
>>> It's just an in-process implementation of the totem protocol, so it does
>>> not have checkpointing, special event messages, or lock management. That
>>> is not part of totem.
>>>
>>>
>>>       
>> ok, have you tried to integrate with JGroups then?
>>     
>
> I am not sure what you mean there. Totem is a completely self-contained
> protocol. It doesn't need any other layer. You can build an electronic
> stock exchange or an air traffic control system just using totem.
>   


Yes but I wouldn't, we can use AMQP all our traffic main traffic and as 
we are a AMQP impl and should, totem is only useful for us for EVT, CLM, 
CPG, CMAN and DLM on top of totem for clustering. Take a look at AIS as 
it also uses totem and how they add these on top of it. JGroups was just 
another example of providing some of these on top of totem.

Carl.

Re: clustering

[lichtner <li...@bway.net>]

On Fri, 22 Dec 2006, Carl Trieloff wrote:

> > It's just an in-process implementation of the totem protocol, so it does
> > not have checkpointing, special event messages, or lock management. That
> > is not part of totem.
> >
> >
>
> ok, have you tried to integrate with JGroups then?

I am not sure what you mean there. Totem is a completely self-contained
protocol. It doesn't need any other layer. You can build an electronic
stock exchange or an air traffic control system just using totem.

Re: clustering

[Carl Trieloff <cc...@redhat.com>]

lichtner wrote:
> On Thu, 21 Dec 2006, Carl Trieloff wrote:
>
>   
>> there is both a server in Java and C++.
>>     
>
> While we are on the topic, why two server implementations?
>   

So that it can also be run as a native daemon service on platforms that 
support that and
take advantage of better IO rates, and so that we can deploy all Java 
where that is desired.

Note that we have clients in Python, Ruby, C#, C++, Java (JMS) and I 
have been asked
for Perl...

>   
>> If you go to the wiki, cwiki.apache.org/qpid under clustering there is
>> quite a bit up there on what we are working on in this regard. C++ is
>> being put together to use OpenAIS (also totem) and Java there are more
>> options for non AMQP cluster comms.
>>     
>
> Yes, I saw the wiki. But it seems to mention that the design is not
> finished - correct me if I am wrong.
>   

We had to wait for 0-9 of the AMQP spec to be published to get some
of the items we need in the protocol. Now that 0-9 is out it should go quite
quickly.


>   
>> How mature is EVS4J on CKPT, EVT, EVS, CLM, CPG,CMAN, DLM ?
>>     
>
> It's just an in-process implementation of the totem protocol, so it does
> not have checkpointing, special event messages, or lock management. That
> is not part of totem.
>
>   

ok, have you tried to integrate with JGroups then?

Re: clustering

[lichtner <li...@bway.net>]

On Thu, 21 Dec 2006, Carl Trieloff wrote:

> there is both a server in Java and C++.

While we are on the topic, why two server implementations?

> If you go to the wiki, cwiki.apache.org/qpid under clustering there is
> quite a bit up there on what we are working on in this regard. C++ is
> being put together to use OpenAIS (also totem) and Java there are more
> options for non AMQP cluster comms.

Yes, I saw the wiki. But it seems to mention that the design is not
finished - correct me if I am wrong.

> How mature is EVS4J on CKPT, EVT, EVS, CLM, CPG,CMAN, DLM ?

It's just an in-process implementation of the totem protocol, so it does
not have checkpointing, special event messages, or lock management. That
is not part of totem.

> Or how do you rate it to something like JGroups?

EVS4J only implements totem (membership and reliable multicast with total
ordering and flow control) - not other protocols. And you do not have to
assemble separate protocols (which means there is nothing to figure out.)
And the code is small and easy to review for correctness. And it's fast.
The api is very, very simple.

Re: clustering

[Carl Trieloff <cc...@redhat.com>]

there is both a server in Java and C++. If you go to the wiki, 
cwiki.apache.org/qpid under clustering there is quite a bit up there on 
what we are working on in this regard. C++ is being put together to use 
OpenAIS (also totem) and Java there are more options for non AMQP 
cluster comms.

How mature is EVS4J on CKPT, EVT, EVS, CLM, CPG,CMAN, DLM ?
Or how do you rate it to something like JGroups?

Carl.


lichtner wrote:
> Does this project need any contributions regarding building symmetric
> clustering which includes high availability?
>
> If the server is going to be in Java, I believe that EVS4J would provide a
> solid foundation. Essentially the idea is that all operations which affect
> the state of the server should be broadcast to all cluster nodes before
> being applied to the server's state, but each node would service a
> different set of client connections.
>
> As the author of EVS4J, I know of at least one app server vendor which has
> built it clustering code on EVS4J and has achieved 25,000 messages per
> second (about 1400 bytes per message.) The throughput per queue would
> therefore be about 25,000/s, (regardless of the size of the cluster.)
> However, the latency would increase linearly with the number of nodes
> (i.e. with the number of client connections to be serviced.)
>