Re: Java atomic bindings Was: Re: Queue creation/deletion

[Robert Greig <ro...@gmail.com>]

2009/1/30 Andrew Wright <at...@mac.com>:

> Actually this was to integrate with Terracotta. TC needs some sort of load
> balancer to get locality of reference (for performance) - qpid would fit the
> bill here, via hierarchical topics. There'd only be a single subscriber on a
> topic at a time. TC clients are notified as nodes join/leave, so can take
> over the processing of a given topic if the primary processor goes away.

So you have a number of nodes which access Terracotta shared storage
and you want to push jobs into those nodes, with some kind of affinity
between job types and nodes in order to be able to get the locality of
reference.

Are you pairing the nodes up so that each node has a "backup" that
will take over if the primary dies?

The issue with using a topic is that recovery is made more complicated
- unless the backup also subscribes to the messages and just throws
them away. I would be tempted to use a queue with the headers exchange
and just load balance between the primary and secondary (effective
having active/active pairs with the clients). I'm obviously making a
lot of assumptions about your application's suitability for that
approach!

As far as the broker goes, I think that is an orthogonal question. I
would run active/active brokers behind a hardware load balancer such
as an F5. That will mean the client doesn't have to deal with failover
plus you don't have to use shared storage so should give higher
throughput (Carl can you confirm?).

> As you say, Coherence has this sort of thing built in. Unfortunately it also
> comes with a fairly hefty Oracle price tag...

Yes it is expensive but it is a much more sophisticated product (or
certainly a "different" product) than Terracotta - I think it would
depend what your non-functionals are whether it is worth using it (or
Gemfire, IBM ObjectGrid or Gigaspaces which are credible Coherence
competitors).

RG

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org

Re: Java atomic bindings Was: Re: Queue creation/deletion

[Andrew Wright <at...@mac.com>]

On 4 Feb 2009, at 22:50, Robert Greig wrote:
>
>
>> The big difficulty we have is a hard requirement for in-order message
>> processing. This unfortunately rules out load balancing from a  
>> queue across
>> active-active client pairs. We'd rather just route incoming msgs to a
>> store-and-forward topic, and accept a small processing pause during  
>> app node
>> failover.
>
> Do you have the requirement for total ordering or is it just a partial
> ordering, i.e. you need to process requests for a given order id in
> the right order (i.e. new/amend/cancel for a given order id)?
>

It's a partial ordering, per instrument. Ie, orders on the same  
instrument need to be processed sequentially.

>> Once a message has been read from the topic, it is written into the  
>> TC
>> cache. At that point we can failover processing to a different node  
>> without
>> any message or data loss.
>
> Does TC now support memory replication between the server nodes?
> Previously it required shared storage which meant the performance
> sucked.
>

I can imagine. It now does in-memory replication between a master &  
slave. They still have the shared storage mode but it's basically  
deprecated these days I believe.

It also does notification of client leave/join (to remaining clients).  
The general idea is we use qpid to split messages between TC client  
nodes (for data locality), and TC notifications for HA (another node  
can take over processing of a given stream of orders).

>>
>> Absolutely - and we haven't ruled anything out yet. We're going to  
>> start
>> with the lowest cost option, and if it doesn't work out, then have a
>> rethink.
>
> What are your throughput and latency requirements?
>
> This is an interesting domain since it blurs the distinction between
> messaging and in-memory data grids.

Most definitely. Like most things, 'as much throughput as fast as  
possible' - according to the business at least! qpid seems more than  
capable of handling our throughput requirements. We're currently  
investigating a few application structure scenarios to see which one  
meets our latency/load balancing needs the best.

Cheers,
Andrew


---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org

Re: Java atomic bindings Was: Re: Queue creation/deletion

[Robert Greig <ro...@gmail.com>]

2009/2/2 Andrew Wright <at...@mac.com>:

> Essentially, yes. Each app node is responsible for a subset of the data
> processing (we're looking to implement an order matching engine/exchange).

OK - a domain quite familiar to me.

> The big difficulty we have is a hard requirement for in-order message
> processing. This unfortunately rules out load balancing from a queue across
> active-active client pairs. We'd rather just route incoming msgs to a
> store-and-forward topic, and accept a small processing pause during app node
> failover.

Do you have the requirement for total ordering or is it just a partial
ordering, i.e. you need to process requests for a given order id in
the right order (i.e. new/amend/cancel for a given order id)?

> Once a message has been read from the topic, it is written into the TC
> cache. At that point we can failover processing to a different node without
> any message or data loss.

Does TC now support memory replication between the server nodes?
Previously it required shared storage which meant the performance
sucked.

> We were rather hoping that qpid clients (eg. the Java client) would be able
> to failover against a HA broker pair without the use of a dedicated load
> balancer.

Yes, it can. It implements heartbeats and can automatically
resubscribe. Carl - does the active/active propagate subscription info
across nodes of the cluster?

>>> As you say, Coherence has this sort of thing built in. Unfortunately it
>>> also
>>> comes with a fairly hefty Oracle price tag...
>>
>> Yes it is expensive but it is a much more sophisticated product (or
>> certainly a "different" product) than Terracotta - I think it would
>> depend what your non-functionals are whether it is worth using it (or
>> Gemfire, IBM ObjectGrid or Gigaspaces which are credible Coherence
>> competitors).
>
> Absolutely - and we haven't ruled anything out yet. We're going to start
> with the lowest cost option, and if it doesn't work out, then have a
> rethink.

What are your throughput and latency requirements?

This is an interesting domain since it blurs the distinction between
messaging and in-memory data grids.

RG

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org

Re: Java atomic bindings Was: Re: Queue creation/deletion

[Andrew Wright <at...@mac.com>]

Thanks for the comments - a few responses below.

On 31 Jan 2009, at 23:20, Robert Greig wrote:

> 2009/1/30 Andrew Wright <at...@mac.com>:
>
>> Actually this was to integrate with Terracotta. TC needs some sort  
>> of load
>> balancer to get locality of reference (for performance) - qpid  
>> would fit the
>> bill here, via hierarchical topics. There'd only be a single  
>> subscriber on a
>> topic at a time. TC clients are notified as nodes join/leave, so  
>> can take
>> over the processing of a given topic if the primary processor goes  
>> away.
>
> So you have a number of nodes which access Terracotta shared storage
> and you want to push jobs into those nodes, with some kind of affinity
> between job types and nodes in order to be able to get the locality of
> reference.
>

Essentially, yes. Each app node is responsible for a subset of the  
data processing (we're looking to implement an order matching engine/ 
exchange).

> The issue with using a topic is that recovery is made more complicated
> - unless the backup also subscribes to the messages and just throws
> them away. I would be tempted to use a queue with the headers exchange
> and just load balance between the primary and secondary (effective
> having active/active pairs with the clients). I'm obviously making a
> lot of assumptions about your application's suitability for that
> approach!
>

The big difficulty we have is a hard requirement for in-order message  
processing. This unfortunately rules out load balancing from a queue  
across active-active client pairs. We'd rather just route incoming  
msgs to a store-and-forward topic, and accept a small processing pause  
during app node failover.

Once a message has been read from the topic, it is written into the TC  
cache. At that point we can failover processing to a different node  
without any message or data loss.


> As far as the broker goes, I think that is an orthogonal question. I
> would run active/active brokers behind a hardware load balancer such
> as an F5. That will mean the client doesn't have to deal with failover
> plus you don't have to use shared storage so should give higher
> throughput (Carl can you confirm?).
>

We were rather hoping that qpid clients (eg. the Java client) would be  
able to failover against a HA broker pair without the use of a  
dedicated load balancer.

>> As you say, Coherence has this sort of thing built in.  
>> Unfortunately it also
>> comes with a fairly hefty Oracle price tag...
>
> Yes it is expensive but it is a much more sophisticated product (or
> certainly a "different" product) than Terracotta - I think it would
> depend what your non-functionals are whether it is worth using it (or
> Gemfire, IBM ObjectGrid or Gigaspaces which are credible Coherence
> competitors).

Absolutely - and we haven't ruled anything out yet. We're going to  
start with the lowest cost option, and if it doesn't work out, then  
have a rethink.

Cheers,
Andrew

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org