Asynchronous architecture

[James Talbut <jt...@tardis.spudsoft>]

Folks,

Another question along similar lines to the System architecture thread.

I need to provide an asynchronous web service that will accept calls
from a client, persist them somewhere, then asynchronously try to call
a corresponding web service implemented by a third party.
If the call succeeds my service should notify the client (via another
web service, probably most REST-like than SOAP-like).
If the call fails I should keep trying periodically for 24 hours.

The two primary requirements are:
1. The initial call from the client should be quick (so don't try
calling the third party straight away).
2. It must be impossible to lose a message or send it to the third
party twice (hence it needs to persist the message somehow before
returning).

I don't know ActiveMQ (but happy to learn) and the third party won't
provide anything more than a simple web service interface (so they
won't hook into an ActiveMQ that we implement).

I'd like to be able to reuse the infrastructure so providing the same
functionality across different web services is trivial.
i.e. I'd like the thing that picks up failed messages to work for any
web service (requiring nothing more than the spring definition).

Is ActiveMQ the right tool for looking after persisting the messages
and making them available?
Should I just write my own component to persist (and recover) a
complete Camel Exchange?
Or is there something else out there that will do what I want?

Any pointers gratefully received.

Thanks

Jim

Re: Asynchronous architecture

[John McDonald <mc...@gmail.com>]

I have seen this kind of operation in place before - SMS providers for example for event driven stuff across the solution boundaries using basically a callback implementation

I would 
* expose an http interface to my solution.  I think it would be RESTian for the usual reasons
* I would use some message broking in the middle because it guarantees the sequence, deliver once, dont lose thing.  ActiveMq seems pretty good - I am no expert though
* For the feedback leg to the client you could
     ** ask them to register an optional http endpoint that you could signal
     ** let them come and get it from you - e.g. you expose an rss feed personalised to that client

I think its a fairly standard way of exposing an inherently async service in a sync manner

On 31 Mar 2011, at 14:41, James Talbut wrote:

> Folks,
> 
> Another question along similar lines to the System architecture thread.
> 
> I need to provide an asynchronous web service that will accept calls
> from a client, persist them somewhere, then asynchronously try to call
> a corresponding web service implemented by a third party.
> If the call succeeds my service should notify the client (via another
> web service, probably most REST-like than SOAP-like).
> If the call fails I should keep trying periodically for 24 hours.
> 
> The two primary requirements are:
> 1. The initial call from the client should be quick (so don't try
> calling the third party straight away).
> 2. It must be impossible to lose a message or send it to the third
> party twice (hence it needs to persist the message somehow before
> returning).
> 
> I don't know ActiveMQ (but happy to learn) and the third party won't
> provide anything more than a simple web service interface (so they
> won't hook into an ActiveMQ that we implement).
> *
> I'd like to be able to reuse the infrastructure so providing the same
> functionality across different web services is trivial.
> i.e. I'd like the thing that picks up failed messages to work for any
> web service (requiring nothing more than the spring definition).
> 
> Is ActiveMQ the right tool for looking after persisting the messages
> and making them available?
> Should I just write my own component to persist (and recover) a
> complete Camel Exchange?
> Or is there something else out there that will do what I want?
> 
> Any pointers gratefully received.
> 
> Thanks
> 
> Jim
> 

Re: Asynchronous architecture

[Hadrian Zbarcea <hz...@gmail.com>]

Hi James,

Camel can help you with that. Looks like you got it covered. For persistence you have a few choices, camel-jms, camel-jpa, camel-sql, depending on your message. But ActiveMQ is probably your best bet. It also scales very well  (you can also cluster it) and is relatively easy to configure. 

1. How quick it is, depends on the size of the message. For best results you can use an embedded broker, in the same container you use for your camel routes. I personally prefer OSGi/Karaf, but app servers and other deployment models work as well. 
2. For not sending it twice, camel supports an idempotent consumer [1].

There is no need to write your own persistence mechanism.

I hope this helps,
Hadrian

[1] http://camel.apache.org/idempotent-consumer.html


On Mar 31, 2011, at 9:41 AM, James Talbut wrote:

> Folks,
> 
> Another question along similar lines to the System architecture thread.
> 
> I need to provide an asynchronous web service that will accept calls
> from a client, persist them somewhere, then asynchronously try to call
> a corresponding web service implemented by a third party.
> If the call succeeds my service should notify the client (via another
> web service, probably most REST-like than SOAP-like).
> If the call fails I should keep trying periodically for 24 hours.
> 
> The two primary requirements are:
> 1. The initial call from the client should be quick (so don't try
> calling the third party straight away).
> 2. It must be impossible to lose a message or send it to the third
> party twice (hence it needs to persist the message somehow before
> returning).
> 
> I don't know ActiveMQ (but happy to learn) and the third party won't
> provide anything more than a simple web service interface (so they
> won't hook into an ActiveMQ that we implement).
> 
> I'd like to be able to reuse the infrastructure so providing the same
> functionality across different web services is trivial.
> i.e. I'd like the thing that picks up failed messages to work for any
> web service (requiring nothing more than the spring definition).
> 
> Is ActiveMQ the right tool for looking after persisting the messages
> and making them available?
> Should I just write my own component to persist (and recover) a
> complete Camel Exchange?
> Or is there something else out there that will do what I want?
> 
> Any pointers gratefully received.
> 
> Thanks
> 
> Jim
>