Enterprise messaging with Atom Feeds

[Bruno Luiz Pereira da Silva <bl...@gmail.com>]

Hi, I'd like to ask you guys about something I'm considering to use in a
project.

We are developing several asynchronous processes here, and we will populate
a queue with items to be processed. This queue will be consumed by lots
(let's say 100-200) of simultaneous threads to process the items. Upon
completion, the processor will publish to another queue, of successfully
processed items.

I'm considering the use of Atom feeds for both queues. An item to be
processed would be published as an entry of the "To be processed" feed. And
after being successfully processed, the item would be published as an item
of the "Successfully processed" feed.

In order to avoid 2 different batch instances processing the same item from
the "To be processed" feed, I thought of using ETag to manage this. A batch
instance trying to get an item to process would try to update a given
element of the entry, informing the ETag of the item. If the batch instance
is able to update the element, it'll then process the item and then publish
this entry to the "Successfully processed" feed, if everything goes well
with the processing.

I'd like to know if you guys think Atom feeds are adequate to do this, or if
this is just not a good idea. I don't know if Atom feeds can handle well
several threads trying to update something at the same time.

I'll appreciate your thoughts and comments.

-- 
Atenciosamente,

Bruno Luiz Pereira da Silva
blpsilva@gmail.com
http://brunopereira.com.br

Re: Enterprise messaging with Atom Feeds

[Bruno Luiz Pereira da Silva <bl...@gmail.com>]

Hi Dan, thanks for answering me. I just took a look at ActiveMQ RESTFul
queue, in the link you sent. I didn't know it had such feature. Nice to know
that.

However, their approach doesn't really match my goals. I posted some time
ago regarding this idea of using an Atom Feed for enterprise messaging, you
can see the post here if you'd like to:
http://blpsilva.wordpress.com/2008/01/27/atom-feeds-for-enterprise-application-messaging/

The basic idea is like the following: suppose you have a given application
doing some things that other applications must be notified about. For
example, let's say the application responsible for Google's users
registration is called "Register". Each user's registration is an event for
the Register app. Now let's consider YouTube, Picasa and Gmail as
applications interested in such events. Let's suppose that when a new user
is created, he must have his Gmail mailbox created, and his filesystem
directory created for Picasa and YouTube. One event in the Register
application generates events for several other applications, and you could
possibly add other applications to be notified whenever you want.

In order to have a nice solution with low coupling I thought of using an
Atom Feed to represent the list of created users. Any application that needs
to do something when a new user is created can simply subscribe to this feed
and do whatever it needs when a new entry is created.

Currently I'm developing several asynchronous processes that would make use
of such queues to replace synchronous processing. We'll have maybe several
hundred threads consuming this queue in order to make the queue processing
very fast. Of course I could put this queue in the database and use some
classic transactional way of reading it to avoid problems. However, if I
could use an Atom feed for that, I'd really like. The design would be
cleaner and it'd be convenient in our RESTful architecture.

I just saw your response in your post about Mule's Jersey connector. I'll
take a look in this HTTP polling component you mentioned. Once I have chosen
a given option I'll reply here.

Thanks for the help!

-- 
Regards,

Bruno Luiz Pereira da Silva
blpsilva@gmail.com
http://brunopereira.com.br


On Tue, Mar 11, 2008 at 9:16 PM, Dan Diephouse <da...@mulesource.com>
wrote:

> There's been some good discussion about how to do this type of thing in
> the past. ActiveMQ implements a RESTful interface via per-consumer queues:
>
> http://activemq.apache.org/restful-queue.html
>
> There's also a good thread on how to do Queueing scenarios RESTfully here:
>
> http://tech.groups.yahoo.com/group/rest-discuss/message/8955
>
> A per-consumer subscription is probably the sanest way to do this with
> Atom without introducing application specific semantics.
>
> Dan
>
> P.S. I just realized I never replied to your blog comment - I'll do that
> now!
>
> Bruno Luiz Pereira da Silva wrote:
> > Hi, I'd like to ask you guys about something I'm considering to use in a
> > project.
> >
> > We are developing several asynchronous processes here, and we will
> populate
> > a queue with items to be processed. This queue will be consumed by lots
> > (let's say 100-200) of simultaneous threads to process the items. Upon
> > completion, the processor will publish to another queue, of successfully
> > processed items.
> >
> > I'm considering the use of Atom feeds for both queues. An item to be
> > processed would be published as an entry of the "To be processed" feed.
> And
> > after being successfully processed, the item would be published as an
> item
> > of the "Successfully processed" feed.
> >
> > In order to avoid 2 different batch instances processing the same item
> from
> > the "To be processed" feed, I thought of using ETag to manage this. A
> batch
> > instance trying to get an item to process would try to update a given
> > element of the entry, informing the ETag of the item. If the batch
> instance
> > is able to update the element, it'll then process the item and then
> publish
> > this entry to the "Successfully processed" feed, if everything goes well
> > with the processing.
> >
> > I'd like to know if you guys think Atom feeds are adequate to do this,
> or if
> > this is just not a good idea. I don't know if Atom feeds can handle well
> > several threads trying to update something at the same time.
> >
> > I'll appreciate your thoughts and comments.
> >
> >
>
>
> --
> Dan Diephouse
> MuleSource
> http://mulesource.com | http://netzooid.com
>
>

Re: Enterprise messaging with Atom Feeds

[Dan Diephouse <da...@mulesource.com>]

There's been some good discussion about how to do this type of thing in 
the past. ActiveMQ implements a RESTful interface via per-consumer queues:

http://activemq.apache.org/restful-queue.html

There's also a good thread on how to do Queueing scenarios RESTfully here:

http://tech.groups.yahoo.com/group/rest-discuss/message/8955

A per-consumer subscription is probably the sanest way to do this with 
Atom without introducing application specific semantics.

Dan

P.S. I just realized I never replied to your blog comment - I'll do that 
now!

Bruno Luiz Pereira da Silva wrote:
> Hi, I'd like to ask you guys about something I'm considering to use in a
> project.
>
> We are developing several asynchronous processes here, and we will populate
> a queue with items to be processed. This queue will be consumed by lots
> (let's say 100-200) of simultaneous threads to process the items. Upon
> completion, the processor will publish to another queue, of successfully
> processed items.
>
> I'm considering the use of Atom feeds for both queues. An item to be
> processed would be published as an entry of the "To be processed" feed. And
> after being successfully processed, the item would be published as an item
> of the "Successfully processed" feed.
>
> In order to avoid 2 different batch instances processing the same item from
> the "To be processed" feed, I thought of using ETag to manage this. A batch
> instance trying to get an item to process would try to update a given
> element of the entry, informing the ETag of the item. If the batch instance
> is able to update the element, it'll then process the item and then publish
> this entry to the "Successfully processed" feed, if everything goes well
> with the processing.
>
> I'd like to know if you guys think Atom feeds are adequate to do this, or if
> this is just not a good idea. I don't know if Atom feeds can handle well
> several threads trying to update something at the same time.
>
> I'll appreciate your thoughts and comments.
>
>   


-- 
Dan Diephouse
MuleSource
http://mulesource.com | http://netzooid.com