Maintaining Message Uniqueness in a Queue

[itamara <av...@gmail.com>]

hi AMQ users

What is the best way to ensure Message Uniqueness in a Queue?

I'm having the following architecture:

A single Scheduler (a quartz scheduled job) is periodically checking a DB
for entities to update, for each entity-to-update it uses a single Sender to
send an Update Message with a property holding the entity ID.

The Sender references an ActiveMQQueue destination and a Spring's
JmsTemplate over a PooledConnectionFactory.

The Consumers - MessageListeners in a Spring's
DefaultMessageListenerContainer container - receive a message and do their
onMessage, which may involves updating the DB.


I wish to avoid more then one Consumer updating the same entity
simultaneously, as it will result in DB errors.

I thought the Scheduler should account for this synchronization, so upon
resolution of the entities to update, it tags them in order to know not to
update them again - a tag that is removed when update is finished. But as
errors may occur, this tag is taken in account only for a certain period of
time. This makes the situation, in which the queue contains two messages
with the same entity ID, possible.

Which may cause exactly what I'm trying to avoid.

How do I pull it off?

I thought knowing which messages are currently in the queue could help the
Scheduler decide.

What about the message ID? Can it help here?

I could also expire the message, but it is not accurate. I might throw away
messages for no good reason.

Maybe it is better to do this synchronization elsewhere?

Thanks a lot for any help or thoughts.
-- 
View this message in context: http://www.nabble.com/Maintaining-Message-Uniqueness-in-a-Queue-tp25660461p25660461.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Maintaining Message Uniqueness in a Queue

[Joe Fernandez <jo...@ttmsolutions.com>]

Prior to sending a message, your scheduler can use a QueueBrowser w/selector
to peek at the queue and make sure a message with the same entity id as the
one about to be sent doesn't already exist. 

Or your consumer can use a selector to fetch any other messages on the queue
that have an entity id that matches the one just received.

But there are obvious performance implications with the above.  

Joe
http://www.ttmsolutions.com


itamara wrote:
> 
> hi AMQ users
> <p />
> What is the best way to ensure Message Uniqueness in a Queue?
> <p />
> <u>I'm having the following architecture:</u><br />
> A single Scheduler (a quartz scheduled job) is periodically checking a DB
> for entities to update, for each entity-to-update it uses a single Sender
> to send an Update Message with a property holding the entity ID.<br />
> The Sender references an ActiveMQQueue destination and a Spring's
> JmsTemplate over a PooledConnectionFactory.<br />
> The Consumers - MessageListeners in a Spring's
> DefaultMessageListenerContainer container - receive a message and do their
> onMessage, which may involves updating the DB.<br />
> <br />
> I wish to avoid more then one Consumer updating the same entity
> simultaneously, as it will result in DB errors.<br />
> I thought the Scheduler should account for this synchronization, so upon
> resolution of the entities to update, it tags them in order to know not to
> update them again - a tag that is removed when update is finished. But as
> errors may occur, this tag is taken in account only for a certain period
> of time. This makes the situation, in which the queue contains two
> messages with the same entity ID, possible.<br />
> Which may cause exactly what I'm trying to avoid.
> <p />
> How do I pull it off?<br />
> I thought knowing which messages are currently in the queue could help the
> Scheduler decide.<br />
> What about the message ID? Can it help here?<br />
> I could also expire the message, but it is not accurate. I might throw
> away messages for no good reason.<br />
> Maybe it is better to do this synchronization elsewhere?
> <p />
> Thanks a lot for any help or thoughts.
> 

-- 
View this message in context: http://www.nabble.com/Maintaining-Message-Uniqueness-in-a-Queue-tp25660461p25661136.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Maintaining Message Uniqueness in a Queue

[itamara <av...@gmail.com>]

As the documentation declares: "Message Groups rock!"

It looks like the right thing for me, though I still have to wait for
version 5.3.
I can do that.

Thanks a lot!
-- 
View this message in context: http://www.nabble.com/Maintaining-Message-Uniqueness-in-a-Queue-tp25660461p25665844.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Maintaining Message Uniqueness in a Queue

[James Strachan <ja...@gmail.com>]

2009/9/29 itamara <av...@gmail.com>:
>
> hi AMQ users
>
> What is the best way to ensure Message Uniqueness in a Queue?
>
> I'm having the following architecture:
>
> A single Scheduler (a quartz scheduled job) is periodically checking a DB
> for entities to update, for each entity-to-update it uses a single Sender to
> send an Update Message with a property holding the entity ID.
>
> The Sender references an ActiveMQQueue destination and a Spring's
> JmsTemplate over a PooledConnectionFactory.
>
> The Consumers - MessageListeners in a Spring's
> DefaultMessageListenerContainer container - receive a message and do their
> onMessage, which may involves updating the DB.
>
>
> I wish to avoid more then one Consumer updating the same entity
> simultaneously, as it will result in DB errors.

Using Message Groups to ensure only one consumer thread processes a
certain entity at once sounds like a simple solution...
http://activemq.apache.org/message-groups.html

> I thought the Scheduler should account for this synchronization, so upon
> resolution of the entities to update, it tags them in order to know not to
> update them again - a tag that is removed when update is finished. But as
> errors may occur, this tag is taken in account only for a certain period of
> time. This makes the situation, in which the queue contains two messages
> with the same entity ID, possible.
>
> Which may cause exactly what I'm trying to avoid.
>
> How do I pull it off?
>
> I thought knowing which messages are currently in the queue could help the
> Scheduler decide.
>
> What about the message ID? Can it help here?

Message IDs are unique and set after you send them

-- 
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://fusesource.com/