What methods have people used to overcome race conditions?

[joshhendo <jo...@outlook.com>]

Hi Everyone,

Firstly, I just wanted to quickly say hi as I'm new to the Active MQ
community.

I'm doing a research project on different ways that people have overcome
race conditions when using message queues. As such, I'm looking to do an
informal survey throughout various communities that use message queues, to
find out what kind of problems people have seen (if any), and what
approaches and designs they've used to help minimize or avoid race
conditions.

An example scenario that I'm thinking of is, the same message is
(accidentally) sent multiple times, and with the receiving program
processing messages in parallel across multiple threads, the same message is
processed multiple times simultaneously, causing unexpected results (this
scenario is based around the message being processed in parallel; a race
condition probably wouldn't occur if the same message was sent a second time
after the first message was fully processed).

It could be that many people haven't experienced such issues, or that
there's a common theme to avoiding such issues (e.g. ensuring every message
is idempotent). However, if anyone has had any experience with such
scenarios, I'd very interested in finding out more about your experiences.

Thanks!
Josh Henderson



--
View this message in context: http://activemq.2283324.n4.nabble.com/What-methods-have-people-used-to-overcome-race-conditions-tp4715676.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: What methods have people used to overcome race conditions?

[Matt Pavlovich <ma...@gmail.com>]

Josh-

There are established patterns for managing race conditions in 
distributed computing-- ActiveMQ or other, the approach is the same. A 
messaging system, such as ActiveMQ, is a solid part of the solution of 
sending messages from system A to B and avoiding race conditions by 
solving for many common scenarios in a consistent manner.

In addition, other technical measures can be layered on top:

1. Application supports duplication detection (or replay of message)
      => In theory, handles all scenarios. Has sharp corners in practice.

2. Idempotent repository/message filter where a history of processed 
messages (using an application-level messageId)
      => Can drastically narrow the surface area of the potential for 
duplication, and is generally easier on the application.

On 8/30/16 4:57 PM, joshhendo wrote:
> Hi Tim,
>
> Thanks for the reply; definitely helpful to get a point of view from someone
> regularly using the product.
>
> My example was probably a bit too simplistic, such that it failed to still
> be a race condition. There are other examples I'm using in my project, of
> which a lot of your comments can apply to.
>
> Once again thanks for taking the time to reply.
>
> Cheers,
> Josh
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/What-methods-have-people-used-to-overcome-race-conditions-tp4715676p4715984.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: What methods have people used to overcome race conditions?

[joshhendo <jo...@outlook.com>]

Hi Tim,

Thanks for the reply; definitely helpful to get a point of view from someone
regularly using the product.

My example was probably a bit too simplistic, such that it failed to still
be a race condition. There are other examples I'm using in my project, of
which a lot of your comments can apply to.

Once again thanks for taking the time to reply.

Cheers,
Josh



--
View this message in context: http://activemq.2283324.n4.nabble.com/What-methods-have-people-used-to-overcome-race-conditions-tp4715676p4715984.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: What methods have people used to overcome race conditions?

[Tim Bain <tb...@alumni.duke.edu>]

Your example race condition isn't a race condition; if you publish two
identical messages, they will both be consumed irrespective of whether that
consumption is concurrent or sequential, because they're two different
messages (even though they have the same content).

In any messaging system, you have the choice between at-least-once
processing and at-most-once processing, where the difference is whether you
should attempt to reprocess messages that failed processing the first time
for whatever reason.  If you're not using an idempotent consumer, your only
safe option is at-most-once processing, which ActiveMQ defaults to by
defaulting to AUTO_ACK acknowledgement mode, but it's always a good
practice to use an idempotent consumer even if you're using AUTO_ACK mode.

The caveat to all of what I just wrote is that if your consumer can be
written in a way that all actions can be wrapped in a transaction that can
be rolled back, you can avoid duplicate results of processing (though of
course you'll have processed more than once, you just will have discarded
all but one of the results when you rolled back the transaction).  But not
all actions can be rolled back, so this doesn't help in all cases.

The most obvious race condition I know of is the case where a topic message
is sent when you're not sure whether the consumer is already connected (and
might or might not get it as a result).  Depending on what you're trying to
do, you can sometimes work around the problem by having the producer wait
to send the message till the consumer is subscribed.

Good luck with your research project.

Tim

On Fri, Aug 19, 2016 at 9:39 PM, joshhendo <jo...@outlook.com> wrote:

> Hi Everyone,
>
> Firstly, I just wanted to quickly say hi as I'm new to the Active MQ
> community.
>
> I'm doing a research project on different ways that people have overcome
> race conditions when using message queues. As such, I'm looking to do an
> informal survey throughout various communities that use message queues, to
> find out what kind of problems people have seen (if any), and what
> approaches and designs they've used to help minimize or avoid race
> conditions.
>
> An example scenario that I'm thinking of is, the same message is
> (accidentally) sent multiple times, and with the receiving program
> processing messages in parallel across multiple threads, the same message
> is
> processed multiple times simultaneously, causing unexpected results (this
> scenario is based around the message being processed in parallel; a race
> condition probably wouldn't occur if the same message was sent a second
> time
> after the first message was fully processed).
>
> It could be that many people haven't experienced such issues, or that
> there's a common theme to avoiding such issues (e.g. ensuring every message
> is idempotent). However, if anyone has had any experience with such
> scenarios, I'd very interested in finding out more about your experiences.
>
> Thanks!
> Josh Henderson
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/What-methods-have-people-used-to-overcome-race-
> conditions-tp4715676.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>