Single message processed by multiple threads

[Ganesh Chandrasekaran <gc...@wayfair.com>]

Hi all,

I am noticing a strange behavior in one of my topology. The configuration is as follows:

-          Using RabbitMQ for publishing/consuming messages

-          Single topology with 2 workers running 4 threads each (Using multilang protocol)

I log messages every time a message gets published to rabbit queue so I queued a message just once. But I noticed 4 processes trying to process the same message within a gap of 60 milli seconds.

I know only during some kind of failures Storm can replay messages based on certain configurations. I would expect to have some time gap in those scenario (like message timeouts interval or something) But is the above scenario possible at all that multiple threads process same message within 60 millisec? If yes, how?

Thanks,
Ganesh

RE: Single message processed by multiple threads

[Ganesh Chandrasekaran <gc...@wayfair.com>]

Ahh nevermind, I made a minor mistake in my observation and I was wrong, it is not processing single message simultaneously.  Thanks for the response and sorry about the confusion.

From: Javier Gonzalez [mailto:jagonzal@gmail.com]
Sent: Thursday, August 27, 2015 11:24 AM
To: user@storm.apache.org
Subject: Re: Single message processed by multiple threads


Is your message source thread safe? It is possible for four spout threads to read the same message from the same source if the source does not guarantee uniqueness across multiple clients.
On Aug 27, 2015 9:59 AM, "Ganesh Chandrasekaran" <gc...@wayfair.com>> wrote:
Hi all,

I am noticing a strange behavior in one of my topology. The configuration is as follows:

-          Using RabbitMQ for publishing/consuming messages

-          Single topology with 2 workers running 4 threads each (Using multilang protocol)

I log messages every time a message gets published to rabbit queue so I queued a message just once. But I noticed 4 processes trying to process the same message within a gap of 60 milli seconds.

I know only during some kind of failures Storm can replay messages based on certain configurations. I would expect to have some time gap in those scenario (like message timeouts interval or something) But is the above scenario possible at all that multiple threads process same message within 60 millisec? If yes, how?

Thanks,
Ganesh

Re: Single message processed by multiple threads

[Javier Gonzalez <ja...@gmail.com>]

Is your message source thread safe? It is possible for four spout threads
to read the same message from the same source if the source does not
guarantee uniqueness across multiple clients.
On Aug 27, 2015 9:59 AM, "Ganesh Chandrasekaran" <
gchandrasekaran@wayfair.com> wrote:

> Hi all,
>
>
>
> I am noticing a strange behavior in one of my topology. The configuration
> is as follows:
>
> -          Using RabbitMQ for publishing/consuming messages
>
> -          Single topology with 2 workers running 4 threads each (Using
> multilang protocol)
>
>
>
> I log messages every time a message gets published to rabbit queue so I
> queued a message just once. But I noticed 4 processes trying to process the
> same message within a gap of 60 milli seconds.
>
>
>
> I know only during some kind of failures Storm can replay messages based
> on certain configurations. I would expect to have some time gap in those
> scenario (like message timeouts interval or something) But is the above
> scenario possible at all that multiple threads process same message within
> 60 millisec? If yes, how?
>
>
>
> Thanks,
>
> Ganesh
>