@ptgoetz storm-jms and ActiveMQ

[Alberto São Marcos <am...@bikeemotion.com>]

Hi,
Im currently using Storm 0.9.x, ActiveMQ and @ptgoetz storm-jms spout.
I have multiple producers sending messages to the same queue.
My topologies run multiple Worker/Executors/Taks.

 From what i understand:
1) One ActiveMQ consumer (storm-jms spout instance) takes messages from 
the queue preserving the order (FIFO);
2) Multiple ActiveMQ consumers/spouts in the same queue will compete for 
messages and order is lost.
3) Storm guarantees that bolts consume tuples in the order they are 
emitted by the topology spout;

With storm-jms spout in a parallel scenario is there any message 
ordering preservation guarantee at all?
In what order are tuple batches emitted to the stream by those multiple 
storm-jms spout instances?
Even if I used a Exclusive Consumer 
<http://activemq.apache.org/exclusive-consumer.html> approach, can i 
assume the order of tuple processing is preserved?
Does it having multiple ActiveMQ consumers in the form of spouts 
instances ultimately translates to a unpredictable tuple processing 
order by the topology bolts?
If so, can I solve this problem switching ActiveMQ for other solution?

Thanks in advance.
Alberto

Re: @ptgoetz storm-jms and ActiveMQ

["P. Taylor Goetz" <pt...@gmail.com>]

Hi Alberto,

With a spout parallelism > 1, you will lose ordering.

The only way to preserve some semblance of order would be to use a spout parallelism of 1, AND use a fields grouping of some kind to partition the stream.

What about your use case requires preservation of message order? There might be an alternative angle of attack…

- Taylor

On Aug 4, 2014, at 12:29 PM, Alberto São Marcos <am...@bikeemotion.com> wrote:

> Hi,
> Im currently using Storm 0.9.x, ActiveMQ and @ptgoetz storm-jms spout.
> I have multiple producers sending messages to the same queue.
> My topologies run multiple Worker/Executors/Taks.
> 
> From what i understand:
> 1) One ActiveMQ consumer (storm-jms spout instance) takes messages from the queue preserving the order (FIFO);
> 2) Multiple ActiveMQ consumers/spouts in the same queue will compete for messages and order is lost.
> 3) Storm guarantees that bolts consume tuples in the order they are emitted by the topology spout;
> 
> With storm-jms spout in a parallel scenario is there any message ordering preservation guarantee at all?
> In what order are tuple batches emitted to the stream by those multiple storm-jms spout instances?
> Even if I used a Exclusive Consumer approach, can i assume the order of tuple processing is preserved?
> Does it having multiple ActiveMQ consumers in the form of spouts instances ultimately translates to a unpredictable tuple processing order by the topology bolts?
> If so, can I solve this problem switching ActiveMQ for other solution?
> 
> Thanks in advance.
> Alberto
>