You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Sidhant Goyal <si...@gmail.com> on 2017/05/16 02:31:11 UTC

Prioritizing input streams

Hello,

I have two Kafka Spout streams which are fed into a common bolt for
processing and then persisting in Cassandra.
But messages from one stream has to be processed with higher priority than
messages from another stream.

One stream is fed by a real-time system and the other stream is fed by a
batch system. So it’s useful to prioritize the real-time stream over the
batch stream, so that the real-time processing doesn’t slow down if there
is a sudden burst of data on the batch stream.

How can I achieve this ? Is using a
java.util.concurrent.PriorityBlockingQueue within the bolt will work ?
Since the both the streams will be shuffled into the bolt tasks have
separate instances of queues also doesn't gurantees that only higher
priority stream tuples will be processed first.

Please suggest alternative approaches.

Thanks
Sidhant Goyal