Design Patterns for Interactive Stream Processing using Storm

[Itai Frenkel <It...@forter.com>]

Hi,


I would like to discuss possible Storm design patterns for the following requirement:

Given a storm topology that is used in production for (automatic) real-time stream processing, a REST API is required by the user interface to interactively (manually) run a subset of the topology and display interim results.

For simplicity let's assume the following topology:

QueueSpout -> (multiple parallel) ProcessingBolt(s) -> Join -> ReduceBolt -> PersistenceBolt

The user interface requires each of the ProcessingBolts to be exposed as a separate REST API.



Design 1:

     Deploy a separate DRPCTopology for each ProcessingBolt.

     REST server acts as a reverse proxy that forwards the requests to the DRPC server.



Design 2:

     REST server puts message in a priority queue with low priority, and subscribes for result in Redis.

    Use OOP to enhance all processing bolts to be aware of toggles in the tuple. Effectively the tupple contains toggles, to disable all Processing bolts but one.

    Another toggle forwards interim results to a (Redis) Publish Bolt instead of the ReduceBolt.



Design 1 Pros:

  1.  Follows the principle of immutable stream processing graph.
  2.  Follows the principle of preferring N simpler systems over 1 complex system.

Design 2 Pros:

  1.  Makes operations life easier. One system to monitor/upgrade.
  2.  Enabler for fine-grained monitoring probe to continuously monitor the real-time system (one subsystem at a time)
  3.  Enabler for customer specific stream processing (instead of topology per tenant).



Thoughts?



Itai