Hooking into the internal messaging system

[Pieter-Jan Van Aeken <pi...@euranova.eu>]

Hi,

I am trying to create a loop in Storm. To do this, I would like to be
able to replay a Tuple across a series of Bolts until certain criteria
are met. One of them being max loops so that it does not go into a
never ending loop. 

The way I would like to do this is by (ab)using the internal messaging
system. Is there a way I can create an OutputCollector which submits
records to a Task ID that I provide? That way, I would not need to
worry if Storm is using ZeroMQ or Netty under the hood.

Thanks in advance,

Pieter-Jan Van Aeken

Re: Hooking into the internal messaging system

[Nathan Leung <nc...@gmail.com>]

It is preferable to emit on a separate stream that the prior bolt
subscribes to. That way you can conditionally emit on the second stream and
avoid infinite loops.
On May 7, 2015 5:55 AM, "Matthias J. Sax" <mj...@informatik.hu-berlin.de>
wrote:

> Hi,
>
> you can user collector.emitDirect(...) to send a tuple to a specific
> task. However, you cannot assign task IDs. You need to get the ID from
> the given topology object in .open()/.prepare()
>
> If you use emitDirect(...), we need to declare the output stream as an
> direct stream of course.
>
>
> -Matthias
>
> On 05/07/2015 11:50 AM, Pieter-Jan Van Aeken wrote:
> > Hi,
> >
> > I am trying to create a loop in Storm. To do this, I would like to be
> > able to replay a Tuple across a series of Bolts until certain criteria
> > are met. One of them being max loops so that it does not go into a
> > never ending loop.
> >
> > The way I would like to do this is by (ab)using the internal messaging
> > system. Is there a way I can create an OutputCollector which submits
> > records to a Task ID that I provide? That way, I would not need to
> > worry if Storm is using ZeroMQ or Netty under the hood.
> >
> > Thanks in advance,
> >
> > Pieter-Jan Van Aeken
> >
> >
>
>

Re: Hooking into the internal messaging system

["Matthias J. Sax" <mj...@informatik.hu-berlin.de>]

Hi,

you can user collector.emitDirect(...) to send a tuple to a specific
task. However, you cannot assign task IDs. You need to get the ID from
the given topology object in .open()/.prepare()

If you use emitDirect(...), we need to declare the output stream as an
direct stream of course.


-Matthias

On 05/07/2015 11:50 AM, Pieter-Jan Van Aeken wrote:
> Hi,
> 
> I am trying to create a loop in Storm. To do this, I would like to be
> able to replay a Tuple across a series of Bolts until certain criteria
> are met. One of them being max loops so that it does not go into a
> never ending loop. 
> 
> The way I would like to do this is by (ab)using the internal messaging
> system. Is there a way I can create an OutputCollector which submits
> records to a Task ID that I provide? That way, I would not need to
> worry if Storm is using ZeroMQ or Netty under the hood.
> 
> Thanks in advance,
> 
> Pieter-Jan Van Aeken
> 
>