Hooks into Shell bolt

[Hemanth Yamijala <yh...@gmail.com>]

Hi,

We are planning to integrate some Python modules with Storm. I have been
able to get the integration going quite easily using the ShellBolt and
python Storm module, following the example in storm-starter.

For productionizing, after basic processing in Python, there are some steps
that we would like to perform that are already implemented as Java
libraries. Instead of re-implementing those, we were thinking of ways we
can reuse the existing implementation. One obvious approach is to implement
these in a downstream bolt. Another approach would involve having an
ability to process these in the handleEmit call of the ShellBolt.

There are no such hooks that currently exist though. Has anyone had such
requirements ? Any thoughts on how this can be achieved without modifying
Shell Bolt ? If such hooks are present in ShellBolt, would it be
interesting to the community in general ?

Thanks
hemanth

Re: Hooks into Shell bolt

[Hemanth Yamijala <yh...@gmail.com>]

Thanks for your inputs, Itai.

On Tue, Jan 6, 2015 at 1:09 AM, Itai Frenkel <It...@forter.com> wrote:

>  You should use a downstream bolt since it is easier to maintain. If you
> have to, you can inject a wrapper to the collector that the shellbolt uses
> and manipulate the emitted list before actually forwarding it to the real
> collector.
>  ------------------------------
> *From:* Hemanth Yamijala <yh...@gmail.com>
> *Sent:* Monday, January 5, 2015 7:20 PM
> *To:* user@storm.apache.org
> *Subject:* Hooks into Shell bolt
>
>  Hi,
>
>  We are planning to integrate some Python modules with Storm. I have been
> able to get the integration going quite easily using the ShellBolt and
> python Storm module, following the example in storm-starter.
>
>  For productionizing, after basic processing in Python, there are some
> steps that we would like to perform that are already implemented as Java
> libraries. Instead of re-implementing those, we were thinking of ways we
> can reuse the existing implementation. One obvious approach is to implement
> these in a downstream bolt. Another approach would involve having an
> ability to process these in the handleEmit call of the ShellBolt.
>
>  There are no such hooks that currently exist though. Has anyone had such
> requirements ? Any thoughts on how this can be achieved without modifying
> Shell Bolt ? If such hooks are present in ShellBolt, would it be
> interesting to the community in general ?
>
>  Thanks
> hemanth
>

Re: Hooks into Shell bolt

[Itai Frenkel <It...@forter.com>]

You should use a downstream bolt since it is easier to maintain. If you have to, you can inject a wrapper to the collector that the shellbolt uses and manipulate the emitted list before actually forwarding it to the real collector.

________________________________
From: Hemanth Yamijala <yh...@gmail.com>
Sent: Monday, January 5, 2015 7:20 PM
To: user@storm.apache.org
Subject: Hooks into Shell bolt

Hi,

We are planning to integrate some Python modules with Storm. I have been able to get the integration going quite easily using the ShellBolt and python Storm module, following the example in storm-starter.

For productionizing, after basic processing in Python, there are some steps that we would like to perform that are already implemented as Java libraries. Instead of re-implementing those, we were thinking of ways we can reuse the existing implementation. One obvious approach is to implement these in a downstream bolt. Another approach would involve having an ability to process these in the handleEmit call of the ShellBolt.

There are no such hooks that currently exist though. Has anyone had such requirements ? Any thoughts on how this can be achieved without modifying Shell Bolt ? If such hooks are present in ShellBolt, would it be interesting to the community in general ?

Thanks
hemanth