Overriding execute method in ShellBolt

[adiya n <na...@yahoo.com>]

I tried out ShellBolt examples and it works like a charm.  I went through the multi-lang protocol doc as well and understand it at a high level. 


Now what I dont understand is the following:
-  With a shell bolt, how can you get the output of the external process (say python process) and do something with it and 

    then emit the tuple from the Java code?
-  This should be possible but somehow I have to make sure there is only one emit that is happening from my shellbolt. 

1. My Shellbolt gets the tuple
2. I then pass the data to the external python process
3. get the result/tuple from python process
4. Do something else with it in my java code and then emit the tuple to downstream bolt
   

How would I be able to do this?  Any examples/pointers would really help. So the flow would be:

public static class SomeBolt extends ShellBolt implements IRichBolt { 
                public SomeBolt(){ 
                        super("python", "some.py"); 

                } 
                @Override 
                public void declareOutputFields(OutputFieldsDeclarer declarer) { 
                        declarer.declare(new Fields("someData")); 
                } 
        } 

thanks
Aditya

Re: Overriding execute method in ShellBolt

[Harsha <st...@harsha.io>]

           I am not sure if shellbolt is the right way to go here. With
shellbolt it allows you to write your processing logic in python or
ruby. Since shellbolt implements the execute method which puts the
incoming tuples into processing queue which is taken by your python or
ruby script does some processing and emits the tuple. overriding
shellbolt execute is not a good idea. If you can push your processing
logic into python script and use shellbolt or implement a IRichBolt
from execute method call your python script catch output from that. You
can probably reuse
ShellProcess [1]http://nathanmarz.github.io/storm/doc/backtype/storm/ut
ils/ShellProcess.html.

-Harsha





On Sat, Jun 7, 2014, at 09:31 PM, adiya n wrote:

I tried out ShellBolt examples and it works like a charm.  I went
through the multi-lang protocol doc as well and understand it at a high
level.

Now what I dont understand is the following:
-  With a shell bolt, how can you get the output of the external
process (say python process) and do something with it and
    then emit the tuple from the Java code?
-  This should be possible but somehow I have to make sure there is
only one emit that is happening from my shellbolt.

1. My Shellbolt gets the tuple
2. I then pass the data to the external python process
3. get the result/tuple from python process
4. Do something else with it in my java code and then emit the tuple to
downstream bolt
How would I be able to do this?  Any examples/pointers would really
help. So the flow would be:


public static class SomeBolt extends ShellBolt implements IRichBolt {
                public SomeBolt(){
                        super("python", "some.py");
                }
                @Override
                public void declareOutputFields(OutputFieldsDeclarer
declarer) {
declarer.declare(new Fields("someData"));
                }
        }

thanks
Aditya

References

1. http://nathanmarz.github.io/storm/doc/backtype/storm/utils/ShellProcess.html