You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Joel Samuelsson <sa...@gmail.com> on 2014/07/01 14:38:25 UTC
Reduce number of imports
We are using storm bolts written in python. In the bolt we have a
dependency that takes a lot of time to import. Is there a way to make the
import only once per instance of the bolt?
Best regards,
Joel
Re: Reduce number of imports
Posted by Mark Hu <ma...@gmail.com>.
Sorry about the confusion - I was responding to only the case of
initialization per instance of a bolt which can be done through
post_intialize() if your python bolt implementation class inherits from
storm.BasicBolt.
In our case, we originally had a python implementation, but have been
transitioning to java for stability and reliability reasons due to our
specific use-case. I apologize, python can be reasonable depending on your
needs.
On Thu, Jul 3, 2014 at 12:34 AM, Joel Samuelsson <sa...@gmail.com>
wrote:
> Thanks for your reply.
>
> I am not sure I understand. How would defining a function make sure it is
> run only once? I want all bolts that run on the same machine (or at least
> in the same process) to share one import.
>
> Is it performance-wise you can't recommend Python or another reason?
>
> Best regards,
> Joel
>
>
> 2014-07-03 4:51 GMT+02:00 Mark Hu <ma...@gmail.com>:
>
> I believe the below is a stub you can flesh out.
>>
>> def post_initialize( self ):
>> <flesh out>
>>
>> Please note that I can't recommend Storm and Python; language layers
>> begin with Java -> Jruby -> Python, and the text pipe in between Jruby and
>> Python makes it really hard to recover from. I recommend either Java, and
>> worst case Jruby, for any production work you're thinking about.
>>
>>
>> On Wed, Jul 2, 2014 at 8:41 AM, Joel Samuelsson <
>> samuelsson.joel@gmail.com> wrote:
>>
>>> Once per supervisor / worker, I should say.
>>>
>>> Best regards,
>>> Joel
>>>
>>>
>>> 2014-07-01 14:38 GMT+02:00 Joel Samuelsson <sa...@gmail.com>:
>>>
>>> We are using storm bolts written in python. In the bolt we have a
>>>> dependency that takes a lot of time to import. Is there a way to make the
>>>> import only once per instance of the bolt?
>>>>
>>>> Best regards,
>>>> Joel
>>>>
>>>
>>>
>>
>
Re: Reduce number of imports
Posted by Joel Samuelsson <sa...@gmail.com>.
Thanks for your reply.
I am not sure I understand. How would defining a function make sure it is
run only once? I want all bolts that run on the same machine (or at least
in the same process) to share one import.
Is it performance-wise you can't recommend Python or another reason?
Best regards,
Joel
2014-07-03 4:51 GMT+02:00 Mark Hu <ma...@gmail.com>:
> I believe the below is a stub you can flesh out.
>
> def post_initialize( self ):
> <flesh out>
>
> Please note that I can't recommend Storm and Python; language layers begin
> with Java -> Jruby -> Python, and the text pipe in between Jruby and Python
> makes it really hard to recover from. I recommend either Java, and worst
> case Jruby, for any production work you're thinking about.
>
>
> On Wed, Jul 2, 2014 at 8:41 AM, Joel Samuelsson <samuelsson.joel@gmail.com
> > wrote:
>
>> Once per supervisor / worker, I should say.
>>
>> Best regards,
>> Joel
>>
>>
>> 2014-07-01 14:38 GMT+02:00 Joel Samuelsson <sa...@gmail.com>:
>>
>> We are using storm bolts written in python. In the bolt we have a
>>> dependency that takes a lot of time to import. Is there a way to make the
>>> import only once per instance of the bolt?
>>>
>>> Best regards,
>>> Joel
>>>
>>
>>
>
Re: Reduce number of imports
Posted by Mark Hu <ma...@gmail.com>.
I believe the below is a stub you can flesh out.
def post_initialize( self ):
<flesh out>
Please note that I can't recommend Storm and Python; language layers begin
with Java -> Jruby -> Python, and the text pipe in between Jruby and Python
makes it really hard to recover from. I recommend either Java, and worst
case Jruby, for any production work you're thinking about.
On Wed, Jul 2, 2014 at 8:41 AM, Joel Samuelsson <sa...@gmail.com>
wrote:
> Once per supervisor / worker, I should say.
>
> Best regards,
> Joel
>
>
> 2014-07-01 14:38 GMT+02:00 Joel Samuelsson <sa...@gmail.com>:
>
> We are using storm bolts written in python. In the bolt we have a
>> dependency that takes a lot of time to import. Is there a way to make the
>> import only once per instance of the bolt?
>>
>> Best regards,
>> Joel
>>
>
>
Re: Reduce number of imports
Posted by Joel Samuelsson <sa...@gmail.com>.
Once per supervisor / worker, I should say.
Best regards,
Joel
2014-07-01 14:38 GMT+02:00 Joel Samuelsson <sa...@gmail.com>:
> We are using storm bolts written in python. In the bolt we have a
> dependency that takes a lot of time to import. Is there a way to make the
> import only once per instance of the bolt?
>
> Best regards,
> Joel
>