You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Swara Desai <sw...@gmail.com> on 2014/03/26 11:43:10 UTC
Using storm for heavy processing
Hi,
I am evaluating storm for some processing that might take upto one hour.
This implies that the bolt would be processing one tuple for an hour.
Can storm handle cases which have long processing time? Other than
configuring things like timeout, have any of you noticed any issues with
heavy processing?
Thanks in advance
Re: Using storm for heavy processing
Posted by Naresh Bhatti <na...@gmail.com>.
Padma, the exception were in our application code. We observed the storm
logs on the supervisors to identify the problems.
~Naresh
On Thu, Mar 27, 2014 at 4:50 AM, Nathan Leung <nc...@gmail.com> wrote:
> Personally I would not use storm for such long running computations. I
> would lean towards a batch processing system such as hadoop.
> On Mar 27, 2014 12:34 AM, "Swara Desai" <sw...@gmail.com> wrote:
>
>> Hi,
>> Can someone please shed some light on this? Has anyone used storm to
>> handle long processing, like video transcoding?
>>
>> Thanks
>>
>>
>> On Wed, Mar 26, 2014 at 4:13 PM, Swara Desai <sw...@gmail.com>wrote:
>>
>>> Hi,
>>> I am evaluating storm for some processing that might take upto one hour.
>>> This implies that the bolt would be processing one tuple for an hour.
>>> Can storm handle cases which have long processing time? Other than
>>> configuring things like timeout, have any of you noticed any issues with
>>> heavy processing?
>>>
>>> Thanks in advance
>>>
>>
>>
Re: Using storm for heavy processing
Posted by Nathan Leung <nc...@gmail.com>.
Personally I would not use storm for such long running computations. I
would lean towards a batch processing system such as hadoop.
On Mar 27, 2014 12:34 AM, "Swara Desai" <sw...@gmail.com> wrote:
> Hi,
> Can someone please shed some light on this? Has anyone used storm to
> handle long processing, like video transcoding?
>
> Thanks
>
>
> On Wed, Mar 26, 2014 at 4:13 PM, Swara Desai <sw...@gmail.com> wrote:
>
>> Hi,
>> I am evaluating storm for some processing that might take upto one hour.
>> This implies that the bolt would be processing one tuple for an hour.
>> Can storm handle cases which have long processing time? Other than
>> configuring things like timeout, have any of you noticed any issues with
>> heavy processing?
>>
>> Thanks in advance
>>
>
>
Re: Using storm for heavy processing
Posted by Swara Desai <sw...@gmail.com>.
Hi,
Can someone please shed some light on this? Has anyone used storm to handle
long processing, like video transcoding?
Thanks
On Wed, Mar 26, 2014 at 4:13 PM, Swara Desai <sw...@gmail.com> wrote:
> Hi,
> I am evaluating storm for some processing that might take upto one hour.
> This implies that the bolt would be processing one tuple for an hour.
> Can storm handle cases which have long processing time? Other than
> configuring things like timeout, have any of you noticed any issues with
> heavy processing?
>
> Thanks in advance
>
Re: Using storm for heavy processing
Posted by padma priya chitturi <pa...@gmail.com>.
Hi Naresh,
May i know what kind of exceptions were thrown by the bolts ? How did you
fix the problem which you faced...your inputs might help me in correlating
with my issue.
--Padma Ch.
On Wed, Mar 26, 2014 at 8:16 PM, Naresh <na...@gmail.com> wrote:
> Hi Padma
> It is hard to guess what problem you are facing.
> I can share a problem that we faced earlier on that was slowing the
> processing. You might have a similar issue.
> Our bolts were throwing exceptions which was causing restart of our
> workers. This slowed down the processing. After we fixed all our bolts the
> topology is working at great speed bounded only by external resources.
> Hope that helps.
> Regards
> Naresh
>
> On Mar 26, 2014, at 5:58 AM, padma priya chitturi <pa...@gmail.com>
> wrote:
>
> Hi,
>
> When using storm to process 1GB data file (has approx 200 million tuples),
> the execution proceeded at first while at a later point, processing has
> been stuck and the workers have restarted processing the file from the
> beginning. Does the issue lie with IO ? As storm can process million of
> tuples per second, 200 million tuples wouldn't be an issue. Can someone
> suggest me on this...
>
>
> Thanks,
> Padma Ch
>
>
> On Wed, Mar 26, 2014 at 4:13 PM, Swara Desai <sw...@gmail.com> wrote:
>
>> Hi,
>> I am evaluating storm for some processing that might take upto one hour.
>> This implies that the bolt would be processing one tuple for an hour.
>> Can storm handle cases which have long processing time? Other than
>> configuring things like timeout, have any of you noticed any issues with
>> heavy processing?
>>
>> Thanks in advance
>>
>
>
Re: Using storm for heavy processing
Posted by Naresh <na...@gmail.com>.
Hi Padma
It is hard to guess what problem you are facing.
I can share a problem that we faced earlier on that was slowing the processing. You might have a similar issue.
Our bolts were throwing exceptions which was causing restart of our workers. This slowed down the processing. After we fixed all our bolts the topology is working at great speed bounded only by external resources.
Hope that helps.
Regards
Naresh
> On Mar 26, 2014, at 5:58 AM, padma priya chitturi <pa...@gmail.com> wrote:
>
> Hi,
>
> When using storm to process 1GB data file (has approx 200 million tuples), the execution proceeded at first while at a later point, processing has been stuck and the workers have restarted processing the file from the beginning. Does the issue lie with IO ? As storm can process million of tuples per second, 200 million tuples wouldn't be an issue. Can someone suggest me on this...
>
>
> Thanks,
> Padma Ch
>
>
>> On Wed, Mar 26, 2014 at 4:13 PM, Swara Desai <sw...@gmail.com> wrote:
>> Hi,
>> I am evaluating storm for some processing that might take upto one hour. This implies that the bolt would be processing one tuple for an hour.
>> Can storm handle cases which have long processing time? Other than configuring things like timeout, have any of you noticed any issues with heavy processing?
>>
>> Thanks in advance
>
Re: Using storm for heavy processing
Posted by padma priya chitturi <pa...@gmail.com>.
Hi,
When using storm to process 1GB data file (has approx 200 million tuples),
the execution proceeded at first while at a later point, processing has
been stuck and the workers have restarted processing the file from the
beginning. Does the issue lie with IO ? As storm can process million of
tuples per second, 200 million tuples wouldn't be an issue. Can someone
suggest me on this...
Thanks,
Padma Ch
On Wed, Mar 26, 2014 at 4:13 PM, Swara Desai <sw...@gmail.com> wrote:
> Hi,
> I am evaluating storm for some processing that might take upto one hour.
> This implies that the bolt would be processing one tuple for an hour.
> Can storm handle cases which have long processing time? Other than
> configuring things like timeout, have any of you noticed any issues with
> heavy processing?
>
> Thanks in advance
>