You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Spico Florin <sp...@gmail.com> on 2014/09/05 10:09:21 UTC
Processing large serial data flow strategy
Hello!
I have a test case that involves a dataflow containing a spout and N
serial bolts. For better understanding the dataflow looks like this:
Spout->B_1->B_2->...->B_N
where the "->" repersents the transition of the data from one bolt to
another.
Given the fact that by default Storm is using the EvenScheduler algorithm
to distribute the workload between the nodes, you'll encounter the network
latency issue.
In this particular test case, what will be your advice to use as a
Scheduler? To implement one by myself (custom scheduler) or check different
approaches?
I look forward for your suggestions/ideas.
Best regards,
Florin
Re: Processing large serial data flow strategy
Posted by Vikas Agarwal <vi...@infoobjects.com>.
That is exactly what the paper, I shared, says. There are two ways for
doing that, offline and online. Offline decides the hot edges only once and
assign bolts on hot edges to same machine while online one takes it to the
next step with adaptive changes for hot edges detection.
On Fri, Sep 5, 2014 at 4:55 PM, Spico Florin <sp...@gmail.com> wrote:
> Hello!
> Thank you for your responses. My concerns are related with the fact that
> the bolts will be placed in a cluster in a round robin fashion. Thus,
> you'll encounter the latency due to the network transfer between nodes. In
> my opinion, in my case the more I collocate the bolts on the same node (and
> also slot) the more I'll decrease the network latency and thus the time.
> I'm I right with my assumptions?
> Regards,
> Florin
>
>
> On Fri, Sep 5, 2014 at 1:36 PM, Vikas Agarwal <vi...@infoobjects.com>
> wrote:
>
>> There are schedulers which can adapt themselves based on topology
>> workload and historical performances.
>>
>> I found a good paper on this:
>> http://www.dis.uniroma1.it/~midlab/articoli/ABQ13storm.pdf
>>
>>
>> On Fri, Sep 5, 2014 at 3:20 PM, Michael Vogiatzis <
>> michaelvogiatzis@gmail.com> wrote:
>>
>>> Hi Spico,
>>>
>>> The Scheduler mechanism assigns tasks to available worker slots, but
>>> does not distribute the workload in real time continuously. Therefore the
>>> scheduler approach is unrelated to the network latency your topology may
>>> encounter.
>>>
>>> An example case for implementing a custom scheduler for network issues
>>> could be when a node with a faster connection than the rest contacts a 3rd
>>> party API and therefore makes sense to assign all tasks that make these
>>> calls to that node.
>>>
>>> Hope that helps.
>>> Regards,
>>> Michael
>>>
>>>
>>> On Fri, Sep 5, 2014 at 9:09 AM, Spico Florin <sp...@gmail.com>
>>> wrote:
>>>
>>>> Hello!
>>>> I have a test case that involves a dataflow containing a spout and N
>>>> serial bolts. For better understanding the dataflow looks like this:
>>>> Spout->B_1->B_2->...->B_N
>>>> where the "->" repersents the transition of the data from one bolt to
>>>> another.
>>>> Given the fact that by default Storm is using the EvenScheduler
>>>> algorithm to distribute the workload between the nodes, you'll encounter
>>>> the network latency issue.
>>>> In this particular test case, what will be your advice to use as a
>>>> Scheduler? To implement one by myself (custom scheduler) or check different
>>>> approaches?
>>>>
>>>> I look forward for your suggestions/ideas.
>>>> Best regards,
>>>> Florin
>>>>
>>>
>>>
>>>
>>> --
>>> Michael Vogiatzis
>>> Twitter: @mvogiatzis <https://twitter.com/mvogiatzis>
>>> http://micvog.com/
>>>
>>
>>
>>
>> --
>> Regards,
>> Vikas Agarwal
>> 91 – 9928301411
>>
>> InfoObjects, Inc.
>> Execution Matters
>> http://www.infoobjects.com
>> 2041 Mission College Boulevard, #280
>> Santa Clara, CA 95054
>> +1 (408) 988-2000 Work
>> +1 (408) 716-2726 Fax
>>
>>
>
--
Regards,
Vikas Agarwal
91 – 9928301411
InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
Re: Processing large serial data flow strategy
Posted by Spico Florin <sp...@gmail.com>.
Hello!
Thank you for your responses. My concerns are related with the fact that
the bolts will be placed in a cluster in a round robin fashion. Thus,
you'll encounter the latency due to the network transfer between nodes. In
my opinion, in my case the more I collocate the bolts on the same node (and
also slot) the more I'll decrease the network latency and thus the time.
I'm I right with my assumptions?
Regards,
Florin
On Fri, Sep 5, 2014 at 1:36 PM, Vikas Agarwal <vi...@infoobjects.com> wrote:
> There are schedulers which can adapt themselves based on topology workload
> and historical performances.
>
> I found a good paper on this:
> http://www.dis.uniroma1.it/~midlab/articoli/ABQ13storm.pdf
>
>
> On Fri, Sep 5, 2014 at 3:20 PM, Michael Vogiatzis <
> michaelvogiatzis@gmail.com> wrote:
>
>> Hi Spico,
>>
>> The Scheduler mechanism assigns tasks to available worker slots, but does
>> not distribute the workload in real time continuously. Therefore the
>> scheduler approach is unrelated to the network latency your topology may
>> encounter.
>>
>> An example case for implementing a custom scheduler for network issues
>> could be when a node with a faster connection than the rest contacts a 3rd
>> party API and therefore makes sense to assign all tasks that make these
>> calls to that node.
>>
>> Hope that helps.
>> Regards,
>> Michael
>>
>>
>> On Fri, Sep 5, 2014 at 9:09 AM, Spico Florin <sp...@gmail.com>
>> wrote:
>>
>>> Hello!
>>> I have a test case that involves a dataflow containing a spout and N
>>> serial bolts. For better understanding the dataflow looks like this:
>>> Spout->B_1->B_2->...->B_N
>>> where the "->" repersents the transition of the data from one bolt to
>>> another.
>>> Given the fact that by default Storm is using the EvenScheduler
>>> algorithm to distribute the workload between the nodes, you'll encounter
>>> the network latency issue.
>>> In this particular test case, what will be your advice to use as a
>>> Scheduler? To implement one by myself (custom scheduler) or check different
>>> approaches?
>>>
>>> I look forward for your suggestions/ideas.
>>> Best regards,
>>> Florin
>>>
>>
>>
>>
>> --
>> Michael Vogiatzis
>> Twitter: @mvogiatzis <https://twitter.com/mvogiatzis>
>> http://micvog.com/
>>
>
>
>
> --
> Regards,
> Vikas Agarwal
> 91 – 9928301411
>
> InfoObjects, Inc.
> Execution Matters
> http://www.infoobjects.com
> 2041 Mission College Boulevard, #280
> Santa Clara, CA 95054
> +1 (408) 988-2000 Work
> +1 (408) 716-2726 Fax
>
>
Re: Processing large serial data flow strategy
Posted by Vikas Agarwal <vi...@infoobjects.com>.
There are schedulers which can adapt themselves based on topology workload
and historical performances.
I found a good paper on this:
http://www.dis.uniroma1.it/~midlab/articoli/ABQ13storm.pdf
On Fri, Sep 5, 2014 at 3:20 PM, Michael Vogiatzis <
michaelvogiatzis@gmail.com> wrote:
> Hi Spico,
>
> The Scheduler mechanism assigns tasks to available worker slots, but does
> not distribute the workload in real time continuously. Therefore the
> scheduler approach is unrelated to the network latency your topology may
> encounter.
>
> An example case for implementing a custom scheduler for network issues
> could be when a node with a faster connection than the rest contacts a 3rd
> party API and therefore makes sense to assign all tasks that make these
> calls to that node.
>
> Hope that helps.
> Regards,
> Michael
>
>
> On Fri, Sep 5, 2014 at 9:09 AM, Spico Florin <sp...@gmail.com>
> wrote:
>
>> Hello!
>> I have a test case that involves a dataflow containing a spout and N
>> serial bolts. For better understanding the dataflow looks like this:
>> Spout->B_1->B_2->...->B_N
>> where the "->" repersents the transition of the data from one bolt to
>> another.
>> Given the fact that by default Storm is using the EvenScheduler algorithm
>> to distribute the workload between the nodes, you'll encounter the network
>> latency issue.
>> In this particular test case, what will be your advice to use as a
>> Scheduler? To implement one by myself (custom scheduler) or check different
>> approaches?
>>
>> I look forward for your suggestions/ideas.
>> Best regards,
>> Florin
>>
>
>
>
> --
> Michael Vogiatzis
> Twitter: @mvogiatzis <https://twitter.com/mvogiatzis>
> http://micvog.com/
>
--
Regards,
Vikas Agarwal
91 – 9928301411
InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
Re: Processing large serial data flow strategy
Posted by Michael Vogiatzis <mi...@gmail.com>.
Hi Spico,
The Scheduler mechanism assigns tasks to available worker slots, but does
not distribute the workload in real time continuously. Therefore the
scheduler approach is unrelated to the network latency your topology may
encounter.
An example case for implementing a custom scheduler for network issues
could be when a node with a faster connection than the rest contacts a 3rd
party API and therefore makes sense to assign all tasks that make these
calls to that node.
Hope that helps.
Regards,
Michael
On Fri, Sep 5, 2014 at 9:09 AM, Spico Florin <sp...@gmail.com> wrote:
> Hello!
> I have a test case that involves a dataflow containing a spout and N
> serial bolts. For better understanding the dataflow looks like this:
> Spout->B_1->B_2->...->B_N
> where the "->" repersents the transition of the data from one bolt to
> another.
> Given the fact that by default Storm is using the EvenScheduler algorithm
> to distribute the workload between the nodes, you'll encounter the network
> latency issue.
> In this particular test case, what will be your advice to use as a
> Scheduler? To implement one by myself (custom scheduler) or check different
> approaches?
>
> I look forward for your suggestions/ideas.
> Best regards,
> Florin
>
--
Michael Vogiatzis
Twitter: @mvogiatzis <https://twitter.com/mvogiatzis>
http://micvog.com/