You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Franca van Kaam <fr...@gmail.com> on 2015/05/06 12:18:30 UTC
Forcing code distribution+geographic position of nodes
Hello,
I am fairly new to Storm and Clojure and I am wondering if there is any way
to force the way Nimbus distributes the code to the worker nodes. Also is
there any way to take geographic position of the nodes into account? If
this does not exist yet could you pinpoint me to the right location in the
source code where this could be implemented?
Thanks in advance and best regards,
Franca van Kaam
Re: Forcing code distribution+geographic position of nodes
Posted by "Matthias J. Sax" <mj...@informatik.hu-berlin.de>.
That is possible, that need very careful design to get it right.
The example show in the link I posted below, uses some meta data as
well. This meta data could be your geographical coordinate I guess.
-Matthias
On 05/06/2015 01:58 PM, Franca van Kaam wrote:
> Ok I will try it, thanks a lot ;)
>
> how about geographical coordinates? is it possible to put this as a
> property of the node and use it in the scheduling as well as in grouping to
> redirect tuples?
>
> On Wed, May 6, 2015 at 1:45 PM, Matthias J. Sax <
> mjsax@informatik.hu-berlin.de> wrote:
>
>> Depends on your implementation ;)
>>
>> Storm's default scheduler is implementing the same interface, so it is
>> possible the do it in an efficient way.
>>
>> -Matthias
>>
>>
>> On 05/06/2015 12:54 PM, Franca van Kaam wrote:
>>> Thanks
>>>
>>> Might be a stupid question but would this be efficient if I want to use
>>> this for all of the nodes in my cluster? certain groups of nodes should
>>> have some of the bolts on them, and other nodes should be executing
>>> different bolts... Is it possible too do this on such a large scale?
>>>
>>> On Wed, May 6, 2015 at 12:42 PM, Matthias J. Sax <
>>> mjsax@informatik.hu-berlin.de> wrote:
>>>
>>>> Hi,
>>>>
>>>> you can implement your own custom scheduler. An example how to do this
>>>> is given here.
>>>>
>>>>
>>>>
>> https://xumingming.sinaapp.com/885/twitter-storm-how-to-develop-a-pluggable-scheduler/
>>>>
>>>>
>>>> -Matthias
>>>>
>>>>
>>>>
>>>> On 05/06/2015 12:18 PM, Franca van Kaam wrote:
>>>>> Hello,
>>>>>
>>>>> I am fairly new to Storm and Clojure and I am wondering if there is any
>>>> way
>>>>> to force the way Nimbus distributes the code to the worker nodes. Also
>> is
>>>>> there any way to take geographic position of the nodes into account? If
>>>>> this does not exist yet could you pinpoint me to the right location in
>>>> the
>>>>> source code where this could be implemented?
>>>>>
>>>>> Thanks in advance and best regards,
>>>>>
>>>>> Franca van Kaam
>>>>>
>>>>
>>>>
>>>
>>
>>
>
Re: Forcing code distribution+geographic position of nodes
Posted by Franca van Kaam <fr...@gmail.com>.
Ok I will try it, thanks a lot ;)
how about geographical coordinates? is it possible to put this as a
property of the node and use it in the scheduling as well as in grouping to
redirect tuples?
On Wed, May 6, 2015 at 1:45 PM, Matthias J. Sax <
mjsax@informatik.hu-berlin.de> wrote:
> Depends on your implementation ;)
>
> Storm's default scheduler is implementing the same interface, so it is
> possible the do it in an efficient way.
>
> -Matthias
>
>
> On 05/06/2015 12:54 PM, Franca van Kaam wrote:
> > Thanks
> >
> > Might be a stupid question but would this be efficient if I want to use
> > this for all of the nodes in my cluster? certain groups of nodes should
> > have some of the bolts on them, and other nodes should be executing
> > different bolts... Is it possible too do this on such a large scale?
> >
> > On Wed, May 6, 2015 at 12:42 PM, Matthias J. Sax <
> > mjsax@informatik.hu-berlin.de> wrote:
> >
> >> Hi,
> >>
> >> you can implement your own custom scheduler. An example how to do this
> >> is given here.
> >>
> >>
> >>
> https://xumingming.sinaapp.com/885/twitter-storm-how-to-develop-a-pluggable-scheduler/
> >>
> >>
> >> -Matthias
> >>
> >>
> >>
> >> On 05/06/2015 12:18 PM, Franca van Kaam wrote:
> >>> Hello,
> >>>
> >>> I am fairly new to Storm and Clojure and I am wondering if there is any
> >> way
> >>> to force the way Nimbus distributes the code to the worker nodes. Also
> is
> >>> there any way to take geographic position of the nodes into account? If
> >>> this does not exist yet could you pinpoint me to the right location in
> >> the
> >>> source code where this could be implemented?
> >>>
> >>> Thanks in advance and best regards,
> >>>
> >>> Franca van Kaam
> >>>
> >>
> >>
> >
>
>
Re: Forcing code distribution+geographic position of nodes
Posted by "Matthias J. Sax" <mj...@informatik.hu-berlin.de>.
Depends on your implementation ;)
Storm's default scheduler is implementing the same interface, so it is
possible the do it in an efficient way.
-Matthias
On 05/06/2015 12:54 PM, Franca van Kaam wrote:
> Thanks
>
> Might be a stupid question but would this be efficient if I want to use
> this for all of the nodes in my cluster? certain groups of nodes should
> have some of the bolts on them, and other nodes should be executing
> different bolts... Is it possible too do this on such a large scale?
>
> On Wed, May 6, 2015 at 12:42 PM, Matthias J. Sax <
> mjsax@informatik.hu-berlin.de> wrote:
>
>> Hi,
>>
>> you can implement your own custom scheduler. An example how to do this
>> is given here.
>>
>>
>> https://xumingming.sinaapp.com/885/twitter-storm-how-to-develop-a-pluggable-scheduler/
>>
>>
>> -Matthias
>>
>>
>>
>> On 05/06/2015 12:18 PM, Franca van Kaam wrote:
>>> Hello,
>>>
>>> I am fairly new to Storm and Clojure and I am wondering if there is any
>> way
>>> to force the way Nimbus distributes the code to the worker nodes. Also is
>>> there any way to take geographic position of the nodes into account? If
>>> this does not exist yet could you pinpoint me to the right location in
>> the
>>> source code where this could be implemented?
>>>
>>> Thanks in advance and best regards,
>>>
>>> Franca van Kaam
>>>
>>
>>
>
Re: Forcing code distribution+geographic position of nodes
Posted by Franca van Kaam <fr...@gmail.com>.
Thanks
Might be a stupid question but would this be efficient if I want to use
this for all of the nodes in my cluster? certain groups of nodes should
have some of the bolts on them, and other nodes should be executing
different bolts... Is it possible too do this on such a large scale?
On Wed, May 6, 2015 at 12:42 PM, Matthias J. Sax <
mjsax@informatik.hu-berlin.de> wrote:
> Hi,
>
> you can implement your own custom scheduler. An example how to do this
> is given here.
>
>
> https://xumingming.sinaapp.com/885/twitter-storm-how-to-develop-a-pluggable-scheduler/
>
>
> -Matthias
>
>
>
> On 05/06/2015 12:18 PM, Franca van Kaam wrote:
> > Hello,
> >
> > I am fairly new to Storm and Clojure and I am wondering if there is any
> way
> > to force the way Nimbus distributes the code to the worker nodes. Also is
> > there any way to take geographic position of the nodes into account? If
> > this does not exist yet could you pinpoint me to the right location in
> the
> > source code where this could be implemented?
> >
> > Thanks in advance and best regards,
> >
> > Franca van Kaam
> >
>
>
Re: Forcing code distribution+geographic position of nodes
Posted by "Matthias J. Sax" <mj...@informatik.hu-berlin.de>.
Hi,
you can implement your own custom scheduler. An example how to do this
is given here.
https://xumingming.sinaapp.com/885/twitter-storm-how-to-develop-a-pluggable-scheduler/
-Matthias
On 05/06/2015 12:18 PM, Franca van Kaam wrote:
> Hello,
>
> I am fairly new to Storm and Clojure and I am wondering if there is any way
> to force the way Nimbus distributes the code to the worker nodes. Also is
> there any way to take geographic position of the nodes into account? If
> this does not exist yet could you pinpoint me to the right location in the
> source code where this could be implemented?
>
> Thanks in advance and best regards,
>
> Franca van Kaam
>