You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gearpump.apache.org by Karol Brejna <ka...@apache.org> on 2016/05/11 07:49:52 UTC
Algorithm for assigning Executor to Workers?
What are the criteria that decide which Executor/Tasks goes to which
Workers right now?
I am considering the following scenario.
Let's assume that we have a cluster with some "specialized" nodes (for
example a worker on IoT device that is responsible for controlling some
actuators, dedicated workers to achieve data locality, some hardware
specific - mem/cpu - tasks, etc.).
Are we able to "hint" Gearpump for certain tasks to prefer some nodes over
others? (For example by labeling tasks and nodes or using some other
"awareness" mechanism?)
Regards,
Karol
Re: Algorithm for assigning Executor to Workers?
Posted by Karol Brejna <ka...@apache.org>.
Huafeng, Manu - thank you for quick response.
As for Task id, I agree it's not ideal :)
Let me read the document to get better understanding of the matter.
Karol
śr., 11.05.2016 o 10:19 użytkownik Manu Zhang <ow...@gmail.com>
napisał:
> Hi Karol,
>
> Weihua has previously written up this nice doc IoT features map with
> Gearpump
> <
> https://docs.google.com/document/d/1kagryYsBmtwVIE5E8afgH9K_O2a0_VdtlAY9auVAQQM/edit?usp=sharing
> >.
> You may find more details about how Gearpump can work in the environment of
> cluster and edge devices and the gaps to fill.
>
> On Wed, May 11, 2016 at 4:11 PM Vincent Wang <fv...@gmail.com> wrote:
>
> > You can add a configuration when submitting the application, the format
> is:
> >
> > gearpump {
> > streaming {
> > localities {
> > app1: { //This is the application name to be submitted
> > workerId: [ //The worker id where to launch Task0_0 and
> Task0_1
> > TaskId(0,0), TaskId(0,1)
> > ]
> > }
> > }
> > }
> > }
> >
> > But I think the current way is not flexible when there are lots of nodes
> in
> > a DAG because user have no idea what each Task's task ID is.
> >
> > Thanks,
> > Huafeng
> >
> > 2016-05-11 15:49 GMT+08:00 Karol Brejna <ka...@apache.org>:
> >
> > > What are the criteria that decide which Executor/Tasks goes to which
> > > Workers right now?
> > >
> > > I am considering the following scenario.
> > >
> > > Let's assume that we have a cluster with some "specialized" nodes (for
> > > example a worker on IoT device that is responsible for controlling some
> > > actuators, dedicated workers to achieve data locality, some hardware
> > > specific - mem/cpu - tasks, etc.).
> > >
> > > Are we able to "hint" Gearpump for certain tasks to prefer some nodes
> > over
> > > others? (For example by labeling tasks and nodes or using some other
> > > "awareness" mechanism?)
> > >
> > > Regards,
> > > Karol
> > >
> >
>
Re: Algorithm for assigning Executor to Workers?
Posted by Manu Zhang <ow...@gmail.com>.
Hi Karol,
Weihua has previously written up this nice doc IoT features map with
Gearpump
<https://docs.google.com/document/d/1kagryYsBmtwVIE5E8afgH9K_O2a0_VdtlAY9auVAQQM/edit?usp=sharing>.
You may find more details about how Gearpump can work in the environment of
cluster and edge devices and the gaps to fill.
On Wed, May 11, 2016 at 4:11 PM Vincent Wang <fv...@gmail.com> wrote:
> You can add a configuration when submitting the application, the format is:
>
> gearpump {
> streaming {
> localities {
> app1: { //This is the application name to be submitted
> workerId: [ //The worker id where to launch Task0_0 and Task0_1
> TaskId(0,0), TaskId(0,1)
> ]
> }
> }
> }
> }
>
> But I think the current way is not flexible when there are lots of nodes in
> a DAG because user have no idea what each Task's task ID is.
>
> Thanks,
> Huafeng
>
> 2016-05-11 15:49 GMT+08:00 Karol Brejna <ka...@apache.org>:
>
> > What are the criteria that decide which Executor/Tasks goes to which
> > Workers right now?
> >
> > I am considering the following scenario.
> >
> > Let's assume that we have a cluster with some "specialized" nodes (for
> > example a worker on IoT device that is responsible for controlling some
> > actuators, dedicated workers to achieve data locality, some hardware
> > specific - mem/cpu - tasks, etc.).
> >
> > Are we able to "hint" Gearpump for certain tasks to prefer some nodes
> over
> > others? (For example by labeling tasks and nodes or using some other
> > "awareness" mechanism?)
> >
> > Regards,
> > Karol
> >
>
Re: Algorithm for assigning Executor to Workers?
Posted by Kam Kasravi <ka...@gmail.com>.
Perhaps we can consider a new feature where a task label can be used instead of the task id.
On Wednesday, May 11, 2016 1:11 AM, Vincent Wang <fv...@gmail.com> wrote:
You can add a configuration when submitting the application, the format is:
gearpump {
streaming {
localities {
app1: { //This is the application name to be submitted
workerId: [ //The worker id where to launch Task0_0 and Task0_1
TaskId(0,0), TaskId(0,1)
]
}
}
}
}
But I think the current way is not flexible when there are lots of nodes in
a DAG because user have no idea what each Task's task ID is.
Thanks,
Huafeng
2016-05-11 15:49 GMT+08:00 Karol Brejna <ka...@apache.org>:
> What are the criteria that decide which Executor/Tasks goes to which
> Workers right now?
>
> I am considering the following scenario.
>
> Let's assume that we have a cluster with some "specialized" nodes (for
> example a worker on IoT device that is responsible for controlling some
> actuators, dedicated workers to achieve data locality, some hardware
> specific - mem/cpu - tasks, etc.).
>
> Are we able to "hint" Gearpump for certain tasks to prefer some nodes over
> others? (For example by labeling tasks and nodes or using some other
> "awareness" mechanism?)
>
> Regards,
> Karol
>
Re: Algorithm for assigning Executor to Workers?
Posted by Vincent Wang <fv...@gmail.com>.
You can add a configuration when submitting the application, the format is:
gearpump {
streaming {
localities {
app1: { //This is the application name to be submitted
workerId: [ //The worker id where to launch Task0_0 and Task0_1
TaskId(0,0), TaskId(0,1)
]
}
}
}
}
But I think the current way is not flexible when there are lots of nodes in
a DAG because user have no idea what each Task's task ID is.
Thanks,
Huafeng
2016-05-11 15:49 GMT+08:00 Karol Brejna <ka...@apache.org>:
> What are the criteria that decide which Executor/Tasks goes to which
> Workers right now?
>
> I am considering the following scenario.
>
> Let's assume that we have a cluster with some "specialized" nodes (for
> example a worker on IoT device that is responsible for controlling some
> actuators, dedicated workers to achieve data locality, some hardware
> specific - mem/cpu - tasks, etc.).
>
> Are we able to "hint" Gearpump for certain tasks to prefer some nodes over
> others? (For example by labeling tasks and nodes or using some other
> "awareness" mechanism?)
>
> Regards,
> Karol
>