You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Hu...@dlr.de on 2016/09/21 05:41:22 UTC
RE: Support for tasks groups aka pods in Mesos
Hi,
Very cool feature.
I’ve seen some recent changes (MESOS-6096<https://issues.apache.org/jira/browse/MESOS-6096>) in the mesos-execute cli supporting grouped tasks which spawns the task-group on an arbitrary or specified agent node. Tested this yesterday and it works great. But if the task group holds a lot of one-off tasks (let’s say 50 tasks each 4 cores,…) how could one achieve that tasks are evenly distributed over several agents, depending on free resources? Don’t know if this is referred to pods, but do you think mesos-execute will support this?
Currently I’m using chronos & cook for batch scheduling, but I’d like to achieve that every batch/group of tasks runs in its own temporary framework like it does with mesos-execute
Thanks in advance & kind regards,
Hubert
——————————————————————————
Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR)
German Aerospace Center
German Remote Sensing Data Center | International Ground Segment | Oberpfaffenhofen | 82234 Wessling | Germany
Hubert Asamer
phone +49 (0) 8153 28-2894 | fax +49 (0) 8153 28-1443 | hubert.asamer@dlr.de<ma...@dlr.de>
www.DLR.de/eoc<http://www.dlr.de/eoc>
From: Vinod Kone [mailto:vinodkone@apache.org]
Sent: Dienstag, 9. August 2016 02:54
To: dev; user
Subject: Support for tasks groups aka pods in Mesos
Hi folks,
One of the most requested features in Mesos has been first class support for managing pod like containers. We finally have some time to focus and shepherd this work.
The epic tracking this work is : https://issues.apache.org/jira/browse/MESOS-2449
Design doc: https://issues.apache.org/jira/browse/MESOS-2449
Your feedback on the design will be most welcome. Once we get agreement on the design, we can start breaking down the epic into tickets.
Thanks,
Vinod & Jie
RE: Support for tasks groups aka pods in Mesos
Posted by Hu...@dlr.de.
Thanks for the clarification – I’ll try to dig into this and use execute.cpp as a starting point for my problem…
Kind Regards,
Hubert
From: Guangya Liu [mailto:gyliu513@gmail.com]
Sent: Mittwoch, 21. September 2016 09:19
To: user@mesos.apache.org
Cc: dev; Vinod Kone
Subject: Re: Support for tasks groups aka pods in Mesos
The answer is No, the taskGroup can be treated as Pod in Kubernetes, it will be a set of containers co-located and co-managed on an agent that share some resources (e.g., network namespace, volumes). For your case, you may want to split your 50 tasks to small task groups.
Also the `execute` cli is just an example framework and it can only launch one task group for now, you may want to enhance it if you want to launch multiple task groups.
On Wed, Sep 21, 2016 at 1:41 PM, <Hu...@dlr.de>> wrote:
Hi,
Very cool feature.
I’ve seen some recent changes (MESOS-6096<https://issues.apache.org/jira/browse/MESOS-6096>) in the mesos-execute cli supporting grouped tasks which spawns the task-group on an arbitrary or specified agent node. Tested this yesterday and it works great. But if the task group holds a lot of one-off tasks (let’s say 50 tasks each 4 cores,…) how could one achieve that tasks are evenly distributed over several agents, depending on free resources? Don’t know if this is referred to pods, but do you think mesos-execute will support this?
Currently I’m using chronos & cook for batch scheduling, but I’d like to achieve that every batch/group of tasks runs in its own temporary framework like it does with mesos-execute
Thanks in advance & kind regards,
Hubert
——————————————————————————
Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR)
German Aerospace Center
German Remote Sensing Data Center | International Ground Segment | Oberpfaffenhofen | 82234 Wessling | Germany
Hubert Asamer
phone +49 (0) 8153 28-2894<tel:%2B49%20%280%29%208153%2028-2894> | fax +49 (0) 8153 28-1443<tel:%2B49%20%280%29%208153%2028-1443> | hubert.asamer@dlr.de<ma...@dlr.de>
www.DLR.de/eoc<http://www.dlr.de/eoc>
From: Vinod Kone [mailto:vinodkone@apache.org<ma...@apache.org>]
Sent: Dienstag, 9. August 2016 02:54
To: dev; user
Subject: Support for tasks groups aka pods in Mesos
Hi folks,
One of the most requested features in Mesos has been first class support for managing pod like containers. We finally have some time to focus and shepherd this work.
The epic tracking this work is : https://issues.apache.org/jira/browse/MESOS-2449
Design doc: https://issues.apache.org/jira/browse/MESOS-2449
Your feedback on the design will be most welcome. Once we get agreement on the design, we can start breaking down the epic into tickets.
Thanks,
Vinod & Jie
RE: Support for tasks groups aka pods in Mesos
Posted by Hu...@dlr.de.
Thanks for the clarification – I’ll try to dig into this and use execute.cpp as a starting point for my problem…
Kind Regards,
Hubert
From: Guangya Liu [mailto:gyliu513@gmail.com]
Sent: Mittwoch, 21. September 2016 09:19
To: user@mesos.apache.org
Cc: dev; Vinod Kone
Subject: Re: Support for tasks groups aka pods in Mesos
The answer is No, the taskGroup can be treated as Pod in Kubernetes, it will be a set of containers co-located and co-managed on an agent that share some resources (e.g., network namespace, volumes). For your case, you may want to split your 50 tasks to small task groups.
Also the `execute` cli is just an example framework and it can only launch one task group for now, you may want to enhance it if you want to launch multiple task groups.
On Wed, Sep 21, 2016 at 1:41 PM, <Hu...@dlr.de>> wrote:
Hi,
Very cool feature.
I’ve seen some recent changes (MESOS-6096<https://issues.apache.org/jira/browse/MESOS-6096>) in the mesos-execute cli supporting grouped tasks which spawns the task-group on an arbitrary or specified agent node. Tested this yesterday and it works great. But if the task group holds a lot of one-off tasks (let’s say 50 tasks each 4 cores,…) how could one achieve that tasks are evenly distributed over several agents, depending on free resources? Don’t know if this is referred to pods, but do you think mesos-execute will support this?
Currently I’m using chronos & cook for batch scheduling, but I’d like to achieve that every batch/group of tasks runs in its own temporary framework like it does with mesos-execute
Thanks in advance & kind regards,
Hubert
——————————————————————————
Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR)
German Aerospace Center
German Remote Sensing Data Center | International Ground Segment | Oberpfaffenhofen | 82234 Wessling | Germany
Hubert Asamer
phone +49 (0) 8153 28-2894<tel:%2B49%20%280%29%208153%2028-2894> | fax +49 (0) 8153 28-1443<tel:%2B49%20%280%29%208153%2028-1443> | hubert.asamer@dlr.de<ma...@dlr.de>
www.DLR.de/eoc<http://www.dlr.de/eoc>
From: Vinod Kone [mailto:vinodkone@apache.org<ma...@apache.org>]
Sent: Dienstag, 9. August 2016 02:54
To: dev; user
Subject: Support for tasks groups aka pods in Mesos
Hi folks,
One of the most requested features in Mesos has been first class support for managing pod like containers. We finally have some time to focus and shepherd this work.
The epic tracking this work is : https://issues.apache.org/jira/browse/MESOS-2449
Design doc: https://issues.apache.org/jira/browse/MESOS-2449
Your feedback on the design will be most welcome. Once we get agreement on the design, we can start breaking down the epic into tickets.
Thanks,
Vinod & Jie
Re: Support for tasks groups aka pods in Mesos
Posted by Guangya Liu <gy...@gmail.com>.
The answer is No, the taskGroup can be treated as Pod in Kubernetes, it
will be a set of containers co-located and co-managed on an agent that
share some resources (e.g., network namespace, volumes). For your case, you
may want to split your 50 tasks to small task groups.
Also the `execute` cli is just an example framework and it can only launch
one task group for now, you may want to enhance it if you want to launch
multiple task groups.
On Wed, Sep 21, 2016 at 1:41 PM, <Hu...@dlr.de> wrote:
> Hi,
>
> Very cool feature.
>
> I’ve seen some recent changes (MESOS-6096
> <https://issues.apache.org/jira/browse/MESOS-6096>) in the mesos-execute
> cli supporting grouped tasks which spawns the task-group on an arbitrary or
> specified agent node. Tested this yesterday and it works great. But if the
> task group holds a lot of one-off tasks (let’s say 50 tasks each 4 cores,…)
> how could one achieve that tasks are evenly distributed over several
> agents, depending on free resources? Don’t know if this is referred to
> pods, but do you think mesos-execute will support this?
>
>
>
> Currently I’m using chronos & cook for batch scheduling, but I’d like to
> achieve that every batch/group of tasks runs in its own temporary
> framework like it does with mesos-execute
>
>
>
> Thanks in advance & kind regards,
>
> Hubert
>
>
>
> ——————————————————————————
>
> *Deutsches Zentrum für Luft- und Raumfahrt* e.V. (DLR)
>
> German Aerospace Center
>
>
>
>
>
> German Remote Sensing Data Center | International Ground Segment |
> Oberpfaffenhofen | 82234 Wessling | Germany
>
>
>
> *Hubert Asamer*
>
> phone +49 (0) 8153 28-2894 | fax +49 (0) 8153 28-1443 |
> hubert.asamer@dlr.de
>
> www.DLR.de/eoc <http://www.dlr.de/eoc>
>
>
>
>
>
>
>
>
>
> *From:* Vinod Kone [mailto:vinodkone@apache.org]
> *Sent:* Dienstag, 9. August 2016 02:54
> *To:* dev; user
> *Subject:* Support for tasks groups aka pods in Mesos
>
>
>
> Hi folks,
>
>
>
> One of the most requested features in Mesos has been first class support
> for managing pod like containers. We finally have some time to focus and
> shepherd this work.
>
>
>
> The epic tracking this work is : https://issues.apache.org/
> jira/browse/MESOS-2449
>
>
>
> Design doc: https://issues.apache.org/jira/browse/MESOS-2449
>
>
>
> Your feedback on the design will be most welcome. Once we get agreement on
> the design, we can start breaking down the epic into tickets.
>
>
>
> Thanks,
>
> Vinod & Jie
>
Re: Support for tasks groups aka pods in Mesos
Posted by Guangya Liu <gy...@gmail.com>.
The answer is No, the taskGroup can be treated as Pod in Kubernetes, it
will be a set of containers co-located and co-managed on an agent that
share some resources (e.g., network namespace, volumes). For your case, you
may want to split your 50 tasks to small task groups.
Also the `execute` cli is just an example framework and it can only launch
one task group for now, you may want to enhance it if you want to launch
multiple task groups.
On Wed, Sep 21, 2016 at 1:41 PM, <Hu...@dlr.de> wrote:
> Hi,
>
> Very cool feature.
>
> I’ve seen some recent changes (MESOS-6096
> <https://issues.apache.org/jira/browse/MESOS-6096>) in the mesos-execute
> cli supporting grouped tasks which spawns the task-group on an arbitrary or
> specified agent node. Tested this yesterday and it works great. But if the
> task group holds a lot of one-off tasks (let’s say 50 tasks each 4 cores,…)
> how could one achieve that tasks are evenly distributed over several
> agents, depending on free resources? Don’t know if this is referred to
> pods, but do you think mesos-execute will support this?
>
>
>
> Currently I’m using chronos & cook for batch scheduling, but I’d like to
> achieve that every batch/group of tasks runs in its own temporary
> framework like it does with mesos-execute
>
>
>
> Thanks in advance & kind regards,
>
> Hubert
>
>
>
> ——————————————————————————
>
> *Deutsches Zentrum für Luft- und Raumfahrt* e.V. (DLR)
>
> German Aerospace Center
>
>
>
>
>
> German Remote Sensing Data Center | International Ground Segment |
> Oberpfaffenhofen | 82234 Wessling | Germany
>
>
>
> *Hubert Asamer*
>
> phone +49 (0) 8153 28-2894 | fax +49 (0) 8153 28-1443 |
> hubert.asamer@dlr.de
>
> www.DLR.de/eoc <http://www.dlr.de/eoc>
>
>
>
>
>
>
>
>
>
> *From:* Vinod Kone [mailto:vinodkone@apache.org]
> *Sent:* Dienstag, 9. August 2016 02:54
> *To:* dev; user
> *Subject:* Support for tasks groups aka pods in Mesos
>
>
>
> Hi folks,
>
>
>
> One of the most requested features in Mesos has been first class support
> for managing pod like containers. We finally have some time to focus and
> shepherd this work.
>
>
>
> The epic tracking this work is : https://issues.apache.org/
> jira/browse/MESOS-2449
>
>
>
> Design doc: https://issues.apache.org/jira/browse/MESOS-2449
>
>
>
> Your feedback on the design will be most welcome. Once we get agreement on
> the design, we can start breaking down the epic into tickets.
>
>
>
> Thanks,
>
> Vinod & Jie
>