***UNCHECKED*** Re: [DISCUSS] Merge YARN-6592 to trunk

[Arun Suresh <ar...@gmail.com>]

Thanks everyone. I've opened a VOTE thread..

@Carlo
Interesting idea to support Kubernates API. Definitely makes sense. My
understanding is that we can emulate specification for a k8 POD within our
scheme of things by assigning the same source tag to a set of scheduling
requests.
Ill dig in to this.

Cheers
-Arun

On Fri, Jan 26, 2018 at 1:34 AM, Carlo Aldo Curino <ca...@gmail.com>
wrote:

> Also I think this can help us close the gap (and surpass) Kubernetes for
> complex services (at least for resource management)... It would be awesome
> to have a compatibility layer so folks can run Kubernetes natives apps on a
> yarn cluster.
>
>
>
>
> On Jan 26, 2018 1:32 AM, "Carlo Aldo Curino" <ca...@gmail.com>
> wrote:
>
> +1. I didn't runs tests, but I like the design, and speaking with ops
> teams that operate large clusters I hear this is a feature they think is
> going to help a lot, so I am very supportive of this effort.
>
> On Jan 25, 2018 7:08 PM, "Konstantinos Karanasos" <kk...@gmail.com>
> wrote:
>
>> Thanks for starting the thread Arun, +1 from me too.
>>
>> Konstantinos
>>
>> On Thu, Jan 25, 2018 at 18:54 Weiwei Yang <ch...@hotmail.com> wrote:
>>
>>> +1, thanks for getting to this milestone Arun.
>>> I’ve done some basic validations on a 4 nodes cluster, with some general
>>> affinity/anti-affinty/cardinality constraints, it worked. I’ve also
>>> reviewed the doc, it’s in good shape and very illustrative.
>>>
>>> Thanks.
>>>
>>> --
>>> Weiwei
>>>
>>> On 26 Jan 2018, 10:44 AM +0800, Sunil G <su...@apache.org>, wrote:
>>> +1.
>>>
>>> Thanks Arun.
>>>
>>> I did manual testing for check affinity and anti-affinity features with
>>> placement allocator. Also checked SLS to see any performance regression,
>>> and there are not much difference as Arun mentioned.
>>>
>>> Thanks all the folks for working on this. Kudos!
>>>
>>> - Sunil
>>>
>>>
>>> On Fri, Jan 26, 2018 at 5:16 AM Arun Suresh <asuresh@apache.org<mailto:
>>> asuresh@apache.org>> wrote:
>>> Hello yarn-dev@
>>>
>>> We feel that the YARN-6592 dev branch mostly in shape to be merged into
>>> trunk. This branch adds support for placing containers in YARN using rich
>>> placement constraints. For example, this can be used by applications to
>>> co-locate containers on a node or rack (affinity constraint), spread
>>> containers across nodes or racks (anti-affinity constraint), or even
>>> specify the maximum number of containers on a node/rack (cardinality
>>> constraint).
>>>
>>> We have integrated this feature into the Distributed-Shell application
>>> for feature testing. We have performed end-to-end testing on
>>> moderately-sized clusters to verify that constraints work fine. Performance
>>> tests have been done via both SLS tests and Capacity Scheduler performance
>>> unit tests, and no regression was found. We have opened a JIRA to track
>>> Jenkins acceptance of the aggregated patch [2]. Documentation is in the
>>> process of being completed [3]. You can also check our design document for
>>> more details [4].
>>>
>>> Config flags are needed to enable this feature and it should not have
>>> any effect on YARN when turned off. Once merged, we plan to work on further
>>> improvements, which can be tracked in the umbrella YARN-7812 [5].
>>>
>>> Kindly do take a look at the branch and raise issues/concerns that need
>>> to be addressed before the merge.
>>>
>>> Many thanks to Konstantinos, Wangda, Panagiotis, Weiwei, and Sunil for
>>> their contributions to this effort, as well as Subru, Chris, Carlo, and
>>> Vinod for their inputs and discussions.
>>>
>>> Cheers
>>> -Arun
>>>
>>>
>>> [1] https://issues.apache.org/jira/browse/YARN-6592
>>> [2] https://issues.apache.org/jira/browse/YARN-7792
>>> [3] https://issues.apache.org/jira/browse/YARN-7780
>>> [4] https://issues.apache.org/jira/secure/attachment/12867869/YA
>>> RN-6592-Rich-Placement-Constraints-Design-V1.pdf
>>> [5] https://issues.apache.org/jira/browse/YARN-7812
>>>
>>> --
>> Konstantinos
>>
>
>