Re: [VOTE] Merge YARN-3926 (resource profile) to trunk

[Wangda Tan <wh...@gmail.com>]

Hi all,

Given we have 3 binding +1s, the vote passes. I just push changes to trunk.
Will update JIRAs accordingly.

Thanks everybody for helping this feature and voting!

Best,
Wangda


On Sat, Aug 26, 2017 at 8:58 AM, Sunil G <su...@apache.org> wrote:

> Hi Daniel
>
> Thank you very much for the support.
>
> * When you say that the feature can be turned
> off, do you mean resource types or resource profiles?  I know there's an
> off-by-default property that governs resource profiles, but I didn't see
> any way to turn off resource types.
> Yes,*yarn.resourcemanager.resource-profiles.enabled* is false by default
> and controls off/on of this feature. Now regarding new resource types, its
> been loaded from "*resource-types.xml"* and by default this XML file is not
> available in the package. Thus prevents any issues in default case. Once
> this file is added to a cluster then new resources will be loaded from
> same.
>
> * Even if only CPU and memory are configured, i.e. no additional resource
> types, the code path is different than it was.
> Earlier primitive data types were used to represent vcores and memory. As
> per resource profile work, all resources under YARN is categorized as
> ResourceInformation and placed under existing Resource object. So memory
> and vcores will be accessible and operable with same set of public apis
> from Resources or ResourceCalculator (DRC) same as earlier even when
> feature is off (Code path is same, but improved to support a unified
> ResourceInformation class instead of memory/vcores primitive types).
>
> Thanks
> Sunil
>
>
>
>
> On Sat, Aug 26, 2017 at 8:10 PM Daniel Templeton <da...@cloudera.com>
> wrote:
>
> > Quick question, Wangda.  When you say that the feature can be turned
> > off, do you mean resource types or resource profiles?  I know there's an
> > off-by-default property that governs resource profiles, but I didn't see
> > any way to turn off resource types.  Even if only CPU and memory are
> > configured, i.e. no additional resource types, the code path is
> > different than it was.  Specifically, where CPU and memory were
> > primitives before, they're now entries in an array whose indexes have to
> > be looked up through the ResourceUtils class.  Did I miss something?
> >
> > For those who haven't followed the feature closely, there are really two
> > features here.  Resource types allows for declarative extension of the
> > resource system in YARN.  Resource profiles builds on top of resource
> > types to allow a user to request a group of resources as a profile, much
> > like EC2 instance types, e.g. "fast-compute" might mean 32GB RAM, 8
> > vcores, and 2 GPUs.
> >
> > Daniel
> >
> > On 8/23/17 11:49 AM, Wangda Tan wrote:
> > >   Hi folks,
> > >
> > > Per earlier discussion [1], I'd like to start a formal vote to merge
> > > feature branch YARN-3926 (Resource profile) to trunk. The vote will run
> > for
> > > 7 days and will end August 30 10:00 AM PDT.
> > >
> > > Briefly, YARN-3926 can extend resource model of YARN to support
> resource
> > > types other than CPU and memory, so it will be a cornerstone of
> features
> > > like GPU support (YARN-6223), disk scheduling/isolation (YARN-2139),
> FPGA
> > > support (YARN-5983), network IO scheduling/isolation (YARN-2140). In
> > > addition to that, YARN-3926 allows admin to preconfigure resource
> > profiles
> > > in the cluster, for example, m3.large means <2 vcores, 8 GB memory, 64
> GB
> > > disk>, so applications can request "m3.large" profile instead of
> > specifying
> > > all resource types’s values.
> > >
> > > There are 32 subtasks that were completed as part of this effort.
> > >
> > > This feature needs to be explicitly turned on before use. We paid close
> > > attention to compatibility, performance, and scalability of this
> feature,
> > > mentioned in [1], we didn't see observable performance regression in
> > large
> > > scale SLS (scheduler load simulator) executions and saw less than 5%
> > > performance regression by using micro benchmark added by YARN-6775.
> > >
> > > This feature works from end-to-end (including
> UI/CLI/application/server),
> > > we have setup a cluster with this feature turned on runs for several
> > weeks,
> > > we didn't see any issues by far.
> > >
> > > Merge JIRA: YARN-7013 (Jenkins gave +1 already).
> > > Documentation: YARN-7056
> > >
> > > Special thanks to a team of folks who worked hard and contributed
> towards
> > > this effort including design discussion/development/reviews, etc.:
> Varun
> > > Vasudev, Sunil Govind, Daniel Templeton, Vinod Vavilapalli, Yufei Gu,
> > > Karthik Kambatla, Jason Lowe, Arun Suresh.
> > >
> > > Regards,
> > > Wangda Tan
> > >
> > > [1]
> > >
> > http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/
> 201708.mbox/%3CCAD%2B%2BeCnjEHU%3D-M33QdjnND0ZL73eKwxRua4%
> 3DBbp4G8inQZmaMg%40mail.gmail.com%3E
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
>