Re: Hadoop 3.2 Release Plan proposal

[Haibo Chen <ha...@cloudera.com.INVALID>]

Hi Sunil,

For YARN-1011, we found a few minor issues that we have not merged our
internal fixes into the upstream feature branch. Hence, given the release
timeline, YARN-1011 will not make it.

Thanks.

On Thu, Aug 30, 2018 at 2:52 PM, Virajith Jalaparti <vi...@gmail.com>
wrote:

> Hi Sunil,
>
> Quick correction on the task list  (missed this earlier) -- HDFS-12615 is
> being done by Inigo Goiri
>
> -Virajith
>
>
>
> On Thu, Aug 30, 2018 at 9:30 AM Sunil G <su...@apache.org> wrote:
>
> > Hi All,
> >
> > Inline with earlier communication dated 17th July 2018, I would like to
> > provide some updates.
> >
> > We are approaching previously proposed code freeze date (Aug 31).
> >
> > One of the critical feature Node Attributes feature merge discussion/vote
> > is ongoing. Also few other Blocker bugs need a bit more time. With regard
> > to this, suggesting to push the feature/code freeze for 2 more weeks to
> > accommodate these jiras too.
> >
> > Proposing Updated changes in plan inline with this:
> > Feature freeze date : all features to merge by September 7, 2018.
> > Code freeze date : blockers/critical only, no improvements and
> >  blocker/critical bug-fixes September 14, 2018.
> > Release date: September 28, 2018
> >
> > If any features in branch which are targeted to 3.2.0, please reply to
> this
> > email thread.
> >
> > *Here's an updated 3.2.0 feature status:*
> >
> > 1. Merged & Completed features:
> >
> > - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> > Initial cut.
> > - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> > - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> > - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> > and CLI.
> >
> > 2. Features close to finish:
> >
> > - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
> > Ongoing.
> > - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> ATSv2.
> > Patch in progress.
> > - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> > - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
> In
> > progress.
> >
> > 3. Tentative features:
> >
> > - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
> be
> > done before Aug 2018.
> > - (Eric) YARN-7129: Application Catalog for YARN applications.
> Challenging
> > as more discussions are on-going.
> >
> > *Summary of 3.2.0 issues status:*
> >
> > 26 Blocker and Critical issues [1] are open, I am following up with
> owners
> > to get status on each of them to get in by Code Freeze date.
> >
> > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> > Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
> ORDER
> > BY priority DESC
> >
> > Thanks,
> > Sunil
> >
> > On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
> >
> > > Hi All,
> > >
> > > Thanks for the feedbacks. Inline with earlier communication dated 17th
> > > July 2018, I would like to provide some updates.
> > >
> > > We are approaching previously proposed feature freeze date (Aug 21,
> about
> > > 7 days from today).
> > > If any features in branch which are targeted to 3.2.0, please reply to
> > > this email thread.
> > > Steve has mentioned about the s3 features which will come close to Code
> > > Freeze Date (Aug 31st).
> > >
> > > *Here's an updated 3.2.0 feature status:*
> > >
> > > 1. Merged & Completed features:
> > >
> > > - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> workloads
> > > Initial cut.
> > > - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> > >
> > > 2. Features close to finish:
> > >
> > > - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
> patches
> > > are all in, only one last
> > > patch is in review state.
> > > - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> Scheduler.
> > > Close to commit.
> > > - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> API
> > > and CLI. 2 patches are pending
> > > which will be closed by Feature freeze date.
> > > - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> > ATSv2.
> > > Patch in progress.
> > > - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
> works.
> > > - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> Storage.
> > > In progress.
> > >
> > > 3. Tentative features:
> > >
> > > - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
> > be
> > > done before Aug 2018.
> > > - (Eric) YARN-7129: Application Catalog for YARN applications.
> > Challenging
> > > as more discussions are on-going.
> > >
> > > *Summary of 3.2.0 issues status:*
> > >
> > > 39 Blocker and Critical issues [1] are open, I am checking with owners
> to
> > > get status on each of them to get in by Code Freeze date.
> > >
> > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> (Blocker,
> > > Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
> > ORDER
> > > BY priority DESC
> > >
> > > Thanks,
> > > Sunil
> > >
> > > On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
> > >
> > >> Thanks Subru for the thoughts.
> > >> One of the main reason for a major release is to push out critical
> > >> features with a faster cadence to the users. If we are pulling more
> and
> > >> more different types of features to a minor release, that branch will
> > >> become more destabilized and it may be tough to say that 3.1.2 is
> stable
> > >> that 3.1.1 for eg. We always tend to improve and stabilize features in
> > >> subsequent minor release.
> > >> For few companies, it makes sense to push out these new features
> faster
> > >> to make a reach to the users. Adding to the point to the backporting
> > >> issues, I agree that its a pain and we can workaround that with some
> git
> > >> scripts. If we can make such scripts available to committers, backport
> > will
> > >> be seem-less across branches and we can achieve the faster release
> > cadence
> > >> also.
> > >>
> > >> Thoughts?
> > >>
> > >> - Sunil
> > >>
> > >>
> > >> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
> > wrote:
> > >>
> > >>> Thanks Sunil for volunteering to lead the release effort. I am
> > generally
> > >>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
> > >>> already
> > >>> have too many branches to be maintained. I already see many commits
> are
> > >>> in
> > >>> different branches with no apparent rationale, for e.g: 3.1 has
> commits
> > >>> which are absent in 3.0 etc.
> > >>>
> > >>> Additionally AFAIK 3.x has not been deployed in any major production
> > >>> setting so the cost of adding features should be minimal.
> > >>>
> > >>> Thoughts?
> > >>>
> > >>> -Subru
> > >>>
> > >>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
> > >>>
> > >>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> > >>> >
> > >>> > Yes, important changes and features are much needed, hence we will
> be
> > >>> > keeping the door open for them as possible. Also considering few
> more
> > >>> > offline requests from other folks, I think extending the timeframe
> by
> > >>> > couple of weeks makes sense (including a second RC buffer) and this
> > >>> should
> > >>> > ideally help us to ship this by September itself.
> > >>> >
> > >>> > Revised dates (I will be updating same in Roadmap wiki as well)
> > >>> >
> > >>> > - Feature freeze date : all features to merge by August 21, 2018.
> > >>> >
> > >>> > - Code freeze date : blockers/critical only, no improvements and
> non
> > >>> > blocker/critical
> > >>> >
> > >>> > bug-fixes  August 31, 2018.
> > >>> >
> > >>> > - Release date: September 15, 2018
> > >>> >
> > >>> > Thank Eric and Zian, I think Wangda has already answered your
> > >>> questions.
> > >>> >
> > >>> > Thanks
> > >>> > Sunil
> > >>> >
> > >>> >
> > >>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
> > >>> wrote:
> > >>> >
> > >>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for
> that.
> > >>> > >
> > >>> > > To concerns from Steve,
> > >>> > >
> > >>> > > It is a good idea to keep the door open to get important changes
> /
> > >>> > > features in before cutoff. I would prefer to keep the proposed
> > >>> release
> > >>> > date
> > >>> > > to make sure things can happen earlier instead of last minute and
> > we
> > >>> all
> > >>> > > know that releases are always get delayed :). I'm also fine if we
> > >>> want
> > >>> > get
> > >>> > > another several weeks time.
> > >>> > >
> > >>> > > Regarding of 3.3 release, I would suggest doing that before
> > >>> thanksgiving.
> > >>> > > Do you think is it good or too early / late?
> > >>> > >
> > >>> > > Eric,
> > >>> > >
> > >>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
> > >>> merged
> > >>> > > in time, we probably not need the YARN-8220.
> > >>> > >
> > >>> > > Sunil,
> > >>> > >
> > >>> > > Could u update https://cwiki.apache.org/
> confluence/display/HADOOP/
> > >>> > Roadmap
> > >>> > > with proposed plan as well? We can fill feature list first before
> > >>> getting
> > >>> > > consensus of time.
> > >>> > >
> > >>> > > Thanks,
> > >>> > > Wangda
> > >>> > >
> > >>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> > >>> <fabbri@cloudera.com.invalid
> > >>> > >
> > >>> > > wrote:
> > >>> > >
> > >>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> > >>> stevel@hortonworks.com>
> > >>> > >> wrote:
> > >>> > >>
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
> > >>> > >> > sunilg@apache.org>> wrote:
> > >>> > >> >
> > >>> > >> > I would also would like to take this opportunity to come up
> > with a
> > >>> > >> detailed
> > >>> > >> > plan.
> > >>> > >> >
> > >>> > >> > - Feature freeze date : all features should be merged by
> August
> > >>> 10,
> > >>> > >> 2018.
> > >>> > >> >
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > <snip>
> > >>> > >>
> > >>> > >> >
> > >>> > >> > Please let me know if I missed any features targeted to 3.2
> per
> > >>> this
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > Well there these big todo lists for S3 & S3Guard.
> > >>> > >> >
> > >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> > >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen
> 2
> > >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> > >>> > >> >
> > >>> > >> > I don't think this is quite ready yet, I've been doing work on
> > >>> it, but
> > >>> > >> if
> > >>> > >> > we have a 3 week deadline, I'm going to expect some timely
> > >>> reviews on
> > >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> > >>> > >> >
> > >>> > >> > I've uprated that to a blocker feature; will review the S3 &
> > >>> S3Guard
> > >>> > >> JIRAs
> > >>> > >> > to see which of those are blocking. Then there are some
> pressing
> > >>> > "guave,
> > >>> > >> > java 9 prep"
> > >>> > >> >
> > >>> > >> >
> > >>> > >>  I can help with this part if you like.
> > >>> > >>
> > >>> > >>
> > >>> > >>
> > >>> > >> >
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > timeline. I would like to volunteer myself as release manager
> of
> > >>> 3.2.0
> > >>> > >> > release.
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > well volunteered!
> > >>> > >> >
> > >>> > >> >
> > >>> > >> >
> > >>> > >> Yes, thank you for stepping up.
> > >>> > >>
> > >>> > >>
> > >>> > >> >
> > >>> > >> > I think this raises a good q: what timetable should we have
> for
> > >>> the
> > >>> > >> 3.2. &
> > >>> > >> > 3.3 releases; if we do want a faster cadence, then having the
> > >>> outline
> > >>> > >> time
> > >>> > >> > from the 3.2 to the 3.3 release means that there's less
> concern
> > >>> about
> > >>> > >> > things not making the 3.2 dealine
> > >>> > >> >
> > >>> > >> > -Steve
> > >>> > >> >
> > >>> > >> >
> > >>> > >> Good idea to mitigate the short deadline.
> > >>> > >>
> > >>> > >> -AF
> > >>> > >>
> > >>> > >
> > >>> >
> > >>>
> > >>
> >
>