You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Sunil G <su...@apache.org> on 2018/08/14 17:00:55 UTC
Re: Hadoop 3.2 Release Plan proposal
Hi All,
Thanks for the feedbacks. Inline with earlier communication dated 17th July
2018, I would like to provide some updates.
We are approaching previously proposed feature freeze date (Aug 21, about 7
days from today).
If any features in branch which are targeted to 3.2.0, please reply to this
email thread.
Steve has mentioned about the s3 features which will come close to Code
Freeze Date (Aug 31st).
*Here's an updated 3.2.0 feature status:*
1. Merged & Completed features:
- (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
Initial cut.
- (Uma) HDFS-10285: HDFS Storage Policy Satisfier
2. Features close to finish:
- (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major patches
are all in, only one last
patch is in review state.
- (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
Close to commit.
- (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
and CLI. 2 patches are pending
which will be closed by Feature freeze date.
- (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
Patch in progress.
- (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
- (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage. In
progress.
3. Tentative features:
- (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
done before Aug 2018.
- (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
as more discussions are on-going.
*Summary of 3.2.0 issues status:*
39 Blocker and Critical issues [1] are open, I am checking with owners to
get status on each of them to get in by Code Freeze date.
[1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
BY priority DESC
Thanks,
Sunil
On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
> Thanks Subru for the thoughts.
> One of the main reason for a major release is to push out critical
> features with a faster cadence to the users. If we are pulling more and
> more different types of features to a minor release, that branch will
> become more destabilized and it may be tough to say that 3.1.2 is stable
> that 3.1.1 for eg. We always tend to improve and stabilize features in
> subsequent minor release.
> For few companies, it makes sense to push out these new features faster to
> make a reach to the users. Adding to the point to the backporting issues, I
> agree that its a pain and we can workaround that with some git scripts. If
> we can make such scripts available to committers, backport will be
> seem-less across branches and we can achieve the faster release cadence
> also.
>
> Thoughts?
>
> - Sunil
>
>
> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org> wrote:
>
>> Thanks Sunil for volunteering to lead the release effort. I am generally
>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
>> already
>> have too many branches to be maintained. I already see many commits are in
>> different branches with no apparent rationale, for e.g: 3.1 has commits
>> which are absent in 3.0 etc.
>>
>> Additionally AFAIK 3.x has not been deployed in any major production
>> setting so the cost of adding features should be minimal.
>>
>> Thoughts?
>>
>> -Subru
>>
>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
>>
>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>> >
>> > Yes, important changes and features are much needed, hence we will be
>> > keeping the door open for them as possible. Also considering few more
>> > offline requests from other folks, I think extending the timeframe by
>> > couple of weeks makes sense (including a second RC buffer) and this
>> should
>> > ideally help us to ship this by September itself.
>> >
>> > Revised dates (I will be updating same in Roadmap wiki as well)
>> >
>> > - Feature freeze date : all features to merge by August 21, 2018.
>> >
>> > - Code freeze date : blockers/critical only, no improvements and non
>> > blocker/critical
>> >
>> > bug-fixes August 31, 2018.
>> >
>> > - Release date: September 15, 2018
>> >
>> > Thank Eric and Zian, I think Wangda has already answered your questions.
>> >
>> > Thanks
>> > Sunil
>> >
>> >
>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
>> wrote:
>> >
>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for that.
>> > >
>> > > To concerns from Steve,
>> > >
>> > > It is a good idea to keep the door open to get important changes /
>> > > features in before cutoff. I would prefer to keep the proposed release
>> > date
>> > > to make sure things can happen earlier instead of last minute and we
>> all
>> > > know that releases are always get delayed :). I'm also fine if we want
>> > get
>> > > another several weeks time.
>> > >
>> > > Regarding of 3.3 release, I would suggest doing that before
>> thanksgiving.
>> > > Do you think is it good or too early / late?
>> > >
>> > > Eric,
>> > >
>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
>> merged
>> > > in time, we probably not need the YARN-8220.
>> > >
>> > > Sunil,
>> > >
>> > > Could u update https://cwiki.apache.org/confluence/display/HADOOP/
>> > Roadmap
>> > > with proposed plan as well? We can fill feature list first before
>> getting
>> > > consensus of time.
>> > >
>> > > Thanks,
>> > > Wangda
>> > >
>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>> <fabbri@cloudera.com.invalid
>> > >
>> > > wrote:
>> > >
>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>> stevel@hortonworks.com>
>> > >> wrote:
>> > >>
>> > >> >
>> > >> >
>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
>> > >> > sunilg@apache.org>> wrote:
>> > >> >
>> > >> > I would also would like to take this opportunity to come up with a
>> > >> detailed
>> > >> > plan.
>> > >> >
>> > >> > - Feature freeze date : all features should be merged by August 10,
>> > >> 2018.
>> > >> >
>> > >> >
>> > >> >
>> > >> > <snip>
>> > >>
>> > >> >
>> > >> > Please let me know if I missed any features targeted to 3.2 per
>> this
>> > >> >
>> > >> >
>> > >> > Well there these big todo lists for S3 & S3Guard.
>> > >> >
>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>> > >> >
>> > >> >
>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen 2
>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>> > >> >
>> > >> > I don't think this is quite ready yet, I've been doing work on it,
>> but
>> > >> if
>> > >> > we have a 3 week deadline, I'm going to expect some timely reviews
>> on
>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>> > >> >
>> > >> > I've uprated that to a blocker feature; will review the S3 &
>> S3Guard
>> > >> JIRAs
>> > >> > to see which of those are blocking. Then there are some pressing
>> > "guave,
>> > >> > java 9 prep"
>> > >> >
>> > >> >
>> > >> I can help with this part if you like.
>> > >>
>> > >>
>> > >>
>> > >> >
>> > >> >
>> > >> >
>> > >> > timeline. I would like to volunteer myself as release manager of
>> 3.2.0
>> > >> > release.
>> > >> >
>> > >> >
>> > >> > well volunteered!
>> > >> >
>> > >> >
>> > >> >
>> > >> Yes, thank you for stepping up.
>> > >>
>> > >>
>> > >> >
>> > >> > I think this raises a good q: what timetable should we have for the
>> > >> 3.2. &
>> > >> > 3.3 releases; if we do want a faster cadence, then having the
>> outline
>> > >> time
>> > >> > from the 3.2 to the 3.3 release means that there's less concern
>> about
>> > >> > things not making the 3.2 dealine
>> > >> >
>> > >> > -Steve
>> > >> >
>> > >> >
>> > >> Good idea to mitigate the short deadline.
>> > >>
>> > >> -AF
>> > >>
>> > >
>> >
>>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Haibo Chen <ha...@cloudera.com.INVALID>.
Hi Sunil,
For YARN-1011, we found a few minor issues that we have not merged our
internal fixes into the upstream feature branch. Hence, given the release
timeline, YARN-1011 will not make it.
Thanks.
On Thu, Aug 30, 2018 at 2:52 PM, Virajith Jalaparti <vi...@gmail.com>
wrote:
> Hi Sunil,
>
> Quick correction on the task list (missed this earlier) -- HDFS-12615 is
> being done by Inigo Goiri
>
> -Virajith
>
>
>
> On Thu, Aug 30, 2018 at 9:30 AM Sunil G <su...@apache.org> wrote:
>
> > Hi All,
> >
> > Inline with earlier communication dated 17th July 2018, I would like to
> > provide some updates.
> >
> > We are approaching previously proposed code freeze date (Aug 31).
> >
> > One of the critical feature Node Attributes feature merge discussion/vote
> > is ongoing. Also few other Blocker bugs need a bit more time. With regard
> > to this, suggesting to push the feature/code freeze for 2 more weeks to
> > accommodate these jiras too.
> >
> > Proposing Updated changes in plan inline with this:
> > Feature freeze date : all features to merge by September 7, 2018.
> > Code freeze date : blockers/critical only, no improvements and
> > blocker/critical bug-fixes September 14, 2018.
> > Release date: September 28, 2018
> >
> > If any features in branch which are targeted to 3.2.0, please reply to
> this
> > email thread.
> >
> > *Here's an updated 3.2.0 feature status:*
> >
> > 1. Merged & Completed features:
> >
> > - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> > Initial cut.
> > - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> > - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> > - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> > and CLI.
> >
> > 2. Features close to finish:
> >
> > - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
> > Ongoing.
> > - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> ATSv2.
> > Patch in progress.
> > - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> > - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
> In
> > progress.
> >
> > 3. Tentative features:
> >
> > - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
> be
> > done before Aug 2018.
> > - (Eric) YARN-7129: Application Catalog for YARN applications.
> Challenging
> > as more discussions are on-going.
> >
> > *Summary of 3.2.0 issues status:*
> >
> > 26 Blocker and Critical issues [1] are open, I am following up with
> owners
> > to get status on each of them to get in by Code Freeze date.
> >
> > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> > Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
> ORDER
> > BY priority DESC
> >
> > Thanks,
> > Sunil
> >
> > On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
> >
> > > Hi All,
> > >
> > > Thanks for the feedbacks. Inline with earlier communication dated 17th
> > > July 2018, I would like to provide some updates.
> > >
> > > We are approaching previously proposed feature freeze date (Aug 21,
> about
> > > 7 days from today).
> > > If any features in branch which are targeted to 3.2.0, please reply to
> > > this email thread.
> > > Steve has mentioned about the s3 features which will come close to Code
> > > Freeze Date (Aug 31st).
> > >
> > > *Here's an updated 3.2.0 feature status:*
> > >
> > > 1. Merged & Completed features:
> > >
> > > - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> workloads
> > > Initial cut.
> > > - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> > >
> > > 2. Features close to finish:
> > >
> > > - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
> patches
> > > are all in, only one last
> > > patch is in review state.
> > > - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> Scheduler.
> > > Close to commit.
> > > - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> API
> > > and CLI. 2 patches are pending
> > > which will be closed by Feature freeze date.
> > > - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> > ATSv2.
> > > Patch in progress.
> > > - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
> works.
> > > - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> Storage.
> > > In progress.
> > >
> > > 3. Tentative features:
> > >
> > > - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
> > be
> > > done before Aug 2018.
> > > - (Eric) YARN-7129: Application Catalog for YARN applications.
> > Challenging
> > > as more discussions are on-going.
> > >
> > > *Summary of 3.2.0 issues status:*
> > >
> > > 39 Blocker and Critical issues [1] are open, I am checking with owners
> to
> > > get status on each of them to get in by Code Freeze date.
> > >
> > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> (Blocker,
> > > Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
> > ORDER
> > > BY priority DESC
> > >
> > > Thanks,
> > > Sunil
> > >
> > > On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
> > >
> > >> Thanks Subru for the thoughts.
> > >> One of the main reason for a major release is to push out critical
> > >> features with a faster cadence to the users. If we are pulling more
> and
> > >> more different types of features to a minor release, that branch will
> > >> become more destabilized and it may be tough to say that 3.1.2 is
> stable
> > >> that 3.1.1 for eg. We always tend to improve and stabilize features in
> > >> subsequent minor release.
> > >> For few companies, it makes sense to push out these new features
> faster
> > >> to make a reach to the users. Adding to the point to the backporting
> > >> issues, I agree that its a pain and we can workaround that with some
> git
> > >> scripts. If we can make such scripts available to committers, backport
> > will
> > >> be seem-less across branches and we can achieve the faster release
> > cadence
> > >> also.
> > >>
> > >> Thoughts?
> > >>
> > >> - Sunil
> > >>
> > >>
> > >> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
> > wrote:
> > >>
> > >>> Thanks Sunil for volunteering to lead the release effort. I am
> > generally
> > >>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
> > >>> already
> > >>> have too many branches to be maintained. I already see many commits
> are
> > >>> in
> > >>> different branches with no apparent rationale, for e.g: 3.1 has
> commits
> > >>> which are absent in 3.0 etc.
> > >>>
> > >>> Additionally AFAIK 3.x has not been deployed in any major production
> > >>> setting so the cost of adding features should be minimal.
> > >>>
> > >>> Thoughts?
> > >>>
> > >>> -Subru
> > >>>
> > >>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
> > >>>
> > >>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> > >>> >
> > >>> > Yes, important changes and features are much needed, hence we will
> be
> > >>> > keeping the door open for them as possible. Also considering few
> more
> > >>> > offline requests from other folks, I think extending the timeframe
> by
> > >>> > couple of weeks makes sense (including a second RC buffer) and this
> > >>> should
> > >>> > ideally help us to ship this by September itself.
> > >>> >
> > >>> > Revised dates (I will be updating same in Roadmap wiki as well)
> > >>> >
> > >>> > - Feature freeze date : all features to merge by August 21, 2018.
> > >>> >
> > >>> > - Code freeze date : blockers/critical only, no improvements and
> non
> > >>> > blocker/critical
> > >>> >
> > >>> > bug-fixes August 31, 2018.
> > >>> >
> > >>> > - Release date: September 15, 2018
> > >>> >
> > >>> > Thank Eric and Zian, I think Wangda has already answered your
> > >>> questions.
> > >>> >
> > >>> > Thanks
> > >>> > Sunil
> > >>> >
> > >>> >
> > >>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
> > >>> wrote:
> > >>> >
> > >>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for
> that.
> > >>> > >
> > >>> > > To concerns from Steve,
> > >>> > >
> > >>> > > It is a good idea to keep the door open to get important changes
> /
> > >>> > > features in before cutoff. I would prefer to keep the proposed
> > >>> release
> > >>> > date
> > >>> > > to make sure things can happen earlier instead of last minute and
> > we
> > >>> all
> > >>> > > know that releases are always get delayed :). I'm also fine if we
> > >>> want
> > >>> > get
> > >>> > > another several weeks time.
> > >>> > >
> > >>> > > Regarding of 3.3 release, I would suggest doing that before
> > >>> thanksgiving.
> > >>> > > Do you think is it good or too early / late?
> > >>> > >
> > >>> > > Eric,
> > >>> > >
> > >>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
> > >>> merged
> > >>> > > in time, we probably not need the YARN-8220.
> > >>> > >
> > >>> > > Sunil,
> > >>> > >
> > >>> > > Could u update https://cwiki.apache.org/
> confluence/display/HADOOP/
> > >>> > Roadmap
> > >>> > > with proposed plan as well? We can fill feature list first before
> > >>> getting
> > >>> > > consensus of time.
> > >>> > >
> > >>> > > Thanks,
> > >>> > > Wangda
> > >>> > >
> > >>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> > >>> <fabbri@cloudera.com.invalid
> > >>> > >
> > >>> > > wrote:
> > >>> > >
> > >>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> > >>> stevel@hortonworks.com>
> > >>> > >> wrote:
> > >>> > >>
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
> > >>> > >> > sunilg@apache.org>> wrote:
> > >>> > >> >
> > >>> > >> > I would also would like to take this opportunity to come up
> > with a
> > >>> > >> detailed
> > >>> > >> > plan.
> > >>> > >> >
> > >>> > >> > - Feature freeze date : all features should be merged by
> August
> > >>> 10,
> > >>> > >> 2018.
> > >>> > >> >
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > <snip>
> > >>> > >>
> > >>> > >> >
> > >>> > >> > Please let me know if I missed any features targeted to 3.2
> per
> > >>> this
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > Well there these big todo lists for S3 & S3Guard.
> > >>> > >> >
> > >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> > >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen
> 2
> > >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> > >>> > >> >
> > >>> > >> > I don't think this is quite ready yet, I've been doing work on
> > >>> it, but
> > >>> > >> if
> > >>> > >> > we have a 3 week deadline, I'm going to expect some timely
> > >>> reviews on
> > >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> > >>> > >> >
> > >>> > >> > I've uprated that to a blocker feature; will review the S3 &
> > >>> S3Guard
> > >>> > >> JIRAs
> > >>> > >> > to see which of those are blocking. Then there are some
> pressing
> > >>> > "guave,
> > >>> > >> > java 9 prep"
> > >>> > >> >
> > >>> > >> >
> > >>> > >> I can help with this part if you like.
> > >>> > >>
> > >>> > >>
> > >>> > >>
> > >>> > >> >
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > timeline. I would like to volunteer myself as release manager
> of
> > >>> 3.2.0
> > >>> > >> > release.
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > well volunteered!
> > >>> > >> >
> > >>> > >> >
> > >>> > >> >
> > >>> > >> Yes, thank you for stepping up.
> > >>> > >>
> > >>> > >>
> > >>> > >> >
> > >>> > >> > I think this raises a good q: what timetable should we have
> for
> > >>> the
> > >>> > >> 3.2. &
> > >>> > >> > 3.3 releases; if we do want a faster cadence, then having the
> > >>> outline
> > >>> > >> time
> > >>> > >> > from the 3.2 to the 3.3 release means that there's less
> concern
> > >>> about
> > >>> > >> > things not making the 3.2 dealine
> > >>> > >> >
> > >>> > >> > -Steve
> > >>> > >> >
> > >>> > >> >
> > >>> > >> Good idea to mitigate the short deadline.
> > >>> > >>
> > >>> > >> -AF
> > >>> > >>
> > >>> > >
> > >>> >
> > >>>
> > >>
> >
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Haibo Chen <ha...@cloudera.com.INVALID>.
Hi Sunil,
For YARN-1011, we found a few minor issues that we have not merged our
internal fixes into the upstream feature branch. Hence, given the release
timeline, YARN-1011 will not make it.
Thanks.
On Thu, Aug 30, 2018 at 2:52 PM, Virajith Jalaparti <vi...@gmail.com>
wrote:
> Hi Sunil,
>
> Quick correction on the task list (missed this earlier) -- HDFS-12615 is
> being done by Inigo Goiri
>
> -Virajith
>
>
>
> On Thu, Aug 30, 2018 at 9:30 AM Sunil G <su...@apache.org> wrote:
>
> > Hi All,
> >
> > Inline with earlier communication dated 17th July 2018, I would like to
> > provide some updates.
> >
> > We are approaching previously proposed code freeze date (Aug 31).
> >
> > One of the critical feature Node Attributes feature merge discussion/vote
> > is ongoing. Also few other Blocker bugs need a bit more time. With regard
> > to this, suggesting to push the feature/code freeze for 2 more weeks to
> > accommodate these jiras too.
> >
> > Proposing Updated changes in plan inline with this:
> > Feature freeze date : all features to merge by September 7, 2018.
> > Code freeze date : blockers/critical only, no improvements and
> > blocker/critical bug-fixes September 14, 2018.
> > Release date: September 28, 2018
> >
> > If any features in branch which are targeted to 3.2.0, please reply to
> this
> > email thread.
> >
> > *Here's an updated 3.2.0 feature status:*
> >
> > 1. Merged & Completed features:
> >
> > - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> > Initial cut.
> > - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> > - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> > - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> > and CLI.
> >
> > 2. Features close to finish:
> >
> > - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
> > Ongoing.
> > - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> ATSv2.
> > Patch in progress.
> > - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> > - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
> In
> > progress.
> >
> > 3. Tentative features:
> >
> > - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
> be
> > done before Aug 2018.
> > - (Eric) YARN-7129: Application Catalog for YARN applications.
> Challenging
> > as more discussions are on-going.
> >
> > *Summary of 3.2.0 issues status:*
> >
> > 26 Blocker and Critical issues [1] are open, I am following up with
> owners
> > to get status on each of them to get in by Code Freeze date.
> >
> > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> > Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
> ORDER
> > BY priority DESC
> >
> > Thanks,
> > Sunil
> >
> > On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
> >
> > > Hi All,
> > >
> > > Thanks for the feedbacks. Inline with earlier communication dated 17th
> > > July 2018, I would like to provide some updates.
> > >
> > > We are approaching previously proposed feature freeze date (Aug 21,
> about
> > > 7 days from today).
> > > If any features in branch which are targeted to 3.2.0, please reply to
> > > this email thread.
> > > Steve has mentioned about the s3 features which will come close to Code
> > > Freeze Date (Aug 31st).
> > >
> > > *Here's an updated 3.2.0 feature status:*
> > >
> > > 1. Merged & Completed features:
> > >
> > > - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> workloads
> > > Initial cut.
> > > - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> > >
> > > 2. Features close to finish:
> > >
> > > - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
> patches
> > > are all in, only one last
> > > patch is in review state.
> > > - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> Scheduler.
> > > Close to commit.
> > > - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> API
> > > and CLI. 2 patches are pending
> > > which will be closed by Feature freeze date.
> > > - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> > ATSv2.
> > > Patch in progress.
> > > - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
> works.
> > > - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> Storage.
> > > In progress.
> > >
> > > 3. Tentative features:
> > >
> > > - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
> > be
> > > done before Aug 2018.
> > > - (Eric) YARN-7129: Application Catalog for YARN applications.
> > Challenging
> > > as more discussions are on-going.
> > >
> > > *Summary of 3.2.0 issues status:*
> > >
> > > 39 Blocker and Critical issues [1] are open, I am checking with owners
> to
> > > get status on each of them to get in by Code Freeze date.
> > >
> > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> (Blocker,
> > > Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
> > ORDER
> > > BY priority DESC
> > >
> > > Thanks,
> > > Sunil
> > >
> > > On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
> > >
> > >> Thanks Subru for the thoughts.
> > >> One of the main reason for a major release is to push out critical
> > >> features with a faster cadence to the users. If we are pulling more
> and
> > >> more different types of features to a minor release, that branch will
> > >> become more destabilized and it may be tough to say that 3.1.2 is
> stable
> > >> that 3.1.1 for eg. We always tend to improve and stabilize features in
> > >> subsequent minor release.
> > >> For few companies, it makes sense to push out these new features
> faster
> > >> to make a reach to the users. Adding to the point to the backporting
> > >> issues, I agree that its a pain and we can workaround that with some
> git
> > >> scripts. If we can make such scripts available to committers, backport
> > will
> > >> be seem-less across branches and we can achieve the faster release
> > cadence
> > >> also.
> > >>
> > >> Thoughts?
> > >>
> > >> - Sunil
> > >>
> > >>
> > >> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
> > wrote:
> > >>
> > >>> Thanks Sunil for volunteering to lead the release effort. I am
> > generally
> > >>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
> > >>> already
> > >>> have too many branches to be maintained. I already see many commits
> are
> > >>> in
> > >>> different branches with no apparent rationale, for e.g: 3.1 has
> commits
> > >>> which are absent in 3.0 etc.
> > >>>
> > >>> Additionally AFAIK 3.x has not been deployed in any major production
> > >>> setting so the cost of adding features should be minimal.
> > >>>
> > >>> Thoughts?
> > >>>
> > >>> -Subru
> > >>>
> > >>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
> > >>>
> > >>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> > >>> >
> > >>> > Yes, important changes and features are much needed, hence we will
> be
> > >>> > keeping the door open for them as possible. Also considering few
> more
> > >>> > offline requests from other folks, I think extending the timeframe
> by
> > >>> > couple of weeks makes sense (including a second RC buffer) and this
> > >>> should
> > >>> > ideally help us to ship this by September itself.
> > >>> >
> > >>> > Revised dates (I will be updating same in Roadmap wiki as well)
> > >>> >
> > >>> > - Feature freeze date : all features to merge by August 21, 2018.
> > >>> >
> > >>> > - Code freeze date : blockers/critical only, no improvements and
> non
> > >>> > blocker/critical
> > >>> >
> > >>> > bug-fixes August 31, 2018.
> > >>> >
> > >>> > - Release date: September 15, 2018
> > >>> >
> > >>> > Thank Eric and Zian, I think Wangda has already answered your
> > >>> questions.
> > >>> >
> > >>> > Thanks
> > >>> > Sunil
> > >>> >
> > >>> >
> > >>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
> > >>> wrote:
> > >>> >
> > >>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for
> that.
> > >>> > >
> > >>> > > To concerns from Steve,
> > >>> > >
> > >>> > > It is a good idea to keep the door open to get important changes
> /
> > >>> > > features in before cutoff. I would prefer to keep the proposed
> > >>> release
> > >>> > date
> > >>> > > to make sure things can happen earlier instead of last minute and
> > we
> > >>> all
> > >>> > > know that releases are always get delayed :). I'm also fine if we
> > >>> want
> > >>> > get
> > >>> > > another several weeks time.
> > >>> > >
> > >>> > > Regarding of 3.3 release, I would suggest doing that before
> > >>> thanksgiving.
> > >>> > > Do you think is it good or too early / late?
> > >>> > >
> > >>> > > Eric,
> > >>> > >
> > >>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
> > >>> merged
> > >>> > > in time, we probably not need the YARN-8220.
> > >>> > >
> > >>> > > Sunil,
> > >>> > >
> > >>> > > Could u update https://cwiki.apache.org/
> confluence/display/HADOOP/
> > >>> > Roadmap
> > >>> > > with proposed plan as well? We can fill feature list first before
> > >>> getting
> > >>> > > consensus of time.
> > >>> > >
> > >>> > > Thanks,
> > >>> > > Wangda
> > >>> > >
> > >>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> > >>> <fabbri@cloudera.com.invalid
> > >>> > >
> > >>> > > wrote:
> > >>> > >
> > >>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> > >>> stevel@hortonworks.com>
> > >>> > >> wrote:
> > >>> > >>
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
> > >>> > >> > sunilg@apache.org>> wrote:
> > >>> > >> >
> > >>> > >> > I would also would like to take this opportunity to come up
> > with a
> > >>> > >> detailed
> > >>> > >> > plan.
> > >>> > >> >
> > >>> > >> > - Feature freeze date : all features should be merged by
> August
> > >>> 10,
> > >>> > >> 2018.
> > >>> > >> >
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > <snip>
> > >>> > >>
> > >>> > >> >
> > >>> > >> > Please let me know if I missed any features targeted to 3.2
> per
> > >>> this
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > Well there these big todo lists for S3 & S3Guard.
> > >>> > >> >
> > >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> > >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen
> 2
> > >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> > >>> > >> >
> > >>> > >> > I don't think this is quite ready yet, I've been doing work on
> > >>> it, but
> > >>> > >> if
> > >>> > >> > we have a 3 week deadline, I'm going to expect some timely
> > >>> reviews on
> > >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> > >>> > >> >
> > >>> > >> > I've uprated that to a blocker feature; will review the S3 &
> > >>> S3Guard
> > >>> > >> JIRAs
> > >>> > >> > to see which of those are blocking. Then there are some
> pressing
> > >>> > "guave,
> > >>> > >> > java 9 prep"
> > >>> > >> >
> > >>> > >> >
> > >>> > >> I can help with this part if you like.
> > >>> > >>
> > >>> > >>
> > >>> > >>
> > >>> > >> >
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > timeline. I would like to volunteer myself as release manager
> of
> > >>> 3.2.0
> > >>> > >> > release.
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > well volunteered!
> > >>> > >> >
> > >>> > >> >
> > >>> > >> >
> > >>> > >> Yes, thank you for stepping up.
> > >>> > >>
> > >>> > >>
> > >>> > >> >
> > >>> > >> > I think this raises a good q: what timetable should we have
> for
> > >>> the
> > >>> > >> 3.2. &
> > >>> > >> > 3.3 releases; if we do want a faster cadence, then having the
> > >>> outline
> > >>> > >> time
> > >>> > >> > from the 3.2 to the 3.3 release means that there's less
> concern
> > >>> about
> > >>> > >> > things not making the 3.2 dealine
> > >>> > >> >
> > >>> > >> > -Steve
> > >>> > >> >
> > >>> > >> >
> > >>> > >> Good idea to mitigate the short deadline.
> > >>> > >>
> > >>> > >> -AF
> > >>> > >>
> > >>> > >
> > >>> >
> > >>>
> > >>
> >
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Haibo Chen <ha...@cloudera.com.INVALID>.
Hi Sunil,
For YARN-1011, we found a few minor issues that we have not merged our
internal fixes into the upstream feature branch. Hence, given the release
timeline, YARN-1011 will not make it.
Thanks.
On Thu, Aug 30, 2018 at 2:52 PM, Virajith Jalaparti <vi...@gmail.com>
wrote:
> Hi Sunil,
>
> Quick correction on the task list (missed this earlier) -- HDFS-12615 is
> being done by Inigo Goiri
>
> -Virajith
>
>
>
> On Thu, Aug 30, 2018 at 9:30 AM Sunil G <su...@apache.org> wrote:
>
> > Hi All,
> >
> > Inline with earlier communication dated 17th July 2018, I would like to
> > provide some updates.
> >
> > We are approaching previously proposed code freeze date (Aug 31).
> >
> > One of the critical feature Node Attributes feature merge discussion/vote
> > is ongoing. Also few other Blocker bugs need a bit more time. With regard
> > to this, suggesting to push the feature/code freeze for 2 more weeks to
> > accommodate these jiras too.
> >
> > Proposing Updated changes in plan inline with this:
> > Feature freeze date : all features to merge by September 7, 2018.
> > Code freeze date : blockers/critical only, no improvements and
> > blocker/critical bug-fixes September 14, 2018.
> > Release date: September 28, 2018
> >
> > If any features in branch which are targeted to 3.2.0, please reply to
> this
> > email thread.
> >
> > *Here's an updated 3.2.0 feature status:*
> >
> > 1. Merged & Completed features:
> >
> > - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> > Initial cut.
> > - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> > - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> > - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> > and CLI.
> >
> > 2. Features close to finish:
> >
> > - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
> > Ongoing.
> > - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> ATSv2.
> > Patch in progress.
> > - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> > - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
> In
> > progress.
> >
> > 3. Tentative features:
> >
> > - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
> be
> > done before Aug 2018.
> > - (Eric) YARN-7129: Application Catalog for YARN applications.
> Challenging
> > as more discussions are on-going.
> >
> > *Summary of 3.2.0 issues status:*
> >
> > 26 Blocker and Critical issues [1] are open, I am following up with
> owners
> > to get status on each of them to get in by Code Freeze date.
> >
> > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> > Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
> ORDER
> > BY priority DESC
> >
> > Thanks,
> > Sunil
> >
> > On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
> >
> > > Hi All,
> > >
> > > Thanks for the feedbacks. Inline with earlier communication dated 17th
> > > July 2018, I would like to provide some updates.
> > >
> > > We are approaching previously proposed feature freeze date (Aug 21,
> about
> > > 7 days from today).
> > > If any features in branch which are targeted to 3.2.0, please reply to
> > > this email thread.
> > > Steve has mentioned about the s3 features which will come close to Code
> > > Freeze Date (Aug 31st).
> > >
> > > *Here's an updated 3.2.0 feature status:*
> > >
> > > 1. Merged & Completed features:
> > >
> > > - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> workloads
> > > Initial cut.
> > > - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> > >
> > > 2. Features close to finish:
> > >
> > > - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
> patches
> > > are all in, only one last
> > > patch is in review state.
> > > - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> Scheduler.
> > > Close to commit.
> > > - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> API
> > > and CLI. 2 patches are pending
> > > which will be closed by Feature freeze date.
> > > - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> > ATSv2.
> > > Patch in progress.
> > > - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
> works.
> > > - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> Storage.
> > > In progress.
> > >
> > > 3. Tentative features:
> > >
> > > - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
> > be
> > > done before Aug 2018.
> > > - (Eric) YARN-7129: Application Catalog for YARN applications.
> > Challenging
> > > as more discussions are on-going.
> > >
> > > *Summary of 3.2.0 issues status:*
> > >
> > > 39 Blocker and Critical issues [1] are open, I am checking with owners
> to
> > > get status on each of them to get in by Code Freeze date.
> > >
> > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> (Blocker,
> > > Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
> > ORDER
> > > BY priority DESC
> > >
> > > Thanks,
> > > Sunil
> > >
> > > On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
> > >
> > >> Thanks Subru for the thoughts.
> > >> One of the main reason for a major release is to push out critical
> > >> features with a faster cadence to the users. If we are pulling more
> and
> > >> more different types of features to a minor release, that branch will
> > >> become more destabilized and it may be tough to say that 3.1.2 is
> stable
> > >> that 3.1.1 for eg. We always tend to improve and stabilize features in
> > >> subsequent minor release.
> > >> For few companies, it makes sense to push out these new features
> faster
> > >> to make a reach to the users. Adding to the point to the backporting
> > >> issues, I agree that its a pain and we can workaround that with some
> git
> > >> scripts. If we can make such scripts available to committers, backport
> > will
> > >> be seem-less across branches and we can achieve the faster release
> > cadence
> > >> also.
> > >>
> > >> Thoughts?
> > >>
> > >> - Sunil
> > >>
> > >>
> > >> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
> > wrote:
> > >>
> > >>> Thanks Sunil for volunteering to lead the release effort. I am
> > generally
> > >>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
> > >>> already
> > >>> have too many branches to be maintained. I already see many commits
> are
> > >>> in
> > >>> different branches with no apparent rationale, for e.g: 3.1 has
> commits
> > >>> which are absent in 3.0 etc.
> > >>>
> > >>> Additionally AFAIK 3.x has not been deployed in any major production
> > >>> setting so the cost of adding features should be minimal.
> > >>>
> > >>> Thoughts?
> > >>>
> > >>> -Subru
> > >>>
> > >>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
> > >>>
> > >>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> > >>> >
> > >>> > Yes, important changes and features are much needed, hence we will
> be
> > >>> > keeping the door open for them as possible. Also considering few
> more
> > >>> > offline requests from other folks, I think extending the timeframe
> by
> > >>> > couple of weeks makes sense (including a second RC buffer) and this
> > >>> should
> > >>> > ideally help us to ship this by September itself.
> > >>> >
> > >>> > Revised dates (I will be updating same in Roadmap wiki as well)
> > >>> >
> > >>> > - Feature freeze date : all features to merge by August 21, 2018.
> > >>> >
> > >>> > - Code freeze date : blockers/critical only, no improvements and
> non
> > >>> > blocker/critical
> > >>> >
> > >>> > bug-fixes August 31, 2018.
> > >>> >
> > >>> > - Release date: September 15, 2018
> > >>> >
> > >>> > Thank Eric and Zian, I think Wangda has already answered your
> > >>> questions.
> > >>> >
> > >>> > Thanks
> > >>> > Sunil
> > >>> >
> > >>> >
> > >>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
> > >>> wrote:
> > >>> >
> > >>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for
> that.
> > >>> > >
> > >>> > > To concerns from Steve,
> > >>> > >
> > >>> > > It is a good idea to keep the door open to get important changes
> /
> > >>> > > features in before cutoff. I would prefer to keep the proposed
> > >>> release
> > >>> > date
> > >>> > > to make sure things can happen earlier instead of last minute and
> > we
> > >>> all
> > >>> > > know that releases are always get delayed :). I'm also fine if we
> > >>> want
> > >>> > get
> > >>> > > another several weeks time.
> > >>> > >
> > >>> > > Regarding of 3.3 release, I would suggest doing that before
> > >>> thanksgiving.
> > >>> > > Do you think is it good or too early / late?
> > >>> > >
> > >>> > > Eric,
> > >>> > >
> > >>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
> > >>> merged
> > >>> > > in time, we probably not need the YARN-8220.
> > >>> > >
> > >>> > > Sunil,
> > >>> > >
> > >>> > > Could u update https://cwiki.apache.org/
> confluence/display/HADOOP/
> > >>> > Roadmap
> > >>> > > with proposed plan as well? We can fill feature list first before
> > >>> getting
> > >>> > > consensus of time.
> > >>> > >
> > >>> > > Thanks,
> > >>> > > Wangda
> > >>> > >
> > >>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> > >>> <fabbri@cloudera.com.invalid
> > >>> > >
> > >>> > > wrote:
> > >>> > >
> > >>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> > >>> stevel@hortonworks.com>
> > >>> > >> wrote:
> > >>> > >>
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
> > >>> > >> > sunilg@apache.org>> wrote:
> > >>> > >> >
> > >>> > >> > I would also would like to take this opportunity to come up
> > with a
> > >>> > >> detailed
> > >>> > >> > plan.
> > >>> > >> >
> > >>> > >> > - Feature freeze date : all features should be merged by
> August
> > >>> 10,
> > >>> > >> 2018.
> > >>> > >> >
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > <snip>
> > >>> > >>
> > >>> > >> >
> > >>> > >> > Please let me know if I missed any features targeted to 3.2
> per
> > >>> this
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > Well there these big todo lists for S3 & S3Guard.
> > >>> > >> >
> > >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> > >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen
> 2
> > >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> > >>> > >> >
> > >>> > >> > I don't think this is quite ready yet, I've been doing work on
> > >>> it, but
> > >>> > >> if
> > >>> > >> > we have a 3 week deadline, I'm going to expect some timely
> > >>> reviews on
> > >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> > >>> > >> >
> > >>> > >> > I've uprated that to a blocker feature; will review the S3 &
> > >>> S3Guard
> > >>> > >> JIRAs
> > >>> > >> > to see which of those are blocking. Then there are some
> pressing
> > >>> > "guave,
> > >>> > >> > java 9 prep"
> > >>> > >> >
> > >>> > >> >
> > >>> > >> I can help with this part if you like.
> > >>> > >>
> > >>> > >>
> > >>> > >>
> > >>> > >> >
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > timeline. I would like to volunteer myself as release manager
> of
> > >>> 3.2.0
> > >>> > >> > release.
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > well volunteered!
> > >>> > >> >
> > >>> > >> >
> > >>> > >> >
> > >>> > >> Yes, thank you for stepping up.
> > >>> > >>
> > >>> > >>
> > >>> > >> >
> > >>> > >> > I think this raises a good q: what timetable should we have
> for
> > >>> the
> > >>> > >> 3.2. &
> > >>> > >> > 3.3 releases; if we do want a faster cadence, then having the
> > >>> outline
> > >>> > >> time
> > >>> > >> > from the 3.2 to the 3.3 release means that there's less
> concern
> > >>> about
> > >>> > >> > things not making the 3.2 dealine
> > >>> > >> >
> > >>> > >> > -Steve
> > >>> > >> >
> > >>> > >> >
> > >>> > >> Good idea to mitigate the short deadline.
> > >>> > >>
> > >>> > >> -AF
> > >>> > >>
> > >>> > >
> > >>> >
> > >>>
> > >>
> >
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Virajith Jalaparti <vi...@gmail.com>.
Hi Sunil,
Quick correction on the task list (missed this earlier) -- HDFS-12615 is
being done by Inigo Goiri
-Virajith
On Thu, Aug 30, 2018 at 9:30 AM Sunil G <su...@apache.org> wrote:
> Hi All,
>
> Inline with earlier communication dated 17th July 2018, I would like to
> provide some updates.
>
> We are approaching previously proposed code freeze date (Aug 31).
>
> One of the critical feature Node Attributes feature merge discussion/vote
> is ongoing. Also few other Blocker bugs need a bit more time. With regard
> to this, suggesting to push the feature/code freeze for 2 more weeks to
> accommodate these jiras too.
>
> Proposing Updated changes in plan inline with this:
> Feature freeze date : all features to merge by September 7, 2018.
> Code freeze date : blockers/critical only, no improvements and
> blocker/critical bug-fixes September 14, 2018.
> Release date: September 28, 2018
>
> If any features in branch which are targeted to 3.2.0, please reply to this
> email thread.
>
> *Here's an updated 3.2.0 feature status:*
>
> 1. Merged & Completed features:
>
> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> Initial cut.
> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> and CLI.
>
> 2. Features close to finish:
>
> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
> Ongoing.
> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
> Patch in progress.
> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage. In
> progress.
>
> 3. Tentative features:
>
> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
> done before Aug 2018.
> - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
> as more discussions are on-going.
>
> *Summary of 3.2.0 issues status:*
>
> 26 Blocker and Critical issues [1] are open, I am following up with owners
> to get status on each of them to get in by Code Freeze date.
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
> BY priority DESC
>
> Thanks,
> Sunil
>
> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
>
> > Hi All,
> >
> > Thanks for the feedbacks. Inline with earlier communication dated 17th
> > July 2018, I would like to provide some updates.
> >
> > We are approaching previously proposed feature freeze date (Aug 21, about
> > 7 days from today).
> > If any features in branch which are targeted to 3.2.0, please reply to
> > this email thread.
> > Steve has mentioned about the s3 features which will come close to Code
> > Freeze Date (Aug 31st).
> >
> > *Here's an updated 3.2.0 feature status:*
> >
> > 1. Merged & Completed features:
> >
> > - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> > Initial cut.
> > - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >
> > 2. Features close to finish:
> >
> > - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major patches
> > are all in, only one last
> > patch is in review state.
> > - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> > Close to commit.
> > - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> > and CLI. 2 patches are pending
> > which will be closed by Feature freeze date.
> > - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> ATSv2.
> > Patch in progress.
> > - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> > - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
> > In progress.
> >
> > 3. Tentative features:
> >
> > - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
> be
> > done before Aug 2018.
> > - (Eric) YARN-7129: Application Catalog for YARN applications.
> Challenging
> > as more discussions are on-going.
> >
> > *Summary of 3.2.0 issues status:*
> >
> > 39 Blocker and Critical issues [1] are open, I am checking with owners to
> > get status on each of them to get in by Code Freeze date.
> >
> > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> > Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
> ORDER
> > BY priority DESC
> >
> > Thanks,
> > Sunil
> >
> > On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
> >
> >> Thanks Subru for the thoughts.
> >> One of the main reason for a major release is to push out critical
> >> features with a faster cadence to the users. If we are pulling more and
> >> more different types of features to a minor release, that branch will
> >> become more destabilized and it may be tough to say that 3.1.2 is stable
> >> that 3.1.1 for eg. We always tend to improve and stabilize features in
> >> subsequent minor release.
> >> For few companies, it makes sense to push out these new features faster
> >> to make a reach to the users. Adding to the point to the backporting
> >> issues, I agree that its a pain and we can workaround that with some git
> >> scripts. If we can make such scripts available to committers, backport
> will
> >> be seem-less across branches and we can achieve the faster release
> cadence
> >> also.
> >>
> >> Thoughts?
> >>
> >> - Sunil
> >>
> >>
> >> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
> wrote:
> >>
> >>> Thanks Sunil for volunteering to lead the release effort. I am
> generally
> >>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
> >>> already
> >>> have too many branches to be maintained. I already see many commits are
> >>> in
> >>> different branches with no apparent rationale, for e.g: 3.1 has commits
> >>> which are absent in 3.0 etc.
> >>>
> >>> Additionally AFAIK 3.x has not been deployed in any major production
> >>> setting so the cost of adding features should be minimal.
> >>>
> >>> Thoughts?
> >>>
> >>> -Subru
> >>>
> >>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
> >>>
> >>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> >>> >
> >>> > Yes, important changes and features are much needed, hence we will be
> >>> > keeping the door open for them as possible. Also considering few more
> >>> > offline requests from other folks, I think extending the timeframe by
> >>> > couple of weeks makes sense (including a second RC buffer) and this
> >>> should
> >>> > ideally help us to ship this by September itself.
> >>> >
> >>> > Revised dates (I will be updating same in Roadmap wiki as well)
> >>> >
> >>> > - Feature freeze date : all features to merge by August 21, 2018.
> >>> >
> >>> > - Code freeze date : blockers/critical only, no improvements and non
> >>> > blocker/critical
> >>> >
> >>> > bug-fixes August 31, 2018.
> >>> >
> >>> > - Release date: September 15, 2018
> >>> >
> >>> > Thank Eric and Zian, I think Wangda has already answered your
> >>> questions.
> >>> >
> >>> > Thanks
> >>> > Sunil
> >>> >
> >>> >
> >>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
> >>> wrote:
> >>> >
> >>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for that.
> >>> > >
> >>> > > To concerns from Steve,
> >>> > >
> >>> > > It is a good idea to keep the door open to get important changes /
> >>> > > features in before cutoff. I would prefer to keep the proposed
> >>> release
> >>> > date
> >>> > > to make sure things can happen earlier instead of last minute and
> we
> >>> all
> >>> > > know that releases are always get delayed :). I'm also fine if we
> >>> want
> >>> > get
> >>> > > another several weeks time.
> >>> > >
> >>> > > Regarding of 3.3 release, I would suggest doing that before
> >>> thanksgiving.
> >>> > > Do you think is it good or too early / late?
> >>> > >
> >>> > > Eric,
> >>> > >
> >>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
> >>> merged
> >>> > > in time, we probably not need the YARN-8220.
> >>> > >
> >>> > > Sunil,
> >>> > >
> >>> > > Could u update https://cwiki.apache.org/confluence/display/HADOOP/
> >>> > Roadmap
> >>> > > with proposed plan as well? We can fill feature list first before
> >>> getting
> >>> > > consensus of time.
> >>> > >
> >>> > > Thanks,
> >>> > > Wangda
> >>> > >
> >>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> >>> <fabbri@cloudera.com.invalid
> >>> > >
> >>> > > wrote:
> >>> > >
> >>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> >>> stevel@hortonworks.com>
> >>> > >> wrote:
> >>> > >>
> >>> > >> >
> >>> > >> >
> >>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
> >>> > >> > sunilg@apache.org>> wrote:
> >>> > >> >
> >>> > >> > I would also would like to take this opportunity to come up
> with a
> >>> > >> detailed
> >>> > >> > plan.
> >>> > >> >
> >>> > >> > - Feature freeze date : all features should be merged by August
> >>> 10,
> >>> > >> 2018.
> >>> > >> >
> >>> > >> >
> >>> > >> >
> >>> > >> > <snip>
> >>> > >>
> >>> > >> >
> >>> > >> > Please let me know if I missed any features targeted to 3.2 per
> >>> this
> >>> > >> >
> >>> > >> >
> >>> > >> > Well there these big todo lists for S3 & S3Guard.
> >>> > >> >
> >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> >>> > >> >
> >>> > >> >
> >>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen 2
> >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> >>> > >> >
> >>> > >> > I don't think this is quite ready yet, I've been doing work on
> >>> it, but
> >>> > >> if
> >>> > >> > we have a 3 week deadline, I'm going to expect some timely
> >>> reviews on
> >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> >>> > >> >
> >>> > >> > I've uprated that to a blocker feature; will review the S3 &
> >>> S3Guard
> >>> > >> JIRAs
> >>> > >> > to see which of those are blocking. Then there are some pressing
> >>> > "guave,
> >>> > >> > java 9 prep"
> >>> > >> >
> >>> > >> >
> >>> > >> I can help with this part if you like.
> >>> > >>
> >>> > >>
> >>> > >>
> >>> > >> >
> >>> > >> >
> >>> > >> >
> >>> > >> > timeline. I would like to volunteer myself as release manager of
> >>> 3.2.0
> >>> > >> > release.
> >>> > >> >
> >>> > >> >
> >>> > >> > well volunteered!
> >>> > >> >
> >>> > >> >
> >>> > >> >
> >>> > >> Yes, thank you for stepping up.
> >>> > >>
> >>> > >>
> >>> > >> >
> >>> > >> > I think this raises a good q: what timetable should we have for
> >>> the
> >>> > >> 3.2. &
> >>> > >> > 3.3 releases; if we do want a faster cadence, then having the
> >>> outline
> >>> > >> time
> >>> > >> > from the 3.2 to the 3.3 release means that there's less concern
> >>> about
> >>> > >> > things not making the 3.2 dealine
> >>> > >> >
> >>> > >> > -Steve
> >>> > >> >
> >>> > >> >
> >>> > >> Good idea to mitigate the short deadline.
> >>> > >>
> >>> > >> -AF
> >>> > >>
> >>> > >
> >>> >
> >>>
> >>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Virajith Jalaparti <vi...@gmail.com>.
Hi Sunil,
Quick correction on the task list (missed this earlier) -- HDFS-12615 is
being done by Inigo Goiri
-Virajith
On Thu, Aug 30, 2018 at 9:30 AM Sunil G <su...@apache.org> wrote:
> Hi All,
>
> Inline with earlier communication dated 17th July 2018, I would like to
> provide some updates.
>
> We are approaching previously proposed code freeze date (Aug 31).
>
> One of the critical feature Node Attributes feature merge discussion/vote
> is ongoing. Also few other Blocker bugs need a bit more time. With regard
> to this, suggesting to push the feature/code freeze for 2 more weeks to
> accommodate these jiras too.
>
> Proposing Updated changes in plan inline with this:
> Feature freeze date : all features to merge by September 7, 2018.
> Code freeze date : blockers/critical only, no improvements and
> blocker/critical bug-fixes September 14, 2018.
> Release date: September 28, 2018
>
> If any features in branch which are targeted to 3.2.0, please reply to this
> email thread.
>
> *Here's an updated 3.2.0 feature status:*
>
> 1. Merged & Completed features:
>
> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> Initial cut.
> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> and CLI.
>
> 2. Features close to finish:
>
> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
> Ongoing.
> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
> Patch in progress.
> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage. In
> progress.
>
> 3. Tentative features:
>
> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
> done before Aug 2018.
> - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
> as more discussions are on-going.
>
> *Summary of 3.2.0 issues status:*
>
> 26 Blocker and Critical issues [1] are open, I am following up with owners
> to get status on each of them to get in by Code Freeze date.
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
> BY priority DESC
>
> Thanks,
> Sunil
>
> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
>
> > Hi All,
> >
> > Thanks for the feedbacks. Inline with earlier communication dated 17th
> > July 2018, I would like to provide some updates.
> >
> > We are approaching previously proposed feature freeze date (Aug 21, about
> > 7 days from today).
> > If any features in branch which are targeted to 3.2.0, please reply to
> > this email thread.
> > Steve has mentioned about the s3 features which will come close to Code
> > Freeze Date (Aug 31st).
> >
> > *Here's an updated 3.2.0 feature status:*
> >
> > 1. Merged & Completed features:
> >
> > - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> > Initial cut.
> > - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >
> > 2. Features close to finish:
> >
> > - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major patches
> > are all in, only one last
> > patch is in review state.
> > - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> > Close to commit.
> > - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> > and CLI. 2 patches are pending
> > which will be closed by Feature freeze date.
> > - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> ATSv2.
> > Patch in progress.
> > - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> > - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
> > In progress.
> >
> > 3. Tentative features:
> >
> > - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
> be
> > done before Aug 2018.
> > - (Eric) YARN-7129: Application Catalog for YARN applications.
> Challenging
> > as more discussions are on-going.
> >
> > *Summary of 3.2.0 issues status:*
> >
> > 39 Blocker and Critical issues [1] are open, I am checking with owners to
> > get status on each of them to get in by Code Freeze date.
> >
> > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> > Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
> ORDER
> > BY priority DESC
> >
> > Thanks,
> > Sunil
> >
> > On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
> >
> >> Thanks Subru for the thoughts.
> >> One of the main reason for a major release is to push out critical
> >> features with a faster cadence to the users. If we are pulling more and
> >> more different types of features to a minor release, that branch will
> >> become more destabilized and it may be tough to say that 3.1.2 is stable
> >> that 3.1.1 for eg. We always tend to improve and stabilize features in
> >> subsequent minor release.
> >> For few companies, it makes sense to push out these new features faster
> >> to make a reach to the users. Adding to the point to the backporting
> >> issues, I agree that its a pain and we can workaround that with some git
> >> scripts. If we can make such scripts available to committers, backport
> will
> >> be seem-less across branches and we can achieve the faster release
> cadence
> >> also.
> >>
> >> Thoughts?
> >>
> >> - Sunil
> >>
> >>
> >> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
> wrote:
> >>
> >>> Thanks Sunil for volunteering to lead the release effort. I am
> generally
> >>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
> >>> already
> >>> have too many branches to be maintained. I already see many commits are
> >>> in
> >>> different branches with no apparent rationale, for e.g: 3.1 has commits
> >>> which are absent in 3.0 etc.
> >>>
> >>> Additionally AFAIK 3.x has not been deployed in any major production
> >>> setting so the cost of adding features should be minimal.
> >>>
> >>> Thoughts?
> >>>
> >>> -Subru
> >>>
> >>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
> >>>
> >>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> >>> >
> >>> > Yes, important changes and features are much needed, hence we will be
> >>> > keeping the door open for them as possible. Also considering few more
> >>> > offline requests from other folks, I think extending the timeframe by
> >>> > couple of weeks makes sense (including a second RC buffer) and this
> >>> should
> >>> > ideally help us to ship this by September itself.
> >>> >
> >>> > Revised dates (I will be updating same in Roadmap wiki as well)
> >>> >
> >>> > - Feature freeze date : all features to merge by August 21, 2018.
> >>> >
> >>> > - Code freeze date : blockers/critical only, no improvements and non
> >>> > blocker/critical
> >>> >
> >>> > bug-fixes August 31, 2018.
> >>> >
> >>> > - Release date: September 15, 2018
> >>> >
> >>> > Thank Eric and Zian, I think Wangda has already answered your
> >>> questions.
> >>> >
> >>> > Thanks
> >>> > Sunil
> >>> >
> >>> >
> >>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
> >>> wrote:
> >>> >
> >>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for that.
> >>> > >
> >>> > > To concerns from Steve,
> >>> > >
> >>> > > It is a good idea to keep the door open to get important changes /
> >>> > > features in before cutoff. I would prefer to keep the proposed
> >>> release
> >>> > date
> >>> > > to make sure things can happen earlier instead of last minute and
> we
> >>> all
> >>> > > know that releases are always get delayed :). I'm also fine if we
> >>> want
> >>> > get
> >>> > > another several weeks time.
> >>> > >
> >>> > > Regarding of 3.3 release, I would suggest doing that before
> >>> thanksgiving.
> >>> > > Do you think is it good or too early / late?
> >>> > >
> >>> > > Eric,
> >>> > >
> >>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
> >>> merged
> >>> > > in time, we probably not need the YARN-8220.
> >>> > >
> >>> > > Sunil,
> >>> > >
> >>> > > Could u update https://cwiki.apache.org/confluence/display/HADOOP/
> >>> > Roadmap
> >>> > > with proposed plan as well? We can fill feature list first before
> >>> getting
> >>> > > consensus of time.
> >>> > >
> >>> > > Thanks,
> >>> > > Wangda
> >>> > >
> >>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> >>> <fabbri@cloudera.com.invalid
> >>> > >
> >>> > > wrote:
> >>> > >
> >>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> >>> stevel@hortonworks.com>
> >>> > >> wrote:
> >>> > >>
> >>> > >> >
> >>> > >> >
> >>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
> >>> > >> > sunilg@apache.org>> wrote:
> >>> > >> >
> >>> > >> > I would also would like to take this opportunity to come up
> with a
> >>> > >> detailed
> >>> > >> > plan.
> >>> > >> >
> >>> > >> > - Feature freeze date : all features should be merged by August
> >>> 10,
> >>> > >> 2018.
> >>> > >> >
> >>> > >> >
> >>> > >> >
> >>> > >> > <snip>
> >>> > >>
> >>> > >> >
> >>> > >> > Please let me know if I missed any features targeted to 3.2 per
> >>> this
> >>> > >> >
> >>> > >> >
> >>> > >> > Well there these big todo lists for S3 & S3Guard.
> >>> > >> >
> >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> >>> > >> >
> >>> > >> >
> >>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen 2
> >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> >>> > >> >
> >>> > >> > I don't think this is quite ready yet, I've been doing work on
> >>> it, but
> >>> > >> if
> >>> > >> > we have a 3 week deadline, I'm going to expect some timely
> >>> reviews on
> >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> >>> > >> >
> >>> > >> > I've uprated that to a blocker feature; will review the S3 &
> >>> S3Guard
> >>> > >> JIRAs
> >>> > >> > to see which of those are blocking. Then there are some pressing
> >>> > "guave,
> >>> > >> > java 9 prep"
> >>> > >> >
> >>> > >> >
> >>> > >> I can help with this part if you like.
> >>> > >>
> >>> > >>
> >>> > >>
> >>> > >> >
> >>> > >> >
> >>> > >> >
> >>> > >> > timeline. I would like to volunteer myself as release manager of
> >>> 3.2.0
> >>> > >> > release.
> >>> > >> >
> >>> > >> >
> >>> > >> > well volunteered!
> >>> > >> >
> >>> > >> >
> >>> > >> >
> >>> > >> Yes, thank you for stepping up.
> >>> > >>
> >>> > >>
> >>> > >> >
> >>> > >> > I think this raises a good q: what timetable should we have for
> >>> the
> >>> > >> 3.2. &
> >>> > >> > 3.3 releases; if we do want a faster cadence, then having the
> >>> outline
> >>> > >> time
> >>> > >> > from the 3.2 to the 3.3 release means that there's less concern
> >>> about
> >>> > >> > things not making the 3.2 dealine
> >>> > >> >
> >>> > >> > -Steve
> >>> > >> >
> >>> > >> >
> >>> > >> Good idea to mitigate the short deadline.
> >>> > >>
> >>> > >> -AF
> >>> > >>
> >>> > >
> >>> >
> >>>
> >>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi Folks,
Given all blockers are closed now, I am planning to do RC1 tomorrow.
branch-3.2.0 remains closed for commit. Please let me know if any issues
are missed.
Thanks,
Sunil G
On Tue, Nov 13, 2018 at 8:11 PM Sunil G <su...@apache.org> wrote:
> Hi Folks,
>
> All blockers are closed by last weekend and corrected jiras as well.
> Preparing RC0 now. Facing some issue with shaded jars file size. Hence
> respinning again.
> Planning to complete this by end of this week.
>
> Thanks
> Sunil
>
> On Fri, Oct 26, 2018 at 2:34 AM Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
>> Another thing is that I see a bunch of jiras under HDFS-8707, which don't
>> have the Fix Version field listing 3.2, some have it just empty.
>> This means they will not be populated into release notes.
>>
>> Thanks,
>> --Konstantin
>>
>> On Thu, Oct 25, 2018 at 7:59 AM Sunil G <su...@apache.org> wrote:
>>
>> > Thanks Konstantin for pointing out.
>> > As 3.2 is pretty much on RC level, its better we try to find a good
>> > solution to this issue.
>> >
>> > I ll follow up on this in the jira.
>> >
>> > - Sunil
>> >
>> > On Thu, Oct 25, 2018 at 11:35 AM Konstantin Shvachko <
>> shv.hadoop@gmail.com>
>> > wrote:
>> >
>> >> I've tried to attract attention to an incompatibility issue through the
>> >> jira, but it didn't work. So pitching in in this thread.
>> >> https://issues.apache.org/jira/browse/HDFS-12026
>> >> It introduced binary incompatibility, which will prevent people from
>> >> upgrading from 3.1 to 3.2.
>> >> I think it can get messy if we release anything with this feature.
>> >>
>> >> Thanks,
>> >> --Konstantin
>> >>
>> >> On Mon, Oct 22, 2018 at 5:01 AM Steve Loughran <stevel@hortonworks.com
>> >
>> >> wrote:
>> >>
>> >>> its in.
>> >>>
>> >>> good catch!
>> >>>
>> >>> On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <weichiu@cloudera.com
>> <mailto:
>> >>> weichiu@cloudera.com>> wrote:
>> >>>
>> >>> Thanks Sunil G for driving the release,
>> >>> I filed HADOOP-15866<
>> https://issues.apache.org/jira/browse/HADOOP-15866>
>> >>> for a compat fix. If any one has cycle please review it, as I think
>> it is
>> >>> needed for 3.2.0.
>> >>>
>> >>> On Thu, Oct 18, 2018 at 4:43 AM Sunil G <sunilg@apache.org<mailto:
>> >>> sunilg@apache.org>> wrote:
>> >>> Hi Folks,
>> >>>
>> >>> As we previously communicated for 3.2.0 release, we have delayed due
>> to
>> >>> few
>> >>> blockers in our gate.
>> >>>
>> >>> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open
>> for
>> >>> all bug fixes.
>> >>>
>> >>> - Sunil
>> >>>
>> >>>
>> >>> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <sunilg@apache.org<mailto:
>> >>> sunilg@apache.org>> wrote:
>> >>>
>> >>> > Hi Folks,
>> >>> >
>> >>> > We are now close to RC as other blocker issues are now merged to
>> trunk
>> >>> and
>> >>> > branch-3.2. Last 2 critical issues are closer to merge and will be
>> >>> > committed in few hours.
>> >>> > With this, I will be creating 3.2.0 branch today and will go ahead
>> >>> with RC
>> >>> > related process.
>> >>> >
>> >>> > - Sunil
>> >>> >
>> >>> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <
>> jonbender@stripe.com
>> >>> <ma...@stripe.com>>
>> >>> > wrote:
>> >>> >
>> >>> >> Hello, were there any updates around the 3.2.0 RC timing? All I
>> see in
>> >>> >> the current blockers are related to the new Submarine subproject,
>> >>> wasn't
>> >>> >> sure if that is what is holding things up.
>> >>> >>
>> >>> >> Cheers,
>> >>> >> Jon
>> >>> >>
>> >>> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <sunilg@apache.org<mailto:
>> >>> sunilg@apache.org>> wrote:
>> >>> >>
>> >>> >>> Thanks Robert and Haibo for quickly correcting same.
>> >>> >>> Sigh, I somehow missed one file while committing the change. Sorry
>> >>> for
>> >>> >>> the
>> >>> >>> trouble.
>> >>> >>>
>> >>> >>> - Sunil
>> >>> >>>
>> >>> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <
>> rkanter@cloudera.com
>> >>> <ma...@cloudera.com>>
>> >>> >>> wrote:
>> >>> >>>
>> >>> >>> > Looks like there's two that weren't updated:
>> >>> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT"
>> .
>> >>> -r
>> >>> >>> > --include=pom.xml
>> >>> >>> > ./hadoop-project/pom.xml:
>> >>> >>> >
>> >>> <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>> >>> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>> >>> >>> >
>> >>> >>> > I've just pushed in an addendum commit to fix those.
>> >>> >>> > In the future, please make sure to do a sanity compile when
>> >>> updating
>> >>> >>> poms.
>> >>> >>> >
>> >>> >>> > thanks
>> >>> >>> > - Robert
>> >>> >>> >
>> >>> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>> >>> >>> <fa...@cloudera.com.invalid>>
>> >>> >>> > wrote:
>> >>> >>> >
>> >>> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in
>> >>> the
>> >>> >>> >> top-level pom.xml?
>> >>> >>> >>
>> >>> >>> >>
>> >>> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <sunilg@apache.org
>> >>> <ma...@apache.org>> wrote:
>> >>> >>> >>
>> >>> >>> >> > Hi All
>> >>> >>> >> >
>> >>> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
>> >>> trunk
>> >>> >>> to
>> >>> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
>> >>> >>> necessary
>> >>> >>> >> > patches are pulled into branch-3.2.
>> >>> >>> >> >
>> >>> >>> >> > Thank You
>> >>> >>> >> > - Sunil
>> >>> >>> >> >
>> >>> >>> >> >
>> >>> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <sunilg@apache.org
>> >>> <ma...@apache.org>> wrote:
>> >>> >>> >> >
>> >>> >>> >> > > Hi All
>> >>> >>> >> > >
>> >>> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is
>> >>> merged to
>> >>> >>> >> trunk.
>> >>> >>> >> > > Thanks for the support.
>> >>> >>> >> > >
>> >>> >>> >> > > *Plan for RC*
>> >>> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>> >>> >>> >> > > RC0 for 3.2: *28th Friday*
>> >>> >>> >> > >
>> >>> >>> >> > > Thank You
>> >>> >>> >> > > Sunil
>> >>> >>> >> > >
>> >>> >>> >> > >
>> >>> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <sunilg@apache.org
>> >>> <ma...@apache.org>>
>> >>> >>> wrote:
>> >>> >>> >> > >
>> >>> >>> >> > >> Hi All
>> >>> >>> >> > >>
>> >>> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all
>> of
>> >>> you
>> >>> >>> for
>> >>> >>> >> > >> helping in this. I am following up on these tickets, once
>> its
>> >>> >>> closed
>> >>> >>> >> we
>> >>> >>> >> > >> will cut the 3.2 branch.
>> >>> >>> >> > >>
>> >>> >>> >> > >> Thanks
>> >>> >>> >> > >> Sunil Govindan
>> >>> >>> >> > >>
>> >>> >>> >> > >>
>> >>> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <
>> sunilg@apache.org
>> >>> <ma...@apache.org>>
>> >>> >>> wrote:
>> >>> >>> >> > >>
>> >>> >>> >> > >>> Hi All,
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> Inline with the original 3.2 communication proposal dated
>> >>> 17th
>> >>> >>> July
>> >>> >>> >> > >>> 2018, I would like to provide more updates.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> We are approaching previously proposed code freeze date
>> >>> >>> (September
>> >>> >>> >> 14,
>> >>> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
>> >>> point
>> >>> >>> >> > existing
>> >>> >>> >> > >>> trunk to 3.3 if there are no issues.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> *Current Release Plan:*
>> >>> >>> >> > >>> Feature freeze date : all features to merge by September
>> 7,
>> >>> >>> 2018.
>> >>> >>> >> > >>> Code freeze date : blockers/critical only, no
>> improvements
>> >>> and
>> >>> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>> >>> >>> >> > >>> Release date: September 28, 2018
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> If any critical/blocker tickets which are targeted to
>> >>> 3.2.0, we
>> >>> >>> >> need to
>> >>> >>> >> > >>> backport to 3.2 post branch cut.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> Here's an updated 3.2.0 feature status:
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> 1. Merged & Completed features:
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> >>> DeepLearning
>> >>> >>> >> > >>> workloads Initial cut.
>> >>> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> >>> Capacity
>> >>> >>> >> > >>> Scheduler.
>> >>> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>> YARN
>> >>> >>> Service
>> >>> >>> >> > API
>> >>> >>> >> > >>> and CLI.
>> >>> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in
>> YARN.
>> >>> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
>> >>> Improvement
>> >>> >>> >> works.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> 2. Features close to finish:
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>> >>> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
>> >>> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>> >>> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>> >>> apps
>> >>> >>> from
>> >>> >>> >> > >>> ATSv2. Patch in progress.
>> >>> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >>> >>> >> challenging to
>> >>> >>> >> > >>> be done before Aug 2018.
>> >>> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN
>> >>> applications.
>> >>> >>> >> > >>> Challenging as more discussions are on-going.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> *Summary of 3.2.0 issues status:*
>> >>> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am
>> following
>> >>> up
>> >>> >>> with
>> >>> >>> >> > >>> owners to get status on each of them to get in by Code
>> >>> Freeze
>> >>> >>> date.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>> priority
>> >>> in
>> >>> >>> >> > (Blocker,
>> >>> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target
>> >>> Version/s" =
>> >>> >>> 3.2.0
>> >>> >>> >> > ORDER
>> >>> >>> >> > >>> BY priority DESC
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> Thanks,
>> >>> >>> >> > >>> Sunil
>> >>> >>> >> > >>>
>> >>> >>> >> > >>>
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <
>> sunilg@apache.org
>> >>> <ma...@apache.org>>
>> >>> >>> wrote:
>> >>> >>> >> > >>>
>> >>> >>> >> > >>>> Hi All,
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> Inline with earlier communication dated 17th July 2018,
>> I
>> >>> would
>> >>> >>> >> like
>> >>> >>> >> > to
>> >>> >>> >> > >>>> provide some updates.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> We are approaching previously proposed code freeze date
>> >>> (Aug
>> >>> >>> 31).
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> One of the critical feature Node Attributes feature
>> merge
>> >>> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs
>> >>> need a
>> >>> >>> bit
>> >>> >>> >> > more
>> >>> >>> >> > >>>> time. With regard to this, suggesting to push the
>> >>> feature/code
>> >>> >>> >> freeze
>> >>> >>> >> > for 2
>> >>> >>> >> > >>>> more weeks to accommodate these jiras too.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> Proposing Updated changes in plan inline with this:
>> >>> >>> >> > >>>> Feature freeze date : all features to merge by
>> September 7,
>> >>> >>> 2018.
>> >>> >>> >> > >>>> Code freeze date : blockers/critical only, no
>> improvements
>> >>> and
>> >>> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>> >>> >>> >> > >>>> Release date: September 28, 2018
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> If any features in branch which are targeted to 3.2.0,
>> >>> please
>> >>> >>> >> reply to
>> >>> >>> >> > >>>> this email thread.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> 1. Merged & Completed features:
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> >>> DeepLearning
>> >>> >>> >> > >>>> workloads Initial cut.
>> >>> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> >>> Capacity
>> >>> >>> >> > >>>> Scheduler.
>> >>> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>> >>> YARN
>> >>> >>> >> Service
>> >>> >>> >> > >>>> API and CLI.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> 2. Features close to finish:
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in
>> YARN.
>> >>> >>> >> Merge/Vote
>> >>> >>> >> > >>>> Ongoing.
>> >>> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of
>> historical
>> >>> apps
>> >>> >>> from
>> >>> >>> >> > >>>> ATSv2. Patch in progress.
>> >>> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> >>> >>> Improvement
>> >>> >>> >> > works.
>> >>> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support
>> Windows
>> >>> Azure
>> >>> >>> >> > >>>> Storage. In progress.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> 3. Tentative features:
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >>> >>> >> challenging
>> >>> >>> >> > to
>> >>> >>> >> > >>>> be done before Aug 2018.
>> >>> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
>> >>> applications.
>> >>> >>> >> > >>>> Challenging as more discussions are on-going.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> *Summary of 3.2.0 issues status:*
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am
>> >>> following up
>> >>> >>> with
>> >>> >>> >> > >>>> owners to get status on each of them to get in by Code
>> >>> Freeze
>> >>> >>> date.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>> >>> priority in
>> >>> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND
>> "Target
>> >>> >>> >> > Version/s" =
>> >>> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> Thanks,
>> >>> >>> >> > >>>> Sunil
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <
>> >>> sunilg@apache.org<ma...@apache.org>>
>> >>> >>> >> wrote:
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>>> Hi All,
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier
>> >>> communication
>> >>> >>> dated
>> >>> >>> >> > 17th
>> >>> >>> >> > >>>>> July 2018, I would like to provide some updates.
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> We are approaching previously proposed feature freeze
>> date
>> >>> >>> (Aug
>> >>> >>> >> 21,
>> >>> >>> >> > >>>>> about 7 days from today).
>> >>> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
>> >>> please
>> >>> >>> >> reply
>> >>> >>> >> > to
>> >>> >>> >> > >>>>> this email thread.
>> >>> >>> >> > >>>>> Steve has mentioned about the s3 features which will
>> come
>> >>> >>> close to
>> >>> >>> >> > >>>>> Code Freeze Date (Aug 31st).
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> 1. Merged & Completed features:
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> >>> >>> DeepLearning
>> >>> >>> >> > >>>>> workloads Initial cut.
>> >>> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> 2. Features close to finish:
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in
>> YARN.
>> >>> >>> Major
>> >>> >>> >> > >>>>> patches are all in, only one last
>> >>> >>> >> > >>>>> patch is in review state.
>> >>> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> >>> Capacity
>> >>> >>> >> > >>>>> Scheduler. Close to commit.
>> >>> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>> >>> YARN
>> >>> >>> >> Service
>> >>> >>> >> > >>>>> API and CLI. 2 patches are pending
>> >>> >>> >> > >>>>> which will be closed by Feature freeze date.
>> >>> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of
>> historical
>> >>> apps
>> >>> >>> >> from
>> >>> >>> >> > >>>>> ATSv2. Patch in progress.
>> >>> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> >>> >>> Improvement
>> >>> >>> >> > >>>>> works.
>> >>> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support
>> Windows
>> >>> >>> Azure
>> >>> >>> >> > >>>>> Storage. In progress.
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> 3. Tentative features:
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment.
>> Looks
>> >>> >>> >> challenging
>> >>> >>> >> > >>>>> to be done before Aug 2018.
>> >>> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
>> >>> applications.
>> >>> >>> >> > >>>>> Challenging as more discussions are on-going.
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am
>> checking
>> >>> >>> with
>> >>> >>> >> > owners
>> >>> >>> >> > >>>>> to get status on each of them to get in by Code Freeze
>> >>> date.
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>> >>> priority in
>> >>> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND
>> >>> "Target
>> >>> >>> >> > Version/s" =
>> >>> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> Thanks,
>> >>> >>> >> > >>>>> Sunil
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <
>> >>> sunilg@apache.org<ma...@apache.org>>
>> >>> >>> >> wrote:
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>>> Thanks Subru for the thoughts.
>> >>> >>> >> > >>>>>> One of the main reason for a major release is to push
>> out
>> >>> >>> >> critical
>> >>> >>> >> > >>>>>> features with a faster cadence to the users. If we are
>> >>> >>> pulling
>> >>> >>> >> more
>> >>> >>> >> > and
>> >>> >>> >> > >>>>>> more different types of features to a minor release,
>> that
>> >>> >>> branch
>> >>> >>> >> > will
>> >>> >>> >> > >>>>>> become more destabilized and it may be tough to say
>> that
>> >>> >>> 3.1.2 is
>> >>> >>> >> > stable
>> >>> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and
>> >>> stabilize
>> >>> >>> >> features
>> >>> >>> >> > in
>> >>> >>> >> > >>>>>> subsequent minor release.
>> >>> >>> >> > >>>>>> For few companies, it makes sense to push out these
>> new
>> >>> >>> features
>> >>> >>> >> > >>>>>> faster to make a reach to the users. Adding to the
>> point
>> >>> to
>> >>> >>> the
>> >>> >>> >> > backporting
>> >>> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround
>> >>> that
>> >>> >>> with
>> >>> >>> >> > some git
>> >>> >>> >> > >>>>>> scripts. If we can make such scripts available to
>> >>> committers,
>> >>> >>> >> > backport will
>> >>> >>> >> > >>>>>> be seem-less across branches and we can achieve the
>> >>> faster
>> >>> >>> >> release
>> >>> >>> >> > cadence
>> >>> >>> >> > >>>>>> also.
>> >>> >>> >> > >>>>>>
>> >>> >>> >> > >>>>>> Thoughts?
>> >>> >>> >> > >>>>>>
>> >>> >>> >> > >>>>>> - Sunil
>> >>> >>> >> > >>>>>>
>> >>> >>> >> > >>>>>>
>> >>> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>> >>> >>> subru@apache.org<ma...@apache.org>
>> >>> >>> >> >
>> >>> >>> >> > >>>>>> wrote:
>> >>> >>> >> > >>>>>>
>> >>> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release
>> >>> effort. I
>> >>> >>> am
>> >>> >>> >> > >>>>>>> generally
>> >>> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a
>> >>> 3.1.x) as
>> >>> >>> >> feel we
>> >>> >>> >> > >>>>>>> already
>> >>> >>> >> > >>>>>>> have too many branches to be maintained. I already
>> see
>> >>> many
>> >>> >>> >> commits
>> >>> >>> >> > >>>>>>> are in
>> >>> >>> >> > >>>>>>> different branches with no apparent rationale, for
>> e.g:
>> >>> 3.1
>> >>> >>> has
>> >>> >>> >> > >>>>>>> commits
>> >>> >>> >> > >>>>>>> which are absent in 3.0 etc.
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any
>> >>> major
>> >>> >>> >> > production
>> >>> >>> >> > >>>>>>> setting so the cost of adding features should be
>> >>> minimal.
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>> Thoughts?
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>> -Subru
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
>> >>> >>> sunilg@apache.org<ma...@apache.org>>
>> >>> >>> >> > wrote:
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > Yes, important changes and features are much
>> needed,
>> >>> >>> hence we
>> >>> >>> >> > will
>> >>> >>> >> > >>>>>>> be
>> >>> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
>> >>> >>> considering
>> >>> >>> >> few
>> >>> >>> >> > >>>>>>> more
>> >>> >>> >> > >>>>>>> > offline requests from other folks, I think
>> extending
>> >>> the
>> >>> >>> >> > timeframe
>> >>> >>> >> > >>>>>>> by
>> >>> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
>> >>> >>> buffer) and
>> >>> >>> >> > >>>>>>> this should
>> >>> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap
>> >>> wiki as
>> >>> >>> >> well)
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by
>> >>> August
>> >>> >>> 21,
>> >>> >>> >> 2018.
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>> >>> >>> improvements
>> >>> >>> >> and
>> >>> >>> >> > >>>>>>> non
>> >>> >>> >> > >>>>>>> > blocker/critical
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > - Release date: September 15, 2018
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already
>> >>> answered
>> >>> >>> your
>> >>> >>> >> > >>>>>>> questions.
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > Thanks
>> >>> >>> >> > >>>>>>> > Sunil
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>> >>> >>> >> wheeleast@gmail.com<ma...@gmail.com>
>> >>> >>> >> > >
>> >>> >>> >> > >>>>>>> wrote:
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2
>> >>> release,
>> >>> >>> +1
>> >>> >>> >> for
>> >>> >>> >> > >>>>>>> that.
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > To concerns from Steve,
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
>> >>> important
>> >>> >>> >> > changes
>> >>> >>> >> > >>>>>>> /
>> >>> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep
>> >>> the
>> >>> >>> >> proposed
>> >>> >>> >> > >>>>>>> release
>> >>> >>> >> > >>>>>>> > date
>> >>> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of
>> >>> last
>> >>> >>> >> minute
>> >>> >>> >> > >>>>>>> and we all
>> >>> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm
>> >>> also
>> >>> >>> fine
>> >>> >>> >> if
>> >>> >>> >> > >>>>>>> we want
>> >>> >>> >> > >>>>>>> > get
>> >>> >>> >> > >>>>>>> > > another several weeks time.
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing
>> that
>> >>> >>> before
>> >>> >>> >> > >>>>>>> thanksgiving.
>> >>> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > Eric,
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
>> >>> >>> YARN-8135
>> >>> >>> >> can
>> >>> >>> >> > >>>>>>> get merged
>> >>> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > Sunil,
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > Could u update
>> >>> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>> >>> >>> >> > >>>>>>> > Roadmap
>> >>> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature
>> list
>> >>> >>> first
>> >>> >>> >> > >>>>>>> before getting
>> >>> >>> >> > >>>>>>> > > consensus of time.
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > Thanks,
>> >>> >>> >> > >>>>>>> > > Wangda
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>> >>> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid<mailto:
>> >>> fabbri@cloudera.com.invalid>
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > wrote:
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>> >>> >>> >> > >>>>>>> stevel@hortonworks.com<mailto:stevel@hortonworks.com
>> >>
>> >>> >>> >> > >>>>>>> > >> wrote:
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
>> >>> sunilg@apache.org<ma...@apache.org>
>> >>> >>> >> > <mailto:
>> >>> >>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>>
>> >>> wrote:
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > I would also would like to take this
>> opportunity
>> >>> to
>> >>> >>> come
>> >>> >>> >> up
>> >>> >>> >> > >>>>>>> with a
>> >>> >>> >> > >>>>>>> > >> detailed
>> >>> >>> >> > >>>>>>> > >> > plan.
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
>> >>> >>> merged by
>> >>> >>> >> > >>>>>>> August 10,
>> >>> >>> >> > >>>>>>> > >> 2018.
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > <snip>
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
>> >>> targeted
>> >>> >>> to
>> >>> >>> >> 3.2
>> >>> >>> >> > >>>>>>> per this
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 &
>> S3Guard.
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> https://issues.apache.org/jira/browse/HADOOP-15226
>> >>> >>> >> > >>>>>>> > >> >
>> >>> https://issues.apache.org/jira/browse/HADOOP-15220
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for
>> Azure
>> >>> >>> Datalake
>> >>> >>> >> > Gen
>> >>> >>> >> > >>>>>>> 2
>> >>> >>> >> > >>>>>>> > >> >
>> >>> https://issues.apache.org/jira/browse/HADOOP-15407
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've
>> been
>> >>> >>> doing
>> >>> >>> >> work
>> >>> >>> >> > >>>>>>> on it, but
>> >>> >>> >> > >>>>>>> > >> if
>> >>> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect
>> >>> some
>> >>> >>> >> timely
>> >>> >>> >> > >>>>>>> reviews on
>> >>> >>> >> > >>>>>>> > >> >
>> >>> https://issues.apache.org/jira/browse/HADOOP-15546
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will
>> >>> review
>> >>> >>> the
>> >>> >>> >> S3 &
>> >>> >>> >> > >>>>>>> S3Guard
>> >>> >>> >> > >>>>>>> > >> JIRAs
>> >>> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there
>> >>> are
>> >>> >>> some
>> >>> >>> >> > >>>>>>> pressing
>> >>> >>> >> > >>>>>>> > "guave,
>> >>> >>> >> > >>>>>>> > >> > java 9 prep"
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> I can help with this part if you like.
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
>> >>> release
>> >>> >>> >> > manager
>> >>> >>> >> > >>>>>>> of 3.2.0
>> >>> >>> >> > >>>>>>> > >> > release.
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > well volunteered!
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable
>> >>> should
>> >>> >>> we
>> >>> >>> >> have
>> >>> >>> >> > >>>>>>> for the
>> >>> >>> >> > >>>>>>> > >> 3.2. &
>> >>> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence,
>> >>> then
>> >>> >>> having
>> >>> >>> >> > the
>> >>> >>> >> > >>>>>>> outline
>> >>> >>> >> > >>>>>>> > >> time
>> >>> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that
>> >>> there's
>> >>> >>> less
>> >>> >>> >> > >>>>>>> concern about
>> >>> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > -Steve
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >> -AF
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>
>> >>> >>> >> >
>> >>> >>> >>
>> >>> >>> >
>> >>> >>>
>> >>> >>
>> >>> >>
>> >>>
>> >>>
>> >>> --
>> >>> A very happy Clouderan
>> >>>
>> >>>
>>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi Folks,
Given all blockers are closed now, I am planning to do RC1 tomorrow.
branch-3.2.0 remains closed for commit. Please let me know if any issues
are missed.
Thanks,
Sunil G
On Tue, Nov 13, 2018 at 8:11 PM Sunil G <su...@apache.org> wrote:
> Hi Folks,
>
> All blockers are closed by last weekend and corrected jiras as well.
> Preparing RC0 now. Facing some issue with shaded jars file size. Hence
> respinning again.
> Planning to complete this by end of this week.
>
> Thanks
> Sunil
>
> On Fri, Oct 26, 2018 at 2:34 AM Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
>> Another thing is that I see a bunch of jiras under HDFS-8707, which don't
>> have the Fix Version field listing 3.2, some have it just empty.
>> This means they will not be populated into release notes.
>>
>> Thanks,
>> --Konstantin
>>
>> On Thu, Oct 25, 2018 at 7:59 AM Sunil G <su...@apache.org> wrote:
>>
>> > Thanks Konstantin for pointing out.
>> > As 3.2 is pretty much on RC level, its better we try to find a good
>> > solution to this issue.
>> >
>> > I ll follow up on this in the jira.
>> >
>> > - Sunil
>> >
>> > On Thu, Oct 25, 2018 at 11:35 AM Konstantin Shvachko <
>> shv.hadoop@gmail.com>
>> > wrote:
>> >
>> >> I've tried to attract attention to an incompatibility issue through the
>> >> jira, but it didn't work. So pitching in in this thread.
>> >> https://issues.apache.org/jira/browse/HDFS-12026
>> >> It introduced binary incompatibility, which will prevent people from
>> >> upgrading from 3.1 to 3.2.
>> >> I think it can get messy if we release anything with this feature.
>> >>
>> >> Thanks,
>> >> --Konstantin
>> >>
>> >> On Mon, Oct 22, 2018 at 5:01 AM Steve Loughran <stevel@hortonworks.com
>> >
>> >> wrote:
>> >>
>> >>> its in.
>> >>>
>> >>> good catch!
>> >>>
>> >>> On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <weichiu@cloudera.com
>> <mailto:
>> >>> weichiu@cloudera.com>> wrote:
>> >>>
>> >>> Thanks Sunil G for driving the release,
>> >>> I filed HADOOP-15866<
>> https://issues.apache.org/jira/browse/HADOOP-15866>
>> >>> for a compat fix. If any one has cycle please review it, as I think
>> it is
>> >>> needed for 3.2.0.
>> >>>
>> >>> On Thu, Oct 18, 2018 at 4:43 AM Sunil G <sunilg@apache.org<mailto:
>> >>> sunilg@apache.org>> wrote:
>> >>> Hi Folks,
>> >>>
>> >>> As we previously communicated for 3.2.0 release, we have delayed due
>> to
>> >>> few
>> >>> blockers in our gate.
>> >>>
>> >>> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open
>> for
>> >>> all bug fixes.
>> >>>
>> >>> - Sunil
>> >>>
>> >>>
>> >>> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <sunilg@apache.org<mailto:
>> >>> sunilg@apache.org>> wrote:
>> >>>
>> >>> > Hi Folks,
>> >>> >
>> >>> > We are now close to RC as other blocker issues are now merged to
>> trunk
>> >>> and
>> >>> > branch-3.2. Last 2 critical issues are closer to merge and will be
>> >>> > committed in few hours.
>> >>> > With this, I will be creating 3.2.0 branch today and will go ahead
>> >>> with RC
>> >>> > related process.
>> >>> >
>> >>> > - Sunil
>> >>> >
>> >>> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <
>> jonbender@stripe.com
>> >>> <ma...@stripe.com>>
>> >>> > wrote:
>> >>> >
>> >>> >> Hello, were there any updates around the 3.2.0 RC timing? All I
>> see in
>> >>> >> the current blockers are related to the new Submarine subproject,
>> >>> wasn't
>> >>> >> sure if that is what is holding things up.
>> >>> >>
>> >>> >> Cheers,
>> >>> >> Jon
>> >>> >>
>> >>> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <sunilg@apache.org<mailto:
>> >>> sunilg@apache.org>> wrote:
>> >>> >>
>> >>> >>> Thanks Robert and Haibo for quickly correcting same.
>> >>> >>> Sigh, I somehow missed one file while committing the change. Sorry
>> >>> for
>> >>> >>> the
>> >>> >>> trouble.
>> >>> >>>
>> >>> >>> - Sunil
>> >>> >>>
>> >>> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <
>> rkanter@cloudera.com
>> >>> <ma...@cloudera.com>>
>> >>> >>> wrote:
>> >>> >>>
>> >>> >>> > Looks like there's two that weren't updated:
>> >>> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT"
>> .
>> >>> -r
>> >>> >>> > --include=pom.xml
>> >>> >>> > ./hadoop-project/pom.xml:
>> >>> >>> >
>> >>> <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>> >>> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>> >>> >>> >
>> >>> >>> > I've just pushed in an addendum commit to fix those.
>> >>> >>> > In the future, please make sure to do a sanity compile when
>> >>> updating
>> >>> >>> poms.
>> >>> >>> >
>> >>> >>> > thanks
>> >>> >>> > - Robert
>> >>> >>> >
>> >>> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>> >>> >>> <fa...@cloudera.com.invalid>>
>> >>> >>> > wrote:
>> >>> >>> >
>> >>> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in
>> >>> the
>> >>> >>> >> top-level pom.xml?
>> >>> >>> >>
>> >>> >>> >>
>> >>> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <sunilg@apache.org
>> >>> <ma...@apache.org>> wrote:
>> >>> >>> >>
>> >>> >>> >> > Hi All
>> >>> >>> >> >
>> >>> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
>> >>> trunk
>> >>> >>> to
>> >>> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
>> >>> >>> necessary
>> >>> >>> >> > patches are pulled into branch-3.2.
>> >>> >>> >> >
>> >>> >>> >> > Thank You
>> >>> >>> >> > - Sunil
>> >>> >>> >> >
>> >>> >>> >> >
>> >>> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <sunilg@apache.org
>> >>> <ma...@apache.org>> wrote:
>> >>> >>> >> >
>> >>> >>> >> > > Hi All
>> >>> >>> >> > >
>> >>> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is
>> >>> merged to
>> >>> >>> >> trunk.
>> >>> >>> >> > > Thanks for the support.
>> >>> >>> >> > >
>> >>> >>> >> > > *Plan for RC*
>> >>> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>> >>> >>> >> > > RC0 for 3.2: *28th Friday*
>> >>> >>> >> > >
>> >>> >>> >> > > Thank You
>> >>> >>> >> > > Sunil
>> >>> >>> >> > >
>> >>> >>> >> > >
>> >>> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <sunilg@apache.org
>> >>> <ma...@apache.org>>
>> >>> >>> wrote:
>> >>> >>> >> > >
>> >>> >>> >> > >> Hi All
>> >>> >>> >> > >>
>> >>> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all
>> of
>> >>> you
>> >>> >>> for
>> >>> >>> >> > >> helping in this. I am following up on these tickets, once
>> its
>> >>> >>> closed
>> >>> >>> >> we
>> >>> >>> >> > >> will cut the 3.2 branch.
>> >>> >>> >> > >>
>> >>> >>> >> > >> Thanks
>> >>> >>> >> > >> Sunil Govindan
>> >>> >>> >> > >>
>> >>> >>> >> > >>
>> >>> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <
>> sunilg@apache.org
>> >>> <ma...@apache.org>>
>> >>> >>> wrote:
>> >>> >>> >> > >>
>> >>> >>> >> > >>> Hi All,
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> Inline with the original 3.2 communication proposal dated
>> >>> 17th
>> >>> >>> July
>> >>> >>> >> > >>> 2018, I would like to provide more updates.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> We are approaching previously proposed code freeze date
>> >>> >>> (September
>> >>> >>> >> 14,
>> >>> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
>> >>> point
>> >>> >>> >> > existing
>> >>> >>> >> > >>> trunk to 3.3 if there are no issues.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> *Current Release Plan:*
>> >>> >>> >> > >>> Feature freeze date : all features to merge by September
>> 7,
>> >>> >>> 2018.
>> >>> >>> >> > >>> Code freeze date : blockers/critical only, no
>> improvements
>> >>> and
>> >>> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>> >>> >>> >> > >>> Release date: September 28, 2018
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> If any critical/blocker tickets which are targeted to
>> >>> 3.2.0, we
>> >>> >>> >> need to
>> >>> >>> >> > >>> backport to 3.2 post branch cut.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> Here's an updated 3.2.0 feature status:
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> 1. Merged & Completed features:
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> >>> DeepLearning
>> >>> >>> >> > >>> workloads Initial cut.
>> >>> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> >>> Capacity
>> >>> >>> >> > >>> Scheduler.
>> >>> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>> YARN
>> >>> >>> Service
>> >>> >>> >> > API
>> >>> >>> >> > >>> and CLI.
>> >>> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in
>> YARN.
>> >>> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
>> >>> Improvement
>> >>> >>> >> works.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> 2. Features close to finish:
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>> >>> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
>> >>> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>> >>> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>> >>> apps
>> >>> >>> from
>> >>> >>> >> > >>> ATSv2. Patch in progress.
>> >>> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >>> >>> >> challenging to
>> >>> >>> >> > >>> be done before Aug 2018.
>> >>> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN
>> >>> applications.
>> >>> >>> >> > >>> Challenging as more discussions are on-going.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> *Summary of 3.2.0 issues status:*
>> >>> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am
>> following
>> >>> up
>> >>> >>> with
>> >>> >>> >> > >>> owners to get status on each of them to get in by Code
>> >>> Freeze
>> >>> >>> date.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>> priority
>> >>> in
>> >>> >>> >> > (Blocker,
>> >>> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target
>> >>> Version/s" =
>> >>> >>> 3.2.0
>> >>> >>> >> > ORDER
>> >>> >>> >> > >>> BY priority DESC
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> Thanks,
>> >>> >>> >> > >>> Sunil
>> >>> >>> >> > >>>
>> >>> >>> >> > >>>
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <
>> sunilg@apache.org
>> >>> <ma...@apache.org>>
>> >>> >>> wrote:
>> >>> >>> >> > >>>
>> >>> >>> >> > >>>> Hi All,
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> Inline with earlier communication dated 17th July 2018,
>> I
>> >>> would
>> >>> >>> >> like
>> >>> >>> >> > to
>> >>> >>> >> > >>>> provide some updates.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> We are approaching previously proposed code freeze date
>> >>> (Aug
>> >>> >>> 31).
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> One of the critical feature Node Attributes feature
>> merge
>> >>> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs
>> >>> need a
>> >>> >>> bit
>> >>> >>> >> > more
>> >>> >>> >> > >>>> time. With regard to this, suggesting to push the
>> >>> feature/code
>> >>> >>> >> freeze
>> >>> >>> >> > for 2
>> >>> >>> >> > >>>> more weeks to accommodate these jiras too.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> Proposing Updated changes in plan inline with this:
>> >>> >>> >> > >>>> Feature freeze date : all features to merge by
>> September 7,
>> >>> >>> 2018.
>> >>> >>> >> > >>>> Code freeze date : blockers/critical only, no
>> improvements
>> >>> and
>> >>> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>> >>> >>> >> > >>>> Release date: September 28, 2018
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> If any features in branch which are targeted to 3.2.0,
>> >>> please
>> >>> >>> >> reply to
>> >>> >>> >> > >>>> this email thread.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> 1. Merged & Completed features:
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> >>> DeepLearning
>> >>> >>> >> > >>>> workloads Initial cut.
>> >>> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> >>> Capacity
>> >>> >>> >> > >>>> Scheduler.
>> >>> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>> >>> YARN
>> >>> >>> >> Service
>> >>> >>> >> > >>>> API and CLI.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> 2. Features close to finish:
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in
>> YARN.
>> >>> >>> >> Merge/Vote
>> >>> >>> >> > >>>> Ongoing.
>> >>> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of
>> historical
>> >>> apps
>> >>> >>> from
>> >>> >>> >> > >>>> ATSv2. Patch in progress.
>> >>> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> >>> >>> Improvement
>> >>> >>> >> > works.
>> >>> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support
>> Windows
>> >>> Azure
>> >>> >>> >> > >>>> Storage. In progress.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> 3. Tentative features:
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >>> >>> >> challenging
>> >>> >>> >> > to
>> >>> >>> >> > >>>> be done before Aug 2018.
>> >>> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
>> >>> applications.
>> >>> >>> >> > >>>> Challenging as more discussions are on-going.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> *Summary of 3.2.0 issues status:*
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am
>> >>> following up
>> >>> >>> with
>> >>> >>> >> > >>>> owners to get status on each of them to get in by Code
>> >>> Freeze
>> >>> >>> date.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>> >>> priority in
>> >>> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND
>> "Target
>> >>> >>> >> > Version/s" =
>> >>> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> Thanks,
>> >>> >>> >> > >>>> Sunil
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <
>> >>> sunilg@apache.org<ma...@apache.org>>
>> >>> >>> >> wrote:
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>>> Hi All,
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier
>> >>> communication
>> >>> >>> dated
>> >>> >>> >> > 17th
>> >>> >>> >> > >>>>> July 2018, I would like to provide some updates.
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> We are approaching previously proposed feature freeze
>> date
>> >>> >>> (Aug
>> >>> >>> >> 21,
>> >>> >>> >> > >>>>> about 7 days from today).
>> >>> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
>> >>> please
>> >>> >>> >> reply
>> >>> >>> >> > to
>> >>> >>> >> > >>>>> this email thread.
>> >>> >>> >> > >>>>> Steve has mentioned about the s3 features which will
>> come
>> >>> >>> close to
>> >>> >>> >> > >>>>> Code Freeze Date (Aug 31st).
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> 1. Merged & Completed features:
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> >>> >>> DeepLearning
>> >>> >>> >> > >>>>> workloads Initial cut.
>> >>> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> 2. Features close to finish:
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in
>> YARN.
>> >>> >>> Major
>> >>> >>> >> > >>>>> patches are all in, only one last
>> >>> >>> >> > >>>>> patch is in review state.
>> >>> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> >>> Capacity
>> >>> >>> >> > >>>>> Scheduler. Close to commit.
>> >>> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>> >>> YARN
>> >>> >>> >> Service
>> >>> >>> >> > >>>>> API and CLI. 2 patches are pending
>> >>> >>> >> > >>>>> which will be closed by Feature freeze date.
>> >>> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of
>> historical
>> >>> apps
>> >>> >>> >> from
>> >>> >>> >> > >>>>> ATSv2. Patch in progress.
>> >>> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> >>> >>> Improvement
>> >>> >>> >> > >>>>> works.
>> >>> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support
>> Windows
>> >>> >>> Azure
>> >>> >>> >> > >>>>> Storage. In progress.
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> 3. Tentative features:
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment.
>> Looks
>> >>> >>> >> challenging
>> >>> >>> >> > >>>>> to be done before Aug 2018.
>> >>> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
>> >>> applications.
>> >>> >>> >> > >>>>> Challenging as more discussions are on-going.
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am
>> checking
>> >>> >>> with
>> >>> >>> >> > owners
>> >>> >>> >> > >>>>> to get status on each of them to get in by Code Freeze
>> >>> date.
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>> >>> priority in
>> >>> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND
>> >>> "Target
>> >>> >>> >> > Version/s" =
>> >>> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> Thanks,
>> >>> >>> >> > >>>>> Sunil
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <
>> >>> sunilg@apache.org<ma...@apache.org>>
>> >>> >>> >> wrote:
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>>> Thanks Subru for the thoughts.
>> >>> >>> >> > >>>>>> One of the main reason for a major release is to push
>> out
>> >>> >>> >> critical
>> >>> >>> >> > >>>>>> features with a faster cadence to the users. If we are
>> >>> >>> pulling
>> >>> >>> >> more
>> >>> >>> >> > and
>> >>> >>> >> > >>>>>> more different types of features to a minor release,
>> that
>> >>> >>> branch
>> >>> >>> >> > will
>> >>> >>> >> > >>>>>> become more destabilized and it may be tough to say
>> that
>> >>> >>> 3.1.2 is
>> >>> >>> >> > stable
>> >>> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and
>> >>> stabilize
>> >>> >>> >> features
>> >>> >>> >> > in
>> >>> >>> >> > >>>>>> subsequent minor release.
>> >>> >>> >> > >>>>>> For few companies, it makes sense to push out these
>> new
>> >>> >>> features
>> >>> >>> >> > >>>>>> faster to make a reach to the users. Adding to the
>> point
>> >>> to
>> >>> >>> the
>> >>> >>> >> > backporting
>> >>> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround
>> >>> that
>> >>> >>> with
>> >>> >>> >> > some git
>> >>> >>> >> > >>>>>> scripts. If we can make such scripts available to
>> >>> committers,
>> >>> >>> >> > backport will
>> >>> >>> >> > >>>>>> be seem-less across branches and we can achieve the
>> >>> faster
>> >>> >>> >> release
>> >>> >>> >> > cadence
>> >>> >>> >> > >>>>>> also.
>> >>> >>> >> > >>>>>>
>> >>> >>> >> > >>>>>> Thoughts?
>> >>> >>> >> > >>>>>>
>> >>> >>> >> > >>>>>> - Sunil
>> >>> >>> >> > >>>>>>
>> >>> >>> >> > >>>>>>
>> >>> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>> >>> >>> subru@apache.org<ma...@apache.org>
>> >>> >>> >> >
>> >>> >>> >> > >>>>>> wrote:
>> >>> >>> >> > >>>>>>
>> >>> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release
>> >>> effort. I
>> >>> >>> am
>> >>> >>> >> > >>>>>>> generally
>> >>> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a
>> >>> 3.1.x) as
>> >>> >>> >> feel we
>> >>> >>> >> > >>>>>>> already
>> >>> >>> >> > >>>>>>> have too many branches to be maintained. I already
>> see
>> >>> many
>> >>> >>> >> commits
>> >>> >>> >> > >>>>>>> are in
>> >>> >>> >> > >>>>>>> different branches with no apparent rationale, for
>> e.g:
>> >>> 3.1
>> >>> >>> has
>> >>> >>> >> > >>>>>>> commits
>> >>> >>> >> > >>>>>>> which are absent in 3.0 etc.
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any
>> >>> major
>> >>> >>> >> > production
>> >>> >>> >> > >>>>>>> setting so the cost of adding features should be
>> >>> minimal.
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>> Thoughts?
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>> -Subru
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
>> >>> >>> sunilg@apache.org<ma...@apache.org>>
>> >>> >>> >> > wrote:
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > Yes, important changes and features are much
>> needed,
>> >>> >>> hence we
>> >>> >>> >> > will
>> >>> >>> >> > >>>>>>> be
>> >>> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
>> >>> >>> considering
>> >>> >>> >> few
>> >>> >>> >> > >>>>>>> more
>> >>> >>> >> > >>>>>>> > offline requests from other folks, I think
>> extending
>> >>> the
>> >>> >>> >> > timeframe
>> >>> >>> >> > >>>>>>> by
>> >>> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
>> >>> >>> buffer) and
>> >>> >>> >> > >>>>>>> this should
>> >>> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap
>> >>> wiki as
>> >>> >>> >> well)
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by
>> >>> August
>> >>> >>> 21,
>> >>> >>> >> 2018.
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>> >>> >>> improvements
>> >>> >>> >> and
>> >>> >>> >> > >>>>>>> non
>> >>> >>> >> > >>>>>>> > blocker/critical
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > - Release date: September 15, 2018
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already
>> >>> answered
>> >>> >>> your
>> >>> >>> >> > >>>>>>> questions.
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > Thanks
>> >>> >>> >> > >>>>>>> > Sunil
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>> >>> >>> >> wheeleast@gmail.com<ma...@gmail.com>
>> >>> >>> >> > >
>> >>> >>> >> > >>>>>>> wrote:
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2
>> >>> release,
>> >>> >>> +1
>> >>> >>> >> for
>> >>> >>> >> > >>>>>>> that.
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > To concerns from Steve,
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
>> >>> important
>> >>> >>> >> > changes
>> >>> >>> >> > >>>>>>> /
>> >>> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep
>> >>> the
>> >>> >>> >> proposed
>> >>> >>> >> > >>>>>>> release
>> >>> >>> >> > >>>>>>> > date
>> >>> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of
>> >>> last
>> >>> >>> >> minute
>> >>> >>> >> > >>>>>>> and we all
>> >>> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm
>> >>> also
>> >>> >>> fine
>> >>> >>> >> if
>> >>> >>> >> > >>>>>>> we want
>> >>> >>> >> > >>>>>>> > get
>> >>> >>> >> > >>>>>>> > > another several weeks time.
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing
>> that
>> >>> >>> before
>> >>> >>> >> > >>>>>>> thanksgiving.
>> >>> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > Eric,
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
>> >>> >>> YARN-8135
>> >>> >>> >> can
>> >>> >>> >> > >>>>>>> get merged
>> >>> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > Sunil,
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > Could u update
>> >>> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>> >>> >>> >> > >>>>>>> > Roadmap
>> >>> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature
>> list
>> >>> >>> first
>> >>> >>> >> > >>>>>>> before getting
>> >>> >>> >> > >>>>>>> > > consensus of time.
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > Thanks,
>> >>> >>> >> > >>>>>>> > > Wangda
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>> >>> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid<mailto:
>> >>> fabbri@cloudera.com.invalid>
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > wrote:
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>> >>> >>> >> > >>>>>>> stevel@hortonworks.com<mailto:stevel@hortonworks.com
>> >>
>> >>> >>> >> > >>>>>>> > >> wrote:
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
>> >>> sunilg@apache.org<ma...@apache.org>
>> >>> >>> >> > <mailto:
>> >>> >>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>>
>> >>> wrote:
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > I would also would like to take this
>> opportunity
>> >>> to
>> >>> >>> come
>> >>> >>> >> up
>> >>> >>> >> > >>>>>>> with a
>> >>> >>> >> > >>>>>>> > >> detailed
>> >>> >>> >> > >>>>>>> > >> > plan.
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
>> >>> >>> merged by
>> >>> >>> >> > >>>>>>> August 10,
>> >>> >>> >> > >>>>>>> > >> 2018.
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > <snip>
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
>> >>> targeted
>> >>> >>> to
>> >>> >>> >> 3.2
>> >>> >>> >> > >>>>>>> per this
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 &
>> S3Guard.
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> https://issues.apache.org/jira/browse/HADOOP-15226
>> >>> >>> >> > >>>>>>> > >> >
>> >>> https://issues.apache.org/jira/browse/HADOOP-15220
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for
>> Azure
>> >>> >>> Datalake
>> >>> >>> >> > Gen
>> >>> >>> >> > >>>>>>> 2
>> >>> >>> >> > >>>>>>> > >> >
>> >>> https://issues.apache.org/jira/browse/HADOOP-15407
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've
>> been
>> >>> >>> doing
>> >>> >>> >> work
>> >>> >>> >> > >>>>>>> on it, but
>> >>> >>> >> > >>>>>>> > >> if
>> >>> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect
>> >>> some
>> >>> >>> >> timely
>> >>> >>> >> > >>>>>>> reviews on
>> >>> >>> >> > >>>>>>> > >> >
>> >>> https://issues.apache.org/jira/browse/HADOOP-15546
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will
>> >>> review
>> >>> >>> the
>> >>> >>> >> S3 &
>> >>> >>> >> > >>>>>>> S3Guard
>> >>> >>> >> > >>>>>>> > >> JIRAs
>> >>> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there
>> >>> are
>> >>> >>> some
>> >>> >>> >> > >>>>>>> pressing
>> >>> >>> >> > >>>>>>> > "guave,
>> >>> >>> >> > >>>>>>> > >> > java 9 prep"
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> I can help with this part if you like.
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
>> >>> release
>> >>> >>> >> > manager
>> >>> >>> >> > >>>>>>> of 3.2.0
>> >>> >>> >> > >>>>>>> > >> > release.
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > well volunteered!
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable
>> >>> should
>> >>> >>> we
>> >>> >>> >> have
>> >>> >>> >> > >>>>>>> for the
>> >>> >>> >> > >>>>>>> > >> 3.2. &
>> >>> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence,
>> >>> then
>> >>> >>> having
>> >>> >>> >> > the
>> >>> >>> >> > >>>>>>> outline
>> >>> >>> >> > >>>>>>> > >> time
>> >>> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that
>> >>> there's
>> >>> >>> less
>> >>> >>> >> > >>>>>>> concern about
>> >>> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > -Steve
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >> -AF
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>
>> >>> >>> >> >
>> >>> >>> >>
>> >>> >>> >
>> >>> >>>
>> >>> >>
>> >>> >>
>> >>>
>> >>>
>> >>> --
>> >>> A very happy Clouderan
>> >>>
>> >>>
>>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi Folks,
Given all blockers are closed now, I am planning to do RC1 tomorrow.
branch-3.2.0 remains closed for commit. Please let me know if any issues
are missed.
Thanks,
Sunil G
On Tue, Nov 13, 2018 at 8:11 PM Sunil G <su...@apache.org> wrote:
> Hi Folks,
>
> All blockers are closed by last weekend and corrected jiras as well.
> Preparing RC0 now. Facing some issue with shaded jars file size. Hence
> respinning again.
> Planning to complete this by end of this week.
>
> Thanks
> Sunil
>
> On Fri, Oct 26, 2018 at 2:34 AM Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
>> Another thing is that I see a bunch of jiras under HDFS-8707, which don't
>> have the Fix Version field listing 3.2, some have it just empty.
>> This means they will not be populated into release notes.
>>
>> Thanks,
>> --Konstantin
>>
>> On Thu, Oct 25, 2018 at 7:59 AM Sunil G <su...@apache.org> wrote:
>>
>> > Thanks Konstantin for pointing out.
>> > As 3.2 is pretty much on RC level, its better we try to find a good
>> > solution to this issue.
>> >
>> > I ll follow up on this in the jira.
>> >
>> > - Sunil
>> >
>> > On Thu, Oct 25, 2018 at 11:35 AM Konstantin Shvachko <
>> shv.hadoop@gmail.com>
>> > wrote:
>> >
>> >> I've tried to attract attention to an incompatibility issue through the
>> >> jira, but it didn't work. So pitching in in this thread.
>> >> https://issues.apache.org/jira/browse/HDFS-12026
>> >> It introduced binary incompatibility, which will prevent people from
>> >> upgrading from 3.1 to 3.2.
>> >> I think it can get messy if we release anything with this feature.
>> >>
>> >> Thanks,
>> >> --Konstantin
>> >>
>> >> On Mon, Oct 22, 2018 at 5:01 AM Steve Loughran <stevel@hortonworks.com
>> >
>> >> wrote:
>> >>
>> >>> its in.
>> >>>
>> >>> good catch!
>> >>>
>> >>> On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <weichiu@cloudera.com
>> <mailto:
>> >>> weichiu@cloudera.com>> wrote:
>> >>>
>> >>> Thanks Sunil G for driving the release,
>> >>> I filed HADOOP-15866<
>> https://issues.apache.org/jira/browse/HADOOP-15866>
>> >>> for a compat fix. If any one has cycle please review it, as I think
>> it is
>> >>> needed for 3.2.0.
>> >>>
>> >>> On Thu, Oct 18, 2018 at 4:43 AM Sunil G <sunilg@apache.org<mailto:
>> >>> sunilg@apache.org>> wrote:
>> >>> Hi Folks,
>> >>>
>> >>> As we previously communicated for 3.2.0 release, we have delayed due
>> to
>> >>> few
>> >>> blockers in our gate.
>> >>>
>> >>> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open
>> for
>> >>> all bug fixes.
>> >>>
>> >>> - Sunil
>> >>>
>> >>>
>> >>> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <sunilg@apache.org<mailto:
>> >>> sunilg@apache.org>> wrote:
>> >>>
>> >>> > Hi Folks,
>> >>> >
>> >>> > We are now close to RC as other blocker issues are now merged to
>> trunk
>> >>> and
>> >>> > branch-3.2. Last 2 critical issues are closer to merge and will be
>> >>> > committed in few hours.
>> >>> > With this, I will be creating 3.2.0 branch today and will go ahead
>> >>> with RC
>> >>> > related process.
>> >>> >
>> >>> > - Sunil
>> >>> >
>> >>> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <
>> jonbender@stripe.com
>> >>> <ma...@stripe.com>>
>> >>> > wrote:
>> >>> >
>> >>> >> Hello, were there any updates around the 3.2.0 RC timing? All I
>> see in
>> >>> >> the current blockers are related to the new Submarine subproject,
>> >>> wasn't
>> >>> >> sure if that is what is holding things up.
>> >>> >>
>> >>> >> Cheers,
>> >>> >> Jon
>> >>> >>
>> >>> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <sunilg@apache.org<mailto:
>> >>> sunilg@apache.org>> wrote:
>> >>> >>
>> >>> >>> Thanks Robert and Haibo for quickly correcting same.
>> >>> >>> Sigh, I somehow missed one file while committing the change. Sorry
>> >>> for
>> >>> >>> the
>> >>> >>> trouble.
>> >>> >>>
>> >>> >>> - Sunil
>> >>> >>>
>> >>> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <
>> rkanter@cloudera.com
>> >>> <ma...@cloudera.com>>
>> >>> >>> wrote:
>> >>> >>>
>> >>> >>> > Looks like there's two that weren't updated:
>> >>> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT"
>> .
>> >>> -r
>> >>> >>> > --include=pom.xml
>> >>> >>> > ./hadoop-project/pom.xml:
>> >>> >>> >
>> >>> <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>> >>> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>> >>> >>> >
>> >>> >>> > I've just pushed in an addendum commit to fix those.
>> >>> >>> > In the future, please make sure to do a sanity compile when
>> >>> updating
>> >>> >>> poms.
>> >>> >>> >
>> >>> >>> > thanks
>> >>> >>> > - Robert
>> >>> >>> >
>> >>> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>> >>> >>> <fa...@cloudera.com.invalid>>
>> >>> >>> > wrote:
>> >>> >>> >
>> >>> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in
>> >>> the
>> >>> >>> >> top-level pom.xml?
>> >>> >>> >>
>> >>> >>> >>
>> >>> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <sunilg@apache.org
>> >>> <ma...@apache.org>> wrote:
>> >>> >>> >>
>> >>> >>> >> > Hi All
>> >>> >>> >> >
>> >>> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
>> >>> trunk
>> >>> >>> to
>> >>> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
>> >>> >>> necessary
>> >>> >>> >> > patches are pulled into branch-3.2.
>> >>> >>> >> >
>> >>> >>> >> > Thank You
>> >>> >>> >> > - Sunil
>> >>> >>> >> >
>> >>> >>> >> >
>> >>> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <sunilg@apache.org
>> >>> <ma...@apache.org>> wrote:
>> >>> >>> >> >
>> >>> >>> >> > > Hi All
>> >>> >>> >> > >
>> >>> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is
>> >>> merged to
>> >>> >>> >> trunk.
>> >>> >>> >> > > Thanks for the support.
>> >>> >>> >> > >
>> >>> >>> >> > > *Plan for RC*
>> >>> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>> >>> >>> >> > > RC0 for 3.2: *28th Friday*
>> >>> >>> >> > >
>> >>> >>> >> > > Thank You
>> >>> >>> >> > > Sunil
>> >>> >>> >> > >
>> >>> >>> >> > >
>> >>> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <sunilg@apache.org
>> >>> <ma...@apache.org>>
>> >>> >>> wrote:
>> >>> >>> >> > >
>> >>> >>> >> > >> Hi All
>> >>> >>> >> > >>
>> >>> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all
>> of
>> >>> you
>> >>> >>> for
>> >>> >>> >> > >> helping in this. I am following up on these tickets, once
>> its
>> >>> >>> closed
>> >>> >>> >> we
>> >>> >>> >> > >> will cut the 3.2 branch.
>> >>> >>> >> > >>
>> >>> >>> >> > >> Thanks
>> >>> >>> >> > >> Sunil Govindan
>> >>> >>> >> > >>
>> >>> >>> >> > >>
>> >>> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <
>> sunilg@apache.org
>> >>> <ma...@apache.org>>
>> >>> >>> wrote:
>> >>> >>> >> > >>
>> >>> >>> >> > >>> Hi All,
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> Inline with the original 3.2 communication proposal dated
>> >>> 17th
>> >>> >>> July
>> >>> >>> >> > >>> 2018, I would like to provide more updates.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> We are approaching previously proposed code freeze date
>> >>> >>> (September
>> >>> >>> >> 14,
>> >>> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
>> >>> point
>> >>> >>> >> > existing
>> >>> >>> >> > >>> trunk to 3.3 if there are no issues.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> *Current Release Plan:*
>> >>> >>> >> > >>> Feature freeze date : all features to merge by September
>> 7,
>> >>> >>> 2018.
>> >>> >>> >> > >>> Code freeze date : blockers/critical only, no
>> improvements
>> >>> and
>> >>> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>> >>> >>> >> > >>> Release date: September 28, 2018
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> If any critical/blocker tickets which are targeted to
>> >>> 3.2.0, we
>> >>> >>> >> need to
>> >>> >>> >> > >>> backport to 3.2 post branch cut.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> Here's an updated 3.2.0 feature status:
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> 1. Merged & Completed features:
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> >>> DeepLearning
>> >>> >>> >> > >>> workloads Initial cut.
>> >>> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> >>> Capacity
>> >>> >>> >> > >>> Scheduler.
>> >>> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>> YARN
>> >>> >>> Service
>> >>> >>> >> > API
>> >>> >>> >> > >>> and CLI.
>> >>> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in
>> YARN.
>> >>> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
>> >>> Improvement
>> >>> >>> >> works.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> 2. Features close to finish:
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>> >>> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
>> >>> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>> >>> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>> >>> apps
>> >>> >>> from
>> >>> >>> >> > >>> ATSv2. Patch in progress.
>> >>> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >>> >>> >> challenging to
>> >>> >>> >> > >>> be done before Aug 2018.
>> >>> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN
>> >>> applications.
>> >>> >>> >> > >>> Challenging as more discussions are on-going.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> *Summary of 3.2.0 issues status:*
>> >>> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am
>> following
>> >>> up
>> >>> >>> with
>> >>> >>> >> > >>> owners to get status on each of them to get in by Code
>> >>> Freeze
>> >>> >>> date.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>> priority
>> >>> in
>> >>> >>> >> > (Blocker,
>> >>> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target
>> >>> Version/s" =
>> >>> >>> 3.2.0
>> >>> >>> >> > ORDER
>> >>> >>> >> > >>> BY priority DESC
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> Thanks,
>> >>> >>> >> > >>> Sunil
>> >>> >>> >> > >>>
>> >>> >>> >> > >>>
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <
>> sunilg@apache.org
>> >>> <ma...@apache.org>>
>> >>> >>> wrote:
>> >>> >>> >> > >>>
>> >>> >>> >> > >>>> Hi All,
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> Inline with earlier communication dated 17th July 2018,
>> I
>> >>> would
>> >>> >>> >> like
>> >>> >>> >> > to
>> >>> >>> >> > >>>> provide some updates.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> We are approaching previously proposed code freeze date
>> >>> (Aug
>> >>> >>> 31).
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> One of the critical feature Node Attributes feature
>> merge
>> >>> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs
>> >>> need a
>> >>> >>> bit
>> >>> >>> >> > more
>> >>> >>> >> > >>>> time. With regard to this, suggesting to push the
>> >>> feature/code
>> >>> >>> >> freeze
>> >>> >>> >> > for 2
>> >>> >>> >> > >>>> more weeks to accommodate these jiras too.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> Proposing Updated changes in plan inline with this:
>> >>> >>> >> > >>>> Feature freeze date : all features to merge by
>> September 7,
>> >>> >>> 2018.
>> >>> >>> >> > >>>> Code freeze date : blockers/critical only, no
>> improvements
>> >>> and
>> >>> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>> >>> >>> >> > >>>> Release date: September 28, 2018
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> If any features in branch which are targeted to 3.2.0,
>> >>> please
>> >>> >>> >> reply to
>> >>> >>> >> > >>>> this email thread.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> 1. Merged & Completed features:
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> >>> DeepLearning
>> >>> >>> >> > >>>> workloads Initial cut.
>> >>> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> >>> Capacity
>> >>> >>> >> > >>>> Scheduler.
>> >>> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>> >>> YARN
>> >>> >>> >> Service
>> >>> >>> >> > >>>> API and CLI.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> 2. Features close to finish:
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in
>> YARN.
>> >>> >>> >> Merge/Vote
>> >>> >>> >> > >>>> Ongoing.
>> >>> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of
>> historical
>> >>> apps
>> >>> >>> from
>> >>> >>> >> > >>>> ATSv2. Patch in progress.
>> >>> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> >>> >>> Improvement
>> >>> >>> >> > works.
>> >>> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support
>> Windows
>> >>> Azure
>> >>> >>> >> > >>>> Storage. In progress.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> 3. Tentative features:
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >>> >>> >> challenging
>> >>> >>> >> > to
>> >>> >>> >> > >>>> be done before Aug 2018.
>> >>> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
>> >>> applications.
>> >>> >>> >> > >>>> Challenging as more discussions are on-going.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> *Summary of 3.2.0 issues status:*
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am
>> >>> following up
>> >>> >>> with
>> >>> >>> >> > >>>> owners to get status on each of them to get in by Code
>> >>> Freeze
>> >>> >>> date.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>> >>> priority in
>> >>> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND
>> "Target
>> >>> >>> >> > Version/s" =
>> >>> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> Thanks,
>> >>> >>> >> > >>>> Sunil
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <
>> >>> sunilg@apache.org<ma...@apache.org>>
>> >>> >>> >> wrote:
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>>> Hi All,
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier
>> >>> communication
>> >>> >>> dated
>> >>> >>> >> > 17th
>> >>> >>> >> > >>>>> July 2018, I would like to provide some updates.
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> We are approaching previously proposed feature freeze
>> date
>> >>> >>> (Aug
>> >>> >>> >> 21,
>> >>> >>> >> > >>>>> about 7 days from today).
>> >>> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
>> >>> please
>> >>> >>> >> reply
>> >>> >>> >> > to
>> >>> >>> >> > >>>>> this email thread.
>> >>> >>> >> > >>>>> Steve has mentioned about the s3 features which will
>> come
>> >>> >>> close to
>> >>> >>> >> > >>>>> Code Freeze Date (Aug 31st).
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> 1. Merged & Completed features:
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> >>> >>> DeepLearning
>> >>> >>> >> > >>>>> workloads Initial cut.
>> >>> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> 2. Features close to finish:
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in
>> YARN.
>> >>> >>> Major
>> >>> >>> >> > >>>>> patches are all in, only one last
>> >>> >>> >> > >>>>> patch is in review state.
>> >>> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> >>> Capacity
>> >>> >>> >> > >>>>> Scheduler. Close to commit.
>> >>> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>> >>> YARN
>> >>> >>> >> Service
>> >>> >>> >> > >>>>> API and CLI. 2 patches are pending
>> >>> >>> >> > >>>>> which will be closed by Feature freeze date.
>> >>> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of
>> historical
>> >>> apps
>> >>> >>> >> from
>> >>> >>> >> > >>>>> ATSv2. Patch in progress.
>> >>> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> >>> >>> Improvement
>> >>> >>> >> > >>>>> works.
>> >>> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support
>> Windows
>> >>> >>> Azure
>> >>> >>> >> > >>>>> Storage. In progress.
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> 3. Tentative features:
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment.
>> Looks
>> >>> >>> >> challenging
>> >>> >>> >> > >>>>> to be done before Aug 2018.
>> >>> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
>> >>> applications.
>> >>> >>> >> > >>>>> Challenging as more discussions are on-going.
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am
>> checking
>> >>> >>> with
>> >>> >>> >> > owners
>> >>> >>> >> > >>>>> to get status on each of them to get in by Code Freeze
>> >>> date.
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>> >>> priority in
>> >>> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND
>> >>> "Target
>> >>> >>> >> > Version/s" =
>> >>> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> Thanks,
>> >>> >>> >> > >>>>> Sunil
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <
>> >>> sunilg@apache.org<ma...@apache.org>>
>> >>> >>> >> wrote:
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>>> Thanks Subru for the thoughts.
>> >>> >>> >> > >>>>>> One of the main reason for a major release is to push
>> out
>> >>> >>> >> critical
>> >>> >>> >> > >>>>>> features with a faster cadence to the users. If we are
>> >>> >>> pulling
>> >>> >>> >> more
>> >>> >>> >> > and
>> >>> >>> >> > >>>>>> more different types of features to a minor release,
>> that
>> >>> >>> branch
>> >>> >>> >> > will
>> >>> >>> >> > >>>>>> become more destabilized and it may be tough to say
>> that
>> >>> >>> 3.1.2 is
>> >>> >>> >> > stable
>> >>> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and
>> >>> stabilize
>> >>> >>> >> features
>> >>> >>> >> > in
>> >>> >>> >> > >>>>>> subsequent minor release.
>> >>> >>> >> > >>>>>> For few companies, it makes sense to push out these
>> new
>> >>> >>> features
>> >>> >>> >> > >>>>>> faster to make a reach to the users. Adding to the
>> point
>> >>> to
>> >>> >>> the
>> >>> >>> >> > backporting
>> >>> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround
>> >>> that
>> >>> >>> with
>> >>> >>> >> > some git
>> >>> >>> >> > >>>>>> scripts. If we can make such scripts available to
>> >>> committers,
>> >>> >>> >> > backport will
>> >>> >>> >> > >>>>>> be seem-less across branches and we can achieve the
>> >>> faster
>> >>> >>> >> release
>> >>> >>> >> > cadence
>> >>> >>> >> > >>>>>> also.
>> >>> >>> >> > >>>>>>
>> >>> >>> >> > >>>>>> Thoughts?
>> >>> >>> >> > >>>>>>
>> >>> >>> >> > >>>>>> - Sunil
>> >>> >>> >> > >>>>>>
>> >>> >>> >> > >>>>>>
>> >>> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>> >>> >>> subru@apache.org<ma...@apache.org>
>> >>> >>> >> >
>> >>> >>> >> > >>>>>> wrote:
>> >>> >>> >> > >>>>>>
>> >>> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release
>> >>> effort. I
>> >>> >>> am
>> >>> >>> >> > >>>>>>> generally
>> >>> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a
>> >>> 3.1.x) as
>> >>> >>> >> feel we
>> >>> >>> >> > >>>>>>> already
>> >>> >>> >> > >>>>>>> have too many branches to be maintained. I already
>> see
>> >>> many
>> >>> >>> >> commits
>> >>> >>> >> > >>>>>>> are in
>> >>> >>> >> > >>>>>>> different branches with no apparent rationale, for
>> e.g:
>> >>> 3.1
>> >>> >>> has
>> >>> >>> >> > >>>>>>> commits
>> >>> >>> >> > >>>>>>> which are absent in 3.0 etc.
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any
>> >>> major
>> >>> >>> >> > production
>> >>> >>> >> > >>>>>>> setting so the cost of adding features should be
>> >>> minimal.
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>> Thoughts?
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>> -Subru
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
>> >>> >>> sunilg@apache.org<ma...@apache.org>>
>> >>> >>> >> > wrote:
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > Yes, important changes and features are much
>> needed,
>> >>> >>> hence we
>> >>> >>> >> > will
>> >>> >>> >> > >>>>>>> be
>> >>> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
>> >>> >>> considering
>> >>> >>> >> few
>> >>> >>> >> > >>>>>>> more
>> >>> >>> >> > >>>>>>> > offline requests from other folks, I think
>> extending
>> >>> the
>> >>> >>> >> > timeframe
>> >>> >>> >> > >>>>>>> by
>> >>> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
>> >>> >>> buffer) and
>> >>> >>> >> > >>>>>>> this should
>> >>> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap
>> >>> wiki as
>> >>> >>> >> well)
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by
>> >>> August
>> >>> >>> 21,
>> >>> >>> >> 2018.
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>> >>> >>> improvements
>> >>> >>> >> and
>> >>> >>> >> > >>>>>>> non
>> >>> >>> >> > >>>>>>> > blocker/critical
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > - Release date: September 15, 2018
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already
>> >>> answered
>> >>> >>> your
>> >>> >>> >> > >>>>>>> questions.
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > Thanks
>> >>> >>> >> > >>>>>>> > Sunil
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>> >>> >>> >> wheeleast@gmail.com<ma...@gmail.com>
>> >>> >>> >> > >
>> >>> >>> >> > >>>>>>> wrote:
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2
>> >>> release,
>> >>> >>> +1
>> >>> >>> >> for
>> >>> >>> >> > >>>>>>> that.
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > To concerns from Steve,
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
>> >>> important
>> >>> >>> >> > changes
>> >>> >>> >> > >>>>>>> /
>> >>> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep
>> >>> the
>> >>> >>> >> proposed
>> >>> >>> >> > >>>>>>> release
>> >>> >>> >> > >>>>>>> > date
>> >>> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of
>> >>> last
>> >>> >>> >> minute
>> >>> >>> >> > >>>>>>> and we all
>> >>> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm
>> >>> also
>> >>> >>> fine
>> >>> >>> >> if
>> >>> >>> >> > >>>>>>> we want
>> >>> >>> >> > >>>>>>> > get
>> >>> >>> >> > >>>>>>> > > another several weeks time.
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing
>> that
>> >>> >>> before
>> >>> >>> >> > >>>>>>> thanksgiving.
>> >>> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > Eric,
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
>> >>> >>> YARN-8135
>> >>> >>> >> can
>> >>> >>> >> > >>>>>>> get merged
>> >>> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > Sunil,
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > Could u update
>> >>> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>> >>> >>> >> > >>>>>>> > Roadmap
>> >>> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature
>> list
>> >>> >>> first
>> >>> >>> >> > >>>>>>> before getting
>> >>> >>> >> > >>>>>>> > > consensus of time.
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > Thanks,
>> >>> >>> >> > >>>>>>> > > Wangda
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>> >>> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid<mailto:
>> >>> fabbri@cloudera.com.invalid>
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > wrote:
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>> >>> >>> >> > >>>>>>> stevel@hortonworks.com<mailto:stevel@hortonworks.com
>> >>
>> >>> >>> >> > >>>>>>> > >> wrote:
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
>> >>> sunilg@apache.org<ma...@apache.org>
>> >>> >>> >> > <mailto:
>> >>> >>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>>
>> >>> wrote:
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > I would also would like to take this
>> opportunity
>> >>> to
>> >>> >>> come
>> >>> >>> >> up
>> >>> >>> >> > >>>>>>> with a
>> >>> >>> >> > >>>>>>> > >> detailed
>> >>> >>> >> > >>>>>>> > >> > plan.
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
>> >>> >>> merged by
>> >>> >>> >> > >>>>>>> August 10,
>> >>> >>> >> > >>>>>>> > >> 2018.
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > <snip>
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
>> >>> targeted
>> >>> >>> to
>> >>> >>> >> 3.2
>> >>> >>> >> > >>>>>>> per this
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 &
>> S3Guard.
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> https://issues.apache.org/jira/browse/HADOOP-15226
>> >>> >>> >> > >>>>>>> > >> >
>> >>> https://issues.apache.org/jira/browse/HADOOP-15220
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for
>> Azure
>> >>> >>> Datalake
>> >>> >>> >> > Gen
>> >>> >>> >> > >>>>>>> 2
>> >>> >>> >> > >>>>>>> > >> >
>> >>> https://issues.apache.org/jira/browse/HADOOP-15407
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've
>> been
>> >>> >>> doing
>> >>> >>> >> work
>> >>> >>> >> > >>>>>>> on it, but
>> >>> >>> >> > >>>>>>> > >> if
>> >>> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect
>> >>> some
>> >>> >>> >> timely
>> >>> >>> >> > >>>>>>> reviews on
>> >>> >>> >> > >>>>>>> > >> >
>> >>> https://issues.apache.org/jira/browse/HADOOP-15546
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will
>> >>> review
>> >>> >>> the
>> >>> >>> >> S3 &
>> >>> >>> >> > >>>>>>> S3Guard
>> >>> >>> >> > >>>>>>> > >> JIRAs
>> >>> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there
>> >>> are
>> >>> >>> some
>> >>> >>> >> > >>>>>>> pressing
>> >>> >>> >> > >>>>>>> > "guave,
>> >>> >>> >> > >>>>>>> > >> > java 9 prep"
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> I can help with this part if you like.
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
>> >>> release
>> >>> >>> >> > manager
>> >>> >>> >> > >>>>>>> of 3.2.0
>> >>> >>> >> > >>>>>>> > >> > release.
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > well volunteered!
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable
>> >>> should
>> >>> >>> we
>> >>> >>> >> have
>> >>> >>> >> > >>>>>>> for the
>> >>> >>> >> > >>>>>>> > >> 3.2. &
>> >>> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence,
>> >>> then
>> >>> >>> having
>> >>> >>> >> > the
>> >>> >>> >> > >>>>>>> outline
>> >>> >>> >> > >>>>>>> > >> time
>> >>> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that
>> >>> there's
>> >>> >>> less
>> >>> >>> >> > >>>>>>> concern about
>> >>> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > -Steve
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >> -AF
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>
>> >>> >>> >> >
>> >>> >>> >>
>> >>> >>> >
>> >>> >>>
>> >>> >>
>> >>> >>
>> >>>
>> >>>
>> >>> --
>> >>> A very happy Clouderan
>> >>>
>> >>>
>>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi Folks,
Given all blockers are closed now, I am planning to do RC1 tomorrow.
branch-3.2.0 remains closed for commit. Please let me know if any issues
are missed.
Thanks,
Sunil G
On Tue, Nov 13, 2018 at 8:11 PM Sunil G <su...@apache.org> wrote:
> Hi Folks,
>
> All blockers are closed by last weekend and corrected jiras as well.
> Preparing RC0 now. Facing some issue with shaded jars file size. Hence
> respinning again.
> Planning to complete this by end of this week.
>
> Thanks
> Sunil
>
> On Fri, Oct 26, 2018 at 2:34 AM Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
>> Another thing is that I see a bunch of jiras under HDFS-8707, which don't
>> have the Fix Version field listing 3.2, some have it just empty.
>> This means they will not be populated into release notes.
>>
>> Thanks,
>> --Konstantin
>>
>> On Thu, Oct 25, 2018 at 7:59 AM Sunil G <su...@apache.org> wrote:
>>
>> > Thanks Konstantin for pointing out.
>> > As 3.2 is pretty much on RC level, its better we try to find a good
>> > solution to this issue.
>> >
>> > I ll follow up on this in the jira.
>> >
>> > - Sunil
>> >
>> > On Thu, Oct 25, 2018 at 11:35 AM Konstantin Shvachko <
>> shv.hadoop@gmail.com>
>> > wrote:
>> >
>> >> I've tried to attract attention to an incompatibility issue through the
>> >> jira, but it didn't work. So pitching in in this thread.
>> >> https://issues.apache.org/jira/browse/HDFS-12026
>> >> It introduced binary incompatibility, which will prevent people from
>> >> upgrading from 3.1 to 3.2.
>> >> I think it can get messy if we release anything with this feature.
>> >>
>> >> Thanks,
>> >> --Konstantin
>> >>
>> >> On Mon, Oct 22, 2018 at 5:01 AM Steve Loughran <stevel@hortonworks.com
>> >
>> >> wrote:
>> >>
>> >>> its in.
>> >>>
>> >>> good catch!
>> >>>
>> >>> On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <weichiu@cloudera.com
>> <mailto:
>> >>> weichiu@cloudera.com>> wrote:
>> >>>
>> >>> Thanks Sunil G for driving the release,
>> >>> I filed HADOOP-15866<
>> https://issues.apache.org/jira/browse/HADOOP-15866>
>> >>> for a compat fix. If any one has cycle please review it, as I think
>> it is
>> >>> needed for 3.2.0.
>> >>>
>> >>> On Thu, Oct 18, 2018 at 4:43 AM Sunil G <sunilg@apache.org<mailto:
>> >>> sunilg@apache.org>> wrote:
>> >>> Hi Folks,
>> >>>
>> >>> As we previously communicated for 3.2.0 release, we have delayed due
>> to
>> >>> few
>> >>> blockers in our gate.
>> >>>
>> >>> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open
>> for
>> >>> all bug fixes.
>> >>>
>> >>> - Sunil
>> >>>
>> >>>
>> >>> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <sunilg@apache.org<mailto:
>> >>> sunilg@apache.org>> wrote:
>> >>>
>> >>> > Hi Folks,
>> >>> >
>> >>> > We are now close to RC as other blocker issues are now merged to
>> trunk
>> >>> and
>> >>> > branch-3.2. Last 2 critical issues are closer to merge and will be
>> >>> > committed in few hours.
>> >>> > With this, I will be creating 3.2.0 branch today and will go ahead
>> >>> with RC
>> >>> > related process.
>> >>> >
>> >>> > - Sunil
>> >>> >
>> >>> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <
>> jonbender@stripe.com
>> >>> <ma...@stripe.com>>
>> >>> > wrote:
>> >>> >
>> >>> >> Hello, were there any updates around the 3.2.0 RC timing? All I
>> see in
>> >>> >> the current blockers are related to the new Submarine subproject,
>> >>> wasn't
>> >>> >> sure if that is what is holding things up.
>> >>> >>
>> >>> >> Cheers,
>> >>> >> Jon
>> >>> >>
>> >>> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <sunilg@apache.org<mailto:
>> >>> sunilg@apache.org>> wrote:
>> >>> >>
>> >>> >>> Thanks Robert and Haibo for quickly correcting same.
>> >>> >>> Sigh, I somehow missed one file while committing the change. Sorry
>> >>> for
>> >>> >>> the
>> >>> >>> trouble.
>> >>> >>>
>> >>> >>> - Sunil
>> >>> >>>
>> >>> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <
>> rkanter@cloudera.com
>> >>> <ma...@cloudera.com>>
>> >>> >>> wrote:
>> >>> >>>
>> >>> >>> > Looks like there's two that weren't updated:
>> >>> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT"
>> .
>> >>> -r
>> >>> >>> > --include=pom.xml
>> >>> >>> > ./hadoop-project/pom.xml:
>> >>> >>> >
>> >>> <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>> >>> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>> >>> >>> >
>> >>> >>> > I've just pushed in an addendum commit to fix those.
>> >>> >>> > In the future, please make sure to do a sanity compile when
>> >>> updating
>> >>> >>> poms.
>> >>> >>> >
>> >>> >>> > thanks
>> >>> >>> > - Robert
>> >>> >>> >
>> >>> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>> >>> >>> <fa...@cloudera.com.invalid>>
>> >>> >>> > wrote:
>> >>> >>> >
>> >>> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in
>> >>> the
>> >>> >>> >> top-level pom.xml?
>> >>> >>> >>
>> >>> >>> >>
>> >>> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <sunilg@apache.org
>> >>> <ma...@apache.org>> wrote:
>> >>> >>> >>
>> >>> >>> >> > Hi All
>> >>> >>> >> >
>> >>> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
>> >>> trunk
>> >>> >>> to
>> >>> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
>> >>> >>> necessary
>> >>> >>> >> > patches are pulled into branch-3.2.
>> >>> >>> >> >
>> >>> >>> >> > Thank You
>> >>> >>> >> > - Sunil
>> >>> >>> >> >
>> >>> >>> >> >
>> >>> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <sunilg@apache.org
>> >>> <ma...@apache.org>> wrote:
>> >>> >>> >> >
>> >>> >>> >> > > Hi All
>> >>> >>> >> > >
>> >>> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is
>> >>> merged to
>> >>> >>> >> trunk.
>> >>> >>> >> > > Thanks for the support.
>> >>> >>> >> > >
>> >>> >>> >> > > *Plan for RC*
>> >>> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>> >>> >>> >> > > RC0 for 3.2: *28th Friday*
>> >>> >>> >> > >
>> >>> >>> >> > > Thank You
>> >>> >>> >> > > Sunil
>> >>> >>> >> > >
>> >>> >>> >> > >
>> >>> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <sunilg@apache.org
>> >>> <ma...@apache.org>>
>> >>> >>> wrote:
>> >>> >>> >> > >
>> >>> >>> >> > >> Hi All
>> >>> >>> >> > >>
>> >>> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all
>> of
>> >>> you
>> >>> >>> for
>> >>> >>> >> > >> helping in this. I am following up on these tickets, once
>> its
>> >>> >>> closed
>> >>> >>> >> we
>> >>> >>> >> > >> will cut the 3.2 branch.
>> >>> >>> >> > >>
>> >>> >>> >> > >> Thanks
>> >>> >>> >> > >> Sunil Govindan
>> >>> >>> >> > >>
>> >>> >>> >> > >>
>> >>> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <
>> sunilg@apache.org
>> >>> <ma...@apache.org>>
>> >>> >>> wrote:
>> >>> >>> >> > >>
>> >>> >>> >> > >>> Hi All,
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> Inline with the original 3.2 communication proposal dated
>> >>> 17th
>> >>> >>> July
>> >>> >>> >> > >>> 2018, I would like to provide more updates.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> We are approaching previously proposed code freeze date
>> >>> >>> (September
>> >>> >>> >> 14,
>> >>> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
>> >>> point
>> >>> >>> >> > existing
>> >>> >>> >> > >>> trunk to 3.3 if there are no issues.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> *Current Release Plan:*
>> >>> >>> >> > >>> Feature freeze date : all features to merge by September
>> 7,
>> >>> >>> 2018.
>> >>> >>> >> > >>> Code freeze date : blockers/critical only, no
>> improvements
>> >>> and
>> >>> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>> >>> >>> >> > >>> Release date: September 28, 2018
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> If any critical/blocker tickets which are targeted to
>> >>> 3.2.0, we
>> >>> >>> >> need to
>> >>> >>> >> > >>> backport to 3.2 post branch cut.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> Here's an updated 3.2.0 feature status:
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> 1. Merged & Completed features:
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> >>> DeepLearning
>> >>> >>> >> > >>> workloads Initial cut.
>> >>> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> >>> Capacity
>> >>> >>> >> > >>> Scheduler.
>> >>> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>> YARN
>> >>> >>> Service
>> >>> >>> >> > API
>> >>> >>> >> > >>> and CLI.
>> >>> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in
>> YARN.
>> >>> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
>> >>> Improvement
>> >>> >>> >> works.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> 2. Features close to finish:
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>> >>> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
>> >>> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>> >>> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>> >>> apps
>> >>> >>> from
>> >>> >>> >> > >>> ATSv2. Patch in progress.
>> >>> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >>> >>> >> challenging to
>> >>> >>> >> > >>> be done before Aug 2018.
>> >>> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN
>> >>> applications.
>> >>> >>> >> > >>> Challenging as more discussions are on-going.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> *Summary of 3.2.0 issues status:*
>> >>> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am
>> following
>> >>> up
>> >>> >>> with
>> >>> >>> >> > >>> owners to get status on each of them to get in by Code
>> >>> Freeze
>> >>> >>> date.
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>> priority
>> >>> in
>> >>> >>> >> > (Blocker,
>> >>> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target
>> >>> Version/s" =
>> >>> >>> 3.2.0
>> >>> >>> >> > ORDER
>> >>> >>> >> > >>> BY priority DESC
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> Thanks,
>> >>> >>> >> > >>> Sunil
>> >>> >>> >> > >>>
>> >>> >>> >> > >>>
>> >>> >>> >> > >>>
>> >>> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <
>> sunilg@apache.org
>> >>> <ma...@apache.org>>
>> >>> >>> wrote:
>> >>> >>> >> > >>>
>> >>> >>> >> > >>>> Hi All,
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> Inline with earlier communication dated 17th July 2018,
>> I
>> >>> would
>> >>> >>> >> like
>> >>> >>> >> > to
>> >>> >>> >> > >>>> provide some updates.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> We are approaching previously proposed code freeze date
>> >>> (Aug
>> >>> >>> 31).
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> One of the critical feature Node Attributes feature
>> merge
>> >>> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs
>> >>> need a
>> >>> >>> bit
>> >>> >>> >> > more
>> >>> >>> >> > >>>> time. With regard to this, suggesting to push the
>> >>> feature/code
>> >>> >>> >> freeze
>> >>> >>> >> > for 2
>> >>> >>> >> > >>>> more weeks to accommodate these jiras too.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> Proposing Updated changes in plan inline with this:
>> >>> >>> >> > >>>> Feature freeze date : all features to merge by
>> September 7,
>> >>> >>> 2018.
>> >>> >>> >> > >>>> Code freeze date : blockers/critical only, no
>> improvements
>> >>> and
>> >>> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>> >>> >>> >> > >>>> Release date: September 28, 2018
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> If any features in branch which are targeted to 3.2.0,
>> >>> please
>> >>> >>> >> reply to
>> >>> >>> >> > >>>> this email thread.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> 1. Merged & Completed features:
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> >>> DeepLearning
>> >>> >>> >> > >>>> workloads Initial cut.
>> >>> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> >>> Capacity
>> >>> >>> >> > >>>> Scheduler.
>> >>> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>> >>> YARN
>> >>> >>> >> Service
>> >>> >>> >> > >>>> API and CLI.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> 2. Features close to finish:
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in
>> YARN.
>> >>> >>> >> Merge/Vote
>> >>> >>> >> > >>>> Ongoing.
>> >>> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of
>> historical
>> >>> apps
>> >>> >>> from
>> >>> >>> >> > >>>> ATSv2. Patch in progress.
>> >>> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> >>> >>> Improvement
>> >>> >>> >> > works.
>> >>> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support
>> Windows
>> >>> Azure
>> >>> >>> >> > >>>> Storage. In progress.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> 3. Tentative features:
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >>> >>> >> challenging
>> >>> >>> >> > to
>> >>> >>> >> > >>>> be done before Aug 2018.
>> >>> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
>> >>> applications.
>> >>> >>> >> > >>>> Challenging as more discussions are on-going.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> *Summary of 3.2.0 issues status:*
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am
>> >>> following up
>> >>> >>> with
>> >>> >>> >> > >>>> owners to get status on each of them to get in by Code
>> >>> Freeze
>> >>> >>> date.
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>> >>> priority in
>> >>> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND
>> "Target
>> >>> >>> >> > Version/s" =
>> >>> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> Thanks,
>> >>> >>> >> > >>>> Sunil
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <
>> >>> sunilg@apache.org<ma...@apache.org>>
>> >>> >>> >> wrote:
>> >>> >>> >> > >>>>
>> >>> >>> >> > >>>>> Hi All,
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier
>> >>> communication
>> >>> >>> dated
>> >>> >>> >> > 17th
>> >>> >>> >> > >>>>> July 2018, I would like to provide some updates.
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> We are approaching previously proposed feature freeze
>> date
>> >>> >>> (Aug
>> >>> >>> >> 21,
>> >>> >>> >> > >>>>> about 7 days from today).
>> >>> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
>> >>> please
>> >>> >>> >> reply
>> >>> >>> >> > to
>> >>> >>> >> > >>>>> this email thread.
>> >>> >>> >> > >>>>> Steve has mentioned about the s3 features which will
>> come
>> >>> >>> close to
>> >>> >>> >> > >>>>> Code Freeze Date (Aug 31st).
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> 1. Merged & Completed features:
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> >>> >>> DeepLearning
>> >>> >>> >> > >>>>> workloads Initial cut.
>> >>> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> 2. Features close to finish:
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in
>> YARN.
>> >>> >>> Major
>> >>> >>> >> > >>>>> patches are all in, only one last
>> >>> >>> >> > >>>>> patch is in review state.
>> >>> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> >>> Capacity
>> >>> >>> >> > >>>>> Scheduler. Close to commit.
>> >>> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>> >>> YARN
>> >>> >>> >> Service
>> >>> >>> >> > >>>>> API and CLI. 2 patches are pending
>> >>> >>> >> > >>>>> which will be closed by Feature freeze date.
>> >>> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of
>> historical
>> >>> apps
>> >>> >>> >> from
>> >>> >>> >> > >>>>> ATSv2. Patch in progress.
>> >>> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> >>> >>> Improvement
>> >>> >>> >> > >>>>> works.
>> >>> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support
>> Windows
>> >>> >>> Azure
>> >>> >>> >> > >>>>> Storage. In progress.
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> 3. Tentative features:
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment.
>> Looks
>> >>> >>> >> challenging
>> >>> >>> >> > >>>>> to be done before Aug 2018.
>> >>> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
>> >>> applications.
>> >>> >>> >> > >>>>> Challenging as more discussions are on-going.
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am
>> checking
>> >>> >>> with
>> >>> >>> >> > owners
>> >>> >>> >> > >>>>> to get status on each of them to get in by Code Freeze
>> >>> date.
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>> >>> priority in
>> >>> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND
>> >>> "Target
>> >>> >>> >> > Version/s" =
>> >>> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> Thanks,
>> >>> >>> >> > >>>>> Sunil
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <
>> >>> sunilg@apache.org<ma...@apache.org>>
>> >>> >>> >> wrote:
>> >>> >>> >> > >>>>>
>> >>> >>> >> > >>>>>> Thanks Subru for the thoughts.
>> >>> >>> >> > >>>>>> One of the main reason for a major release is to push
>> out
>> >>> >>> >> critical
>> >>> >>> >> > >>>>>> features with a faster cadence to the users. If we are
>> >>> >>> pulling
>> >>> >>> >> more
>> >>> >>> >> > and
>> >>> >>> >> > >>>>>> more different types of features to a minor release,
>> that
>> >>> >>> branch
>> >>> >>> >> > will
>> >>> >>> >> > >>>>>> become more destabilized and it may be tough to say
>> that
>> >>> >>> 3.1.2 is
>> >>> >>> >> > stable
>> >>> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and
>> >>> stabilize
>> >>> >>> >> features
>> >>> >>> >> > in
>> >>> >>> >> > >>>>>> subsequent minor release.
>> >>> >>> >> > >>>>>> For few companies, it makes sense to push out these
>> new
>> >>> >>> features
>> >>> >>> >> > >>>>>> faster to make a reach to the users. Adding to the
>> point
>> >>> to
>> >>> >>> the
>> >>> >>> >> > backporting
>> >>> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround
>> >>> that
>> >>> >>> with
>> >>> >>> >> > some git
>> >>> >>> >> > >>>>>> scripts. If we can make such scripts available to
>> >>> committers,
>> >>> >>> >> > backport will
>> >>> >>> >> > >>>>>> be seem-less across branches and we can achieve the
>> >>> faster
>> >>> >>> >> release
>> >>> >>> >> > cadence
>> >>> >>> >> > >>>>>> also.
>> >>> >>> >> > >>>>>>
>> >>> >>> >> > >>>>>> Thoughts?
>> >>> >>> >> > >>>>>>
>> >>> >>> >> > >>>>>> - Sunil
>> >>> >>> >> > >>>>>>
>> >>> >>> >> > >>>>>>
>> >>> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>> >>> >>> subru@apache.org<ma...@apache.org>
>> >>> >>> >> >
>> >>> >>> >> > >>>>>> wrote:
>> >>> >>> >> > >>>>>>
>> >>> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release
>> >>> effort. I
>> >>> >>> am
>> >>> >>> >> > >>>>>>> generally
>> >>> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a
>> >>> 3.1.x) as
>> >>> >>> >> feel we
>> >>> >>> >> > >>>>>>> already
>> >>> >>> >> > >>>>>>> have too many branches to be maintained. I already
>> see
>> >>> many
>> >>> >>> >> commits
>> >>> >>> >> > >>>>>>> are in
>> >>> >>> >> > >>>>>>> different branches with no apparent rationale, for
>> e.g:
>> >>> 3.1
>> >>> >>> has
>> >>> >>> >> > >>>>>>> commits
>> >>> >>> >> > >>>>>>> which are absent in 3.0 etc.
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any
>> >>> major
>> >>> >>> >> > production
>> >>> >>> >> > >>>>>>> setting so the cost of adding features should be
>> >>> minimal.
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>> Thoughts?
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>> -Subru
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
>> >>> >>> sunilg@apache.org<ma...@apache.org>>
>> >>> >>> >> > wrote:
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > Yes, important changes and features are much
>> needed,
>> >>> >>> hence we
>> >>> >>> >> > will
>> >>> >>> >> > >>>>>>> be
>> >>> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
>> >>> >>> considering
>> >>> >>> >> few
>> >>> >>> >> > >>>>>>> more
>> >>> >>> >> > >>>>>>> > offline requests from other folks, I think
>> extending
>> >>> the
>> >>> >>> >> > timeframe
>> >>> >>> >> > >>>>>>> by
>> >>> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
>> >>> >>> buffer) and
>> >>> >>> >> > >>>>>>> this should
>> >>> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap
>> >>> wiki as
>> >>> >>> >> well)
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by
>> >>> August
>> >>> >>> 21,
>> >>> >>> >> 2018.
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>> >>> >>> improvements
>> >>> >>> >> and
>> >>> >>> >> > >>>>>>> non
>> >>> >>> >> > >>>>>>> > blocker/critical
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > - Release date: September 15, 2018
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already
>> >>> answered
>> >>> >>> your
>> >>> >>> >> > >>>>>>> questions.
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > Thanks
>> >>> >>> >> > >>>>>>> > Sunil
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>> >>> >>> >> wheeleast@gmail.com<ma...@gmail.com>
>> >>> >>> >> > >
>> >>> >>> >> > >>>>>>> wrote:
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2
>> >>> release,
>> >>> >>> +1
>> >>> >>> >> for
>> >>> >>> >> > >>>>>>> that.
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > To concerns from Steve,
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
>> >>> important
>> >>> >>> >> > changes
>> >>> >>> >> > >>>>>>> /
>> >>> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep
>> >>> the
>> >>> >>> >> proposed
>> >>> >>> >> > >>>>>>> release
>> >>> >>> >> > >>>>>>> > date
>> >>> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of
>> >>> last
>> >>> >>> >> minute
>> >>> >>> >> > >>>>>>> and we all
>> >>> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm
>> >>> also
>> >>> >>> fine
>> >>> >>> >> if
>> >>> >>> >> > >>>>>>> we want
>> >>> >>> >> > >>>>>>> > get
>> >>> >>> >> > >>>>>>> > > another several weeks time.
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing
>> that
>> >>> >>> before
>> >>> >>> >> > >>>>>>> thanksgiving.
>> >>> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > Eric,
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
>> >>> >>> YARN-8135
>> >>> >>> >> can
>> >>> >>> >> > >>>>>>> get merged
>> >>> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > Sunil,
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > Could u update
>> >>> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>> >>> >>> >> > >>>>>>> > Roadmap
>> >>> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature
>> list
>> >>> >>> first
>> >>> >>> >> > >>>>>>> before getting
>> >>> >>> >> > >>>>>>> > > consensus of time.
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > Thanks,
>> >>> >>> >> > >>>>>>> > > Wangda
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>> >>> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid<mailto:
>> >>> fabbri@cloudera.com.invalid>
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > > wrote:
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>> >>> >>> >> > >>>>>>> stevel@hortonworks.com<mailto:stevel@hortonworks.com
>> >>
>> >>> >>> >> > >>>>>>> > >> wrote:
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
>> >>> sunilg@apache.org<ma...@apache.org>
>> >>> >>> >> > <mailto:
>> >>> >>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>>
>> >>> wrote:
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > I would also would like to take this
>> opportunity
>> >>> to
>> >>> >>> come
>> >>> >>> >> up
>> >>> >>> >> > >>>>>>> with a
>> >>> >>> >> > >>>>>>> > >> detailed
>> >>> >>> >> > >>>>>>> > >> > plan.
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
>> >>> >>> merged by
>> >>> >>> >> > >>>>>>> August 10,
>> >>> >>> >> > >>>>>>> > >> 2018.
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > <snip>
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
>> >>> targeted
>> >>> >>> to
>> >>> >>> >> 3.2
>> >>> >>> >> > >>>>>>> per this
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 &
>> S3Guard.
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> https://issues.apache.org/jira/browse/HADOOP-15226
>> >>> >>> >> > >>>>>>> > >> >
>> >>> https://issues.apache.org/jira/browse/HADOOP-15220
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for
>> Azure
>> >>> >>> Datalake
>> >>> >>> >> > Gen
>> >>> >>> >> > >>>>>>> 2
>> >>> >>> >> > >>>>>>> > >> >
>> >>> https://issues.apache.org/jira/browse/HADOOP-15407
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've
>> been
>> >>> >>> doing
>> >>> >>> >> work
>> >>> >>> >> > >>>>>>> on it, but
>> >>> >>> >> > >>>>>>> > >> if
>> >>> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect
>> >>> some
>> >>> >>> >> timely
>> >>> >>> >> > >>>>>>> reviews on
>> >>> >>> >> > >>>>>>> > >> >
>> >>> https://issues.apache.org/jira/browse/HADOOP-15546
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will
>> >>> review
>> >>> >>> the
>> >>> >>> >> S3 &
>> >>> >>> >> > >>>>>>> S3Guard
>> >>> >>> >> > >>>>>>> > >> JIRAs
>> >>> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there
>> >>> are
>> >>> >>> some
>> >>> >>> >> > >>>>>>> pressing
>> >>> >>> >> > >>>>>>> > "guave,
>> >>> >>> >> > >>>>>>> > >> > java 9 prep"
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> I can help with this part if you like.
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
>> >>> release
>> >>> >>> >> > manager
>> >>> >>> >> > >>>>>>> of 3.2.0
>> >>> >>> >> > >>>>>>> > >> > release.
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > well volunteered!
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable
>> >>> should
>> >>> >>> we
>> >>> >>> >> have
>> >>> >>> >> > >>>>>>> for the
>> >>> >>> >> > >>>>>>> > >> 3.2. &
>> >>> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence,
>> >>> then
>> >>> >>> having
>> >>> >>> >> > the
>> >>> >>> >> > >>>>>>> outline
>> >>> >>> >> > >>>>>>> > >> time
>> >>> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that
>> >>> there's
>> >>> >>> less
>> >>> >>> >> > >>>>>>> concern about
>> >>> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> > -Steve
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> >
>> >>> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >> -AF
>> >>> >>> >> > >>>>>>> > >>
>> >>> >>> >> > >>>>>>> > >
>> >>> >>> >> > >>>>>>> >
>> >>> >>> >> > >>>>>>>
>> >>> >>> >> > >>>>>>
>> >>> >>> >> >
>> >>> >>> >>
>> >>> >>> >
>> >>> >>>
>> >>> >>
>> >>> >>
>> >>>
>> >>>
>> >>> --
>> >>> A very happy Clouderan
>> >>>
>> >>>
>>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi Folks,
All blockers are closed by last weekend and corrected jiras as well.
Preparing RC0 now. Facing some issue with shaded jars file size. Hence
respinning again.
Planning to complete this by end of this week.
Thanks
Sunil
On Fri, Oct 26, 2018 at 2:34 AM Konstantin Shvachko <sh...@gmail.com>
wrote:
> Another thing is that I see a bunch of jiras under HDFS-8707, which don't
> have the Fix Version field listing 3.2, some have it just empty.
> This means they will not be populated into release notes.
>
> Thanks,
> --Konstantin
>
> On Thu, Oct 25, 2018 at 7:59 AM Sunil G <su...@apache.org> wrote:
>
> > Thanks Konstantin for pointing out.
> > As 3.2 is pretty much on RC level, its better we try to find a good
> > solution to this issue.
> >
> > I ll follow up on this in the jira.
> >
> > - Sunil
> >
> > On Thu, Oct 25, 2018 at 11:35 AM Konstantin Shvachko <
> shv.hadoop@gmail.com>
> > wrote:
> >
> >> I've tried to attract attention to an incompatibility issue through the
> >> jira, but it didn't work. So pitching in in this thread.
> >> https://issues.apache.org/jira/browse/HDFS-12026
> >> It introduced binary incompatibility, which will prevent people from
> >> upgrading from 3.1 to 3.2.
> >> I think it can get messy if we release anything with this feature.
> >>
> >> Thanks,
> >> --Konstantin
> >>
> >> On Mon, Oct 22, 2018 at 5:01 AM Steve Loughran <st...@hortonworks.com>
> >> wrote:
> >>
> >>> its in.
> >>>
> >>> good catch!
> >>>
> >>> On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <weichiu@cloudera.com
> <mailto:
> >>> weichiu@cloudera.com>> wrote:
> >>>
> >>> Thanks Sunil G for driving the release,
> >>> I filed HADOOP-15866<
> https://issues.apache.org/jira/browse/HADOOP-15866>
> >>> for a compat fix. If any one has cycle please review it, as I think it
> is
> >>> needed for 3.2.0.
> >>>
> >>> On Thu, Oct 18, 2018 at 4:43 AM Sunil G <sunilg@apache.org<mailto:
> >>> sunilg@apache.org>> wrote:
> >>> Hi Folks,
> >>>
> >>> As we previously communicated for 3.2.0 release, we have delayed due to
> >>> few
> >>> blockers in our gate.
> >>>
> >>> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open
> for
> >>> all bug fixes.
> >>>
> >>> - Sunil
> >>>
> >>>
> >>> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <sunilg@apache.org<mailto:
> >>> sunilg@apache.org>> wrote:
> >>>
> >>> > Hi Folks,
> >>> >
> >>> > We are now close to RC as other blocker issues are now merged to
> trunk
> >>> and
> >>> > branch-3.2. Last 2 critical issues are closer to merge and will be
> >>> > committed in few hours.
> >>> > With this, I will be creating 3.2.0 branch today and will go ahead
> >>> with RC
> >>> > related process.
> >>> >
> >>> > - Sunil
> >>> >
> >>> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <
> jonbender@stripe.com
> >>> <ma...@stripe.com>>
> >>> > wrote:
> >>> >
> >>> >> Hello, were there any updates around the 3.2.0 RC timing? All I see
> in
> >>> >> the current blockers are related to the new Submarine subproject,
> >>> wasn't
> >>> >> sure if that is what is holding things up.
> >>> >>
> >>> >> Cheers,
> >>> >> Jon
> >>> >>
> >>> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <sunilg@apache.org<mailto:
> >>> sunilg@apache.org>> wrote:
> >>> >>
> >>> >>> Thanks Robert and Haibo for quickly correcting same.
> >>> >>> Sigh, I somehow missed one file while committing the change. Sorry
> >>> for
> >>> >>> the
> >>> >>> trouble.
> >>> >>>
> >>> >>> - Sunil
> >>> >>>
> >>> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rkanter@cloudera.com
> >>> <ma...@cloudera.com>>
> >>> >>> wrote:
> >>> >>>
> >>> >>> > Looks like there's two that weren't updated:
> >>> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" .
> >>> -r
> >>> >>> > --include=pom.xml
> >>> >>> > ./hadoop-project/pom.xml:
> >>> >>> >
> >>> <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
> >>> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
> >>> >>> >
> >>> >>> > I've just pushed in an addendum commit to fix those.
> >>> >>> > In the future, please make sure to do a sanity compile when
> >>> updating
> >>> >>> poms.
> >>> >>> >
> >>> >>> > thanks
> >>> >>> > - Robert
> >>> >>> >
> >>> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
> >>> >>> <fa...@cloudera.com.invalid>>
> >>> >>> > wrote:
> >>> >>> >
> >>> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in
> >>> the
> >>> >>> >> top-level pom.xml?
> >>> >>> >>
> >>> >>> >>
> >>> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <sunilg@apache.org
> >>> <ma...@apache.org>> wrote:
> >>> >>> >>
> >>> >>> >> > Hi All
> >>> >>> >> >
> >>> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
> >>> trunk
> >>> >>> to
> >>> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
> >>> >>> necessary
> >>> >>> >> > patches are pulled into branch-3.2.
> >>> >>> >> >
> >>> >>> >> > Thank You
> >>> >>> >> > - Sunil
> >>> >>> >> >
> >>> >>> >> >
> >>> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <sunilg@apache.org
> >>> <ma...@apache.org>> wrote:
> >>> >>> >> >
> >>> >>> >> > > Hi All
> >>> >>> >> > >
> >>> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is
> >>> merged to
> >>> >>> >> trunk.
> >>> >>> >> > > Thanks for the support.
> >>> >>> >> > >
> >>> >>> >> > > *Plan for RC*
> >>> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
> >>> >>> >> > > RC0 for 3.2: *28th Friday*
> >>> >>> >> > >
> >>> >>> >> > > Thank You
> >>> >>> >> > > Sunil
> >>> >>> >> > >
> >>> >>> >> > >
> >>> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <sunilg@apache.org
> >>> <ma...@apache.org>>
> >>> >>> wrote:
> >>> >>> >> > >
> >>> >>> >> > >> Hi All
> >>> >>> >> > >>
> >>> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of
> >>> you
> >>> >>> for
> >>> >>> >> > >> helping in this. I am following up on these tickets, once
> its
> >>> >>> closed
> >>> >>> >> we
> >>> >>> >> > >> will cut the 3.2 branch.
> >>> >>> >> > >>
> >>> >>> >> > >> Thanks
> >>> >>> >> > >> Sunil Govindan
> >>> >>> >> > >>
> >>> >>> >> > >>
> >>> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <sunilg@apache.org
> >>> <ma...@apache.org>>
> >>> >>> wrote:
> >>> >>> >> > >>
> >>> >>> >> > >>> Hi All,
> >>> >>> >> > >>>
> >>> >>> >> > >>> Inline with the original 3.2 communication proposal dated
> >>> 17th
> >>> >>> July
> >>> >>> >> > >>> 2018, I would like to provide more updates.
> >>> >>> >> > >>>
> >>> >>> >> > >>> We are approaching previously proposed code freeze date
> >>> >>> (September
> >>> >>> >> 14,
> >>> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
> >>> point
> >>> >>> >> > existing
> >>> >>> >> > >>> trunk to 3.3 if there are no issues.
> >>> >>> >> > >>>
> >>> >>> >> > >>> *Current Release Plan:*
> >>> >>> >> > >>> Feature freeze date : all features to merge by September
> 7,
> >>> >>> 2018.
> >>> >>> >> > >>> Code freeze date : blockers/critical only, no improvements
> >>> and
> >>> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
> >>> >>> >> > >>> Release date: September 28, 2018
> >>> >>> >> > >>>
> >>> >>> >> > >>> If any critical/blocker tickets which are targeted to
> >>> 3.2.0, we
> >>> >>> >> need to
> >>> >>> >> > >>> backport to 3.2 post branch cut.
> >>> >>> >> > >>>
> >>> >>> >> > >>> Here's an updated 3.2.0 feature status:
> >>> >>> >> > >>>
> >>> >>> >> > >>> 1. Merged & Completed features:
> >>> >>> >> > >>>
> >>> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
> >>> DeepLearning
> >>> >>> >> > >>> workloads Initial cut.
> >>> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in
> >>> Capacity
> >>> >>> >> > >>> Scheduler.
> >>> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via
> YARN
> >>> >>> Service
> >>> >>> >> > API
> >>> >>> >> > >>> and CLI.
> >>> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
> >>> Improvement
> >>> >>> >> works.
> >>> >>> >> > >>>
> >>> >>> >> > >>> 2. Features close to finish:
> >>> >>> >> > >>>
> >>> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
> >>> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
> >>> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
> >>> >>> >> > >>>
> >>> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
> >>> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> >>> apps
> >>> >>> from
> >>> >>> >> > >>> ATSv2. Patch in progress.
> >>> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >>> >> challenging to
> >>> >>> >> > >>> be done before Aug 2018.
> >>> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN
> >>> applications.
> >>> >>> >> > >>> Challenging as more discussions are on-going.
> >>> >>> >> > >>>
> >>> >>> >> > >>> *Summary of 3.2.0 issues status:*
> >>> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am
> following
> >>> up
> >>> >>> with
> >>> >>> >> > >>> owners to get status on each of them to get in by Code
> >>> Freeze
> >>> >>> date.
> >>> >>> >> > >>>
> >>> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
> priority
> >>> in
> >>> >>> >> > (Blocker,
> >>> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target
> >>> Version/s" =
> >>> >>> 3.2.0
> >>> >>> >> > ORDER
> >>> >>> >> > >>> BY priority DESC
> >>> >>> >> > >>>
> >>> >>> >> > >>> Thanks,
> >>> >>> >> > >>> Sunil
> >>> >>> >> > >>>
> >>> >>> >> > >>>
> >>> >>> >> > >>>
> >>> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <
> sunilg@apache.org
> >>> <ma...@apache.org>>
> >>> >>> wrote:
> >>> >>> >> > >>>
> >>> >>> >> > >>>> Hi All,
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> Inline with earlier communication dated 17th July 2018, I
> >>> would
> >>> >>> >> like
> >>> >>> >> > to
> >>> >>> >> > >>>> provide some updates.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> We are approaching previously proposed code freeze date
> >>> (Aug
> >>> >>> 31).
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> One of the critical feature Node Attributes feature merge
> >>> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs
> >>> need a
> >>> >>> bit
> >>> >>> >> > more
> >>> >>> >> > >>>> time. With regard to this, suggesting to push the
> >>> feature/code
> >>> >>> >> freeze
> >>> >>> >> > for 2
> >>> >>> >> > >>>> more weeks to accommodate these jiras too.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> Proposing Updated changes in plan inline with this:
> >>> >>> >> > >>>> Feature freeze date : all features to merge by September
> 7,
> >>> >>> 2018.
> >>> >>> >> > >>>> Code freeze date : blockers/critical only, no
> improvements
> >>> and
> >>> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
> >>> >>> >> > >>>> Release date: September 28, 2018
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> If any features in branch which are targeted to 3.2.0,
> >>> please
> >>> >>> >> reply to
> >>> >>> >> > >>>> this email thread.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> 1. Merged & Completed features:
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> >>> DeepLearning
> >>> >>> >> > >>>> workloads Initial cut.
> >>> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> >>> Capacity
> >>> >>> >> > >>>> Scheduler.
> >>> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
> >>> YARN
> >>> >>> >> Service
> >>> >>> >> > >>>> API and CLI.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> 2. Features close to finish:
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in
> YARN.
> >>> >>> >> Merge/Vote
> >>> >>> >> > >>>> Ongoing.
> >>> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> >>> apps
> >>> >>> from
> >>> >>> >> > >>>> ATSv2. Patch in progress.
> >>> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> >>> Improvement
> >>> >>> >> > works.
> >>> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
> >>> Azure
> >>> >>> >> > >>>> Storage. In progress.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> 3. Tentative features:
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >>> >> challenging
> >>> >>> >> > to
> >>> >>> >> > >>>> be done before Aug 2018.
> >>> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
> >>> applications.
> >>> >>> >> > >>>> Challenging as more discussions are on-going.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> *Summary of 3.2.0 issues status:*
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am
> >>> following up
> >>> >>> with
> >>> >>> >> > >>>> owners to get status on each of them to get in by Code
> >>> Freeze
> >>> >>> date.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
> >>> priority in
> >>> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND
> "Target
> >>> >>> >> > Version/s" =
> >>> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> Thanks,
> >>> >>> >> > >>>> Sunil
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <
> >>> sunilg@apache.org<ma...@apache.org>>
> >>> >>> >> wrote:
> >>> >>> >> > >>>>
> >>> >>> >> > >>>>> Hi All,
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier
> >>> communication
> >>> >>> dated
> >>> >>> >> > 17th
> >>> >>> >> > >>>>> July 2018, I would like to provide some updates.
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> We are approaching previously proposed feature freeze
> date
> >>> >>> (Aug
> >>> >>> >> 21,
> >>> >>> >> > >>>>> about 7 days from today).
> >>> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
> >>> please
> >>> >>> >> reply
> >>> >>> >> > to
> >>> >>> >> > >>>>> this email thread.
> >>> >>> >> > >>>>> Steve has mentioned about the s3 features which will
> come
> >>> >>> close to
> >>> >>> >> > >>>>> Code Freeze Date (Aug 31st).
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> 1. Merged & Completed features:
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> >>> >>> DeepLearning
> >>> >>> >> > >>>>> workloads Initial cut.
> >>> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> 2. Features close to finish:
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in
> YARN.
> >>> >>> Major
> >>> >>> >> > >>>>> patches are all in, only one last
> >>> >>> >> > >>>>> patch is in review state.
> >>> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> >>> Capacity
> >>> >>> >> > >>>>> Scheduler. Close to commit.
> >>> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
> >>> YARN
> >>> >>> >> Service
> >>> >>> >> > >>>>> API and CLI. 2 patches are pending
> >>> >>> >> > >>>>> which will be closed by Feature freeze date.
> >>> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of
> historical
> >>> apps
> >>> >>> >> from
> >>> >>> >> > >>>>> ATSv2. Patch in progress.
> >>> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> >>> Improvement
> >>> >>> >> > >>>>> works.
> >>> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support
> Windows
> >>> >>> Azure
> >>> >>> >> > >>>>> Storage. In progress.
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> 3. Tentative features:
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >>> >> challenging
> >>> >>> >> > >>>>> to be done before Aug 2018.
> >>> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
> >>> applications.
> >>> >>> >> > >>>>> Challenging as more discussions are on-going.
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am
> checking
> >>> >>> with
> >>> >>> >> > owners
> >>> >>> >> > >>>>> to get status on each of them to get in by Code Freeze
> >>> date.
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
> >>> priority in
> >>> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND
> >>> "Target
> >>> >>> >> > Version/s" =
> >>> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> Thanks,
> >>> >>> >> > >>>>> Sunil
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <
> >>> sunilg@apache.org<ma...@apache.org>>
> >>> >>> >> wrote:
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>>> Thanks Subru for the thoughts.
> >>> >>> >> > >>>>>> One of the main reason for a major release is to push
> out
> >>> >>> >> critical
> >>> >>> >> > >>>>>> features with a faster cadence to the users. If we are
> >>> >>> pulling
> >>> >>> >> more
> >>> >>> >> > and
> >>> >>> >> > >>>>>> more different types of features to a minor release,
> that
> >>> >>> branch
> >>> >>> >> > will
> >>> >>> >> > >>>>>> become more destabilized and it may be tough to say
> that
> >>> >>> 3.1.2 is
> >>> >>> >> > stable
> >>> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and
> >>> stabilize
> >>> >>> >> features
> >>> >>> >> > in
> >>> >>> >> > >>>>>> subsequent minor release.
> >>> >>> >> > >>>>>> For few companies, it makes sense to push out these new
> >>> >>> features
> >>> >>> >> > >>>>>> faster to make a reach to the users. Adding to the
> point
> >>> to
> >>> >>> the
> >>> >>> >> > backporting
> >>> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround
> >>> that
> >>> >>> with
> >>> >>> >> > some git
> >>> >>> >> > >>>>>> scripts. If we can make such scripts available to
> >>> committers,
> >>> >>> >> > backport will
> >>> >>> >> > >>>>>> be seem-less across branches and we can achieve the
> >>> faster
> >>> >>> >> release
> >>> >>> >> > cadence
> >>> >>> >> > >>>>>> also.
> >>> >>> >> > >>>>>>
> >>> >>> >> > >>>>>> Thoughts?
> >>> >>> >> > >>>>>>
> >>> >>> >> > >>>>>> - Sunil
> >>> >>> >> > >>>>>>
> >>> >>> >> > >>>>>>
> >>> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
> >>> >>> subru@apache.org<ma...@apache.org>
> >>> >>> >> >
> >>> >>> >> > >>>>>> wrote:
> >>> >>> >> > >>>>>>
> >>> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release
> >>> effort. I
> >>> >>> am
> >>> >>> >> > >>>>>>> generally
> >>> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a
> >>> 3.1.x) as
> >>> >>> >> feel we
> >>> >>> >> > >>>>>>> already
> >>> >>> >> > >>>>>>> have too many branches to be maintained. I already see
> >>> many
> >>> >>> >> commits
> >>> >>> >> > >>>>>>> are in
> >>> >>> >> > >>>>>>> different branches with no apparent rationale, for
> e.g:
> >>> 3.1
> >>> >>> has
> >>> >>> >> > >>>>>>> commits
> >>> >>> >> > >>>>>>> which are absent in 3.0 etc.
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any
> >>> major
> >>> >>> >> > production
> >>> >>> >> > >>>>>>> setting so the cost of adding features should be
> >>> minimal.
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>> Thoughts?
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>> -Subru
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
> >>> >>> sunilg@apache.org<ma...@apache.org>>
> >>> >>> >> > wrote:
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > Yes, important changes and features are much needed,
> >>> >>> hence we
> >>> >>> >> > will
> >>> >>> >> > >>>>>>> be
> >>> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
> >>> >>> considering
> >>> >>> >> few
> >>> >>> >> > >>>>>>> more
> >>> >>> >> > >>>>>>> > offline requests from other folks, I think extending
> >>> the
> >>> >>> >> > timeframe
> >>> >>> >> > >>>>>>> by
> >>> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
> >>> >>> buffer) and
> >>> >>> >> > >>>>>>> this should
> >>> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap
> >>> wiki as
> >>> >>> >> well)
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by
> >>> August
> >>> >>> 21,
> >>> >>> >> 2018.
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
> >>> >>> improvements
> >>> >>> >> and
> >>> >>> >> > >>>>>>> non
> >>> >>> >> > >>>>>>> > blocker/critical
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > - Release date: September 15, 2018
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already
> >>> answered
> >>> >>> your
> >>> >>> >> > >>>>>>> questions.
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > Thanks
> >>> >>> >> > >>>>>>> > Sunil
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
> >>> >>> >> wheeleast@gmail.com<ma...@gmail.com>
> >>> >>> >> > >
> >>> >>> >> > >>>>>>> wrote:
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2
> >>> release,
> >>> >>> +1
> >>> >>> >> for
> >>> >>> >> > >>>>>>> that.
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > To concerns from Steve,
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
> >>> important
> >>> >>> >> > changes
> >>> >>> >> > >>>>>>> /
> >>> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep
> >>> the
> >>> >>> >> proposed
> >>> >>> >> > >>>>>>> release
> >>> >>> >> > >>>>>>> > date
> >>> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of
> >>> last
> >>> >>> >> minute
> >>> >>> >> > >>>>>>> and we all
> >>> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm
> >>> also
> >>> >>> fine
> >>> >>> >> if
> >>> >>> >> > >>>>>>> we want
> >>> >>> >> > >>>>>>> > get
> >>> >>> >> > >>>>>>> > > another several weeks time.
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing
> that
> >>> >>> before
> >>> >>> >> > >>>>>>> thanksgiving.
> >>> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > Eric,
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
> >>> >>> YARN-8135
> >>> >>> >> can
> >>> >>> >> > >>>>>>> get merged
> >>> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > Sunil,
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > Could u update
> >>> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
> >>> >>> >> > >>>>>>> > Roadmap
> >>> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature
> list
> >>> >>> first
> >>> >>> >> > >>>>>>> before getting
> >>> >>> >> > >>>>>>> > > consensus of time.
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > Thanks,
> >>> >>> >> > >>>>>>> > > Wangda
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> >>> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid<mailto:
> >>> fabbri@cloudera.com.invalid>
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > wrote:
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> >>> >>> >> > >>>>>>> stevel@hortonworks.com<mailto:stevel@hortonworks.com
> >>
> >>> >>> >> > >>>>>>> > >> wrote:
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
> >>> sunilg@apache.org<ma...@apache.org>
> >>> >>> >> > <mailto:
> >>> >>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>>
> >>> wrote:
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > I would also would like to take this
> opportunity
> >>> to
> >>> >>> come
> >>> >>> >> up
> >>> >>> >> > >>>>>>> with a
> >>> >>> >> > >>>>>>> > >> detailed
> >>> >>> >> > >>>>>>> > >> > plan.
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
> >>> >>> merged by
> >>> >>> >> > >>>>>>> August 10,
> >>> >>> >> > >>>>>>> > >> 2018.
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > <snip>
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
> >>> targeted
> >>> >>> to
> >>> >>> >> 3.2
> >>> >>> >> > >>>>>>> per this
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 &
> S3Guard.
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> https://issues.apache.org/jira/browse/HADOOP-15226
> >>> >>> >> > >>>>>>> > >> >
> >>> https://issues.apache.org/jira/browse/HADOOP-15220
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for
> Azure
> >>> >>> Datalake
> >>> >>> >> > Gen
> >>> >>> >> > >>>>>>> 2
> >>> >>> >> > >>>>>>> > >> >
> >>> https://issues.apache.org/jira/browse/HADOOP-15407
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've
> been
> >>> >>> doing
> >>> >>> >> work
> >>> >>> >> > >>>>>>> on it, but
> >>> >>> >> > >>>>>>> > >> if
> >>> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect
> >>> some
> >>> >>> >> timely
> >>> >>> >> > >>>>>>> reviews on
> >>> >>> >> > >>>>>>> > >> >
> >>> https://issues.apache.org/jira/browse/HADOOP-15546
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will
> >>> review
> >>> >>> the
> >>> >>> >> S3 &
> >>> >>> >> > >>>>>>> S3Guard
> >>> >>> >> > >>>>>>> > >> JIRAs
> >>> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there
> >>> are
> >>> >>> some
> >>> >>> >> > >>>>>>> pressing
> >>> >>> >> > >>>>>>> > "guave,
> >>> >>> >> > >>>>>>> > >> > java 9 prep"
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> I can help with this part if you like.
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
> >>> release
> >>> >>> >> > manager
> >>> >>> >> > >>>>>>> of 3.2.0
> >>> >>> >> > >>>>>>> > >> > release.
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > well volunteered!
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable
> >>> should
> >>> >>> we
> >>> >>> >> have
> >>> >>> >> > >>>>>>> for the
> >>> >>> >> > >>>>>>> > >> 3.2. &
> >>> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence,
> >>> then
> >>> >>> having
> >>> >>> >> > the
> >>> >>> >> > >>>>>>> outline
> >>> >>> >> > >>>>>>> > >> time
> >>> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that
> >>> there's
> >>> >>> less
> >>> >>> >> > >>>>>>> concern about
> >>> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > -Steve
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >> -AF
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>
> >>> >>> >> >
> >>> >>> >>
> >>> >>> >
> >>> >>>
> >>> >>
> >>> >>
> >>>
> >>>
> >>> --
> >>> A very happy Clouderan
> >>>
> >>>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi Folks,
All blockers are closed by last weekend and corrected jiras as well.
Preparing RC0 now. Facing some issue with shaded jars file size. Hence
respinning again.
Planning to complete this by end of this week.
Thanks
Sunil
On Fri, Oct 26, 2018 at 2:34 AM Konstantin Shvachko <sh...@gmail.com>
wrote:
> Another thing is that I see a bunch of jiras under HDFS-8707, which don't
> have the Fix Version field listing 3.2, some have it just empty.
> This means they will not be populated into release notes.
>
> Thanks,
> --Konstantin
>
> On Thu, Oct 25, 2018 at 7:59 AM Sunil G <su...@apache.org> wrote:
>
> > Thanks Konstantin for pointing out.
> > As 3.2 is pretty much on RC level, its better we try to find a good
> > solution to this issue.
> >
> > I ll follow up on this in the jira.
> >
> > - Sunil
> >
> > On Thu, Oct 25, 2018 at 11:35 AM Konstantin Shvachko <
> shv.hadoop@gmail.com>
> > wrote:
> >
> >> I've tried to attract attention to an incompatibility issue through the
> >> jira, but it didn't work. So pitching in in this thread.
> >> https://issues.apache.org/jira/browse/HDFS-12026
> >> It introduced binary incompatibility, which will prevent people from
> >> upgrading from 3.1 to 3.2.
> >> I think it can get messy if we release anything with this feature.
> >>
> >> Thanks,
> >> --Konstantin
> >>
> >> On Mon, Oct 22, 2018 at 5:01 AM Steve Loughran <st...@hortonworks.com>
> >> wrote:
> >>
> >>> its in.
> >>>
> >>> good catch!
> >>>
> >>> On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <weichiu@cloudera.com
> <mailto:
> >>> weichiu@cloudera.com>> wrote:
> >>>
> >>> Thanks Sunil G for driving the release,
> >>> I filed HADOOP-15866<
> https://issues.apache.org/jira/browse/HADOOP-15866>
> >>> for a compat fix. If any one has cycle please review it, as I think it
> is
> >>> needed for 3.2.0.
> >>>
> >>> On Thu, Oct 18, 2018 at 4:43 AM Sunil G <sunilg@apache.org<mailto:
> >>> sunilg@apache.org>> wrote:
> >>> Hi Folks,
> >>>
> >>> As we previously communicated for 3.2.0 release, we have delayed due to
> >>> few
> >>> blockers in our gate.
> >>>
> >>> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open
> for
> >>> all bug fixes.
> >>>
> >>> - Sunil
> >>>
> >>>
> >>> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <sunilg@apache.org<mailto:
> >>> sunilg@apache.org>> wrote:
> >>>
> >>> > Hi Folks,
> >>> >
> >>> > We are now close to RC as other blocker issues are now merged to
> trunk
> >>> and
> >>> > branch-3.2. Last 2 critical issues are closer to merge and will be
> >>> > committed in few hours.
> >>> > With this, I will be creating 3.2.0 branch today and will go ahead
> >>> with RC
> >>> > related process.
> >>> >
> >>> > - Sunil
> >>> >
> >>> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <
> jonbender@stripe.com
> >>> <ma...@stripe.com>>
> >>> > wrote:
> >>> >
> >>> >> Hello, were there any updates around the 3.2.0 RC timing? All I see
> in
> >>> >> the current blockers are related to the new Submarine subproject,
> >>> wasn't
> >>> >> sure if that is what is holding things up.
> >>> >>
> >>> >> Cheers,
> >>> >> Jon
> >>> >>
> >>> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <sunilg@apache.org<mailto:
> >>> sunilg@apache.org>> wrote:
> >>> >>
> >>> >>> Thanks Robert and Haibo for quickly correcting same.
> >>> >>> Sigh, I somehow missed one file while committing the change. Sorry
> >>> for
> >>> >>> the
> >>> >>> trouble.
> >>> >>>
> >>> >>> - Sunil
> >>> >>>
> >>> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rkanter@cloudera.com
> >>> <ma...@cloudera.com>>
> >>> >>> wrote:
> >>> >>>
> >>> >>> > Looks like there's two that weren't updated:
> >>> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" .
> >>> -r
> >>> >>> > --include=pom.xml
> >>> >>> > ./hadoop-project/pom.xml:
> >>> >>> >
> >>> <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
> >>> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
> >>> >>> >
> >>> >>> > I've just pushed in an addendum commit to fix those.
> >>> >>> > In the future, please make sure to do a sanity compile when
> >>> updating
> >>> >>> poms.
> >>> >>> >
> >>> >>> > thanks
> >>> >>> > - Robert
> >>> >>> >
> >>> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
> >>> >>> <fa...@cloudera.com.invalid>>
> >>> >>> > wrote:
> >>> >>> >
> >>> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in
> >>> the
> >>> >>> >> top-level pom.xml?
> >>> >>> >>
> >>> >>> >>
> >>> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <sunilg@apache.org
> >>> <ma...@apache.org>> wrote:
> >>> >>> >>
> >>> >>> >> > Hi All
> >>> >>> >> >
> >>> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
> >>> trunk
> >>> >>> to
> >>> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
> >>> >>> necessary
> >>> >>> >> > patches are pulled into branch-3.2.
> >>> >>> >> >
> >>> >>> >> > Thank You
> >>> >>> >> > - Sunil
> >>> >>> >> >
> >>> >>> >> >
> >>> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <sunilg@apache.org
> >>> <ma...@apache.org>> wrote:
> >>> >>> >> >
> >>> >>> >> > > Hi All
> >>> >>> >> > >
> >>> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is
> >>> merged to
> >>> >>> >> trunk.
> >>> >>> >> > > Thanks for the support.
> >>> >>> >> > >
> >>> >>> >> > > *Plan for RC*
> >>> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
> >>> >>> >> > > RC0 for 3.2: *28th Friday*
> >>> >>> >> > >
> >>> >>> >> > > Thank You
> >>> >>> >> > > Sunil
> >>> >>> >> > >
> >>> >>> >> > >
> >>> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <sunilg@apache.org
> >>> <ma...@apache.org>>
> >>> >>> wrote:
> >>> >>> >> > >
> >>> >>> >> > >> Hi All
> >>> >>> >> > >>
> >>> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of
> >>> you
> >>> >>> for
> >>> >>> >> > >> helping in this. I am following up on these tickets, once
> its
> >>> >>> closed
> >>> >>> >> we
> >>> >>> >> > >> will cut the 3.2 branch.
> >>> >>> >> > >>
> >>> >>> >> > >> Thanks
> >>> >>> >> > >> Sunil Govindan
> >>> >>> >> > >>
> >>> >>> >> > >>
> >>> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <sunilg@apache.org
> >>> <ma...@apache.org>>
> >>> >>> wrote:
> >>> >>> >> > >>
> >>> >>> >> > >>> Hi All,
> >>> >>> >> > >>>
> >>> >>> >> > >>> Inline with the original 3.2 communication proposal dated
> >>> 17th
> >>> >>> July
> >>> >>> >> > >>> 2018, I would like to provide more updates.
> >>> >>> >> > >>>
> >>> >>> >> > >>> We are approaching previously proposed code freeze date
> >>> >>> (September
> >>> >>> >> 14,
> >>> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
> >>> point
> >>> >>> >> > existing
> >>> >>> >> > >>> trunk to 3.3 if there are no issues.
> >>> >>> >> > >>>
> >>> >>> >> > >>> *Current Release Plan:*
> >>> >>> >> > >>> Feature freeze date : all features to merge by September
> 7,
> >>> >>> 2018.
> >>> >>> >> > >>> Code freeze date : blockers/critical only, no improvements
> >>> and
> >>> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
> >>> >>> >> > >>> Release date: September 28, 2018
> >>> >>> >> > >>>
> >>> >>> >> > >>> If any critical/blocker tickets which are targeted to
> >>> 3.2.0, we
> >>> >>> >> need to
> >>> >>> >> > >>> backport to 3.2 post branch cut.
> >>> >>> >> > >>>
> >>> >>> >> > >>> Here's an updated 3.2.0 feature status:
> >>> >>> >> > >>>
> >>> >>> >> > >>> 1. Merged & Completed features:
> >>> >>> >> > >>>
> >>> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
> >>> DeepLearning
> >>> >>> >> > >>> workloads Initial cut.
> >>> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in
> >>> Capacity
> >>> >>> >> > >>> Scheduler.
> >>> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via
> YARN
> >>> >>> Service
> >>> >>> >> > API
> >>> >>> >> > >>> and CLI.
> >>> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
> >>> Improvement
> >>> >>> >> works.
> >>> >>> >> > >>>
> >>> >>> >> > >>> 2. Features close to finish:
> >>> >>> >> > >>>
> >>> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
> >>> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
> >>> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
> >>> >>> >> > >>>
> >>> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
> >>> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> >>> apps
> >>> >>> from
> >>> >>> >> > >>> ATSv2. Patch in progress.
> >>> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >>> >> challenging to
> >>> >>> >> > >>> be done before Aug 2018.
> >>> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN
> >>> applications.
> >>> >>> >> > >>> Challenging as more discussions are on-going.
> >>> >>> >> > >>>
> >>> >>> >> > >>> *Summary of 3.2.0 issues status:*
> >>> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am
> following
> >>> up
> >>> >>> with
> >>> >>> >> > >>> owners to get status on each of them to get in by Code
> >>> Freeze
> >>> >>> date.
> >>> >>> >> > >>>
> >>> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
> priority
> >>> in
> >>> >>> >> > (Blocker,
> >>> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target
> >>> Version/s" =
> >>> >>> 3.2.0
> >>> >>> >> > ORDER
> >>> >>> >> > >>> BY priority DESC
> >>> >>> >> > >>>
> >>> >>> >> > >>> Thanks,
> >>> >>> >> > >>> Sunil
> >>> >>> >> > >>>
> >>> >>> >> > >>>
> >>> >>> >> > >>>
> >>> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <
> sunilg@apache.org
> >>> <ma...@apache.org>>
> >>> >>> wrote:
> >>> >>> >> > >>>
> >>> >>> >> > >>>> Hi All,
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> Inline with earlier communication dated 17th July 2018, I
> >>> would
> >>> >>> >> like
> >>> >>> >> > to
> >>> >>> >> > >>>> provide some updates.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> We are approaching previously proposed code freeze date
> >>> (Aug
> >>> >>> 31).
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> One of the critical feature Node Attributes feature merge
> >>> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs
> >>> need a
> >>> >>> bit
> >>> >>> >> > more
> >>> >>> >> > >>>> time. With regard to this, suggesting to push the
> >>> feature/code
> >>> >>> >> freeze
> >>> >>> >> > for 2
> >>> >>> >> > >>>> more weeks to accommodate these jiras too.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> Proposing Updated changes in plan inline with this:
> >>> >>> >> > >>>> Feature freeze date : all features to merge by September
> 7,
> >>> >>> 2018.
> >>> >>> >> > >>>> Code freeze date : blockers/critical only, no
> improvements
> >>> and
> >>> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
> >>> >>> >> > >>>> Release date: September 28, 2018
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> If any features in branch which are targeted to 3.2.0,
> >>> please
> >>> >>> >> reply to
> >>> >>> >> > >>>> this email thread.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> 1. Merged & Completed features:
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> >>> DeepLearning
> >>> >>> >> > >>>> workloads Initial cut.
> >>> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> >>> Capacity
> >>> >>> >> > >>>> Scheduler.
> >>> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
> >>> YARN
> >>> >>> >> Service
> >>> >>> >> > >>>> API and CLI.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> 2. Features close to finish:
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in
> YARN.
> >>> >>> >> Merge/Vote
> >>> >>> >> > >>>> Ongoing.
> >>> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> >>> apps
> >>> >>> from
> >>> >>> >> > >>>> ATSv2. Patch in progress.
> >>> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> >>> Improvement
> >>> >>> >> > works.
> >>> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
> >>> Azure
> >>> >>> >> > >>>> Storage. In progress.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> 3. Tentative features:
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >>> >> challenging
> >>> >>> >> > to
> >>> >>> >> > >>>> be done before Aug 2018.
> >>> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
> >>> applications.
> >>> >>> >> > >>>> Challenging as more discussions are on-going.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> *Summary of 3.2.0 issues status:*
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am
> >>> following up
> >>> >>> with
> >>> >>> >> > >>>> owners to get status on each of them to get in by Code
> >>> Freeze
> >>> >>> date.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
> >>> priority in
> >>> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND
> "Target
> >>> >>> >> > Version/s" =
> >>> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> Thanks,
> >>> >>> >> > >>>> Sunil
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <
> >>> sunilg@apache.org<ma...@apache.org>>
> >>> >>> >> wrote:
> >>> >>> >> > >>>>
> >>> >>> >> > >>>>> Hi All,
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier
> >>> communication
> >>> >>> dated
> >>> >>> >> > 17th
> >>> >>> >> > >>>>> July 2018, I would like to provide some updates.
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> We are approaching previously proposed feature freeze
> date
> >>> >>> (Aug
> >>> >>> >> 21,
> >>> >>> >> > >>>>> about 7 days from today).
> >>> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
> >>> please
> >>> >>> >> reply
> >>> >>> >> > to
> >>> >>> >> > >>>>> this email thread.
> >>> >>> >> > >>>>> Steve has mentioned about the s3 features which will
> come
> >>> >>> close to
> >>> >>> >> > >>>>> Code Freeze Date (Aug 31st).
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> 1. Merged & Completed features:
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> >>> >>> DeepLearning
> >>> >>> >> > >>>>> workloads Initial cut.
> >>> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> 2. Features close to finish:
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in
> YARN.
> >>> >>> Major
> >>> >>> >> > >>>>> patches are all in, only one last
> >>> >>> >> > >>>>> patch is in review state.
> >>> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> >>> Capacity
> >>> >>> >> > >>>>> Scheduler. Close to commit.
> >>> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
> >>> YARN
> >>> >>> >> Service
> >>> >>> >> > >>>>> API and CLI. 2 patches are pending
> >>> >>> >> > >>>>> which will be closed by Feature freeze date.
> >>> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of
> historical
> >>> apps
> >>> >>> >> from
> >>> >>> >> > >>>>> ATSv2. Patch in progress.
> >>> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> >>> Improvement
> >>> >>> >> > >>>>> works.
> >>> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support
> Windows
> >>> >>> Azure
> >>> >>> >> > >>>>> Storage. In progress.
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> 3. Tentative features:
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >>> >> challenging
> >>> >>> >> > >>>>> to be done before Aug 2018.
> >>> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
> >>> applications.
> >>> >>> >> > >>>>> Challenging as more discussions are on-going.
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am
> checking
> >>> >>> with
> >>> >>> >> > owners
> >>> >>> >> > >>>>> to get status on each of them to get in by Code Freeze
> >>> date.
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
> >>> priority in
> >>> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND
> >>> "Target
> >>> >>> >> > Version/s" =
> >>> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> Thanks,
> >>> >>> >> > >>>>> Sunil
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <
> >>> sunilg@apache.org<ma...@apache.org>>
> >>> >>> >> wrote:
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>>> Thanks Subru for the thoughts.
> >>> >>> >> > >>>>>> One of the main reason for a major release is to push
> out
> >>> >>> >> critical
> >>> >>> >> > >>>>>> features with a faster cadence to the users. If we are
> >>> >>> pulling
> >>> >>> >> more
> >>> >>> >> > and
> >>> >>> >> > >>>>>> more different types of features to a minor release,
> that
> >>> >>> branch
> >>> >>> >> > will
> >>> >>> >> > >>>>>> become more destabilized and it may be tough to say
> that
> >>> >>> 3.1.2 is
> >>> >>> >> > stable
> >>> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and
> >>> stabilize
> >>> >>> >> features
> >>> >>> >> > in
> >>> >>> >> > >>>>>> subsequent minor release.
> >>> >>> >> > >>>>>> For few companies, it makes sense to push out these new
> >>> >>> features
> >>> >>> >> > >>>>>> faster to make a reach to the users. Adding to the
> point
> >>> to
> >>> >>> the
> >>> >>> >> > backporting
> >>> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround
> >>> that
> >>> >>> with
> >>> >>> >> > some git
> >>> >>> >> > >>>>>> scripts. If we can make such scripts available to
> >>> committers,
> >>> >>> >> > backport will
> >>> >>> >> > >>>>>> be seem-less across branches and we can achieve the
> >>> faster
> >>> >>> >> release
> >>> >>> >> > cadence
> >>> >>> >> > >>>>>> also.
> >>> >>> >> > >>>>>>
> >>> >>> >> > >>>>>> Thoughts?
> >>> >>> >> > >>>>>>
> >>> >>> >> > >>>>>> - Sunil
> >>> >>> >> > >>>>>>
> >>> >>> >> > >>>>>>
> >>> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
> >>> >>> subru@apache.org<ma...@apache.org>
> >>> >>> >> >
> >>> >>> >> > >>>>>> wrote:
> >>> >>> >> > >>>>>>
> >>> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release
> >>> effort. I
> >>> >>> am
> >>> >>> >> > >>>>>>> generally
> >>> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a
> >>> 3.1.x) as
> >>> >>> >> feel we
> >>> >>> >> > >>>>>>> already
> >>> >>> >> > >>>>>>> have too many branches to be maintained. I already see
> >>> many
> >>> >>> >> commits
> >>> >>> >> > >>>>>>> are in
> >>> >>> >> > >>>>>>> different branches with no apparent rationale, for
> e.g:
> >>> 3.1
> >>> >>> has
> >>> >>> >> > >>>>>>> commits
> >>> >>> >> > >>>>>>> which are absent in 3.0 etc.
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any
> >>> major
> >>> >>> >> > production
> >>> >>> >> > >>>>>>> setting so the cost of adding features should be
> >>> minimal.
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>> Thoughts?
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>> -Subru
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
> >>> >>> sunilg@apache.org<ma...@apache.org>>
> >>> >>> >> > wrote:
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > Yes, important changes and features are much needed,
> >>> >>> hence we
> >>> >>> >> > will
> >>> >>> >> > >>>>>>> be
> >>> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
> >>> >>> considering
> >>> >>> >> few
> >>> >>> >> > >>>>>>> more
> >>> >>> >> > >>>>>>> > offline requests from other folks, I think extending
> >>> the
> >>> >>> >> > timeframe
> >>> >>> >> > >>>>>>> by
> >>> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
> >>> >>> buffer) and
> >>> >>> >> > >>>>>>> this should
> >>> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap
> >>> wiki as
> >>> >>> >> well)
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by
> >>> August
> >>> >>> 21,
> >>> >>> >> 2018.
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
> >>> >>> improvements
> >>> >>> >> and
> >>> >>> >> > >>>>>>> non
> >>> >>> >> > >>>>>>> > blocker/critical
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > - Release date: September 15, 2018
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already
> >>> answered
> >>> >>> your
> >>> >>> >> > >>>>>>> questions.
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > Thanks
> >>> >>> >> > >>>>>>> > Sunil
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
> >>> >>> >> wheeleast@gmail.com<ma...@gmail.com>
> >>> >>> >> > >
> >>> >>> >> > >>>>>>> wrote:
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2
> >>> release,
> >>> >>> +1
> >>> >>> >> for
> >>> >>> >> > >>>>>>> that.
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > To concerns from Steve,
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
> >>> important
> >>> >>> >> > changes
> >>> >>> >> > >>>>>>> /
> >>> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep
> >>> the
> >>> >>> >> proposed
> >>> >>> >> > >>>>>>> release
> >>> >>> >> > >>>>>>> > date
> >>> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of
> >>> last
> >>> >>> >> minute
> >>> >>> >> > >>>>>>> and we all
> >>> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm
> >>> also
> >>> >>> fine
> >>> >>> >> if
> >>> >>> >> > >>>>>>> we want
> >>> >>> >> > >>>>>>> > get
> >>> >>> >> > >>>>>>> > > another several weeks time.
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing
> that
> >>> >>> before
> >>> >>> >> > >>>>>>> thanksgiving.
> >>> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > Eric,
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
> >>> >>> YARN-8135
> >>> >>> >> can
> >>> >>> >> > >>>>>>> get merged
> >>> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > Sunil,
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > Could u update
> >>> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
> >>> >>> >> > >>>>>>> > Roadmap
> >>> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature
> list
> >>> >>> first
> >>> >>> >> > >>>>>>> before getting
> >>> >>> >> > >>>>>>> > > consensus of time.
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > Thanks,
> >>> >>> >> > >>>>>>> > > Wangda
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> >>> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid<mailto:
> >>> fabbri@cloudera.com.invalid>
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > wrote:
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> >>> >>> >> > >>>>>>> stevel@hortonworks.com<mailto:stevel@hortonworks.com
> >>
> >>> >>> >> > >>>>>>> > >> wrote:
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
> >>> sunilg@apache.org<ma...@apache.org>
> >>> >>> >> > <mailto:
> >>> >>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>>
> >>> wrote:
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > I would also would like to take this
> opportunity
> >>> to
> >>> >>> come
> >>> >>> >> up
> >>> >>> >> > >>>>>>> with a
> >>> >>> >> > >>>>>>> > >> detailed
> >>> >>> >> > >>>>>>> > >> > plan.
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
> >>> >>> merged by
> >>> >>> >> > >>>>>>> August 10,
> >>> >>> >> > >>>>>>> > >> 2018.
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > <snip>
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
> >>> targeted
> >>> >>> to
> >>> >>> >> 3.2
> >>> >>> >> > >>>>>>> per this
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 &
> S3Guard.
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> https://issues.apache.org/jira/browse/HADOOP-15226
> >>> >>> >> > >>>>>>> > >> >
> >>> https://issues.apache.org/jira/browse/HADOOP-15220
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for
> Azure
> >>> >>> Datalake
> >>> >>> >> > Gen
> >>> >>> >> > >>>>>>> 2
> >>> >>> >> > >>>>>>> > >> >
> >>> https://issues.apache.org/jira/browse/HADOOP-15407
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've
> been
> >>> >>> doing
> >>> >>> >> work
> >>> >>> >> > >>>>>>> on it, but
> >>> >>> >> > >>>>>>> > >> if
> >>> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect
> >>> some
> >>> >>> >> timely
> >>> >>> >> > >>>>>>> reviews on
> >>> >>> >> > >>>>>>> > >> >
> >>> https://issues.apache.org/jira/browse/HADOOP-15546
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will
> >>> review
> >>> >>> the
> >>> >>> >> S3 &
> >>> >>> >> > >>>>>>> S3Guard
> >>> >>> >> > >>>>>>> > >> JIRAs
> >>> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there
> >>> are
> >>> >>> some
> >>> >>> >> > >>>>>>> pressing
> >>> >>> >> > >>>>>>> > "guave,
> >>> >>> >> > >>>>>>> > >> > java 9 prep"
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> I can help with this part if you like.
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
> >>> release
> >>> >>> >> > manager
> >>> >>> >> > >>>>>>> of 3.2.0
> >>> >>> >> > >>>>>>> > >> > release.
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > well volunteered!
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable
> >>> should
> >>> >>> we
> >>> >>> >> have
> >>> >>> >> > >>>>>>> for the
> >>> >>> >> > >>>>>>> > >> 3.2. &
> >>> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence,
> >>> then
> >>> >>> having
> >>> >>> >> > the
> >>> >>> >> > >>>>>>> outline
> >>> >>> >> > >>>>>>> > >> time
> >>> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that
> >>> there's
> >>> >>> less
> >>> >>> >> > >>>>>>> concern about
> >>> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > -Steve
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >> -AF
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>
> >>> >>> >> >
> >>> >>> >>
> >>> >>> >
> >>> >>>
> >>> >>
> >>> >>
> >>>
> >>>
> >>> --
> >>> A very happy Clouderan
> >>>
> >>>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi Folks,
All blockers are closed by last weekend and corrected jiras as well.
Preparing RC0 now. Facing some issue with shaded jars file size. Hence
respinning again.
Planning to complete this by end of this week.
Thanks
Sunil
On Fri, Oct 26, 2018 at 2:34 AM Konstantin Shvachko <sh...@gmail.com>
wrote:
> Another thing is that I see a bunch of jiras under HDFS-8707, which don't
> have the Fix Version field listing 3.2, some have it just empty.
> This means they will not be populated into release notes.
>
> Thanks,
> --Konstantin
>
> On Thu, Oct 25, 2018 at 7:59 AM Sunil G <su...@apache.org> wrote:
>
> > Thanks Konstantin for pointing out.
> > As 3.2 is pretty much on RC level, its better we try to find a good
> > solution to this issue.
> >
> > I ll follow up on this in the jira.
> >
> > - Sunil
> >
> > On Thu, Oct 25, 2018 at 11:35 AM Konstantin Shvachko <
> shv.hadoop@gmail.com>
> > wrote:
> >
> >> I've tried to attract attention to an incompatibility issue through the
> >> jira, but it didn't work. So pitching in in this thread.
> >> https://issues.apache.org/jira/browse/HDFS-12026
> >> It introduced binary incompatibility, which will prevent people from
> >> upgrading from 3.1 to 3.2.
> >> I think it can get messy if we release anything with this feature.
> >>
> >> Thanks,
> >> --Konstantin
> >>
> >> On Mon, Oct 22, 2018 at 5:01 AM Steve Loughran <st...@hortonworks.com>
> >> wrote:
> >>
> >>> its in.
> >>>
> >>> good catch!
> >>>
> >>> On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <weichiu@cloudera.com
> <mailto:
> >>> weichiu@cloudera.com>> wrote:
> >>>
> >>> Thanks Sunil G for driving the release,
> >>> I filed HADOOP-15866<
> https://issues.apache.org/jira/browse/HADOOP-15866>
> >>> for a compat fix. If any one has cycle please review it, as I think it
> is
> >>> needed for 3.2.0.
> >>>
> >>> On Thu, Oct 18, 2018 at 4:43 AM Sunil G <sunilg@apache.org<mailto:
> >>> sunilg@apache.org>> wrote:
> >>> Hi Folks,
> >>>
> >>> As we previously communicated for 3.2.0 release, we have delayed due to
> >>> few
> >>> blockers in our gate.
> >>>
> >>> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open
> for
> >>> all bug fixes.
> >>>
> >>> - Sunil
> >>>
> >>>
> >>> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <sunilg@apache.org<mailto:
> >>> sunilg@apache.org>> wrote:
> >>>
> >>> > Hi Folks,
> >>> >
> >>> > We are now close to RC as other blocker issues are now merged to
> trunk
> >>> and
> >>> > branch-3.2. Last 2 critical issues are closer to merge and will be
> >>> > committed in few hours.
> >>> > With this, I will be creating 3.2.0 branch today and will go ahead
> >>> with RC
> >>> > related process.
> >>> >
> >>> > - Sunil
> >>> >
> >>> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <
> jonbender@stripe.com
> >>> <ma...@stripe.com>>
> >>> > wrote:
> >>> >
> >>> >> Hello, were there any updates around the 3.2.0 RC timing? All I see
> in
> >>> >> the current blockers are related to the new Submarine subproject,
> >>> wasn't
> >>> >> sure if that is what is holding things up.
> >>> >>
> >>> >> Cheers,
> >>> >> Jon
> >>> >>
> >>> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <sunilg@apache.org<mailto:
> >>> sunilg@apache.org>> wrote:
> >>> >>
> >>> >>> Thanks Robert and Haibo for quickly correcting same.
> >>> >>> Sigh, I somehow missed one file while committing the change. Sorry
> >>> for
> >>> >>> the
> >>> >>> trouble.
> >>> >>>
> >>> >>> - Sunil
> >>> >>>
> >>> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rkanter@cloudera.com
> >>> <ma...@cloudera.com>>
> >>> >>> wrote:
> >>> >>>
> >>> >>> > Looks like there's two that weren't updated:
> >>> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" .
> >>> -r
> >>> >>> > --include=pom.xml
> >>> >>> > ./hadoop-project/pom.xml:
> >>> >>> >
> >>> <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
> >>> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
> >>> >>> >
> >>> >>> > I've just pushed in an addendum commit to fix those.
> >>> >>> > In the future, please make sure to do a sanity compile when
> >>> updating
> >>> >>> poms.
> >>> >>> >
> >>> >>> > thanks
> >>> >>> > - Robert
> >>> >>> >
> >>> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
> >>> >>> <fa...@cloudera.com.invalid>>
> >>> >>> > wrote:
> >>> >>> >
> >>> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in
> >>> the
> >>> >>> >> top-level pom.xml?
> >>> >>> >>
> >>> >>> >>
> >>> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <sunilg@apache.org
> >>> <ma...@apache.org>> wrote:
> >>> >>> >>
> >>> >>> >> > Hi All
> >>> >>> >> >
> >>> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
> >>> trunk
> >>> >>> to
> >>> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
> >>> >>> necessary
> >>> >>> >> > patches are pulled into branch-3.2.
> >>> >>> >> >
> >>> >>> >> > Thank You
> >>> >>> >> > - Sunil
> >>> >>> >> >
> >>> >>> >> >
> >>> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <sunilg@apache.org
> >>> <ma...@apache.org>> wrote:
> >>> >>> >> >
> >>> >>> >> > > Hi All
> >>> >>> >> > >
> >>> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is
> >>> merged to
> >>> >>> >> trunk.
> >>> >>> >> > > Thanks for the support.
> >>> >>> >> > >
> >>> >>> >> > > *Plan for RC*
> >>> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
> >>> >>> >> > > RC0 for 3.2: *28th Friday*
> >>> >>> >> > >
> >>> >>> >> > > Thank You
> >>> >>> >> > > Sunil
> >>> >>> >> > >
> >>> >>> >> > >
> >>> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <sunilg@apache.org
> >>> <ma...@apache.org>>
> >>> >>> wrote:
> >>> >>> >> > >
> >>> >>> >> > >> Hi All
> >>> >>> >> > >>
> >>> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of
> >>> you
> >>> >>> for
> >>> >>> >> > >> helping in this. I am following up on these tickets, once
> its
> >>> >>> closed
> >>> >>> >> we
> >>> >>> >> > >> will cut the 3.2 branch.
> >>> >>> >> > >>
> >>> >>> >> > >> Thanks
> >>> >>> >> > >> Sunil Govindan
> >>> >>> >> > >>
> >>> >>> >> > >>
> >>> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <sunilg@apache.org
> >>> <ma...@apache.org>>
> >>> >>> wrote:
> >>> >>> >> > >>
> >>> >>> >> > >>> Hi All,
> >>> >>> >> > >>>
> >>> >>> >> > >>> Inline with the original 3.2 communication proposal dated
> >>> 17th
> >>> >>> July
> >>> >>> >> > >>> 2018, I would like to provide more updates.
> >>> >>> >> > >>>
> >>> >>> >> > >>> We are approaching previously proposed code freeze date
> >>> >>> (September
> >>> >>> >> 14,
> >>> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
> >>> point
> >>> >>> >> > existing
> >>> >>> >> > >>> trunk to 3.3 if there are no issues.
> >>> >>> >> > >>>
> >>> >>> >> > >>> *Current Release Plan:*
> >>> >>> >> > >>> Feature freeze date : all features to merge by September
> 7,
> >>> >>> 2018.
> >>> >>> >> > >>> Code freeze date : blockers/critical only, no improvements
> >>> and
> >>> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
> >>> >>> >> > >>> Release date: September 28, 2018
> >>> >>> >> > >>>
> >>> >>> >> > >>> If any critical/blocker tickets which are targeted to
> >>> 3.2.0, we
> >>> >>> >> need to
> >>> >>> >> > >>> backport to 3.2 post branch cut.
> >>> >>> >> > >>>
> >>> >>> >> > >>> Here's an updated 3.2.0 feature status:
> >>> >>> >> > >>>
> >>> >>> >> > >>> 1. Merged & Completed features:
> >>> >>> >> > >>>
> >>> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
> >>> DeepLearning
> >>> >>> >> > >>> workloads Initial cut.
> >>> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in
> >>> Capacity
> >>> >>> >> > >>> Scheduler.
> >>> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via
> YARN
> >>> >>> Service
> >>> >>> >> > API
> >>> >>> >> > >>> and CLI.
> >>> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
> >>> Improvement
> >>> >>> >> works.
> >>> >>> >> > >>>
> >>> >>> >> > >>> 2. Features close to finish:
> >>> >>> >> > >>>
> >>> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
> >>> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
> >>> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
> >>> >>> >> > >>>
> >>> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
> >>> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> >>> apps
> >>> >>> from
> >>> >>> >> > >>> ATSv2. Patch in progress.
> >>> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >>> >> challenging to
> >>> >>> >> > >>> be done before Aug 2018.
> >>> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN
> >>> applications.
> >>> >>> >> > >>> Challenging as more discussions are on-going.
> >>> >>> >> > >>>
> >>> >>> >> > >>> *Summary of 3.2.0 issues status:*
> >>> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am
> following
> >>> up
> >>> >>> with
> >>> >>> >> > >>> owners to get status on each of them to get in by Code
> >>> Freeze
> >>> >>> date.
> >>> >>> >> > >>>
> >>> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
> priority
> >>> in
> >>> >>> >> > (Blocker,
> >>> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target
> >>> Version/s" =
> >>> >>> 3.2.0
> >>> >>> >> > ORDER
> >>> >>> >> > >>> BY priority DESC
> >>> >>> >> > >>>
> >>> >>> >> > >>> Thanks,
> >>> >>> >> > >>> Sunil
> >>> >>> >> > >>>
> >>> >>> >> > >>>
> >>> >>> >> > >>>
> >>> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <
> sunilg@apache.org
> >>> <ma...@apache.org>>
> >>> >>> wrote:
> >>> >>> >> > >>>
> >>> >>> >> > >>>> Hi All,
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> Inline with earlier communication dated 17th July 2018, I
> >>> would
> >>> >>> >> like
> >>> >>> >> > to
> >>> >>> >> > >>>> provide some updates.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> We are approaching previously proposed code freeze date
> >>> (Aug
> >>> >>> 31).
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> One of the critical feature Node Attributes feature merge
> >>> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs
> >>> need a
> >>> >>> bit
> >>> >>> >> > more
> >>> >>> >> > >>>> time. With regard to this, suggesting to push the
> >>> feature/code
> >>> >>> >> freeze
> >>> >>> >> > for 2
> >>> >>> >> > >>>> more weeks to accommodate these jiras too.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> Proposing Updated changes in plan inline with this:
> >>> >>> >> > >>>> Feature freeze date : all features to merge by September
> 7,
> >>> >>> 2018.
> >>> >>> >> > >>>> Code freeze date : blockers/critical only, no
> improvements
> >>> and
> >>> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
> >>> >>> >> > >>>> Release date: September 28, 2018
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> If any features in branch which are targeted to 3.2.0,
> >>> please
> >>> >>> >> reply to
> >>> >>> >> > >>>> this email thread.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> 1. Merged & Completed features:
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> >>> DeepLearning
> >>> >>> >> > >>>> workloads Initial cut.
> >>> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> >>> Capacity
> >>> >>> >> > >>>> Scheduler.
> >>> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
> >>> YARN
> >>> >>> >> Service
> >>> >>> >> > >>>> API and CLI.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> 2. Features close to finish:
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in
> YARN.
> >>> >>> >> Merge/Vote
> >>> >>> >> > >>>> Ongoing.
> >>> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> >>> apps
> >>> >>> from
> >>> >>> >> > >>>> ATSv2. Patch in progress.
> >>> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> >>> Improvement
> >>> >>> >> > works.
> >>> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
> >>> Azure
> >>> >>> >> > >>>> Storage. In progress.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> 3. Tentative features:
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >>> >> challenging
> >>> >>> >> > to
> >>> >>> >> > >>>> be done before Aug 2018.
> >>> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
> >>> applications.
> >>> >>> >> > >>>> Challenging as more discussions are on-going.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> *Summary of 3.2.0 issues status:*
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am
> >>> following up
> >>> >>> with
> >>> >>> >> > >>>> owners to get status on each of them to get in by Code
> >>> Freeze
> >>> >>> date.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
> >>> priority in
> >>> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND
> "Target
> >>> >>> >> > Version/s" =
> >>> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> Thanks,
> >>> >>> >> > >>>> Sunil
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <
> >>> sunilg@apache.org<ma...@apache.org>>
> >>> >>> >> wrote:
> >>> >>> >> > >>>>
> >>> >>> >> > >>>>> Hi All,
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier
> >>> communication
> >>> >>> dated
> >>> >>> >> > 17th
> >>> >>> >> > >>>>> July 2018, I would like to provide some updates.
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> We are approaching previously proposed feature freeze
> date
> >>> >>> (Aug
> >>> >>> >> 21,
> >>> >>> >> > >>>>> about 7 days from today).
> >>> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
> >>> please
> >>> >>> >> reply
> >>> >>> >> > to
> >>> >>> >> > >>>>> this email thread.
> >>> >>> >> > >>>>> Steve has mentioned about the s3 features which will
> come
> >>> >>> close to
> >>> >>> >> > >>>>> Code Freeze Date (Aug 31st).
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> 1. Merged & Completed features:
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> >>> >>> DeepLearning
> >>> >>> >> > >>>>> workloads Initial cut.
> >>> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> 2. Features close to finish:
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in
> YARN.
> >>> >>> Major
> >>> >>> >> > >>>>> patches are all in, only one last
> >>> >>> >> > >>>>> patch is in review state.
> >>> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> >>> Capacity
> >>> >>> >> > >>>>> Scheduler. Close to commit.
> >>> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
> >>> YARN
> >>> >>> >> Service
> >>> >>> >> > >>>>> API and CLI. 2 patches are pending
> >>> >>> >> > >>>>> which will be closed by Feature freeze date.
> >>> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of
> historical
> >>> apps
> >>> >>> >> from
> >>> >>> >> > >>>>> ATSv2. Patch in progress.
> >>> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> >>> Improvement
> >>> >>> >> > >>>>> works.
> >>> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support
> Windows
> >>> >>> Azure
> >>> >>> >> > >>>>> Storage. In progress.
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> 3. Tentative features:
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >>> >> challenging
> >>> >>> >> > >>>>> to be done before Aug 2018.
> >>> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
> >>> applications.
> >>> >>> >> > >>>>> Challenging as more discussions are on-going.
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am
> checking
> >>> >>> with
> >>> >>> >> > owners
> >>> >>> >> > >>>>> to get status on each of them to get in by Code Freeze
> >>> date.
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
> >>> priority in
> >>> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND
> >>> "Target
> >>> >>> >> > Version/s" =
> >>> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> Thanks,
> >>> >>> >> > >>>>> Sunil
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <
> >>> sunilg@apache.org<ma...@apache.org>>
> >>> >>> >> wrote:
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>>> Thanks Subru for the thoughts.
> >>> >>> >> > >>>>>> One of the main reason for a major release is to push
> out
> >>> >>> >> critical
> >>> >>> >> > >>>>>> features with a faster cadence to the users. If we are
> >>> >>> pulling
> >>> >>> >> more
> >>> >>> >> > and
> >>> >>> >> > >>>>>> more different types of features to a minor release,
> that
> >>> >>> branch
> >>> >>> >> > will
> >>> >>> >> > >>>>>> become more destabilized and it may be tough to say
> that
> >>> >>> 3.1.2 is
> >>> >>> >> > stable
> >>> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and
> >>> stabilize
> >>> >>> >> features
> >>> >>> >> > in
> >>> >>> >> > >>>>>> subsequent minor release.
> >>> >>> >> > >>>>>> For few companies, it makes sense to push out these new
> >>> >>> features
> >>> >>> >> > >>>>>> faster to make a reach to the users. Adding to the
> point
> >>> to
> >>> >>> the
> >>> >>> >> > backporting
> >>> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround
> >>> that
> >>> >>> with
> >>> >>> >> > some git
> >>> >>> >> > >>>>>> scripts. If we can make such scripts available to
> >>> committers,
> >>> >>> >> > backport will
> >>> >>> >> > >>>>>> be seem-less across branches and we can achieve the
> >>> faster
> >>> >>> >> release
> >>> >>> >> > cadence
> >>> >>> >> > >>>>>> also.
> >>> >>> >> > >>>>>>
> >>> >>> >> > >>>>>> Thoughts?
> >>> >>> >> > >>>>>>
> >>> >>> >> > >>>>>> - Sunil
> >>> >>> >> > >>>>>>
> >>> >>> >> > >>>>>>
> >>> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
> >>> >>> subru@apache.org<ma...@apache.org>
> >>> >>> >> >
> >>> >>> >> > >>>>>> wrote:
> >>> >>> >> > >>>>>>
> >>> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release
> >>> effort. I
> >>> >>> am
> >>> >>> >> > >>>>>>> generally
> >>> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a
> >>> 3.1.x) as
> >>> >>> >> feel we
> >>> >>> >> > >>>>>>> already
> >>> >>> >> > >>>>>>> have too many branches to be maintained. I already see
> >>> many
> >>> >>> >> commits
> >>> >>> >> > >>>>>>> are in
> >>> >>> >> > >>>>>>> different branches with no apparent rationale, for
> e.g:
> >>> 3.1
> >>> >>> has
> >>> >>> >> > >>>>>>> commits
> >>> >>> >> > >>>>>>> which are absent in 3.0 etc.
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any
> >>> major
> >>> >>> >> > production
> >>> >>> >> > >>>>>>> setting so the cost of adding features should be
> >>> minimal.
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>> Thoughts?
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>> -Subru
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
> >>> >>> sunilg@apache.org<ma...@apache.org>>
> >>> >>> >> > wrote:
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > Yes, important changes and features are much needed,
> >>> >>> hence we
> >>> >>> >> > will
> >>> >>> >> > >>>>>>> be
> >>> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
> >>> >>> considering
> >>> >>> >> few
> >>> >>> >> > >>>>>>> more
> >>> >>> >> > >>>>>>> > offline requests from other folks, I think extending
> >>> the
> >>> >>> >> > timeframe
> >>> >>> >> > >>>>>>> by
> >>> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
> >>> >>> buffer) and
> >>> >>> >> > >>>>>>> this should
> >>> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap
> >>> wiki as
> >>> >>> >> well)
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by
> >>> August
> >>> >>> 21,
> >>> >>> >> 2018.
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
> >>> >>> improvements
> >>> >>> >> and
> >>> >>> >> > >>>>>>> non
> >>> >>> >> > >>>>>>> > blocker/critical
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > - Release date: September 15, 2018
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already
> >>> answered
> >>> >>> your
> >>> >>> >> > >>>>>>> questions.
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > Thanks
> >>> >>> >> > >>>>>>> > Sunil
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
> >>> >>> >> wheeleast@gmail.com<ma...@gmail.com>
> >>> >>> >> > >
> >>> >>> >> > >>>>>>> wrote:
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2
> >>> release,
> >>> >>> +1
> >>> >>> >> for
> >>> >>> >> > >>>>>>> that.
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > To concerns from Steve,
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
> >>> important
> >>> >>> >> > changes
> >>> >>> >> > >>>>>>> /
> >>> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep
> >>> the
> >>> >>> >> proposed
> >>> >>> >> > >>>>>>> release
> >>> >>> >> > >>>>>>> > date
> >>> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of
> >>> last
> >>> >>> >> minute
> >>> >>> >> > >>>>>>> and we all
> >>> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm
> >>> also
> >>> >>> fine
> >>> >>> >> if
> >>> >>> >> > >>>>>>> we want
> >>> >>> >> > >>>>>>> > get
> >>> >>> >> > >>>>>>> > > another several weeks time.
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing
> that
> >>> >>> before
> >>> >>> >> > >>>>>>> thanksgiving.
> >>> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > Eric,
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
> >>> >>> YARN-8135
> >>> >>> >> can
> >>> >>> >> > >>>>>>> get merged
> >>> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > Sunil,
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > Could u update
> >>> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
> >>> >>> >> > >>>>>>> > Roadmap
> >>> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature
> list
> >>> >>> first
> >>> >>> >> > >>>>>>> before getting
> >>> >>> >> > >>>>>>> > > consensus of time.
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > Thanks,
> >>> >>> >> > >>>>>>> > > Wangda
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> >>> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid<mailto:
> >>> fabbri@cloudera.com.invalid>
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > wrote:
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> >>> >>> >> > >>>>>>> stevel@hortonworks.com<mailto:stevel@hortonworks.com
> >>
> >>> >>> >> > >>>>>>> > >> wrote:
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
> >>> sunilg@apache.org<ma...@apache.org>
> >>> >>> >> > <mailto:
> >>> >>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>>
> >>> wrote:
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > I would also would like to take this
> opportunity
> >>> to
> >>> >>> come
> >>> >>> >> up
> >>> >>> >> > >>>>>>> with a
> >>> >>> >> > >>>>>>> > >> detailed
> >>> >>> >> > >>>>>>> > >> > plan.
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
> >>> >>> merged by
> >>> >>> >> > >>>>>>> August 10,
> >>> >>> >> > >>>>>>> > >> 2018.
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > <snip>
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
> >>> targeted
> >>> >>> to
> >>> >>> >> 3.2
> >>> >>> >> > >>>>>>> per this
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 &
> S3Guard.
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> https://issues.apache.org/jira/browse/HADOOP-15226
> >>> >>> >> > >>>>>>> > >> >
> >>> https://issues.apache.org/jira/browse/HADOOP-15220
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for
> Azure
> >>> >>> Datalake
> >>> >>> >> > Gen
> >>> >>> >> > >>>>>>> 2
> >>> >>> >> > >>>>>>> > >> >
> >>> https://issues.apache.org/jira/browse/HADOOP-15407
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've
> been
> >>> >>> doing
> >>> >>> >> work
> >>> >>> >> > >>>>>>> on it, but
> >>> >>> >> > >>>>>>> > >> if
> >>> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect
> >>> some
> >>> >>> >> timely
> >>> >>> >> > >>>>>>> reviews on
> >>> >>> >> > >>>>>>> > >> >
> >>> https://issues.apache.org/jira/browse/HADOOP-15546
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will
> >>> review
> >>> >>> the
> >>> >>> >> S3 &
> >>> >>> >> > >>>>>>> S3Guard
> >>> >>> >> > >>>>>>> > >> JIRAs
> >>> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there
> >>> are
> >>> >>> some
> >>> >>> >> > >>>>>>> pressing
> >>> >>> >> > >>>>>>> > "guave,
> >>> >>> >> > >>>>>>> > >> > java 9 prep"
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> I can help with this part if you like.
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
> >>> release
> >>> >>> >> > manager
> >>> >>> >> > >>>>>>> of 3.2.0
> >>> >>> >> > >>>>>>> > >> > release.
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > well volunteered!
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable
> >>> should
> >>> >>> we
> >>> >>> >> have
> >>> >>> >> > >>>>>>> for the
> >>> >>> >> > >>>>>>> > >> 3.2. &
> >>> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence,
> >>> then
> >>> >>> having
> >>> >>> >> > the
> >>> >>> >> > >>>>>>> outline
> >>> >>> >> > >>>>>>> > >> time
> >>> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that
> >>> there's
> >>> >>> less
> >>> >>> >> > >>>>>>> concern about
> >>> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > -Steve
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >> -AF
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>
> >>> >>> >> >
> >>> >>> >>
> >>> >>> >
> >>> >>>
> >>> >>
> >>> >>
> >>>
> >>>
> >>> --
> >>> A very happy Clouderan
> >>>
> >>>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi Folks,
All blockers are closed by last weekend and corrected jiras as well.
Preparing RC0 now. Facing some issue with shaded jars file size. Hence
respinning again.
Planning to complete this by end of this week.
Thanks
Sunil
On Fri, Oct 26, 2018 at 2:34 AM Konstantin Shvachko <sh...@gmail.com>
wrote:
> Another thing is that I see a bunch of jiras under HDFS-8707, which don't
> have the Fix Version field listing 3.2, some have it just empty.
> This means they will not be populated into release notes.
>
> Thanks,
> --Konstantin
>
> On Thu, Oct 25, 2018 at 7:59 AM Sunil G <su...@apache.org> wrote:
>
> > Thanks Konstantin for pointing out.
> > As 3.2 is pretty much on RC level, its better we try to find a good
> > solution to this issue.
> >
> > I ll follow up on this in the jira.
> >
> > - Sunil
> >
> > On Thu, Oct 25, 2018 at 11:35 AM Konstantin Shvachko <
> shv.hadoop@gmail.com>
> > wrote:
> >
> >> I've tried to attract attention to an incompatibility issue through the
> >> jira, but it didn't work. So pitching in in this thread.
> >> https://issues.apache.org/jira/browse/HDFS-12026
> >> It introduced binary incompatibility, which will prevent people from
> >> upgrading from 3.1 to 3.2.
> >> I think it can get messy if we release anything with this feature.
> >>
> >> Thanks,
> >> --Konstantin
> >>
> >> On Mon, Oct 22, 2018 at 5:01 AM Steve Loughran <st...@hortonworks.com>
> >> wrote:
> >>
> >>> its in.
> >>>
> >>> good catch!
> >>>
> >>> On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <weichiu@cloudera.com
> <mailto:
> >>> weichiu@cloudera.com>> wrote:
> >>>
> >>> Thanks Sunil G for driving the release,
> >>> I filed HADOOP-15866<
> https://issues.apache.org/jira/browse/HADOOP-15866>
> >>> for a compat fix. If any one has cycle please review it, as I think it
> is
> >>> needed for 3.2.0.
> >>>
> >>> On Thu, Oct 18, 2018 at 4:43 AM Sunil G <sunilg@apache.org<mailto:
> >>> sunilg@apache.org>> wrote:
> >>> Hi Folks,
> >>>
> >>> As we previously communicated for 3.2.0 release, we have delayed due to
> >>> few
> >>> blockers in our gate.
> >>>
> >>> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open
> for
> >>> all bug fixes.
> >>>
> >>> - Sunil
> >>>
> >>>
> >>> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <sunilg@apache.org<mailto:
> >>> sunilg@apache.org>> wrote:
> >>>
> >>> > Hi Folks,
> >>> >
> >>> > We are now close to RC as other blocker issues are now merged to
> trunk
> >>> and
> >>> > branch-3.2. Last 2 critical issues are closer to merge and will be
> >>> > committed in few hours.
> >>> > With this, I will be creating 3.2.0 branch today and will go ahead
> >>> with RC
> >>> > related process.
> >>> >
> >>> > - Sunil
> >>> >
> >>> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <
> jonbender@stripe.com
> >>> <ma...@stripe.com>>
> >>> > wrote:
> >>> >
> >>> >> Hello, were there any updates around the 3.2.0 RC timing? All I see
> in
> >>> >> the current blockers are related to the new Submarine subproject,
> >>> wasn't
> >>> >> sure if that is what is holding things up.
> >>> >>
> >>> >> Cheers,
> >>> >> Jon
> >>> >>
> >>> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <sunilg@apache.org<mailto:
> >>> sunilg@apache.org>> wrote:
> >>> >>
> >>> >>> Thanks Robert and Haibo for quickly correcting same.
> >>> >>> Sigh, I somehow missed one file while committing the change. Sorry
> >>> for
> >>> >>> the
> >>> >>> trouble.
> >>> >>>
> >>> >>> - Sunil
> >>> >>>
> >>> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rkanter@cloudera.com
> >>> <ma...@cloudera.com>>
> >>> >>> wrote:
> >>> >>>
> >>> >>> > Looks like there's two that weren't updated:
> >>> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" .
> >>> -r
> >>> >>> > --include=pom.xml
> >>> >>> > ./hadoop-project/pom.xml:
> >>> >>> >
> >>> <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
> >>> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
> >>> >>> >
> >>> >>> > I've just pushed in an addendum commit to fix those.
> >>> >>> > In the future, please make sure to do a sanity compile when
> >>> updating
> >>> >>> poms.
> >>> >>> >
> >>> >>> > thanks
> >>> >>> > - Robert
> >>> >>> >
> >>> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
> >>> >>> <fa...@cloudera.com.invalid>>
> >>> >>> > wrote:
> >>> >>> >
> >>> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in
> >>> the
> >>> >>> >> top-level pom.xml?
> >>> >>> >>
> >>> >>> >>
> >>> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <sunilg@apache.org
> >>> <ma...@apache.org>> wrote:
> >>> >>> >>
> >>> >>> >> > Hi All
> >>> >>> >> >
> >>> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
> >>> trunk
> >>> >>> to
> >>> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
> >>> >>> necessary
> >>> >>> >> > patches are pulled into branch-3.2.
> >>> >>> >> >
> >>> >>> >> > Thank You
> >>> >>> >> > - Sunil
> >>> >>> >> >
> >>> >>> >> >
> >>> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <sunilg@apache.org
> >>> <ma...@apache.org>> wrote:
> >>> >>> >> >
> >>> >>> >> > > Hi All
> >>> >>> >> > >
> >>> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is
> >>> merged to
> >>> >>> >> trunk.
> >>> >>> >> > > Thanks for the support.
> >>> >>> >> > >
> >>> >>> >> > > *Plan for RC*
> >>> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
> >>> >>> >> > > RC0 for 3.2: *28th Friday*
> >>> >>> >> > >
> >>> >>> >> > > Thank You
> >>> >>> >> > > Sunil
> >>> >>> >> > >
> >>> >>> >> > >
> >>> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <sunilg@apache.org
> >>> <ma...@apache.org>>
> >>> >>> wrote:
> >>> >>> >> > >
> >>> >>> >> > >> Hi All
> >>> >>> >> > >>
> >>> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of
> >>> you
> >>> >>> for
> >>> >>> >> > >> helping in this. I am following up on these tickets, once
> its
> >>> >>> closed
> >>> >>> >> we
> >>> >>> >> > >> will cut the 3.2 branch.
> >>> >>> >> > >>
> >>> >>> >> > >> Thanks
> >>> >>> >> > >> Sunil Govindan
> >>> >>> >> > >>
> >>> >>> >> > >>
> >>> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <sunilg@apache.org
> >>> <ma...@apache.org>>
> >>> >>> wrote:
> >>> >>> >> > >>
> >>> >>> >> > >>> Hi All,
> >>> >>> >> > >>>
> >>> >>> >> > >>> Inline with the original 3.2 communication proposal dated
> >>> 17th
> >>> >>> July
> >>> >>> >> > >>> 2018, I would like to provide more updates.
> >>> >>> >> > >>>
> >>> >>> >> > >>> We are approaching previously proposed code freeze date
> >>> >>> (September
> >>> >>> >> 14,
> >>> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
> >>> point
> >>> >>> >> > existing
> >>> >>> >> > >>> trunk to 3.3 if there are no issues.
> >>> >>> >> > >>>
> >>> >>> >> > >>> *Current Release Plan:*
> >>> >>> >> > >>> Feature freeze date : all features to merge by September
> 7,
> >>> >>> 2018.
> >>> >>> >> > >>> Code freeze date : blockers/critical only, no improvements
> >>> and
> >>> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
> >>> >>> >> > >>> Release date: September 28, 2018
> >>> >>> >> > >>>
> >>> >>> >> > >>> If any critical/blocker tickets which are targeted to
> >>> 3.2.0, we
> >>> >>> >> need to
> >>> >>> >> > >>> backport to 3.2 post branch cut.
> >>> >>> >> > >>>
> >>> >>> >> > >>> Here's an updated 3.2.0 feature status:
> >>> >>> >> > >>>
> >>> >>> >> > >>> 1. Merged & Completed features:
> >>> >>> >> > >>>
> >>> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
> >>> DeepLearning
> >>> >>> >> > >>> workloads Initial cut.
> >>> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in
> >>> Capacity
> >>> >>> >> > >>> Scheduler.
> >>> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via
> YARN
> >>> >>> Service
> >>> >>> >> > API
> >>> >>> >> > >>> and CLI.
> >>> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
> >>> Improvement
> >>> >>> >> works.
> >>> >>> >> > >>>
> >>> >>> >> > >>> 2. Features close to finish:
> >>> >>> >> > >>>
> >>> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
> >>> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
> >>> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
> >>> >>> >> > >>>
> >>> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
> >>> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> >>> apps
> >>> >>> from
> >>> >>> >> > >>> ATSv2. Patch in progress.
> >>> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >>> >> challenging to
> >>> >>> >> > >>> be done before Aug 2018.
> >>> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN
> >>> applications.
> >>> >>> >> > >>> Challenging as more discussions are on-going.
> >>> >>> >> > >>>
> >>> >>> >> > >>> *Summary of 3.2.0 issues status:*
> >>> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am
> following
> >>> up
> >>> >>> with
> >>> >>> >> > >>> owners to get status on each of them to get in by Code
> >>> Freeze
> >>> >>> date.
> >>> >>> >> > >>>
> >>> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
> priority
> >>> in
> >>> >>> >> > (Blocker,
> >>> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target
> >>> Version/s" =
> >>> >>> 3.2.0
> >>> >>> >> > ORDER
> >>> >>> >> > >>> BY priority DESC
> >>> >>> >> > >>>
> >>> >>> >> > >>> Thanks,
> >>> >>> >> > >>> Sunil
> >>> >>> >> > >>>
> >>> >>> >> > >>>
> >>> >>> >> > >>>
> >>> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <
> sunilg@apache.org
> >>> <ma...@apache.org>>
> >>> >>> wrote:
> >>> >>> >> > >>>
> >>> >>> >> > >>>> Hi All,
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> Inline with earlier communication dated 17th July 2018, I
> >>> would
> >>> >>> >> like
> >>> >>> >> > to
> >>> >>> >> > >>>> provide some updates.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> We are approaching previously proposed code freeze date
> >>> (Aug
> >>> >>> 31).
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> One of the critical feature Node Attributes feature merge
> >>> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs
> >>> need a
> >>> >>> bit
> >>> >>> >> > more
> >>> >>> >> > >>>> time. With regard to this, suggesting to push the
> >>> feature/code
> >>> >>> >> freeze
> >>> >>> >> > for 2
> >>> >>> >> > >>>> more weeks to accommodate these jiras too.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> Proposing Updated changes in plan inline with this:
> >>> >>> >> > >>>> Feature freeze date : all features to merge by September
> 7,
> >>> >>> 2018.
> >>> >>> >> > >>>> Code freeze date : blockers/critical only, no
> improvements
> >>> and
> >>> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
> >>> >>> >> > >>>> Release date: September 28, 2018
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> If any features in branch which are targeted to 3.2.0,
> >>> please
> >>> >>> >> reply to
> >>> >>> >> > >>>> this email thread.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> 1. Merged & Completed features:
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> >>> DeepLearning
> >>> >>> >> > >>>> workloads Initial cut.
> >>> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> >>> Capacity
> >>> >>> >> > >>>> Scheduler.
> >>> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
> >>> YARN
> >>> >>> >> Service
> >>> >>> >> > >>>> API and CLI.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> 2. Features close to finish:
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in
> YARN.
> >>> >>> >> Merge/Vote
> >>> >>> >> > >>>> Ongoing.
> >>> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> >>> apps
> >>> >>> from
> >>> >>> >> > >>>> ATSv2. Patch in progress.
> >>> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> >>> Improvement
> >>> >>> >> > works.
> >>> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
> >>> Azure
> >>> >>> >> > >>>> Storage. In progress.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> 3. Tentative features:
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >>> >> challenging
> >>> >>> >> > to
> >>> >>> >> > >>>> be done before Aug 2018.
> >>> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
> >>> applications.
> >>> >>> >> > >>>> Challenging as more discussions are on-going.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> *Summary of 3.2.0 issues status:*
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am
> >>> following up
> >>> >>> with
> >>> >>> >> > >>>> owners to get status on each of them to get in by Code
> >>> Freeze
> >>> >>> date.
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
> >>> priority in
> >>> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND
> "Target
> >>> >>> >> > Version/s" =
> >>> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> Thanks,
> >>> >>> >> > >>>> Sunil
> >>> >>> >> > >>>>
> >>> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <
> >>> sunilg@apache.org<ma...@apache.org>>
> >>> >>> >> wrote:
> >>> >>> >> > >>>>
> >>> >>> >> > >>>>> Hi All,
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier
> >>> communication
> >>> >>> dated
> >>> >>> >> > 17th
> >>> >>> >> > >>>>> July 2018, I would like to provide some updates.
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> We are approaching previously proposed feature freeze
> date
> >>> >>> (Aug
> >>> >>> >> 21,
> >>> >>> >> > >>>>> about 7 days from today).
> >>> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
> >>> please
> >>> >>> >> reply
> >>> >>> >> > to
> >>> >>> >> > >>>>> this email thread.
> >>> >>> >> > >>>>> Steve has mentioned about the s3 features which will
> come
> >>> >>> close to
> >>> >>> >> > >>>>> Code Freeze Date (Aug 31st).
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> 1. Merged & Completed features:
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> >>> >>> DeepLearning
> >>> >>> >> > >>>>> workloads Initial cut.
> >>> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> 2. Features close to finish:
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in
> YARN.
> >>> >>> Major
> >>> >>> >> > >>>>> patches are all in, only one last
> >>> >>> >> > >>>>> patch is in review state.
> >>> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> >>> Capacity
> >>> >>> >> > >>>>> Scheduler. Close to commit.
> >>> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
> >>> YARN
> >>> >>> >> Service
> >>> >>> >> > >>>>> API and CLI. 2 patches are pending
> >>> >>> >> > >>>>> which will be closed by Feature freeze date.
> >>> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of
> historical
> >>> apps
> >>> >>> >> from
> >>> >>> >> > >>>>> ATSv2. Patch in progress.
> >>> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> >>> Improvement
> >>> >>> >> > >>>>> works.
> >>> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support
> Windows
> >>> >>> Azure
> >>> >>> >> > >>>>> Storage. In progress.
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> 3. Tentative features:
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >>> >> challenging
> >>> >>> >> > >>>>> to be done before Aug 2018.
> >>> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
> >>> applications.
> >>> >>> >> > >>>>> Challenging as more discussions are on-going.
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am
> checking
> >>> >>> with
> >>> >>> >> > owners
> >>> >>> >> > >>>>> to get status on each of them to get in by Code Freeze
> >>> date.
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
> >>> priority in
> >>> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND
> >>> "Target
> >>> >>> >> > Version/s" =
> >>> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> Thanks,
> >>> >>> >> > >>>>> Sunil
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <
> >>> sunilg@apache.org<ma...@apache.org>>
> >>> >>> >> wrote:
> >>> >>> >> > >>>>>
> >>> >>> >> > >>>>>> Thanks Subru for the thoughts.
> >>> >>> >> > >>>>>> One of the main reason for a major release is to push
> out
> >>> >>> >> critical
> >>> >>> >> > >>>>>> features with a faster cadence to the users. If we are
> >>> >>> pulling
> >>> >>> >> more
> >>> >>> >> > and
> >>> >>> >> > >>>>>> more different types of features to a minor release,
> that
> >>> >>> branch
> >>> >>> >> > will
> >>> >>> >> > >>>>>> become more destabilized and it may be tough to say
> that
> >>> >>> 3.1.2 is
> >>> >>> >> > stable
> >>> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and
> >>> stabilize
> >>> >>> >> features
> >>> >>> >> > in
> >>> >>> >> > >>>>>> subsequent minor release.
> >>> >>> >> > >>>>>> For few companies, it makes sense to push out these new
> >>> >>> features
> >>> >>> >> > >>>>>> faster to make a reach to the users. Adding to the
> point
> >>> to
> >>> >>> the
> >>> >>> >> > backporting
> >>> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround
> >>> that
> >>> >>> with
> >>> >>> >> > some git
> >>> >>> >> > >>>>>> scripts. If we can make such scripts available to
> >>> committers,
> >>> >>> >> > backport will
> >>> >>> >> > >>>>>> be seem-less across branches and we can achieve the
> >>> faster
> >>> >>> >> release
> >>> >>> >> > cadence
> >>> >>> >> > >>>>>> also.
> >>> >>> >> > >>>>>>
> >>> >>> >> > >>>>>> Thoughts?
> >>> >>> >> > >>>>>>
> >>> >>> >> > >>>>>> - Sunil
> >>> >>> >> > >>>>>>
> >>> >>> >> > >>>>>>
> >>> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
> >>> >>> subru@apache.org<ma...@apache.org>
> >>> >>> >> >
> >>> >>> >> > >>>>>> wrote:
> >>> >>> >> > >>>>>>
> >>> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release
> >>> effort. I
> >>> >>> am
> >>> >>> >> > >>>>>>> generally
> >>> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a
> >>> 3.1.x) as
> >>> >>> >> feel we
> >>> >>> >> > >>>>>>> already
> >>> >>> >> > >>>>>>> have too many branches to be maintained. I already see
> >>> many
> >>> >>> >> commits
> >>> >>> >> > >>>>>>> are in
> >>> >>> >> > >>>>>>> different branches with no apparent rationale, for
> e.g:
> >>> 3.1
> >>> >>> has
> >>> >>> >> > >>>>>>> commits
> >>> >>> >> > >>>>>>> which are absent in 3.0 etc.
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any
> >>> major
> >>> >>> >> > production
> >>> >>> >> > >>>>>>> setting so the cost of adding features should be
> >>> minimal.
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>> Thoughts?
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>> -Subru
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
> >>> >>> sunilg@apache.org<ma...@apache.org>>
> >>> >>> >> > wrote:
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > Yes, important changes and features are much needed,
> >>> >>> hence we
> >>> >>> >> > will
> >>> >>> >> > >>>>>>> be
> >>> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
> >>> >>> considering
> >>> >>> >> few
> >>> >>> >> > >>>>>>> more
> >>> >>> >> > >>>>>>> > offline requests from other folks, I think extending
> >>> the
> >>> >>> >> > timeframe
> >>> >>> >> > >>>>>>> by
> >>> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
> >>> >>> buffer) and
> >>> >>> >> > >>>>>>> this should
> >>> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap
> >>> wiki as
> >>> >>> >> well)
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by
> >>> August
> >>> >>> 21,
> >>> >>> >> 2018.
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
> >>> >>> improvements
> >>> >>> >> and
> >>> >>> >> > >>>>>>> non
> >>> >>> >> > >>>>>>> > blocker/critical
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > - Release date: September 15, 2018
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already
> >>> answered
> >>> >>> your
> >>> >>> >> > >>>>>>> questions.
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > Thanks
> >>> >>> >> > >>>>>>> > Sunil
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
> >>> >>> >> wheeleast@gmail.com<ma...@gmail.com>
> >>> >>> >> > >
> >>> >>> >> > >>>>>>> wrote:
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2
> >>> release,
> >>> >>> +1
> >>> >>> >> for
> >>> >>> >> > >>>>>>> that.
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > To concerns from Steve,
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
> >>> important
> >>> >>> >> > changes
> >>> >>> >> > >>>>>>> /
> >>> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep
> >>> the
> >>> >>> >> proposed
> >>> >>> >> > >>>>>>> release
> >>> >>> >> > >>>>>>> > date
> >>> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of
> >>> last
> >>> >>> >> minute
> >>> >>> >> > >>>>>>> and we all
> >>> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm
> >>> also
> >>> >>> fine
> >>> >>> >> if
> >>> >>> >> > >>>>>>> we want
> >>> >>> >> > >>>>>>> > get
> >>> >>> >> > >>>>>>> > > another several weeks time.
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing
> that
> >>> >>> before
> >>> >>> >> > >>>>>>> thanksgiving.
> >>> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > Eric,
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
> >>> >>> YARN-8135
> >>> >>> >> can
> >>> >>> >> > >>>>>>> get merged
> >>> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > Sunil,
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > Could u update
> >>> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
> >>> >>> >> > >>>>>>> > Roadmap
> >>> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature
> list
> >>> >>> first
> >>> >>> >> > >>>>>>> before getting
> >>> >>> >> > >>>>>>> > > consensus of time.
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > Thanks,
> >>> >>> >> > >>>>>>> > > Wangda
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> >>> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid<mailto:
> >>> fabbri@cloudera.com.invalid>
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > > wrote:
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> >>> >>> >> > >>>>>>> stevel@hortonworks.com<mailto:stevel@hortonworks.com
> >>
> >>> >>> >> > >>>>>>> > >> wrote:
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
> >>> sunilg@apache.org<ma...@apache.org>
> >>> >>> >> > <mailto:
> >>> >>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>>
> >>> wrote:
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > I would also would like to take this
> opportunity
> >>> to
> >>> >>> come
> >>> >>> >> up
> >>> >>> >> > >>>>>>> with a
> >>> >>> >> > >>>>>>> > >> detailed
> >>> >>> >> > >>>>>>> > >> > plan.
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
> >>> >>> merged by
> >>> >>> >> > >>>>>>> August 10,
> >>> >>> >> > >>>>>>> > >> 2018.
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > <snip>
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
> >>> targeted
> >>> >>> to
> >>> >>> >> 3.2
> >>> >>> >> > >>>>>>> per this
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 &
> S3Guard.
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> https://issues.apache.org/jira/browse/HADOOP-15226
> >>> >>> >> > >>>>>>> > >> >
> >>> https://issues.apache.org/jira/browse/HADOOP-15220
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for
> Azure
> >>> >>> Datalake
> >>> >>> >> > Gen
> >>> >>> >> > >>>>>>> 2
> >>> >>> >> > >>>>>>> > >> >
> >>> https://issues.apache.org/jira/browse/HADOOP-15407
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've
> been
> >>> >>> doing
> >>> >>> >> work
> >>> >>> >> > >>>>>>> on it, but
> >>> >>> >> > >>>>>>> > >> if
> >>> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect
> >>> some
> >>> >>> >> timely
> >>> >>> >> > >>>>>>> reviews on
> >>> >>> >> > >>>>>>> > >> >
> >>> https://issues.apache.org/jira/browse/HADOOP-15546
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will
> >>> review
> >>> >>> the
> >>> >>> >> S3 &
> >>> >>> >> > >>>>>>> S3Guard
> >>> >>> >> > >>>>>>> > >> JIRAs
> >>> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there
> >>> are
> >>> >>> some
> >>> >>> >> > >>>>>>> pressing
> >>> >>> >> > >>>>>>> > "guave,
> >>> >>> >> > >>>>>>> > >> > java 9 prep"
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> I can help with this part if you like.
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
> >>> release
> >>> >>> >> > manager
> >>> >>> >> > >>>>>>> of 3.2.0
> >>> >>> >> > >>>>>>> > >> > release.
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > well volunteered!
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable
> >>> should
> >>> >>> we
> >>> >>> >> have
> >>> >>> >> > >>>>>>> for the
> >>> >>> >> > >>>>>>> > >> 3.2. &
> >>> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence,
> >>> then
> >>> >>> having
> >>> >>> >> > the
> >>> >>> >> > >>>>>>> outline
> >>> >>> >> > >>>>>>> > >> time
> >>> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that
> >>> there's
> >>> >>> less
> >>> >>> >> > >>>>>>> concern about
> >>> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> > -Steve
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> >
> >>> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >> -AF
> >>> >>> >> > >>>>>>> > >>
> >>> >>> >> > >>>>>>> > >
> >>> >>> >> > >>>>>>> >
> >>> >>> >> > >>>>>>>
> >>> >>> >> > >>>>>>
> >>> >>> >> >
> >>> >>> >>
> >>> >>> >
> >>> >>>
> >>> >>
> >>> >>
> >>>
> >>>
> >>> --
> >>> A very happy Clouderan
> >>>
> >>>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Konstantin Shvachko <sh...@gmail.com>.
Another thing is that I see a bunch of jiras under HDFS-8707, which don't
have the Fix Version field listing 3.2, some have it just empty.
This means they will not be populated into release notes.
Thanks,
--Konstantin
On Thu, Oct 25, 2018 at 7:59 AM Sunil G <su...@apache.org> wrote:
> Thanks Konstantin for pointing out.
> As 3.2 is pretty much on RC level, its better we try to find a good
> solution to this issue.
>
> I ll follow up on this in the jira.
>
> - Sunil
>
> On Thu, Oct 25, 2018 at 11:35 AM Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
>> I've tried to attract attention to an incompatibility issue through the
>> jira, but it didn't work. So pitching in in this thread.
>> https://issues.apache.org/jira/browse/HDFS-12026
>> It introduced binary incompatibility, which will prevent people from
>> upgrading from 3.1 to 3.2.
>> I think it can get messy if we release anything with this feature.
>>
>> Thanks,
>> --Konstantin
>>
>> On Mon, Oct 22, 2018 at 5:01 AM Steve Loughran <st...@hortonworks.com>
>> wrote:
>>
>>> its in.
>>>
>>> good catch!
>>>
>>> On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <weichiu@cloudera.com<mailto:
>>> weichiu@cloudera.com>> wrote:
>>>
>>> Thanks Sunil G for driving the release,
>>> I filed HADOOP-15866<https://issues.apache.org/jira/browse/HADOOP-15866>
>>> for a compat fix. If any one has cycle please review it, as I think it is
>>> needed for 3.2.0.
>>>
>>> On Thu, Oct 18, 2018 at 4:43 AM Sunil G <sunilg@apache.org<mailto:
>>> sunilg@apache.org>> wrote:
>>> Hi Folks,
>>>
>>> As we previously communicated for 3.2.0 release, we have delayed due to
>>> few
>>> blockers in our gate.
>>>
>>> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
>>> all bug fixes.
>>>
>>> - Sunil
>>>
>>>
>>> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <sunilg@apache.org<mailto:
>>> sunilg@apache.org>> wrote:
>>>
>>> > Hi Folks,
>>> >
>>> > We are now close to RC as other blocker issues are now merged to trunk
>>> and
>>> > branch-3.2. Last 2 critical issues are closer to merge and will be
>>> > committed in few hours.
>>> > With this, I will be creating 3.2.0 branch today and will go ahead
>>> with RC
>>> > related process.
>>> >
>>> > - Sunil
>>> >
>>> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jonbender@stripe.com
>>> <ma...@stripe.com>>
>>> > wrote:
>>> >
>>> >> Hello, were there any updates around the 3.2.0 RC timing? All I see in
>>> >> the current blockers are related to the new Submarine subproject,
>>> wasn't
>>> >> sure if that is what is holding things up.
>>> >>
>>> >> Cheers,
>>> >> Jon
>>> >>
>>> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <sunilg@apache.org<mailto:
>>> sunilg@apache.org>> wrote:
>>> >>
>>> >>> Thanks Robert and Haibo for quickly correcting same.
>>> >>> Sigh, I somehow missed one file while committing the change. Sorry
>>> for
>>> >>> the
>>> >>> trouble.
>>> >>>
>>> >>> - Sunil
>>> >>>
>>> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rkanter@cloudera.com
>>> <ma...@cloudera.com>>
>>> >>> wrote:
>>> >>>
>>> >>> > Looks like there's two that weren't updated:
>>> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" .
>>> -r
>>> >>> > --include=pom.xml
>>> >>> > ./hadoop-project/pom.xml:
>>> >>> >
>>> <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>>> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>>> >>> >
>>> >>> > I've just pushed in an addendum commit to fix those.
>>> >>> > In the future, please make sure to do a sanity compile when
>>> updating
>>> >>> poms.
>>> >>> >
>>> >>> > thanks
>>> >>> > - Robert
>>> >>> >
>>> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>>> >>> <fa...@cloudera.com.invalid>>
>>> >>> > wrote:
>>> >>> >
>>> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in
>>> the
>>> >>> >> top-level pom.xml?
>>> >>> >>
>>> >>> >>
>>> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <sunilg@apache.org
>>> <ma...@apache.org>> wrote:
>>> >>> >>
>>> >>> >> > Hi All
>>> >>> >> >
>>> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
>>> trunk
>>> >>> to
>>> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
>>> >>> necessary
>>> >>> >> > patches are pulled into branch-3.2.
>>> >>> >> >
>>> >>> >> > Thank You
>>> >>> >> > - Sunil
>>> >>> >> >
>>> >>> >> >
>>> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <sunilg@apache.org
>>> <ma...@apache.org>> wrote:
>>> >>> >> >
>>> >>> >> > > Hi All
>>> >>> >> > >
>>> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is
>>> merged to
>>> >>> >> trunk.
>>> >>> >> > > Thanks for the support.
>>> >>> >> > >
>>> >>> >> > > *Plan for RC*
>>> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>>> >>> >> > > RC0 for 3.2: *28th Friday*
>>> >>> >> > >
>>> >>> >> > > Thank You
>>> >>> >> > > Sunil
>>> >>> >> > >
>>> >>> >> > >
>>> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <sunilg@apache.org
>>> <ma...@apache.org>>
>>> >>> wrote:
>>> >>> >> > >
>>> >>> >> > >> Hi All
>>> >>> >> > >>
>>> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of
>>> you
>>> >>> for
>>> >>> >> > >> helping in this. I am following up on these tickets, once its
>>> >>> closed
>>> >>> >> we
>>> >>> >> > >> will cut the 3.2 branch.
>>> >>> >> > >>
>>> >>> >> > >> Thanks
>>> >>> >> > >> Sunil Govindan
>>> >>> >> > >>
>>> >>> >> > >>
>>> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <sunilg@apache.org
>>> <ma...@apache.org>>
>>> >>> wrote:
>>> >>> >> > >>
>>> >>> >> > >>> Hi All,
>>> >>> >> > >>>
>>> >>> >> > >>> Inline with the original 3.2 communication proposal dated
>>> 17th
>>> >>> July
>>> >>> >> > >>> 2018, I would like to provide more updates.
>>> >>> >> > >>>
>>> >>> >> > >>> We are approaching previously proposed code freeze date
>>> >>> (September
>>> >>> >> 14,
>>> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
>>> point
>>> >>> >> > existing
>>> >>> >> > >>> trunk to 3.3 if there are no issues.
>>> >>> >> > >>>
>>> >>> >> > >>> *Current Release Plan:*
>>> >>> >> > >>> Feature freeze date : all features to merge by September 7,
>>> >>> 2018.
>>> >>> >> > >>> Code freeze date : blockers/critical only, no improvements
>>> and
>>> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>>> >>> >> > >>> Release date: September 28, 2018
>>> >>> >> > >>>
>>> >>> >> > >>> If any critical/blocker tickets which are targeted to
>>> 3.2.0, we
>>> >>> >> need to
>>> >>> >> > >>> backport to 3.2 post branch cut.
>>> >>> >> > >>>
>>> >>> >> > >>> Here's an updated 3.2.0 feature status:
>>> >>> >> > >>>
>>> >>> >> > >>> 1. Merged & Completed features:
>>> >>> >> > >>>
>>> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
>>> DeepLearning
>>> >>> >> > >>> workloads Initial cut.
>>> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in
>>> Capacity
>>> >>> >> > >>> Scheduler.
>>> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> >>> Service
>>> >>> >> > API
>>> >>> >> > >>> and CLI.
>>> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
>>> Improvement
>>> >>> >> works.
>>> >>> >> > >>>
>>> >>> >> > >>> 2. Features close to finish:
>>> >>> >> > >>>
>>> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>>> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
>>> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>>> >>> >> > >>>
>>> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>>> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>>> apps
>>> >>> from
>>> >>> >> > >>> ATSv2. Patch in progress.
>>> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >>> >> challenging to
>>> >>> >> > >>> be done before Aug 2018.
>>> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN
>>> applications.
>>> >>> >> > >>> Challenging as more discussions are on-going.
>>> >>> >> > >>>
>>> >>> >> > >>> *Summary of 3.2.0 issues status:*
>>> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following
>>> up
>>> >>> with
>>> >>> >> > >>> owners to get status on each of them to get in by Code
>>> Freeze
>>> >>> date.
>>> >>> >> > >>>
>>> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
>>> in
>>> >>> >> > (Blocker,
>>> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target
>>> Version/s" =
>>> >>> 3.2.0
>>> >>> >> > ORDER
>>> >>> >> > >>> BY priority DESC
>>> >>> >> > >>>
>>> >>> >> > >>> Thanks,
>>> >>> >> > >>> Sunil
>>> >>> >> > >>>
>>> >>> >> > >>>
>>> >>> >> > >>>
>>> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <sunilg@apache.org
>>> <ma...@apache.org>>
>>> >>> wrote:
>>> >>> >> > >>>
>>> >>> >> > >>>> Hi All,
>>> >>> >> > >>>>
>>> >>> >> > >>>> Inline with earlier communication dated 17th July 2018, I
>>> would
>>> >>> >> like
>>> >>> >> > to
>>> >>> >> > >>>> provide some updates.
>>> >>> >> > >>>>
>>> >>> >> > >>>> We are approaching previously proposed code freeze date
>>> (Aug
>>> >>> 31).
>>> >>> >> > >>>>
>>> >>> >> > >>>> One of the critical feature Node Attributes feature merge
>>> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs
>>> need a
>>> >>> bit
>>> >>> >> > more
>>> >>> >> > >>>> time. With regard to this, suggesting to push the
>>> feature/code
>>> >>> >> freeze
>>> >>> >> > for 2
>>> >>> >> > >>>> more weeks to accommodate these jiras too.
>>> >>> >> > >>>>
>>> >>> >> > >>>> Proposing Updated changes in plan inline with this:
>>> >>> >> > >>>> Feature freeze date : all features to merge by September 7,
>>> >>> 2018.
>>> >>> >> > >>>> Code freeze date : blockers/critical only, no improvements
>>> and
>>> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>>> >>> >> > >>>> Release date: September 28, 2018
>>> >>> >> > >>>>
>>> >>> >> > >>>> If any features in branch which are targeted to 3.2.0,
>>> please
>>> >>> >> reply to
>>> >>> >> > >>>> this email thread.
>>> >>> >> > >>>>
>>> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>>> >>> >> > >>>>
>>> >>> >> > >>>> 1. Merged & Completed features:
>>> >>> >> > >>>>
>>> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>>> DeepLearning
>>> >>> >> > >>>> workloads Initial cut.
>>> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>>> Capacity
>>> >>> >> > >>>> Scheduler.
>>> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>>> YARN
>>> >>> >> Service
>>> >>> >> > >>>> API and CLI.
>>> >>> >> > >>>>
>>> >>> >> > >>>> 2. Features close to finish:
>>> >>> >> > >>>>
>>> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >>> >> Merge/Vote
>>> >>> >> > >>>> Ongoing.
>>> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>>> apps
>>> >>> from
>>> >>> >> > >>>> ATSv2. Patch in progress.
>>> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> >>> Improvement
>>> >>> >> > works.
>>> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>>> Azure
>>> >>> >> > >>>> Storage. In progress.
>>> >>> >> > >>>>
>>> >>> >> > >>>> 3. Tentative features:
>>> >>> >> > >>>>
>>> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >>> >> challenging
>>> >>> >> > to
>>> >>> >> > >>>> be done before Aug 2018.
>>> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
>>> applications.
>>> >>> >> > >>>> Challenging as more discussions are on-going.
>>> >>> >> > >>>>
>>> >>> >> > >>>> *Summary of 3.2.0 issues status:*
>>> >>> >> > >>>>
>>> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am
>>> following up
>>> >>> with
>>> >>> >> > >>>> owners to get status on each of them to get in by Code
>>> Freeze
>>> >>> date.
>>> >>> >> > >>>>
>>> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>>> priority in
>>> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>>> >>> >> > Version/s" =
>>> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
>>> >>> >> > >>>>
>>> >>> >> > >>>> Thanks,
>>> >>> >> > >>>> Sunil
>>> >>> >> > >>>>
>>> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <
>>> sunilg@apache.org<ma...@apache.org>>
>>> >>> >> wrote:
>>> >>> >> > >>>>
>>> >>> >> > >>>>> Hi All,
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier
>>> communication
>>> >>> dated
>>> >>> >> > 17th
>>> >>> >> > >>>>> July 2018, I would like to provide some updates.
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> We are approaching previously proposed feature freeze date
>>> >>> (Aug
>>> >>> >> 21,
>>> >>> >> > >>>>> about 7 days from today).
>>> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
>>> please
>>> >>> >> reply
>>> >>> >> > to
>>> >>> >> > >>>>> this email thread.
>>> >>> >> > >>>>> Steve has mentioned about the s3 features which will come
>>> >>> close to
>>> >>> >> > >>>>> Code Freeze Date (Aug 31st).
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> 1. Merged & Completed features:
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>>> >>> DeepLearning
>>> >>> >> > >>>>> workloads Initial cut.
>>> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> 2. Features close to finish:
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >>> Major
>>> >>> >> > >>>>> patches are all in, only one last
>>> >>> >> > >>>>> patch is in review state.
>>> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>>> Capacity
>>> >>> >> > >>>>> Scheduler. Close to commit.
>>> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>>> YARN
>>> >>> >> Service
>>> >>> >> > >>>>> API and CLI. 2 patches are pending
>>> >>> >> > >>>>> which will be closed by Feature freeze date.
>>> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>>> apps
>>> >>> >> from
>>> >>> >> > >>>>> ATSv2. Patch in progress.
>>> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> >>> Improvement
>>> >>> >> > >>>>> works.
>>> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>>> >>> Azure
>>> >>> >> > >>>>> Storage. In progress.
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> 3. Tentative features:
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >>> >> challenging
>>> >>> >> > >>>>> to be done before Aug 2018.
>>> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
>>> applications.
>>> >>> >> > >>>>> Challenging as more discussions are on-going.
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
>>> >>> with
>>> >>> >> > owners
>>> >>> >> > >>>>> to get status on each of them to get in by Code Freeze
>>> date.
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>>> priority in
>>> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND
>>> "Target
>>> >>> >> > Version/s" =
>>> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> Thanks,
>>> >>> >> > >>>>> Sunil
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <
>>> sunilg@apache.org<ma...@apache.org>>
>>> >>> >> wrote:
>>> >>> >> > >>>>>
>>> >>> >> > >>>>>> Thanks Subru for the thoughts.
>>> >>> >> > >>>>>> One of the main reason for a major release is to push out
>>> >>> >> critical
>>> >>> >> > >>>>>> features with a faster cadence to the users. If we are
>>> >>> pulling
>>> >>> >> more
>>> >>> >> > and
>>> >>> >> > >>>>>> more different types of features to a minor release, that
>>> >>> branch
>>> >>> >> > will
>>> >>> >> > >>>>>> become more destabilized and it may be tough to say that
>>> >>> 3.1.2 is
>>> >>> >> > stable
>>> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and
>>> stabilize
>>> >>> >> features
>>> >>> >> > in
>>> >>> >> > >>>>>> subsequent minor release.
>>> >>> >> > >>>>>> For few companies, it makes sense to push out these new
>>> >>> features
>>> >>> >> > >>>>>> faster to make a reach to the users. Adding to the point
>>> to
>>> >>> the
>>> >>> >> > backporting
>>> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround
>>> that
>>> >>> with
>>> >>> >> > some git
>>> >>> >> > >>>>>> scripts. If we can make such scripts available to
>>> committers,
>>> >>> >> > backport will
>>> >>> >> > >>>>>> be seem-less across branches and we can achieve the
>>> faster
>>> >>> >> release
>>> >>> >> > cadence
>>> >>> >> > >>>>>> also.
>>> >>> >> > >>>>>>
>>> >>> >> > >>>>>> Thoughts?
>>> >>> >> > >>>>>>
>>> >>> >> > >>>>>> - Sunil
>>> >>> >> > >>>>>>
>>> >>> >> > >>>>>>
>>> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>>> >>> subru@apache.org<ma...@apache.org>
>>> >>> >> >
>>> >>> >> > >>>>>> wrote:
>>> >>> >> > >>>>>>
>>> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release
>>> effort. I
>>> >>> am
>>> >>> >> > >>>>>>> generally
>>> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a
>>> 3.1.x) as
>>> >>> >> feel we
>>> >>> >> > >>>>>>> already
>>> >>> >> > >>>>>>> have too many branches to be maintained. I already see
>>> many
>>> >>> >> commits
>>> >>> >> > >>>>>>> are in
>>> >>> >> > >>>>>>> different branches with no apparent rationale, for e.g:
>>> 3.1
>>> >>> has
>>> >>> >> > >>>>>>> commits
>>> >>> >> > >>>>>>> which are absent in 3.0 etc.
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any
>>> major
>>> >>> >> > production
>>> >>> >> > >>>>>>> setting so the cost of adding features should be
>>> minimal.
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>> Thoughts?
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>> -Subru
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
>>> >>> sunilg@apache.org<ma...@apache.org>>
>>> >>> >> > wrote:
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > Yes, important changes and features are much needed,
>>> >>> hence we
>>> >>> >> > will
>>> >>> >> > >>>>>>> be
>>> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
>>> >>> considering
>>> >>> >> few
>>> >>> >> > >>>>>>> more
>>> >>> >> > >>>>>>> > offline requests from other folks, I think extending
>>> the
>>> >>> >> > timeframe
>>> >>> >> > >>>>>>> by
>>> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
>>> >>> buffer) and
>>> >>> >> > >>>>>>> this should
>>> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap
>>> wiki as
>>> >>> >> well)
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by
>>> August
>>> >>> 21,
>>> >>> >> 2018.
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>>> >>> improvements
>>> >>> >> and
>>> >>> >> > >>>>>>> non
>>> >>> >> > >>>>>>> > blocker/critical
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > - Release date: September 15, 2018
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already
>>> answered
>>> >>> your
>>> >>> >> > >>>>>>> questions.
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > Thanks
>>> >>> >> > >>>>>>> > Sunil
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>>> >>> >> wheeleast@gmail.com<ma...@gmail.com>
>>> >>> >> > >
>>> >>> >> > >>>>>>> wrote:
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2
>>> release,
>>> >>> +1
>>> >>> >> for
>>> >>> >> > >>>>>>> that.
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > To concerns from Steve,
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
>>> important
>>> >>> >> > changes
>>> >>> >> > >>>>>>> /
>>> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep
>>> the
>>> >>> >> proposed
>>> >>> >> > >>>>>>> release
>>> >>> >> > >>>>>>> > date
>>> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of
>>> last
>>> >>> >> minute
>>> >>> >> > >>>>>>> and we all
>>> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm
>>> also
>>> >>> fine
>>> >>> >> if
>>> >>> >> > >>>>>>> we want
>>> >>> >> > >>>>>>> > get
>>> >>> >> > >>>>>>> > > another several weeks time.
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
>>> >>> before
>>> >>> >> > >>>>>>> thanksgiving.
>>> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > Eric,
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
>>> >>> YARN-8135
>>> >>> >> can
>>> >>> >> > >>>>>>> get merged
>>> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > Sunil,
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > Could u update
>>> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>>> >>> >> > >>>>>>> > Roadmap
>>> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
>>> >>> first
>>> >>> >> > >>>>>>> before getting
>>> >>> >> > >>>>>>> > > consensus of time.
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > Thanks,
>>> >>> >> > >>>>>>> > > Wangda
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid<mailto:
>>> fabbri@cloudera.com.invalid>
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > wrote:
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>> >>> >> > >>>>>>> stevel@hortonworks.com<ma...@hortonworks.com>>
>>> >>> >> > >>>>>>> > >> wrote:
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
>>> sunilg@apache.org<ma...@apache.org>
>>> >>> >> > <mailto:
>>> >>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>>
>>> wrote:
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > I would also would like to take this opportunity
>>> to
>>> >>> come
>>> >>> >> up
>>> >>> >> > >>>>>>> with a
>>> >>> >> > >>>>>>> > >> detailed
>>> >>> >> > >>>>>>> > >> > plan.
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
>>> >>> merged by
>>> >>> >> > >>>>>>> August 10,
>>> >>> >> > >>>>>>> > >> 2018.
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > <snip>
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
>>> targeted
>>> >>> to
>>> >>> >> 3.2
>>> >>> >> > >>>>>>> per this
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> https://issues.apache.org/jira/browse/HADOOP-15226
>>> >>> >> > >>>>>>> > >> >
>>> https://issues.apache.org/jira/browse/HADOOP-15220
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
>>> >>> Datalake
>>> >>> >> > Gen
>>> >>> >> > >>>>>>> 2
>>> >>> >> > >>>>>>> > >> >
>>> https://issues.apache.org/jira/browse/HADOOP-15407
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
>>> >>> doing
>>> >>> >> work
>>> >>> >> > >>>>>>> on it, but
>>> >>> >> > >>>>>>> > >> if
>>> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect
>>> some
>>> >>> >> timely
>>> >>> >> > >>>>>>> reviews on
>>> >>> >> > >>>>>>> > >> >
>>> https://issues.apache.org/jira/browse/HADOOP-15546
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will
>>> review
>>> >>> the
>>> >>> >> S3 &
>>> >>> >> > >>>>>>> S3Guard
>>> >>> >> > >>>>>>> > >> JIRAs
>>> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there
>>> are
>>> >>> some
>>> >>> >> > >>>>>>> pressing
>>> >>> >> > >>>>>>> > "guave,
>>> >>> >> > >>>>>>> > >> > java 9 prep"
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> I can help with this part if you like.
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
>>> release
>>> >>> >> > manager
>>> >>> >> > >>>>>>> of 3.2.0
>>> >>> >> > >>>>>>> > >> > release.
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > well volunteered!
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable
>>> should
>>> >>> we
>>> >>> >> have
>>> >>> >> > >>>>>>> for the
>>> >>> >> > >>>>>>> > >> 3.2. &
>>> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence,
>>> then
>>> >>> having
>>> >>> >> > the
>>> >>> >> > >>>>>>> outline
>>> >>> >> > >>>>>>> > >> time
>>> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that
>>> there's
>>> >>> less
>>> >>> >> > >>>>>>> concern about
>>> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > -Steve
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >> -AF
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>
>>> >>> >> >
>>> >>> >>
>>> >>> >
>>> >>>
>>> >>
>>> >>
>>>
>>>
>>> --
>>> A very happy Clouderan
>>>
>>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Konstantin Shvachko <sh...@gmail.com>.
Another thing is that I see a bunch of jiras under HDFS-8707, which don't
have the Fix Version field listing 3.2, some have it just empty.
This means they will not be populated into release notes.
Thanks,
--Konstantin
On Thu, Oct 25, 2018 at 7:59 AM Sunil G <su...@apache.org> wrote:
> Thanks Konstantin for pointing out.
> As 3.2 is pretty much on RC level, its better we try to find a good
> solution to this issue.
>
> I ll follow up on this in the jira.
>
> - Sunil
>
> On Thu, Oct 25, 2018 at 11:35 AM Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
>> I've tried to attract attention to an incompatibility issue through the
>> jira, but it didn't work. So pitching in in this thread.
>> https://issues.apache.org/jira/browse/HDFS-12026
>> It introduced binary incompatibility, which will prevent people from
>> upgrading from 3.1 to 3.2.
>> I think it can get messy if we release anything with this feature.
>>
>> Thanks,
>> --Konstantin
>>
>> On Mon, Oct 22, 2018 at 5:01 AM Steve Loughran <st...@hortonworks.com>
>> wrote:
>>
>>> its in.
>>>
>>> good catch!
>>>
>>> On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <weichiu@cloudera.com<mailto:
>>> weichiu@cloudera.com>> wrote:
>>>
>>> Thanks Sunil G for driving the release,
>>> I filed HADOOP-15866<https://issues.apache.org/jira/browse/HADOOP-15866>
>>> for a compat fix. If any one has cycle please review it, as I think it is
>>> needed for 3.2.0.
>>>
>>> On Thu, Oct 18, 2018 at 4:43 AM Sunil G <sunilg@apache.org<mailto:
>>> sunilg@apache.org>> wrote:
>>> Hi Folks,
>>>
>>> As we previously communicated for 3.2.0 release, we have delayed due to
>>> few
>>> blockers in our gate.
>>>
>>> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
>>> all bug fixes.
>>>
>>> - Sunil
>>>
>>>
>>> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <sunilg@apache.org<mailto:
>>> sunilg@apache.org>> wrote:
>>>
>>> > Hi Folks,
>>> >
>>> > We are now close to RC as other blocker issues are now merged to trunk
>>> and
>>> > branch-3.2. Last 2 critical issues are closer to merge and will be
>>> > committed in few hours.
>>> > With this, I will be creating 3.2.0 branch today and will go ahead
>>> with RC
>>> > related process.
>>> >
>>> > - Sunil
>>> >
>>> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jonbender@stripe.com
>>> <ma...@stripe.com>>
>>> > wrote:
>>> >
>>> >> Hello, were there any updates around the 3.2.0 RC timing? All I see in
>>> >> the current blockers are related to the new Submarine subproject,
>>> wasn't
>>> >> sure if that is what is holding things up.
>>> >>
>>> >> Cheers,
>>> >> Jon
>>> >>
>>> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <sunilg@apache.org<mailto:
>>> sunilg@apache.org>> wrote:
>>> >>
>>> >>> Thanks Robert and Haibo for quickly correcting same.
>>> >>> Sigh, I somehow missed one file while committing the change. Sorry
>>> for
>>> >>> the
>>> >>> trouble.
>>> >>>
>>> >>> - Sunil
>>> >>>
>>> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rkanter@cloudera.com
>>> <ma...@cloudera.com>>
>>> >>> wrote:
>>> >>>
>>> >>> > Looks like there's two that weren't updated:
>>> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" .
>>> -r
>>> >>> > --include=pom.xml
>>> >>> > ./hadoop-project/pom.xml:
>>> >>> >
>>> <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>>> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>>> >>> >
>>> >>> > I've just pushed in an addendum commit to fix those.
>>> >>> > In the future, please make sure to do a sanity compile when
>>> updating
>>> >>> poms.
>>> >>> >
>>> >>> > thanks
>>> >>> > - Robert
>>> >>> >
>>> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>>> >>> <fa...@cloudera.com.invalid>>
>>> >>> > wrote:
>>> >>> >
>>> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in
>>> the
>>> >>> >> top-level pom.xml?
>>> >>> >>
>>> >>> >>
>>> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <sunilg@apache.org
>>> <ma...@apache.org>> wrote:
>>> >>> >>
>>> >>> >> > Hi All
>>> >>> >> >
>>> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
>>> trunk
>>> >>> to
>>> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
>>> >>> necessary
>>> >>> >> > patches are pulled into branch-3.2.
>>> >>> >> >
>>> >>> >> > Thank You
>>> >>> >> > - Sunil
>>> >>> >> >
>>> >>> >> >
>>> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <sunilg@apache.org
>>> <ma...@apache.org>> wrote:
>>> >>> >> >
>>> >>> >> > > Hi All
>>> >>> >> > >
>>> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is
>>> merged to
>>> >>> >> trunk.
>>> >>> >> > > Thanks for the support.
>>> >>> >> > >
>>> >>> >> > > *Plan for RC*
>>> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>>> >>> >> > > RC0 for 3.2: *28th Friday*
>>> >>> >> > >
>>> >>> >> > > Thank You
>>> >>> >> > > Sunil
>>> >>> >> > >
>>> >>> >> > >
>>> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <sunilg@apache.org
>>> <ma...@apache.org>>
>>> >>> wrote:
>>> >>> >> > >
>>> >>> >> > >> Hi All
>>> >>> >> > >>
>>> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of
>>> you
>>> >>> for
>>> >>> >> > >> helping in this. I am following up on these tickets, once its
>>> >>> closed
>>> >>> >> we
>>> >>> >> > >> will cut the 3.2 branch.
>>> >>> >> > >>
>>> >>> >> > >> Thanks
>>> >>> >> > >> Sunil Govindan
>>> >>> >> > >>
>>> >>> >> > >>
>>> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <sunilg@apache.org
>>> <ma...@apache.org>>
>>> >>> wrote:
>>> >>> >> > >>
>>> >>> >> > >>> Hi All,
>>> >>> >> > >>>
>>> >>> >> > >>> Inline with the original 3.2 communication proposal dated
>>> 17th
>>> >>> July
>>> >>> >> > >>> 2018, I would like to provide more updates.
>>> >>> >> > >>>
>>> >>> >> > >>> We are approaching previously proposed code freeze date
>>> >>> (September
>>> >>> >> 14,
>>> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
>>> point
>>> >>> >> > existing
>>> >>> >> > >>> trunk to 3.3 if there are no issues.
>>> >>> >> > >>>
>>> >>> >> > >>> *Current Release Plan:*
>>> >>> >> > >>> Feature freeze date : all features to merge by September 7,
>>> >>> 2018.
>>> >>> >> > >>> Code freeze date : blockers/critical only, no improvements
>>> and
>>> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>>> >>> >> > >>> Release date: September 28, 2018
>>> >>> >> > >>>
>>> >>> >> > >>> If any critical/blocker tickets which are targeted to
>>> 3.2.0, we
>>> >>> >> need to
>>> >>> >> > >>> backport to 3.2 post branch cut.
>>> >>> >> > >>>
>>> >>> >> > >>> Here's an updated 3.2.0 feature status:
>>> >>> >> > >>>
>>> >>> >> > >>> 1. Merged & Completed features:
>>> >>> >> > >>>
>>> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
>>> DeepLearning
>>> >>> >> > >>> workloads Initial cut.
>>> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in
>>> Capacity
>>> >>> >> > >>> Scheduler.
>>> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> >>> Service
>>> >>> >> > API
>>> >>> >> > >>> and CLI.
>>> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
>>> Improvement
>>> >>> >> works.
>>> >>> >> > >>>
>>> >>> >> > >>> 2. Features close to finish:
>>> >>> >> > >>>
>>> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>>> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
>>> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>>> >>> >> > >>>
>>> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>>> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>>> apps
>>> >>> from
>>> >>> >> > >>> ATSv2. Patch in progress.
>>> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >>> >> challenging to
>>> >>> >> > >>> be done before Aug 2018.
>>> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN
>>> applications.
>>> >>> >> > >>> Challenging as more discussions are on-going.
>>> >>> >> > >>>
>>> >>> >> > >>> *Summary of 3.2.0 issues status:*
>>> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following
>>> up
>>> >>> with
>>> >>> >> > >>> owners to get status on each of them to get in by Code
>>> Freeze
>>> >>> date.
>>> >>> >> > >>>
>>> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
>>> in
>>> >>> >> > (Blocker,
>>> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target
>>> Version/s" =
>>> >>> 3.2.0
>>> >>> >> > ORDER
>>> >>> >> > >>> BY priority DESC
>>> >>> >> > >>>
>>> >>> >> > >>> Thanks,
>>> >>> >> > >>> Sunil
>>> >>> >> > >>>
>>> >>> >> > >>>
>>> >>> >> > >>>
>>> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <sunilg@apache.org
>>> <ma...@apache.org>>
>>> >>> wrote:
>>> >>> >> > >>>
>>> >>> >> > >>>> Hi All,
>>> >>> >> > >>>>
>>> >>> >> > >>>> Inline with earlier communication dated 17th July 2018, I
>>> would
>>> >>> >> like
>>> >>> >> > to
>>> >>> >> > >>>> provide some updates.
>>> >>> >> > >>>>
>>> >>> >> > >>>> We are approaching previously proposed code freeze date
>>> (Aug
>>> >>> 31).
>>> >>> >> > >>>>
>>> >>> >> > >>>> One of the critical feature Node Attributes feature merge
>>> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs
>>> need a
>>> >>> bit
>>> >>> >> > more
>>> >>> >> > >>>> time. With regard to this, suggesting to push the
>>> feature/code
>>> >>> >> freeze
>>> >>> >> > for 2
>>> >>> >> > >>>> more weeks to accommodate these jiras too.
>>> >>> >> > >>>>
>>> >>> >> > >>>> Proposing Updated changes in plan inline with this:
>>> >>> >> > >>>> Feature freeze date : all features to merge by September 7,
>>> >>> 2018.
>>> >>> >> > >>>> Code freeze date : blockers/critical only, no improvements
>>> and
>>> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>>> >>> >> > >>>> Release date: September 28, 2018
>>> >>> >> > >>>>
>>> >>> >> > >>>> If any features in branch which are targeted to 3.2.0,
>>> please
>>> >>> >> reply to
>>> >>> >> > >>>> this email thread.
>>> >>> >> > >>>>
>>> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>>> >>> >> > >>>>
>>> >>> >> > >>>> 1. Merged & Completed features:
>>> >>> >> > >>>>
>>> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>>> DeepLearning
>>> >>> >> > >>>> workloads Initial cut.
>>> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>>> Capacity
>>> >>> >> > >>>> Scheduler.
>>> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>>> YARN
>>> >>> >> Service
>>> >>> >> > >>>> API and CLI.
>>> >>> >> > >>>>
>>> >>> >> > >>>> 2. Features close to finish:
>>> >>> >> > >>>>
>>> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >>> >> Merge/Vote
>>> >>> >> > >>>> Ongoing.
>>> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>>> apps
>>> >>> from
>>> >>> >> > >>>> ATSv2. Patch in progress.
>>> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> >>> Improvement
>>> >>> >> > works.
>>> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>>> Azure
>>> >>> >> > >>>> Storage. In progress.
>>> >>> >> > >>>>
>>> >>> >> > >>>> 3. Tentative features:
>>> >>> >> > >>>>
>>> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >>> >> challenging
>>> >>> >> > to
>>> >>> >> > >>>> be done before Aug 2018.
>>> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
>>> applications.
>>> >>> >> > >>>> Challenging as more discussions are on-going.
>>> >>> >> > >>>>
>>> >>> >> > >>>> *Summary of 3.2.0 issues status:*
>>> >>> >> > >>>>
>>> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am
>>> following up
>>> >>> with
>>> >>> >> > >>>> owners to get status on each of them to get in by Code
>>> Freeze
>>> >>> date.
>>> >>> >> > >>>>
>>> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>>> priority in
>>> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>>> >>> >> > Version/s" =
>>> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
>>> >>> >> > >>>>
>>> >>> >> > >>>> Thanks,
>>> >>> >> > >>>> Sunil
>>> >>> >> > >>>>
>>> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <
>>> sunilg@apache.org<ma...@apache.org>>
>>> >>> >> wrote:
>>> >>> >> > >>>>
>>> >>> >> > >>>>> Hi All,
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier
>>> communication
>>> >>> dated
>>> >>> >> > 17th
>>> >>> >> > >>>>> July 2018, I would like to provide some updates.
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> We are approaching previously proposed feature freeze date
>>> >>> (Aug
>>> >>> >> 21,
>>> >>> >> > >>>>> about 7 days from today).
>>> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
>>> please
>>> >>> >> reply
>>> >>> >> > to
>>> >>> >> > >>>>> this email thread.
>>> >>> >> > >>>>> Steve has mentioned about the s3 features which will come
>>> >>> close to
>>> >>> >> > >>>>> Code Freeze Date (Aug 31st).
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> 1. Merged & Completed features:
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>>> >>> DeepLearning
>>> >>> >> > >>>>> workloads Initial cut.
>>> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> 2. Features close to finish:
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >>> Major
>>> >>> >> > >>>>> patches are all in, only one last
>>> >>> >> > >>>>> patch is in review state.
>>> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>>> Capacity
>>> >>> >> > >>>>> Scheduler. Close to commit.
>>> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>>> YARN
>>> >>> >> Service
>>> >>> >> > >>>>> API and CLI. 2 patches are pending
>>> >>> >> > >>>>> which will be closed by Feature freeze date.
>>> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>>> apps
>>> >>> >> from
>>> >>> >> > >>>>> ATSv2. Patch in progress.
>>> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> >>> Improvement
>>> >>> >> > >>>>> works.
>>> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>>> >>> Azure
>>> >>> >> > >>>>> Storage. In progress.
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> 3. Tentative features:
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >>> >> challenging
>>> >>> >> > >>>>> to be done before Aug 2018.
>>> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
>>> applications.
>>> >>> >> > >>>>> Challenging as more discussions are on-going.
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
>>> >>> with
>>> >>> >> > owners
>>> >>> >> > >>>>> to get status on each of them to get in by Code Freeze
>>> date.
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>>> priority in
>>> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND
>>> "Target
>>> >>> >> > Version/s" =
>>> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> Thanks,
>>> >>> >> > >>>>> Sunil
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <
>>> sunilg@apache.org<ma...@apache.org>>
>>> >>> >> wrote:
>>> >>> >> > >>>>>
>>> >>> >> > >>>>>> Thanks Subru for the thoughts.
>>> >>> >> > >>>>>> One of the main reason for a major release is to push out
>>> >>> >> critical
>>> >>> >> > >>>>>> features with a faster cadence to the users. If we are
>>> >>> pulling
>>> >>> >> more
>>> >>> >> > and
>>> >>> >> > >>>>>> more different types of features to a minor release, that
>>> >>> branch
>>> >>> >> > will
>>> >>> >> > >>>>>> become more destabilized and it may be tough to say that
>>> >>> 3.1.2 is
>>> >>> >> > stable
>>> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and
>>> stabilize
>>> >>> >> features
>>> >>> >> > in
>>> >>> >> > >>>>>> subsequent minor release.
>>> >>> >> > >>>>>> For few companies, it makes sense to push out these new
>>> >>> features
>>> >>> >> > >>>>>> faster to make a reach to the users. Adding to the point
>>> to
>>> >>> the
>>> >>> >> > backporting
>>> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround
>>> that
>>> >>> with
>>> >>> >> > some git
>>> >>> >> > >>>>>> scripts. If we can make such scripts available to
>>> committers,
>>> >>> >> > backport will
>>> >>> >> > >>>>>> be seem-less across branches and we can achieve the
>>> faster
>>> >>> >> release
>>> >>> >> > cadence
>>> >>> >> > >>>>>> also.
>>> >>> >> > >>>>>>
>>> >>> >> > >>>>>> Thoughts?
>>> >>> >> > >>>>>>
>>> >>> >> > >>>>>> - Sunil
>>> >>> >> > >>>>>>
>>> >>> >> > >>>>>>
>>> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>>> >>> subru@apache.org<ma...@apache.org>
>>> >>> >> >
>>> >>> >> > >>>>>> wrote:
>>> >>> >> > >>>>>>
>>> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release
>>> effort. I
>>> >>> am
>>> >>> >> > >>>>>>> generally
>>> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a
>>> 3.1.x) as
>>> >>> >> feel we
>>> >>> >> > >>>>>>> already
>>> >>> >> > >>>>>>> have too many branches to be maintained. I already see
>>> many
>>> >>> >> commits
>>> >>> >> > >>>>>>> are in
>>> >>> >> > >>>>>>> different branches with no apparent rationale, for e.g:
>>> 3.1
>>> >>> has
>>> >>> >> > >>>>>>> commits
>>> >>> >> > >>>>>>> which are absent in 3.0 etc.
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any
>>> major
>>> >>> >> > production
>>> >>> >> > >>>>>>> setting so the cost of adding features should be
>>> minimal.
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>> Thoughts?
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>> -Subru
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
>>> >>> sunilg@apache.org<ma...@apache.org>>
>>> >>> >> > wrote:
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > Yes, important changes and features are much needed,
>>> >>> hence we
>>> >>> >> > will
>>> >>> >> > >>>>>>> be
>>> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
>>> >>> considering
>>> >>> >> few
>>> >>> >> > >>>>>>> more
>>> >>> >> > >>>>>>> > offline requests from other folks, I think extending
>>> the
>>> >>> >> > timeframe
>>> >>> >> > >>>>>>> by
>>> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
>>> >>> buffer) and
>>> >>> >> > >>>>>>> this should
>>> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap
>>> wiki as
>>> >>> >> well)
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by
>>> August
>>> >>> 21,
>>> >>> >> 2018.
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>>> >>> improvements
>>> >>> >> and
>>> >>> >> > >>>>>>> non
>>> >>> >> > >>>>>>> > blocker/critical
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > - Release date: September 15, 2018
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already
>>> answered
>>> >>> your
>>> >>> >> > >>>>>>> questions.
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > Thanks
>>> >>> >> > >>>>>>> > Sunil
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>>> >>> >> wheeleast@gmail.com<ma...@gmail.com>
>>> >>> >> > >
>>> >>> >> > >>>>>>> wrote:
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2
>>> release,
>>> >>> +1
>>> >>> >> for
>>> >>> >> > >>>>>>> that.
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > To concerns from Steve,
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
>>> important
>>> >>> >> > changes
>>> >>> >> > >>>>>>> /
>>> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep
>>> the
>>> >>> >> proposed
>>> >>> >> > >>>>>>> release
>>> >>> >> > >>>>>>> > date
>>> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of
>>> last
>>> >>> >> minute
>>> >>> >> > >>>>>>> and we all
>>> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm
>>> also
>>> >>> fine
>>> >>> >> if
>>> >>> >> > >>>>>>> we want
>>> >>> >> > >>>>>>> > get
>>> >>> >> > >>>>>>> > > another several weeks time.
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
>>> >>> before
>>> >>> >> > >>>>>>> thanksgiving.
>>> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > Eric,
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
>>> >>> YARN-8135
>>> >>> >> can
>>> >>> >> > >>>>>>> get merged
>>> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > Sunil,
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > Could u update
>>> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>>> >>> >> > >>>>>>> > Roadmap
>>> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
>>> >>> first
>>> >>> >> > >>>>>>> before getting
>>> >>> >> > >>>>>>> > > consensus of time.
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > Thanks,
>>> >>> >> > >>>>>>> > > Wangda
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid<mailto:
>>> fabbri@cloudera.com.invalid>
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > wrote:
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>> >>> >> > >>>>>>> stevel@hortonworks.com<ma...@hortonworks.com>>
>>> >>> >> > >>>>>>> > >> wrote:
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
>>> sunilg@apache.org<ma...@apache.org>
>>> >>> >> > <mailto:
>>> >>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>>
>>> wrote:
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > I would also would like to take this opportunity
>>> to
>>> >>> come
>>> >>> >> up
>>> >>> >> > >>>>>>> with a
>>> >>> >> > >>>>>>> > >> detailed
>>> >>> >> > >>>>>>> > >> > plan.
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
>>> >>> merged by
>>> >>> >> > >>>>>>> August 10,
>>> >>> >> > >>>>>>> > >> 2018.
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > <snip>
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
>>> targeted
>>> >>> to
>>> >>> >> 3.2
>>> >>> >> > >>>>>>> per this
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> https://issues.apache.org/jira/browse/HADOOP-15226
>>> >>> >> > >>>>>>> > >> >
>>> https://issues.apache.org/jira/browse/HADOOP-15220
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
>>> >>> Datalake
>>> >>> >> > Gen
>>> >>> >> > >>>>>>> 2
>>> >>> >> > >>>>>>> > >> >
>>> https://issues.apache.org/jira/browse/HADOOP-15407
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
>>> >>> doing
>>> >>> >> work
>>> >>> >> > >>>>>>> on it, but
>>> >>> >> > >>>>>>> > >> if
>>> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect
>>> some
>>> >>> >> timely
>>> >>> >> > >>>>>>> reviews on
>>> >>> >> > >>>>>>> > >> >
>>> https://issues.apache.org/jira/browse/HADOOP-15546
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will
>>> review
>>> >>> the
>>> >>> >> S3 &
>>> >>> >> > >>>>>>> S3Guard
>>> >>> >> > >>>>>>> > >> JIRAs
>>> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there
>>> are
>>> >>> some
>>> >>> >> > >>>>>>> pressing
>>> >>> >> > >>>>>>> > "guave,
>>> >>> >> > >>>>>>> > >> > java 9 prep"
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> I can help with this part if you like.
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
>>> release
>>> >>> >> > manager
>>> >>> >> > >>>>>>> of 3.2.0
>>> >>> >> > >>>>>>> > >> > release.
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > well volunteered!
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable
>>> should
>>> >>> we
>>> >>> >> have
>>> >>> >> > >>>>>>> for the
>>> >>> >> > >>>>>>> > >> 3.2. &
>>> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence,
>>> then
>>> >>> having
>>> >>> >> > the
>>> >>> >> > >>>>>>> outline
>>> >>> >> > >>>>>>> > >> time
>>> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that
>>> there's
>>> >>> less
>>> >>> >> > >>>>>>> concern about
>>> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > -Steve
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >> -AF
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>
>>> >>> >> >
>>> >>> >>
>>> >>> >
>>> >>>
>>> >>
>>> >>
>>>
>>>
>>> --
>>> A very happy Clouderan
>>>
>>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Konstantin Shvachko <sh...@gmail.com>.
Another thing is that I see a bunch of jiras under HDFS-8707, which don't
have the Fix Version field listing 3.2, some have it just empty.
This means they will not be populated into release notes.
Thanks,
--Konstantin
On Thu, Oct 25, 2018 at 7:59 AM Sunil G <su...@apache.org> wrote:
> Thanks Konstantin for pointing out.
> As 3.2 is pretty much on RC level, its better we try to find a good
> solution to this issue.
>
> I ll follow up on this in the jira.
>
> - Sunil
>
> On Thu, Oct 25, 2018 at 11:35 AM Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
>> I've tried to attract attention to an incompatibility issue through the
>> jira, but it didn't work. So pitching in in this thread.
>> https://issues.apache.org/jira/browse/HDFS-12026
>> It introduced binary incompatibility, which will prevent people from
>> upgrading from 3.1 to 3.2.
>> I think it can get messy if we release anything with this feature.
>>
>> Thanks,
>> --Konstantin
>>
>> On Mon, Oct 22, 2018 at 5:01 AM Steve Loughran <st...@hortonworks.com>
>> wrote:
>>
>>> its in.
>>>
>>> good catch!
>>>
>>> On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <weichiu@cloudera.com<mailto:
>>> weichiu@cloudera.com>> wrote:
>>>
>>> Thanks Sunil G for driving the release,
>>> I filed HADOOP-15866<https://issues.apache.org/jira/browse/HADOOP-15866>
>>> for a compat fix. If any one has cycle please review it, as I think it is
>>> needed for 3.2.0.
>>>
>>> On Thu, Oct 18, 2018 at 4:43 AM Sunil G <sunilg@apache.org<mailto:
>>> sunilg@apache.org>> wrote:
>>> Hi Folks,
>>>
>>> As we previously communicated for 3.2.0 release, we have delayed due to
>>> few
>>> blockers in our gate.
>>>
>>> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
>>> all bug fixes.
>>>
>>> - Sunil
>>>
>>>
>>> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <sunilg@apache.org<mailto:
>>> sunilg@apache.org>> wrote:
>>>
>>> > Hi Folks,
>>> >
>>> > We are now close to RC as other blocker issues are now merged to trunk
>>> and
>>> > branch-3.2. Last 2 critical issues are closer to merge and will be
>>> > committed in few hours.
>>> > With this, I will be creating 3.2.0 branch today and will go ahead
>>> with RC
>>> > related process.
>>> >
>>> > - Sunil
>>> >
>>> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jonbender@stripe.com
>>> <ma...@stripe.com>>
>>> > wrote:
>>> >
>>> >> Hello, were there any updates around the 3.2.0 RC timing? All I see in
>>> >> the current blockers are related to the new Submarine subproject,
>>> wasn't
>>> >> sure if that is what is holding things up.
>>> >>
>>> >> Cheers,
>>> >> Jon
>>> >>
>>> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <sunilg@apache.org<mailto:
>>> sunilg@apache.org>> wrote:
>>> >>
>>> >>> Thanks Robert and Haibo for quickly correcting same.
>>> >>> Sigh, I somehow missed one file while committing the change. Sorry
>>> for
>>> >>> the
>>> >>> trouble.
>>> >>>
>>> >>> - Sunil
>>> >>>
>>> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rkanter@cloudera.com
>>> <ma...@cloudera.com>>
>>> >>> wrote:
>>> >>>
>>> >>> > Looks like there's two that weren't updated:
>>> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" .
>>> -r
>>> >>> > --include=pom.xml
>>> >>> > ./hadoop-project/pom.xml:
>>> >>> >
>>> <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>>> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>>> >>> >
>>> >>> > I've just pushed in an addendum commit to fix those.
>>> >>> > In the future, please make sure to do a sanity compile when
>>> updating
>>> >>> poms.
>>> >>> >
>>> >>> > thanks
>>> >>> > - Robert
>>> >>> >
>>> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>>> >>> <fa...@cloudera.com.invalid>>
>>> >>> > wrote:
>>> >>> >
>>> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in
>>> the
>>> >>> >> top-level pom.xml?
>>> >>> >>
>>> >>> >>
>>> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <sunilg@apache.org
>>> <ma...@apache.org>> wrote:
>>> >>> >>
>>> >>> >> > Hi All
>>> >>> >> >
>>> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
>>> trunk
>>> >>> to
>>> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
>>> >>> necessary
>>> >>> >> > patches are pulled into branch-3.2.
>>> >>> >> >
>>> >>> >> > Thank You
>>> >>> >> > - Sunil
>>> >>> >> >
>>> >>> >> >
>>> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <sunilg@apache.org
>>> <ma...@apache.org>> wrote:
>>> >>> >> >
>>> >>> >> > > Hi All
>>> >>> >> > >
>>> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is
>>> merged to
>>> >>> >> trunk.
>>> >>> >> > > Thanks for the support.
>>> >>> >> > >
>>> >>> >> > > *Plan for RC*
>>> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>>> >>> >> > > RC0 for 3.2: *28th Friday*
>>> >>> >> > >
>>> >>> >> > > Thank You
>>> >>> >> > > Sunil
>>> >>> >> > >
>>> >>> >> > >
>>> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <sunilg@apache.org
>>> <ma...@apache.org>>
>>> >>> wrote:
>>> >>> >> > >
>>> >>> >> > >> Hi All
>>> >>> >> > >>
>>> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of
>>> you
>>> >>> for
>>> >>> >> > >> helping in this. I am following up on these tickets, once its
>>> >>> closed
>>> >>> >> we
>>> >>> >> > >> will cut the 3.2 branch.
>>> >>> >> > >>
>>> >>> >> > >> Thanks
>>> >>> >> > >> Sunil Govindan
>>> >>> >> > >>
>>> >>> >> > >>
>>> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <sunilg@apache.org
>>> <ma...@apache.org>>
>>> >>> wrote:
>>> >>> >> > >>
>>> >>> >> > >>> Hi All,
>>> >>> >> > >>>
>>> >>> >> > >>> Inline with the original 3.2 communication proposal dated
>>> 17th
>>> >>> July
>>> >>> >> > >>> 2018, I would like to provide more updates.
>>> >>> >> > >>>
>>> >>> >> > >>> We are approaching previously proposed code freeze date
>>> >>> (September
>>> >>> >> 14,
>>> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
>>> point
>>> >>> >> > existing
>>> >>> >> > >>> trunk to 3.3 if there are no issues.
>>> >>> >> > >>>
>>> >>> >> > >>> *Current Release Plan:*
>>> >>> >> > >>> Feature freeze date : all features to merge by September 7,
>>> >>> 2018.
>>> >>> >> > >>> Code freeze date : blockers/critical only, no improvements
>>> and
>>> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>>> >>> >> > >>> Release date: September 28, 2018
>>> >>> >> > >>>
>>> >>> >> > >>> If any critical/blocker tickets which are targeted to
>>> 3.2.0, we
>>> >>> >> need to
>>> >>> >> > >>> backport to 3.2 post branch cut.
>>> >>> >> > >>>
>>> >>> >> > >>> Here's an updated 3.2.0 feature status:
>>> >>> >> > >>>
>>> >>> >> > >>> 1. Merged & Completed features:
>>> >>> >> > >>>
>>> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
>>> DeepLearning
>>> >>> >> > >>> workloads Initial cut.
>>> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in
>>> Capacity
>>> >>> >> > >>> Scheduler.
>>> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> >>> Service
>>> >>> >> > API
>>> >>> >> > >>> and CLI.
>>> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
>>> Improvement
>>> >>> >> works.
>>> >>> >> > >>>
>>> >>> >> > >>> 2. Features close to finish:
>>> >>> >> > >>>
>>> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>>> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
>>> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>>> >>> >> > >>>
>>> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>>> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>>> apps
>>> >>> from
>>> >>> >> > >>> ATSv2. Patch in progress.
>>> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >>> >> challenging to
>>> >>> >> > >>> be done before Aug 2018.
>>> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN
>>> applications.
>>> >>> >> > >>> Challenging as more discussions are on-going.
>>> >>> >> > >>>
>>> >>> >> > >>> *Summary of 3.2.0 issues status:*
>>> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following
>>> up
>>> >>> with
>>> >>> >> > >>> owners to get status on each of them to get in by Code
>>> Freeze
>>> >>> date.
>>> >>> >> > >>>
>>> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
>>> in
>>> >>> >> > (Blocker,
>>> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target
>>> Version/s" =
>>> >>> 3.2.0
>>> >>> >> > ORDER
>>> >>> >> > >>> BY priority DESC
>>> >>> >> > >>>
>>> >>> >> > >>> Thanks,
>>> >>> >> > >>> Sunil
>>> >>> >> > >>>
>>> >>> >> > >>>
>>> >>> >> > >>>
>>> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <sunilg@apache.org
>>> <ma...@apache.org>>
>>> >>> wrote:
>>> >>> >> > >>>
>>> >>> >> > >>>> Hi All,
>>> >>> >> > >>>>
>>> >>> >> > >>>> Inline with earlier communication dated 17th July 2018, I
>>> would
>>> >>> >> like
>>> >>> >> > to
>>> >>> >> > >>>> provide some updates.
>>> >>> >> > >>>>
>>> >>> >> > >>>> We are approaching previously proposed code freeze date
>>> (Aug
>>> >>> 31).
>>> >>> >> > >>>>
>>> >>> >> > >>>> One of the critical feature Node Attributes feature merge
>>> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs
>>> need a
>>> >>> bit
>>> >>> >> > more
>>> >>> >> > >>>> time. With regard to this, suggesting to push the
>>> feature/code
>>> >>> >> freeze
>>> >>> >> > for 2
>>> >>> >> > >>>> more weeks to accommodate these jiras too.
>>> >>> >> > >>>>
>>> >>> >> > >>>> Proposing Updated changes in plan inline with this:
>>> >>> >> > >>>> Feature freeze date : all features to merge by September 7,
>>> >>> 2018.
>>> >>> >> > >>>> Code freeze date : blockers/critical only, no improvements
>>> and
>>> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>>> >>> >> > >>>> Release date: September 28, 2018
>>> >>> >> > >>>>
>>> >>> >> > >>>> If any features in branch which are targeted to 3.2.0,
>>> please
>>> >>> >> reply to
>>> >>> >> > >>>> this email thread.
>>> >>> >> > >>>>
>>> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>>> >>> >> > >>>>
>>> >>> >> > >>>> 1. Merged & Completed features:
>>> >>> >> > >>>>
>>> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>>> DeepLearning
>>> >>> >> > >>>> workloads Initial cut.
>>> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>>> Capacity
>>> >>> >> > >>>> Scheduler.
>>> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>>> YARN
>>> >>> >> Service
>>> >>> >> > >>>> API and CLI.
>>> >>> >> > >>>>
>>> >>> >> > >>>> 2. Features close to finish:
>>> >>> >> > >>>>
>>> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >>> >> Merge/Vote
>>> >>> >> > >>>> Ongoing.
>>> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>>> apps
>>> >>> from
>>> >>> >> > >>>> ATSv2. Patch in progress.
>>> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> >>> Improvement
>>> >>> >> > works.
>>> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>>> Azure
>>> >>> >> > >>>> Storage. In progress.
>>> >>> >> > >>>>
>>> >>> >> > >>>> 3. Tentative features:
>>> >>> >> > >>>>
>>> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >>> >> challenging
>>> >>> >> > to
>>> >>> >> > >>>> be done before Aug 2018.
>>> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
>>> applications.
>>> >>> >> > >>>> Challenging as more discussions are on-going.
>>> >>> >> > >>>>
>>> >>> >> > >>>> *Summary of 3.2.0 issues status:*
>>> >>> >> > >>>>
>>> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am
>>> following up
>>> >>> with
>>> >>> >> > >>>> owners to get status on each of them to get in by Code
>>> Freeze
>>> >>> date.
>>> >>> >> > >>>>
>>> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>>> priority in
>>> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>>> >>> >> > Version/s" =
>>> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
>>> >>> >> > >>>>
>>> >>> >> > >>>> Thanks,
>>> >>> >> > >>>> Sunil
>>> >>> >> > >>>>
>>> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <
>>> sunilg@apache.org<ma...@apache.org>>
>>> >>> >> wrote:
>>> >>> >> > >>>>
>>> >>> >> > >>>>> Hi All,
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier
>>> communication
>>> >>> dated
>>> >>> >> > 17th
>>> >>> >> > >>>>> July 2018, I would like to provide some updates.
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> We are approaching previously proposed feature freeze date
>>> >>> (Aug
>>> >>> >> 21,
>>> >>> >> > >>>>> about 7 days from today).
>>> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
>>> please
>>> >>> >> reply
>>> >>> >> > to
>>> >>> >> > >>>>> this email thread.
>>> >>> >> > >>>>> Steve has mentioned about the s3 features which will come
>>> >>> close to
>>> >>> >> > >>>>> Code Freeze Date (Aug 31st).
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> 1. Merged & Completed features:
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>>> >>> DeepLearning
>>> >>> >> > >>>>> workloads Initial cut.
>>> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> 2. Features close to finish:
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >>> Major
>>> >>> >> > >>>>> patches are all in, only one last
>>> >>> >> > >>>>> patch is in review state.
>>> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>>> Capacity
>>> >>> >> > >>>>> Scheduler. Close to commit.
>>> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>>> YARN
>>> >>> >> Service
>>> >>> >> > >>>>> API and CLI. 2 patches are pending
>>> >>> >> > >>>>> which will be closed by Feature freeze date.
>>> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>>> apps
>>> >>> >> from
>>> >>> >> > >>>>> ATSv2. Patch in progress.
>>> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> >>> Improvement
>>> >>> >> > >>>>> works.
>>> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>>> >>> Azure
>>> >>> >> > >>>>> Storage. In progress.
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> 3. Tentative features:
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >>> >> challenging
>>> >>> >> > >>>>> to be done before Aug 2018.
>>> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
>>> applications.
>>> >>> >> > >>>>> Challenging as more discussions are on-going.
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
>>> >>> with
>>> >>> >> > owners
>>> >>> >> > >>>>> to get status on each of them to get in by Code Freeze
>>> date.
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>>> priority in
>>> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND
>>> "Target
>>> >>> >> > Version/s" =
>>> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> Thanks,
>>> >>> >> > >>>>> Sunil
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <
>>> sunilg@apache.org<ma...@apache.org>>
>>> >>> >> wrote:
>>> >>> >> > >>>>>
>>> >>> >> > >>>>>> Thanks Subru for the thoughts.
>>> >>> >> > >>>>>> One of the main reason for a major release is to push out
>>> >>> >> critical
>>> >>> >> > >>>>>> features with a faster cadence to the users. If we are
>>> >>> pulling
>>> >>> >> more
>>> >>> >> > and
>>> >>> >> > >>>>>> more different types of features to a minor release, that
>>> >>> branch
>>> >>> >> > will
>>> >>> >> > >>>>>> become more destabilized and it may be tough to say that
>>> >>> 3.1.2 is
>>> >>> >> > stable
>>> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and
>>> stabilize
>>> >>> >> features
>>> >>> >> > in
>>> >>> >> > >>>>>> subsequent minor release.
>>> >>> >> > >>>>>> For few companies, it makes sense to push out these new
>>> >>> features
>>> >>> >> > >>>>>> faster to make a reach to the users. Adding to the point
>>> to
>>> >>> the
>>> >>> >> > backporting
>>> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround
>>> that
>>> >>> with
>>> >>> >> > some git
>>> >>> >> > >>>>>> scripts. If we can make such scripts available to
>>> committers,
>>> >>> >> > backport will
>>> >>> >> > >>>>>> be seem-less across branches and we can achieve the
>>> faster
>>> >>> >> release
>>> >>> >> > cadence
>>> >>> >> > >>>>>> also.
>>> >>> >> > >>>>>>
>>> >>> >> > >>>>>> Thoughts?
>>> >>> >> > >>>>>>
>>> >>> >> > >>>>>> - Sunil
>>> >>> >> > >>>>>>
>>> >>> >> > >>>>>>
>>> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>>> >>> subru@apache.org<ma...@apache.org>
>>> >>> >> >
>>> >>> >> > >>>>>> wrote:
>>> >>> >> > >>>>>>
>>> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release
>>> effort. I
>>> >>> am
>>> >>> >> > >>>>>>> generally
>>> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a
>>> 3.1.x) as
>>> >>> >> feel we
>>> >>> >> > >>>>>>> already
>>> >>> >> > >>>>>>> have too many branches to be maintained. I already see
>>> many
>>> >>> >> commits
>>> >>> >> > >>>>>>> are in
>>> >>> >> > >>>>>>> different branches with no apparent rationale, for e.g:
>>> 3.1
>>> >>> has
>>> >>> >> > >>>>>>> commits
>>> >>> >> > >>>>>>> which are absent in 3.0 etc.
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any
>>> major
>>> >>> >> > production
>>> >>> >> > >>>>>>> setting so the cost of adding features should be
>>> minimal.
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>> Thoughts?
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>> -Subru
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
>>> >>> sunilg@apache.org<ma...@apache.org>>
>>> >>> >> > wrote:
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > Yes, important changes and features are much needed,
>>> >>> hence we
>>> >>> >> > will
>>> >>> >> > >>>>>>> be
>>> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
>>> >>> considering
>>> >>> >> few
>>> >>> >> > >>>>>>> more
>>> >>> >> > >>>>>>> > offline requests from other folks, I think extending
>>> the
>>> >>> >> > timeframe
>>> >>> >> > >>>>>>> by
>>> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
>>> >>> buffer) and
>>> >>> >> > >>>>>>> this should
>>> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap
>>> wiki as
>>> >>> >> well)
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by
>>> August
>>> >>> 21,
>>> >>> >> 2018.
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>>> >>> improvements
>>> >>> >> and
>>> >>> >> > >>>>>>> non
>>> >>> >> > >>>>>>> > blocker/critical
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > - Release date: September 15, 2018
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already
>>> answered
>>> >>> your
>>> >>> >> > >>>>>>> questions.
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > Thanks
>>> >>> >> > >>>>>>> > Sunil
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>>> >>> >> wheeleast@gmail.com<ma...@gmail.com>
>>> >>> >> > >
>>> >>> >> > >>>>>>> wrote:
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2
>>> release,
>>> >>> +1
>>> >>> >> for
>>> >>> >> > >>>>>>> that.
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > To concerns from Steve,
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
>>> important
>>> >>> >> > changes
>>> >>> >> > >>>>>>> /
>>> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep
>>> the
>>> >>> >> proposed
>>> >>> >> > >>>>>>> release
>>> >>> >> > >>>>>>> > date
>>> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of
>>> last
>>> >>> >> minute
>>> >>> >> > >>>>>>> and we all
>>> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm
>>> also
>>> >>> fine
>>> >>> >> if
>>> >>> >> > >>>>>>> we want
>>> >>> >> > >>>>>>> > get
>>> >>> >> > >>>>>>> > > another several weeks time.
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
>>> >>> before
>>> >>> >> > >>>>>>> thanksgiving.
>>> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > Eric,
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
>>> >>> YARN-8135
>>> >>> >> can
>>> >>> >> > >>>>>>> get merged
>>> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > Sunil,
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > Could u update
>>> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>>> >>> >> > >>>>>>> > Roadmap
>>> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
>>> >>> first
>>> >>> >> > >>>>>>> before getting
>>> >>> >> > >>>>>>> > > consensus of time.
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > Thanks,
>>> >>> >> > >>>>>>> > > Wangda
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid<mailto:
>>> fabbri@cloudera.com.invalid>
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > wrote:
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>> >>> >> > >>>>>>> stevel@hortonworks.com<ma...@hortonworks.com>>
>>> >>> >> > >>>>>>> > >> wrote:
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
>>> sunilg@apache.org<ma...@apache.org>
>>> >>> >> > <mailto:
>>> >>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>>
>>> wrote:
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > I would also would like to take this opportunity
>>> to
>>> >>> come
>>> >>> >> up
>>> >>> >> > >>>>>>> with a
>>> >>> >> > >>>>>>> > >> detailed
>>> >>> >> > >>>>>>> > >> > plan.
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
>>> >>> merged by
>>> >>> >> > >>>>>>> August 10,
>>> >>> >> > >>>>>>> > >> 2018.
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > <snip>
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
>>> targeted
>>> >>> to
>>> >>> >> 3.2
>>> >>> >> > >>>>>>> per this
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> https://issues.apache.org/jira/browse/HADOOP-15226
>>> >>> >> > >>>>>>> > >> >
>>> https://issues.apache.org/jira/browse/HADOOP-15220
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
>>> >>> Datalake
>>> >>> >> > Gen
>>> >>> >> > >>>>>>> 2
>>> >>> >> > >>>>>>> > >> >
>>> https://issues.apache.org/jira/browse/HADOOP-15407
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
>>> >>> doing
>>> >>> >> work
>>> >>> >> > >>>>>>> on it, but
>>> >>> >> > >>>>>>> > >> if
>>> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect
>>> some
>>> >>> >> timely
>>> >>> >> > >>>>>>> reviews on
>>> >>> >> > >>>>>>> > >> >
>>> https://issues.apache.org/jira/browse/HADOOP-15546
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will
>>> review
>>> >>> the
>>> >>> >> S3 &
>>> >>> >> > >>>>>>> S3Guard
>>> >>> >> > >>>>>>> > >> JIRAs
>>> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there
>>> are
>>> >>> some
>>> >>> >> > >>>>>>> pressing
>>> >>> >> > >>>>>>> > "guave,
>>> >>> >> > >>>>>>> > >> > java 9 prep"
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> I can help with this part if you like.
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
>>> release
>>> >>> >> > manager
>>> >>> >> > >>>>>>> of 3.2.0
>>> >>> >> > >>>>>>> > >> > release.
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > well volunteered!
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable
>>> should
>>> >>> we
>>> >>> >> have
>>> >>> >> > >>>>>>> for the
>>> >>> >> > >>>>>>> > >> 3.2. &
>>> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence,
>>> then
>>> >>> having
>>> >>> >> > the
>>> >>> >> > >>>>>>> outline
>>> >>> >> > >>>>>>> > >> time
>>> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that
>>> there's
>>> >>> less
>>> >>> >> > >>>>>>> concern about
>>> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > -Steve
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >> -AF
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>
>>> >>> >> >
>>> >>> >>
>>> >>> >
>>> >>>
>>> >>
>>> >>
>>>
>>>
>>> --
>>> A very happy Clouderan
>>>
>>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Konstantin Shvachko <sh...@gmail.com>.
Another thing is that I see a bunch of jiras under HDFS-8707, which don't
have the Fix Version field listing 3.2, some have it just empty.
This means they will not be populated into release notes.
Thanks,
--Konstantin
On Thu, Oct 25, 2018 at 7:59 AM Sunil G <su...@apache.org> wrote:
> Thanks Konstantin for pointing out.
> As 3.2 is pretty much on RC level, its better we try to find a good
> solution to this issue.
>
> I ll follow up on this in the jira.
>
> - Sunil
>
> On Thu, Oct 25, 2018 at 11:35 AM Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
>> I've tried to attract attention to an incompatibility issue through the
>> jira, but it didn't work. So pitching in in this thread.
>> https://issues.apache.org/jira/browse/HDFS-12026
>> It introduced binary incompatibility, which will prevent people from
>> upgrading from 3.1 to 3.2.
>> I think it can get messy if we release anything with this feature.
>>
>> Thanks,
>> --Konstantin
>>
>> On Mon, Oct 22, 2018 at 5:01 AM Steve Loughran <st...@hortonworks.com>
>> wrote:
>>
>>> its in.
>>>
>>> good catch!
>>>
>>> On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <weichiu@cloudera.com<mailto:
>>> weichiu@cloudera.com>> wrote:
>>>
>>> Thanks Sunil G for driving the release,
>>> I filed HADOOP-15866<https://issues.apache.org/jira/browse/HADOOP-15866>
>>> for a compat fix. If any one has cycle please review it, as I think it is
>>> needed for 3.2.0.
>>>
>>> On Thu, Oct 18, 2018 at 4:43 AM Sunil G <sunilg@apache.org<mailto:
>>> sunilg@apache.org>> wrote:
>>> Hi Folks,
>>>
>>> As we previously communicated for 3.2.0 release, we have delayed due to
>>> few
>>> blockers in our gate.
>>>
>>> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
>>> all bug fixes.
>>>
>>> - Sunil
>>>
>>>
>>> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <sunilg@apache.org<mailto:
>>> sunilg@apache.org>> wrote:
>>>
>>> > Hi Folks,
>>> >
>>> > We are now close to RC as other blocker issues are now merged to trunk
>>> and
>>> > branch-3.2. Last 2 critical issues are closer to merge and will be
>>> > committed in few hours.
>>> > With this, I will be creating 3.2.0 branch today and will go ahead
>>> with RC
>>> > related process.
>>> >
>>> > - Sunil
>>> >
>>> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jonbender@stripe.com
>>> <ma...@stripe.com>>
>>> > wrote:
>>> >
>>> >> Hello, were there any updates around the 3.2.0 RC timing? All I see in
>>> >> the current blockers are related to the new Submarine subproject,
>>> wasn't
>>> >> sure if that is what is holding things up.
>>> >>
>>> >> Cheers,
>>> >> Jon
>>> >>
>>> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <sunilg@apache.org<mailto:
>>> sunilg@apache.org>> wrote:
>>> >>
>>> >>> Thanks Robert and Haibo for quickly correcting same.
>>> >>> Sigh, I somehow missed one file while committing the change. Sorry
>>> for
>>> >>> the
>>> >>> trouble.
>>> >>>
>>> >>> - Sunil
>>> >>>
>>> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rkanter@cloudera.com
>>> <ma...@cloudera.com>>
>>> >>> wrote:
>>> >>>
>>> >>> > Looks like there's two that weren't updated:
>>> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" .
>>> -r
>>> >>> > --include=pom.xml
>>> >>> > ./hadoop-project/pom.xml:
>>> >>> >
>>> <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>>> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>>> >>> >
>>> >>> > I've just pushed in an addendum commit to fix those.
>>> >>> > In the future, please make sure to do a sanity compile when
>>> updating
>>> >>> poms.
>>> >>> >
>>> >>> > thanks
>>> >>> > - Robert
>>> >>> >
>>> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>>> >>> <fa...@cloudera.com.invalid>>
>>> >>> > wrote:
>>> >>> >
>>> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in
>>> the
>>> >>> >> top-level pom.xml?
>>> >>> >>
>>> >>> >>
>>> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <sunilg@apache.org
>>> <ma...@apache.org>> wrote:
>>> >>> >>
>>> >>> >> > Hi All
>>> >>> >> >
>>> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
>>> trunk
>>> >>> to
>>> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
>>> >>> necessary
>>> >>> >> > patches are pulled into branch-3.2.
>>> >>> >> >
>>> >>> >> > Thank You
>>> >>> >> > - Sunil
>>> >>> >> >
>>> >>> >> >
>>> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <sunilg@apache.org
>>> <ma...@apache.org>> wrote:
>>> >>> >> >
>>> >>> >> > > Hi All
>>> >>> >> > >
>>> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is
>>> merged to
>>> >>> >> trunk.
>>> >>> >> > > Thanks for the support.
>>> >>> >> > >
>>> >>> >> > > *Plan for RC*
>>> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>>> >>> >> > > RC0 for 3.2: *28th Friday*
>>> >>> >> > >
>>> >>> >> > > Thank You
>>> >>> >> > > Sunil
>>> >>> >> > >
>>> >>> >> > >
>>> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <sunilg@apache.org
>>> <ma...@apache.org>>
>>> >>> wrote:
>>> >>> >> > >
>>> >>> >> > >> Hi All
>>> >>> >> > >>
>>> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of
>>> you
>>> >>> for
>>> >>> >> > >> helping in this. I am following up on these tickets, once its
>>> >>> closed
>>> >>> >> we
>>> >>> >> > >> will cut the 3.2 branch.
>>> >>> >> > >>
>>> >>> >> > >> Thanks
>>> >>> >> > >> Sunil Govindan
>>> >>> >> > >>
>>> >>> >> > >>
>>> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <sunilg@apache.org
>>> <ma...@apache.org>>
>>> >>> wrote:
>>> >>> >> > >>
>>> >>> >> > >>> Hi All,
>>> >>> >> > >>>
>>> >>> >> > >>> Inline with the original 3.2 communication proposal dated
>>> 17th
>>> >>> July
>>> >>> >> > >>> 2018, I would like to provide more updates.
>>> >>> >> > >>>
>>> >>> >> > >>> We are approaching previously proposed code freeze date
>>> >>> (September
>>> >>> >> 14,
>>> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
>>> point
>>> >>> >> > existing
>>> >>> >> > >>> trunk to 3.3 if there are no issues.
>>> >>> >> > >>>
>>> >>> >> > >>> *Current Release Plan:*
>>> >>> >> > >>> Feature freeze date : all features to merge by September 7,
>>> >>> 2018.
>>> >>> >> > >>> Code freeze date : blockers/critical only, no improvements
>>> and
>>> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>>> >>> >> > >>> Release date: September 28, 2018
>>> >>> >> > >>>
>>> >>> >> > >>> If any critical/blocker tickets which are targeted to
>>> 3.2.0, we
>>> >>> >> need to
>>> >>> >> > >>> backport to 3.2 post branch cut.
>>> >>> >> > >>>
>>> >>> >> > >>> Here's an updated 3.2.0 feature status:
>>> >>> >> > >>>
>>> >>> >> > >>> 1. Merged & Completed features:
>>> >>> >> > >>>
>>> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
>>> DeepLearning
>>> >>> >> > >>> workloads Initial cut.
>>> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in
>>> Capacity
>>> >>> >> > >>> Scheduler.
>>> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> >>> Service
>>> >>> >> > API
>>> >>> >> > >>> and CLI.
>>> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
>>> Improvement
>>> >>> >> works.
>>> >>> >> > >>>
>>> >>> >> > >>> 2. Features close to finish:
>>> >>> >> > >>>
>>> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>>> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
>>> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>>> >>> >> > >>>
>>> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>>> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>>> apps
>>> >>> from
>>> >>> >> > >>> ATSv2. Patch in progress.
>>> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >>> >> challenging to
>>> >>> >> > >>> be done before Aug 2018.
>>> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN
>>> applications.
>>> >>> >> > >>> Challenging as more discussions are on-going.
>>> >>> >> > >>>
>>> >>> >> > >>> *Summary of 3.2.0 issues status:*
>>> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following
>>> up
>>> >>> with
>>> >>> >> > >>> owners to get status on each of them to get in by Code
>>> Freeze
>>> >>> date.
>>> >>> >> > >>>
>>> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
>>> in
>>> >>> >> > (Blocker,
>>> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target
>>> Version/s" =
>>> >>> 3.2.0
>>> >>> >> > ORDER
>>> >>> >> > >>> BY priority DESC
>>> >>> >> > >>>
>>> >>> >> > >>> Thanks,
>>> >>> >> > >>> Sunil
>>> >>> >> > >>>
>>> >>> >> > >>>
>>> >>> >> > >>>
>>> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <sunilg@apache.org
>>> <ma...@apache.org>>
>>> >>> wrote:
>>> >>> >> > >>>
>>> >>> >> > >>>> Hi All,
>>> >>> >> > >>>>
>>> >>> >> > >>>> Inline with earlier communication dated 17th July 2018, I
>>> would
>>> >>> >> like
>>> >>> >> > to
>>> >>> >> > >>>> provide some updates.
>>> >>> >> > >>>>
>>> >>> >> > >>>> We are approaching previously proposed code freeze date
>>> (Aug
>>> >>> 31).
>>> >>> >> > >>>>
>>> >>> >> > >>>> One of the critical feature Node Attributes feature merge
>>> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs
>>> need a
>>> >>> bit
>>> >>> >> > more
>>> >>> >> > >>>> time. With regard to this, suggesting to push the
>>> feature/code
>>> >>> >> freeze
>>> >>> >> > for 2
>>> >>> >> > >>>> more weeks to accommodate these jiras too.
>>> >>> >> > >>>>
>>> >>> >> > >>>> Proposing Updated changes in plan inline with this:
>>> >>> >> > >>>> Feature freeze date : all features to merge by September 7,
>>> >>> 2018.
>>> >>> >> > >>>> Code freeze date : blockers/critical only, no improvements
>>> and
>>> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>>> >>> >> > >>>> Release date: September 28, 2018
>>> >>> >> > >>>>
>>> >>> >> > >>>> If any features in branch which are targeted to 3.2.0,
>>> please
>>> >>> >> reply to
>>> >>> >> > >>>> this email thread.
>>> >>> >> > >>>>
>>> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>>> >>> >> > >>>>
>>> >>> >> > >>>> 1. Merged & Completed features:
>>> >>> >> > >>>>
>>> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>>> DeepLearning
>>> >>> >> > >>>> workloads Initial cut.
>>> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>>> Capacity
>>> >>> >> > >>>> Scheduler.
>>> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>>> YARN
>>> >>> >> Service
>>> >>> >> > >>>> API and CLI.
>>> >>> >> > >>>>
>>> >>> >> > >>>> 2. Features close to finish:
>>> >>> >> > >>>>
>>> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >>> >> Merge/Vote
>>> >>> >> > >>>> Ongoing.
>>> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>>> apps
>>> >>> from
>>> >>> >> > >>>> ATSv2. Patch in progress.
>>> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> >>> Improvement
>>> >>> >> > works.
>>> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>>> Azure
>>> >>> >> > >>>> Storage. In progress.
>>> >>> >> > >>>>
>>> >>> >> > >>>> 3. Tentative features:
>>> >>> >> > >>>>
>>> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >>> >> challenging
>>> >>> >> > to
>>> >>> >> > >>>> be done before Aug 2018.
>>> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
>>> applications.
>>> >>> >> > >>>> Challenging as more discussions are on-going.
>>> >>> >> > >>>>
>>> >>> >> > >>>> *Summary of 3.2.0 issues status:*
>>> >>> >> > >>>>
>>> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am
>>> following up
>>> >>> with
>>> >>> >> > >>>> owners to get status on each of them to get in by Code
>>> Freeze
>>> >>> date.
>>> >>> >> > >>>>
>>> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>>> priority in
>>> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>>> >>> >> > Version/s" =
>>> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
>>> >>> >> > >>>>
>>> >>> >> > >>>> Thanks,
>>> >>> >> > >>>> Sunil
>>> >>> >> > >>>>
>>> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <
>>> sunilg@apache.org<ma...@apache.org>>
>>> >>> >> wrote:
>>> >>> >> > >>>>
>>> >>> >> > >>>>> Hi All,
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier
>>> communication
>>> >>> dated
>>> >>> >> > 17th
>>> >>> >> > >>>>> July 2018, I would like to provide some updates.
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> We are approaching previously proposed feature freeze date
>>> >>> (Aug
>>> >>> >> 21,
>>> >>> >> > >>>>> about 7 days from today).
>>> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
>>> please
>>> >>> >> reply
>>> >>> >> > to
>>> >>> >> > >>>>> this email thread.
>>> >>> >> > >>>>> Steve has mentioned about the s3 features which will come
>>> >>> close to
>>> >>> >> > >>>>> Code Freeze Date (Aug 31st).
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> 1. Merged & Completed features:
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>>> >>> DeepLearning
>>> >>> >> > >>>>> workloads Initial cut.
>>> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> 2. Features close to finish:
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >>> Major
>>> >>> >> > >>>>> patches are all in, only one last
>>> >>> >> > >>>>> patch is in review state.
>>> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>>> Capacity
>>> >>> >> > >>>>> Scheduler. Close to commit.
>>> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>>> YARN
>>> >>> >> Service
>>> >>> >> > >>>>> API and CLI. 2 patches are pending
>>> >>> >> > >>>>> which will be closed by Feature freeze date.
>>> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>>> apps
>>> >>> >> from
>>> >>> >> > >>>>> ATSv2. Patch in progress.
>>> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> >>> Improvement
>>> >>> >> > >>>>> works.
>>> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>>> >>> Azure
>>> >>> >> > >>>>> Storage. In progress.
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> 3. Tentative features:
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >>> >> challenging
>>> >>> >> > >>>>> to be done before Aug 2018.
>>> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
>>> applications.
>>> >>> >> > >>>>> Challenging as more discussions are on-going.
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
>>> >>> with
>>> >>> >> > owners
>>> >>> >> > >>>>> to get status on each of them to get in by Code Freeze
>>> date.
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>>> priority in
>>> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND
>>> "Target
>>> >>> >> > Version/s" =
>>> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> Thanks,
>>> >>> >> > >>>>> Sunil
>>> >>> >> > >>>>>
>>> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <
>>> sunilg@apache.org<ma...@apache.org>>
>>> >>> >> wrote:
>>> >>> >> > >>>>>
>>> >>> >> > >>>>>> Thanks Subru for the thoughts.
>>> >>> >> > >>>>>> One of the main reason for a major release is to push out
>>> >>> >> critical
>>> >>> >> > >>>>>> features with a faster cadence to the users. If we are
>>> >>> pulling
>>> >>> >> more
>>> >>> >> > and
>>> >>> >> > >>>>>> more different types of features to a minor release, that
>>> >>> branch
>>> >>> >> > will
>>> >>> >> > >>>>>> become more destabilized and it may be tough to say that
>>> >>> 3.1.2 is
>>> >>> >> > stable
>>> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and
>>> stabilize
>>> >>> >> features
>>> >>> >> > in
>>> >>> >> > >>>>>> subsequent minor release.
>>> >>> >> > >>>>>> For few companies, it makes sense to push out these new
>>> >>> features
>>> >>> >> > >>>>>> faster to make a reach to the users. Adding to the point
>>> to
>>> >>> the
>>> >>> >> > backporting
>>> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround
>>> that
>>> >>> with
>>> >>> >> > some git
>>> >>> >> > >>>>>> scripts. If we can make such scripts available to
>>> committers,
>>> >>> >> > backport will
>>> >>> >> > >>>>>> be seem-less across branches and we can achieve the
>>> faster
>>> >>> >> release
>>> >>> >> > cadence
>>> >>> >> > >>>>>> also.
>>> >>> >> > >>>>>>
>>> >>> >> > >>>>>> Thoughts?
>>> >>> >> > >>>>>>
>>> >>> >> > >>>>>> - Sunil
>>> >>> >> > >>>>>>
>>> >>> >> > >>>>>>
>>> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>>> >>> subru@apache.org<ma...@apache.org>
>>> >>> >> >
>>> >>> >> > >>>>>> wrote:
>>> >>> >> > >>>>>>
>>> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release
>>> effort. I
>>> >>> am
>>> >>> >> > >>>>>>> generally
>>> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a
>>> 3.1.x) as
>>> >>> >> feel we
>>> >>> >> > >>>>>>> already
>>> >>> >> > >>>>>>> have too many branches to be maintained. I already see
>>> many
>>> >>> >> commits
>>> >>> >> > >>>>>>> are in
>>> >>> >> > >>>>>>> different branches with no apparent rationale, for e.g:
>>> 3.1
>>> >>> has
>>> >>> >> > >>>>>>> commits
>>> >>> >> > >>>>>>> which are absent in 3.0 etc.
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any
>>> major
>>> >>> >> > production
>>> >>> >> > >>>>>>> setting so the cost of adding features should be
>>> minimal.
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>> Thoughts?
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>> -Subru
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
>>> >>> sunilg@apache.org<ma...@apache.org>>
>>> >>> >> > wrote:
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > Yes, important changes and features are much needed,
>>> >>> hence we
>>> >>> >> > will
>>> >>> >> > >>>>>>> be
>>> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
>>> >>> considering
>>> >>> >> few
>>> >>> >> > >>>>>>> more
>>> >>> >> > >>>>>>> > offline requests from other folks, I think extending
>>> the
>>> >>> >> > timeframe
>>> >>> >> > >>>>>>> by
>>> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
>>> >>> buffer) and
>>> >>> >> > >>>>>>> this should
>>> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap
>>> wiki as
>>> >>> >> well)
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by
>>> August
>>> >>> 21,
>>> >>> >> 2018.
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>>> >>> improvements
>>> >>> >> and
>>> >>> >> > >>>>>>> non
>>> >>> >> > >>>>>>> > blocker/critical
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > - Release date: September 15, 2018
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already
>>> answered
>>> >>> your
>>> >>> >> > >>>>>>> questions.
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > Thanks
>>> >>> >> > >>>>>>> > Sunil
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>>> >>> >> wheeleast@gmail.com<ma...@gmail.com>
>>> >>> >> > >
>>> >>> >> > >>>>>>> wrote:
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2
>>> release,
>>> >>> +1
>>> >>> >> for
>>> >>> >> > >>>>>>> that.
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > To concerns from Steve,
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
>>> important
>>> >>> >> > changes
>>> >>> >> > >>>>>>> /
>>> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep
>>> the
>>> >>> >> proposed
>>> >>> >> > >>>>>>> release
>>> >>> >> > >>>>>>> > date
>>> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of
>>> last
>>> >>> >> minute
>>> >>> >> > >>>>>>> and we all
>>> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm
>>> also
>>> >>> fine
>>> >>> >> if
>>> >>> >> > >>>>>>> we want
>>> >>> >> > >>>>>>> > get
>>> >>> >> > >>>>>>> > > another several weeks time.
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
>>> >>> before
>>> >>> >> > >>>>>>> thanksgiving.
>>> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > Eric,
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
>>> >>> YARN-8135
>>> >>> >> can
>>> >>> >> > >>>>>>> get merged
>>> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > Sunil,
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > Could u update
>>> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>>> >>> >> > >>>>>>> > Roadmap
>>> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
>>> >>> first
>>> >>> >> > >>>>>>> before getting
>>> >>> >> > >>>>>>> > > consensus of time.
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > Thanks,
>>> >>> >> > >>>>>>> > > Wangda
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid<mailto:
>>> fabbri@cloudera.com.invalid>
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > > wrote:
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>> >>> >> > >>>>>>> stevel@hortonworks.com<ma...@hortonworks.com>>
>>> >>> >> > >>>>>>> > >> wrote:
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
>>> sunilg@apache.org<ma...@apache.org>
>>> >>> >> > <mailto:
>>> >>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>>
>>> wrote:
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > I would also would like to take this opportunity
>>> to
>>> >>> come
>>> >>> >> up
>>> >>> >> > >>>>>>> with a
>>> >>> >> > >>>>>>> > >> detailed
>>> >>> >> > >>>>>>> > >> > plan.
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
>>> >>> merged by
>>> >>> >> > >>>>>>> August 10,
>>> >>> >> > >>>>>>> > >> 2018.
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > <snip>
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
>>> targeted
>>> >>> to
>>> >>> >> 3.2
>>> >>> >> > >>>>>>> per this
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> https://issues.apache.org/jira/browse/HADOOP-15226
>>> >>> >> > >>>>>>> > >> >
>>> https://issues.apache.org/jira/browse/HADOOP-15220
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
>>> >>> Datalake
>>> >>> >> > Gen
>>> >>> >> > >>>>>>> 2
>>> >>> >> > >>>>>>> > >> >
>>> https://issues.apache.org/jira/browse/HADOOP-15407
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
>>> >>> doing
>>> >>> >> work
>>> >>> >> > >>>>>>> on it, but
>>> >>> >> > >>>>>>> > >> if
>>> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect
>>> some
>>> >>> >> timely
>>> >>> >> > >>>>>>> reviews on
>>> >>> >> > >>>>>>> > >> >
>>> https://issues.apache.org/jira/browse/HADOOP-15546
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will
>>> review
>>> >>> the
>>> >>> >> S3 &
>>> >>> >> > >>>>>>> S3Guard
>>> >>> >> > >>>>>>> > >> JIRAs
>>> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there
>>> are
>>> >>> some
>>> >>> >> > >>>>>>> pressing
>>> >>> >> > >>>>>>> > "guave,
>>> >>> >> > >>>>>>> > >> > java 9 prep"
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> I can help with this part if you like.
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
>>> release
>>> >>> >> > manager
>>> >>> >> > >>>>>>> of 3.2.0
>>> >>> >> > >>>>>>> > >> > release.
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > well volunteered!
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable
>>> should
>>> >>> we
>>> >>> >> have
>>> >>> >> > >>>>>>> for the
>>> >>> >> > >>>>>>> > >> 3.2. &
>>> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence,
>>> then
>>> >>> having
>>> >>> >> > the
>>> >>> >> > >>>>>>> outline
>>> >>> >> > >>>>>>> > >> time
>>> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that
>>> there's
>>> >>> less
>>> >>> >> > >>>>>>> concern about
>>> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> > -Steve
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> >
>>> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >> -AF
>>> >>> >> > >>>>>>> > >>
>>> >>> >> > >>>>>>> > >
>>> >>> >> > >>>>>>> >
>>> >>> >> > >>>>>>>
>>> >>> >> > >>>>>>
>>> >>> >> >
>>> >>> >>
>>> >>> >
>>> >>>
>>> >>
>>> >>
>>>
>>>
>>> --
>>> A very happy Clouderan
>>>
>>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Thanks Konstantin for pointing out.
As 3.2 is pretty much on RC level, its better we try to find a good
solution to this issue.
I ll follow up on this in the jira.
- Sunil
On Thu, Oct 25, 2018 at 11:35 AM Konstantin Shvachko <sh...@gmail.com>
wrote:
> I've tried to attract attention to an incompatibility issue through the
> jira, but it didn't work. So pitching in in this thread.
> https://issues.apache.org/jira/browse/HDFS-12026
> It introduced binary incompatibility, which will prevent people from
> upgrading from 3.1 to 3.2.
> I think it can get messy if we release anything with this feature.
>
> Thanks,
> --Konstantin
>
> On Mon, Oct 22, 2018 at 5:01 AM Steve Loughran <st...@hortonworks.com>
> wrote:
>
>> its in.
>>
>> good catch!
>>
>> On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <weichiu@cloudera.com<mailto:
>> weichiu@cloudera.com>> wrote:
>>
>> Thanks Sunil G for driving the release,
>> I filed HADOOP-15866<https://issues.apache.org/jira/browse/HADOOP-15866>
>> for a compat fix. If any one has cycle please review it, as I think it is
>> needed for 3.2.0.
>>
>> On Thu, Oct 18, 2018 at 4:43 AM Sunil G <sunilg@apache.org<mailto:
>> sunilg@apache.org>> wrote:
>> Hi Folks,
>>
>> As we previously communicated for 3.2.0 release, we have delayed due to
>> few
>> blockers in our gate.
>>
>> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
>> all bug fixes.
>>
>> - Sunil
>>
>>
>> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <sunilg@apache.org<mailto:
>> sunilg@apache.org>> wrote:
>>
>> > Hi Folks,
>> >
>> > We are now close to RC as other blocker issues are now merged to trunk
>> and
>> > branch-3.2. Last 2 critical issues are closer to merge and will be
>> > committed in few hours.
>> > With this, I will be creating 3.2.0 branch today and will go ahead with
>> RC
>> > related process.
>> >
>> > - Sunil
>> >
>> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jonbender@stripe.com
>> <ma...@stripe.com>>
>> > wrote:
>> >
>> >> Hello, were there any updates around the 3.2.0 RC timing? All I see in
>> >> the current blockers are related to the new Submarine subproject,
>> wasn't
>> >> sure if that is what is holding things up.
>> >>
>> >> Cheers,
>> >> Jon
>> >>
>> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <sunilg@apache.org<mailto:
>> sunilg@apache.org>> wrote:
>> >>
>> >>> Thanks Robert and Haibo for quickly correcting same.
>> >>> Sigh, I somehow missed one file while committing the change. Sorry for
>> >>> the
>> >>> trouble.
>> >>>
>> >>> - Sunil
>> >>>
>> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rkanter@cloudera.com
>> <ma...@cloudera.com>>
>> >>> wrote:
>> >>>
>> >>> > Looks like there's two that weren't updated:
>> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
>> >>> > --include=pom.xml
>> >>> > ./hadoop-project/pom.xml:
>> >>> >
>> <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>> >>> >
>> >>> > I've just pushed in an addendum commit to fix those.
>> >>> > In the future, please make sure to do a sanity compile when updating
>> >>> poms.
>> >>> >
>> >>> > thanks
>> >>> > - Robert
>> >>> >
>> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>> >>> <fa...@cloudera.com.invalid>>
>> >>> > wrote:
>> >>> >
>> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
>> >>> >> top-level pom.xml?
>> >>> >>
>> >>> >>
>> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <sunilg@apache.org<mailto:
>> sunilg@apache.org>> wrote:
>> >>> >>
>> >>> >> > Hi All
>> >>> >> >
>> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
>> trunk
>> >>> to
>> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
>> >>> necessary
>> >>> >> > patches are pulled into branch-3.2.
>> >>> >> >
>> >>> >> > Thank You
>> >>> >> > - Sunil
>> >>> >> >
>> >>> >> >
>> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <sunilg@apache.org
>> <ma...@apache.org>> wrote:
>> >>> >> >
>> >>> >> > > Hi All
>> >>> >> > >
>> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged
>> to
>> >>> >> trunk.
>> >>> >> > > Thanks for the support.
>> >>> >> > >
>> >>> >> > > *Plan for RC*
>> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>> >>> >> > > RC0 for 3.2: *28th Friday*
>> >>> >> > >
>> >>> >> > > Thank You
>> >>> >> > > Sunil
>> >>> >> > >
>> >>> >> > >
>> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <sunilg@apache.org
>> <ma...@apache.org>>
>> >>> wrote:
>> >>> >> > >
>> >>> >> > >> Hi All
>> >>> >> > >>
>> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of
>> you
>> >>> for
>> >>> >> > >> helping in this. I am following up on these tickets, once its
>> >>> closed
>> >>> >> we
>> >>> >> > >> will cut the 3.2 branch.
>> >>> >> > >>
>> >>> >> > >> Thanks
>> >>> >> > >> Sunil Govindan
>> >>> >> > >>
>> >>> >> > >>
>> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <sunilg@apache.org
>> <ma...@apache.org>>
>> >>> wrote:
>> >>> >> > >>
>> >>> >> > >>> Hi All,
>> >>> >> > >>>
>> >>> >> > >>> Inline with the original 3.2 communication proposal dated
>> 17th
>> >>> July
>> >>> >> > >>> 2018, I would like to provide more updates.
>> >>> >> > >>>
>> >>> >> > >>> We are approaching previously proposed code freeze date
>> >>> (September
>> >>> >> 14,
>> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
>> point
>> >>> >> > existing
>> >>> >> > >>> trunk to 3.3 if there are no issues.
>> >>> >> > >>>
>> >>> >> > >>> *Current Release Plan:*
>> >>> >> > >>> Feature freeze date : all features to merge by September 7,
>> >>> 2018.
>> >>> >> > >>> Code freeze date : blockers/critical only, no improvements
>> and
>> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>> >>> >> > >>> Release date: September 28, 2018
>> >>> >> > >>>
>> >>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0,
>> we
>> >>> >> need to
>> >>> >> > >>> backport to 3.2 post branch cut.
>> >>> >> > >>>
>> >>> >> > >>> Here's an updated 3.2.0 feature status:
>> >>> >> > >>>
>> >>> >> > >>> 1. Merged & Completed features:
>> >>> >> > >>>
>> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> DeepLearning
>> >>> >> > >>> workloads Initial cut.
>> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> Capacity
>> >>> >> > >>> Scheduler.
>> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> >>> Service
>> >>> >> > API
>> >>> >> > >>> and CLI.
>> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
>> Improvement
>> >>> >> works.
>> >>> >> > >>>
>> >>> >> > >>> 2. Features close to finish:
>> >>> >> > >>>
>> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
>> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>> >>> >> > >>>
>> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>> apps
>> >>> from
>> >>> >> > >>> ATSv2. Patch in progress.
>> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >>> >> challenging to
>> >>> >> > >>> be done before Aug 2018.
>> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN
>> applications.
>> >>> >> > >>> Challenging as more discussions are on-going.
>> >>> >> > >>>
>> >>> >> > >>> *Summary of 3.2.0 issues status:*
>> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following
>> up
>> >>> with
>> >>> >> > >>> owners to get status on each of them to get in by Code Freeze
>> >>> date.
>> >>> >> > >>>
>> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
>> in
>> >>> >> > (Blocker,
>> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s"
>> =
>> >>> 3.2.0
>> >>> >> > ORDER
>> >>> >> > >>> BY priority DESC
>> >>> >> > >>>
>> >>> >> > >>> Thanks,
>> >>> >> > >>> Sunil
>> >>> >> > >>>
>> >>> >> > >>>
>> >>> >> > >>>
>> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <sunilg@apache.org
>> <ma...@apache.org>>
>> >>> wrote:
>> >>> >> > >>>
>> >>> >> > >>>> Hi All,
>> >>> >> > >>>>
>> >>> >> > >>>> Inline with earlier communication dated 17th July 2018, I
>> would
>> >>> >> like
>> >>> >> > to
>> >>> >> > >>>> provide some updates.
>> >>> >> > >>>>
>> >>> >> > >>>> We are approaching previously proposed code freeze date (Aug
>> >>> 31).
>> >>> >> > >>>>
>> >>> >> > >>>> One of the critical feature Node Attributes feature merge
>> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs
>> need a
>> >>> bit
>> >>> >> > more
>> >>> >> > >>>> time. With regard to this, suggesting to push the
>> feature/code
>> >>> >> freeze
>> >>> >> > for 2
>> >>> >> > >>>> more weeks to accommodate these jiras too.
>> >>> >> > >>>>
>> >>> >> > >>>> Proposing Updated changes in plan inline with this:
>> >>> >> > >>>> Feature freeze date : all features to merge by September 7,
>> >>> 2018.
>> >>> >> > >>>> Code freeze date : blockers/critical only, no improvements
>> and
>> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>> >>> >> > >>>> Release date: September 28, 2018
>> >>> >> > >>>>
>> >>> >> > >>>> If any features in branch which are targeted to 3.2.0,
>> please
>> >>> >> reply to
>> >>> >> > >>>> this email thread.
>> >>> >> > >>>>
>> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>> >>> >> > >>>>
>> >>> >> > >>>> 1. Merged & Completed features:
>> >>> >> > >>>>
>> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> DeepLearning
>> >>> >> > >>>> workloads Initial cut.
>> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> Capacity
>> >>> >> > >>>> Scheduler.
>> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> >>> >> Service
>> >>> >> > >>>> API and CLI.
>> >>> >> > >>>>
>> >>> >> > >>>> 2. Features close to finish:
>> >>> >> > >>>>
>> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> >>> >> Merge/Vote
>> >>> >> > >>>> Ongoing.
>> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>> apps
>> >>> from
>> >>> >> > >>>> ATSv2. Patch in progress.
>> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> >>> Improvement
>> >>> >> > works.
>> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>> Azure
>> >>> >> > >>>> Storage. In progress.
>> >>> >> > >>>>
>> >>> >> > >>>> 3. Tentative features:
>> >>> >> > >>>>
>> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >>> >> challenging
>> >>> >> > to
>> >>> >> > >>>> be done before Aug 2018.
>> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
>> applications.
>> >>> >> > >>>> Challenging as more discussions are on-going.
>> >>> >> > >>>>
>> >>> >> > >>>> *Summary of 3.2.0 issues status:*
>> >>> >> > >>>>
>> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following
>> up
>> >>> with
>> >>> >> > >>>> owners to get status on each of them to get in by Code
>> Freeze
>> >>> date.
>> >>> >> > >>>>
>> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
>> in
>> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> >>> >> > Version/s" =
>> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
>> >>> >> > >>>>
>> >>> >> > >>>> Thanks,
>> >>> >> > >>>> Sunil
>> >>> >> > >>>>
>> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <sunilg@apache.org
>> <ma...@apache.org>>
>> >>> >> wrote:
>> >>> >> > >>>>
>> >>> >> > >>>>> Hi All,
>> >>> >> > >>>>>
>> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
>> >>> dated
>> >>> >> > 17th
>> >>> >> > >>>>> July 2018, I would like to provide some updates.
>> >>> >> > >>>>>
>> >>> >> > >>>>> We are approaching previously proposed feature freeze date
>> >>> (Aug
>> >>> >> 21,
>> >>> >> > >>>>> about 7 days from today).
>> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
>> please
>> >>> >> reply
>> >>> >> > to
>> >>> >> > >>>>> this email thread.
>> >>> >> > >>>>> Steve has mentioned about the s3 features which will come
>> >>> close to
>> >>> >> > >>>>> Code Freeze Date (Aug 31st).
>> >>> >> > >>>>>
>> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>> >>> >> > >>>>>
>> >>> >> > >>>>> 1. Merged & Completed features:
>> >>> >> > >>>>>
>> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> >>> DeepLearning
>> >>> >> > >>>>> workloads Initial cut.
>> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >> > >>>>>
>> >>> >> > >>>>> 2. Features close to finish:
>> >>> >> > >>>>>
>> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> >>> Major
>> >>> >> > >>>>> patches are all in, only one last
>> >>> >> > >>>>> patch is in review state.
>> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> Capacity
>> >>> >> > >>>>> Scheduler. Close to commit.
>> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>> YARN
>> >>> >> Service
>> >>> >> > >>>>> API and CLI. 2 patches are pending
>> >>> >> > >>>>> which will be closed by Feature freeze date.
>> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>> apps
>> >>> >> from
>> >>> >> > >>>>> ATSv2. Patch in progress.
>> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> >>> Improvement
>> >>> >> > >>>>> works.
>> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>> >>> Azure
>> >>> >> > >>>>> Storage. In progress.
>> >>> >> > >>>>>
>> >>> >> > >>>>> 3. Tentative features:
>> >>> >> > >>>>>
>> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >>> >> challenging
>> >>> >> > >>>>> to be done before Aug 2018.
>> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
>> applications.
>> >>> >> > >>>>> Challenging as more discussions are on-going.
>> >>> >> > >>>>>
>> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
>> >>> >> > >>>>>
>> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
>> >>> with
>> >>> >> > owners
>> >>> >> > >>>>> to get status on each of them to get in by Code Freeze
>> date.
>> >>> >> > >>>>>
>> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>> priority in
>> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> >>> >> > Version/s" =
>> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>> >>> >> > >>>>>
>> >>> >> > >>>>> Thanks,
>> >>> >> > >>>>> Sunil
>> >>> >> > >>>>>
>> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <sunilg@apache.org
>> <ma...@apache.org>>
>> >>> >> wrote:
>> >>> >> > >>>>>
>> >>> >> > >>>>>> Thanks Subru for the thoughts.
>> >>> >> > >>>>>> One of the main reason for a major release is to push out
>> >>> >> critical
>> >>> >> > >>>>>> features with a faster cadence to the users. If we are
>> >>> pulling
>> >>> >> more
>> >>> >> > and
>> >>> >> > >>>>>> more different types of features to a minor release, that
>> >>> branch
>> >>> >> > will
>> >>> >> > >>>>>> become more destabilized and it may be tough to say that
>> >>> 3.1.2 is
>> >>> >> > stable
>> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
>> >>> >> features
>> >>> >> > in
>> >>> >> > >>>>>> subsequent minor release.
>> >>> >> > >>>>>> For few companies, it makes sense to push out these new
>> >>> features
>> >>> >> > >>>>>> faster to make a reach to the users. Adding to the point
>> to
>> >>> the
>> >>> >> > backporting
>> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
>> >>> with
>> >>> >> > some git
>> >>> >> > >>>>>> scripts. If we can make such scripts available to
>> committers,
>> >>> >> > backport will
>> >>> >> > >>>>>> be seem-less across branches and we can achieve the faster
>> >>> >> release
>> >>> >> > cadence
>> >>> >> > >>>>>> also.
>> >>> >> > >>>>>>
>> >>> >> > >>>>>> Thoughts?
>> >>> >> > >>>>>>
>> >>> >> > >>>>>> - Sunil
>> >>> >> > >>>>>>
>> >>> >> > >>>>>>
>> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>> >>> subru@apache.org<ma...@apache.org>
>> >>> >> >
>> >>> >> > >>>>>> wrote:
>> >>> >> > >>>>>>
>> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release
>> effort. I
>> >>> am
>> >>> >> > >>>>>>> generally
>> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x)
>> as
>> >>> >> feel we
>> >>> >> > >>>>>>> already
>> >>> >> > >>>>>>> have too many branches to be maintained. I already see
>> many
>> >>> >> commits
>> >>> >> > >>>>>>> are in
>> >>> >> > >>>>>>> different branches with no apparent rationale, for e.g:
>> 3.1
>> >>> has
>> >>> >> > >>>>>>> commits
>> >>> >> > >>>>>>> which are absent in 3.0 etc.
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
>> >>> >> > production
>> >>> >> > >>>>>>> setting so the cost of adding features should be minimal.
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>> Thoughts?
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>> -Subru
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
>> >>> sunilg@apache.org<ma...@apache.org>>
>> >>> >> > wrote:
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > Yes, important changes and features are much needed,
>> >>> hence we
>> >>> >> > will
>> >>> >> > >>>>>>> be
>> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
>> >>> considering
>> >>> >> few
>> >>> >> > >>>>>>> more
>> >>> >> > >>>>>>> > offline requests from other folks, I think extending
>> the
>> >>> >> > timeframe
>> >>> >> > >>>>>>> by
>> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
>> >>> buffer) and
>> >>> >> > >>>>>>> this should
>> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki
>> as
>> >>> >> well)
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by August
>> >>> 21,
>> >>> >> 2018.
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>> >>> improvements
>> >>> >> and
>> >>> >> > >>>>>>> non
>> >>> >> > >>>>>>> > blocker/critical
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > - Release date: September 15, 2018
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already
>> answered
>> >>> your
>> >>> >> > >>>>>>> questions.
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > Thanks
>> >>> >> > >>>>>>> > Sunil
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>> >>> >> wheeleast@gmail.com<ma...@gmail.com>
>> >>> >> > >
>> >>> >> > >>>>>>> wrote:
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2
>> release,
>> >>> +1
>> >>> >> for
>> >>> >> > >>>>>>> that.
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > To concerns from Steve,
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
>> important
>> >>> >> > changes
>> >>> >> > >>>>>>> /
>> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
>> >>> >> proposed
>> >>> >> > >>>>>>> release
>> >>> >> > >>>>>>> > date
>> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of
>> last
>> >>> >> minute
>> >>> >> > >>>>>>> and we all
>> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm
>> also
>> >>> fine
>> >>> >> if
>> >>> >> > >>>>>>> we want
>> >>> >> > >>>>>>> > get
>> >>> >> > >>>>>>> > > another several weeks time.
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
>> >>> before
>> >>> >> > >>>>>>> thanksgiving.
>> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > Eric,
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
>> >>> YARN-8135
>> >>> >> can
>> >>> >> > >>>>>>> get merged
>> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > Sunil,
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > Could u update
>> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>> >>> >> > >>>>>>> > Roadmap
>> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
>> >>> first
>> >>> >> > >>>>>>> before getting
>> >>> >> > >>>>>>> > > consensus of time.
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > Thanks,
>> >>> >> > >>>>>>> > > Wangda
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid<mailto:
>> fabbri@cloudera.com.invalid>
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > wrote:
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>> >>> >> > >>>>>>> stevel@hortonworks.com<ma...@hortonworks.com>>
>> >>> >> > >>>>>>> > >> wrote:
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
>> sunilg@apache.org<ma...@apache.org>
>> >>> >> > <mailto:
>> >>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>>
>> wrote:
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > I would also would like to take this opportunity
>> to
>> >>> come
>> >>> >> up
>> >>> >> > >>>>>>> with a
>> >>> >> > >>>>>>> > >> detailed
>> >>> >> > >>>>>>> > >> > plan.
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
>> >>> merged by
>> >>> >> > >>>>>>> August 10,
>> >>> >> > >>>>>>> > >> 2018.
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > <snip>
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
>> targeted
>> >>> to
>> >>> >> 3.2
>> >>> >> > >>>>>>> per this
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> https://issues.apache.org/jira/browse/HADOOP-15226
>> >>> >> > >>>>>>> > >> >
>> https://issues.apache.org/jira/browse/HADOOP-15220
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
>> >>> Datalake
>> >>> >> > Gen
>> >>> >> > >>>>>>> 2
>> >>> >> > >>>>>>> > >> >
>> https://issues.apache.org/jira/browse/HADOOP-15407
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
>> >>> doing
>> >>> >> work
>> >>> >> > >>>>>>> on it, but
>> >>> >> > >>>>>>> > >> if
>> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect
>> some
>> >>> >> timely
>> >>> >> > >>>>>>> reviews on
>> >>> >> > >>>>>>> > >> >
>> https://issues.apache.org/jira/browse/HADOOP-15546
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will
>> review
>> >>> the
>> >>> >> S3 &
>> >>> >> > >>>>>>> S3Guard
>> >>> >> > >>>>>>> > >> JIRAs
>> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
>> >>> some
>> >>> >> > >>>>>>> pressing
>> >>> >> > >>>>>>> > "guave,
>> >>> >> > >>>>>>> > >> > java 9 prep"
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> I can help with this part if you like.
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
>> release
>> >>> >> > manager
>> >>> >> > >>>>>>> of 3.2.0
>> >>> >> > >>>>>>> > >> > release.
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > well volunteered!
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable
>> should
>> >>> we
>> >>> >> have
>> >>> >> > >>>>>>> for the
>> >>> >> > >>>>>>> > >> 3.2. &
>> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
>> >>> having
>> >>> >> > the
>> >>> >> > >>>>>>> outline
>> >>> >> > >>>>>>> > >> time
>> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
>> >>> less
>> >>> >> > >>>>>>> concern about
>> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > -Steve
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >> -AF
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>
>> >>> >> >
>> >>> >>
>> >>> >
>> >>>
>> >>
>> >>
>>
>>
>> --
>> A very happy Clouderan
>>
>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Thanks Konstantin for pointing out.
As 3.2 is pretty much on RC level, its better we try to find a good
solution to this issue.
I ll follow up on this in the jira.
- Sunil
On Thu, Oct 25, 2018 at 11:35 AM Konstantin Shvachko <sh...@gmail.com>
wrote:
> I've tried to attract attention to an incompatibility issue through the
> jira, but it didn't work. So pitching in in this thread.
> https://issues.apache.org/jira/browse/HDFS-12026
> It introduced binary incompatibility, which will prevent people from
> upgrading from 3.1 to 3.2.
> I think it can get messy if we release anything with this feature.
>
> Thanks,
> --Konstantin
>
> On Mon, Oct 22, 2018 at 5:01 AM Steve Loughran <st...@hortonworks.com>
> wrote:
>
>> its in.
>>
>> good catch!
>>
>> On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <weichiu@cloudera.com<mailto:
>> weichiu@cloudera.com>> wrote:
>>
>> Thanks Sunil G for driving the release,
>> I filed HADOOP-15866<https://issues.apache.org/jira/browse/HADOOP-15866>
>> for a compat fix. If any one has cycle please review it, as I think it is
>> needed for 3.2.0.
>>
>> On Thu, Oct 18, 2018 at 4:43 AM Sunil G <sunilg@apache.org<mailto:
>> sunilg@apache.org>> wrote:
>> Hi Folks,
>>
>> As we previously communicated for 3.2.0 release, we have delayed due to
>> few
>> blockers in our gate.
>>
>> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
>> all bug fixes.
>>
>> - Sunil
>>
>>
>> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <sunilg@apache.org<mailto:
>> sunilg@apache.org>> wrote:
>>
>> > Hi Folks,
>> >
>> > We are now close to RC as other blocker issues are now merged to trunk
>> and
>> > branch-3.2. Last 2 critical issues are closer to merge and will be
>> > committed in few hours.
>> > With this, I will be creating 3.2.0 branch today and will go ahead with
>> RC
>> > related process.
>> >
>> > - Sunil
>> >
>> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jonbender@stripe.com
>> <ma...@stripe.com>>
>> > wrote:
>> >
>> >> Hello, were there any updates around the 3.2.0 RC timing? All I see in
>> >> the current blockers are related to the new Submarine subproject,
>> wasn't
>> >> sure if that is what is holding things up.
>> >>
>> >> Cheers,
>> >> Jon
>> >>
>> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <sunilg@apache.org<mailto:
>> sunilg@apache.org>> wrote:
>> >>
>> >>> Thanks Robert and Haibo for quickly correcting same.
>> >>> Sigh, I somehow missed one file while committing the change. Sorry for
>> >>> the
>> >>> trouble.
>> >>>
>> >>> - Sunil
>> >>>
>> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rkanter@cloudera.com
>> <ma...@cloudera.com>>
>> >>> wrote:
>> >>>
>> >>> > Looks like there's two that weren't updated:
>> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
>> >>> > --include=pom.xml
>> >>> > ./hadoop-project/pom.xml:
>> >>> >
>> <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>> >>> >
>> >>> > I've just pushed in an addendum commit to fix those.
>> >>> > In the future, please make sure to do a sanity compile when updating
>> >>> poms.
>> >>> >
>> >>> > thanks
>> >>> > - Robert
>> >>> >
>> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>> >>> <fa...@cloudera.com.invalid>>
>> >>> > wrote:
>> >>> >
>> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
>> >>> >> top-level pom.xml?
>> >>> >>
>> >>> >>
>> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <sunilg@apache.org<mailto:
>> sunilg@apache.org>> wrote:
>> >>> >>
>> >>> >> > Hi All
>> >>> >> >
>> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
>> trunk
>> >>> to
>> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
>> >>> necessary
>> >>> >> > patches are pulled into branch-3.2.
>> >>> >> >
>> >>> >> > Thank You
>> >>> >> > - Sunil
>> >>> >> >
>> >>> >> >
>> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <sunilg@apache.org
>> <ma...@apache.org>> wrote:
>> >>> >> >
>> >>> >> > > Hi All
>> >>> >> > >
>> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged
>> to
>> >>> >> trunk.
>> >>> >> > > Thanks for the support.
>> >>> >> > >
>> >>> >> > > *Plan for RC*
>> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>> >>> >> > > RC0 for 3.2: *28th Friday*
>> >>> >> > >
>> >>> >> > > Thank You
>> >>> >> > > Sunil
>> >>> >> > >
>> >>> >> > >
>> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <sunilg@apache.org
>> <ma...@apache.org>>
>> >>> wrote:
>> >>> >> > >
>> >>> >> > >> Hi All
>> >>> >> > >>
>> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of
>> you
>> >>> for
>> >>> >> > >> helping in this. I am following up on these tickets, once its
>> >>> closed
>> >>> >> we
>> >>> >> > >> will cut the 3.2 branch.
>> >>> >> > >>
>> >>> >> > >> Thanks
>> >>> >> > >> Sunil Govindan
>> >>> >> > >>
>> >>> >> > >>
>> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <sunilg@apache.org
>> <ma...@apache.org>>
>> >>> wrote:
>> >>> >> > >>
>> >>> >> > >>> Hi All,
>> >>> >> > >>>
>> >>> >> > >>> Inline with the original 3.2 communication proposal dated
>> 17th
>> >>> July
>> >>> >> > >>> 2018, I would like to provide more updates.
>> >>> >> > >>>
>> >>> >> > >>> We are approaching previously proposed code freeze date
>> >>> (September
>> >>> >> 14,
>> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
>> point
>> >>> >> > existing
>> >>> >> > >>> trunk to 3.3 if there are no issues.
>> >>> >> > >>>
>> >>> >> > >>> *Current Release Plan:*
>> >>> >> > >>> Feature freeze date : all features to merge by September 7,
>> >>> 2018.
>> >>> >> > >>> Code freeze date : blockers/critical only, no improvements
>> and
>> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>> >>> >> > >>> Release date: September 28, 2018
>> >>> >> > >>>
>> >>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0,
>> we
>> >>> >> need to
>> >>> >> > >>> backport to 3.2 post branch cut.
>> >>> >> > >>>
>> >>> >> > >>> Here's an updated 3.2.0 feature status:
>> >>> >> > >>>
>> >>> >> > >>> 1. Merged & Completed features:
>> >>> >> > >>>
>> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> DeepLearning
>> >>> >> > >>> workloads Initial cut.
>> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> Capacity
>> >>> >> > >>> Scheduler.
>> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> >>> Service
>> >>> >> > API
>> >>> >> > >>> and CLI.
>> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
>> Improvement
>> >>> >> works.
>> >>> >> > >>>
>> >>> >> > >>> 2. Features close to finish:
>> >>> >> > >>>
>> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
>> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>> >>> >> > >>>
>> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>> apps
>> >>> from
>> >>> >> > >>> ATSv2. Patch in progress.
>> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >>> >> challenging to
>> >>> >> > >>> be done before Aug 2018.
>> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN
>> applications.
>> >>> >> > >>> Challenging as more discussions are on-going.
>> >>> >> > >>>
>> >>> >> > >>> *Summary of 3.2.0 issues status:*
>> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following
>> up
>> >>> with
>> >>> >> > >>> owners to get status on each of them to get in by Code Freeze
>> >>> date.
>> >>> >> > >>>
>> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
>> in
>> >>> >> > (Blocker,
>> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s"
>> =
>> >>> 3.2.0
>> >>> >> > ORDER
>> >>> >> > >>> BY priority DESC
>> >>> >> > >>>
>> >>> >> > >>> Thanks,
>> >>> >> > >>> Sunil
>> >>> >> > >>>
>> >>> >> > >>>
>> >>> >> > >>>
>> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <sunilg@apache.org
>> <ma...@apache.org>>
>> >>> wrote:
>> >>> >> > >>>
>> >>> >> > >>>> Hi All,
>> >>> >> > >>>>
>> >>> >> > >>>> Inline with earlier communication dated 17th July 2018, I
>> would
>> >>> >> like
>> >>> >> > to
>> >>> >> > >>>> provide some updates.
>> >>> >> > >>>>
>> >>> >> > >>>> We are approaching previously proposed code freeze date (Aug
>> >>> 31).
>> >>> >> > >>>>
>> >>> >> > >>>> One of the critical feature Node Attributes feature merge
>> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs
>> need a
>> >>> bit
>> >>> >> > more
>> >>> >> > >>>> time. With regard to this, suggesting to push the
>> feature/code
>> >>> >> freeze
>> >>> >> > for 2
>> >>> >> > >>>> more weeks to accommodate these jiras too.
>> >>> >> > >>>>
>> >>> >> > >>>> Proposing Updated changes in plan inline with this:
>> >>> >> > >>>> Feature freeze date : all features to merge by September 7,
>> >>> 2018.
>> >>> >> > >>>> Code freeze date : blockers/critical only, no improvements
>> and
>> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>> >>> >> > >>>> Release date: September 28, 2018
>> >>> >> > >>>>
>> >>> >> > >>>> If any features in branch which are targeted to 3.2.0,
>> please
>> >>> >> reply to
>> >>> >> > >>>> this email thread.
>> >>> >> > >>>>
>> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>> >>> >> > >>>>
>> >>> >> > >>>> 1. Merged & Completed features:
>> >>> >> > >>>>
>> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> DeepLearning
>> >>> >> > >>>> workloads Initial cut.
>> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> Capacity
>> >>> >> > >>>> Scheduler.
>> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> >>> >> Service
>> >>> >> > >>>> API and CLI.
>> >>> >> > >>>>
>> >>> >> > >>>> 2. Features close to finish:
>> >>> >> > >>>>
>> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> >>> >> Merge/Vote
>> >>> >> > >>>> Ongoing.
>> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>> apps
>> >>> from
>> >>> >> > >>>> ATSv2. Patch in progress.
>> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> >>> Improvement
>> >>> >> > works.
>> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>> Azure
>> >>> >> > >>>> Storage. In progress.
>> >>> >> > >>>>
>> >>> >> > >>>> 3. Tentative features:
>> >>> >> > >>>>
>> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >>> >> challenging
>> >>> >> > to
>> >>> >> > >>>> be done before Aug 2018.
>> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
>> applications.
>> >>> >> > >>>> Challenging as more discussions are on-going.
>> >>> >> > >>>>
>> >>> >> > >>>> *Summary of 3.2.0 issues status:*
>> >>> >> > >>>>
>> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following
>> up
>> >>> with
>> >>> >> > >>>> owners to get status on each of them to get in by Code
>> Freeze
>> >>> date.
>> >>> >> > >>>>
>> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
>> in
>> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> >>> >> > Version/s" =
>> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
>> >>> >> > >>>>
>> >>> >> > >>>> Thanks,
>> >>> >> > >>>> Sunil
>> >>> >> > >>>>
>> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <sunilg@apache.org
>> <ma...@apache.org>>
>> >>> >> wrote:
>> >>> >> > >>>>
>> >>> >> > >>>>> Hi All,
>> >>> >> > >>>>>
>> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
>> >>> dated
>> >>> >> > 17th
>> >>> >> > >>>>> July 2018, I would like to provide some updates.
>> >>> >> > >>>>>
>> >>> >> > >>>>> We are approaching previously proposed feature freeze date
>> >>> (Aug
>> >>> >> 21,
>> >>> >> > >>>>> about 7 days from today).
>> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
>> please
>> >>> >> reply
>> >>> >> > to
>> >>> >> > >>>>> this email thread.
>> >>> >> > >>>>> Steve has mentioned about the s3 features which will come
>> >>> close to
>> >>> >> > >>>>> Code Freeze Date (Aug 31st).
>> >>> >> > >>>>>
>> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>> >>> >> > >>>>>
>> >>> >> > >>>>> 1. Merged & Completed features:
>> >>> >> > >>>>>
>> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> >>> DeepLearning
>> >>> >> > >>>>> workloads Initial cut.
>> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >> > >>>>>
>> >>> >> > >>>>> 2. Features close to finish:
>> >>> >> > >>>>>
>> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> >>> Major
>> >>> >> > >>>>> patches are all in, only one last
>> >>> >> > >>>>> patch is in review state.
>> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> Capacity
>> >>> >> > >>>>> Scheduler. Close to commit.
>> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>> YARN
>> >>> >> Service
>> >>> >> > >>>>> API and CLI. 2 patches are pending
>> >>> >> > >>>>> which will be closed by Feature freeze date.
>> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>> apps
>> >>> >> from
>> >>> >> > >>>>> ATSv2. Patch in progress.
>> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> >>> Improvement
>> >>> >> > >>>>> works.
>> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>> >>> Azure
>> >>> >> > >>>>> Storage. In progress.
>> >>> >> > >>>>>
>> >>> >> > >>>>> 3. Tentative features:
>> >>> >> > >>>>>
>> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >>> >> challenging
>> >>> >> > >>>>> to be done before Aug 2018.
>> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
>> applications.
>> >>> >> > >>>>> Challenging as more discussions are on-going.
>> >>> >> > >>>>>
>> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
>> >>> >> > >>>>>
>> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
>> >>> with
>> >>> >> > owners
>> >>> >> > >>>>> to get status on each of them to get in by Code Freeze
>> date.
>> >>> >> > >>>>>
>> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>> priority in
>> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> >>> >> > Version/s" =
>> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>> >>> >> > >>>>>
>> >>> >> > >>>>> Thanks,
>> >>> >> > >>>>> Sunil
>> >>> >> > >>>>>
>> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <sunilg@apache.org
>> <ma...@apache.org>>
>> >>> >> wrote:
>> >>> >> > >>>>>
>> >>> >> > >>>>>> Thanks Subru for the thoughts.
>> >>> >> > >>>>>> One of the main reason for a major release is to push out
>> >>> >> critical
>> >>> >> > >>>>>> features with a faster cadence to the users. If we are
>> >>> pulling
>> >>> >> more
>> >>> >> > and
>> >>> >> > >>>>>> more different types of features to a minor release, that
>> >>> branch
>> >>> >> > will
>> >>> >> > >>>>>> become more destabilized and it may be tough to say that
>> >>> 3.1.2 is
>> >>> >> > stable
>> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
>> >>> >> features
>> >>> >> > in
>> >>> >> > >>>>>> subsequent minor release.
>> >>> >> > >>>>>> For few companies, it makes sense to push out these new
>> >>> features
>> >>> >> > >>>>>> faster to make a reach to the users. Adding to the point
>> to
>> >>> the
>> >>> >> > backporting
>> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
>> >>> with
>> >>> >> > some git
>> >>> >> > >>>>>> scripts. If we can make such scripts available to
>> committers,
>> >>> >> > backport will
>> >>> >> > >>>>>> be seem-less across branches and we can achieve the faster
>> >>> >> release
>> >>> >> > cadence
>> >>> >> > >>>>>> also.
>> >>> >> > >>>>>>
>> >>> >> > >>>>>> Thoughts?
>> >>> >> > >>>>>>
>> >>> >> > >>>>>> - Sunil
>> >>> >> > >>>>>>
>> >>> >> > >>>>>>
>> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>> >>> subru@apache.org<ma...@apache.org>
>> >>> >> >
>> >>> >> > >>>>>> wrote:
>> >>> >> > >>>>>>
>> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release
>> effort. I
>> >>> am
>> >>> >> > >>>>>>> generally
>> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x)
>> as
>> >>> >> feel we
>> >>> >> > >>>>>>> already
>> >>> >> > >>>>>>> have too many branches to be maintained. I already see
>> many
>> >>> >> commits
>> >>> >> > >>>>>>> are in
>> >>> >> > >>>>>>> different branches with no apparent rationale, for e.g:
>> 3.1
>> >>> has
>> >>> >> > >>>>>>> commits
>> >>> >> > >>>>>>> which are absent in 3.0 etc.
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
>> >>> >> > production
>> >>> >> > >>>>>>> setting so the cost of adding features should be minimal.
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>> Thoughts?
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>> -Subru
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
>> >>> sunilg@apache.org<ma...@apache.org>>
>> >>> >> > wrote:
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > Yes, important changes and features are much needed,
>> >>> hence we
>> >>> >> > will
>> >>> >> > >>>>>>> be
>> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
>> >>> considering
>> >>> >> few
>> >>> >> > >>>>>>> more
>> >>> >> > >>>>>>> > offline requests from other folks, I think extending
>> the
>> >>> >> > timeframe
>> >>> >> > >>>>>>> by
>> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
>> >>> buffer) and
>> >>> >> > >>>>>>> this should
>> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki
>> as
>> >>> >> well)
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by August
>> >>> 21,
>> >>> >> 2018.
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>> >>> improvements
>> >>> >> and
>> >>> >> > >>>>>>> non
>> >>> >> > >>>>>>> > blocker/critical
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > - Release date: September 15, 2018
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already
>> answered
>> >>> your
>> >>> >> > >>>>>>> questions.
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > Thanks
>> >>> >> > >>>>>>> > Sunil
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>> >>> >> wheeleast@gmail.com<ma...@gmail.com>
>> >>> >> > >
>> >>> >> > >>>>>>> wrote:
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2
>> release,
>> >>> +1
>> >>> >> for
>> >>> >> > >>>>>>> that.
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > To concerns from Steve,
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
>> important
>> >>> >> > changes
>> >>> >> > >>>>>>> /
>> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
>> >>> >> proposed
>> >>> >> > >>>>>>> release
>> >>> >> > >>>>>>> > date
>> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of
>> last
>> >>> >> minute
>> >>> >> > >>>>>>> and we all
>> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm
>> also
>> >>> fine
>> >>> >> if
>> >>> >> > >>>>>>> we want
>> >>> >> > >>>>>>> > get
>> >>> >> > >>>>>>> > > another several weeks time.
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
>> >>> before
>> >>> >> > >>>>>>> thanksgiving.
>> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > Eric,
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
>> >>> YARN-8135
>> >>> >> can
>> >>> >> > >>>>>>> get merged
>> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > Sunil,
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > Could u update
>> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>> >>> >> > >>>>>>> > Roadmap
>> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
>> >>> first
>> >>> >> > >>>>>>> before getting
>> >>> >> > >>>>>>> > > consensus of time.
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > Thanks,
>> >>> >> > >>>>>>> > > Wangda
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid<mailto:
>> fabbri@cloudera.com.invalid>
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > wrote:
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>> >>> >> > >>>>>>> stevel@hortonworks.com<ma...@hortonworks.com>>
>> >>> >> > >>>>>>> > >> wrote:
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
>> sunilg@apache.org<ma...@apache.org>
>> >>> >> > <mailto:
>> >>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>>
>> wrote:
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > I would also would like to take this opportunity
>> to
>> >>> come
>> >>> >> up
>> >>> >> > >>>>>>> with a
>> >>> >> > >>>>>>> > >> detailed
>> >>> >> > >>>>>>> > >> > plan.
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
>> >>> merged by
>> >>> >> > >>>>>>> August 10,
>> >>> >> > >>>>>>> > >> 2018.
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > <snip>
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
>> targeted
>> >>> to
>> >>> >> 3.2
>> >>> >> > >>>>>>> per this
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> https://issues.apache.org/jira/browse/HADOOP-15226
>> >>> >> > >>>>>>> > >> >
>> https://issues.apache.org/jira/browse/HADOOP-15220
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
>> >>> Datalake
>> >>> >> > Gen
>> >>> >> > >>>>>>> 2
>> >>> >> > >>>>>>> > >> >
>> https://issues.apache.org/jira/browse/HADOOP-15407
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
>> >>> doing
>> >>> >> work
>> >>> >> > >>>>>>> on it, but
>> >>> >> > >>>>>>> > >> if
>> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect
>> some
>> >>> >> timely
>> >>> >> > >>>>>>> reviews on
>> >>> >> > >>>>>>> > >> >
>> https://issues.apache.org/jira/browse/HADOOP-15546
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will
>> review
>> >>> the
>> >>> >> S3 &
>> >>> >> > >>>>>>> S3Guard
>> >>> >> > >>>>>>> > >> JIRAs
>> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
>> >>> some
>> >>> >> > >>>>>>> pressing
>> >>> >> > >>>>>>> > "guave,
>> >>> >> > >>>>>>> > >> > java 9 prep"
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> I can help with this part if you like.
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
>> release
>> >>> >> > manager
>> >>> >> > >>>>>>> of 3.2.0
>> >>> >> > >>>>>>> > >> > release.
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > well volunteered!
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable
>> should
>> >>> we
>> >>> >> have
>> >>> >> > >>>>>>> for the
>> >>> >> > >>>>>>> > >> 3.2. &
>> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
>> >>> having
>> >>> >> > the
>> >>> >> > >>>>>>> outline
>> >>> >> > >>>>>>> > >> time
>> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
>> >>> less
>> >>> >> > >>>>>>> concern about
>> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > -Steve
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >> -AF
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>
>> >>> >> >
>> >>> >>
>> >>> >
>> >>>
>> >>
>> >>
>>
>>
>> --
>> A very happy Clouderan
>>
>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Thanks Konstantin for pointing out.
As 3.2 is pretty much on RC level, its better we try to find a good
solution to this issue.
I ll follow up on this in the jira.
- Sunil
On Thu, Oct 25, 2018 at 11:35 AM Konstantin Shvachko <sh...@gmail.com>
wrote:
> I've tried to attract attention to an incompatibility issue through the
> jira, but it didn't work. So pitching in in this thread.
> https://issues.apache.org/jira/browse/HDFS-12026
> It introduced binary incompatibility, which will prevent people from
> upgrading from 3.1 to 3.2.
> I think it can get messy if we release anything with this feature.
>
> Thanks,
> --Konstantin
>
> On Mon, Oct 22, 2018 at 5:01 AM Steve Loughran <st...@hortonworks.com>
> wrote:
>
>> its in.
>>
>> good catch!
>>
>> On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <weichiu@cloudera.com<mailto:
>> weichiu@cloudera.com>> wrote:
>>
>> Thanks Sunil G for driving the release,
>> I filed HADOOP-15866<https://issues.apache.org/jira/browse/HADOOP-15866>
>> for a compat fix. If any one has cycle please review it, as I think it is
>> needed for 3.2.0.
>>
>> On Thu, Oct 18, 2018 at 4:43 AM Sunil G <sunilg@apache.org<mailto:
>> sunilg@apache.org>> wrote:
>> Hi Folks,
>>
>> As we previously communicated for 3.2.0 release, we have delayed due to
>> few
>> blockers in our gate.
>>
>> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
>> all bug fixes.
>>
>> - Sunil
>>
>>
>> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <sunilg@apache.org<mailto:
>> sunilg@apache.org>> wrote:
>>
>> > Hi Folks,
>> >
>> > We are now close to RC as other blocker issues are now merged to trunk
>> and
>> > branch-3.2. Last 2 critical issues are closer to merge and will be
>> > committed in few hours.
>> > With this, I will be creating 3.2.0 branch today and will go ahead with
>> RC
>> > related process.
>> >
>> > - Sunil
>> >
>> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jonbender@stripe.com
>> <ma...@stripe.com>>
>> > wrote:
>> >
>> >> Hello, were there any updates around the 3.2.0 RC timing? All I see in
>> >> the current blockers are related to the new Submarine subproject,
>> wasn't
>> >> sure if that is what is holding things up.
>> >>
>> >> Cheers,
>> >> Jon
>> >>
>> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <sunilg@apache.org<mailto:
>> sunilg@apache.org>> wrote:
>> >>
>> >>> Thanks Robert and Haibo for quickly correcting same.
>> >>> Sigh, I somehow missed one file while committing the change. Sorry for
>> >>> the
>> >>> trouble.
>> >>>
>> >>> - Sunil
>> >>>
>> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rkanter@cloudera.com
>> <ma...@cloudera.com>>
>> >>> wrote:
>> >>>
>> >>> > Looks like there's two that weren't updated:
>> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
>> >>> > --include=pom.xml
>> >>> > ./hadoop-project/pom.xml:
>> >>> >
>> <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>> >>> >
>> >>> > I've just pushed in an addendum commit to fix those.
>> >>> > In the future, please make sure to do a sanity compile when updating
>> >>> poms.
>> >>> >
>> >>> > thanks
>> >>> > - Robert
>> >>> >
>> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>> >>> <fa...@cloudera.com.invalid>>
>> >>> > wrote:
>> >>> >
>> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
>> >>> >> top-level pom.xml?
>> >>> >>
>> >>> >>
>> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <sunilg@apache.org<mailto:
>> sunilg@apache.org>> wrote:
>> >>> >>
>> >>> >> > Hi All
>> >>> >> >
>> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
>> trunk
>> >>> to
>> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
>> >>> necessary
>> >>> >> > patches are pulled into branch-3.2.
>> >>> >> >
>> >>> >> > Thank You
>> >>> >> > - Sunil
>> >>> >> >
>> >>> >> >
>> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <sunilg@apache.org
>> <ma...@apache.org>> wrote:
>> >>> >> >
>> >>> >> > > Hi All
>> >>> >> > >
>> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged
>> to
>> >>> >> trunk.
>> >>> >> > > Thanks for the support.
>> >>> >> > >
>> >>> >> > > *Plan for RC*
>> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>> >>> >> > > RC0 for 3.2: *28th Friday*
>> >>> >> > >
>> >>> >> > > Thank You
>> >>> >> > > Sunil
>> >>> >> > >
>> >>> >> > >
>> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <sunilg@apache.org
>> <ma...@apache.org>>
>> >>> wrote:
>> >>> >> > >
>> >>> >> > >> Hi All
>> >>> >> > >>
>> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of
>> you
>> >>> for
>> >>> >> > >> helping in this. I am following up on these tickets, once its
>> >>> closed
>> >>> >> we
>> >>> >> > >> will cut the 3.2 branch.
>> >>> >> > >>
>> >>> >> > >> Thanks
>> >>> >> > >> Sunil Govindan
>> >>> >> > >>
>> >>> >> > >>
>> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <sunilg@apache.org
>> <ma...@apache.org>>
>> >>> wrote:
>> >>> >> > >>
>> >>> >> > >>> Hi All,
>> >>> >> > >>>
>> >>> >> > >>> Inline with the original 3.2 communication proposal dated
>> 17th
>> >>> July
>> >>> >> > >>> 2018, I would like to provide more updates.
>> >>> >> > >>>
>> >>> >> > >>> We are approaching previously proposed code freeze date
>> >>> (September
>> >>> >> 14,
>> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
>> point
>> >>> >> > existing
>> >>> >> > >>> trunk to 3.3 if there are no issues.
>> >>> >> > >>>
>> >>> >> > >>> *Current Release Plan:*
>> >>> >> > >>> Feature freeze date : all features to merge by September 7,
>> >>> 2018.
>> >>> >> > >>> Code freeze date : blockers/critical only, no improvements
>> and
>> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>> >>> >> > >>> Release date: September 28, 2018
>> >>> >> > >>>
>> >>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0,
>> we
>> >>> >> need to
>> >>> >> > >>> backport to 3.2 post branch cut.
>> >>> >> > >>>
>> >>> >> > >>> Here's an updated 3.2.0 feature status:
>> >>> >> > >>>
>> >>> >> > >>> 1. Merged & Completed features:
>> >>> >> > >>>
>> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> DeepLearning
>> >>> >> > >>> workloads Initial cut.
>> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> Capacity
>> >>> >> > >>> Scheduler.
>> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> >>> Service
>> >>> >> > API
>> >>> >> > >>> and CLI.
>> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
>> Improvement
>> >>> >> works.
>> >>> >> > >>>
>> >>> >> > >>> 2. Features close to finish:
>> >>> >> > >>>
>> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
>> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>> >>> >> > >>>
>> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>> apps
>> >>> from
>> >>> >> > >>> ATSv2. Patch in progress.
>> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >>> >> challenging to
>> >>> >> > >>> be done before Aug 2018.
>> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN
>> applications.
>> >>> >> > >>> Challenging as more discussions are on-going.
>> >>> >> > >>>
>> >>> >> > >>> *Summary of 3.2.0 issues status:*
>> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following
>> up
>> >>> with
>> >>> >> > >>> owners to get status on each of them to get in by Code Freeze
>> >>> date.
>> >>> >> > >>>
>> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
>> in
>> >>> >> > (Blocker,
>> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s"
>> =
>> >>> 3.2.0
>> >>> >> > ORDER
>> >>> >> > >>> BY priority DESC
>> >>> >> > >>>
>> >>> >> > >>> Thanks,
>> >>> >> > >>> Sunil
>> >>> >> > >>>
>> >>> >> > >>>
>> >>> >> > >>>
>> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <sunilg@apache.org
>> <ma...@apache.org>>
>> >>> wrote:
>> >>> >> > >>>
>> >>> >> > >>>> Hi All,
>> >>> >> > >>>>
>> >>> >> > >>>> Inline with earlier communication dated 17th July 2018, I
>> would
>> >>> >> like
>> >>> >> > to
>> >>> >> > >>>> provide some updates.
>> >>> >> > >>>>
>> >>> >> > >>>> We are approaching previously proposed code freeze date (Aug
>> >>> 31).
>> >>> >> > >>>>
>> >>> >> > >>>> One of the critical feature Node Attributes feature merge
>> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs
>> need a
>> >>> bit
>> >>> >> > more
>> >>> >> > >>>> time. With regard to this, suggesting to push the
>> feature/code
>> >>> >> freeze
>> >>> >> > for 2
>> >>> >> > >>>> more weeks to accommodate these jiras too.
>> >>> >> > >>>>
>> >>> >> > >>>> Proposing Updated changes in plan inline with this:
>> >>> >> > >>>> Feature freeze date : all features to merge by September 7,
>> >>> 2018.
>> >>> >> > >>>> Code freeze date : blockers/critical only, no improvements
>> and
>> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>> >>> >> > >>>> Release date: September 28, 2018
>> >>> >> > >>>>
>> >>> >> > >>>> If any features in branch which are targeted to 3.2.0,
>> please
>> >>> >> reply to
>> >>> >> > >>>> this email thread.
>> >>> >> > >>>>
>> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>> >>> >> > >>>>
>> >>> >> > >>>> 1. Merged & Completed features:
>> >>> >> > >>>>
>> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> DeepLearning
>> >>> >> > >>>> workloads Initial cut.
>> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> Capacity
>> >>> >> > >>>> Scheduler.
>> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> >>> >> Service
>> >>> >> > >>>> API and CLI.
>> >>> >> > >>>>
>> >>> >> > >>>> 2. Features close to finish:
>> >>> >> > >>>>
>> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> >>> >> Merge/Vote
>> >>> >> > >>>> Ongoing.
>> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>> apps
>> >>> from
>> >>> >> > >>>> ATSv2. Patch in progress.
>> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> >>> Improvement
>> >>> >> > works.
>> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>> Azure
>> >>> >> > >>>> Storage. In progress.
>> >>> >> > >>>>
>> >>> >> > >>>> 3. Tentative features:
>> >>> >> > >>>>
>> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >>> >> challenging
>> >>> >> > to
>> >>> >> > >>>> be done before Aug 2018.
>> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
>> applications.
>> >>> >> > >>>> Challenging as more discussions are on-going.
>> >>> >> > >>>>
>> >>> >> > >>>> *Summary of 3.2.0 issues status:*
>> >>> >> > >>>>
>> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following
>> up
>> >>> with
>> >>> >> > >>>> owners to get status on each of them to get in by Code
>> Freeze
>> >>> date.
>> >>> >> > >>>>
>> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
>> in
>> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> >>> >> > Version/s" =
>> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
>> >>> >> > >>>>
>> >>> >> > >>>> Thanks,
>> >>> >> > >>>> Sunil
>> >>> >> > >>>>
>> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <sunilg@apache.org
>> <ma...@apache.org>>
>> >>> >> wrote:
>> >>> >> > >>>>
>> >>> >> > >>>>> Hi All,
>> >>> >> > >>>>>
>> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
>> >>> dated
>> >>> >> > 17th
>> >>> >> > >>>>> July 2018, I would like to provide some updates.
>> >>> >> > >>>>>
>> >>> >> > >>>>> We are approaching previously proposed feature freeze date
>> >>> (Aug
>> >>> >> 21,
>> >>> >> > >>>>> about 7 days from today).
>> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
>> please
>> >>> >> reply
>> >>> >> > to
>> >>> >> > >>>>> this email thread.
>> >>> >> > >>>>> Steve has mentioned about the s3 features which will come
>> >>> close to
>> >>> >> > >>>>> Code Freeze Date (Aug 31st).
>> >>> >> > >>>>>
>> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>> >>> >> > >>>>>
>> >>> >> > >>>>> 1. Merged & Completed features:
>> >>> >> > >>>>>
>> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> >>> DeepLearning
>> >>> >> > >>>>> workloads Initial cut.
>> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >> > >>>>>
>> >>> >> > >>>>> 2. Features close to finish:
>> >>> >> > >>>>>
>> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> >>> Major
>> >>> >> > >>>>> patches are all in, only one last
>> >>> >> > >>>>> patch is in review state.
>> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> Capacity
>> >>> >> > >>>>> Scheduler. Close to commit.
>> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>> YARN
>> >>> >> Service
>> >>> >> > >>>>> API and CLI. 2 patches are pending
>> >>> >> > >>>>> which will be closed by Feature freeze date.
>> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>> apps
>> >>> >> from
>> >>> >> > >>>>> ATSv2. Patch in progress.
>> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> >>> Improvement
>> >>> >> > >>>>> works.
>> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>> >>> Azure
>> >>> >> > >>>>> Storage. In progress.
>> >>> >> > >>>>>
>> >>> >> > >>>>> 3. Tentative features:
>> >>> >> > >>>>>
>> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >>> >> challenging
>> >>> >> > >>>>> to be done before Aug 2018.
>> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
>> applications.
>> >>> >> > >>>>> Challenging as more discussions are on-going.
>> >>> >> > >>>>>
>> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
>> >>> >> > >>>>>
>> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
>> >>> with
>> >>> >> > owners
>> >>> >> > >>>>> to get status on each of them to get in by Code Freeze
>> date.
>> >>> >> > >>>>>
>> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>> priority in
>> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> >>> >> > Version/s" =
>> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>> >>> >> > >>>>>
>> >>> >> > >>>>> Thanks,
>> >>> >> > >>>>> Sunil
>> >>> >> > >>>>>
>> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <sunilg@apache.org
>> <ma...@apache.org>>
>> >>> >> wrote:
>> >>> >> > >>>>>
>> >>> >> > >>>>>> Thanks Subru for the thoughts.
>> >>> >> > >>>>>> One of the main reason for a major release is to push out
>> >>> >> critical
>> >>> >> > >>>>>> features with a faster cadence to the users. If we are
>> >>> pulling
>> >>> >> more
>> >>> >> > and
>> >>> >> > >>>>>> more different types of features to a minor release, that
>> >>> branch
>> >>> >> > will
>> >>> >> > >>>>>> become more destabilized and it may be tough to say that
>> >>> 3.1.2 is
>> >>> >> > stable
>> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
>> >>> >> features
>> >>> >> > in
>> >>> >> > >>>>>> subsequent minor release.
>> >>> >> > >>>>>> For few companies, it makes sense to push out these new
>> >>> features
>> >>> >> > >>>>>> faster to make a reach to the users. Adding to the point
>> to
>> >>> the
>> >>> >> > backporting
>> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
>> >>> with
>> >>> >> > some git
>> >>> >> > >>>>>> scripts. If we can make such scripts available to
>> committers,
>> >>> >> > backport will
>> >>> >> > >>>>>> be seem-less across branches and we can achieve the faster
>> >>> >> release
>> >>> >> > cadence
>> >>> >> > >>>>>> also.
>> >>> >> > >>>>>>
>> >>> >> > >>>>>> Thoughts?
>> >>> >> > >>>>>>
>> >>> >> > >>>>>> - Sunil
>> >>> >> > >>>>>>
>> >>> >> > >>>>>>
>> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>> >>> subru@apache.org<ma...@apache.org>
>> >>> >> >
>> >>> >> > >>>>>> wrote:
>> >>> >> > >>>>>>
>> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release
>> effort. I
>> >>> am
>> >>> >> > >>>>>>> generally
>> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x)
>> as
>> >>> >> feel we
>> >>> >> > >>>>>>> already
>> >>> >> > >>>>>>> have too many branches to be maintained. I already see
>> many
>> >>> >> commits
>> >>> >> > >>>>>>> are in
>> >>> >> > >>>>>>> different branches with no apparent rationale, for e.g:
>> 3.1
>> >>> has
>> >>> >> > >>>>>>> commits
>> >>> >> > >>>>>>> which are absent in 3.0 etc.
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
>> >>> >> > production
>> >>> >> > >>>>>>> setting so the cost of adding features should be minimal.
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>> Thoughts?
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>> -Subru
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
>> >>> sunilg@apache.org<ma...@apache.org>>
>> >>> >> > wrote:
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > Yes, important changes and features are much needed,
>> >>> hence we
>> >>> >> > will
>> >>> >> > >>>>>>> be
>> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
>> >>> considering
>> >>> >> few
>> >>> >> > >>>>>>> more
>> >>> >> > >>>>>>> > offline requests from other folks, I think extending
>> the
>> >>> >> > timeframe
>> >>> >> > >>>>>>> by
>> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
>> >>> buffer) and
>> >>> >> > >>>>>>> this should
>> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki
>> as
>> >>> >> well)
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by August
>> >>> 21,
>> >>> >> 2018.
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>> >>> improvements
>> >>> >> and
>> >>> >> > >>>>>>> non
>> >>> >> > >>>>>>> > blocker/critical
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > - Release date: September 15, 2018
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already
>> answered
>> >>> your
>> >>> >> > >>>>>>> questions.
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > Thanks
>> >>> >> > >>>>>>> > Sunil
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>> >>> >> wheeleast@gmail.com<ma...@gmail.com>
>> >>> >> > >
>> >>> >> > >>>>>>> wrote:
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2
>> release,
>> >>> +1
>> >>> >> for
>> >>> >> > >>>>>>> that.
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > To concerns from Steve,
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
>> important
>> >>> >> > changes
>> >>> >> > >>>>>>> /
>> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
>> >>> >> proposed
>> >>> >> > >>>>>>> release
>> >>> >> > >>>>>>> > date
>> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of
>> last
>> >>> >> minute
>> >>> >> > >>>>>>> and we all
>> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm
>> also
>> >>> fine
>> >>> >> if
>> >>> >> > >>>>>>> we want
>> >>> >> > >>>>>>> > get
>> >>> >> > >>>>>>> > > another several weeks time.
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
>> >>> before
>> >>> >> > >>>>>>> thanksgiving.
>> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > Eric,
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
>> >>> YARN-8135
>> >>> >> can
>> >>> >> > >>>>>>> get merged
>> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > Sunil,
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > Could u update
>> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>> >>> >> > >>>>>>> > Roadmap
>> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
>> >>> first
>> >>> >> > >>>>>>> before getting
>> >>> >> > >>>>>>> > > consensus of time.
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > Thanks,
>> >>> >> > >>>>>>> > > Wangda
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid<mailto:
>> fabbri@cloudera.com.invalid>
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > wrote:
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>> >>> >> > >>>>>>> stevel@hortonworks.com<ma...@hortonworks.com>>
>> >>> >> > >>>>>>> > >> wrote:
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
>> sunilg@apache.org<ma...@apache.org>
>> >>> >> > <mailto:
>> >>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>>
>> wrote:
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > I would also would like to take this opportunity
>> to
>> >>> come
>> >>> >> up
>> >>> >> > >>>>>>> with a
>> >>> >> > >>>>>>> > >> detailed
>> >>> >> > >>>>>>> > >> > plan.
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
>> >>> merged by
>> >>> >> > >>>>>>> August 10,
>> >>> >> > >>>>>>> > >> 2018.
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > <snip>
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
>> targeted
>> >>> to
>> >>> >> 3.2
>> >>> >> > >>>>>>> per this
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> https://issues.apache.org/jira/browse/HADOOP-15226
>> >>> >> > >>>>>>> > >> >
>> https://issues.apache.org/jira/browse/HADOOP-15220
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
>> >>> Datalake
>> >>> >> > Gen
>> >>> >> > >>>>>>> 2
>> >>> >> > >>>>>>> > >> >
>> https://issues.apache.org/jira/browse/HADOOP-15407
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
>> >>> doing
>> >>> >> work
>> >>> >> > >>>>>>> on it, but
>> >>> >> > >>>>>>> > >> if
>> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect
>> some
>> >>> >> timely
>> >>> >> > >>>>>>> reviews on
>> >>> >> > >>>>>>> > >> >
>> https://issues.apache.org/jira/browse/HADOOP-15546
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will
>> review
>> >>> the
>> >>> >> S3 &
>> >>> >> > >>>>>>> S3Guard
>> >>> >> > >>>>>>> > >> JIRAs
>> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
>> >>> some
>> >>> >> > >>>>>>> pressing
>> >>> >> > >>>>>>> > "guave,
>> >>> >> > >>>>>>> > >> > java 9 prep"
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> I can help with this part if you like.
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
>> release
>> >>> >> > manager
>> >>> >> > >>>>>>> of 3.2.0
>> >>> >> > >>>>>>> > >> > release.
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > well volunteered!
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable
>> should
>> >>> we
>> >>> >> have
>> >>> >> > >>>>>>> for the
>> >>> >> > >>>>>>> > >> 3.2. &
>> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
>> >>> having
>> >>> >> > the
>> >>> >> > >>>>>>> outline
>> >>> >> > >>>>>>> > >> time
>> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
>> >>> less
>> >>> >> > >>>>>>> concern about
>> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > -Steve
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >> -AF
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>
>> >>> >> >
>> >>> >>
>> >>> >
>> >>>
>> >>
>> >>
>>
>>
>> --
>> A very happy Clouderan
>>
>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Thanks Konstantin for pointing out.
As 3.2 is pretty much on RC level, its better we try to find a good
solution to this issue.
I ll follow up on this in the jira.
- Sunil
On Thu, Oct 25, 2018 at 11:35 AM Konstantin Shvachko <sh...@gmail.com>
wrote:
> I've tried to attract attention to an incompatibility issue through the
> jira, but it didn't work. So pitching in in this thread.
> https://issues.apache.org/jira/browse/HDFS-12026
> It introduced binary incompatibility, which will prevent people from
> upgrading from 3.1 to 3.2.
> I think it can get messy if we release anything with this feature.
>
> Thanks,
> --Konstantin
>
> On Mon, Oct 22, 2018 at 5:01 AM Steve Loughran <st...@hortonworks.com>
> wrote:
>
>> its in.
>>
>> good catch!
>>
>> On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <weichiu@cloudera.com<mailto:
>> weichiu@cloudera.com>> wrote:
>>
>> Thanks Sunil G for driving the release,
>> I filed HADOOP-15866<https://issues.apache.org/jira/browse/HADOOP-15866>
>> for a compat fix. If any one has cycle please review it, as I think it is
>> needed for 3.2.0.
>>
>> On Thu, Oct 18, 2018 at 4:43 AM Sunil G <sunilg@apache.org<mailto:
>> sunilg@apache.org>> wrote:
>> Hi Folks,
>>
>> As we previously communicated for 3.2.0 release, we have delayed due to
>> few
>> blockers in our gate.
>>
>> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
>> all bug fixes.
>>
>> - Sunil
>>
>>
>> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <sunilg@apache.org<mailto:
>> sunilg@apache.org>> wrote:
>>
>> > Hi Folks,
>> >
>> > We are now close to RC as other blocker issues are now merged to trunk
>> and
>> > branch-3.2. Last 2 critical issues are closer to merge and will be
>> > committed in few hours.
>> > With this, I will be creating 3.2.0 branch today and will go ahead with
>> RC
>> > related process.
>> >
>> > - Sunil
>> >
>> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jonbender@stripe.com
>> <ma...@stripe.com>>
>> > wrote:
>> >
>> >> Hello, were there any updates around the 3.2.0 RC timing? All I see in
>> >> the current blockers are related to the new Submarine subproject,
>> wasn't
>> >> sure if that is what is holding things up.
>> >>
>> >> Cheers,
>> >> Jon
>> >>
>> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <sunilg@apache.org<mailto:
>> sunilg@apache.org>> wrote:
>> >>
>> >>> Thanks Robert and Haibo for quickly correcting same.
>> >>> Sigh, I somehow missed one file while committing the change. Sorry for
>> >>> the
>> >>> trouble.
>> >>>
>> >>> - Sunil
>> >>>
>> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rkanter@cloudera.com
>> <ma...@cloudera.com>>
>> >>> wrote:
>> >>>
>> >>> > Looks like there's two that weren't updated:
>> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
>> >>> > --include=pom.xml
>> >>> > ./hadoop-project/pom.xml:
>> >>> >
>> <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>> >>> >
>> >>> > I've just pushed in an addendum commit to fix those.
>> >>> > In the future, please make sure to do a sanity compile when updating
>> >>> poms.
>> >>> >
>> >>> > thanks
>> >>> > - Robert
>> >>> >
>> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>> >>> <fa...@cloudera.com.invalid>>
>> >>> > wrote:
>> >>> >
>> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
>> >>> >> top-level pom.xml?
>> >>> >>
>> >>> >>
>> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <sunilg@apache.org<mailto:
>> sunilg@apache.org>> wrote:
>> >>> >>
>> >>> >> > Hi All
>> >>> >> >
>> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
>> trunk
>> >>> to
>> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
>> >>> necessary
>> >>> >> > patches are pulled into branch-3.2.
>> >>> >> >
>> >>> >> > Thank You
>> >>> >> > - Sunil
>> >>> >> >
>> >>> >> >
>> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <sunilg@apache.org
>> <ma...@apache.org>> wrote:
>> >>> >> >
>> >>> >> > > Hi All
>> >>> >> > >
>> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged
>> to
>> >>> >> trunk.
>> >>> >> > > Thanks for the support.
>> >>> >> > >
>> >>> >> > > *Plan for RC*
>> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>> >>> >> > > RC0 for 3.2: *28th Friday*
>> >>> >> > >
>> >>> >> > > Thank You
>> >>> >> > > Sunil
>> >>> >> > >
>> >>> >> > >
>> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <sunilg@apache.org
>> <ma...@apache.org>>
>> >>> wrote:
>> >>> >> > >
>> >>> >> > >> Hi All
>> >>> >> > >>
>> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of
>> you
>> >>> for
>> >>> >> > >> helping in this. I am following up on these tickets, once its
>> >>> closed
>> >>> >> we
>> >>> >> > >> will cut the 3.2 branch.
>> >>> >> > >>
>> >>> >> > >> Thanks
>> >>> >> > >> Sunil Govindan
>> >>> >> > >>
>> >>> >> > >>
>> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <sunilg@apache.org
>> <ma...@apache.org>>
>> >>> wrote:
>> >>> >> > >>
>> >>> >> > >>> Hi All,
>> >>> >> > >>>
>> >>> >> > >>> Inline with the original 3.2 communication proposal dated
>> 17th
>> >>> July
>> >>> >> > >>> 2018, I would like to provide more updates.
>> >>> >> > >>>
>> >>> >> > >>> We are approaching previously proposed code freeze date
>> >>> (September
>> >>> >> 14,
>> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
>> point
>> >>> >> > existing
>> >>> >> > >>> trunk to 3.3 if there are no issues.
>> >>> >> > >>>
>> >>> >> > >>> *Current Release Plan:*
>> >>> >> > >>> Feature freeze date : all features to merge by September 7,
>> >>> 2018.
>> >>> >> > >>> Code freeze date : blockers/critical only, no improvements
>> and
>> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>> >>> >> > >>> Release date: September 28, 2018
>> >>> >> > >>>
>> >>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0,
>> we
>> >>> >> need to
>> >>> >> > >>> backport to 3.2 post branch cut.
>> >>> >> > >>>
>> >>> >> > >>> Here's an updated 3.2.0 feature status:
>> >>> >> > >>>
>> >>> >> > >>> 1. Merged & Completed features:
>> >>> >> > >>>
>> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> DeepLearning
>> >>> >> > >>> workloads Initial cut.
>> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> Capacity
>> >>> >> > >>> Scheduler.
>> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> >>> Service
>> >>> >> > API
>> >>> >> > >>> and CLI.
>> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
>> Improvement
>> >>> >> works.
>> >>> >> > >>>
>> >>> >> > >>> 2. Features close to finish:
>> >>> >> > >>>
>> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
>> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>> >>> >> > >>>
>> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>> apps
>> >>> from
>> >>> >> > >>> ATSv2. Patch in progress.
>> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >>> >> challenging to
>> >>> >> > >>> be done before Aug 2018.
>> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN
>> applications.
>> >>> >> > >>> Challenging as more discussions are on-going.
>> >>> >> > >>>
>> >>> >> > >>> *Summary of 3.2.0 issues status:*
>> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following
>> up
>> >>> with
>> >>> >> > >>> owners to get status on each of them to get in by Code Freeze
>> >>> date.
>> >>> >> > >>>
>> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
>> in
>> >>> >> > (Blocker,
>> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s"
>> =
>> >>> 3.2.0
>> >>> >> > ORDER
>> >>> >> > >>> BY priority DESC
>> >>> >> > >>>
>> >>> >> > >>> Thanks,
>> >>> >> > >>> Sunil
>> >>> >> > >>>
>> >>> >> > >>>
>> >>> >> > >>>
>> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <sunilg@apache.org
>> <ma...@apache.org>>
>> >>> wrote:
>> >>> >> > >>>
>> >>> >> > >>>> Hi All,
>> >>> >> > >>>>
>> >>> >> > >>>> Inline with earlier communication dated 17th July 2018, I
>> would
>> >>> >> like
>> >>> >> > to
>> >>> >> > >>>> provide some updates.
>> >>> >> > >>>>
>> >>> >> > >>>> We are approaching previously proposed code freeze date (Aug
>> >>> 31).
>> >>> >> > >>>>
>> >>> >> > >>>> One of the critical feature Node Attributes feature merge
>> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs
>> need a
>> >>> bit
>> >>> >> > more
>> >>> >> > >>>> time. With regard to this, suggesting to push the
>> feature/code
>> >>> >> freeze
>> >>> >> > for 2
>> >>> >> > >>>> more weeks to accommodate these jiras too.
>> >>> >> > >>>>
>> >>> >> > >>>> Proposing Updated changes in plan inline with this:
>> >>> >> > >>>> Feature freeze date : all features to merge by September 7,
>> >>> 2018.
>> >>> >> > >>>> Code freeze date : blockers/critical only, no improvements
>> and
>> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>> >>> >> > >>>> Release date: September 28, 2018
>> >>> >> > >>>>
>> >>> >> > >>>> If any features in branch which are targeted to 3.2.0,
>> please
>> >>> >> reply to
>> >>> >> > >>>> this email thread.
>> >>> >> > >>>>
>> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>> >>> >> > >>>>
>> >>> >> > >>>> 1. Merged & Completed features:
>> >>> >> > >>>>
>> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> DeepLearning
>> >>> >> > >>>> workloads Initial cut.
>> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> Capacity
>> >>> >> > >>>> Scheduler.
>> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> >>> >> Service
>> >>> >> > >>>> API and CLI.
>> >>> >> > >>>>
>> >>> >> > >>>> 2. Features close to finish:
>> >>> >> > >>>>
>> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> >>> >> Merge/Vote
>> >>> >> > >>>> Ongoing.
>> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>> apps
>> >>> from
>> >>> >> > >>>> ATSv2. Patch in progress.
>> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> >>> Improvement
>> >>> >> > works.
>> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>> Azure
>> >>> >> > >>>> Storage. In progress.
>> >>> >> > >>>>
>> >>> >> > >>>> 3. Tentative features:
>> >>> >> > >>>>
>> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >>> >> challenging
>> >>> >> > to
>> >>> >> > >>>> be done before Aug 2018.
>> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
>> applications.
>> >>> >> > >>>> Challenging as more discussions are on-going.
>> >>> >> > >>>>
>> >>> >> > >>>> *Summary of 3.2.0 issues status:*
>> >>> >> > >>>>
>> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following
>> up
>> >>> with
>> >>> >> > >>>> owners to get status on each of them to get in by Code
>> Freeze
>> >>> date.
>> >>> >> > >>>>
>> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
>> in
>> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> >>> >> > Version/s" =
>> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
>> >>> >> > >>>>
>> >>> >> > >>>> Thanks,
>> >>> >> > >>>> Sunil
>> >>> >> > >>>>
>> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <sunilg@apache.org
>> <ma...@apache.org>>
>> >>> >> wrote:
>> >>> >> > >>>>
>> >>> >> > >>>>> Hi All,
>> >>> >> > >>>>>
>> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
>> >>> dated
>> >>> >> > 17th
>> >>> >> > >>>>> July 2018, I would like to provide some updates.
>> >>> >> > >>>>>
>> >>> >> > >>>>> We are approaching previously proposed feature freeze date
>> >>> (Aug
>> >>> >> 21,
>> >>> >> > >>>>> about 7 days from today).
>> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
>> please
>> >>> >> reply
>> >>> >> > to
>> >>> >> > >>>>> this email thread.
>> >>> >> > >>>>> Steve has mentioned about the s3 features which will come
>> >>> close to
>> >>> >> > >>>>> Code Freeze Date (Aug 31st).
>> >>> >> > >>>>>
>> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>> >>> >> > >>>>>
>> >>> >> > >>>>> 1. Merged & Completed features:
>> >>> >> > >>>>>
>> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>> >>> DeepLearning
>> >>> >> > >>>>> workloads Initial cut.
>> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >>> >> > >>>>>
>> >>> >> > >>>>> 2. Features close to finish:
>> >>> >> > >>>>>
>> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> >>> Major
>> >>> >> > >>>>> patches are all in, only one last
>> >>> >> > >>>>> patch is in review state.
>> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
>> Capacity
>> >>> >> > >>>>> Scheduler. Close to commit.
>> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via
>> YARN
>> >>> >> Service
>> >>> >> > >>>>> API and CLI. 2 patches are pending
>> >>> >> > >>>>> which will be closed by Feature freeze date.
>> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
>> apps
>> >>> >> from
>> >>> >> > >>>>> ATSv2. Patch in progress.
>> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> >>> Improvement
>> >>> >> > >>>>> works.
>> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>> >>> Azure
>> >>> >> > >>>>> Storage. In progress.
>> >>> >> > >>>>>
>> >>> >> > >>>>> 3. Tentative features:
>> >>> >> > >>>>>
>> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >>> >> challenging
>> >>> >> > >>>>> to be done before Aug 2018.
>> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
>> applications.
>> >>> >> > >>>>> Challenging as more discussions are on-going.
>> >>> >> > >>>>>
>> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
>> >>> >> > >>>>>
>> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
>> >>> with
>> >>> >> > owners
>> >>> >> > >>>>> to get status on each of them to get in by Code Freeze
>> date.
>> >>> >> > >>>>>
>> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>> priority in
>> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> >>> >> > Version/s" =
>> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>> >>> >> > >>>>>
>> >>> >> > >>>>> Thanks,
>> >>> >> > >>>>> Sunil
>> >>> >> > >>>>>
>> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <sunilg@apache.org
>> <ma...@apache.org>>
>> >>> >> wrote:
>> >>> >> > >>>>>
>> >>> >> > >>>>>> Thanks Subru for the thoughts.
>> >>> >> > >>>>>> One of the main reason for a major release is to push out
>> >>> >> critical
>> >>> >> > >>>>>> features with a faster cadence to the users. If we are
>> >>> pulling
>> >>> >> more
>> >>> >> > and
>> >>> >> > >>>>>> more different types of features to a minor release, that
>> >>> branch
>> >>> >> > will
>> >>> >> > >>>>>> become more destabilized and it may be tough to say that
>> >>> 3.1.2 is
>> >>> >> > stable
>> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
>> >>> >> features
>> >>> >> > in
>> >>> >> > >>>>>> subsequent minor release.
>> >>> >> > >>>>>> For few companies, it makes sense to push out these new
>> >>> features
>> >>> >> > >>>>>> faster to make a reach to the users. Adding to the point
>> to
>> >>> the
>> >>> >> > backporting
>> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
>> >>> with
>> >>> >> > some git
>> >>> >> > >>>>>> scripts. If we can make such scripts available to
>> committers,
>> >>> >> > backport will
>> >>> >> > >>>>>> be seem-less across branches and we can achieve the faster
>> >>> >> release
>> >>> >> > cadence
>> >>> >> > >>>>>> also.
>> >>> >> > >>>>>>
>> >>> >> > >>>>>> Thoughts?
>> >>> >> > >>>>>>
>> >>> >> > >>>>>> - Sunil
>> >>> >> > >>>>>>
>> >>> >> > >>>>>>
>> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>> >>> subru@apache.org<ma...@apache.org>
>> >>> >> >
>> >>> >> > >>>>>> wrote:
>> >>> >> > >>>>>>
>> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release
>> effort. I
>> >>> am
>> >>> >> > >>>>>>> generally
>> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x)
>> as
>> >>> >> feel we
>> >>> >> > >>>>>>> already
>> >>> >> > >>>>>>> have too many branches to be maintained. I already see
>> many
>> >>> >> commits
>> >>> >> > >>>>>>> are in
>> >>> >> > >>>>>>> different branches with no apparent rationale, for e.g:
>> 3.1
>> >>> has
>> >>> >> > >>>>>>> commits
>> >>> >> > >>>>>>> which are absent in 3.0 etc.
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
>> >>> >> > production
>> >>> >> > >>>>>>> setting so the cost of adding features should be minimal.
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>> Thoughts?
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>> -Subru
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
>> >>> sunilg@apache.org<ma...@apache.org>>
>> >>> >> > wrote:
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > Yes, important changes and features are much needed,
>> >>> hence we
>> >>> >> > will
>> >>> >> > >>>>>>> be
>> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
>> >>> considering
>> >>> >> few
>> >>> >> > >>>>>>> more
>> >>> >> > >>>>>>> > offline requests from other folks, I think extending
>> the
>> >>> >> > timeframe
>> >>> >> > >>>>>>> by
>> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
>> >>> buffer) and
>> >>> >> > >>>>>>> this should
>> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki
>> as
>> >>> >> well)
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by August
>> >>> 21,
>> >>> >> 2018.
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>> >>> improvements
>> >>> >> and
>> >>> >> > >>>>>>> non
>> >>> >> > >>>>>>> > blocker/critical
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > - Release date: September 15, 2018
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already
>> answered
>> >>> your
>> >>> >> > >>>>>>> questions.
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > Thanks
>> >>> >> > >>>>>>> > Sunil
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>> >>> >> wheeleast@gmail.com<ma...@gmail.com>
>> >>> >> > >
>> >>> >> > >>>>>>> wrote:
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2
>> release,
>> >>> +1
>> >>> >> for
>> >>> >> > >>>>>>> that.
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > To concerns from Steve,
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
>> important
>> >>> >> > changes
>> >>> >> > >>>>>>> /
>> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
>> >>> >> proposed
>> >>> >> > >>>>>>> release
>> >>> >> > >>>>>>> > date
>> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of
>> last
>> >>> >> minute
>> >>> >> > >>>>>>> and we all
>> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm
>> also
>> >>> fine
>> >>> >> if
>> >>> >> > >>>>>>> we want
>> >>> >> > >>>>>>> > get
>> >>> >> > >>>>>>> > > another several weeks time.
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
>> >>> before
>> >>> >> > >>>>>>> thanksgiving.
>> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > Eric,
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
>> >>> YARN-8135
>> >>> >> can
>> >>> >> > >>>>>>> get merged
>> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > Sunil,
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > Could u update
>> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>> >>> >> > >>>>>>> > Roadmap
>> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
>> >>> first
>> >>> >> > >>>>>>> before getting
>> >>> >> > >>>>>>> > > consensus of time.
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > Thanks,
>> >>> >> > >>>>>>> > > Wangda
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid<mailto:
>> fabbri@cloudera.com.invalid>
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > > wrote:
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>> >>> >> > >>>>>>> stevel@hortonworks.com<ma...@hortonworks.com>>
>> >>> >> > >>>>>>> > >> wrote:
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
>> sunilg@apache.org<ma...@apache.org>
>> >>> >> > <mailto:
>> >>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>>
>> wrote:
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > I would also would like to take this opportunity
>> to
>> >>> come
>> >>> >> up
>> >>> >> > >>>>>>> with a
>> >>> >> > >>>>>>> > >> detailed
>> >>> >> > >>>>>>> > >> > plan.
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
>> >>> merged by
>> >>> >> > >>>>>>> August 10,
>> >>> >> > >>>>>>> > >> 2018.
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > <snip>
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
>> targeted
>> >>> to
>> >>> >> 3.2
>> >>> >> > >>>>>>> per this
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> https://issues.apache.org/jira/browse/HADOOP-15226
>> >>> >> > >>>>>>> > >> >
>> https://issues.apache.org/jira/browse/HADOOP-15220
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
>> >>> Datalake
>> >>> >> > Gen
>> >>> >> > >>>>>>> 2
>> >>> >> > >>>>>>> > >> >
>> https://issues.apache.org/jira/browse/HADOOP-15407
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
>> >>> doing
>> >>> >> work
>> >>> >> > >>>>>>> on it, but
>> >>> >> > >>>>>>> > >> if
>> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect
>> some
>> >>> >> timely
>> >>> >> > >>>>>>> reviews on
>> >>> >> > >>>>>>> > >> >
>> https://issues.apache.org/jira/browse/HADOOP-15546
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will
>> review
>> >>> the
>> >>> >> S3 &
>> >>> >> > >>>>>>> S3Guard
>> >>> >> > >>>>>>> > >> JIRAs
>> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
>> >>> some
>> >>> >> > >>>>>>> pressing
>> >>> >> > >>>>>>> > "guave,
>> >>> >> > >>>>>>> > >> > java 9 prep"
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> I can help with this part if you like.
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
>> release
>> >>> >> > manager
>> >>> >> > >>>>>>> of 3.2.0
>> >>> >> > >>>>>>> > >> > release.
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > well volunteered!
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable
>> should
>> >>> we
>> >>> >> have
>> >>> >> > >>>>>>> for the
>> >>> >> > >>>>>>> > >> 3.2. &
>> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
>> >>> having
>> >>> >> > the
>> >>> >> > >>>>>>> outline
>> >>> >> > >>>>>>> > >> time
>> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
>> >>> less
>> >>> >> > >>>>>>> concern about
>> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> > -Steve
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> >
>> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >> -AF
>> >>> >> > >>>>>>> > >>
>> >>> >> > >>>>>>> > >
>> >>> >> > >>>>>>> >
>> >>> >> > >>>>>>>
>> >>> >> > >>>>>>
>> >>> >> >
>> >>> >>
>> >>> >
>> >>>
>> >>
>> >>
>>
>>
>> --
>> A very happy Clouderan
>>
>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Konstantin Shvachko <sh...@gmail.com>.
I've tried to attract attention to an incompatibility issue through the
jira, but it didn't work. So pitching in in this thread.
https://issues.apache.org/jira/browse/HDFS-12026
It introduced binary incompatibility, which will prevent people from
upgrading from 3.1 to 3.2.
I think it can get messy if we release anything with this feature.
Thanks,
--Konstantin
On Mon, Oct 22, 2018 at 5:01 AM Steve Loughran <st...@hortonworks.com>
wrote:
> its in.
>
> good catch!
>
> On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <weichiu@cloudera.com<mailto:
> weichiu@cloudera.com>> wrote:
>
> Thanks Sunil G for driving the release,
> I filed HADOOP-15866<https://issues.apache.org/jira/browse/HADOOP-15866>
> for a compat fix. If any one has cycle please review it, as I think it is
> needed for 3.2.0.
>
> On Thu, Oct 18, 2018 at 4:43 AM Sunil G <sunilg@apache.org<mailto:
> sunilg@apache.org>> wrote:
> Hi Folks,
>
> As we previously communicated for 3.2.0 release, we have delayed due to few
> blockers in our gate.
>
> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
> all bug fixes.
>
> - Sunil
>
>
> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <sunilg@apache.org<mailto:
> sunilg@apache.org>> wrote:
>
> > Hi Folks,
> >
> > We are now close to RC as other blocker issues are now merged to trunk
> and
> > branch-3.2. Last 2 critical issues are closer to merge and will be
> > committed in few hours.
> > With this, I will be creating 3.2.0 branch today and will go ahead with
> RC
> > related process.
> >
> > - Sunil
> >
> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jonbender@stripe.com
> <ma...@stripe.com>>
> > wrote:
> >
> >> Hello, were there any updates around the 3.2.0 RC timing? All I see in
> >> the current blockers are related to the new Submarine subproject, wasn't
> >> sure if that is what is holding things up.
> >>
> >> Cheers,
> >> Jon
> >>
> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <sunilg@apache.org<mailto:
> sunilg@apache.org>> wrote:
> >>
> >>> Thanks Robert and Haibo for quickly correcting same.
> >>> Sigh, I somehow missed one file while committing the change. Sorry for
> >>> the
> >>> trouble.
> >>>
> >>> - Sunil
> >>>
> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rkanter@cloudera.com
> <ma...@cloudera.com>>
> >>> wrote:
> >>>
> >>> > Looks like there's two that weren't updated:
> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
> >>> > --include=pom.xml
> >>> > ./hadoop-project/pom.xml:
> >>> > <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
> >>> >
> >>> > I've just pushed in an addendum commit to fix those.
> >>> > In the future, please make sure to do a sanity compile when updating
> >>> poms.
> >>> >
> >>> > thanks
> >>> > - Robert
> >>> >
> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
> >>> <fa...@cloudera.com.invalid>>
> >>> > wrote:
> >>> >
> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
> >>> >> top-level pom.xml?
> >>> >>
> >>> >>
> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <sunilg@apache.org<mailto:
> sunilg@apache.org>> wrote:
> >>> >>
> >>> >> > Hi All
> >>> >> >
> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
> trunk
> >>> to
> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
> >>> necessary
> >>> >> > patches are pulled into branch-3.2.
> >>> >> >
> >>> >> > Thank You
> >>> >> > - Sunil
> >>> >> >
> >>> >> >
> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <sunilg@apache.org
> <ma...@apache.org>> wrote:
> >>> >> >
> >>> >> > > Hi All
> >>> >> > >
> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged
> to
> >>> >> trunk.
> >>> >> > > Thanks for the support.
> >>> >> > >
> >>> >> > > *Plan for RC*
> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
> >>> >> > > RC0 for 3.2: *28th Friday*
> >>> >> > >
> >>> >> > > Thank You
> >>> >> > > Sunil
> >>> >> > >
> >>> >> > >
> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <sunilg@apache.org
> <ma...@apache.org>>
> >>> wrote:
> >>> >> > >
> >>> >> > >> Hi All
> >>> >> > >>
> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you
> >>> for
> >>> >> > >> helping in this. I am following up on these tickets, once its
> >>> closed
> >>> >> we
> >>> >> > >> will cut the 3.2 branch.
> >>> >> > >>
> >>> >> > >> Thanks
> >>> >> > >> Sunil Govindan
> >>> >> > >>
> >>> >> > >>
> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <sunilg@apache.org
> <ma...@apache.org>>
> >>> wrote:
> >>> >> > >>
> >>> >> > >>> Hi All,
> >>> >> > >>>
> >>> >> > >>> Inline with the original 3.2 communication proposal dated 17th
> >>> July
> >>> >> > >>> 2018, I would like to provide more updates.
> >>> >> > >>>
> >>> >> > >>> We are approaching previously proposed code freeze date
> >>> (September
> >>> >> 14,
> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
> point
> >>> >> > existing
> >>> >> > >>> trunk to 3.3 if there are no issues.
> >>> >> > >>>
> >>> >> > >>> *Current Release Plan:*
> >>> >> > >>> Feature freeze date : all features to merge by September 7,
> >>> 2018.
> >>> >> > >>> Code freeze date : blockers/critical only, no improvements and
> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
> >>> >> > >>> Release date: September 28, 2018
> >>> >> > >>>
> >>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0,
> we
> >>> >> need to
> >>> >> > >>> backport to 3.2 post branch cut.
> >>> >> > >>>
> >>> >> > >>> Here's an updated 3.2.0 feature status:
> >>> >> > >>>
> >>> >> > >>> 1. Merged & Completed features:
> >>> >> > >>>
> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
> DeepLearning
> >>> >> > >>> workloads Initial cut.
> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >>> >> > >>> Scheduler.
> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> Service
> >>> >> > API
> >>> >> > >>> and CLI.
> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
> Improvement
> >>> >> works.
> >>> >> > >>>
> >>> >> > >>> 2. Features close to finish:
> >>> >> > >>>
> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
> >>> >> > >>>
> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
> >>> from
> >>> >> > >>> ATSv2. Patch in progress.
> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging to
> >>> >> > >>> be done before Aug 2018.
> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >>> >> > >>> Challenging as more discussions are on-going.
> >>> >> > >>>
> >>> >> > >>> *Summary of 3.2.0 issues status:*
> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following up
> >>> with
> >>> >> > >>> owners to get status on each of them to get in by Code Freeze
> >>> date.
> >>> >> > >>>
> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> >>> >> > (Blocker,
> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" =
> >>> 3.2.0
> >>> >> > ORDER
> >>> >> > >>> BY priority DESC
> >>> >> > >>>
> >>> >> > >>> Thanks,
> >>> >> > >>> Sunil
> >>> >> > >>>
> >>> >> > >>>
> >>> >> > >>>
> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <sunilg@apache.org
> <ma...@apache.org>>
> >>> wrote:
> >>> >> > >>>
> >>> >> > >>>> Hi All,
> >>> >> > >>>>
> >>> >> > >>>> Inline with earlier communication dated 17th July 2018, I
> would
> >>> >> like
> >>> >> > to
> >>> >> > >>>> provide some updates.
> >>> >> > >>>>
> >>> >> > >>>> We are approaching previously proposed code freeze date (Aug
> >>> 31).
> >>> >> > >>>>
> >>> >> > >>>> One of the critical feature Node Attributes feature merge
> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need
> a
> >>> bit
> >>> >> > more
> >>> >> > >>>> time. With regard to this, suggesting to push the
> feature/code
> >>> >> freeze
> >>> >> > for 2
> >>> >> > >>>> more weeks to accommodate these jiras too.
> >>> >> > >>>>
> >>> >> > >>>> Proposing Updated changes in plan inline with this:
> >>> >> > >>>> Feature freeze date : all features to merge by September 7,
> >>> 2018.
> >>> >> > >>>> Code freeze date : blockers/critical only, no improvements
> and
> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
> >>> >> > >>>> Release date: September 28, 2018
> >>> >> > >>>>
> >>> >> > >>>> If any features in branch which are targeted to 3.2.0, please
> >>> >> reply to
> >>> >> > >>>> this email thread.
> >>> >> > >>>>
> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
> >>> >> > >>>>
> >>> >> > >>>> 1. Merged & Completed features:
> >>> >> > >>>>
> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> DeepLearning
> >>> >> > >>>> workloads Initial cut.
> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> Capacity
> >>> >> > >>>> Scheduler.
> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> >> Service
> >>> >> > >>>> API and CLI.
> >>> >> > >>>>
> >>> >> > >>>> 2. Features close to finish:
> >>> >> > >>>>
> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> >> Merge/Vote
> >>> >> > >>>> Ongoing.
> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> apps
> >>> from
> >>> >> > >>>> ATSv2. Patch in progress.
> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> Improvement
> >>> >> > works.
> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
> Azure
> >>> >> > >>>> Storage. In progress.
> >>> >> > >>>>
> >>> >> > >>>> 3. Tentative features:
> >>> >> > >>>>
> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging
> >>> >> > to
> >>> >> > >>>> be done before Aug 2018.
> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
> applications.
> >>> >> > >>>> Challenging as more discussions are on-going.
> >>> >> > >>>>
> >>> >> > >>>> *Summary of 3.2.0 issues status:*
> >>> >> > >>>>
> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following
> up
> >>> with
> >>> >> > >>>> owners to get status on each of them to get in by Code Freeze
> >>> date.
> >>> >> > >>>>
> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
> in
> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> >>> >> > Version/s" =
> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
> >>> >> > >>>>
> >>> >> > >>>> Thanks,
> >>> >> > >>>> Sunil
> >>> >> > >>>>
> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <sunilg@apache.org
> <ma...@apache.org>>
> >>> >> wrote:
> >>> >> > >>>>
> >>> >> > >>>>> Hi All,
> >>> >> > >>>>>
> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
> >>> dated
> >>> >> > 17th
> >>> >> > >>>>> July 2018, I would like to provide some updates.
> >>> >> > >>>>>
> >>> >> > >>>>> We are approaching previously proposed feature freeze date
> >>> (Aug
> >>> >> 21,
> >>> >> > >>>>> about 7 days from today).
> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
> please
> >>> >> reply
> >>> >> > to
> >>> >> > >>>>> this email thread.
> >>> >> > >>>>> Steve has mentioned about the s3 features which will come
> >>> close to
> >>> >> > >>>>> Code Freeze Date (Aug 31st).
> >>> >> > >>>>>
> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
> >>> >> > >>>>>
> >>> >> > >>>>> 1. Merged & Completed features:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> >>> DeepLearning
> >>> >> > >>>>> workloads Initial cut.
> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>>>>
> >>> >> > >>>>> 2. Features close to finish:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> Major
> >>> >> > >>>>> patches are all in, only one last
> >>> >> > >>>>> patch is in review state.
> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> Capacity
> >>> >> > >>>>> Scheduler. Close to commit.
> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> >> Service
> >>> >> > >>>>> API and CLI. 2 patches are pending
> >>> >> > >>>>> which will be closed by Feature freeze date.
> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> apps
> >>> >> from
> >>> >> > >>>>> ATSv2. Patch in progress.
> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> Improvement
> >>> >> > >>>>> works.
> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
> >>> Azure
> >>> >> > >>>>> Storage. In progress.
> >>> >> > >>>>>
> >>> >> > >>>>> 3. Tentative features:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging
> >>> >> > >>>>> to be done before Aug 2018.
> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
> applications.
> >>> >> > >>>>> Challenging as more discussions are on-going.
> >>> >> > >>>>>
> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
> >>> >> > >>>>>
> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
> >>> with
> >>> >> > owners
> >>> >> > >>>>> to get status on each of them to get in by Code Freeze date.
> >>> >> > >>>>>
> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
> in
> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> >>> >> > Version/s" =
> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
> >>> >> > >>>>>
> >>> >> > >>>>> Thanks,
> >>> >> > >>>>> Sunil
> >>> >> > >>>>>
> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <sunilg@apache.org
> <ma...@apache.org>>
> >>> >> wrote:
> >>> >> > >>>>>
> >>> >> > >>>>>> Thanks Subru for the thoughts.
> >>> >> > >>>>>> One of the main reason for a major release is to push out
> >>> >> critical
> >>> >> > >>>>>> features with a faster cadence to the users. If we are
> >>> pulling
> >>> >> more
> >>> >> > and
> >>> >> > >>>>>> more different types of features to a minor release, that
> >>> branch
> >>> >> > will
> >>> >> > >>>>>> become more destabilized and it may be tough to say that
> >>> 3.1.2 is
> >>> >> > stable
> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
> >>> >> features
> >>> >> > in
> >>> >> > >>>>>> subsequent minor release.
> >>> >> > >>>>>> For few companies, it makes sense to push out these new
> >>> features
> >>> >> > >>>>>> faster to make a reach to the users. Adding to the point to
> >>> the
> >>> >> > backporting
> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
> >>> with
> >>> >> > some git
> >>> >> > >>>>>> scripts. If we can make such scripts available to
> committers,
> >>> >> > backport will
> >>> >> > >>>>>> be seem-less across branches and we can achieve the faster
> >>> >> release
> >>> >> > cadence
> >>> >> > >>>>>> also.
> >>> >> > >>>>>>
> >>> >> > >>>>>> Thoughts?
> >>> >> > >>>>>>
> >>> >> > >>>>>> - Sunil
> >>> >> > >>>>>>
> >>> >> > >>>>>>
> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
> >>> subru@apache.org<ma...@apache.org>
> >>> >> >
> >>> >> > >>>>>> wrote:
> >>> >> > >>>>>>
> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release effort.
> I
> >>> am
> >>> >> > >>>>>>> generally
> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x)
> as
> >>> >> feel we
> >>> >> > >>>>>>> already
> >>> >> > >>>>>>> have too many branches to be maintained. I already see
> many
> >>> >> commits
> >>> >> > >>>>>>> are in
> >>> >> > >>>>>>> different branches with no apparent rationale, for e.g:
> 3.1
> >>> has
> >>> >> > >>>>>>> commits
> >>> >> > >>>>>>> which are absent in 3.0 etc.
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
> >>> >> > production
> >>> >> > >>>>>>> setting so the cost of adding features should be minimal.
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> Thoughts?
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> -Subru
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
> >>> sunilg@apache.org<ma...@apache.org>>
> >>> >> > wrote:
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Yes, important changes and features are much needed,
> >>> hence we
> >>> >> > will
> >>> >> > >>>>>>> be
> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
> >>> considering
> >>> >> few
> >>> >> > >>>>>>> more
> >>> >> > >>>>>>> > offline requests from other folks, I think extending the
> >>> >> > timeframe
> >>> >> > >>>>>>> by
> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
> >>> buffer) and
> >>> >> > >>>>>>> this should
> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki
> as
> >>> >> well)
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by August
> >>> 21,
> >>> >> 2018.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
> >>> improvements
> >>> >> and
> >>> >> > >>>>>>> non
> >>> >> > >>>>>>> > blocker/critical
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Release date: September 15, 2018
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered
> >>> your
> >>> >> > >>>>>>> questions.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Thanks
> >>> >> > >>>>>>> > Sunil
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
> >>> >> wheeleast@gmail.com<ma...@gmail.com>
> >>> >> > >
> >>> >> > >>>>>>> wrote:
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release,
> >>> +1
> >>> >> for
> >>> >> > >>>>>>> that.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > To concerns from Steve,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
> important
> >>> >> > changes
> >>> >> > >>>>>>> /
> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
> >>> >> proposed
> >>> >> > >>>>>>> release
> >>> >> > >>>>>>> > date
> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of last
> >>> >> minute
> >>> >> > >>>>>>> and we all
> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm also
> >>> fine
> >>> >> if
> >>> >> > >>>>>>> we want
> >>> >> > >>>>>>> > get
> >>> >> > >>>>>>> > > another several weeks time.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
> >>> before
> >>> >> > >>>>>>> thanksgiving.
> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Eric,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
> >>> YARN-8135
> >>> >> can
> >>> >> > >>>>>>> get merged
> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Sunil,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Could u update
> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
> >>> >> > >>>>>>> > Roadmap
> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
> >>> first
> >>> >> > >>>>>>> before getting
> >>> >> > >>>>>>> > > consensus of time.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Thanks,
> >>> >> > >>>>>>> > > Wangda
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid<mailto:
> fabbri@cloudera.com.invalid>
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > wrote:
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> >>> >> > >>>>>>> stevel@hortonworks.com<ma...@hortonworks.com>>
> >>> >> > >>>>>>> > >> wrote:
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
> sunilg@apache.org<ma...@apache.org>
> >>> >> > <mailto:
> >>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>>
> wrote:
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I would also would like to take this opportunity to
> >>> come
> >>> >> up
> >>> >> > >>>>>>> with a
> >>> >> > >>>>>>> > >> detailed
> >>> >> > >>>>>>> > >> > plan.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
> >>> merged by
> >>> >> > >>>>>>> August 10,
> >>> >> > >>>>>>> > >> 2018.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > <snip>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
> targeted
> >>> to
> >>> >> 3.2
> >>> >> > >>>>>>> per this
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
> >>> Datalake
> >>> >> > Gen
> >>> >> > >>>>>>> 2
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
> >>> doing
> >>> >> work
> >>> >> > >>>>>>> on it, but
> >>> >> > >>>>>>> > >> if
> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
> >>> >> timely
> >>> >> > >>>>>>> reviews on
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will review
> >>> the
> >>> >> S3 &
> >>> >> > >>>>>>> S3Guard
> >>> >> > >>>>>>> > >> JIRAs
> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
> >>> some
> >>> >> > >>>>>>> pressing
> >>> >> > >>>>>>> > "guave,
> >>> >> > >>>>>>> > >> > java 9 prep"
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> I can help with this part if you like.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
> release
> >>> >> > manager
> >>> >> > >>>>>>> of 3.2.0
> >>> >> > >>>>>>> > >> > release.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > well volunteered!
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable should
> >>> we
> >>> >> have
> >>> >> > >>>>>>> for the
> >>> >> > >>>>>>> > >> 3.2. &
> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
> >>> having
> >>> >> > the
> >>> >> > >>>>>>> outline
> >>> >> > >>>>>>> > >> time
> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
> >>> less
> >>> >> > >>>>>>> concern about
> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > -Steve
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> -AF
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>>
> >>> >> > >>>>>>
> >>> >> >
> >>> >>
> >>> >
> >>>
> >>
> >>
>
>
> --
> A very happy Clouderan
>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Konstantin Shvachko <sh...@gmail.com>.
I've tried to attract attention to an incompatibility issue through the
jira, but it didn't work. So pitching in in this thread.
https://issues.apache.org/jira/browse/HDFS-12026
It introduced binary incompatibility, which will prevent people from
upgrading from 3.1 to 3.2.
I think it can get messy if we release anything with this feature.
Thanks,
--Konstantin
On Mon, Oct 22, 2018 at 5:01 AM Steve Loughran <st...@hortonworks.com>
wrote:
> its in.
>
> good catch!
>
> On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <weichiu@cloudera.com<mailto:
> weichiu@cloudera.com>> wrote:
>
> Thanks Sunil G for driving the release,
> I filed HADOOP-15866<https://issues.apache.org/jira/browse/HADOOP-15866>
> for a compat fix. If any one has cycle please review it, as I think it is
> needed for 3.2.0.
>
> On Thu, Oct 18, 2018 at 4:43 AM Sunil G <sunilg@apache.org<mailto:
> sunilg@apache.org>> wrote:
> Hi Folks,
>
> As we previously communicated for 3.2.0 release, we have delayed due to few
> blockers in our gate.
>
> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
> all bug fixes.
>
> - Sunil
>
>
> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <sunilg@apache.org<mailto:
> sunilg@apache.org>> wrote:
>
> > Hi Folks,
> >
> > We are now close to RC as other blocker issues are now merged to trunk
> and
> > branch-3.2. Last 2 critical issues are closer to merge and will be
> > committed in few hours.
> > With this, I will be creating 3.2.0 branch today and will go ahead with
> RC
> > related process.
> >
> > - Sunil
> >
> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jonbender@stripe.com
> <ma...@stripe.com>>
> > wrote:
> >
> >> Hello, were there any updates around the 3.2.0 RC timing? All I see in
> >> the current blockers are related to the new Submarine subproject, wasn't
> >> sure if that is what is holding things up.
> >>
> >> Cheers,
> >> Jon
> >>
> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <sunilg@apache.org<mailto:
> sunilg@apache.org>> wrote:
> >>
> >>> Thanks Robert and Haibo for quickly correcting same.
> >>> Sigh, I somehow missed one file while committing the change. Sorry for
> >>> the
> >>> trouble.
> >>>
> >>> - Sunil
> >>>
> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rkanter@cloudera.com
> <ma...@cloudera.com>>
> >>> wrote:
> >>>
> >>> > Looks like there's two that weren't updated:
> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
> >>> > --include=pom.xml
> >>> > ./hadoop-project/pom.xml:
> >>> > <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
> >>> >
> >>> > I've just pushed in an addendum commit to fix those.
> >>> > In the future, please make sure to do a sanity compile when updating
> >>> poms.
> >>> >
> >>> > thanks
> >>> > - Robert
> >>> >
> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
> >>> <fa...@cloudera.com.invalid>>
> >>> > wrote:
> >>> >
> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
> >>> >> top-level pom.xml?
> >>> >>
> >>> >>
> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <sunilg@apache.org<mailto:
> sunilg@apache.org>> wrote:
> >>> >>
> >>> >> > Hi All
> >>> >> >
> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
> trunk
> >>> to
> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
> >>> necessary
> >>> >> > patches are pulled into branch-3.2.
> >>> >> >
> >>> >> > Thank You
> >>> >> > - Sunil
> >>> >> >
> >>> >> >
> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <sunilg@apache.org
> <ma...@apache.org>> wrote:
> >>> >> >
> >>> >> > > Hi All
> >>> >> > >
> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged
> to
> >>> >> trunk.
> >>> >> > > Thanks for the support.
> >>> >> > >
> >>> >> > > *Plan for RC*
> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
> >>> >> > > RC0 for 3.2: *28th Friday*
> >>> >> > >
> >>> >> > > Thank You
> >>> >> > > Sunil
> >>> >> > >
> >>> >> > >
> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <sunilg@apache.org
> <ma...@apache.org>>
> >>> wrote:
> >>> >> > >
> >>> >> > >> Hi All
> >>> >> > >>
> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you
> >>> for
> >>> >> > >> helping in this. I am following up on these tickets, once its
> >>> closed
> >>> >> we
> >>> >> > >> will cut the 3.2 branch.
> >>> >> > >>
> >>> >> > >> Thanks
> >>> >> > >> Sunil Govindan
> >>> >> > >>
> >>> >> > >>
> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <sunilg@apache.org
> <ma...@apache.org>>
> >>> wrote:
> >>> >> > >>
> >>> >> > >>> Hi All,
> >>> >> > >>>
> >>> >> > >>> Inline with the original 3.2 communication proposal dated 17th
> >>> July
> >>> >> > >>> 2018, I would like to provide more updates.
> >>> >> > >>>
> >>> >> > >>> We are approaching previously proposed code freeze date
> >>> (September
> >>> >> 14,
> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
> point
> >>> >> > existing
> >>> >> > >>> trunk to 3.3 if there are no issues.
> >>> >> > >>>
> >>> >> > >>> *Current Release Plan:*
> >>> >> > >>> Feature freeze date : all features to merge by September 7,
> >>> 2018.
> >>> >> > >>> Code freeze date : blockers/critical only, no improvements and
> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
> >>> >> > >>> Release date: September 28, 2018
> >>> >> > >>>
> >>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0,
> we
> >>> >> need to
> >>> >> > >>> backport to 3.2 post branch cut.
> >>> >> > >>>
> >>> >> > >>> Here's an updated 3.2.0 feature status:
> >>> >> > >>>
> >>> >> > >>> 1. Merged & Completed features:
> >>> >> > >>>
> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
> DeepLearning
> >>> >> > >>> workloads Initial cut.
> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >>> >> > >>> Scheduler.
> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> Service
> >>> >> > API
> >>> >> > >>> and CLI.
> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
> Improvement
> >>> >> works.
> >>> >> > >>>
> >>> >> > >>> 2. Features close to finish:
> >>> >> > >>>
> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
> >>> >> > >>>
> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
> >>> from
> >>> >> > >>> ATSv2. Patch in progress.
> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging to
> >>> >> > >>> be done before Aug 2018.
> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >>> >> > >>> Challenging as more discussions are on-going.
> >>> >> > >>>
> >>> >> > >>> *Summary of 3.2.0 issues status:*
> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following up
> >>> with
> >>> >> > >>> owners to get status on each of them to get in by Code Freeze
> >>> date.
> >>> >> > >>>
> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> >>> >> > (Blocker,
> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" =
> >>> 3.2.0
> >>> >> > ORDER
> >>> >> > >>> BY priority DESC
> >>> >> > >>>
> >>> >> > >>> Thanks,
> >>> >> > >>> Sunil
> >>> >> > >>>
> >>> >> > >>>
> >>> >> > >>>
> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <sunilg@apache.org
> <ma...@apache.org>>
> >>> wrote:
> >>> >> > >>>
> >>> >> > >>>> Hi All,
> >>> >> > >>>>
> >>> >> > >>>> Inline with earlier communication dated 17th July 2018, I
> would
> >>> >> like
> >>> >> > to
> >>> >> > >>>> provide some updates.
> >>> >> > >>>>
> >>> >> > >>>> We are approaching previously proposed code freeze date (Aug
> >>> 31).
> >>> >> > >>>>
> >>> >> > >>>> One of the critical feature Node Attributes feature merge
> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need
> a
> >>> bit
> >>> >> > more
> >>> >> > >>>> time. With regard to this, suggesting to push the
> feature/code
> >>> >> freeze
> >>> >> > for 2
> >>> >> > >>>> more weeks to accommodate these jiras too.
> >>> >> > >>>>
> >>> >> > >>>> Proposing Updated changes in plan inline with this:
> >>> >> > >>>> Feature freeze date : all features to merge by September 7,
> >>> 2018.
> >>> >> > >>>> Code freeze date : blockers/critical only, no improvements
> and
> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
> >>> >> > >>>> Release date: September 28, 2018
> >>> >> > >>>>
> >>> >> > >>>> If any features in branch which are targeted to 3.2.0, please
> >>> >> reply to
> >>> >> > >>>> this email thread.
> >>> >> > >>>>
> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
> >>> >> > >>>>
> >>> >> > >>>> 1. Merged & Completed features:
> >>> >> > >>>>
> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> DeepLearning
> >>> >> > >>>> workloads Initial cut.
> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> Capacity
> >>> >> > >>>> Scheduler.
> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> >> Service
> >>> >> > >>>> API and CLI.
> >>> >> > >>>>
> >>> >> > >>>> 2. Features close to finish:
> >>> >> > >>>>
> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> >> Merge/Vote
> >>> >> > >>>> Ongoing.
> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> apps
> >>> from
> >>> >> > >>>> ATSv2. Patch in progress.
> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> Improvement
> >>> >> > works.
> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
> Azure
> >>> >> > >>>> Storage. In progress.
> >>> >> > >>>>
> >>> >> > >>>> 3. Tentative features:
> >>> >> > >>>>
> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging
> >>> >> > to
> >>> >> > >>>> be done before Aug 2018.
> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
> applications.
> >>> >> > >>>> Challenging as more discussions are on-going.
> >>> >> > >>>>
> >>> >> > >>>> *Summary of 3.2.0 issues status:*
> >>> >> > >>>>
> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following
> up
> >>> with
> >>> >> > >>>> owners to get status on each of them to get in by Code Freeze
> >>> date.
> >>> >> > >>>>
> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
> in
> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> >>> >> > Version/s" =
> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
> >>> >> > >>>>
> >>> >> > >>>> Thanks,
> >>> >> > >>>> Sunil
> >>> >> > >>>>
> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <sunilg@apache.org
> <ma...@apache.org>>
> >>> >> wrote:
> >>> >> > >>>>
> >>> >> > >>>>> Hi All,
> >>> >> > >>>>>
> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
> >>> dated
> >>> >> > 17th
> >>> >> > >>>>> July 2018, I would like to provide some updates.
> >>> >> > >>>>>
> >>> >> > >>>>> We are approaching previously proposed feature freeze date
> >>> (Aug
> >>> >> 21,
> >>> >> > >>>>> about 7 days from today).
> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
> please
> >>> >> reply
> >>> >> > to
> >>> >> > >>>>> this email thread.
> >>> >> > >>>>> Steve has mentioned about the s3 features which will come
> >>> close to
> >>> >> > >>>>> Code Freeze Date (Aug 31st).
> >>> >> > >>>>>
> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
> >>> >> > >>>>>
> >>> >> > >>>>> 1. Merged & Completed features:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> >>> DeepLearning
> >>> >> > >>>>> workloads Initial cut.
> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>>>>
> >>> >> > >>>>> 2. Features close to finish:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> Major
> >>> >> > >>>>> patches are all in, only one last
> >>> >> > >>>>> patch is in review state.
> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> Capacity
> >>> >> > >>>>> Scheduler. Close to commit.
> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> >> Service
> >>> >> > >>>>> API and CLI. 2 patches are pending
> >>> >> > >>>>> which will be closed by Feature freeze date.
> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> apps
> >>> >> from
> >>> >> > >>>>> ATSv2. Patch in progress.
> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> Improvement
> >>> >> > >>>>> works.
> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
> >>> Azure
> >>> >> > >>>>> Storage. In progress.
> >>> >> > >>>>>
> >>> >> > >>>>> 3. Tentative features:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging
> >>> >> > >>>>> to be done before Aug 2018.
> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
> applications.
> >>> >> > >>>>> Challenging as more discussions are on-going.
> >>> >> > >>>>>
> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
> >>> >> > >>>>>
> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
> >>> with
> >>> >> > owners
> >>> >> > >>>>> to get status on each of them to get in by Code Freeze date.
> >>> >> > >>>>>
> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
> in
> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> >>> >> > Version/s" =
> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
> >>> >> > >>>>>
> >>> >> > >>>>> Thanks,
> >>> >> > >>>>> Sunil
> >>> >> > >>>>>
> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <sunilg@apache.org
> <ma...@apache.org>>
> >>> >> wrote:
> >>> >> > >>>>>
> >>> >> > >>>>>> Thanks Subru for the thoughts.
> >>> >> > >>>>>> One of the main reason for a major release is to push out
> >>> >> critical
> >>> >> > >>>>>> features with a faster cadence to the users. If we are
> >>> pulling
> >>> >> more
> >>> >> > and
> >>> >> > >>>>>> more different types of features to a minor release, that
> >>> branch
> >>> >> > will
> >>> >> > >>>>>> become more destabilized and it may be tough to say that
> >>> 3.1.2 is
> >>> >> > stable
> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
> >>> >> features
> >>> >> > in
> >>> >> > >>>>>> subsequent minor release.
> >>> >> > >>>>>> For few companies, it makes sense to push out these new
> >>> features
> >>> >> > >>>>>> faster to make a reach to the users. Adding to the point to
> >>> the
> >>> >> > backporting
> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
> >>> with
> >>> >> > some git
> >>> >> > >>>>>> scripts. If we can make such scripts available to
> committers,
> >>> >> > backport will
> >>> >> > >>>>>> be seem-less across branches and we can achieve the faster
> >>> >> release
> >>> >> > cadence
> >>> >> > >>>>>> also.
> >>> >> > >>>>>>
> >>> >> > >>>>>> Thoughts?
> >>> >> > >>>>>>
> >>> >> > >>>>>> - Sunil
> >>> >> > >>>>>>
> >>> >> > >>>>>>
> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
> >>> subru@apache.org<ma...@apache.org>
> >>> >> >
> >>> >> > >>>>>> wrote:
> >>> >> > >>>>>>
> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release effort.
> I
> >>> am
> >>> >> > >>>>>>> generally
> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x)
> as
> >>> >> feel we
> >>> >> > >>>>>>> already
> >>> >> > >>>>>>> have too many branches to be maintained. I already see
> many
> >>> >> commits
> >>> >> > >>>>>>> are in
> >>> >> > >>>>>>> different branches with no apparent rationale, for e.g:
> 3.1
> >>> has
> >>> >> > >>>>>>> commits
> >>> >> > >>>>>>> which are absent in 3.0 etc.
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
> >>> >> > production
> >>> >> > >>>>>>> setting so the cost of adding features should be minimal.
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> Thoughts?
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> -Subru
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
> >>> sunilg@apache.org<ma...@apache.org>>
> >>> >> > wrote:
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Yes, important changes and features are much needed,
> >>> hence we
> >>> >> > will
> >>> >> > >>>>>>> be
> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
> >>> considering
> >>> >> few
> >>> >> > >>>>>>> more
> >>> >> > >>>>>>> > offline requests from other folks, I think extending the
> >>> >> > timeframe
> >>> >> > >>>>>>> by
> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
> >>> buffer) and
> >>> >> > >>>>>>> this should
> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki
> as
> >>> >> well)
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by August
> >>> 21,
> >>> >> 2018.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
> >>> improvements
> >>> >> and
> >>> >> > >>>>>>> non
> >>> >> > >>>>>>> > blocker/critical
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Release date: September 15, 2018
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered
> >>> your
> >>> >> > >>>>>>> questions.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Thanks
> >>> >> > >>>>>>> > Sunil
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
> >>> >> wheeleast@gmail.com<ma...@gmail.com>
> >>> >> > >
> >>> >> > >>>>>>> wrote:
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release,
> >>> +1
> >>> >> for
> >>> >> > >>>>>>> that.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > To concerns from Steve,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
> important
> >>> >> > changes
> >>> >> > >>>>>>> /
> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
> >>> >> proposed
> >>> >> > >>>>>>> release
> >>> >> > >>>>>>> > date
> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of last
> >>> >> minute
> >>> >> > >>>>>>> and we all
> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm also
> >>> fine
> >>> >> if
> >>> >> > >>>>>>> we want
> >>> >> > >>>>>>> > get
> >>> >> > >>>>>>> > > another several weeks time.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
> >>> before
> >>> >> > >>>>>>> thanksgiving.
> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Eric,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
> >>> YARN-8135
> >>> >> can
> >>> >> > >>>>>>> get merged
> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Sunil,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Could u update
> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
> >>> >> > >>>>>>> > Roadmap
> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
> >>> first
> >>> >> > >>>>>>> before getting
> >>> >> > >>>>>>> > > consensus of time.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Thanks,
> >>> >> > >>>>>>> > > Wangda
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid<mailto:
> fabbri@cloudera.com.invalid>
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > wrote:
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> >>> >> > >>>>>>> stevel@hortonworks.com<ma...@hortonworks.com>>
> >>> >> > >>>>>>> > >> wrote:
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
> sunilg@apache.org<ma...@apache.org>
> >>> >> > <mailto:
> >>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>>
> wrote:
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I would also would like to take this opportunity to
> >>> come
> >>> >> up
> >>> >> > >>>>>>> with a
> >>> >> > >>>>>>> > >> detailed
> >>> >> > >>>>>>> > >> > plan.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
> >>> merged by
> >>> >> > >>>>>>> August 10,
> >>> >> > >>>>>>> > >> 2018.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > <snip>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
> targeted
> >>> to
> >>> >> 3.2
> >>> >> > >>>>>>> per this
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
> >>> Datalake
> >>> >> > Gen
> >>> >> > >>>>>>> 2
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
> >>> doing
> >>> >> work
> >>> >> > >>>>>>> on it, but
> >>> >> > >>>>>>> > >> if
> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
> >>> >> timely
> >>> >> > >>>>>>> reviews on
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will review
> >>> the
> >>> >> S3 &
> >>> >> > >>>>>>> S3Guard
> >>> >> > >>>>>>> > >> JIRAs
> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
> >>> some
> >>> >> > >>>>>>> pressing
> >>> >> > >>>>>>> > "guave,
> >>> >> > >>>>>>> > >> > java 9 prep"
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> I can help with this part if you like.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
> release
> >>> >> > manager
> >>> >> > >>>>>>> of 3.2.0
> >>> >> > >>>>>>> > >> > release.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > well volunteered!
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable should
> >>> we
> >>> >> have
> >>> >> > >>>>>>> for the
> >>> >> > >>>>>>> > >> 3.2. &
> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
> >>> having
> >>> >> > the
> >>> >> > >>>>>>> outline
> >>> >> > >>>>>>> > >> time
> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
> >>> less
> >>> >> > >>>>>>> concern about
> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > -Steve
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> -AF
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>>
> >>> >> > >>>>>>
> >>> >> >
> >>> >>
> >>> >
> >>>
> >>
> >>
>
>
> --
> A very happy Clouderan
>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Konstantin Shvachko <sh...@gmail.com>.
I've tried to attract attention to an incompatibility issue through the
jira, but it didn't work. So pitching in in this thread.
https://issues.apache.org/jira/browse/HDFS-12026
It introduced binary incompatibility, which will prevent people from
upgrading from 3.1 to 3.2.
I think it can get messy if we release anything with this feature.
Thanks,
--Konstantin
On Mon, Oct 22, 2018 at 5:01 AM Steve Loughran <st...@hortonworks.com>
wrote:
> its in.
>
> good catch!
>
> On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <weichiu@cloudera.com<mailto:
> weichiu@cloudera.com>> wrote:
>
> Thanks Sunil G for driving the release,
> I filed HADOOP-15866<https://issues.apache.org/jira/browse/HADOOP-15866>
> for a compat fix. If any one has cycle please review it, as I think it is
> needed for 3.2.0.
>
> On Thu, Oct 18, 2018 at 4:43 AM Sunil G <sunilg@apache.org<mailto:
> sunilg@apache.org>> wrote:
> Hi Folks,
>
> As we previously communicated for 3.2.0 release, we have delayed due to few
> blockers in our gate.
>
> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
> all bug fixes.
>
> - Sunil
>
>
> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <sunilg@apache.org<mailto:
> sunilg@apache.org>> wrote:
>
> > Hi Folks,
> >
> > We are now close to RC as other blocker issues are now merged to trunk
> and
> > branch-3.2. Last 2 critical issues are closer to merge and will be
> > committed in few hours.
> > With this, I will be creating 3.2.0 branch today and will go ahead with
> RC
> > related process.
> >
> > - Sunil
> >
> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jonbender@stripe.com
> <ma...@stripe.com>>
> > wrote:
> >
> >> Hello, were there any updates around the 3.2.0 RC timing? All I see in
> >> the current blockers are related to the new Submarine subproject, wasn't
> >> sure if that is what is holding things up.
> >>
> >> Cheers,
> >> Jon
> >>
> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <sunilg@apache.org<mailto:
> sunilg@apache.org>> wrote:
> >>
> >>> Thanks Robert and Haibo for quickly correcting same.
> >>> Sigh, I somehow missed one file while committing the change. Sorry for
> >>> the
> >>> trouble.
> >>>
> >>> - Sunil
> >>>
> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rkanter@cloudera.com
> <ma...@cloudera.com>>
> >>> wrote:
> >>>
> >>> > Looks like there's two that weren't updated:
> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
> >>> > --include=pom.xml
> >>> > ./hadoop-project/pom.xml:
> >>> > <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
> >>> >
> >>> > I've just pushed in an addendum commit to fix those.
> >>> > In the future, please make sure to do a sanity compile when updating
> >>> poms.
> >>> >
> >>> > thanks
> >>> > - Robert
> >>> >
> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
> >>> <fa...@cloudera.com.invalid>>
> >>> > wrote:
> >>> >
> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
> >>> >> top-level pom.xml?
> >>> >>
> >>> >>
> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <sunilg@apache.org<mailto:
> sunilg@apache.org>> wrote:
> >>> >>
> >>> >> > Hi All
> >>> >> >
> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
> trunk
> >>> to
> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
> >>> necessary
> >>> >> > patches are pulled into branch-3.2.
> >>> >> >
> >>> >> > Thank You
> >>> >> > - Sunil
> >>> >> >
> >>> >> >
> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <sunilg@apache.org
> <ma...@apache.org>> wrote:
> >>> >> >
> >>> >> > > Hi All
> >>> >> > >
> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged
> to
> >>> >> trunk.
> >>> >> > > Thanks for the support.
> >>> >> > >
> >>> >> > > *Plan for RC*
> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
> >>> >> > > RC0 for 3.2: *28th Friday*
> >>> >> > >
> >>> >> > > Thank You
> >>> >> > > Sunil
> >>> >> > >
> >>> >> > >
> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <sunilg@apache.org
> <ma...@apache.org>>
> >>> wrote:
> >>> >> > >
> >>> >> > >> Hi All
> >>> >> > >>
> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you
> >>> for
> >>> >> > >> helping in this. I am following up on these tickets, once its
> >>> closed
> >>> >> we
> >>> >> > >> will cut the 3.2 branch.
> >>> >> > >>
> >>> >> > >> Thanks
> >>> >> > >> Sunil Govindan
> >>> >> > >>
> >>> >> > >>
> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <sunilg@apache.org
> <ma...@apache.org>>
> >>> wrote:
> >>> >> > >>
> >>> >> > >>> Hi All,
> >>> >> > >>>
> >>> >> > >>> Inline with the original 3.2 communication proposal dated 17th
> >>> July
> >>> >> > >>> 2018, I would like to provide more updates.
> >>> >> > >>>
> >>> >> > >>> We are approaching previously proposed code freeze date
> >>> (September
> >>> >> 14,
> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
> point
> >>> >> > existing
> >>> >> > >>> trunk to 3.3 if there are no issues.
> >>> >> > >>>
> >>> >> > >>> *Current Release Plan:*
> >>> >> > >>> Feature freeze date : all features to merge by September 7,
> >>> 2018.
> >>> >> > >>> Code freeze date : blockers/critical only, no improvements and
> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
> >>> >> > >>> Release date: September 28, 2018
> >>> >> > >>>
> >>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0,
> we
> >>> >> need to
> >>> >> > >>> backport to 3.2 post branch cut.
> >>> >> > >>>
> >>> >> > >>> Here's an updated 3.2.0 feature status:
> >>> >> > >>>
> >>> >> > >>> 1. Merged & Completed features:
> >>> >> > >>>
> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
> DeepLearning
> >>> >> > >>> workloads Initial cut.
> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >>> >> > >>> Scheduler.
> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> Service
> >>> >> > API
> >>> >> > >>> and CLI.
> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
> Improvement
> >>> >> works.
> >>> >> > >>>
> >>> >> > >>> 2. Features close to finish:
> >>> >> > >>>
> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
> >>> >> > >>>
> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
> >>> from
> >>> >> > >>> ATSv2. Patch in progress.
> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging to
> >>> >> > >>> be done before Aug 2018.
> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >>> >> > >>> Challenging as more discussions are on-going.
> >>> >> > >>>
> >>> >> > >>> *Summary of 3.2.0 issues status:*
> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following up
> >>> with
> >>> >> > >>> owners to get status on each of them to get in by Code Freeze
> >>> date.
> >>> >> > >>>
> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> >>> >> > (Blocker,
> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" =
> >>> 3.2.0
> >>> >> > ORDER
> >>> >> > >>> BY priority DESC
> >>> >> > >>>
> >>> >> > >>> Thanks,
> >>> >> > >>> Sunil
> >>> >> > >>>
> >>> >> > >>>
> >>> >> > >>>
> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <sunilg@apache.org
> <ma...@apache.org>>
> >>> wrote:
> >>> >> > >>>
> >>> >> > >>>> Hi All,
> >>> >> > >>>>
> >>> >> > >>>> Inline with earlier communication dated 17th July 2018, I
> would
> >>> >> like
> >>> >> > to
> >>> >> > >>>> provide some updates.
> >>> >> > >>>>
> >>> >> > >>>> We are approaching previously proposed code freeze date (Aug
> >>> 31).
> >>> >> > >>>>
> >>> >> > >>>> One of the critical feature Node Attributes feature merge
> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need
> a
> >>> bit
> >>> >> > more
> >>> >> > >>>> time. With regard to this, suggesting to push the
> feature/code
> >>> >> freeze
> >>> >> > for 2
> >>> >> > >>>> more weeks to accommodate these jiras too.
> >>> >> > >>>>
> >>> >> > >>>> Proposing Updated changes in plan inline with this:
> >>> >> > >>>> Feature freeze date : all features to merge by September 7,
> >>> 2018.
> >>> >> > >>>> Code freeze date : blockers/critical only, no improvements
> and
> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
> >>> >> > >>>> Release date: September 28, 2018
> >>> >> > >>>>
> >>> >> > >>>> If any features in branch which are targeted to 3.2.0, please
> >>> >> reply to
> >>> >> > >>>> this email thread.
> >>> >> > >>>>
> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
> >>> >> > >>>>
> >>> >> > >>>> 1. Merged & Completed features:
> >>> >> > >>>>
> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> DeepLearning
> >>> >> > >>>> workloads Initial cut.
> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> Capacity
> >>> >> > >>>> Scheduler.
> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> >> Service
> >>> >> > >>>> API and CLI.
> >>> >> > >>>>
> >>> >> > >>>> 2. Features close to finish:
> >>> >> > >>>>
> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> >> Merge/Vote
> >>> >> > >>>> Ongoing.
> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> apps
> >>> from
> >>> >> > >>>> ATSv2. Patch in progress.
> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> Improvement
> >>> >> > works.
> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
> Azure
> >>> >> > >>>> Storage. In progress.
> >>> >> > >>>>
> >>> >> > >>>> 3. Tentative features:
> >>> >> > >>>>
> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging
> >>> >> > to
> >>> >> > >>>> be done before Aug 2018.
> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
> applications.
> >>> >> > >>>> Challenging as more discussions are on-going.
> >>> >> > >>>>
> >>> >> > >>>> *Summary of 3.2.0 issues status:*
> >>> >> > >>>>
> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following
> up
> >>> with
> >>> >> > >>>> owners to get status on each of them to get in by Code Freeze
> >>> date.
> >>> >> > >>>>
> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
> in
> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> >>> >> > Version/s" =
> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
> >>> >> > >>>>
> >>> >> > >>>> Thanks,
> >>> >> > >>>> Sunil
> >>> >> > >>>>
> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <sunilg@apache.org
> <ma...@apache.org>>
> >>> >> wrote:
> >>> >> > >>>>
> >>> >> > >>>>> Hi All,
> >>> >> > >>>>>
> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
> >>> dated
> >>> >> > 17th
> >>> >> > >>>>> July 2018, I would like to provide some updates.
> >>> >> > >>>>>
> >>> >> > >>>>> We are approaching previously proposed feature freeze date
> >>> (Aug
> >>> >> 21,
> >>> >> > >>>>> about 7 days from today).
> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
> please
> >>> >> reply
> >>> >> > to
> >>> >> > >>>>> this email thread.
> >>> >> > >>>>> Steve has mentioned about the s3 features which will come
> >>> close to
> >>> >> > >>>>> Code Freeze Date (Aug 31st).
> >>> >> > >>>>>
> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
> >>> >> > >>>>>
> >>> >> > >>>>> 1. Merged & Completed features:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> >>> DeepLearning
> >>> >> > >>>>> workloads Initial cut.
> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>>>>
> >>> >> > >>>>> 2. Features close to finish:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> Major
> >>> >> > >>>>> patches are all in, only one last
> >>> >> > >>>>> patch is in review state.
> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> Capacity
> >>> >> > >>>>> Scheduler. Close to commit.
> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> >> Service
> >>> >> > >>>>> API and CLI. 2 patches are pending
> >>> >> > >>>>> which will be closed by Feature freeze date.
> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> apps
> >>> >> from
> >>> >> > >>>>> ATSv2. Patch in progress.
> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> Improvement
> >>> >> > >>>>> works.
> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
> >>> Azure
> >>> >> > >>>>> Storage. In progress.
> >>> >> > >>>>>
> >>> >> > >>>>> 3. Tentative features:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging
> >>> >> > >>>>> to be done before Aug 2018.
> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
> applications.
> >>> >> > >>>>> Challenging as more discussions are on-going.
> >>> >> > >>>>>
> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
> >>> >> > >>>>>
> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
> >>> with
> >>> >> > owners
> >>> >> > >>>>> to get status on each of them to get in by Code Freeze date.
> >>> >> > >>>>>
> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
> in
> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> >>> >> > Version/s" =
> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
> >>> >> > >>>>>
> >>> >> > >>>>> Thanks,
> >>> >> > >>>>> Sunil
> >>> >> > >>>>>
> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <sunilg@apache.org
> <ma...@apache.org>>
> >>> >> wrote:
> >>> >> > >>>>>
> >>> >> > >>>>>> Thanks Subru for the thoughts.
> >>> >> > >>>>>> One of the main reason for a major release is to push out
> >>> >> critical
> >>> >> > >>>>>> features with a faster cadence to the users. If we are
> >>> pulling
> >>> >> more
> >>> >> > and
> >>> >> > >>>>>> more different types of features to a minor release, that
> >>> branch
> >>> >> > will
> >>> >> > >>>>>> become more destabilized and it may be tough to say that
> >>> 3.1.2 is
> >>> >> > stable
> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
> >>> >> features
> >>> >> > in
> >>> >> > >>>>>> subsequent minor release.
> >>> >> > >>>>>> For few companies, it makes sense to push out these new
> >>> features
> >>> >> > >>>>>> faster to make a reach to the users. Adding to the point to
> >>> the
> >>> >> > backporting
> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
> >>> with
> >>> >> > some git
> >>> >> > >>>>>> scripts. If we can make such scripts available to
> committers,
> >>> >> > backport will
> >>> >> > >>>>>> be seem-less across branches and we can achieve the faster
> >>> >> release
> >>> >> > cadence
> >>> >> > >>>>>> also.
> >>> >> > >>>>>>
> >>> >> > >>>>>> Thoughts?
> >>> >> > >>>>>>
> >>> >> > >>>>>> - Sunil
> >>> >> > >>>>>>
> >>> >> > >>>>>>
> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
> >>> subru@apache.org<ma...@apache.org>
> >>> >> >
> >>> >> > >>>>>> wrote:
> >>> >> > >>>>>>
> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release effort.
> I
> >>> am
> >>> >> > >>>>>>> generally
> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x)
> as
> >>> >> feel we
> >>> >> > >>>>>>> already
> >>> >> > >>>>>>> have too many branches to be maintained. I already see
> many
> >>> >> commits
> >>> >> > >>>>>>> are in
> >>> >> > >>>>>>> different branches with no apparent rationale, for e.g:
> 3.1
> >>> has
> >>> >> > >>>>>>> commits
> >>> >> > >>>>>>> which are absent in 3.0 etc.
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
> >>> >> > production
> >>> >> > >>>>>>> setting so the cost of adding features should be minimal.
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> Thoughts?
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> -Subru
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
> >>> sunilg@apache.org<ma...@apache.org>>
> >>> >> > wrote:
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Yes, important changes and features are much needed,
> >>> hence we
> >>> >> > will
> >>> >> > >>>>>>> be
> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
> >>> considering
> >>> >> few
> >>> >> > >>>>>>> more
> >>> >> > >>>>>>> > offline requests from other folks, I think extending the
> >>> >> > timeframe
> >>> >> > >>>>>>> by
> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
> >>> buffer) and
> >>> >> > >>>>>>> this should
> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki
> as
> >>> >> well)
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by August
> >>> 21,
> >>> >> 2018.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
> >>> improvements
> >>> >> and
> >>> >> > >>>>>>> non
> >>> >> > >>>>>>> > blocker/critical
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Release date: September 15, 2018
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered
> >>> your
> >>> >> > >>>>>>> questions.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Thanks
> >>> >> > >>>>>>> > Sunil
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
> >>> >> wheeleast@gmail.com<ma...@gmail.com>
> >>> >> > >
> >>> >> > >>>>>>> wrote:
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release,
> >>> +1
> >>> >> for
> >>> >> > >>>>>>> that.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > To concerns from Steve,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
> important
> >>> >> > changes
> >>> >> > >>>>>>> /
> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
> >>> >> proposed
> >>> >> > >>>>>>> release
> >>> >> > >>>>>>> > date
> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of last
> >>> >> minute
> >>> >> > >>>>>>> and we all
> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm also
> >>> fine
> >>> >> if
> >>> >> > >>>>>>> we want
> >>> >> > >>>>>>> > get
> >>> >> > >>>>>>> > > another several weeks time.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
> >>> before
> >>> >> > >>>>>>> thanksgiving.
> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Eric,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
> >>> YARN-8135
> >>> >> can
> >>> >> > >>>>>>> get merged
> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Sunil,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Could u update
> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
> >>> >> > >>>>>>> > Roadmap
> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
> >>> first
> >>> >> > >>>>>>> before getting
> >>> >> > >>>>>>> > > consensus of time.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Thanks,
> >>> >> > >>>>>>> > > Wangda
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid<mailto:
> fabbri@cloudera.com.invalid>
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > wrote:
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> >>> >> > >>>>>>> stevel@hortonworks.com<ma...@hortonworks.com>>
> >>> >> > >>>>>>> > >> wrote:
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
> sunilg@apache.org<ma...@apache.org>
> >>> >> > <mailto:
> >>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>>
> wrote:
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I would also would like to take this opportunity to
> >>> come
> >>> >> up
> >>> >> > >>>>>>> with a
> >>> >> > >>>>>>> > >> detailed
> >>> >> > >>>>>>> > >> > plan.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
> >>> merged by
> >>> >> > >>>>>>> August 10,
> >>> >> > >>>>>>> > >> 2018.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > <snip>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
> targeted
> >>> to
> >>> >> 3.2
> >>> >> > >>>>>>> per this
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
> >>> Datalake
> >>> >> > Gen
> >>> >> > >>>>>>> 2
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
> >>> doing
> >>> >> work
> >>> >> > >>>>>>> on it, but
> >>> >> > >>>>>>> > >> if
> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
> >>> >> timely
> >>> >> > >>>>>>> reviews on
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will review
> >>> the
> >>> >> S3 &
> >>> >> > >>>>>>> S3Guard
> >>> >> > >>>>>>> > >> JIRAs
> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
> >>> some
> >>> >> > >>>>>>> pressing
> >>> >> > >>>>>>> > "guave,
> >>> >> > >>>>>>> > >> > java 9 prep"
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> I can help with this part if you like.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
> release
> >>> >> > manager
> >>> >> > >>>>>>> of 3.2.0
> >>> >> > >>>>>>> > >> > release.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > well volunteered!
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable should
> >>> we
> >>> >> have
> >>> >> > >>>>>>> for the
> >>> >> > >>>>>>> > >> 3.2. &
> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
> >>> having
> >>> >> > the
> >>> >> > >>>>>>> outline
> >>> >> > >>>>>>> > >> time
> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
> >>> less
> >>> >> > >>>>>>> concern about
> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > -Steve
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> -AF
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>>
> >>> >> > >>>>>>
> >>> >> >
> >>> >>
> >>> >
> >>>
> >>
> >>
>
>
> --
> A very happy Clouderan
>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Konstantin Shvachko <sh...@gmail.com>.
I've tried to attract attention to an incompatibility issue through the
jira, but it didn't work. So pitching in in this thread.
https://issues.apache.org/jira/browse/HDFS-12026
It introduced binary incompatibility, which will prevent people from
upgrading from 3.1 to 3.2.
I think it can get messy if we release anything with this feature.
Thanks,
--Konstantin
On Mon, Oct 22, 2018 at 5:01 AM Steve Loughran <st...@hortonworks.com>
wrote:
> its in.
>
> good catch!
>
> On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <weichiu@cloudera.com<mailto:
> weichiu@cloudera.com>> wrote:
>
> Thanks Sunil G for driving the release,
> I filed HADOOP-15866<https://issues.apache.org/jira/browse/HADOOP-15866>
> for a compat fix. If any one has cycle please review it, as I think it is
> needed for 3.2.0.
>
> On Thu, Oct 18, 2018 at 4:43 AM Sunil G <sunilg@apache.org<mailto:
> sunilg@apache.org>> wrote:
> Hi Folks,
>
> As we previously communicated for 3.2.0 release, we have delayed due to few
> blockers in our gate.
>
> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
> all bug fixes.
>
> - Sunil
>
>
> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <sunilg@apache.org<mailto:
> sunilg@apache.org>> wrote:
>
> > Hi Folks,
> >
> > We are now close to RC as other blocker issues are now merged to trunk
> and
> > branch-3.2. Last 2 critical issues are closer to merge and will be
> > committed in few hours.
> > With this, I will be creating 3.2.0 branch today and will go ahead with
> RC
> > related process.
> >
> > - Sunil
> >
> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jonbender@stripe.com
> <ma...@stripe.com>>
> > wrote:
> >
> >> Hello, were there any updates around the 3.2.0 RC timing? All I see in
> >> the current blockers are related to the new Submarine subproject, wasn't
> >> sure if that is what is holding things up.
> >>
> >> Cheers,
> >> Jon
> >>
> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <sunilg@apache.org<mailto:
> sunilg@apache.org>> wrote:
> >>
> >>> Thanks Robert and Haibo for quickly correcting same.
> >>> Sigh, I somehow missed one file while committing the change. Sorry for
> >>> the
> >>> trouble.
> >>>
> >>> - Sunil
> >>>
> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rkanter@cloudera.com
> <ma...@cloudera.com>>
> >>> wrote:
> >>>
> >>> > Looks like there's two that weren't updated:
> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
> >>> > --include=pom.xml
> >>> > ./hadoop-project/pom.xml:
> >>> > <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
> >>> >
> >>> > I've just pushed in an addendum commit to fix those.
> >>> > In the future, please make sure to do a sanity compile when updating
> >>> poms.
> >>> >
> >>> > thanks
> >>> > - Robert
> >>> >
> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
> >>> <fa...@cloudera.com.invalid>>
> >>> > wrote:
> >>> >
> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
> >>> >> top-level pom.xml?
> >>> >>
> >>> >>
> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <sunilg@apache.org<mailto:
> sunilg@apache.org>> wrote:
> >>> >>
> >>> >> > Hi All
> >>> >> >
> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
> trunk
> >>> to
> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
> >>> necessary
> >>> >> > patches are pulled into branch-3.2.
> >>> >> >
> >>> >> > Thank You
> >>> >> > - Sunil
> >>> >> >
> >>> >> >
> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <sunilg@apache.org
> <ma...@apache.org>> wrote:
> >>> >> >
> >>> >> > > Hi All
> >>> >> > >
> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged
> to
> >>> >> trunk.
> >>> >> > > Thanks for the support.
> >>> >> > >
> >>> >> > > *Plan for RC*
> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
> >>> >> > > RC0 for 3.2: *28th Friday*
> >>> >> > >
> >>> >> > > Thank You
> >>> >> > > Sunil
> >>> >> > >
> >>> >> > >
> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <sunilg@apache.org
> <ma...@apache.org>>
> >>> wrote:
> >>> >> > >
> >>> >> > >> Hi All
> >>> >> > >>
> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you
> >>> for
> >>> >> > >> helping in this. I am following up on these tickets, once its
> >>> closed
> >>> >> we
> >>> >> > >> will cut the 3.2 branch.
> >>> >> > >>
> >>> >> > >> Thanks
> >>> >> > >> Sunil Govindan
> >>> >> > >>
> >>> >> > >>
> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <sunilg@apache.org
> <ma...@apache.org>>
> >>> wrote:
> >>> >> > >>
> >>> >> > >>> Hi All,
> >>> >> > >>>
> >>> >> > >>> Inline with the original 3.2 communication proposal dated 17th
> >>> July
> >>> >> > >>> 2018, I would like to provide more updates.
> >>> >> > >>>
> >>> >> > >>> We are approaching previously proposed code freeze date
> >>> (September
> >>> >> 14,
> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
> point
> >>> >> > existing
> >>> >> > >>> trunk to 3.3 if there are no issues.
> >>> >> > >>>
> >>> >> > >>> *Current Release Plan:*
> >>> >> > >>> Feature freeze date : all features to merge by September 7,
> >>> 2018.
> >>> >> > >>> Code freeze date : blockers/critical only, no improvements and
> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
> >>> >> > >>> Release date: September 28, 2018
> >>> >> > >>>
> >>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0,
> we
> >>> >> need to
> >>> >> > >>> backport to 3.2 post branch cut.
> >>> >> > >>>
> >>> >> > >>> Here's an updated 3.2.0 feature status:
> >>> >> > >>>
> >>> >> > >>> 1. Merged & Completed features:
> >>> >> > >>>
> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
> DeepLearning
> >>> >> > >>> workloads Initial cut.
> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >>> >> > >>> Scheduler.
> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> Service
> >>> >> > API
> >>> >> > >>> and CLI.
> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
> Improvement
> >>> >> works.
> >>> >> > >>>
> >>> >> > >>> 2. Features close to finish:
> >>> >> > >>>
> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
> >>> >> > >>>
> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
> >>> from
> >>> >> > >>> ATSv2. Patch in progress.
> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging to
> >>> >> > >>> be done before Aug 2018.
> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >>> >> > >>> Challenging as more discussions are on-going.
> >>> >> > >>>
> >>> >> > >>> *Summary of 3.2.0 issues status:*
> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following up
> >>> with
> >>> >> > >>> owners to get status on each of them to get in by Code Freeze
> >>> date.
> >>> >> > >>>
> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> >>> >> > (Blocker,
> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" =
> >>> 3.2.0
> >>> >> > ORDER
> >>> >> > >>> BY priority DESC
> >>> >> > >>>
> >>> >> > >>> Thanks,
> >>> >> > >>> Sunil
> >>> >> > >>>
> >>> >> > >>>
> >>> >> > >>>
> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <sunilg@apache.org
> <ma...@apache.org>>
> >>> wrote:
> >>> >> > >>>
> >>> >> > >>>> Hi All,
> >>> >> > >>>>
> >>> >> > >>>> Inline with earlier communication dated 17th July 2018, I
> would
> >>> >> like
> >>> >> > to
> >>> >> > >>>> provide some updates.
> >>> >> > >>>>
> >>> >> > >>>> We are approaching previously proposed code freeze date (Aug
> >>> 31).
> >>> >> > >>>>
> >>> >> > >>>> One of the critical feature Node Attributes feature merge
> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need
> a
> >>> bit
> >>> >> > more
> >>> >> > >>>> time. With regard to this, suggesting to push the
> feature/code
> >>> >> freeze
> >>> >> > for 2
> >>> >> > >>>> more weeks to accommodate these jiras too.
> >>> >> > >>>>
> >>> >> > >>>> Proposing Updated changes in plan inline with this:
> >>> >> > >>>> Feature freeze date : all features to merge by September 7,
> >>> 2018.
> >>> >> > >>>> Code freeze date : blockers/critical only, no improvements
> and
> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
> >>> >> > >>>> Release date: September 28, 2018
> >>> >> > >>>>
> >>> >> > >>>> If any features in branch which are targeted to 3.2.0, please
> >>> >> reply to
> >>> >> > >>>> this email thread.
> >>> >> > >>>>
> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
> >>> >> > >>>>
> >>> >> > >>>> 1. Merged & Completed features:
> >>> >> > >>>>
> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> DeepLearning
> >>> >> > >>>> workloads Initial cut.
> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> Capacity
> >>> >> > >>>> Scheduler.
> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> >> Service
> >>> >> > >>>> API and CLI.
> >>> >> > >>>>
> >>> >> > >>>> 2. Features close to finish:
> >>> >> > >>>>
> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> >> Merge/Vote
> >>> >> > >>>> Ongoing.
> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> apps
> >>> from
> >>> >> > >>>> ATSv2. Patch in progress.
> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> Improvement
> >>> >> > works.
> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
> Azure
> >>> >> > >>>> Storage. In progress.
> >>> >> > >>>>
> >>> >> > >>>> 3. Tentative features:
> >>> >> > >>>>
> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging
> >>> >> > to
> >>> >> > >>>> be done before Aug 2018.
> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
> applications.
> >>> >> > >>>> Challenging as more discussions are on-going.
> >>> >> > >>>>
> >>> >> > >>>> *Summary of 3.2.0 issues status:*
> >>> >> > >>>>
> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following
> up
> >>> with
> >>> >> > >>>> owners to get status on each of them to get in by Code Freeze
> >>> date.
> >>> >> > >>>>
> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
> in
> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> >>> >> > Version/s" =
> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
> >>> >> > >>>>
> >>> >> > >>>> Thanks,
> >>> >> > >>>> Sunil
> >>> >> > >>>>
> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <sunilg@apache.org
> <ma...@apache.org>>
> >>> >> wrote:
> >>> >> > >>>>
> >>> >> > >>>>> Hi All,
> >>> >> > >>>>>
> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
> >>> dated
> >>> >> > 17th
> >>> >> > >>>>> July 2018, I would like to provide some updates.
> >>> >> > >>>>>
> >>> >> > >>>>> We are approaching previously proposed feature freeze date
> >>> (Aug
> >>> >> 21,
> >>> >> > >>>>> about 7 days from today).
> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
> please
> >>> >> reply
> >>> >> > to
> >>> >> > >>>>> this email thread.
> >>> >> > >>>>> Steve has mentioned about the s3 features which will come
> >>> close to
> >>> >> > >>>>> Code Freeze Date (Aug 31st).
> >>> >> > >>>>>
> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
> >>> >> > >>>>>
> >>> >> > >>>>> 1. Merged & Completed features:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> >>> DeepLearning
> >>> >> > >>>>> workloads Initial cut.
> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>>>>
> >>> >> > >>>>> 2. Features close to finish:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> Major
> >>> >> > >>>>> patches are all in, only one last
> >>> >> > >>>>> patch is in review state.
> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> Capacity
> >>> >> > >>>>> Scheduler. Close to commit.
> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> >> Service
> >>> >> > >>>>> API and CLI. 2 patches are pending
> >>> >> > >>>>> which will be closed by Feature freeze date.
> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> apps
> >>> >> from
> >>> >> > >>>>> ATSv2. Patch in progress.
> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> Improvement
> >>> >> > >>>>> works.
> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
> >>> Azure
> >>> >> > >>>>> Storage. In progress.
> >>> >> > >>>>>
> >>> >> > >>>>> 3. Tentative features:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging
> >>> >> > >>>>> to be done before Aug 2018.
> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
> applications.
> >>> >> > >>>>> Challenging as more discussions are on-going.
> >>> >> > >>>>>
> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
> >>> >> > >>>>>
> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
> >>> with
> >>> >> > owners
> >>> >> > >>>>> to get status on each of them to get in by Code Freeze date.
> >>> >> > >>>>>
> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
> in
> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> >>> >> > Version/s" =
> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
> >>> >> > >>>>>
> >>> >> > >>>>> Thanks,
> >>> >> > >>>>> Sunil
> >>> >> > >>>>>
> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <sunilg@apache.org
> <ma...@apache.org>>
> >>> >> wrote:
> >>> >> > >>>>>
> >>> >> > >>>>>> Thanks Subru for the thoughts.
> >>> >> > >>>>>> One of the main reason for a major release is to push out
> >>> >> critical
> >>> >> > >>>>>> features with a faster cadence to the users. If we are
> >>> pulling
> >>> >> more
> >>> >> > and
> >>> >> > >>>>>> more different types of features to a minor release, that
> >>> branch
> >>> >> > will
> >>> >> > >>>>>> become more destabilized and it may be tough to say that
> >>> 3.1.2 is
> >>> >> > stable
> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
> >>> >> features
> >>> >> > in
> >>> >> > >>>>>> subsequent minor release.
> >>> >> > >>>>>> For few companies, it makes sense to push out these new
> >>> features
> >>> >> > >>>>>> faster to make a reach to the users. Adding to the point to
> >>> the
> >>> >> > backporting
> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
> >>> with
> >>> >> > some git
> >>> >> > >>>>>> scripts. If we can make such scripts available to
> committers,
> >>> >> > backport will
> >>> >> > >>>>>> be seem-less across branches and we can achieve the faster
> >>> >> release
> >>> >> > cadence
> >>> >> > >>>>>> also.
> >>> >> > >>>>>>
> >>> >> > >>>>>> Thoughts?
> >>> >> > >>>>>>
> >>> >> > >>>>>> - Sunil
> >>> >> > >>>>>>
> >>> >> > >>>>>>
> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
> >>> subru@apache.org<ma...@apache.org>
> >>> >> >
> >>> >> > >>>>>> wrote:
> >>> >> > >>>>>>
> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release effort.
> I
> >>> am
> >>> >> > >>>>>>> generally
> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x)
> as
> >>> >> feel we
> >>> >> > >>>>>>> already
> >>> >> > >>>>>>> have too many branches to be maintained. I already see
> many
> >>> >> commits
> >>> >> > >>>>>>> are in
> >>> >> > >>>>>>> different branches with no apparent rationale, for e.g:
> 3.1
> >>> has
> >>> >> > >>>>>>> commits
> >>> >> > >>>>>>> which are absent in 3.0 etc.
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
> >>> >> > production
> >>> >> > >>>>>>> setting so the cost of adding features should be minimal.
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> Thoughts?
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> -Subru
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
> >>> sunilg@apache.org<ma...@apache.org>>
> >>> >> > wrote:
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Yes, important changes and features are much needed,
> >>> hence we
> >>> >> > will
> >>> >> > >>>>>>> be
> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
> >>> considering
> >>> >> few
> >>> >> > >>>>>>> more
> >>> >> > >>>>>>> > offline requests from other folks, I think extending the
> >>> >> > timeframe
> >>> >> > >>>>>>> by
> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
> >>> buffer) and
> >>> >> > >>>>>>> this should
> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki
> as
> >>> >> well)
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by August
> >>> 21,
> >>> >> 2018.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
> >>> improvements
> >>> >> and
> >>> >> > >>>>>>> non
> >>> >> > >>>>>>> > blocker/critical
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Release date: September 15, 2018
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered
> >>> your
> >>> >> > >>>>>>> questions.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Thanks
> >>> >> > >>>>>>> > Sunil
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
> >>> >> wheeleast@gmail.com<ma...@gmail.com>
> >>> >> > >
> >>> >> > >>>>>>> wrote:
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release,
> >>> +1
> >>> >> for
> >>> >> > >>>>>>> that.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > To concerns from Steve,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
> important
> >>> >> > changes
> >>> >> > >>>>>>> /
> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
> >>> >> proposed
> >>> >> > >>>>>>> release
> >>> >> > >>>>>>> > date
> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of last
> >>> >> minute
> >>> >> > >>>>>>> and we all
> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm also
> >>> fine
> >>> >> if
> >>> >> > >>>>>>> we want
> >>> >> > >>>>>>> > get
> >>> >> > >>>>>>> > > another several weeks time.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
> >>> before
> >>> >> > >>>>>>> thanksgiving.
> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Eric,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
> >>> YARN-8135
> >>> >> can
> >>> >> > >>>>>>> get merged
> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Sunil,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Could u update
> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
> >>> >> > >>>>>>> > Roadmap
> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
> >>> first
> >>> >> > >>>>>>> before getting
> >>> >> > >>>>>>> > > consensus of time.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Thanks,
> >>> >> > >>>>>>> > > Wangda
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid<mailto:
> fabbri@cloudera.com.invalid>
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > wrote:
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> >>> >> > >>>>>>> stevel@hortonworks.com<ma...@hortonworks.com>>
> >>> >> > >>>>>>> > >> wrote:
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
> sunilg@apache.org<ma...@apache.org>
> >>> >> > <mailto:
> >>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>>
> wrote:
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I would also would like to take this opportunity to
> >>> come
> >>> >> up
> >>> >> > >>>>>>> with a
> >>> >> > >>>>>>> > >> detailed
> >>> >> > >>>>>>> > >> > plan.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
> >>> merged by
> >>> >> > >>>>>>> August 10,
> >>> >> > >>>>>>> > >> 2018.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > <snip>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
> targeted
> >>> to
> >>> >> 3.2
> >>> >> > >>>>>>> per this
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
> >>> Datalake
> >>> >> > Gen
> >>> >> > >>>>>>> 2
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
> >>> doing
> >>> >> work
> >>> >> > >>>>>>> on it, but
> >>> >> > >>>>>>> > >> if
> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
> >>> >> timely
> >>> >> > >>>>>>> reviews on
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will review
> >>> the
> >>> >> S3 &
> >>> >> > >>>>>>> S3Guard
> >>> >> > >>>>>>> > >> JIRAs
> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
> >>> some
> >>> >> > >>>>>>> pressing
> >>> >> > >>>>>>> > "guave,
> >>> >> > >>>>>>> > >> > java 9 prep"
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> I can help with this part if you like.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
> release
> >>> >> > manager
> >>> >> > >>>>>>> of 3.2.0
> >>> >> > >>>>>>> > >> > release.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > well volunteered!
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable should
> >>> we
> >>> >> have
> >>> >> > >>>>>>> for the
> >>> >> > >>>>>>> > >> 3.2. &
> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
> >>> having
> >>> >> > the
> >>> >> > >>>>>>> outline
> >>> >> > >>>>>>> > >> time
> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
> >>> less
> >>> >> > >>>>>>> concern about
> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > -Steve
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> -AF
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>>
> >>> >> > >>>>>>
> >>> >> >
> >>> >>
> >>> >
> >>>
> >>
> >>
>
>
> --
> A very happy Clouderan
>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Steve Loughran <st...@hortonworks.com>.
its in.
good catch!
On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <we...@cloudera.com>> wrote:
Thanks Sunil G for driving the release,
I filed HADOOP-15866<https://issues.apache.org/jira/browse/HADOOP-15866> for a compat fix. If any one has cycle please review it, as I think it is needed for 3.2.0.
On Thu, Oct 18, 2018 at 4:43 AM Sunil G <su...@apache.org>> wrote:
Hi Folks,
As we previously communicated for 3.2.0 release, we have delayed due to few
blockers in our gate.
I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
all bug fixes.
- Sunil
On Tue, Oct 16, 2018 at 8:59 AM Sunil G <su...@apache.org>> wrote:
> Hi Folks,
>
> We are now close to RC as other blocker issues are now merged to trunk and
> branch-3.2. Last 2 critical issues are closer to merge and will be
> committed in few hours.
> With this, I will be creating 3.2.0 branch today and will go ahead with RC
> related process.
>
> - Sunil
>
> On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jo...@stripe.com>>
> wrote:
>
>> Hello, were there any updates around the 3.2.0 RC timing? All I see in
>> the current blockers are related to the new Submarine subproject, wasn't
>> sure if that is what is holding things up.
>>
>> Cheers,
>> Jon
>>
>> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <su...@apache.org>> wrote:
>>
>>> Thanks Robert and Haibo for quickly correcting same.
>>> Sigh, I somehow missed one file while committing the change. Sorry for
>>> the
>>> trouble.
>>>
>>> - Sunil
>>>
>>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rk...@cloudera.com>>
>>> wrote:
>>>
>>> > Looks like there's two that weren't updated:
>>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
>>> > --include=pom.xml
>>> > ./hadoop-project/pom.xml:
>>> > <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>>> >
>>> > I've just pushed in an addendum commit to fix those.
>>> > In the future, please make sure to do a sanity compile when updating
>>> poms.
>>> >
>>> > thanks
>>> > - Robert
>>> >
>>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>>> <fa...@cloudera.com.invalid>>
>>> > wrote:
>>> >
>>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
>>> >> top-level pom.xml?
>>> >>
>>> >>
>>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org>> wrote:
>>> >>
>>> >> > Hi All
>>> >> >
>>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset trunk
>>> to
>>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
>>> necessary
>>> >> > patches are pulled into branch-3.2.
>>> >> >
>>> >> > Thank You
>>> >> > - Sunil
>>> >> >
>>> >> >
>>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org>> wrote:
>>> >> >
>>> >> > > Hi All
>>> >> > >
>>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged to
>>> >> trunk.
>>> >> > > Thanks for the support.
>>> >> > >
>>> >> > > *Plan for RC*
>>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>>> >> > > RC0 for 3.2: *28th Friday*
>>> >> > >
>>> >> > > Thank You
>>> >> > > Sunil
>>> >> > >
>>> >> > >
>>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org>>
>>> wrote:
>>> >> > >
>>> >> > >> Hi All
>>> >> > >>
>>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you
>>> for
>>> >> > >> helping in this. I am following up on these tickets, once its
>>> closed
>>> >> we
>>> >> > >> will cut the 3.2 branch.
>>> >> > >>
>>> >> > >> Thanks
>>> >> > >> Sunil Govindan
>>> >> > >>
>>> >> > >>
>>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org>>
>>> wrote:
>>> >> > >>
>>> >> > >>> Hi All,
>>> >> > >>>
>>> >> > >>> Inline with the original 3.2 communication proposal dated 17th
>>> July
>>> >> > >>> 2018, I would like to provide more updates.
>>> >> > >>>
>>> >> > >>> We are approaching previously proposed code freeze date
>>> (September
>>> >> 14,
>>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
>>> >> > existing
>>> >> > >>> trunk to 3.3 if there are no issues.
>>> >> > >>>
>>> >> > >>> *Current Release Plan:*
>>> >> > >>> Feature freeze date : all features to merge by September 7,
>>> 2018.
>>> >> > >>> Code freeze date : blockers/critical only, no improvements and
>>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>>> >> > >>> Release date: September 28, 2018
>>> >> > >>>
>>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0, we
>>> >> need to
>>> >> > >>> backport to 3.2 post branch cut.
>>> >> > >>>
>>> >> > >>> Here's an updated 3.2.0 feature status:
>>> >> > >>>
>>> >> > >>> 1. Merged & Completed features:
>>> >> > >>>
>>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> >> > >>> workloads Initial cut.
>>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>> Scheduler.
>>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> Service
>>> >> > API
>>> >> > >>> and CLI.
>>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement
>>> >> works.
>>> >> > >>>
>>> >> > >>> 2. Features close to finish:
>>> >> > >>>
>>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>>> >> > >>> - (Steve) S3a phase V. Close to commit.
>>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>>> >> > >>>
>>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> from
>>> >> > >>> ATSv2. Patch in progress.
>>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging to
>>> >> > >>> be done before Aug 2018.
>>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>> Challenging as more discussions are on-going.
>>> >> > >>>
>>> >> > >>> *Summary of 3.2.0 issues status:*
>>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following up
>>> with
>>> >> > >>> owners to get status on each of them to get in by Code Freeze
>>> date.
>>> >> > >>>
>>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > (Blocker,
>>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" =
>>> 3.2.0
>>> >> > ORDER
>>> >> > >>> BY priority DESC
>>> >> > >>>
>>> >> > >>> Thanks,
>>> >> > >>> Sunil
>>> >> > >>>
>>> >> > >>>
>>> >> > >>>
>>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org>>
>>> wrote:
>>> >> > >>>
>>> >> > >>>> Hi All,
>>> >> > >>>>
>>> >> > >>>> Inline with earlier communication dated 17th July 2018, I would
>>> >> like
>>> >> > to
>>> >> > >>>> provide some updates.
>>> >> > >>>>
>>> >> > >>>> We are approaching previously proposed code freeze date (Aug
>>> 31).
>>> >> > >>>>
>>> >> > >>>> One of the critical feature Node Attributes feature merge
>>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need a
>>> bit
>>> >> > more
>>> >> > >>>> time. With regard to this, suggesting to push the feature/code
>>> >> freeze
>>> >> > for 2
>>> >> > >>>> more weeks to accommodate these jiras too.
>>> >> > >>>>
>>> >> > >>>> Proposing Updated changes in plan inline with this:
>>> >> > >>>> Feature freeze date : all features to merge by September 7,
>>> 2018.
>>> >> > >>>> Code freeze date : blockers/critical only, no improvements and
>>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>>> >> > >>>> Release date: September 28, 2018
>>> >> > >>>>
>>> >> > >>>> If any features in branch which are targeted to 3.2.0, please
>>> >> reply to
>>> >> > >>>> this email thread.
>>> >> > >>>>
>>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>>> >> > >>>>
>>> >> > >>>> 1. Merged & Completed features:
>>> >> > >>>>
>>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> >> > >>>> workloads Initial cut.
>>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>>> Scheduler.
>>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> >> Service
>>> >> > >>>> API and CLI.
>>> >> > >>>>
>>> >> > >>>> 2. Features close to finish:
>>> >> > >>>>
>>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >> Merge/Vote
>>> >> > >>>> Ongoing.
>>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> from
>>> >> > >>>> ATSv2. Patch in progress.
>>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> Improvement
>>> >> > works.
>>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>>> >> > >>>> Storage. In progress.
>>> >> > >>>>
>>> >> > >>>> 3. Tentative features:
>>> >> > >>>>
>>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging
>>> >> > to
>>> >> > >>>> be done before Aug 2018.
>>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>>> Challenging as more discussions are on-going.
>>> >> > >>>>
>>> >> > >>>> *Summary of 3.2.0 issues status:*
>>> >> > >>>>
>>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following up
>>> with
>>> >> > >>>> owners to get status on each of them to get in by Code Freeze
>>> date.
>>> >> > >>>>
>>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>>> >> > Version/s" =
>>> >> > >>>> 3.2.0 ORDER BY priority DESC
>>> >> > >>>>
>>> >> > >>>> Thanks,
>>> >> > >>>> Sunil
>>> >> > >>>>
>>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org>>
>>> >> wrote:
>>> >> > >>>>
>>> >> > >>>>> Hi All,
>>> >> > >>>>>
>>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
>>> dated
>>> >> > 17th
>>> >> > >>>>> July 2018, I would like to provide some updates.
>>> >> > >>>>>
>>> >> > >>>>> We are approaching previously proposed feature freeze date
>>> (Aug
>>> >> 21,
>>> >> > >>>>> about 7 days from today).
>>> >> > >>>>> If any features in branch which are targeted to 3.2.0, please
>>> >> reply
>>> >> > to
>>> >> > >>>>> this email thread.
>>> >> > >>>>> Steve has mentioned about the s3 features which will come
>>> close to
>>> >> > >>>>> Code Freeze Date (Aug 31st).
>>> >> > >>>>>
>>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>>> >> > >>>>>
>>> >> > >>>>> 1. Merged & Completed features:
>>> >> > >>>>>
>>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>>> DeepLearning
>>> >> > >>>>> workloads Initial cut.
>>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>>>>
>>> >> > >>>>> 2. Features close to finish:
>>> >> > >>>>>
>>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> Major
>>> >> > >>>>> patches are all in, only one last
>>> >> > >>>>> patch is in review state.
>>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>>>> Scheduler. Close to commit.
>>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> >> Service
>>> >> > >>>>> API and CLI. 2 patches are pending
>>> >> > >>>>> which will be closed by Feature freeze date.
>>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> >> from
>>> >> > >>>>> ATSv2. Patch in progress.
>>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> Improvement
>>> >> > >>>>> works.
>>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>>> Azure
>>> >> > >>>>> Storage. In progress.
>>> >> > >>>>>
>>> >> > >>>>> 3. Tentative features:
>>> >> > >>>>>
>>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging
>>> >> > >>>>> to be done before Aug 2018.
>>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>>>> Challenging as more discussions are on-going.
>>> >> > >>>>>
>>> >> > >>>>> *Summary of 3.2.0 issues status:*
>>> >> > >>>>>
>>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
>>> with
>>> >> > owners
>>> >> > >>>>> to get status on each of them to get in by Code Freeze date.
>>> >> > >>>>>
>>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>>> >> > Version/s" =
>>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>>> >> > >>>>>
>>> >> > >>>>> Thanks,
>>> >> > >>>>> Sunil
>>> >> > >>>>>
>>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org>>
>>> >> wrote:
>>> >> > >>>>>
>>> >> > >>>>>> Thanks Subru for the thoughts.
>>> >> > >>>>>> One of the main reason for a major release is to push out
>>> >> critical
>>> >> > >>>>>> features with a faster cadence to the users. If we are
>>> pulling
>>> >> more
>>> >> > and
>>> >> > >>>>>> more different types of features to a minor release, that
>>> branch
>>> >> > will
>>> >> > >>>>>> become more destabilized and it may be tough to say that
>>> 3.1.2 is
>>> >> > stable
>>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
>>> >> features
>>> >> > in
>>> >> > >>>>>> subsequent minor release.
>>> >> > >>>>>> For few companies, it makes sense to push out these new
>>> features
>>> >> > >>>>>> faster to make a reach to the users. Adding to the point to
>>> the
>>> >> > backporting
>>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
>>> with
>>> >> > some git
>>> >> > >>>>>> scripts. If we can make such scripts available to committers,
>>> >> > backport will
>>> >> > >>>>>> be seem-less across branches and we can achieve the faster
>>> >> release
>>> >> > cadence
>>> >> > >>>>>> also.
>>> >> > >>>>>>
>>> >> > >>>>>> Thoughts?
>>> >> > >>>>>>
>>> >> > >>>>>> - Sunil
>>> >> > >>>>>>
>>> >> > >>>>>>
>>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>>> subru@apache.org<ma...@apache.org>
>>> >> >
>>> >> > >>>>>> wrote:
>>> >> > >>>>>>
>>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release effort. I
>>> am
>>> >> > >>>>>>> generally
>>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as
>>> >> feel we
>>> >> > >>>>>>> already
>>> >> > >>>>>>> have too many branches to be maintained. I already see many
>>> >> commits
>>> >> > >>>>>>> are in
>>> >> > >>>>>>> different branches with no apparent rationale, for e.g: 3.1
>>> has
>>> >> > >>>>>>> commits
>>> >> > >>>>>>> which are absent in 3.0 etc.
>>> >> > >>>>>>>
>>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
>>> >> > production
>>> >> > >>>>>>> setting so the cost of adding features should be minimal.
>>> >> > >>>>>>>
>>> >> > >>>>>>> Thoughts?
>>> >> > >>>>>>>
>>> >> > >>>>>>> -Subru
>>> >> > >>>>>>>
>>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
>>> sunilg@apache.org<ma...@apache.org>>
>>> >> > wrote:
>>> >> > >>>>>>>
>>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Yes, important changes and features are much needed,
>>> hence we
>>> >> > will
>>> >> > >>>>>>> be
>>> >> > >>>>>>> > keeping the door open for them as possible. Also
>>> considering
>>> >> few
>>> >> > >>>>>>> more
>>> >> > >>>>>>> > offline requests from other folks, I think extending the
>>> >> > timeframe
>>> >> > >>>>>>> by
>>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
>>> buffer) and
>>> >> > >>>>>>> this should
>>> >> > >>>>>>> > ideally help us to ship this by September itself.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as
>>> >> well)
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Feature freeze date : all features to merge by August
>>> 21,
>>> >> 2018.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>>> improvements
>>> >> and
>>> >> > >>>>>>> non
>>> >> > >>>>>>> > blocker/critical
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > bug-fixes August 31, 2018.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Release date: September 15, 2018
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered
>>> your
>>> >> > >>>>>>> questions.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Thanks
>>> >> > >>>>>>> > Sunil
>>> >> > >>>>>>> >
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>>> >> wheeleast@gmail.com<ma...@gmail.com>
>>> >> > >
>>> >> > >>>>>>> wrote:
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release,
>>> +1
>>> >> for
>>> >> > >>>>>>> that.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > To concerns from Steve,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > It is a good idea to keep the door open to get important
>>> >> > changes
>>> >> > >>>>>>> /
>>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
>>> >> proposed
>>> >> > >>>>>>> release
>>> >> > >>>>>>> > date
>>> >> > >>>>>>> > > to make sure things can happen earlier instead of last
>>> >> minute
>>> >> > >>>>>>> and we all
>>> >> > >>>>>>> > > know that releases are always get delayed :). I'm also
>>> fine
>>> >> if
>>> >> > >>>>>>> we want
>>> >> > >>>>>>> > get
>>> >> > >>>>>>> > > another several weeks time.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
>>> before
>>> >> > >>>>>>> thanksgiving.
>>> >> > >>>>>>> > > Do you think is it good or too early / late?
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Eric,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
>>> YARN-8135
>>> >> can
>>> >> > >>>>>>> get merged
>>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Sunil,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Could u update
>>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>>> >> > >>>>>>> > Roadmap
>>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
>>> first
>>> >> > >>>>>>> before getting
>>> >> > >>>>>>> > > consensus of time.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Thanks,
>>> >> > >>>>>>> > > Wangda
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>> >> > >>>>>>> <fa...@cloudera.com.invalid>
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > wrote:
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>> >> > >>>>>>> stevel@hortonworks.com<ma...@hortonworks.com>>
>>> >> > >>>>>>> > >> wrote:
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <su...@apache.org>
>>> >> > <mailto:
>>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>> wrote:
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I would also would like to take this opportunity to
>>> come
>>> >> up
>>> >> > >>>>>>> with a
>>> >> > >>>>>>> > >> detailed
>>> >> > >>>>>>> > >> > plan.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
>>> merged by
>>> >> > >>>>>>> August 10,
>>> >> > >>>>>>> > >> 2018.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > <snip>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > Please let me know if I missed any features targeted
>>> to
>>> >> 3.2
>>> >> > >>>>>>> per this
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
>>> Datalake
>>> >> > Gen
>>> >> > >>>>>>> 2
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
>>> doing
>>> >> work
>>> >> > >>>>>>> on it, but
>>> >> > >>>>>>> > >> if
>>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
>>> >> timely
>>> >> > >>>>>>> reviews on
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will review
>>> the
>>> >> S3 &
>>> >> > >>>>>>> S3Guard
>>> >> > >>>>>>> > >> JIRAs
>>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
>>> some
>>> >> > >>>>>>> pressing
>>> >> > >>>>>>> > "guave,
>>> >> > >>>>>>> > >> > java 9 prep"
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> I can help with this part if you like.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as release
>>> >> > manager
>>> >> > >>>>>>> of 3.2.0
>>> >> > >>>>>>> > >> > release.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > well volunteered!
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I think this raises a good q: what timetable should
>>> we
>>> >> have
>>> >> > >>>>>>> for the
>>> >> > >>>>>>> > >> 3.2. &
>>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
>>> having
>>> >> > the
>>> >> > >>>>>>> outline
>>> >> > >>>>>>> > >> time
>>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
>>> less
>>> >> > >>>>>>> concern about
>>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > -Steve
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> -AF
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> >
>>> >> > >>>>>>>
>>> >> > >>>>>>
>>> >> >
>>> >>
>>> >
>>>
>>
>>
--
A very happy Clouderan
Re: Hadoop 3.2 Release Plan proposal
Posted by Steve Loughran <st...@hortonworks.com>.
its in.
good catch!
On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <we...@cloudera.com>> wrote:
Thanks Sunil G for driving the release,
I filed HADOOP-15866<https://issues.apache.org/jira/browse/HADOOP-15866> for a compat fix. If any one has cycle please review it, as I think it is needed for 3.2.0.
On Thu, Oct 18, 2018 at 4:43 AM Sunil G <su...@apache.org>> wrote:
Hi Folks,
As we previously communicated for 3.2.0 release, we have delayed due to few
blockers in our gate.
I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
all bug fixes.
- Sunil
On Tue, Oct 16, 2018 at 8:59 AM Sunil G <su...@apache.org>> wrote:
> Hi Folks,
>
> We are now close to RC as other blocker issues are now merged to trunk and
> branch-3.2. Last 2 critical issues are closer to merge and will be
> committed in few hours.
> With this, I will be creating 3.2.0 branch today and will go ahead with RC
> related process.
>
> - Sunil
>
> On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jo...@stripe.com>>
> wrote:
>
>> Hello, were there any updates around the 3.2.0 RC timing? All I see in
>> the current blockers are related to the new Submarine subproject, wasn't
>> sure if that is what is holding things up.
>>
>> Cheers,
>> Jon
>>
>> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <su...@apache.org>> wrote:
>>
>>> Thanks Robert and Haibo for quickly correcting same.
>>> Sigh, I somehow missed one file while committing the change. Sorry for
>>> the
>>> trouble.
>>>
>>> - Sunil
>>>
>>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rk...@cloudera.com>>
>>> wrote:
>>>
>>> > Looks like there's two that weren't updated:
>>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
>>> > --include=pom.xml
>>> > ./hadoop-project/pom.xml:
>>> > <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>>> >
>>> > I've just pushed in an addendum commit to fix those.
>>> > In the future, please make sure to do a sanity compile when updating
>>> poms.
>>> >
>>> > thanks
>>> > - Robert
>>> >
>>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>>> <fa...@cloudera.com.invalid>>
>>> > wrote:
>>> >
>>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
>>> >> top-level pom.xml?
>>> >>
>>> >>
>>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org>> wrote:
>>> >>
>>> >> > Hi All
>>> >> >
>>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset trunk
>>> to
>>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
>>> necessary
>>> >> > patches are pulled into branch-3.2.
>>> >> >
>>> >> > Thank You
>>> >> > - Sunil
>>> >> >
>>> >> >
>>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org>> wrote:
>>> >> >
>>> >> > > Hi All
>>> >> > >
>>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged to
>>> >> trunk.
>>> >> > > Thanks for the support.
>>> >> > >
>>> >> > > *Plan for RC*
>>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>>> >> > > RC0 for 3.2: *28th Friday*
>>> >> > >
>>> >> > > Thank You
>>> >> > > Sunil
>>> >> > >
>>> >> > >
>>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org>>
>>> wrote:
>>> >> > >
>>> >> > >> Hi All
>>> >> > >>
>>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you
>>> for
>>> >> > >> helping in this. I am following up on these tickets, once its
>>> closed
>>> >> we
>>> >> > >> will cut the 3.2 branch.
>>> >> > >>
>>> >> > >> Thanks
>>> >> > >> Sunil Govindan
>>> >> > >>
>>> >> > >>
>>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org>>
>>> wrote:
>>> >> > >>
>>> >> > >>> Hi All,
>>> >> > >>>
>>> >> > >>> Inline with the original 3.2 communication proposal dated 17th
>>> July
>>> >> > >>> 2018, I would like to provide more updates.
>>> >> > >>>
>>> >> > >>> We are approaching previously proposed code freeze date
>>> (September
>>> >> 14,
>>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
>>> >> > existing
>>> >> > >>> trunk to 3.3 if there are no issues.
>>> >> > >>>
>>> >> > >>> *Current Release Plan:*
>>> >> > >>> Feature freeze date : all features to merge by September 7,
>>> 2018.
>>> >> > >>> Code freeze date : blockers/critical only, no improvements and
>>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>>> >> > >>> Release date: September 28, 2018
>>> >> > >>>
>>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0, we
>>> >> need to
>>> >> > >>> backport to 3.2 post branch cut.
>>> >> > >>>
>>> >> > >>> Here's an updated 3.2.0 feature status:
>>> >> > >>>
>>> >> > >>> 1. Merged & Completed features:
>>> >> > >>>
>>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> >> > >>> workloads Initial cut.
>>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>> Scheduler.
>>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> Service
>>> >> > API
>>> >> > >>> and CLI.
>>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement
>>> >> works.
>>> >> > >>>
>>> >> > >>> 2. Features close to finish:
>>> >> > >>>
>>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>>> >> > >>> - (Steve) S3a phase V. Close to commit.
>>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>>> >> > >>>
>>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> from
>>> >> > >>> ATSv2. Patch in progress.
>>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging to
>>> >> > >>> be done before Aug 2018.
>>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>> Challenging as more discussions are on-going.
>>> >> > >>>
>>> >> > >>> *Summary of 3.2.0 issues status:*
>>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following up
>>> with
>>> >> > >>> owners to get status on each of them to get in by Code Freeze
>>> date.
>>> >> > >>>
>>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > (Blocker,
>>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" =
>>> 3.2.0
>>> >> > ORDER
>>> >> > >>> BY priority DESC
>>> >> > >>>
>>> >> > >>> Thanks,
>>> >> > >>> Sunil
>>> >> > >>>
>>> >> > >>>
>>> >> > >>>
>>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org>>
>>> wrote:
>>> >> > >>>
>>> >> > >>>> Hi All,
>>> >> > >>>>
>>> >> > >>>> Inline with earlier communication dated 17th July 2018, I would
>>> >> like
>>> >> > to
>>> >> > >>>> provide some updates.
>>> >> > >>>>
>>> >> > >>>> We are approaching previously proposed code freeze date (Aug
>>> 31).
>>> >> > >>>>
>>> >> > >>>> One of the critical feature Node Attributes feature merge
>>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need a
>>> bit
>>> >> > more
>>> >> > >>>> time. With regard to this, suggesting to push the feature/code
>>> >> freeze
>>> >> > for 2
>>> >> > >>>> more weeks to accommodate these jiras too.
>>> >> > >>>>
>>> >> > >>>> Proposing Updated changes in plan inline with this:
>>> >> > >>>> Feature freeze date : all features to merge by September 7,
>>> 2018.
>>> >> > >>>> Code freeze date : blockers/critical only, no improvements and
>>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>>> >> > >>>> Release date: September 28, 2018
>>> >> > >>>>
>>> >> > >>>> If any features in branch which are targeted to 3.2.0, please
>>> >> reply to
>>> >> > >>>> this email thread.
>>> >> > >>>>
>>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>>> >> > >>>>
>>> >> > >>>> 1. Merged & Completed features:
>>> >> > >>>>
>>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> >> > >>>> workloads Initial cut.
>>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>>> Scheduler.
>>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> >> Service
>>> >> > >>>> API and CLI.
>>> >> > >>>>
>>> >> > >>>> 2. Features close to finish:
>>> >> > >>>>
>>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >> Merge/Vote
>>> >> > >>>> Ongoing.
>>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> from
>>> >> > >>>> ATSv2. Patch in progress.
>>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> Improvement
>>> >> > works.
>>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>>> >> > >>>> Storage. In progress.
>>> >> > >>>>
>>> >> > >>>> 3. Tentative features:
>>> >> > >>>>
>>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging
>>> >> > to
>>> >> > >>>> be done before Aug 2018.
>>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>>> Challenging as more discussions are on-going.
>>> >> > >>>>
>>> >> > >>>> *Summary of 3.2.0 issues status:*
>>> >> > >>>>
>>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following up
>>> with
>>> >> > >>>> owners to get status on each of them to get in by Code Freeze
>>> date.
>>> >> > >>>>
>>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>>> >> > Version/s" =
>>> >> > >>>> 3.2.0 ORDER BY priority DESC
>>> >> > >>>>
>>> >> > >>>> Thanks,
>>> >> > >>>> Sunil
>>> >> > >>>>
>>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org>>
>>> >> wrote:
>>> >> > >>>>
>>> >> > >>>>> Hi All,
>>> >> > >>>>>
>>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
>>> dated
>>> >> > 17th
>>> >> > >>>>> July 2018, I would like to provide some updates.
>>> >> > >>>>>
>>> >> > >>>>> We are approaching previously proposed feature freeze date
>>> (Aug
>>> >> 21,
>>> >> > >>>>> about 7 days from today).
>>> >> > >>>>> If any features in branch which are targeted to 3.2.0, please
>>> >> reply
>>> >> > to
>>> >> > >>>>> this email thread.
>>> >> > >>>>> Steve has mentioned about the s3 features which will come
>>> close to
>>> >> > >>>>> Code Freeze Date (Aug 31st).
>>> >> > >>>>>
>>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>>> >> > >>>>>
>>> >> > >>>>> 1. Merged & Completed features:
>>> >> > >>>>>
>>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>>> DeepLearning
>>> >> > >>>>> workloads Initial cut.
>>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>>>>
>>> >> > >>>>> 2. Features close to finish:
>>> >> > >>>>>
>>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> Major
>>> >> > >>>>> patches are all in, only one last
>>> >> > >>>>> patch is in review state.
>>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>>>> Scheduler. Close to commit.
>>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> >> Service
>>> >> > >>>>> API and CLI. 2 patches are pending
>>> >> > >>>>> which will be closed by Feature freeze date.
>>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> >> from
>>> >> > >>>>> ATSv2. Patch in progress.
>>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> Improvement
>>> >> > >>>>> works.
>>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>>> Azure
>>> >> > >>>>> Storage. In progress.
>>> >> > >>>>>
>>> >> > >>>>> 3. Tentative features:
>>> >> > >>>>>
>>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging
>>> >> > >>>>> to be done before Aug 2018.
>>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>>>> Challenging as more discussions are on-going.
>>> >> > >>>>>
>>> >> > >>>>> *Summary of 3.2.0 issues status:*
>>> >> > >>>>>
>>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
>>> with
>>> >> > owners
>>> >> > >>>>> to get status on each of them to get in by Code Freeze date.
>>> >> > >>>>>
>>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>>> >> > Version/s" =
>>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>>> >> > >>>>>
>>> >> > >>>>> Thanks,
>>> >> > >>>>> Sunil
>>> >> > >>>>>
>>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org>>
>>> >> wrote:
>>> >> > >>>>>
>>> >> > >>>>>> Thanks Subru for the thoughts.
>>> >> > >>>>>> One of the main reason for a major release is to push out
>>> >> critical
>>> >> > >>>>>> features with a faster cadence to the users. If we are
>>> pulling
>>> >> more
>>> >> > and
>>> >> > >>>>>> more different types of features to a minor release, that
>>> branch
>>> >> > will
>>> >> > >>>>>> become more destabilized and it may be tough to say that
>>> 3.1.2 is
>>> >> > stable
>>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
>>> >> features
>>> >> > in
>>> >> > >>>>>> subsequent minor release.
>>> >> > >>>>>> For few companies, it makes sense to push out these new
>>> features
>>> >> > >>>>>> faster to make a reach to the users. Adding to the point to
>>> the
>>> >> > backporting
>>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
>>> with
>>> >> > some git
>>> >> > >>>>>> scripts. If we can make such scripts available to committers,
>>> >> > backport will
>>> >> > >>>>>> be seem-less across branches and we can achieve the faster
>>> >> release
>>> >> > cadence
>>> >> > >>>>>> also.
>>> >> > >>>>>>
>>> >> > >>>>>> Thoughts?
>>> >> > >>>>>>
>>> >> > >>>>>> - Sunil
>>> >> > >>>>>>
>>> >> > >>>>>>
>>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>>> subru@apache.org<ma...@apache.org>
>>> >> >
>>> >> > >>>>>> wrote:
>>> >> > >>>>>>
>>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release effort. I
>>> am
>>> >> > >>>>>>> generally
>>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as
>>> >> feel we
>>> >> > >>>>>>> already
>>> >> > >>>>>>> have too many branches to be maintained. I already see many
>>> >> commits
>>> >> > >>>>>>> are in
>>> >> > >>>>>>> different branches with no apparent rationale, for e.g: 3.1
>>> has
>>> >> > >>>>>>> commits
>>> >> > >>>>>>> which are absent in 3.0 etc.
>>> >> > >>>>>>>
>>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
>>> >> > production
>>> >> > >>>>>>> setting so the cost of adding features should be minimal.
>>> >> > >>>>>>>
>>> >> > >>>>>>> Thoughts?
>>> >> > >>>>>>>
>>> >> > >>>>>>> -Subru
>>> >> > >>>>>>>
>>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
>>> sunilg@apache.org<ma...@apache.org>>
>>> >> > wrote:
>>> >> > >>>>>>>
>>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Yes, important changes and features are much needed,
>>> hence we
>>> >> > will
>>> >> > >>>>>>> be
>>> >> > >>>>>>> > keeping the door open for them as possible. Also
>>> considering
>>> >> few
>>> >> > >>>>>>> more
>>> >> > >>>>>>> > offline requests from other folks, I think extending the
>>> >> > timeframe
>>> >> > >>>>>>> by
>>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
>>> buffer) and
>>> >> > >>>>>>> this should
>>> >> > >>>>>>> > ideally help us to ship this by September itself.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as
>>> >> well)
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Feature freeze date : all features to merge by August
>>> 21,
>>> >> 2018.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>>> improvements
>>> >> and
>>> >> > >>>>>>> non
>>> >> > >>>>>>> > blocker/critical
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > bug-fixes August 31, 2018.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Release date: September 15, 2018
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered
>>> your
>>> >> > >>>>>>> questions.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Thanks
>>> >> > >>>>>>> > Sunil
>>> >> > >>>>>>> >
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>>> >> wheeleast@gmail.com<ma...@gmail.com>
>>> >> > >
>>> >> > >>>>>>> wrote:
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release,
>>> +1
>>> >> for
>>> >> > >>>>>>> that.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > To concerns from Steve,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > It is a good idea to keep the door open to get important
>>> >> > changes
>>> >> > >>>>>>> /
>>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
>>> >> proposed
>>> >> > >>>>>>> release
>>> >> > >>>>>>> > date
>>> >> > >>>>>>> > > to make sure things can happen earlier instead of last
>>> >> minute
>>> >> > >>>>>>> and we all
>>> >> > >>>>>>> > > know that releases are always get delayed :). I'm also
>>> fine
>>> >> if
>>> >> > >>>>>>> we want
>>> >> > >>>>>>> > get
>>> >> > >>>>>>> > > another several weeks time.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
>>> before
>>> >> > >>>>>>> thanksgiving.
>>> >> > >>>>>>> > > Do you think is it good or too early / late?
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Eric,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
>>> YARN-8135
>>> >> can
>>> >> > >>>>>>> get merged
>>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Sunil,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Could u update
>>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>>> >> > >>>>>>> > Roadmap
>>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
>>> first
>>> >> > >>>>>>> before getting
>>> >> > >>>>>>> > > consensus of time.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Thanks,
>>> >> > >>>>>>> > > Wangda
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>> >> > >>>>>>> <fa...@cloudera.com.invalid>
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > wrote:
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>> >> > >>>>>>> stevel@hortonworks.com<ma...@hortonworks.com>>
>>> >> > >>>>>>> > >> wrote:
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <su...@apache.org>
>>> >> > <mailto:
>>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>> wrote:
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I would also would like to take this opportunity to
>>> come
>>> >> up
>>> >> > >>>>>>> with a
>>> >> > >>>>>>> > >> detailed
>>> >> > >>>>>>> > >> > plan.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
>>> merged by
>>> >> > >>>>>>> August 10,
>>> >> > >>>>>>> > >> 2018.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > <snip>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > Please let me know if I missed any features targeted
>>> to
>>> >> 3.2
>>> >> > >>>>>>> per this
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
>>> Datalake
>>> >> > Gen
>>> >> > >>>>>>> 2
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
>>> doing
>>> >> work
>>> >> > >>>>>>> on it, but
>>> >> > >>>>>>> > >> if
>>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
>>> >> timely
>>> >> > >>>>>>> reviews on
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will review
>>> the
>>> >> S3 &
>>> >> > >>>>>>> S3Guard
>>> >> > >>>>>>> > >> JIRAs
>>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
>>> some
>>> >> > >>>>>>> pressing
>>> >> > >>>>>>> > "guave,
>>> >> > >>>>>>> > >> > java 9 prep"
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> I can help with this part if you like.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as release
>>> >> > manager
>>> >> > >>>>>>> of 3.2.0
>>> >> > >>>>>>> > >> > release.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > well volunteered!
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I think this raises a good q: what timetable should
>>> we
>>> >> have
>>> >> > >>>>>>> for the
>>> >> > >>>>>>> > >> 3.2. &
>>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
>>> having
>>> >> > the
>>> >> > >>>>>>> outline
>>> >> > >>>>>>> > >> time
>>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
>>> less
>>> >> > >>>>>>> concern about
>>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > -Steve
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> -AF
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> >
>>> >> > >>>>>>>
>>> >> > >>>>>>
>>> >> >
>>> >>
>>> >
>>>
>>
>>
--
A very happy Clouderan
Re: Hadoop 3.2 Release Plan proposal
Posted by Steve Loughran <st...@hortonworks.com>.
its in.
good catch!
On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <we...@cloudera.com>> wrote:
Thanks Sunil G for driving the release,
I filed HADOOP-15866<https://issues.apache.org/jira/browse/HADOOP-15866> for a compat fix. If any one has cycle please review it, as I think it is needed for 3.2.0.
On Thu, Oct 18, 2018 at 4:43 AM Sunil G <su...@apache.org>> wrote:
Hi Folks,
As we previously communicated for 3.2.0 release, we have delayed due to few
blockers in our gate.
I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
all bug fixes.
- Sunil
On Tue, Oct 16, 2018 at 8:59 AM Sunil G <su...@apache.org>> wrote:
> Hi Folks,
>
> We are now close to RC as other blocker issues are now merged to trunk and
> branch-3.2. Last 2 critical issues are closer to merge and will be
> committed in few hours.
> With this, I will be creating 3.2.0 branch today and will go ahead with RC
> related process.
>
> - Sunil
>
> On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jo...@stripe.com>>
> wrote:
>
>> Hello, were there any updates around the 3.2.0 RC timing? All I see in
>> the current blockers are related to the new Submarine subproject, wasn't
>> sure if that is what is holding things up.
>>
>> Cheers,
>> Jon
>>
>> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <su...@apache.org>> wrote:
>>
>>> Thanks Robert and Haibo for quickly correcting same.
>>> Sigh, I somehow missed one file while committing the change. Sorry for
>>> the
>>> trouble.
>>>
>>> - Sunil
>>>
>>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rk...@cloudera.com>>
>>> wrote:
>>>
>>> > Looks like there's two that weren't updated:
>>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
>>> > --include=pom.xml
>>> > ./hadoop-project/pom.xml:
>>> > <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>>> >
>>> > I've just pushed in an addendum commit to fix those.
>>> > In the future, please make sure to do a sanity compile when updating
>>> poms.
>>> >
>>> > thanks
>>> > - Robert
>>> >
>>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>>> <fa...@cloudera.com.invalid>>
>>> > wrote:
>>> >
>>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
>>> >> top-level pom.xml?
>>> >>
>>> >>
>>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org>> wrote:
>>> >>
>>> >> > Hi All
>>> >> >
>>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset trunk
>>> to
>>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
>>> necessary
>>> >> > patches are pulled into branch-3.2.
>>> >> >
>>> >> > Thank You
>>> >> > - Sunil
>>> >> >
>>> >> >
>>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org>> wrote:
>>> >> >
>>> >> > > Hi All
>>> >> > >
>>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged to
>>> >> trunk.
>>> >> > > Thanks for the support.
>>> >> > >
>>> >> > > *Plan for RC*
>>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>>> >> > > RC0 for 3.2: *28th Friday*
>>> >> > >
>>> >> > > Thank You
>>> >> > > Sunil
>>> >> > >
>>> >> > >
>>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org>>
>>> wrote:
>>> >> > >
>>> >> > >> Hi All
>>> >> > >>
>>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you
>>> for
>>> >> > >> helping in this. I am following up on these tickets, once its
>>> closed
>>> >> we
>>> >> > >> will cut the 3.2 branch.
>>> >> > >>
>>> >> > >> Thanks
>>> >> > >> Sunil Govindan
>>> >> > >>
>>> >> > >>
>>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org>>
>>> wrote:
>>> >> > >>
>>> >> > >>> Hi All,
>>> >> > >>>
>>> >> > >>> Inline with the original 3.2 communication proposal dated 17th
>>> July
>>> >> > >>> 2018, I would like to provide more updates.
>>> >> > >>>
>>> >> > >>> We are approaching previously proposed code freeze date
>>> (September
>>> >> 14,
>>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
>>> >> > existing
>>> >> > >>> trunk to 3.3 if there are no issues.
>>> >> > >>>
>>> >> > >>> *Current Release Plan:*
>>> >> > >>> Feature freeze date : all features to merge by September 7,
>>> 2018.
>>> >> > >>> Code freeze date : blockers/critical only, no improvements and
>>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>>> >> > >>> Release date: September 28, 2018
>>> >> > >>>
>>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0, we
>>> >> need to
>>> >> > >>> backport to 3.2 post branch cut.
>>> >> > >>>
>>> >> > >>> Here's an updated 3.2.0 feature status:
>>> >> > >>>
>>> >> > >>> 1. Merged & Completed features:
>>> >> > >>>
>>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> >> > >>> workloads Initial cut.
>>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>> Scheduler.
>>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> Service
>>> >> > API
>>> >> > >>> and CLI.
>>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement
>>> >> works.
>>> >> > >>>
>>> >> > >>> 2. Features close to finish:
>>> >> > >>>
>>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>>> >> > >>> - (Steve) S3a phase V. Close to commit.
>>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>>> >> > >>>
>>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> from
>>> >> > >>> ATSv2. Patch in progress.
>>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging to
>>> >> > >>> be done before Aug 2018.
>>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>> Challenging as more discussions are on-going.
>>> >> > >>>
>>> >> > >>> *Summary of 3.2.0 issues status:*
>>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following up
>>> with
>>> >> > >>> owners to get status on each of them to get in by Code Freeze
>>> date.
>>> >> > >>>
>>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > (Blocker,
>>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" =
>>> 3.2.0
>>> >> > ORDER
>>> >> > >>> BY priority DESC
>>> >> > >>>
>>> >> > >>> Thanks,
>>> >> > >>> Sunil
>>> >> > >>>
>>> >> > >>>
>>> >> > >>>
>>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org>>
>>> wrote:
>>> >> > >>>
>>> >> > >>>> Hi All,
>>> >> > >>>>
>>> >> > >>>> Inline with earlier communication dated 17th July 2018, I would
>>> >> like
>>> >> > to
>>> >> > >>>> provide some updates.
>>> >> > >>>>
>>> >> > >>>> We are approaching previously proposed code freeze date (Aug
>>> 31).
>>> >> > >>>>
>>> >> > >>>> One of the critical feature Node Attributes feature merge
>>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need a
>>> bit
>>> >> > more
>>> >> > >>>> time. With regard to this, suggesting to push the feature/code
>>> >> freeze
>>> >> > for 2
>>> >> > >>>> more weeks to accommodate these jiras too.
>>> >> > >>>>
>>> >> > >>>> Proposing Updated changes in plan inline with this:
>>> >> > >>>> Feature freeze date : all features to merge by September 7,
>>> 2018.
>>> >> > >>>> Code freeze date : blockers/critical only, no improvements and
>>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>>> >> > >>>> Release date: September 28, 2018
>>> >> > >>>>
>>> >> > >>>> If any features in branch which are targeted to 3.2.0, please
>>> >> reply to
>>> >> > >>>> this email thread.
>>> >> > >>>>
>>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>>> >> > >>>>
>>> >> > >>>> 1. Merged & Completed features:
>>> >> > >>>>
>>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> >> > >>>> workloads Initial cut.
>>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>>> Scheduler.
>>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> >> Service
>>> >> > >>>> API and CLI.
>>> >> > >>>>
>>> >> > >>>> 2. Features close to finish:
>>> >> > >>>>
>>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >> Merge/Vote
>>> >> > >>>> Ongoing.
>>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> from
>>> >> > >>>> ATSv2. Patch in progress.
>>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> Improvement
>>> >> > works.
>>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>>> >> > >>>> Storage. In progress.
>>> >> > >>>>
>>> >> > >>>> 3. Tentative features:
>>> >> > >>>>
>>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging
>>> >> > to
>>> >> > >>>> be done before Aug 2018.
>>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>>> Challenging as more discussions are on-going.
>>> >> > >>>>
>>> >> > >>>> *Summary of 3.2.0 issues status:*
>>> >> > >>>>
>>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following up
>>> with
>>> >> > >>>> owners to get status on each of them to get in by Code Freeze
>>> date.
>>> >> > >>>>
>>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>>> >> > Version/s" =
>>> >> > >>>> 3.2.0 ORDER BY priority DESC
>>> >> > >>>>
>>> >> > >>>> Thanks,
>>> >> > >>>> Sunil
>>> >> > >>>>
>>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org>>
>>> >> wrote:
>>> >> > >>>>
>>> >> > >>>>> Hi All,
>>> >> > >>>>>
>>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
>>> dated
>>> >> > 17th
>>> >> > >>>>> July 2018, I would like to provide some updates.
>>> >> > >>>>>
>>> >> > >>>>> We are approaching previously proposed feature freeze date
>>> (Aug
>>> >> 21,
>>> >> > >>>>> about 7 days from today).
>>> >> > >>>>> If any features in branch which are targeted to 3.2.0, please
>>> >> reply
>>> >> > to
>>> >> > >>>>> this email thread.
>>> >> > >>>>> Steve has mentioned about the s3 features which will come
>>> close to
>>> >> > >>>>> Code Freeze Date (Aug 31st).
>>> >> > >>>>>
>>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>>> >> > >>>>>
>>> >> > >>>>> 1. Merged & Completed features:
>>> >> > >>>>>
>>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>>> DeepLearning
>>> >> > >>>>> workloads Initial cut.
>>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>>>>
>>> >> > >>>>> 2. Features close to finish:
>>> >> > >>>>>
>>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> Major
>>> >> > >>>>> patches are all in, only one last
>>> >> > >>>>> patch is in review state.
>>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>>>> Scheduler. Close to commit.
>>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> >> Service
>>> >> > >>>>> API and CLI. 2 patches are pending
>>> >> > >>>>> which will be closed by Feature freeze date.
>>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> >> from
>>> >> > >>>>> ATSv2. Patch in progress.
>>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> Improvement
>>> >> > >>>>> works.
>>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>>> Azure
>>> >> > >>>>> Storage. In progress.
>>> >> > >>>>>
>>> >> > >>>>> 3. Tentative features:
>>> >> > >>>>>
>>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging
>>> >> > >>>>> to be done before Aug 2018.
>>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>>>> Challenging as more discussions are on-going.
>>> >> > >>>>>
>>> >> > >>>>> *Summary of 3.2.0 issues status:*
>>> >> > >>>>>
>>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
>>> with
>>> >> > owners
>>> >> > >>>>> to get status on each of them to get in by Code Freeze date.
>>> >> > >>>>>
>>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>>> >> > Version/s" =
>>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>>> >> > >>>>>
>>> >> > >>>>> Thanks,
>>> >> > >>>>> Sunil
>>> >> > >>>>>
>>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org>>
>>> >> wrote:
>>> >> > >>>>>
>>> >> > >>>>>> Thanks Subru for the thoughts.
>>> >> > >>>>>> One of the main reason for a major release is to push out
>>> >> critical
>>> >> > >>>>>> features with a faster cadence to the users. If we are
>>> pulling
>>> >> more
>>> >> > and
>>> >> > >>>>>> more different types of features to a minor release, that
>>> branch
>>> >> > will
>>> >> > >>>>>> become more destabilized and it may be tough to say that
>>> 3.1.2 is
>>> >> > stable
>>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
>>> >> features
>>> >> > in
>>> >> > >>>>>> subsequent minor release.
>>> >> > >>>>>> For few companies, it makes sense to push out these new
>>> features
>>> >> > >>>>>> faster to make a reach to the users. Adding to the point to
>>> the
>>> >> > backporting
>>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
>>> with
>>> >> > some git
>>> >> > >>>>>> scripts. If we can make such scripts available to committers,
>>> >> > backport will
>>> >> > >>>>>> be seem-less across branches and we can achieve the faster
>>> >> release
>>> >> > cadence
>>> >> > >>>>>> also.
>>> >> > >>>>>>
>>> >> > >>>>>> Thoughts?
>>> >> > >>>>>>
>>> >> > >>>>>> - Sunil
>>> >> > >>>>>>
>>> >> > >>>>>>
>>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>>> subru@apache.org<ma...@apache.org>
>>> >> >
>>> >> > >>>>>> wrote:
>>> >> > >>>>>>
>>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release effort. I
>>> am
>>> >> > >>>>>>> generally
>>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as
>>> >> feel we
>>> >> > >>>>>>> already
>>> >> > >>>>>>> have too many branches to be maintained. I already see many
>>> >> commits
>>> >> > >>>>>>> are in
>>> >> > >>>>>>> different branches with no apparent rationale, for e.g: 3.1
>>> has
>>> >> > >>>>>>> commits
>>> >> > >>>>>>> which are absent in 3.0 etc.
>>> >> > >>>>>>>
>>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
>>> >> > production
>>> >> > >>>>>>> setting so the cost of adding features should be minimal.
>>> >> > >>>>>>>
>>> >> > >>>>>>> Thoughts?
>>> >> > >>>>>>>
>>> >> > >>>>>>> -Subru
>>> >> > >>>>>>>
>>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
>>> sunilg@apache.org<ma...@apache.org>>
>>> >> > wrote:
>>> >> > >>>>>>>
>>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Yes, important changes and features are much needed,
>>> hence we
>>> >> > will
>>> >> > >>>>>>> be
>>> >> > >>>>>>> > keeping the door open for them as possible. Also
>>> considering
>>> >> few
>>> >> > >>>>>>> more
>>> >> > >>>>>>> > offline requests from other folks, I think extending the
>>> >> > timeframe
>>> >> > >>>>>>> by
>>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
>>> buffer) and
>>> >> > >>>>>>> this should
>>> >> > >>>>>>> > ideally help us to ship this by September itself.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as
>>> >> well)
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Feature freeze date : all features to merge by August
>>> 21,
>>> >> 2018.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>>> improvements
>>> >> and
>>> >> > >>>>>>> non
>>> >> > >>>>>>> > blocker/critical
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > bug-fixes August 31, 2018.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Release date: September 15, 2018
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered
>>> your
>>> >> > >>>>>>> questions.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Thanks
>>> >> > >>>>>>> > Sunil
>>> >> > >>>>>>> >
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>>> >> wheeleast@gmail.com<ma...@gmail.com>
>>> >> > >
>>> >> > >>>>>>> wrote:
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release,
>>> +1
>>> >> for
>>> >> > >>>>>>> that.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > To concerns from Steve,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > It is a good idea to keep the door open to get important
>>> >> > changes
>>> >> > >>>>>>> /
>>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
>>> >> proposed
>>> >> > >>>>>>> release
>>> >> > >>>>>>> > date
>>> >> > >>>>>>> > > to make sure things can happen earlier instead of last
>>> >> minute
>>> >> > >>>>>>> and we all
>>> >> > >>>>>>> > > know that releases are always get delayed :). I'm also
>>> fine
>>> >> if
>>> >> > >>>>>>> we want
>>> >> > >>>>>>> > get
>>> >> > >>>>>>> > > another several weeks time.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
>>> before
>>> >> > >>>>>>> thanksgiving.
>>> >> > >>>>>>> > > Do you think is it good or too early / late?
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Eric,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
>>> YARN-8135
>>> >> can
>>> >> > >>>>>>> get merged
>>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Sunil,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Could u update
>>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>>> >> > >>>>>>> > Roadmap
>>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
>>> first
>>> >> > >>>>>>> before getting
>>> >> > >>>>>>> > > consensus of time.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Thanks,
>>> >> > >>>>>>> > > Wangda
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>> >> > >>>>>>> <fa...@cloudera.com.invalid>
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > wrote:
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>> >> > >>>>>>> stevel@hortonworks.com<ma...@hortonworks.com>>
>>> >> > >>>>>>> > >> wrote:
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <su...@apache.org>
>>> >> > <mailto:
>>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>> wrote:
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I would also would like to take this opportunity to
>>> come
>>> >> up
>>> >> > >>>>>>> with a
>>> >> > >>>>>>> > >> detailed
>>> >> > >>>>>>> > >> > plan.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
>>> merged by
>>> >> > >>>>>>> August 10,
>>> >> > >>>>>>> > >> 2018.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > <snip>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > Please let me know if I missed any features targeted
>>> to
>>> >> 3.2
>>> >> > >>>>>>> per this
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
>>> Datalake
>>> >> > Gen
>>> >> > >>>>>>> 2
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
>>> doing
>>> >> work
>>> >> > >>>>>>> on it, but
>>> >> > >>>>>>> > >> if
>>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
>>> >> timely
>>> >> > >>>>>>> reviews on
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will review
>>> the
>>> >> S3 &
>>> >> > >>>>>>> S3Guard
>>> >> > >>>>>>> > >> JIRAs
>>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
>>> some
>>> >> > >>>>>>> pressing
>>> >> > >>>>>>> > "guave,
>>> >> > >>>>>>> > >> > java 9 prep"
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> I can help with this part if you like.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as release
>>> >> > manager
>>> >> > >>>>>>> of 3.2.0
>>> >> > >>>>>>> > >> > release.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > well volunteered!
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I think this raises a good q: what timetable should
>>> we
>>> >> have
>>> >> > >>>>>>> for the
>>> >> > >>>>>>> > >> 3.2. &
>>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
>>> having
>>> >> > the
>>> >> > >>>>>>> outline
>>> >> > >>>>>>> > >> time
>>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
>>> less
>>> >> > >>>>>>> concern about
>>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > -Steve
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> -AF
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> >
>>> >> > >>>>>>>
>>> >> > >>>>>>
>>> >> >
>>> >>
>>> >
>>>
>>
>>
--
A very happy Clouderan
Re: Hadoop 3.2 Release Plan proposal
Posted by Steve Loughran <st...@hortonworks.com>.
its in.
good catch!
On 20 Oct 2018, at 01:35, Wei-Chiu Chuang <we...@cloudera.com>> wrote:
Thanks Sunil G for driving the release,
I filed HADOOP-15866<https://issues.apache.org/jira/browse/HADOOP-15866> for a compat fix. If any one has cycle please review it, as I think it is needed for 3.2.0.
On Thu, Oct 18, 2018 at 4:43 AM Sunil G <su...@apache.org>> wrote:
Hi Folks,
As we previously communicated for 3.2.0 release, we have delayed due to few
blockers in our gate.
I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
all bug fixes.
- Sunil
On Tue, Oct 16, 2018 at 8:59 AM Sunil G <su...@apache.org>> wrote:
> Hi Folks,
>
> We are now close to RC as other blocker issues are now merged to trunk and
> branch-3.2. Last 2 critical issues are closer to merge and will be
> committed in few hours.
> With this, I will be creating 3.2.0 branch today and will go ahead with RC
> related process.
>
> - Sunil
>
> On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jo...@stripe.com>>
> wrote:
>
>> Hello, were there any updates around the 3.2.0 RC timing? All I see in
>> the current blockers are related to the new Submarine subproject, wasn't
>> sure if that is what is holding things up.
>>
>> Cheers,
>> Jon
>>
>> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <su...@apache.org>> wrote:
>>
>>> Thanks Robert and Haibo for quickly correcting same.
>>> Sigh, I somehow missed one file while committing the change. Sorry for
>>> the
>>> trouble.
>>>
>>> - Sunil
>>>
>>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rk...@cloudera.com>>
>>> wrote:
>>>
>>> > Looks like there's two that weren't updated:
>>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
>>> > --include=pom.xml
>>> > ./hadoop-project/pom.xml:
>>> > <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>>> >
>>> > I've just pushed in an addendum commit to fix those.
>>> > In the future, please make sure to do a sanity compile when updating
>>> poms.
>>> >
>>> > thanks
>>> > - Robert
>>> >
>>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>>> <fa...@cloudera.com.invalid>>
>>> > wrote:
>>> >
>>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
>>> >> top-level pom.xml?
>>> >>
>>> >>
>>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org>> wrote:
>>> >>
>>> >> > Hi All
>>> >> >
>>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset trunk
>>> to
>>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
>>> necessary
>>> >> > patches are pulled into branch-3.2.
>>> >> >
>>> >> > Thank You
>>> >> > - Sunil
>>> >> >
>>> >> >
>>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org>> wrote:
>>> >> >
>>> >> > > Hi All
>>> >> > >
>>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged to
>>> >> trunk.
>>> >> > > Thanks for the support.
>>> >> > >
>>> >> > > *Plan for RC*
>>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>>> >> > > RC0 for 3.2: *28th Friday*
>>> >> > >
>>> >> > > Thank You
>>> >> > > Sunil
>>> >> > >
>>> >> > >
>>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org>>
>>> wrote:
>>> >> > >
>>> >> > >> Hi All
>>> >> > >>
>>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you
>>> for
>>> >> > >> helping in this. I am following up on these tickets, once its
>>> closed
>>> >> we
>>> >> > >> will cut the 3.2 branch.
>>> >> > >>
>>> >> > >> Thanks
>>> >> > >> Sunil Govindan
>>> >> > >>
>>> >> > >>
>>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org>>
>>> wrote:
>>> >> > >>
>>> >> > >>> Hi All,
>>> >> > >>>
>>> >> > >>> Inline with the original 3.2 communication proposal dated 17th
>>> July
>>> >> > >>> 2018, I would like to provide more updates.
>>> >> > >>>
>>> >> > >>> We are approaching previously proposed code freeze date
>>> (September
>>> >> 14,
>>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
>>> >> > existing
>>> >> > >>> trunk to 3.3 if there are no issues.
>>> >> > >>>
>>> >> > >>> *Current Release Plan:*
>>> >> > >>> Feature freeze date : all features to merge by September 7,
>>> 2018.
>>> >> > >>> Code freeze date : blockers/critical only, no improvements and
>>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>>> >> > >>> Release date: September 28, 2018
>>> >> > >>>
>>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0, we
>>> >> need to
>>> >> > >>> backport to 3.2 post branch cut.
>>> >> > >>>
>>> >> > >>> Here's an updated 3.2.0 feature status:
>>> >> > >>>
>>> >> > >>> 1. Merged & Completed features:
>>> >> > >>>
>>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> >> > >>> workloads Initial cut.
>>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>> Scheduler.
>>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> Service
>>> >> > API
>>> >> > >>> and CLI.
>>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement
>>> >> works.
>>> >> > >>>
>>> >> > >>> 2. Features close to finish:
>>> >> > >>>
>>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>>> >> > >>> - (Steve) S3a phase V. Close to commit.
>>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>>> >> > >>>
>>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> from
>>> >> > >>> ATSv2. Patch in progress.
>>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging to
>>> >> > >>> be done before Aug 2018.
>>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>> Challenging as more discussions are on-going.
>>> >> > >>>
>>> >> > >>> *Summary of 3.2.0 issues status:*
>>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following up
>>> with
>>> >> > >>> owners to get status on each of them to get in by Code Freeze
>>> date.
>>> >> > >>>
>>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > (Blocker,
>>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" =
>>> 3.2.0
>>> >> > ORDER
>>> >> > >>> BY priority DESC
>>> >> > >>>
>>> >> > >>> Thanks,
>>> >> > >>> Sunil
>>> >> > >>>
>>> >> > >>>
>>> >> > >>>
>>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org>>
>>> wrote:
>>> >> > >>>
>>> >> > >>>> Hi All,
>>> >> > >>>>
>>> >> > >>>> Inline with earlier communication dated 17th July 2018, I would
>>> >> like
>>> >> > to
>>> >> > >>>> provide some updates.
>>> >> > >>>>
>>> >> > >>>> We are approaching previously proposed code freeze date (Aug
>>> 31).
>>> >> > >>>>
>>> >> > >>>> One of the critical feature Node Attributes feature merge
>>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need a
>>> bit
>>> >> > more
>>> >> > >>>> time. With regard to this, suggesting to push the feature/code
>>> >> freeze
>>> >> > for 2
>>> >> > >>>> more weeks to accommodate these jiras too.
>>> >> > >>>>
>>> >> > >>>> Proposing Updated changes in plan inline with this:
>>> >> > >>>> Feature freeze date : all features to merge by September 7,
>>> 2018.
>>> >> > >>>> Code freeze date : blockers/critical only, no improvements and
>>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>>> >> > >>>> Release date: September 28, 2018
>>> >> > >>>>
>>> >> > >>>> If any features in branch which are targeted to 3.2.0, please
>>> >> reply to
>>> >> > >>>> this email thread.
>>> >> > >>>>
>>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>>> >> > >>>>
>>> >> > >>>> 1. Merged & Completed features:
>>> >> > >>>>
>>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> >> > >>>> workloads Initial cut.
>>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>>> Scheduler.
>>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> >> Service
>>> >> > >>>> API and CLI.
>>> >> > >>>>
>>> >> > >>>> 2. Features close to finish:
>>> >> > >>>>
>>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >> Merge/Vote
>>> >> > >>>> Ongoing.
>>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> from
>>> >> > >>>> ATSv2. Patch in progress.
>>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> Improvement
>>> >> > works.
>>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>>> >> > >>>> Storage. In progress.
>>> >> > >>>>
>>> >> > >>>> 3. Tentative features:
>>> >> > >>>>
>>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging
>>> >> > to
>>> >> > >>>> be done before Aug 2018.
>>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>>> Challenging as more discussions are on-going.
>>> >> > >>>>
>>> >> > >>>> *Summary of 3.2.0 issues status:*
>>> >> > >>>>
>>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following up
>>> with
>>> >> > >>>> owners to get status on each of them to get in by Code Freeze
>>> date.
>>> >> > >>>>
>>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>>> >> > Version/s" =
>>> >> > >>>> 3.2.0 ORDER BY priority DESC
>>> >> > >>>>
>>> >> > >>>> Thanks,
>>> >> > >>>> Sunil
>>> >> > >>>>
>>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org>>
>>> >> wrote:
>>> >> > >>>>
>>> >> > >>>>> Hi All,
>>> >> > >>>>>
>>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
>>> dated
>>> >> > 17th
>>> >> > >>>>> July 2018, I would like to provide some updates.
>>> >> > >>>>>
>>> >> > >>>>> We are approaching previously proposed feature freeze date
>>> (Aug
>>> >> 21,
>>> >> > >>>>> about 7 days from today).
>>> >> > >>>>> If any features in branch which are targeted to 3.2.0, please
>>> >> reply
>>> >> > to
>>> >> > >>>>> this email thread.
>>> >> > >>>>> Steve has mentioned about the s3 features which will come
>>> close to
>>> >> > >>>>> Code Freeze Date (Aug 31st).
>>> >> > >>>>>
>>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>>> >> > >>>>>
>>> >> > >>>>> 1. Merged & Completed features:
>>> >> > >>>>>
>>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>>> DeepLearning
>>> >> > >>>>> workloads Initial cut.
>>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>>>>
>>> >> > >>>>> 2. Features close to finish:
>>> >> > >>>>>
>>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> Major
>>> >> > >>>>> patches are all in, only one last
>>> >> > >>>>> patch is in review state.
>>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>>>> Scheduler. Close to commit.
>>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> >> Service
>>> >> > >>>>> API and CLI. 2 patches are pending
>>> >> > >>>>> which will be closed by Feature freeze date.
>>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> >> from
>>> >> > >>>>> ATSv2. Patch in progress.
>>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> Improvement
>>> >> > >>>>> works.
>>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>>> Azure
>>> >> > >>>>> Storage. In progress.
>>> >> > >>>>>
>>> >> > >>>>> 3. Tentative features:
>>> >> > >>>>>
>>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging
>>> >> > >>>>> to be done before Aug 2018.
>>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>>>> Challenging as more discussions are on-going.
>>> >> > >>>>>
>>> >> > >>>>> *Summary of 3.2.0 issues status:*
>>> >> > >>>>>
>>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
>>> with
>>> >> > owners
>>> >> > >>>>> to get status on each of them to get in by Code Freeze date.
>>> >> > >>>>>
>>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>>> >> > Version/s" =
>>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>>> >> > >>>>>
>>> >> > >>>>> Thanks,
>>> >> > >>>>> Sunil
>>> >> > >>>>>
>>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org>>
>>> >> wrote:
>>> >> > >>>>>
>>> >> > >>>>>> Thanks Subru for the thoughts.
>>> >> > >>>>>> One of the main reason for a major release is to push out
>>> >> critical
>>> >> > >>>>>> features with a faster cadence to the users. If we are
>>> pulling
>>> >> more
>>> >> > and
>>> >> > >>>>>> more different types of features to a minor release, that
>>> branch
>>> >> > will
>>> >> > >>>>>> become more destabilized and it may be tough to say that
>>> 3.1.2 is
>>> >> > stable
>>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
>>> >> features
>>> >> > in
>>> >> > >>>>>> subsequent minor release.
>>> >> > >>>>>> For few companies, it makes sense to push out these new
>>> features
>>> >> > >>>>>> faster to make a reach to the users. Adding to the point to
>>> the
>>> >> > backporting
>>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
>>> with
>>> >> > some git
>>> >> > >>>>>> scripts. If we can make such scripts available to committers,
>>> >> > backport will
>>> >> > >>>>>> be seem-less across branches and we can achieve the faster
>>> >> release
>>> >> > cadence
>>> >> > >>>>>> also.
>>> >> > >>>>>>
>>> >> > >>>>>> Thoughts?
>>> >> > >>>>>>
>>> >> > >>>>>> - Sunil
>>> >> > >>>>>>
>>> >> > >>>>>>
>>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>>> subru@apache.org<ma...@apache.org>
>>> >> >
>>> >> > >>>>>> wrote:
>>> >> > >>>>>>
>>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release effort. I
>>> am
>>> >> > >>>>>>> generally
>>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as
>>> >> feel we
>>> >> > >>>>>>> already
>>> >> > >>>>>>> have too many branches to be maintained. I already see many
>>> >> commits
>>> >> > >>>>>>> are in
>>> >> > >>>>>>> different branches with no apparent rationale, for e.g: 3.1
>>> has
>>> >> > >>>>>>> commits
>>> >> > >>>>>>> which are absent in 3.0 etc.
>>> >> > >>>>>>>
>>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
>>> >> > production
>>> >> > >>>>>>> setting so the cost of adding features should be minimal.
>>> >> > >>>>>>>
>>> >> > >>>>>>> Thoughts?
>>> >> > >>>>>>>
>>> >> > >>>>>>> -Subru
>>> >> > >>>>>>>
>>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
>>> sunilg@apache.org<ma...@apache.org>>
>>> >> > wrote:
>>> >> > >>>>>>>
>>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Yes, important changes and features are much needed,
>>> hence we
>>> >> > will
>>> >> > >>>>>>> be
>>> >> > >>>>>>> > keeping the door open for them as possible. Also
>>> considering
>>> >> few
>>> >> > >>>>>>> more
>>> >> > >>>>>>> > offline requests from other folks, I think extending the
>>> >> > timeframe
>>> >> > >>>>>>> by
>>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
>>> buffer) and
>>> >> > >>>>>>> this should
>>> >> > >>>>>>> > ideally help us to ship this by September itself.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as
>>> >> well)
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Feature freeze date : all features to merge by August
>>> 21,
>>> >> 2018.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>>> improvements
>>> >> and
>>> >> > >>>>>>> non
>>> >> > >>>>>>> > blocker/critical
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > bug-fixes August 31, 2018.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Release date: September 15, 2018
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered
>>> your
>>> >> > >>>>>>> questions.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Thanks
>>> >> > >>>>>>> > Sunil
>>> >> > >>>>>>> >
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>>> >> wheeleast@gmail.com<ma...@gmail.com>
>>> >> > >
>>> >> > >>>>>>> wrote:
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release,
>>> +1
>>> >> for
>>> >> > >>>>>>> that.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > To concerns from Steve,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > It is a good idea to keep the door open to get important
>>> >> > changes
>>> >> > >>>>>>> /
>>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
>>> >> proposed
>>> >> > >>>>>>> release
>>> >> > >>>>>>> > date
>>> >> > >>>>>>> > > to make sure things can happen earlier instead of last
>>> >> minute
>>> >> > >>>>>>> and we all
>>> >> > >>>>>>> > > know that releases are always get delayed :). I'm also
>>> fine
>>> >> if
>>> >> > >>>>>>> we want
>>> >> > >>>>>>> > get
>>> >> > >>>>>>> > > another several weeks time.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
>>> before
>>> >> > >>>>>>> thanksgiving.
>>> >> > >>>>>>> > > Do you think is it good or too early / late?
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Eric,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
>>> YARN-8135
>>> >> can
>>> >> > >>>>>>> get merged
>>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Sunil,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Could u update
>>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>>> >> > >>>>>>> > Roadmap
>>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
>>> first
>>> >> > >>>>>>> before getting
>>> >> > >>>>>>> > > consensus of time.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Thanks,
>>> >> > >>>>>>> > > Wangda
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>> >> > >>>>>>> <fa...@cloudera.com.invalid>
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > wrote:
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>> >> > >>>>>>> stevel@hortonworks.com<ma...@hortonworks.com>>
>>> >> > >>>>>>> > >> wrote:
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <su...@apache.org>
>>> >> > <mailto:
>>> >> > >>>>>>> > >> > sunilg@apache.org<ma...@apache.org>>> wrote:
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I would also would like to take this opportunity to
>>> come
>>> >> up
>>> >> > >>>>>>> with a
>>> >> > >>>>>>> > >> detailed
>>> >> > >>>>>>> > >> > plan.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
>>> merged by
>>> >> > >>>>>>> August 10,
>>> >> > >>>>>>> > >> 2018.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > <snip>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > Please let me know if I missed any features targeted
>>> to
>>> >> 3.2
>>> >> > >>>>>>> per this
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
>>> Datalake
>>> >> > Gen
>>> >> > >>>>>>> 2
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
>>> doing
>>> >> work
>>> >> > >>>>>>> on it, but
>>> >> > >>>>>>> > >> if
>>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
>>> >> timely
>>> >> > >>>>>>> reviews on
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will review
>>> the
>>> >> S3 &
>>> >> > >>>>>>> S3Guard
>>> >> > >>>>>>> > >> JIRAs
>>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
>>> some
>>> >> > >>>>>>> pressing
>>> >> > >>>>>>> > "guave,
>>> >> > >>>>>>> > >> > java 9 prep"
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> I can help with this part if you like.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as release
>>> >> > manager
>>> >> > >>>>>>> of 3.2.0
>>> >> > >>>>>>> > >> > release.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > well volunteered!
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I think this raises a good q: what timetable should
>>> we
>>> >> have
>>> >> > >>>>>>> for the
>>> >> > >>>>>>> > >> 3.2. &
>>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
>>> having
>>> >> > the
>>> >> > >>>>>>> outline
>>> >> > >>>>>>> > >> time
>>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
>>> less
>>> >> > >>>>>>> concern about
>>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > -Steve
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> -AF
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> >
>>> >> > >>>>>>>
>>> >> > >>>>>>
>>> >> >
>>> >>
>>> >
>>>
>>
>>
--
A very happy Clouderan
Re: Hadoop 3.2 Release Plan proposal
Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
Thanks Sunil G for driving the release,
I filed HADOOP-15866 <https://issues.apache.org/jira/browse/HADOOP-15866> for
a compat fix. If any one has cycle please review it, as I think it is
needed for 3.2.0.
On Thu, Oct 18, 2018 at 4:43 AM Sunil G <su...@apache.org> wrote:
> Hi Folks,
>
> As we previously communicated for 3.2.0 release, we have delayed due to few
> blockers in our gate.
>
> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
> all bug fixes.
>
> - Sunil
>
>
> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <su...@apache.org> wrote:
>
> > Hi Folks,
> >
> > We are now close to RC as other blocker issues are now merged to trunk
> and
> > branch-3.2. Last 2 critical issues are closer to merge and will be
> > committed in few hours.
> > With this, I will be creating 3.2.0 branch today and will go ahead with
> RC
> > related process.
> >
> > - Sunil
> >
> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jo...@stripe.com>
> > wrote:
> >
> >> Hello, were there any updates around the 3.2.0 RC timing? All I see in
> >> the current blockers are related to the new Submarine subproject, wasn't
> >> sure if that is what is holding things up.
> >>
> >> Cheers,
> >> Jon
> >>
> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <su...@apache.org> wrote:
> >>
> >>> Thanks Robert and Haibo for quickly correcting same.
> >>> Sigh, I somehow missed one file while committing the change. Sorry for
> >>> the
> >>> trouble.
> >>>
> >>> - Sunil
> >>>
> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rk...@cloudera.com>
> >>> wrote:
> >>>
> >>> > Looks like there's two that weren't updated:
> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
> >>> > --include=pom.xml
> >>> > ./hadoop-project/pom.xml:
> >>> > <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
> >>> >
> >>> > I've just pushed in an addendum commit to fix those.
> >>> > In the future, please make sure to do a sanity compile when updating
> >>> poms.
> >>> >
> >>> > thanks
> >>> > - Robert
> >>> >
> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
> >>> <fa...@cloudera.com.invalid>
> >>> > wrote:
> >>> >
> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
> >>> >> top-level pom.xml?
> >>> >>
> >>> >>
> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
> >>> >>
> >>> >> > Hi All
> >>> >> >
> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
> trunk
> >>> to
> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
> >>> necessary
> >>> >> > patches are pulled into branch-3.2.
> >>> >> >
> >>> >> > Thank You
> >>> >> > - Sunil
> >>> >> >
> >>> >> >
> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org>
> wrote:
> >>> >> >
> >>> >> > > Hi All
> >>> >> > >
> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged
> to
> >>> >> trunk.
> >>> >> > > Thanks for the support.
> >>> >> > >
> >>> >> > > *Plan for RC*
> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
> >>> >> > > RC0 for 3.2: *28th Friday*
> >>> >> > >
> >>> >> > > Thank You
> >>> >> > > Sunil
> >>> >> > >
> >>> >> > >
> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org>
> >>> wrote:
> >>> >> > >
> >>> >> > >> Hi All
> >>> >> > >>
> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you
> >>> for
> >>> >> > >> helping in this. I am following up on these tickets, once its
> >>> closed
> >>> >> we
> >>> >> > >> will cut the 3.2 branch.
> >>> >> > >>
> >>> >> > >> Thanks
> >>> >> > >> Sunil Govindan
> >>> >> > >>
> >>> >> > >>
> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org>
> >>> wrote:
> >>> >> > >>
> >>> >> > >>> Hi All,
> >>> >> > >>>
> >>> >> > >>> Inline with the original 3.2 communication proposal dated 17th
> >>> July
> >>> >> > >>> 2018, I would like to provide more updates.
> >>> >> > >>>
> >>> >> > >>> We are approaching previously proposed code freeze date
> >>> (September
> >>> >> 14,
> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
> point
> >>> >> > existing
> >>> >> > >>> trunk to 3.3 if there are no issues.
> >>> >> > >>>
> >>> >> > >>> *Current Release Plan:*
> >>> >> > >>> Feature freeze date : all features to merge by September 7,
> >>> 2018.
> >>> >> > >>> Code freeze date : blockers/critical only, no improvements and
> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
> >>> >> > >>> Release date: September 28, 2018
> >>> >> > >>>
> >>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0,
> we
> >>> >> need to
> >>> >> > >>> backport to 3.2 post branch cut.
> >>> >> > >>>
> >>> >> > >>> Here's an updated 3.2.0 feature status:
> >>> >> > >>>
> >>> >> > >>> 1. Merged & Completed features:
> >>> >> > >>>
> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
> DeepLearning
> >>> >> > >>> workloads Initial cut.
> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >>> >> > >>> Scheduler.
> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> Service
> >>> >> > API
> >>> >> > >>> and CLI.
> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
> Improvement
> >>> >> works.
> >>> >> > >>>
> >>> >> > >>> 2. Features close to finish:
> >>> >> > >>>
> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
> >>> >> > >>>
> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
> >>> from
> >>> >> > >>> ATSv2. Patch in progress.
> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging to
> >>> >> > >>> be done before Aug 2018.
> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >>> >> > >>> Challenging as more discussions are on-going.
> >>> >> > >>>
> >>> >> > >>> *Summary of 3.2.0 issues status:*
> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following up
> >>> with
> >>> >> > >>> owners to get status on each of them to get in by Code Freeze
> >>> date.
> >>> >> > >>>
> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> >>> >> > (Blocker,
> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" =
> >>> 3.2.0
> >>> >> > ORDER
> >>> >> > >>> BY priority DESC
> >>> >> > >>>
> >>> >> > >>> Thanks,
> >>> >> > >>> Sunil
> >>> >> > >>>
> >>> >> > >>>
> >>> >> > >>>
> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org>
> >>> wrote:
> >>> >> > >>>
> >>> >> > >>>> Hi All,
> >>> >> > >>>>
> >>> >> > >>>> Inline with earlier communication dated 17th July 2018, I
> would
> >>> >> like
> >>> >> > to
> >>> >> > >>>> provide some updates.
> >>> >> > >>>>
> >>> >> > >>>> We are approaching previously proposed code freeze date (Aug
> >>> 31).
> >>> >> > >>>>
> >>> >> > >>>> One of the critical feature Node Attributes feature merge
> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need
> a
> >>> bit
> >>> >> > more
> >>> >> > >>>> time. With regard to this, suggesting to push the
> feature/code
> >>> >> freeze
> >>> >> > for 2
> >>> >> > >>>> more weeks to accommodate these jiras too.
> >>> >> > >>>>
> >>> >> > >>>> Proposing Updated changes in plan inline with this:
> >>> >> > >>>> Feature freeze date : all features to merge by September 7,
> >>> 2018.
> >>> >> > >>>> Code freeze date : blockers/critical only, no improvements
> and
> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
> >>> >> > >>>> Release date: September 28, 2018
> >>> >> > >>>>
> >>> >> > >>>> If any features in branch which are targeted to 3.2.0, please
> >>> >> reply to
> >>> >> > >>>> this email thread.
> >>> >> > >>>>
> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
> >>> >> > >>>>
> >>> >> > >>>> 1. Merged & Completed features:
> >>> >> > >>>>
> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> DeepLearning
> >>> >> > >>>> workloads Initial cut.
> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> Capacity
> >>> >> > >>>> Scheduler.
> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> >> Service
> >>> >> > >>>> API and CLI.
> >>> >> > >>>>
> >>> >> > >>>> 2. Features close to finish:
> >>> >> > >>>>
> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> >> Merge/Vote
> >>> >> > >>>> Ongoing.
> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> apps
> >>> from
> >>> >> > >>>> ATSv2. Patch in progress.
> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> Improvement
> >>> >> > works.
> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
> Azure
> >>> >> > >>>> Storage. In progress.
> >>> >> > >>>>
> >>> >> > >>>> 3. Tentative features:
> >>> >> > >>>>
> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging
> >>> >> > to
> >>> >> > >>>> be done before Aug 2018.
> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
> applications.
> >>> >> > >>>> Challenging as more discussions are on-going.
> >>> >> > >>>>
> >>> >> > >>>> *Summary of 3.2.0 issues status:*
> >>> >> > >>>>
> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following
> up
> >>> with
> >>> >> > >>>> owners to get status on each of them to get in by Code Freeze
> >>> date.
> >>> >> > >>>>
> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
> in
> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> >>> >> > Version/s" =
> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
> >>> >> > >>>>
> >>> >> > >>>> Thanks,
> >>> >> > >>>> Sunil
> >>> >> > >>>>
> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org>
> >>> >> wrote:
> >>> >> > >>>>
> >>> >> > >>>>> Hi All,
> >>> >> > >>>>>
> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
> >>> dated
> >>> >> > 17th
> >>> >> > >>>>> July 2018, I would like to provide some updates.
> >>> >> > >>>>>
> >>> >> > >>>>> We are approaching previously proposed feature freeze date
> >>> (Aug
> >>> >> 21,
> >>> >> > >>>>> about 7 days from today).
> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
> please
> >>> >> reply
> >>> >> > to
> >>> >> > >>>>> this email thread.
> >>> >> > >>>>> Steve has mentioned about the s3 features which will come
> >>> close to
> >>> >> > >>>>> Code Freeze Date (Aug 31st).
> >>> >> > >>>>>
> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
> >>> >> > >>>>>
> >>> >> > >>>>> 1. Merged & Completed features:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> >>> DeepLearning
> >>> >> > >>>>> workloads Initial cut.
> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>>>>
> >>> >> > >>>>> 2. Features close to finish:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> Major
> >>> >> > >>>>> patches are all in, only one last
> >>> >> > >>>>> patch is in review state.
> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> Capacity
> >>> >> > >>>>> Scheduler. Close to commit.
> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> >> Service
> >>> >> > >>>>> API and CLI. 2 patches are pending
> >>> >> > >>>>> which will be closed by Feature freeze date.
> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> apps
> >>> >> from
> >>> >> > >>>>> ATSv2. Patch in progress.
> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> Improvement
> >>> >> > >>>>> works.
> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
> >>> Azure
> >>> >> > >>>>> Storage. In progress.
> >>> >> > >>>>>
> >>> >> > >>>>> 3. Tentative features:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging
> >>> >> > >>>>> to be done before Aug 2018.
> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
> applications.
> >>> >> > >>>>> Challenging as more discussions are on-going.
> >>> >> > >>>>>
> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
> >>> >> > >>>>>
> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
> >>> with
> >>> >> > owners
> >>> >> > >>>>> to get status on each of them to get in by Code Freeze date.
> >>> >> > >>>>>
> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
> in
> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> >>> >> > Version/s" =
> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
> >>> >> > >>>>>
> >>> >> > >>>>> Thanks,
> >>> >> > >>>>> Sunil
> >>> >> > >>>>>
> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org>
> >>> >> wrote:
> >>> >> > >>>>>
> >>> >> > >>>>>> Thanks Subru for the thoughts.
> >>> >> > >>>>>> One of the main reason for a major release is to push out
> >>> >> critical
> >>> >> > >>>>>> features with a faster cadence to the users. If we are
> >>> pulling
> >>> >> more
> >>> >> > and
> >>> >> > >>>>>> more different types of features to a minor release, that
> >>> branch
> >>> >> > will
> >>> >> > >>>>>> become more destabilized and it may be tough to say that
> >>> 3.1.2 is
> >>> >> > stable
> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
> >>> >> features
> >>> >> > in
> >>> >> > >>>>>> subsequent minor release.
> >>> >> > >>>>>> For few companies, it makes sense to push out these new
> >>> features
> >>> >> > >>>>>> faster to make a reach to the users. Adding to the point to
> >>> the
> >>> >> > backporting
> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
> >>> with
> >>> >> > some git
> >>> >> > >>>>>> scripts. If we can make such scripts available to
> committers,
> >>> >> > backport will
> >>> >> > >>>>>> be seem-less across branches and we can achieve the faster
> >>> >> release
> >>> >> > cadence
> >>> >> > >>>>>> also.
> >>> >> > >>>>>>
> >>> >> > >>>>>> Thoughts?
> >>> >> > >>>>>>
> >>> >> > >>>>>> - Sunil
> >>> >> > >>>>>>
> >>> >> > >>>>>>
> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
> >>> subru@apache.org
> >>> >> >
> >>> >> > >>>>>> wrote:
> >>> >> > >>>>>>
> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release effort.
> I
> >>> am
> >>> >> > >>>>>>> generally
> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x)
> as
> >>> >> feel we
> >>> >> > >>>>>>> already
> >>> >> > >>>>>>> have too many branches to be maintained. I already see
> many
> >>> >> commits
> >>> >> > >>>>>>> are in
> >>> >> > >>>>>>> different branches with no apparent rationale, for e.g:
> 3.1
> >>> has
> >>> >> > >>>>>>> commits
> >>> >> > >>>>>>> which are absent in 3.0 etc.
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
> >>> >> > production
> >>> >> > >>>>>>> setting so the cost of adding features should be minimal.
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> Thoughts?
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> -Subru
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
> >>> sunilg@apache.org>
> >>> >> > wrote:
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Yes, important changes and features are much needed,
> >>> hence we
> >>> >> > will
> >>> >> > >>>>>>> be
> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
> >>> considering
> >>> >> few
> >>> >> > >>>>>>> more
> >>> >> > >>>>>>> > offline requests from other folks, I think extending the
> >>> >> > timeframe
> >>> >> > >>>>>>> by
> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
> >>> buffer) and
> >>> >> > >>>>>>> this should
> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki
> as
> >>> >> well)
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by August
> >>> 21,
> >>> >> 2018.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
> >>> improvements
> >>> >> and
> >>> >> > >>>>>>> non
> >>> >> > >>>>>>> > blocker/critical
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Release date: September 15, 2018
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered
> >>> your
> >>> >> > >>>>>>> questions.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Thanks
> >>> >> > >>>>>>> > Sunil
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
> >>> >> wheeleast@gmail.com
> >>> >> > >
> >>> >> > >>>>>>> wrote:
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release,
> >>> +1
> >>> >> for
> >>> >> > >>>>>>> that.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > To concerns from Steve,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
> important
> >>> >> > changes
> >>> >> > >>>>>>> /
> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
> >>> >> proposed
> >>> >> > >>>>>>> release
> >>> >> > >>>>>>> > date
> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of last
> >>> >> minute
> >>> >> > >>>>>>> and we all
> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm also
> >>> fine
> >>> >> if
> >>> >> > >>>>>>> we want
> >>> >> > >>>>>>> > get
> >>> >> > >>>>>>> > > another several weeks time.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
> >>> before
> >>> >> > >>>>>>> thanksgiving.
> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Eric,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
> >>> YARN-8135
> >>> >> can
> >>> >> > >>>>>>> get merged
> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Sunil,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Could u update
> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
> >>> >> > >>>>>>> > Roadmap
> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
> >>> first
> >>> >> > >>>>>>> before getting
> >>> >> > >>>>>>> > > consensus of time.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Thanks,
> >>> >> > >>>>>>> > > Wangda
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > wrote:
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> >>> >> > >>>>>>> stevel@hortonworks.com>
> >>> >> > >>>>>>> > >> wrote:
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
> sunilg@apache.org
> >>> >> > <mailto:
> >>> >> > >>>>>>> > >> > sunilg@apache.org>> wrote:
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I would also would like to take this opportunity to
> >>> come
> >>> >> up
> >>> >> > >>>>>>> with a
> >>> >> > >>>>>>> > >> detailed
> >>> >> > >>>>>>> > >> > plan.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
> >>> merged by
> >>> >> > >>>>>>> August 10,
> >>> >> > >>>>>>> > >> 2018.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > <snip>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
> targeted
> >>> to
> >>> >> 3.2
> >>> >> > >>>>>>> per this
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
> >>> Datalake
> >>> >> > Gen
> >>> >> > >>>>>>> 2
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
> >>> doing
> >>> >> work
> >>> >> > >>>>>>> on it, but
> >>> >> > >>>>>>> > >> if
> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
> >>> >> timely
> >>> >> > >>>>>>> reviews on
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will review
> >>> the
> >>> >> S3 &
> >>> >> > >>>>>>> S3Guard
> >>> >> > >>>>>>> > >> JIRAs
> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
> >>> some
> >>> >> > >>>>>>> pressing
> >>> >> > >>>>>>> > "guave,
> >>> >> > >>>>>>> > >> > java 9 prep"
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> I can help with this part if you like.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
> release
> >>> >> > manager
> >>> >> > >>>>>>> of 3.2.0
> >>> >> > >>>>>>> > >> > release.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > well volunteered!
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable should
> >>> we
> >>> >> have
> >>> >> > >>>>>>> for the
> >>> >> > >>>>>>> > >> 3.2. &
> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
> >>> having
> >>> >> > the
> >>> >> > >>>>>>> outline
> >>> >> > >>>>>>> > >> time
> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
> >>> less
> >>> >> > >>>>>>> concern about
> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > -Steve
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> -AF
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>>
> >>> >> > >>>>>>
> >>> >> >
> >>> >>
> >>> >
> >>>
> >>
> >>
>
--
A very happy Clouderan
Re: Hadoop 3.2 Release Plan proposal
Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
Thanks Sunil G for driving the release,
I filed HADOOP-15866 <https://issues.apache.org/jira/browse/HADOOP-15866> for
a compat fix. If any one has cycle please review it, as I think it is
needed for 3.2.0.
On Thu, Oct 18, 2018 at 4:43 AM Sunil G <su...@apache.org> wrote:
> Hi Folks,
>
> As we previously communicated for 3.2.0 release, we have delayed due to few
> blockers in our gate.
>
> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
> all bug fixes.
>
> - Sunil
>
>
> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <su...@apache.org> wrote:
>
> > Hi Folks,
> >
> > We are now close to RC as other blocker issues are now merged to trunk
> and
> > branch-3.2. Last 2 critical issues are closer to merge and will be
> > committed in few hours.
> > With this, I will be creating 3.2.0 branch today and will go ahead with
> RC
> > related process.
> >
> > - Sunil
> >
> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jo...@stripe.com>
> > wrote:
> >
> >> Hello, were there any updates around the 3.2.0 RC timing? All I see in
> >> the current blockers are related to the new Submarine subproject, wasn't
> >> sure if that is what is holding things up.
> >>
> >> Cheers,
> >> Jon
> >>
> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <su...@apache.org> wrote:
> >>
> >>> Thanks Robert and Haibo for quickly correcting same.
> >>> Sigh, I somehow missed one file while committing the change. Sorry for
> >>> the
> >>> trouble.
> >>>
> >>> - Sunil
> >>>
> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rk...@cloudera.com>
> >>> wrote:
> >>>
> >>> > Looks like there's two that weren't updated:
> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
> >>> > --include=pom.xml
> >>> > ./hadoop-project/pom.xml:
> >>> > <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
> >>> >
> >>> > I've just pushed in an addendum commit to fix those.
> >>> > In the future, please make sure to do a sanity compile when updating
> >>> poms.
> >>> >
> >>> > thanks
> >>> > - Robert
> >>> >
> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
> >>> <fa...@cloudera.com.invalid>
> >>> > wrote:
> >>> >
> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
> >>> >> top-level pom.xml?
> >>> >>
> >>> >>
> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
> >>> >>
> >>> >> > Hi All
> >>> >> >
> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
> trunk
> >>> to
> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
> >>> necessary
> >>> >> > patches are pulled into branch-3.2.
> >>> >> >
> >>> >> > Thank You
> >>> >> > - Sunil
> >>> >> >
> >>> >> >
> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org>
> wrote:
> >>> >> >
> >>> >> > > Hi All
> >>> >> > >
> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged
> to
> >>> >> trunk.
> >>> >> > > Thanks for the support.
> >>> >> > >
> >>> >> > > *Plan for RC*
> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
> >>> >> > > RC0 for 3.2: *28th Friday*
> >>> >> > >
> >>> >> > > Thank You
> >>> >> > > Sunil
> >>> >> > >
> >>> >> > >
> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org>
> >>> wrote:
> >>> >> > >
> >>> >> > >> Hi All
> >>> >> > >>
> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you
> >>> for
> >>> >> > >> helping in this. I am following up on these tickets, once its
> >>> closed
> >>> >> we
> >>> >> > >> will cut the 3.2 branch.
> >>> >> > >>
> >>> >> > >> Thanks
> >>> >> > >> Sunil Govindan
> >>> >> > >>
> >>> >> > >>
> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org>
> >>> wrote:
> >>> >> > >>
> >>> >> > >>> Hi All,
> >>> >> > >>>
> >>> >> > >>> Inline with the original 3.2 communication proposal dated 17th
> >>> July
> >>> >> > >>> 2018, I would like to provide more updates.
> >>> >> > >>>
> >>> >> > >>> We are approaching previously proposed code freeze date
> >>> (September
> >>> >> 14,
> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
> point
> >>> >> > existing
> >>> >> > >>> trunk to 3.3 if there are no issues.
> >>> >> > >>>
> >>> >> > >>> *Current Release Plan:*
> >>> >> > >>> Feature freeze date : all features to merge by September 7,
> >>> 2018.
> >>> >> > >>> Code freeze date : blockers/critical only, no improvements and
> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
> >>> >> > >>> Release date: September 28, 2018
> >>> >> > >>>
> >>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0,
> we
> >>> >> need to
> >>> >> > >>> backport to 3.2 post branch cut.
> >>> >> > >>>
> >>> >> > >>> Here's an updated 3.2.0 feature status:
> >>> >> > >>>
> >>> >> > >>> 1. Merged & Completed features:
> >>> >> > >>>
> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
> DeepLearning
> >>> >> > >>> workloads Initial cut.
> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >>> >> > >>> Scheduler.
> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> Service
> >>> >> > API
> >>> >> > >>> and CLI.
> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
> Improvement
> >>> >> works.
> >>> >> > >>>
> >>> >> > >>> 2. Features close to finish:
> >>> >> > >>>
> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
> >>> >> > >>>
> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
> >>> from
> >>> >> > >>> ATSv2. Patch in progress.
> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging to
> >>> >> > >>> be done before Aug 2018.
> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >>> >> > >>> Challenging as more discussions are on-going.
> >>> >> > >>>
> >>> >> > >>> *Summary of 3.2.0 issues status:*
> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following up
> >>> with
> >>> >> > >>> owners to get status on each of them to get in by Code Freeze
> >>> date.
> >>> >> > >>>
> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> >>> >> > (Blocker,
> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" =
> >>> 3.2.0
> >>> >> > ORDER
> >>> >> > >>> BY priority DESC
> >>> >> > >>>
> >>> >> > >>> Thanks,
> >>> >> > >>> Sunil
> >>> >> > >>>
> >>> >> > >>>
> >>> >> > >>>
> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org>
> >>> wrote:
> >>> >> > >>>
> >>> >> > >>>> Hi All,
> >>> >> > >>>>
> >>> >> > >>>> Inline with earlier communication dated 17th July 2018, I
> would
> >>> >> like
> >>> >> > to
> >>> >> > >>>> provide some updates.
> >>> >> > >>>>
> >>> >> > >>>> We are approaching previously proposed code freeze date (Aug
> >>> 31).
> >>> >> > >>>>
> >>> >> > >>>> One of the critical feature Node Attributes feature merge
> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need
> a
> >>> bit
> >>> >> > more
> >>> >> > >>>> time. With regard to this, suggesting to push the
> feature/code
> >>> >> freeze
> >>> >> > for 2
> >>> >> > >>>> more weeks to accommodate these jiras too.
> >>> >> > >>>>
> >>> >> > >>>> Proposing Updated changes in plan inline with this:
> >>> >> > >>>> Feature freeze date : all features to merge by September 7,
> >>> 2018.
> >>> >> > >>>> Code freeze date : blockers/critical only, no improvements
> and
> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
> >>> >> > >>>> Release date: September 28, 2018
> >>> >> > >>>>
> >>> >> > >>>> If any features in branch which are targeted to 3.2.0, please
> >>> >> reply to
> >>> >> > >>>> this email thread.
> >>> >> > >>>>
> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
> >>> >> > >>>>
> >>> >> > >>>> 1. Merged & Completed features:
> >>> >> > >>>>
> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> DeepLearning
> >>> >> > >>>> workloads Initial cut.
> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> Capacity
> >>> >> > >>>> Scheduler.
> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> >> Service
> >>> >> > >>>> API and CLI.
> >>> >> > >>>>
> >>> >> > >>>> 2. Features close to finish:
> >>> >> > >>>>
> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> >> Merge/Vote
> >>> >> > >>>> Ongoing.
> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> apps
> >>> from
> >>> >> > >>>> ATSv2. Patch in progress.
> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> Improvement
> >>> >> > works.
> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
> Azure
> >>> >> > >>>> Storage. In progress.
> >>> >> > >>>>
> >>> >> > >>>> 3. Tentative features:
> >>> >> > >>>>
> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging
> >>> >> > to
> >>> >> > >>>> be done before Aug 2018.
> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
> applications.
> >>> >> > >>>> Challenging as more discussions are on-going.
> >>> >> > >>>>
> >>> >> > >>>> *Summary of 3.2.0 issues status:*
> >>> >> > >>>>
> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following
> up
> >>> with
> >>> >> > >>>> owners to get status on each of them to get in by Code Freeze
> >>> date.
> >>> >> > >>>>
> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
> in
> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> >>> >> > Version/s" =
> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
> >>> >> > >>>>
> >>> >> > >>>> Thanks,
> >>> >> > >>>> Sunil
> >>> >> > >>>>
> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org>
> >>> >> wrote:
> >>> >> > >>>>
> >>> >> > >>>>> Hi All,
> >>> >> > >>>>>
> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
> >>> dated
> >>> >> > 17th
> >>> >> > >>>>> July 2018, I would like to provide some updates.
> >>> >> > >>>>>
> >>> >> > >>>>> We are approaching previously proposed feature freeze date
> >>> (Aug
> >>> >> 21,
> >>> >> > >>>>> about 7 days from today).
> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
> please
> >>> >> reply
> >>> >> > to
> >>> >> > >>>>> this email thread.
> >>> >> > >>>>> Steve has mentioned about the s3 features which will come
> >>> close to
> >>> >> > >>>>> Code Freeze Date (Aug 31st).
> >>> >> > >>>>>
> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
> >>> >> > >>>>>
> >>> >> > >>>>> 1. Merged & Completed features:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> >>> DeepLearning
> >>> >> > >>>>> workloads Initial cut.
> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>>>>
> >>> >> > >>>>> 2. Features close to finish:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> Major
> >>> >> > >>>>> patches are all in, only one last
> >>> >> > >>>>> patch is in review state.
> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> Capacity
> >>> >> > >>>>> Scheduler. Close to commit.
> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> >> Service
> >>> >> > >>>>> API and CLI. 2 patches are pending
> >>> >> > >>>>> which will be closed by Feature freeze date.
> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> apps
> >>> >> from
> >>> >> > >>>>> ATSv2. Patch in progress.
> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> Improvement
> >>> >> > >>>>> works.
> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
> >>> Azure
> >>> >> > >>>>> Storage. In progress.
> >>> >> > >>>>>
> >>> >> > >>>>> 3. Tentative features:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging
> >>> >> > >>>>> to be done before Aug 2018.
> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
> applications.
> >>> >> > >>>>> Challenging as more discussions are on-going.
> >>> >> > >>>>>
> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
> >>> >> > >>>>>
> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
> >>> with
> >>> >> > owners
> >>> >> > >>>>> to get status on each of them to get in by Code Freeze date.
> >>> >> > >>>>>
> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
> in
> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> >>> >> > Version/s" =
> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
> >>> >> > >>>>>
> >>> >> > >>>>> Thanks,
> >>> >> > >>>>> Sunil
> >>> >> > >>>>>
> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org>
> >>> >> wrote:
> >>> >> > >>>>>
> >>> >> > >>>>>> Thanks Subru for the thoughts.
> >>> >> > >>>>>> One of the main reason for a major release is to push out
> >>> >> critical
> >>> >> > >>>>>> features with a faster cadence to the users. If we are
> >>> pulling
> >>> >> more
> >>> >> > and
> >>> >> > >>>>>> more different types of features to a minor release, that
> >>> branch
> >>> >> > will
> >>> >> > >>>>>> become more destabilized and it may be tough to say that
> >>> 3.1.2 is
> >>> >> > stable
> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
> >>> >> features
> >>> >> > in
> >>> >> > >>>>>> subsequent minor release.
> >>> >> > >>>>>> For few companies, it makes sense to push out these new
> >>> features
> >>> >> > >>>>>> faster to make a reach to the users. Adding to the point to
> >>> the
> >>> >> > backporting
> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
> >>> with
> >>> >> > some git
> >>> >> > >>>>>> scripts. If we can make such scripts available to
> committers,
> >>> >> > backport will
> >>> >> > >>>>>> be seem-less across branches and we can achieve the faster
> >>> >> release
> >>> >> > cadence
> >>> >> > >>>>>> also.
> >>> >> > >>>>>>
> >>> >> > >>>>>> Thoughts?
> >>> >> > >>>>>>
> >>> >> > >>>>>> - Sunil
> >>> >> > >>>>>>
> >>> >> > >>>>>>
> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
> >>> subru@apache.org
> >>> >> >
> >>> >> > >>>>>> wrote:
> >>> >> > >>>>>>
> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release effort.
> I
> >>> am
> >>> >> > >>>>>>> generally
> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x)
> as
> >>> >> feel we
> >>> >> > >>>>>>> already
> >>> >> > >>>>>>> have too many branches to be maintained. I already see
> many
> >>> >> commits
> >>> >> > >>>>>>> are in
> >>> >> > >>>>>>> different branches with no apparent rationale, for e.g:
> 3.1
> >>> has
> >>> >> > >>>>>>> commits
> >>> >> > >>>>>>> which are absent in 3.0 etc.
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
> >>> >> > production
> >>> >> > >>>>>>> setting so the cost of adding features should be minimal.
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> Thoughts?
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> -Subru
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
> >>> sunilg@apache.org>
> >>> >> > wrote:
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Yes, important changes and features are much needed,
> >>> hence we
> >>> >> > will
> >>> >> > >>>>>>> be
> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
> >>> considering
> >>> >> few
> >>> >> > >>>>>>> more
> >>> >> > >>>>>>> > offline requests from other folks, I think extending the
> >>> >> > timeframe
> >>> >> > >>>>>>> by
> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
> >>> buffer) and
> >>> >> > >>>>>>> this should
> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki
> as
> >>> >> well)
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by August
> >>> 21,
> >>> >> 2018.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
> >>> improvements
> >>> >> and
> >>> >> > >>>>>>> non
> >>> >> > >>>>>>> > blocker/critical
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Release date: September 15, 2018
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered
> >>> your
> >>> >> > >>>>>>> questions.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Thanks
> >>> >> > >>>>>>> > Sunil
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
> >>> >> wheeleast@gmail.com
> >>> >> > >
> >>> >> > >>>>>>> wrote:
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release,
> >>> +1
> >>> >> for
> >>> >> > >>>>>>> that.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > To concerns from Steve,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
> important
> >>> >> > changes
> >>> >> > >>>>>>> /
> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
> >>> >> proposed
> >>> >> > >>>>>>> release
> >>> >> > >>>>>>> > date
> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of last
> >>> >> minute
> >>> >> > >>>>>>> and we all
> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm also
> >>> fine
> >>> >> if
> >>> >> > >>>>>>> we want
> >>> >> > >>>>>>> > get
> >>> >> > >>>>>>> > > another several weeks time.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
> >>> before
> >>> >> > >>>>>>> thanksgiving.
> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Eric,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
> >>> YARN-8135
> >>> >> can
> >>> >> > >>>>>>> get merged
> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Sunil,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Could u update
> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
> >>> >> > >>>>>>> > Roadmap
> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
> >>> first
> >>> >> > >>>>>>> before getting
> >>> >> > >>>>>>> > > consensus of time.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Thanks,
> >>> >> > >>>>>>> > > Wangda
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > wrote:
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> >>> >> > >>>>>>> stevel@hortonworks.com>
> >>> >> > >>>>>>> > >> wrote:
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
> sunilg@apache.org
> >>> >> > <mailto:
> >>> >> > >>>>>>> > >> > sunilg@apache.org>> wrote:
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I would also would like to take this opportunity to
> >>> come
> >>> >> up
> >>> >> > >>>>>>> with a
> >>> >> > >>>>>>> > >> detailed
> >>> >> > >>>>>>> > >> > plan.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
> >>> merged by
> >>> >> > >>>>>>> August 10,
> >>> >> > >>>>>>> > >> 2018.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > <snip>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
> targeted
> >>> to
> >>> >> 3.2
> >>> >> > >>>>>>> per this
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
> >>> Datalake
> >>> >> > Gen
> >>> >> > >>>>>>> 2
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
> >>> doing
> >>> >> work
> >>> >> > >>>>>>> on it, but
> >>> >> > >>>>>>> > >> if
> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
> >>> >> timely
> >>> >> > >>>>>>> reviews on
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will review
> >>> the
> >>> >> S3 &
> >>> >> > >>>>>>> S3Guard
> >>> >> > >>>>>>> > >> JIRAs
> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
> >>> some
> >>> >> > >>>>>>> pressing
> >>> >> > >>>>>>> > "guave,
> >>> >> > >>>>>>> > >> > java 9 prep"
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> I can help with this part if you like.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
> release
> >>> >> > manager
> >>> >> > >>>>>>> of 3.2.0
> >>> >> > >>>>>>> > >> > release.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > well volunteered!
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable should
> >>> we
> >>> >> have
> >>> >> > >>>>>>> for the
> >>> >> > >>>>>>> > >> 3.2. &
> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
> >>> having
> >>> >> > the
> >>> >> > >>>>>>> outline
> >>> >> > >>>>>>> > >> time
> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
> >>> less
> >>> >> > >>>>>>> concern about
> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > -Steve
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> -AF
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>>
> >>> >> > >>>>>>
> >>> >> >
> >>> >>
> >>> >
> >>>
> >>
> >>
>
--
A very happy Clouderan
Re: Hadoop 3.2 Release Plan proposal
Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
Thanks Sunil G for driving the release,
I filed HADOOP-15866 <https://issues.apache.org/jira/browse/HADOOP-15866> for
a compat fix. If any one has cycle please review it, as I think it is
needed for 3.2.0.
On Thu, Oct 18, 2018 at 4:43 AM Sunil G <su...@apache.org> wrote:
> Hi Folks,
>
> As we previously communicated for 3.2.0 release, we have delayed due to few
> blockers in our gate.
>
> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
> all bug fixes.
>
> - Sunil
>
>
> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <su...@apache.org> wrote:
>
> > Hi Folks,
> >
> > We are now close to RC as other blocker issues are now merged to trunk
> and
> > branch-3.2. Last 2 critical issues are closer to merge and will be
> > committed in few hours.
> > With this, I will be creating 3.2.0 branch today and will go ahead with
> RC
> > related process.
> >
> > - Sunil
> >
> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jo...@stripe.com>
> > wrote:
> >
> >> Hello, were there any updates around the 3.2.0 RC timing? All I see in
> >> the current blockers are related to the new Submarine subproject, wasn't
> >> sure if that is what is holding things up.
> >>
> >> Cheers,
> >> Jon
> >>
> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <su...@apache.org> wrote:
> >>
> >>> Thanks Robert and Haibo for quickly correcting same.
> >>> Sigh, I somehow missed one file while committing the change. Sorry for
> >>> the
> >>> trouble.
> >>>
> >>> - Sunil
> >>>
> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rk...@cloudera.com>
> >>> wrote:
> >>>
> >>> > Looks like there's two that weren't updated:
> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
> >>> > --include=pom.xml
> >>> > ./hadoop-project/pom.xml:
> >>> > <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
> >>> >
> >>> > I've just pushed in an addendum commit to fix those.
> >>> > In the future, please make sure to do a sanity compile when updating
> >>> poms.
> >>> >
> >>> > thanks
> >>> > - Robert
> >>> >
> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
> >>> <fa...@cloudera.com.invalid>
> >>> > wrote:
> >>> >
> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
> >>> >> top-level pom.xml?
> >>> >>
> >>> >>
> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
> >>> >>
> >>> >> > Hi All
> >>> >> >
> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
> trunk
> >>> to
> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
> >>> necessary
> >>> >> > patches are pulled into branch-3.2.
> >>> >> >
> >>> >> > Thank You
> >>> >> > - Sunil
> >>> >> >
> >>> >> >
> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org>
> wrote:
> >>> >> >
> >>> >> > > Hi All
> >>> >> > >
> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged
> to
> >>> >> trunk.
> >>> >> > > Thanks for the support.
> >>> >> > >
> >>> >> > > *Plan for RC*
> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
> >>> >> > > RC0 for 3.2: *28th Friday*
> >>> >> > >
> >>> >> > > Thank You
> >>> >> > > Sunil
> >>> >> > >
> >>> >> > >
> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org>
> >>> wrote:
> >>> >> > >
> >>> >> > >> Hi All
> >>> >> > >>
> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you
> >>> for
> >>> >> > >> helping in this. I am following up on these tickets, once its
> >>> closed
> >>> >> we
> >>> >> > >> will cut the 3.2 branch.
> >>> >> > >>
> >>> >> > >> Thanks
> >>> >> > >> Sunil Govindan
> >>> >> > >>
> >>> >> > >>
> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org>
> >>> wrote:
> >>> >> > >>
> >>> >> > >>> Hi All,
> >>> >> > >>>
> >>> >> > >>> Inline with the original 3.2 communication proposal dated 17th
> >>> July
> >>> >> > >>> 2018, I would like to provide more updates.
> >>> >> > >>>
> >>> >> > >>> We are approaching previously proposed code freeze date
> >>> (September
> >>> >> 14,
> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
> point
> >>> >> > existing
> >>> >> > >>> trunk to 3.3 if there are no issues.
> >>> >> > >>>
> >>> >> > >>> *Current Release Plan:*
> >>> >> > >>> Feature freeze date : all features to merge by September 7,
> >>> 2018.
> >>> >> > >>> Code freeze date : blockers/critical only, no improvements and
> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
> >>> >> > >>> Release date: September 28, 2018
> >>> >> > >>>
> >>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0,
> we
> >>> >> need to
> >>> >> > >>> backport to 3.2 post branch cut.
> >>> >> > >>>
> >>> >> > >>> Here's an updated 3.2.0 feature status:
> >>> >> > >>>
> >>> >> > >>> 1. Merged & Completed features:
> >>> >> > >>>
> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
> DeepLearning
> >>> >> > >>> workloads Initial cut.
> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >>> >> > >>> Scheduler.
> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> Service
> >>> >> > API
> >>> >> > >>> and CLI.
> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
> Improvement
> >>> >> works.
> >>> >> > >>>
> >>> >> > >>> 2. Features close to finish:
> >>> >> > >>>
> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
> >>> >> > >>>
> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
> >>> from
> >>> >> > >>> ATSv2. Patch in progress.
> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging to
> >>> >> > >>> be done before Aug 2018.
> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >>> >> > >>> Challenging as more discussions are on-going.
> >>> >> > >>>
> >>> >> > >>> *Summary of 3.2.0 issues status:*
> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following up
> >>> with
> >>> >> > >>> owners to get status on each of them to get in by Code Freeze
> >>> date.
> >>> >> > >>>
> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> >>> >> > (Blocker,
> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" =
> >>> 3.2.0
> >>> >> > ORDER
> >>> >> > >>> BY priority DESC
> >>> >> > >>>
> >>> >> > >>> Thanks,
> >>> >> > >>> Sunil
> >>> >> > >>>
> >>> >> > >>>
> >>> >> > >>>
> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org>
> >>> wrote:
> >>> >> > >>>
> >>> >> > >>>> Hi All,
> >>> >> > >>>>
> >>> >> > >>>> Inline with earlier communication dated 17th July 2018, I
> would
> >>> >> like
> >>> >> > to
> >>> >> > >>>> provide some updates.
> >>> >> > >>>>
> >>> >> > >>>> We are approaching previously proposed code freeze date (Aug
> >>> 31).
> >>> >> > >>>>
> >>> >> > >>>> One of the critical feature Node Attributes feature merge
> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need
> a
> >>> bit
> >>> >> > more
> >>> >> > >>>> time. With regard to this, suggesting to push the
> feature/code
> >>> >> freeze
> >>> >> > for 2
> >>> >> > >>>> more weeks to accommodate these jiras too.
> >>> >> > >>>>
> >>> >> > >>>> Proposing Updated changes in plan inline with this:
> >>> >> > >>>> Feature freeze date : all features to merge by September 7,
> >>> 2018.
> >>> >> > >>>> Code freeze date : blockers/critical only, no improvements
> and
> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
> >>> >> > >>>> Release date: September 28, 2018
> >>> >> > >>>>
> >>> >> > >>>> If any features in branch which are targeted to 3.2.0, please
> >>> >> reply to
> >>> >> > >>>> this email thread.
> >>> >> > >>>>
> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
> >>> >> > >>>>
> >>> >> > >>>> 1. Merged & Completed features:
> >>> >> > >>>>
> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> DeepLearning
> >>> >> > >>>> workloads Initial cut.
> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> Capacity
> >>> >> > >>>> Scheduler.
> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> >> Service
> >>> >> > >>>> API and CLI.
> >>> >> > >>>>
> >>> >> > >>>> 2. Features close to finish:
> >>> >> > >>>>
> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> >> Merge/Vote
> >>> >> > >>>> Ongoing.
> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> apps
> >>> from
> >>> >> > >>>> ATSv2. Patch in progress.
> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> Improvement
> >>> >> > works.
> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
> Azure
> >>> >> > >>>> Storage. In progress.
> >>> >> > >>>>
> >>> >> > >>>> 3. Tentative features:
> >>> >> > >>>>
> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging
> >>> >> > to
> >>> >> > >>>> be done before Aug 2018.
> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
> applications.
> >>> >> > >>>> Challenging as more discussions are on-going.
> >>> >> > >>>>
> >>> >> > >>>> *Summary of 3.2.0 issues status:*
> >>> >> > >>>>
> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following
> up
> >>> with
> >>> >> > >>>> owners to get status on each of them to get in by Code Freeze
> >>> date.
> >>> >> > >>>>
> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
> in
> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> >>> >> > Version/s" =
> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
> >>> >> > >>>>
> >>> >> > >>>> Thanks,
> >>> >> > >>>> Sunil
> >>> >> > >>>>
> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org>
> >>> >> wrote:
> >>> >> > >>>>
> >>> >> > >>>>> Hi All,
> >>> >> > >>>>>
> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
> >>> dated
> >>> >> > 17th
> >>> >> > >>>>> July 2018, I would like to provide some updates.
> >>> >> > >>>>>
> >>> >> > >>>>> We are approaching previously proposed feature freeze date
> >>> (Aug
> >>> >> 21,
> >>> >> > >>>>> about 7 days from today).
> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
> please
> >>> >> reply
> >>> >> > to
> >>> >> > >>>>> this email thread.
> >>> >> > >>>>> Steve has mentioned about the s3 features which will come
> >>> close to
> >>> >> > >>>>> Code Freeze Date (Aug 31st).
> >>> >> > >>>>>
> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
> >>> >> > >>>>>
> >>> >> > >>>>> 1. Merged & Completed features:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> >>> DeepLearning
> >>> >> > >>>>> workloads Initial cut.
> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>>>>
> >>> >> > >>>>> 2. Features close to finish:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> Major
> >>> >> > >>>>> patches are all in, only one last
> >>> >> > >>>>> patch is in review state.
> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> Capacity
> >>> >> > >>>>> Scheduler. Close to commit.
> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> >> Service
> >>> >> > >>>>> API and CLI. 2 patches are pending
> >>> >> > >>>>> which will be closed by Feature freeze date.
> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> apps
> >>> >> from
> >>> >> > >>>>> ATSv2. Patch in progress.
> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> Improvement
> >>> >> > >>>>> works.
> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
> >>> Azure
> >>> >> > >>>>> Storage. In progress.
> >>> >> > >>>>>
> >>> >> > >>>>> 3. Tentative features:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging
> >>> >> > >>>>> to be done before Aug 2018.
> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
> applications.
> >>> >> > >>>>> Challenging as more discussions are on-going.
> >>> >> > >>>>>
> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
> >>> >> > >>>>>
> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
> >>> with
> >>> >> > owners
> >>> >> > >>>>> to get status on each of them to get in by Code Freeze date.
> >>> >> > >>>>>
> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
> in
> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> >>> >> > Version/s" =
> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
> >>> >> > >>>>>
> >>> >> > >>>>> Thanks,
> >>> >> > >>>>> Sunil
> >>> >> > >>>>>
> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org>
> >>> >> wrote:
> >>> >> > >>>>>
> >>> >> > >>>>>> Thanks Subru for the thoughts.
> >>> >> > >>>>>> One of the main reason for a major release is to push out
> >>> >> critical
> >>> >> > >>>>>> features with a faster cadence to the users. If we are
> >>> pulling
> >>> >> more
> >>> >> > and
> >>> >> > >>>>>> more different types of features to a minor release, that
> >>> branch
> >>> >> > will
> >>> >> > >>>>>> become more destabilized and it may be tough to say that
> >>> 3.1.2 is
> >>> >> > stable
> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
> >>> >> features
> >>> >> > in
> >>> >> > >>>>>> subsequent minor release.
> >>> >> > >>>>>> For few companies, it makes sense to push out these new
> >>> features
> >>> >> > >>>>>> faster to make a reach to the users. Adding to the point to
> >>> the
> >>> >> > backporting
> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
> >>> with
> >>> >> > some git
> >>> >> > >>>>>> scripts. If we can make such scripts available to
> committers,
> >>> >> > backport will
> >>> >> > >>>>>> be seem-less across branches and we can achieve the faster
> >>> >> release
> >>> >> > cadence
> >>> >> > >>>>>> also.
> >>> >> > >>>>>>
> >>> >> > >>>>>> Thoughts?
> >>> >> > >>>>>>
> >>> >> > >>>>>> - Sunil
> >>> >> > >>>>>>
> >>> >> > >>>>>>
> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
> >>> subru@apache.org
> >>> >> >
> >>> >> > >>>>>> wrote:
> >>> >> > >>>>>>
> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release effort.
> I
> >>> am
> >>> >> > >>>>>>> generally
> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x)
> as
> >>> >> feel we
> >>> >> > >>>>>>> already
> >>> >> > >>>>>>> have too many branches to be maintained. I already see
> many
> >>> >> commits
> >>> >> > >>>>>>> are in
> >>> >> > >>>>>>> different branches with no apparent rationale, for e.g:
> 3.1
> >>> has
> >>> >> > >>>>>>> commits
> >>> >> > >>>>>>> which are absent in 3.0 etc.
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
> >>> >> > production
> >>> >> > >>>>>>> setting so the cost of adding features should be minimal.
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> Thoughts?
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> -Subru
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
> >>> sunilg@apache.org>
> >>> >> > wrote:
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Yes, important changes and features are much needed,
> >>> hence we
> >>> >> > will
> >>> >> > >>>>>>> be
> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
> >>> considering
> >>> >> few
> >>> >> > >>>>>>> more
> >>> >> > >>>>>>> > offline requests from other folks, I think extending the
> >>> >> > timeframe
> >>> >> > >>>>>>> by
> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
> >>> buffer) and
> >>> >> > >>>>>>> this should
> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki
> as
> >>> >> well)
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by August
> >>> 21,
> >>> >> 2018.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
> >>> improvements
> >>> >> and
> >>> >> > >>>>>>> non
> >>> >> > >>>>>>> > blocker/critical
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Release date: September 15, 2018
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered
> >>> your
> >>> >> > >>>>>>> questions.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Thanks
> >>> >> > >>>>>>> > Sunil
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
> >>> >> wheeleast@gmail.com
> >>> >> > >
> >>> >> > >>>>>>> wrote:
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release,
> >>> +1
> >>> >> for
> >>> >> > >>>>>>> that.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > To concerns from Steve,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
> important
> >>> >> > changes
> >>> >> > >>>>>>> /
> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
> >>> >> proposed
> >>> >> > >>>>>>> release
> >>> >> > >>>>>>> > date
> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of last
> >>> >> minute
> >>> >> > >>>>>>> and we all
> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm also
> >>> fine
> >>> >> if
> >>> >> > >>>>>>> we want
> >>> >> > >>>>>>> > get
> >>> >> > >>>>>>> > > another several weeks time.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
> >>> before
> >>> >> > >>>>>>> thanksgiving.
> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Eric,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
> >>> YARN-8135
> >>> >> can
> >>> >> > >>>>>>> get merged
> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Sunil,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Could u update
> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
> >>> >> > >>>>>>> > Roadmap
> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
> >>> first
> >>> >> > >>>>>>> before getting
> >>> >> > >>>>>>> > > consensus of time.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Thanks,
> >>> >> > >>>>>>> > > Wangda
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > wrote:
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> >>> >> > >>>>>>> stevel@hortonworks.com>
> >>> >> > >>>>>>> > >> wrote:
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
> sunilg@apache.org
> >>> >> > <mailto:
> >>> >> > >>>>>>> > >> > sunilg@apache.org>> wrote:
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I would also would like to take this opportunity to
> >>> come
> >>> >> up
> >>> >> > >>>>>>> with a
> >>> >> > >>>>>>> > >> detailed
> >>> >> > >>>>>>> > >> > plan.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
> >>> merged by
> >>> >> > >>>>>>> August 10,
> >>> >> > >>>>>>> > >> 2018.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > <snip>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
> targeted
> >>> to
> >>> >> 3.2
> >>> >> > >>>>>>> per this
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
> >>> Datalake
> >>> >> > Gen
> >>> >> > >>>>>>> 2
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
> >>> doing
> >>> >> work
> >>> >> > >>>>>>> on it, but
> >>> >> > >>>>>>> > >> if
> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
> >>> >> timely
> >>> >> > >>>>>>> reviews on
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will review
> >>> the
> >>> >> S3 &
> >>> >> > >>>>>>> S3Guard
> >>> >> > >>>>>>> > >> JIRAs
> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
> >>> some
> >>> >> > >>>>>>> pressing
> >>> >> > >>>>>>> > "guave,
> >>> >> > >>>>>>> > >> > java 9 prep"
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> I can help with this part if you like.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
> release
> >>> >> > manager
> >>> >> > >>>>>>> of 3.2.0
> >>> >> > >>>>>>> > >> > release.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > well volunteered!
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable should
> >>> we
> >>> >> have
> >>> >> > >>>>>>> for the
> >>> >> > >>>>>>> > >> 3.2. &
> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
> >>> having
> >>> >> > the
> >>> >> > >>>>>>> outline
> >>> >> > >>>>>>> > >> time
> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
> >>> less
> >>> >> > >>>>>>> concern about
> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > -Steve
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> -AF
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>>
> >>> >> > >>>>>>
> >>> >> >
> >>> >>
> >>> >
> >>>
> >>
> >>
>
--
A very happy Clouderan
Re: Hadoop 3.2 Release Plan proposal
Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
Thanks Sunil G for driving the release,
I filed HADOOP-15866 <https://issues.apache.org/jira/browse/HADOOP-15866> for
a compat fix. If any one has cycle please review it, as I think it is
needed for 3.2.0.
On Thu, Oct 18, 2018 at 4:43 AM Sunil G <su...@apache.org> wrote:
> Hi Folks,
>
> As we previously communicated for 3.2.0 release, we have delayed due to few
> blockers in our gate.
>
> I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
> all bug fixes.
>
> - Sunil
>
>
> On Tue, Oct 16, 2018 at 8:59 AM Sunil G <su...@apache.org> wrote:
>
> > Hi Folks,
> >
> > We are now close to RC as other blocker issues are now merged to trunk
> and
> > branch-3.2. Last 2 critical issues are closer to merge and will be
> > committed in few hours.
> > With this, I will be creating 3.2.0 branch today and will go ahead with
> RC
> > related process.
> >
> > - Sunil
> >
> > On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jo...@stripe.com>
> > wrote:
> >
> >> Hello, were there any updates around the 3.2.0 RC timing? All I see in
> >> the current blockers are related to the new Submarine subproject, wasn't
> >> sure if that is what is holding things up.
> >>
> >> Cheers,
> >> Jon
> >>
> >> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <su...@apache.org> wrote:
> >>
> >>> Thanks Robert and Haibo for quickly correcting same.
> >>> Sigh, I somehow missed one file while committing the change. Sorry for
> >>> the
> >>> trouble.
> >>>
> >>> - Sunil
> >>>
> >>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rk...@cloudera.com>
> >>> wrote:
> >>>
> >>> > Looks like there's two that weren't updated:
> >>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
> >>> > --include=pom.xml
> >>> > ./hadoop-project/pom.xml:
> >>> > <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
> >>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
> >>> >
> >>> > I've just pushed in an addendum commit to fix those.
> >>> > In the future, please make sure to do a sanity compile when updating
> >>> poms.
> >>> >
> >>> > thanks
> >>> > - Robert
> >>> >
> >>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
> >>> <fa...@cloudera.com.invalid>
> >>> > wrote:
> >>> >
> >>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
> >>> >> top-level pom.xml?
> >>> >>
> >>> >>
> >>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
> >>> >>
> >>> >> > Hi All
> >>> >> >
> >>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset
> trunk
> >>> to
> >>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
> >>> necessary
> >>> >> > patches are pulled into branch-3.2.
> >>> >> >
> >>> >> > Thank You
> >>> >> > - Sunil
> >>> >> >
> >>> >> >
> >>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org>
> wrote:
> >>> >> >
> >>> >> > > Hi All
> >>> >> > >
> >>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged
> to
> >>> >> trunk.
> >>> >> > > Thanks for the support.
> >>> >> > >
> >>> >> > > *Plan for RC*
> >>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
> >>> >> > > RC0 for 3.2: *28th Friday*
> >>> >> > >
> >>> >> > > Thank You
> >>> >> > > Sunil
> >>> >> > >
> >>> >> > >
> >>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org>
> >>> wrote:
> >>> >> > >
> >>> >> > >> Hi All
> >>> >> > >>
> >>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you
> >>> for
> >>> >> > >> helping in this. I am following up on these tickets, once its
> >>> closed
> >>> >> we
> >>> >> > >> will cut the 3.2 branch.
> >>> >> > >>
> >>> >> > >> Thanks
> >>> >> > >> Sunil Govindan
> >>> >> > >>
> >>> >> > >>
> >>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org>
> >>> wrote:
> >>> >> > >>
> >>> >> > >>> Hi All,
> >>> >> > >>>
> >>> >> > >>> Inline with the original 3.2 communication proposal dated 17th
> >>> July
> >>> >> > >>> 2018, I would like to provide more updates.
> >>> >> > >>>
> >>> >> > >>> We are approaching previously proposed code freeze date
> >>> (September
> >>> >> 14,
> >>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and
> point
> >>> >> > existing
> >>> >> > >>> trunk to 3.3 if there are no issues.
> >>> >> > >>>
> >>> >> > >>> *Current Release Plan:*
> >>> >> > >>> Feature freeze date : all features to merge by September 7,
> >>> 2018.
> >>> >> > >>> Code freeze date : blockers/critical only, no improvements and
> >>> >> > >>> blocker/critical bug-fixes September 14, 2018.
> >>> >> > >>> Release date: September 28, 2018
> >>> >> > >>>
> >>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0,
> we
> >>> >> need to
> >>> >> > >>> backport to 3.2 post branch cut.
> >>> >> > >>>
> >>> >> > >>> Here's an updated 3.2.0 feature status:
> >>> >> > >>>
> >>> >> > >>> 1. Merged & Completed features:
> >>> >> > >>>
> >>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for
> DeepLearning
> >>> >> > >>> workloads Initial cut.
> >>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >>> >> > >>> Scheduler.
> >>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> Service
> >>> >> > API
> >>> >> > >>> and CLI.
> >>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation.
> Improvement
> >>> >> works.
> >>> >> > >>>
> >>> >> > >>> 2. Features close to finish:
> >>> >> > >>>
> >>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
> >>> >> > >>> - (Steve) S3a phase V. Close to commit.
> >>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
> >>> >> > >>>
> >>> >> > >>> 3. Tentative/Cancelled features for 3.2:
> >>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
> >>> from
> >>> >> > >>> ATSv2. Patch in progress.
> >>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging to
> >>> >> > >>> be done before Aug 2018.
> >>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >>> >> > >>> Challenging as more discussions are on-going.
> >>> >> > >>>
> >>> >> > >>> *Summary of 3.2.0 issues status:*
> >>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following up
> >>> with
> >>> >> > >>> owners to get status on each of them to get in by Code Freeze
> >>> date.
> >>> >> > >>>
> >>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> >>> >> > (Blocker,
> >>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" =
> >>> 3.2.0
> >>> >> > ORDER
> >>> >> > >>> BY priority DESC
> >>> >> > >>>
> >>> >> > >>> Thanks,
> >>> >> > >>> Sunil
> >>> >> > >>>
> >>> >> > >>>
> >>> >> > >>>
> >>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org>
> >>> wrote:
> >>> >> > >>>
> >>> >> > >>>> Hi All,
> >>> >> > >>>>
> >>> >> > >>>> Inline with earlier communication dated 17th July 2018, I
> would
> >>> >> like
> >>> >> > to
> >>> >> > >>>> provide some updates.
> >>> >> > >>>>
> >>> >> > >>>> We are approaching previously proposed code freeze date (Aug
> >>> 31).
> >>> >> > >>>>
> >>> >> > >>>> One of the critical feature Node Attributes feature merge
> >>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need
> a
> >>> bit
> >>> >> > more
> >>> >> > >>>> time. With regard to this, suggesting to push the
> feature/code
> >>> >> freeze
> >>> >> > for 2
> >>> >> > >>>> more weeks to accommodate these jiras too.
> >>> >> > >>>>
> >>> >> > >>>> Proposing Updated changes in plan inline with this:
> >>> >> > >>>> Feature freeze date : all features to merge by September 7,
> >>> 2018.
> >>> >> > >>>> Code freeze date : blockers/critical only, no improvements
> and
> >>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
> >>> >> > >>>> Release date: September 28, 2018
> >>> >> > >>>>
> >>> >> > >>>> If any features in branch which are targeted to 3.2.0, please
> >>> >> reply to
> >>> >> > >>>> this email thread.
> >>> >> > >>>>
> >>> >> > >>>> *Here's an updated 3.2.0 feature status:*
> >>> >> > >>>>
> >>> >> > >>>> 1. Merged & Completed features:
> >>> >> > >>>>
> >>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> DeepLearning
> >>> >> > >>>> workloads Initial cut.
> >>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> Capacity
> >>> >> > >>>> Scheduler.
> >>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> >> Service
> >>> >> > >>>> API and CLI.
> >>> >> > >>>>
> >>> >> > >>>> 2. Features close to finish:
> >>> >> > >>>>
> >>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> >> Merge/Vote
> >>> >> > >>>> Ongoing.
> >>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> apps
> >>> from
> >>> >> > >>>> ATSv2. Patch in progress.
> >>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> Improvement
> >>> >> > works.
> >>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
> Azure
> >>> >> > >>>> Storage. In progress.
> >>> >> > >>>>
> >>> >> > >>>> 3. Tentative features:
> >>> >> > >>>>
> >>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging
> >>> >> > to
> >>> >> > >>>> be done before Aug 2018.
> >>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN
> applications.
> >>> >> > >>>> Challenging as more discussions are on-going.
> >>> >> > >>>>
> >>> >> > >>>> *Summary of 3.2.0 issues status:*
> >>> >> > >>>>
> >>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following
> up
> >>> with
> >>> >> > >>>> owners to get status on each of them to get in by Code Freeze
> >>> date.
> >>> >> > >>>>
> >>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
> in
> >>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> >>> >> > Version/s" =
> >>> >> > >>>> 3.2.0 ORDER BY priority DESC
> >>> >> > >>>>
> >>> >> > >>>> Thanks,
> >>> >> > >>>> Sunil
> >>> >> > >>>>
> >>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org>
> >>> >> wrote:
> >>> >> > >>>>
> >>> >> > >>>>> Hi All,
> >>> >> > >>>>>
> >>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
> >>> dated
> >>> >> > 17th
> >>> >> > >>>>> July 2018, I would like to provide some updates.
> >>> >> > >>>>>
> >>> >> > >>>>> We are approaching previously proposed feature freeze date
> >>> (Aug
> >>> >> 21,
> >>> >> > >>>>> about 7 days from today).
> >>> >> > >>>>> If any features in branch which are targeted to 3.2.0,
> please
> >>> >> reply
> >>> >> > to
> >>> >> > >>>>> this email thread.
> >>> >> > >>>>> Steve has mentioned about the s3 features which will come
> >>> close to
> >>> >> > >>>>> Code Freeze Date (Aug 31st).
> >>> >> > >>>>>
> >>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
> >>> >> > >>>>>
> >>> >> > >>>>> 1. Merged & Completed features:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
> >>> DeepLearning
> >>> >> > >>>>> workloads Initial cut.
> >>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> >> > >>>>>
> >>> >> > >>>>> 2. Features close to finish:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> Major
> >>> >> > >>>>> patches are all in, only one last
> >>> >> > >>>>> patch is in review state.
> >>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in
> Capacity
> >>> >> > >>>>> Scheduler. Close to commit.
> >>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >>> >> Service
> >>> >> > >>>>> API and CLI. 2 patches are pending
> >>> >> > >>>>> which will be closed by Feature freeze date.
> >>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical
> apps
> >>> >> from
> >>> >> > >>>>> ATSv2. Patch in progress.
> >>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> >>> Improvement
> >>> >> > >>>>> works.
> >>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
> >>> Azure
> >>> >> > >>>>> Storage. In progress.
> >>> >> > >>>>>
> >>> >> > >>>>> 3. Tentative features:
> >>> >> > >>>>>
> >>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >>> >> challenging
> >>> >> > >>>>> to be done before Aug 2018.
> >>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN
> applications.
> >>> >> > >>>>> Challenging as more discussions are on-going.
> >>> >> > >>>>>
> >>> >> > >>>>> *Summary of 3.2.0 issues status:*
> >>> >> > >>>>>
> >>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
> >>> with
> >>> >> > owners
> >>> >> > >>>>> to get status on each of them to get in by Code Freeze date.
> >>> >> > >>>>>
> >>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority
> in
> >>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> >>> >> > Version/s" =
> >>> >> > >>>>> 3.2.0 ORDER BY priority DESC
> >>> >> > >>>>>
> >>> >> > >>>>> Thanks,
> >>> >> > >>>>> Sunil
> >>> >> > >>>>>
> >>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org>
> >>> >> wrote:
> >>> >> > >>>>>
> >>> >> > >>>>>> Thanks Subru for the thoughts.
> >>> >> > >>>>>> One of the main reason for a major release is to push out
> >>> >> critical
> >>> >> > >>>>>> features with a faster cadence to the users. If we are
> >>> pulling
> >>> >> more
> >>> >> > and
> >>> >> > >>>>>> more different types of features to a minor release, that
> >>> branch
> >>> >> > will
> >>> >> > >>>>>> become more destabilized and it may be tough to say that
> >>> 3.1.2 is
> >>> >> > stable
> >>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
> >>> >> features
> >>> >> > in
> >>> >> > >>>>>> subsequent minor release.
> >>> >> > >>>>>> For few companies, it makes sense to push out these new
> >>> features
> >>> >> > >>>>>> faster to make a reach to the users. Adding to the point to
> >>> the
> >>> >> > backporting
> >>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
> >>> with
> >>> >> > some git
> >>> >> > >>>>>> scripts. If we can make such scripts available to
> committers,
> >>> >> > backport will
> >>> >> > >>>>>> be seem-less across branches and we can achieve the faster
> >>> >> release
> >>> >> > cadence
> >>> >> > >>>>>> also.
> >>> >> > >>>>>>
> >>> >> > >>>>>> Thoughts?
> >>> >> > >>>>>>
> >>> >> > >>>>>> - Sunil
> >>> >> > >>>>>>
> >>> >> > >>>>>>
> >>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
> >>> subru@apache.org
> >>> >> >
> >>> >> > >>>>>> wrote:
> >>> >> > >>>>>>
> >>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release effort.
> I
> >>> am
> >>> >> > >>>>>>> generally
> >>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x)
> as
> >>> >> feel we
> >>> >> > >>>>>>> already
> >>> >> > >>>>>>> have too many branches to be maintained. I already see
> many
> >>> >> commits
> >>> >> > >>>>>>> are in
> >>> >> > >>>>>>> different branches with no apparent rationale, for e.g:
> 3.1
> >>> has
> >>> >> > >>>>>>> commits
> >>> >> > >>>>>>> which are absent in 3.0 etc.
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
> >>> >> > production
> >>> >> > >>>>>>> setting so the cost of adding features should be minimal.
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> Thoughts?
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> -Subru
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
> >>> sunilg@apache.org>
> >>> >> > wrote:
> >>> >> > >>>>>>>
> >>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Yes, important changes and features are much needed,
> >>> hence we
> >>> >> > will
> >>> >> > >>>>>>> be
> >>> >> > >>>>>>> > keeping the door open for them as possible. Also
> >>> considering
> >>> >> few
> >>> >> > >>>>>>> more
> >>> >> > >>>>>>> > offline requests from other folks, I think extending the
> >>> >> > timeframe
> >>> >> > >>>>>>> by
> >>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
> >>> buffer) and
> >>> >> > >>>>>>> this should
> >>> >> > >>>>>>> > ideally help us to ship this by September itself.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki
> as
> >>> >> well)
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Feature freeze date : all features to merge by August
> >>> 21,
> >>> >> 2018.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
> >>> improvements
> >>> >> and
> >>> >> > >>>>>>> non
> >>> >> > >>>>>>> > blocker/critical
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > bug-fixes August 31, 2018.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > - Release date: September 15, 2018
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered
> >>> your
> >>> >> > >>>>>>> questions.
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > Thanks
> >>> >> > >>>>>>> > Sunil
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
> >>> >> wheeleast@gmail.com
> >>> >> > >
> >>> >> > >>>>>>> wrote:
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release,
> >>> +1
> >>> >> for
> >>> >> > >>>>>>> that.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > To concerns from Steve,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > It is a good idea to keep the door open to get
> important
> >>> >> > changes
> >>> >> > >>>>>>> /
> >>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
> >>> >> proposed
> >>> >> > >>>>>>> release
> >>> >> > >>>>>>> > date
> >>> >> > >>>>>>> > > to make sure things can happen earlier instead of last
> >>> >> minute
> >>> >> > >>>>>>> and we all
> >>> >> > >>>>>>> > > know that releases are always get delayed :). I'm also
> >>> fine
> >>> >> if
> >>> >> > >>>>>>> we want
> >>> >> > >>>>>>> > get
> >>> >> > >>>>>>> > > another several weeks time.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
> >>> before
> >>> >> > >>>>>>> thanksgiving.
> >>> >> > >>>>>>> > > Do you think is it good or too early / late?
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Eric,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
> >>> YARN-8135
> >>> >> can
> >>> >> > >>>>>>> get merged
> >>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Sunil,
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Could u update
> >>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
> >>> >> > >>>>>>> > Roadmap
> >>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
> >>> first
> >>> >> > >>>>>>> before getting
> >>> >> > >>>>>>> > > consensus of time.
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > Thanks,
> >>> >> > >>>>>>> > > Wangda
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> >>> >> > >>>>>>> <fabbri@cloudera.com.invalid
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > > wrote:
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> >>> >> > >>>>>>> stevel@hortonworks.com>
> >>> >> > >>>>>>> > >> wrote:
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <
> sunilg@apache.org
> >>> >> > <mailto:
> >>> >> > >>>>>>> > >> > sunilg@apache.org>> wrote:
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I would also would like to take this opportunity to
> >>> come
> >>> >> up
> >>> >> > >>>>>>> with a
> >>> >> > >>>>>>> > >> detailed
> >>> >> > >>>>>>> > >> > plan.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
> >>> merged by
> >>> >> > >>>>>>> August 10,
> >>> >> > >>>>>>> > >> 2018.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > <snip>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > Please let me know if I missed any features
> targeted
> >>> to
> >>> >> 3.2
> >>> >> > >>>>>>> per this
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
> >>> Datalake
> >>> >> > Gen
> >>> >> > >>>>>>> 2
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
> >>> doing
> >>> >> work
> >>> >> > >>>>>>> on it, but
> >>> >> > >>>>>>> > >> if
> >>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
> >>> >> timely
> >>> >> > >>>>>>> reviews on
> >>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will review
> >>> the
> >>> >> S3 &
> >>> >> > >>>>>>> S3Guard
> >>> >> > >>>>>>> > >> JIRAs
> >>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
> >>> some
> >>> >> > >>>>>>> pressing
> >>> >> > >>>>>>> > "guave,
> >>> >> > >>>>>>> > >> > java 9 prep"
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> I can help with this part if you like.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as
> release
> >>> >> > manager
> >>> >> > >>>>>>> of 3.2.0
> >>> >> > >>>>>>> > >> > release.
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > well volunteered!
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> Yes, thank you for stepping up.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > I think this raises a good q: what timetable should
> >>> we
> >>> >> have
> >>> >> > >>>>>>> for the
> >>> >> > >>>>>>> > >> 3.2. &
> >>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
> >>> having
> >>> >> > the
> >>> >> > >>>>>>> outline
> >>> >> > >>>>>>> > >> time
> >>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
> >>> less
> >>> >> > >>>>>>> concern about
> >>> >> > >>>>>>> > >> > things not making the 3.2 dealine
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> > -Steve
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> >
> >>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >> -AF
> >>> >> > >>>>>>> > >>
> >>> >> > >>>>>>> > >
> >>> >> > >>>>>>> >
> >>> >> > >>>>>>>
> >>> >> > >>>>>>
> >>> >> >
> >>> >>
> >>> >
> >>>
> >>
> >>
>
--
A very happy Clouderan
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi Folks,
As we previously communicated for 3.2.0 release, we have delayed due to few
blockers in our gate.
I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
all bug fixes.
- Sunil
On Tue, Oct 16, 2018 at 8:59 AM Sunil G <su...@apache.org> wrote:
> Hi Folks,
>
> We are now close to RC as other blocker issues are now merged to trunk and
> branch-3.2. Last 2 critical issues are closer to merge and will be
> committed in few hours.
> With this, I will be creating 3.2.0 branch today and will go ahead with RC
> related process.
>
> - Sunil
>
> On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jo...@stripe.com>
> wrote:
>
>> Hello, were there any updates around the 3.2.0 RC timing? All I see in
>> the current blockers are related to the new Submarine subproject, wasn't
>> sure if that is what is holding things up.
>>
>> Cheers,
>> Jon
>>
>> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <su...@apache.org> wrote:
>>
>>> Thanks Robert and Haibo for quickly correcting same.
>>> Sigh, I somehow missed one file while committing the change. Sorry for
>>> the
>>> trouble.
>>>
>>> - Sunil
>>>
>>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rk...@cloudera.com>
>>> wrote:
>>>
>>> > Looks like there's two that weren't updated:
>>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
>>> > --include=pom.xml
>>> > ./hadoop-project/pom.xml:
>>> > <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>>> >
>>> > I've just pushed in an addendum commit to fix those.
>>> > In the future, please make sure to do a sanity compile when updating
>>> poms.
>>> >
>>> > thanks
>>> > - Robert
>>> >
>>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>>> <fa...@cloudera.com.invalid>
>>> > wrote:
>>> >
>>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
>>> >> top-level pom.xml?
>>> >>
>>> >>
>>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
>>> >>
>>> >> > Hi All
>>> >> >
>>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset trunk
>>> to
>>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
>>> necessary
>>> >> > patches are pulled into branch-3.2.
>>> >> >
>>> >> > Thank You
>>> >> > - Sunil
>>> >> >
>>> >> >
>>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
>>> >> >
>>> >> > > Hi All
>>> >> > >
>>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged to
>>> >> trunk.
>>> >> > > Thanks for the support.
>>> >> > >
>>> >> > > *Plan for RC*
>>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>>> >> > > RC0 for 3.2: *28th Friday*
>>> >> > >
>>> >> > > Thank You
>>> >> > > Sunil
>>> >> > >
>>> >> > >
>>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org>
>>> wrote:
>>> >> > >
>>> >> > >> Hi All
>>> >> > >>
>>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you
>>> for
>>> >> > >> helping in this. I am following up on these tickets, once its
>>> closed
>>> >> we
>>> >> > >> will cut the 3.2 branch.
>>> >> > >>
>>> >> > >> Thanks
>>> >> > >> Sunil Govindan
>>> >> > >>
>>> >> > >>
>>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org>
>>> wrote:
>>> >> > >>
>>> >> > >>> Hi All,
>>> >> > >>>
>>> >> > >>> Inline with the original 3.2 communication proposal dated 17th
>>> July
>>> >> > >>> 2018, I would like to provide more updates.
>>> >> > >>>
>>> >> > >>> We are approaching previously proposed code freeze date
>>> (September
>>> >> 14,
>>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
>>> >> > existing
>>> >> > >>> trunk to 3.3 if there are no issues.
>>> >> > >>>
>>> >> > >>> *Current Release Plan:*
>>> >> > >>> Feature freeze date : all features to merge by September 7,
>>> 2018.
>>> >> > >>> Code freeze date : blockers/critical only, no improvements and
>>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>>> >> > >>> Release date: September 28, 2018
>>> >> > >>>
>>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0, we
>>> >> need to
>>> >> > >>> backport to 3.2 post branch cut.
>>> >> > >>>
>>> >> > >>> Here's an updated 3.2.0 feature status:
>>> >> > >>>
>>> >> > >>> 1. Merged & Completed features:
>>> >> > >>>
>>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> >> > >>> workloads Initial cut.
>>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>> Scheduler.
>>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> Service
>>> >> > API
>>> >> > >>> and CLI.
>>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement
>>> >> works.
>>> >> > >>>
>>> >> > >>> 2. Features close to finish:
>>> >> > >>>
>>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>>> >> > >>> - (Steve) S3a phase V. Close to commit.
>>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>>> >> > >>>
>>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> from
>>> >> > >>> ATSv2. Patch in progress.
>>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging to
>>> >> > >>> be done before Aug 2018.
>>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>> Challenging as more discussions are on-going.
>>> >> > >>>
>>> >> > >>> *Summary of 3.2.0 issues status:*
>>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following up
>>> with
>>> >> > >>> owners to get status on each of them to get in by Code Freeze
>>> date.
>>> >> > >>>
>>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > (Blocker,
>>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" =
>>> 3.2.0
>>> >> > ORDER
>>> >> > >>> BY priority DESC
>>> >> > >>>
>>> >> > >>> Thanks,
>>> >> > >>> Sunil
>>> >> > >>>
>>> >> > >>>
>>> >> > >>>
>>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org>
>>> wrote:
>>> >> > >>>
>>> >> > >>>> Hi All,
>>> >> > >>>>
>>> >> > >>>> Inline with earlier communication dated 17th July 2018, I would
>>> >> like
>>> >> > to
>>> >> > >>>> provide some updates.
>>> >> > >>>>
>>> >> > >>>> We are approaching previously proposed code freeze date (Aug
>>> 31).
>>> >> > >>>>
>>> >> > >>>> One of the critical feature Node Attributes feature merge
>>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need a
>>> bit
>>> >> > more
>>> >> > >>>> time. With regard to this, suggesting to push the feature/code
>>> >> freeze
>>> >> > for 2
>>> >> > >>>> more weeks to accommodate these jiras too.
>>> >> > >>>>
>>> >> > >>>> Proposing Updated changes in plan inline with this:
>>> >> > >>>> Feature freeze date : all features to merge by September 7,
>>> 2018.
>>> >> > >>>> Code freeze date : blockers/critical only, no improvements and
>>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>>> >> > >>>> Release date: September 28, 2018
>>> >> > >>>>
>>> >> > >>>> If any features in branch which are targeted to 3.2.0, please
>>> >> reply to
>>> >> > >>>> this email thread.
>>> >> > >>>>
>>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>>> >> > >>>>
>>> >> > >>>> 1. Merged & Completed features:
>>> >> > >>>>
>>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> >> > >>>> workloads Initial cut.
>>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>>> Scheduler.
>>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> >> Service
>>> >> > >>>> API and CLI.
>>> >> > >>>>
>>> >> > >>>> 2. Features close to finish:
>>> >> > >>>>
>>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >> Merge/Vote
>>> >> > >>>> Ongoing.
>>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> from
>>> >> > >>>> ATSv2. Patch in progress.
>>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> Improvement
>>> >> > works.
>>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>>> >> > >>>> Storage. In progress.
>>> >> > >>>>
>>> >> > >>>> 3. Tentative features:
>>> >> > >>>>
>>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging
>>> >> > to
>>> >> > >>>> be done before Aug 2018.
>>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>>> Challenging as more discussions are on-going.
>>> >> > >>>>
>>> >> > >>>> *Summary of 3.2.0 issues status:*
>>> >> > >>>>
>>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following up
>>> with
>>> >> > >>>> owners to get status on each of them to get in by Code Freeze
>>> date.
>>> >> > >>>>
>>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>>> >> > Version/s" =
>>> >> > >>>> 3.2.0 ORDER BY priority DESC
>>> >> > >>>>
>>> >> > >>>> Thanks,
>>> >> > >>>> Sunil
>>> >> > >>>>
>>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org>
>>> >> wrote:
>>> >> > >>>>
>>> >> > >>>>> Hi All,
>>> >> > >>>>>
>>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
>>> dated
>>> >> > 17th
>>> >> > >>>>> July 2018, I would like to provide some updates.
>>> >> > >>>>>
>>> >> > >>>>> We are approaching previously proposed feature freeze date
>>> (Aug
>>> >> 21,
>>> >> > >>>>> about 7 days from today).
>>> >> > >>>>> If any features in branch which are targeted to 3.2.0, please
>>> >> reply
>>> >> > to
>>> >> > >>>>> this email thread.
>>> >> > >>>>> Steve has mentioned about the s3 features which will come
>>> close to
>>> >> > >>>>> Code Freeze Date (Aug 31st).
>>> >> > >>>>>
>>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>>> >> > >>>>>
>>> >> > >>>>> 1. Merged & Completed features:
>>> >> > >>>>>
>>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>>> DeepLearning
>>> >> > >>>>> workloads Initial cut.
>>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>>>>
>>> >> > >>>>> 2. Features close to finish:
>>> >> > >>>>>
>>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> Major
>>> >> > >>>>> patches are all in, only one last
>>> >> > >>>>> patch is in review state.
>>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>>>> Scheduler. Close to commit.
>>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> >> Service
>>> >> > >>>>> API and CLI. 2 patches are pending
>>> >> > >>>>> which will be closed by Feature freeze date.
>>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> >> from
>>> >> > >>>>> ATSv2. Patch in progress.
>>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> Improvement
>>> >> > >>>>> works.
>>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>>> Azure
>>> >> > >>>>> Storage. In progress.
>>> >> > >>>>>
>>> >> > >>>>> 3. Tentative features:
>>> >> > >>>>>
>>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging
>>> >> > >>>>> to be done before Aug 2018.
>>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>>>> Challenging as more discussions are on-going.
>>> >> > >>>>>
>>> >> > >>>>> *Summary of 3.2.0 issues status:*
>>> >> > >>>>>
>>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
>>> with
>>> >> > owners
>>> >> > >>>>> to get status on each of them to get in by Code Freeze date.
>>> >> > >>>>>
>>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>>> >> > Version/s" =
>>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>>> >> > >>>>>
>>> >> > >>>>> Thanks,
>>> >> > >>>>> Sunil
>>> >> > >>>>>
>>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org>
>>> >> wrote:
>>> >> > >>>>>
>>> >> > >>>>>> Thanks Subru for the thoughts.
>>> >> > >>>>>> One of the main reason for a major release is to push out
>>> >> critical
>>> >> > >>>>>> features with a faster cadence to the users. If we are
>>> pulling
>>> >> more
>>> >> > and
>>> >> > >>>>>> more different types of features to a minor release, that
>>> branch
>>> >> > will
>>> >> > >>>>>> become more destabilized and it may be tough to say that
>>> 3.1.2 is
>>> >> > stable
>>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
>>> >> features
>>> >> > in
>>> >> > >>>>>> subsequent minor release.
>>> >> > >>>>>> For few companies, it makes sense to push out these new
>>> features
>>> >> > >>>>>> faster to make a reach to the users. Adding to the point to
>>> the
>>> >> > backporting
>>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
>>> with
>>> >> > some git
>>> >> > >>>>>> scripts. If we can make such scripts available to committers,
>>> >> > backport will
>>> >> > >>>>>> be seem-less across branches and we can achieve the faster
>>> >> release
>>> >> > cadence
>>> >> > >>>>>> also.
>>> >> > >>>>>>
>>> >> > >>>>>> Thoughts?
>>> >> > >>>>>>
>>> >> > >>>>>> - Sunil
>>> >> > >>>>>>
>>> >> > >>>>>>
>>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>>> subru@apache.org
>>> >> >
>>> >> > >>>>>> wrote:
>>> >> > >>>>>>
>>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release effort. I
>>> am
>>> >> > >>>>>>> generally
>>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as
>>> >> feel we
>>> >> > >>>>>>> already
>>> >> > >>>>>>> have too many branches to be maintained. I already see many
>>> >> commits
>>> >> > >>>>>>> are in
>>> >> > >>>>>>> different branches with no apparent rationale, for e.g: 3.1
>>> has
>>> >> > >>>>>>> commits
>>> >> > >>>>>>> which are absent in 3.0 etc.
>>> >> > >>>>>>>
>>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
>>> >> > production
>>> >> > >>>>>>> setting so the cost of adding features should be minimal.
>>> >> > >>>>>>>
>>> >> > >>>>>>> Thoughts?
>>> >> > >>>>>>>
>>> >> > >>>>>>> -Subru
>>> >> > >>>>>>>
>>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
>>> sunilg@apache.org>
>>> >> > wrote:
>>> >> > >>>>>>>
>>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Yes, important changes and features are much needed,
>>> hence we
>>> >> > will
>>> >> > >>>>>>> be
>>> >> > >>>>>>> > keeping the door open for them as possible. Also
>>> considering
>>> >> few
>>> >> > >>>>>>> more
>>> >> > >>>>>>> > offline requests from other folks, I think extending the
>>> >> > timeframe
>>> >> > >>>>>>> by
>>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
>>> buffer) and
>>> >> > >>>>>>> this should
>>> >> > >>>>>>> > ideally help us to ship this by September itself.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as
>>> >> well)
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Feature freeze date : all features to merge by August
>>> 21,
>>> >> 2018.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>>> improvements
>>> >> and
>>> >> > >>>>>>> non
>>> >> > >>>>>>> > blocker/critical
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > bug-fixes August 31, 2018.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Release date: September 15, 2018
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered
>>> your
>>> >> > >>>>>>> questions.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Thanks
>>> >> > >>>>>>> > Sunil
>>> >> > >>>>>>> >
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>>> >> wheeleast@gmail.com
>>> >> > >
>>> >> > >>>>>>> wrote:
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release,
>>> +1
>>> >> for
>>> >> > >>>>>>> that.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > To concerns from Steve,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > It is a good idea to keep the door open to get important
>>> >> > changes
>>> >> > >>>>>>> /
>>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
>>> >> proposed
>>> >> > >>>>>>> release
>>> >> > >>>>>>> > date
>>> >> > >>>>>>> > > to make sure things can happen earlier instead of last
>>> >> minute
>>> >> > >>>>>>> and we all
>>> >> > >>>>>>> > > know that releases are always get delayed :). I'm also
>>> fine
>>> >> if
>>> >> > >>>>>>> we want
>>> >> > >>>>>>> > get
>>> >> > >>>>>>> > > another several weeks time.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
>>> before
>>> >> > >>>>>>> thanksgiving.
>>> >> > >>>>>>> > > Do you think is it good or too early / late?
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Eric,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
>>> YARN-8135
>>> >> can
>>> >> > >>>>>>> get merged
>>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Sunil,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Could u update
>>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>>> >> > >>>>>>> > Roadmap
>>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
>>> first
>>> >> > >>>>>>> before getting
>>> >> > >>>>>>> > > consensus of time.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Thanks,
>>> >> > >>>>>>> > > Wangda
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>> >> > >>>>>>> <fabbri@cloudera.com.invalid
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > wrote:
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>> >> > >>>>>>> stevel@hortonworks.com>
>>> >> > >>>>>>> > >> wrote:
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org
>>> >> > <mailto:
>>> >> > >>>>>>> > >> > sunilg@apache.org>> wrote:
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I would also would like to take this opportunity to
>>> come
>>> >> up
>>> >> > >>>>>>> with a
>>> >> > >>>>>>> > >> detailed
>>> >> > >>>>>>> > >> > plan.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
>>> merged by
>>> >> > >>>>>>> August 10,
>>> >> > >>>>>>> > >> 2018.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > <snip>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > Please let me know if I missed any features targeted
>>> to
>>> >> 3.2
>>> >> > >>>>>>> per this
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
>>> Datalake
>>> >> > Gen
>>> >> > >>>>>>> 2
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
>>> doing
>>> >> work
>>> >> > >>>>>>> on it, but
>>> >> > >>>>>>> > >> if
>>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
>>> >> timely
>>> >> > >>>>>>> reviews on
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will review
>>> the
>>> >> S3 &
>>> >> > >>>>>>> S3Guard
>>> >> > >>>>>>> > >> JIRAs
>>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
>>> some
>>> >> > >>>>>>> pressing
>>> >> > >>>>>>> > "guave,
>>> >> > >>>>>>> > >> > java 9 prep"
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> I can help with this part if you like.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as release
>>> >> > manager
>>> >> > >>>>>>> of 3.2.0
>>> >> > >>>>>>> > >> > release.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > well volunteered!
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I think this raises a good q: what timetable should
>>> we
>>> >> have
>>> >> > >>>>>>> for the
>>> >> > >>>>>>> > >> 3.2. &
>>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
>>> having
>>> >> > the
>>> >> > >>>>>>> outline
>>> >> > >>>>>>> > >> time
>>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
>>> less
>>> >> > >>>>>>> concern about
>>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > -Steve
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> -AF
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> >
>>> >> > >>>>>>>
>>> >> > >>>>>>
>>> >> >
>>> >>
>>> >
>>>
>>
>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi Folks,
As we previously communicated for 3.2.0 release, we have delayed due to few
blockers in our gate.
I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
all bug fixes.
- Sunil
On Tue, Oct 16, 2018 at 8:59 AM Sunil G <su...@apache.org> wrote:
> Hi Folks,
>
> We are now close to RC as other blocker issues are now merged to trunk and
> branch-3.2. Last 2 critical issues are closer to merge and will be
> committed in few hours.
> With this, I will be creating 3.2.0 branch today and will go ahead with RC
> related process.
>
> - Sunil
>
> On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jo...@stripe.com>
> wrote:
>
>> Hello, were there any updates around the 3.2.0 RC timing? All I see in
>> the current blockers are related to the new Submarine subproject, wasn't
>> sure if that is what is holding things up.
>>
>> Cheers,
>> Jon
>>
>> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <su...@apache.org> wrote:
>>
>>> Thanks Robert and Haibo for quickly correcting same.
>>> Sigh, I somehow missed one file while committing the change. Sorry for
>>> the
>>> trouble.
>>>
>>> - Sunil
>>>
>>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rk...@cloudera.com>
>>> wrote:
>>>
>>> > Looks like there's two that weren't updated:
>>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
>>> > --include=pom.xml
>>> > ./hadoop-project/pom.xml:
>>> > <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>>> >
>>> > I've just pushed in an addendum commit to fix those.
>>> > In the future, please make sure to do a sanity compile when updating
>>> poms.
>>> >
>>> > thanks
>>> > - Robert
>>> >
>>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>>> <fa...@cloudera.com.invalid>
>>> > wrote:
>>> >
>>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
>>> >> top-level pom.xml?
>>> >>
>>> >>
>>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
>>> >>
>>> >> > Hi All
>>> >> >
>>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset trunk
>>> to
>>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
>>> necessary
>>> >> > patches are pulled into branch-3.2.
>>> >> >
>>> >> > Thank You
>>> >> > - Sunil
>>> >> >
>>> >> >
>>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
>>> >> >
>>> >> > > Hi All
>>> >> > >
>>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged to
>>> >> trunk.
>>> >> > > Thanks for the support.
>>> >> > >
>>> >> > > *Plan for RC*
>>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>>> >> > > RC0 for 3.2: *28th Friday*
>>> >> > >
>>> >> > > Thank You
>>> >> > > Sunil
>>> >> > >
>>> >> > >
>>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org>
>>> wrote:
>>> >> > >
>>> >> > >> Hi All
>>> >> > >>
>>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you
>>> for
>>> >> > >> helping in this. I am following up on these tickets, once its
>>> closed
>>> >> we
>>> >> > >> will cut the 3.2 branch.
>>> >> > >>
>>> >> > >> Thanks
>>> >> > >> Sunil Govindan
>>> >> > >>
>>> >> > >>
>>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org>
>>> wrote:
>>> >> > >>
>>> >> > >>> Hi All,
>>> >> > >>>
>>> >> > >>> Inline with the original 3.2 communication proposal dated 17th
>>> July
>>> >> > >>> 2018, I would like to provide more updates.
>>> >> > >>>
>>> >> > >>> We are approaching previously proposed code freeze date
>>> (September
>>> >> 14,
>>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
>>> >> > existing
>>> >> > >>> trunk to 3.3 if there are no issues.
>>> >> > >>>
>>> >> > >>> *Current Release Plan:*
>>> >> > >>> Feature freeze date : all features to merge by September 7,
>>> 2018.
>>> >> > >>> Code freeze date : blockers/critical only, no improvements and
>>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>>> >> > >>> Release date: September 28, 2018
>>> >> > >>>
>>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0, we
>>> >> need to
>>> >> > >>> backport to 3.2 post branch cut.
>>> >> > >>>
>>> >> > >>> Here's an updated 3.2.0 feature status:
>>> >> > >>>
>>> >> > >>> 1. Merged & Completed features:
>>> >> > >>>
>>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> >> > >>> workloads Initial cut.
>>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>> Scheduler.
>>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> Service
>>> >> > API
>>> >> > >>> and CLI.
>>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement
>>> >> works.
>>> >> > >>>
>>> >> > >>> 2. Features close to finish:
>>> >> > >>>
>>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>>> >> > >>> - (Steve) S3a phase V. Close to commit.
>>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>>> >> > >>>
>>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> from
>>> >> > >>> ATSv2. Patch in progress.
>>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging to
>>> >> > >>> be done before Aug 2018.
>>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>> Challenging as more discussions are on-going.
>>> >> > >>>
>>> >> > >>> *Summary of 3.2.0 issues status:*
>>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following up
>>> with
>>> >> > >>> owners to get status on each of them to get in by Code Freeze
>>> date.
>>> >> > >>>
>>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > (Blocker,
>>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" =
>>> 3.2.0
>>> >> > ORDER
>>> >> > >>> BY priority DESC
>>> >> > >>>
>>> >> > >>> Thanks,
>>> >> > >>> Sunil
>>> >> > >>>
>>> >> > >>>
>>> >> > >>>
>>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org>
>>> wrote:
>>> >> > >>>
>>> >> > >>>> Hi All,
>>> >> > >>>>
>>> >> > >>>> Inline with earlier communication dated 17th July 2018, I would
>>> >> like
>>> >> > to
>>> >> > >>>> provide some updates.
>>> >> > >>>>
>>> >> > >>>> We are approaching previously proposed code freeze date (Aug
>>> 31).
>>> >> > >>>>
>>> >> > >>>> One of the critical feature Node Attributes feature merge
>>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need a
>>> bit
>>> >> > more
>>> >> > >>>> time. With regard to this, suggesting to push the feature/code
>>> >> freeze
>>> >> > for 2
>>> >> > >>>> more weeks to accommodate these jiras too.
>>> >> > >>>>
>>> >> > >>>> Proposing Updated changes in plan inline with this:
>>> >> > >>>> Feature freeze date : all features to merge by September 7,
>>> 2018.
>>> >> > >>>> Code freeze date : blockers/critical only, no improvements and
>>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>>> >> > >>>> Release date: September 28, 2018
>>> >> > >>>>
>>> >> > >>>> If any features in branch which are targeted to 3.2.0, please
>>> >> reply to
>>> >> > >>>> this email thread.
>>> >> > >>>>
>>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>>> >> > >>>>
>>> >> > >>>> 1. Merged & Completed features:
>>> >> > >>>>
>>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> >> > >>>> workloads Initial cut.
>>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>>> Scheduler.
>>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> >> Service
>>> >> > >>>> API and CLI.
>>> >> > >>>>
>>> >> > >>>> 2. Features close to finish:
>>> >> > >>>>
>>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >> Merge/Vote
>>> >> > >>>> Ongoing.
>>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> from
>>> >> > >>>> ATSv2. Patch in progress.
>>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> Improvement
>>> >> > works.
>>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>>> >> > >>>> Storage. In progress.
>>> >> > >>>>
>>> >> > >>>> 3. Tentative features:
>>> >> > >>>>
>>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging
>>> >> > to
>>> >> > >>>> be done before Aug 2018.
>>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>>> Challenging as more discussions are on-going.
>>> >> > >>>>
>>> >> > >>>> *Summary of 3.2.0 issues status:*
>>> >> > >>>>
>>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following up
>>> with
>>> >> > >>>> owners to get status on each of them to get in by Code Freeze
>>> date.
>>> >> > >>>>
>>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>>> >> > Version/s" =
>>> >> > >>>> 3.2.0 ORDER BY priority DESC
>>> >> > >>>>
>>> >> > >>>> Thanks,
>>> >> > >>>> Sunil
>>> >> > >>>>
>>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org>
>>> >> wrote:
>>> >> > >>>>
>>> >> > >>>>> Hi All,
>>> >> > >>>>>
>>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
>>> dated
>>> >> > 17th
>>> >> > >>>>> July 2018, I would like to provide some updates.
>>> >> > >>>>>
>>> >> > >>>>> We are approaching previously proposed feature freeze date
>>> (Aug
>>> >> 21,
>>> >> > >>>>> about 7 days from today).
>>> >> > >>>>> If any features in branch which are targeted to 3.2.0, please
>>> >> reply
>>> >> > to
>>> >> > >>>>> this email thread.
>>> >> > >>>>> Steve has mentioned about the s3 features which will come
>>> close to
>>> >> > >>>>> Code Freeze Date (Aug 31st).
>>> >> > >>>>>
>>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>>> >> > >>>>>
>>> >> > >>>>> 1. Merged & Completed features:
>>> >> > >>>>>
>>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>>> DeepLearning
>>> >> > >>>>> workloads Initial cut.
>>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>>>>
>>> >> > >>>>> 2. Features close to finish:
>>> >> > >>>>>
>>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> Major
>>> >> > >>>>> patches are all in, only one last
>>> >> > >>>>> patch is in review state.
>>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>>>> Scheduler. Close to commit.
>>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> >> Service
>>> >> > >>>>> API and CLI. 2 patches are pending
>>> >> > >>>>> which will be closed by Feature freeze date.
>>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> >> from
>>> >> > >>>>> ATSv2. Patch in progress.
>>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> Improvement
>>> >> > >>>>> works.
>>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>>> Azure
>>> >> > >>>>> Storage. In progress.
>>> >> > >>>>>
>>> >> > >>>>> 3. Tentative features:
>>> >> > >>>>>
>>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging
>>> >> > >>>>> to be done before Aug 2018.
>>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>>>> Challenging as more discussions are on-going.
>>> >> > >>>>>
>>> >> > >>>>> *Summary of 3.2.0 issues status:*
>>> >> > >>>>>
>>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
>>> with
>>> >> > owners
>>> >> > >>>>> to get status on each of them to get in by Code Freeze date.
>>> >> > >>>>>
>>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>>> >> > Version/s" =
>>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>>> >> > >>>>>
>>> >> > >>>>> Thanks,
>>> >> > >>>>> Sunil
>>> >> > >>>>>
>>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org>
>>> >> wrote:
>>> >> > >>>>>
>>> >> > >>>>>> Thanks Subru for the thoughts.
>>> >> > >>>>>> One of the main reason for a major release is to push out
>>> >> critical
>>> >> > >>>>>> features with a faster cadence to the users. If we are
>>> pulling
>>> >> more
>>> >> > and
>>> >> > >>>>>> more different types of features to a minor release, that
>>> branch
>>> >> > will
>>> >> > >>>>>> become more destabilized and it may be tough to say that
>>> 3.1.2 is
>>> >> > stable
>>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
>>> >> features
>>> >> > in
>>> >> > >>>>>> subsequent minor release.
>>> >> > >>>>>> For few companies, it makes sense to push out these new
>>> features
>>> >> > >>>>>> faster to make a reach to the users. Adding to the point to
>>> the
>>> >> > backporting
>>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
>>> with
>>> >> > some git
>>> >> > >>>>>> scripts. If we can make such scripts available to committers,
>>> >> > backport will
>>> >> > >>>>>> be seem-less across branches and we can achieve the faster
>>> >> release
>>> >> > cadence
>>> >> > >>>>>> also.
>>> >> > >>>>>>
>>> >> > >>>>>> Thoughts?
>>> >> > >>>>>>
>>> >> > >>>>>> - Sunil
>>> >> > >>>>>>
>>> >> > >>>>>>
>>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>>> subru@apache.org
>>> >> >
>>> >> > >>>>>> wrote:
>>> >> > >>>>>>
>>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release effort. I
>>> am
>>> >> > >>>>>>> generally
>>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as
>>> >> feel we
>>> >> > >>>>>>> already
>>> >> > >>>>>>> have too many branches to be maintained. I already see many
>>> >> commits
>>> >> > >>>>>>> are in
>>> >> > >>>>>>> different branches with no apparent rationale, for e.g: 3.1
>>> has
>>> >> > >>>>>>> commits
>>> >> > >>>>>>> which are absent in 3.0 etc.
>>> >> > >>>>>>>
>>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
>>> >> > production
>>> >> > >>>>>>> setting so the cost of adding features should be minimal.
>>> >> > >>>>>>>
>>> >> > >>>>>>> Thoughts?
>>> >> > >>>>>>>
>>> >> > >>>>>>> -Subru
>>> >> > >>>>>>>
>>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
>>> sunilg@apache.org>
>>> >> > wrote:
>>> >> > >>>>>>>
>>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Yes, important changes and features are much needed,
>>> hence we
>>> >> > will
>>> >> > >>>>>>> be
>>> >> > >>>>>>> > keeping the door open for them as possible. Also
>>> considering
>>> >> few
>>> >> > >>>>>>> more
>>> >> > >>>>>>> > offline requests from other folks, I think extending the
>>> >> > timeframe
>>> >> > >>>>>>> by
>>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
>>> buffer) and
>>> >> > >>>>>>> this should
>>> >> > >>>>>>> > ideally help us to ship this by September itself.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as
>>> >> well)
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Feature freeze date : all features to merge by August
>>> 21,
>>> >> 2018.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>>> improvements
>>> >> and
>>> >> > >>>>>>> non
>>> >> > >>>>>>> > blocker/critical
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > bug-fixes August 31, 2018.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Release date: September 15, 2018
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered
>>> your
>>> >> > >>>>>>> questions.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Thanks
>>> >> > >>>>>>> > Sunil
>>> >> > >>>>>>> >
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>>> >> wheeleast@gmail.com
>>> >> > >
>>> >> > >>>>>>> wrote:
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release,
>>> +1
>>> >> for
>>> >> > >>>>>>> that.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > To concerns from Steve,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > It is a good idea to keep the door open to get important
>>> >> > changes
>>> >> > >>>>>>> /
>>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
>>> >> proposed
>>> >> > >>>>>>> release
>>> >> > >>>>>>> > date
>>> >> > >>>>>>> > > to make sure things can happen earlier instead of last
>>> >> minute
>>> >> > >>>>>>> and we all
>>> >> > >>>>>>> > > know that releases are always get delayed :). I'm also
>>> fine
>>> >> if
>>> >> > >>>>>>> we want
>>> >> > >>>>>>> > get
>>> >> > >>>>>>> > > another several weeks time.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
>>> before
>>> >> > >>>>>>> thanksgiving.
>>> >> > >>>>>>> > > Do you think is it good or too early / late?
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Eric,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
>>> YARN-8135
>>> >> can
>>> >> > >>>>>>> get merged
>>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Sunil,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Could u update
>>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>>> >> > >>>>>>> > Roadmap
>>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
>>> first
>>> >> > >>>>>>> before getting
>>> >> > >>>>>>> > > consensus of time.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Thanks,
>>> >> > >>>>>>> > > Wangda
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>> >> > >>>>>>> <fabbri@cloudera.com.invalid
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > wrote:
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>> >> > >>>>>>> stevel@hortonworks.com>
>>> >> > >>>>>>> > >> wrote:
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org
>>> >> > <mailto:
>>> >> > >>>>>>> > >> > sunilg@apache.org>> wrote:
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I would also would like to take this opportunity to
>>> come
>>> >> up
>>> >> > >>>>>>> with a
>>> >> > >>>>>>> > >> detailed
>>> >> > >>>>>>> > >> > plan.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
>>> merged by
>>> >> > >>>>>>> August 10,
>>> >> > >>>>>>> > >> 2018.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > <snip>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > Please let me know if I missed any features targeted
>>> to
>>> >> 3.2
>>> >> > >>>>>>> per this
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
>>> Datalake
>>> >> > Gen
>>> >> > >>>>>>> 2
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
>>> doing
>>> >> work
>>> >> > >>>>>>> on it, but
>>> >> > >>>>>>> > >> if
>>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
>>> >> timely
>>> >> > >>>>>>> reviews on
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will review
>>> the
>>> >> S3 &
>>> >> > >>>>>>> S3Guard
>>> >> > >>>>>>> > >> JIRAs
>>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
>>> some
>>> >> > >>>>>>> pressing
>>> >> > >>>>>>> > "guave,
>>> >> > >>>>>>> > >> > java 9 prep"
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> I can help with this part if you like.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as release
>>> >> > manager
>>> >> > >>>>>>> of 3.2.0
>>> >> > >>>>>>> > >> > release.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > well volunteered!
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I think this raises a good q: what timetable should
>>> we
>>> >> have
>>> >> > >>>>>>> for the
>>> >> > >>>>>>> > >> 3.2. &
>>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
>>> having
>>> >> > the
>>> >> > >>>>>>> outline
>>> >> > >>>>>>> > >> time
>>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
>>> less
>>> >> > >>>>>>> concern about
>>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > -Steve
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> -AF
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> >
>>> >> > >>>>>>>
>>> >> > >>>>>>
>>> >> >
>>> >>
>>> >
>>>
>>
>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi Folks,
As we previously communicated for 3.2.0 release, we have delayed due to few
blockers in our gate.
I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
all bug fixes.
- Sunil
On Tue, Oct 16, 2018 at 8:59 AM Sunil G <su...@apache.org> wrote:
> Hi Folks,
>
> We are now close to RC as other blocker issues are now merged to trunk and
> branch-3.2. Last 2 critical issues are closer to merge and will be
> committed in few hours.
> With this, I will be creating 3.2.0 branch today and will go ahead with RC
> related process.
>
> - Sunil
>
> On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jo...@stripe.com>
> wrote:
>
>> Hello, were there any updates around the 3.2.0 RC timing? All I see in
>> the current blockers are related to the new Submarine subproject, wasn't
>> sure if that is what is holding things up.
>>
>> Cheers,
>> Jon
>>
>> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <su...@apache.org> wrote:
>>
>>> Thanks Robert and Haibo for quickly correcting same.
>>> Sigh, I somehow missed one file while committing the change. Sorry for
>>> the
>>> trouble.
>>>
>>> - Sunil
>>>
>>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rk...@cloudera.com>
>>> wrote:
>>>
>>> > Looks like there's two that weren't updated:
>>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
>>> > --include=pom.xml
>>> > ./hadoop-project/pom.xml:
>>> > <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>>> >
>>> > I've just pushed in an addendum commit to fix those.
>>> > In the future, please make sure to do a sanity compile when updating
>>> poms.
>>> >
>>> > thanks
>>> > - Robert
>>> >
>>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>>> <fa...@cloudera.com.invalid>
>>> > wrote:
>>> >
>>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
>>> >> top-level pom.xml?
>>> >>
>>> >>
>>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
>>> >>
>>> >> > Hi All
>>> >> >
>>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset trunk
>>> to
>>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
>>> necessary
>>> >> > patches are pulled into branch-3.2.
>>> >> >
>>> >> > Thank You
>>> >> > - Sunil
>>> >> >
>>> >> >
>>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
>>> >> >
>>> >> > > Hi All
>>> >> > >
>>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged to
>>> >> trunk.
>>> >> > > Thanks for the support.
>>> >> > >
>>> >> > > *Plan for RC*
>>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>>> >> > > RC0 for 3.2: *28th Friday*
>>> >> > >
>>> >> > > Thank You
>>> >> > > Sunil
>>> >> > >
>>> >> > >
>>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org>
>>> wrote:
>>> >> > >
>>> >> > >> Hi All
>>> >> > >>
>>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you
>>> for
>>> >> > >> helping in this. I am following up on these tickets, once its
>>> closed
>>> >> we
>>> >> > >> will cut the 3.2 branch.
>>> >> > >>
>>> >> > >> Thanks
>>> >> > >> Sunil Govindan
>>> >> > >>
>>> >> > >>
>>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org>
>>> wrote:
>>> >> > >>
>>> >> > >>> Hi All,
>>> >> > >>>
>>> >> > >>> Inline with the original 3.2 communication proposal dated 17th
>>> July
>>> >> > >>> 2018, I would like to provide more updates.
>>> >> > >>>
>>> >> > >>> We are approaching previously proposed code freeze date
>>> (September
>>> >> 14,
>>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
>>> >> > existing
>>> >> > >>> trunk to 3.3 if there are no issues.
>>> >> > >>>
>>> >> > >>> *Current Release Plan:*
>>> >> > >>> Feature freeze date : all features to merge by September 7,
>>> 2018.
>>> >> > >>> Code freeze date : blockers/critical only, no improvements and
>>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>>> >> > >>> Release date: September 28, 2018
>>> >> > >>>
>>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0, we
>>> >> need to
>>> >> > >>> backport to 3.2 post branch cut.
>>> >> > >>>
>>> >> > >>> Here's an updated 3.2.0 feature status:
>>> >> > >>>
>>> >> > >>> 1. Merged & Completed features:
>>> >> > >>>
>>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> >> > >>> workloads Initial cut.
>>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>> Scheduler.
>>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> Service
>>> >> > API
>>> >> > >>> and CLI.
>>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement
>>> >> works.
>>> >> > >>>
>>> >> > >>> 2. Features close to finish:
>>> >> > >>>
>>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>>> >> > >>> - (Steve) S3a phase V. Close to commit.
>>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>>> >> > >>>
>>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> from
>>> >> > >>> ATSv2. Patch in progress.
>>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging to
>>> >> > >>> be done before Aug 2018.
>>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>> Challenging as more discussions are on-going.
>>> >> > >>>
>>> >> > >>> *Summary of 3.2.0 issues status:*
>>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following up
>>> with
>>> >> > >>> owners to get status on each of them to get in by Code Freeze
>>> date.
>>> >> > >>>
>>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > (Blocker,
>>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" =
>>> 3.2.0
>>> >> > ORDER
>>> >> > >>> BY priority DESC
>>> >> > >>>
>>> >> > >>> Thanks,
>>> >> > >>> Sunil
>>> >> > >>>
>>> >> > >>>
>>> >> > >>>
>>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org>
>>> wrote:
>>> >> > >>>
>>> >> > >>>> Hi All,
>>> >> > >>>>
>>> >> > >>>> Inline with earlier communication dated 17th July 2018, I would
>>> >> like
>>> >> > to
>>> >> > >>>> provide some updates.
>>> >> > >>>>
>>> >> > >>>> We are approaching previously proposed code freeze date (Aug
>>> 31).
>>> >> > >>>>
>>> >> > >>>> One of the critical feature Node Attributes feature merge
>>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need a
>>> bit
>>> >> > more
>>> >> > >>>> time. With regard to this, suggesting to push the feature/code
>>> >> freeze
>>> >> > for 2
>>> >> > >>>> more weeks to accommodate these jiras too.
>>> >> > >>>>
>>> >> > >>>> Proposing Updated changes in plan inline with this:
>>> >> > >>>> Feature freeze date : all features to merge by September 7,
>>> 2018.
>>> >> > >>>> Code freeze date : blockers/critical only, no improvements and
>>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>>> >> > >>>> Release date: September 28, 2018
>>> >> > >>>>
>>> >> > >>>> If any features in branch which are targeted to 3.2.0, please
>>> >> reply to
>>> >> > >>>> this email thread.
>>> >> > >>>>
>>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>>> >> > >>>>
>>> >> > >>>> 1. Merged & Completed features:
>>> >> > >>>>
>>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> >> > >>>> workloads Initial cut.
>>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>>> Scheduler.
>>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> >> Service
>>> >> > >>>> API and CLI.
>>> >> > >>>>
>>> >> > >>>> 2. Features close to finish:
>>> >> > >>>>
>>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >> Merge/Vote
>>> >> > >>>> Ongoing.
>>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> from
>>> >> > >>>> ATSv2. Patch in progress.
>>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> Improvement
>>> >> > works.
>>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>>> >> > >>>> Storage. In progress.
>>> >> > >>>>
>>> >> > >>>> 3. Tentative features:
>>> >> > >>>>
>>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging
>>> >> > to
>>> >> > >>>> be done before Aug 2018.
>>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>>> Challenging as more discussions are on-going.
>>> >> > >>>>
>>> >> > >>>> *Summary of 3.2.0 issues status:*
>>> >> > >>>>
>>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following up
>>> with
>>> >> > >>>> owners to get status on each of them to get in by Code Freeze
>>> date.
>>> >> > >>>>
>>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>>> >> > Version/s" =
>>> >> > >>>> 3.2.0 ORDER BY priority DESC
>>> >> > >>>>
>>> >> > >>>> Thanks,
>>> >> > >>>> Sunil
>>> >> > >>>>
>>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org>
>>> >> wrote:
>>> >> > >>>>
>>> >> > >>>>> Hi All,
>>> >> > >>>>>
>>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
>>> dated
>>> >> > 17th
>>> >> > >>>>> July 2018, I would like to provide some updates.
>>> >> > >>>>>
>>> >> > >>>>> We are approaching previously proposed feature freeze date
>>> (Aug
>>> >> 21,
>>> >> > >>>>> about 7 days from today).
>>> >> > >>>>> If any features in branch which are targeted to 3.2.0, please
>>> >> reply
>>> >> > to
>>> >> > >>>>> this email thread.
>>> >> > >>>>> Steve has mentioned about the s3 features which will come
>>> close to
>>> >> > >>>>> Code Freeze Date (Aug 31st).
>>> >> > >>>>>
>>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>>> >> > >>>>>
>>> >> > >>>>> 1. Merged & Completed features:
>>> >> > >>>>>
>>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>>> DeepLearning
>>> >> > >>>>> workloads Initial cut.
>>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>>>>
>>> >> > >>>>> 2. Features close to finish:
>>> >> > >>>>>
>>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> Major
>>> >> > >>>>> patches are all in, only one last
>>> >> > >>>>> patch is in review state.
>>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>>>> Scheduler. Close to commit.
>>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> >> Service
>>> >> > >>>>> API and CLI. 2 patches are pending
>>> >> > >>>>> which will be closed by Feature freeze date.
>>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> >> from
>>> >> > >>>>> ATSv2. Patch in progress.
>>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> Improvement
>>> >> > >>>>> works.
>>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>>> Azure
>>> >> > >>>>> Storage. In progress.
>>> >> > >>>>>
>>> >> > >>>>> 3. Tentative features:
>>> >> > >>>>>
>>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging
>>> >> > >>>>> to be done before Aug 2018.
>>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>>>> Challenging as more discussions are on-going.
>>> >> > >>>>>
>>> >> > >>>>> *Summary of 3.2.0 issues status:*
>>> >> > >>>>>
>>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
>>> with
>>> >> > owners
>>> >> > >>>>> to get status on each of them to get in by Code Freeze date.
>>> >> > >>>>>
>>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>>> >> > Version/s" =
>>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>>> >> > >>>>>
>>> >> > >>>>> Thanks,
>>> >> > >>>>> Sunil
>>> >> > >>>>>
>>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org>
>>> >> wrote:
>>> >> > >>>>>
>>> >> > >>>>>> Thanks Subru for the thoughts.
>>> >> > >>>>>> One of the main reason for a major release is to push out
>>> >> critical
>>> >> > >>>>>> features with a faster cadence to the users. If we are
>>> pulling
>>> >> more
>>> >> > and
>>> >> > >>>>>> more different types of features to a minor release, that
>>> branch
>>> >> > will
>>> >> > >>>>>> become more destabilized and it may be tough to say that
>>> 3.1.2 is
>>> >> > stable
>>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
>>> >> features
>>> >> > in
>>> >> > >>>>>> subsequent minor release.
>>> >> > >>>>>> For few companies, it makes sense to push out these new
>>> features
>>> >> > >>>>>> faster to make a reach to the users. Adding to the point to
>>> the
>>> >> > backporting
>>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
>>> with
>>> >> > some git
>>> >> > >>>>>> scripts. If we can make such scripts available to committers,
>>> >> > backport will
>>> >> > >>>>>> be seem-less across branches and we can achieve the faster
>>> >> release
>>> >> > cadence
>>> >> > >>>>>> also.
>>> >> > >>>>>>
>>> >> > >>>>>> Thoughts?
>>> >> > >>>>>>
>>> >> > >>>>>> - Sunil
>>> >> > >>>>>>
>>> >> > >>>>>>
>>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>>> subru@apache.org
>>> >> >
>>> >> > >>>>>> wrote:
>>> >> > >>>>>>
>>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release effort. I
>>> am
>>> >> > >>>>>>> generally
>>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as
>>> >> feel we
>>> >> > >>>>>>> already
>>> >> > >>>>>>> have too many branches to be maintained. I already see many
>>> >> commits
>>> >> > >>>>>>> are in
>>> >> > >>>>>>> different branches with no apparent rationale, for e.g: 3.1
>>> has
>>> >> > >>>>>>> commits
>>> >> > >>>>>>> which are absent in 3.0 etc.
>>> >> > >>>>>>>
>>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
>>> >> > production
>>> >> > >>>>>>> setting so the cost of adding features should be minimal.
>>> >> > >>>>>>>
>>> >> > >>>>>>> Thoughts?
>>> >> > >>>>>>>
>>> >> > >>>>>>> -Subru
>>> >> > >>>>>>>
>>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
>>> sunilg@apache.org>
>>> >> > wrote:
>>> >> > >>>>>>>
>>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Yes, important changes and features are much needed,
>>> hence we
>>> >> > will
>>> >> > >>>>>>> be
>>> >> > >>>>>>> > keeping the door open for them as possible. Also
>>> considering
>>> >> few
>>> >> > >>>>>>> more
>>> >> > >>>>>>> > offline requests from other folks, I think extending the
>>> >> > timeframe
>>> >> > >>>>>>> by
>>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
>>> buffer) and
>>> >> > >>>>>>> this should
>>> >> > >>>>>>> > ideally help us to ship this by September itself.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as
>>> >> well)
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Feature freeze date : all features to merge by August
>>> 21,
>>> >> 2018.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>>> improvements
>>> >> and
>>> >> > >>>>>>> non
>>> >> > >>>>>>> > blocker/critical
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > bug-fixes August 31, 2018.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Release date: September 15, 2018
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered
>>> your
>>> >> > >>>>>>> questions.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Thanks
>>> >> > >>>>>>> > Sunil
>>> >> > >>>>>>> >
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>>> >> wheeleast@gmail.com
>>> >> > >
>>> >> > >>>>>>> wrote:
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release,
>>> +1
>>> >> for
>>> >> > >>>>>>> that.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > To concerns from Steve,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > It is a good idea to keep the door open to get important
>>> >> > changes
>>> >> > >>>>>>> /
>>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
>>> >> proposed
>>> >> > >>>>>>> release
>>> >> > >>>>>>> > date
>>> >> > >>>>>>> > > to make sure things can happen earlier instead of last
>>> >> minute
>>> >> > >>>>>>> and we all
>>> >> > >>>>>>> > > know that releases are always get delayed :). I'm also
>>> fine
>>> >> if
>>> >> > >>>>>>> we want
>>> >> > >>>>>>> > get
>>> >> > >>>>>>> > > another several weeks time.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
>>> before
>>> >> > >>>>>>> thanksgiving.
>>> >> > >>>>>>> > > Do you think is it good or too early / late?
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Eric,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
>>> YARN-8135
>>> >> can
>>> >> > >>>>>>> get merged
>>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Sunil,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Could u update
>>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>>> >> > >>>>>>> > Roadmap
>>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
>>> first
>>> >> > >>>>>>> before getting
>>> >> > >>>>>>> > > consensus of time.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Thanks,
>>> >> > >>>>>>> > > Wangda
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>> >> > >>>>>>> <fabbri@cloudera.com.invalid
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > wrote:
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>> >> > >>>>>>> stevel@hortonworks.com>
>>> >> > >>>>>>> > >> wrote:
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org
>>> >> > <mailto:
>>> >> > >>>>>>> > >> > sunilg@apache.org>> wrote:
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I would also would like to take this opportunity to
>>> come
>>> >> up
>>> >> > >>>>>>> with a
>>> >> > >>>>>>> > >> detailed
>>> >> > >>>>>>> > >> > plan.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
>>> merged by
>>> >> > >>>>>>> August 10,
>>> >> > >>>>>>> > >> 2018.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > <snip>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > Please let me know if I missed any features targeted
>>> to
>>> >> 3.2
>>> >> > >>>>>>> per this
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
>>> Datalake
>>> >> > Gen
>>> >> > >>>>>>> 2
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
>>> doing
>>> >> work
>>> >> > >>>>>>> on it, but
>>> >> > >>>>>>> > >> if
>>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
>>> >> timely
>>> >> > >>>>>>> reviews on
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will review
>>> the
>>> >> S3 &
>>> >> > >>>>>>> S3Guard
>>> >> > >>>>>>> > >> JIRAs
>>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
>>> some
>>> >> > >>>>>>> pressing
>>> >> > >>>>>>> > "guave,
>>> >> > >>>>>>> > >> > java 9 prep"
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> I can help with this part if you like.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as release
>>> >> > manager
>>> >> > >>>>>>> of 3.2.0
>>> >> > >>>>>>> > >> > release.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > well volunteered!
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I think this raises a good q: what timetable should
>>> we
>>> >> have
>>> >> > >>>>>>> for the
>>> >> > >>>>>>> > >> 3.2. &
>>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
>>> having
>>> >> > the
>>> >> > >>>>>>> outline
>>> >> > >>>>>>> > >> time
>>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
>>> less
>>> >> > >>>>>>> concern about
>>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > -Steve
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> -AF
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> >
>>> >> > >>>>>>>
>>> >> > >>>>>>
>>> >> >
>>> >>
>>> >
>>>
>>
>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi Folks,
As we previously communicated for 3.2.0 release, we have delayed due to few
blockers in our gate.
I just cut branch-3.2.0 for release purpose. branch-3.2 will be open for
all bug fixes.
- Sunil
On Tue, Oct 16, 2018 at 8:59 AM Sunil G <su...@apache.org> wrote:
> Hi Folks,
>
> We are now close to RC as other blocker issues are now merged to trunk and
> branch-3.2. Last 2 critical issues are closer to merge and will be
> committed in few hours.
> With this, I will be creating 3.2.0 branch today and will go ahead with RC
> related process.
>
> - Sunil
>
> On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jo...@stripe.com>
> wrote:
>
>> Hello, were there any updates around the 3.2.0 RC timing? All I see in
>> the current blockers are related to the new Submarine subproject, wasn't
>> sure if that is what is holding things up.
>>
>> Cheers,
>> Jon
>>
>> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <su...@apache.org> wrote:
>>
>>> Thanks Robert and Haibo for quickly correcting same.
>>> Sigh, I somehow missed one file while committing the change. Sorry for
>>> the
>>> trouble.
>>>
>>> - Sunil
>>>
>>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rk...@cloudera.com>
>>> wrote:
>>>
>>> > Looks like there's two that weren't updated:
>>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
>>> > --include=pom.xml
>>> > ./hadoop-project/pom.xml:
>>> > <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>>> >
>>> > I've just pushed in an addendum commit to fix those.
>>> > In the future, please make sure to do a sanity compile when updating
>>> poms.
>>> >
>>> > thanks
>>> > - Robert
>>> >
>>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>>> <fa...@cloudera.com.invalid>
>>> > wrote:
>>> >
>>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
>>> >> top-level pom.xml?
>>> >>
>>> >>
>>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
>>> >>
>>> >> > Hi All
>>> >> >
>>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset trunk
>>> to
>>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all
>>> necessary
>>> >> > patches are pulled into branch-3.2.
>>> >> >
>>> >> > Thank You
>>> >> > - Sunil
>>> >> >
>>> >> >
>>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
>>> >> >
>>> >> > > Hi All
>>> >> > >
>>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged to
>>> >> trunk.
>>> >> > > Thanks for the support.
>>> >> > >
>>> >> > > *Plan for RC*
>>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>>> >> > > RC0 for 3.2: *28th Friday*
>>> >> > >
>>> >> > > Thank You
>>> >> > > Sunil
>>> >> > >
>>> >> > >
>>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org>
>>> wrote:
>>> >> > >
>>> >> > >> Hi All
>>> >> > >>
>>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you
>>> for
>>> >> > >> helping in this. I am following up on these tickets, once its
>>> closed
>>> >> we
>>> >> > >> will cut the 3.2 branch.
>>> >> > >>
>>> >> > >> Thanks
>>> >> > >> Sunil Govindan
>>> >> > >>
>>> >> > >>
>>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org>
>>> wrote:
>>> >> > >>
>>> >> > >>> Hi All,
>>> >> > >>>
>>> >> > >>> Inline with the original 3.2 communication proposal dated 17th
>>> July
>>> >> > >>> 2018, I would like to provide more updates.
>>> >> > >>>
>>> >> > >>> We are approaching previously proposed code freeze date
>>> (September
>>> >> 14,
>>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
>>> >> > existing
>>> >> > >>> trunk to 3.3 if there are no issues.
>>> >> > >>>
>>> >> > >>> *Current Release Plan:*
>>> >> > >>> Feature freeze date : all features to merge by September 7,
>>> 2018.
>>> >> > >>> Code freeze date : blockers/critical only, no improvements and
>>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>>> >> > >>> Release date: September 28, 2018
>>> >> > >>>
>>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0, we
>>> >> need to
>>> >> > >>> backport to 3.2 post branch cut.
>>> >> > >>>
>>> >> > >>> Here's an updated 3.2.0 feature status:
>>> >> > >>>
>>> >> > >>> 1. Merged & Completed features:
>>> >> > >>>
>>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> >> > >>> workloads Initial cut.
>>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>> Scheduler.
>>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> Service
>>> >> > API
>>> >> > >>> and CLI.
>>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement
>>> >> works.
>>> >> > >>>
>>> >> > >>> 2. Features close to finish:
>>> >> > >>>
>>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>>> >> > >>> - (Steve) S3a phase V. Close to commit.
>>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>>> >> > >>>
>>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> from
>>> >> > >>> ATSv2. Patch in progress.
>>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging to
>>> >> > >>> be done before Aug 2018.
>>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>> Challenging as more discussions are on-going.
>>> >> > >>>
>>> >> > >>> *Summary of 3.2.0 issues status:*
>>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following up
>>> with
>>> >> > >>> owners to get status on each of them to get in by Code Freeze
>>> date.
>>> >> > >>>
>>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > (Blocker,
>>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" =
>>> 3.2.0
>>> >> > ORDER
>>> >> > >>> BY priority DESC
>>> >> > >>>
>>> >> > >>> Thanks,
>>> >> > >>> Sunil
>>> >> > >>>
>>> >> > >>>
>>> >> > >>>
>>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org>
>>> wrote:
>>> >> > >>>
>>> >> > >>>> Hi All,
>>> >> > >>>>
>>> >> > >>>> Inline with earlier communication dated 17th July 2018, I would
>>> >> like
>>> >> > to
>>> >> > >>>> provide some updates.
>>> >> > >>>>
>>> >> > >>>> We are approaching previously proposed code freeze date (Aug
>>> 31).
>>> >> > >>>>
>>> >> > >>>> One of the critical feature Node Attributes feature merge
>>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need a
>>> bit
>>> >> > more
>>> >> > >>>> time. With regard to this, suggesting to push the feature/code
>>> >> freeze
>>> >> > for 2
>>> >> > >>>> more weeks to accommodate these jiras too.
>>> >> > >>>>
>>> >> > >>>> Proposing Updated changes in plan inline with this:
>>> >> > >>>> Feature freeze date : all features to merge by September 7,
>>> 2018.
>>> >> > >>>> Code freeze date : blockers/critical only, no improvements and
>>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>>> >> > >>>> Release date: September 28, 2018
>>> >> > >>>>
>>> >> > >>>> If any features in branch which are targeted to 3.2.0, please
>>> >> reply to
>>> >> > >>>> this email thread.
>>> >> > >>>>
>>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>>> >> > >>>>
>>> >> > >>>> 1. Merged & Completed features:
>>> >> > >>>>
>>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> >> > >>>> workloads Initial cut.
>>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>>> Scheduler.
>>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> >> Service
>>> >> > >>>> API and CLI.
>>> >> > >>>>
>>> >> > >>>> 2. Features close to finish:
>>> >> > >>>>
>>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> >> Merge/Vote
>>> >> > >>>> Ongoing.
>>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> from
>>> >> > >>>> ATSv2. Patch in progress.
>>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> Improvement
>>> >> > works.
>>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>>> >> > >>>> Storage. In progress.
>>> >> > >>>>
>>> >> > >>>> 3. Tentative features:
>>> >> > >>>>
>>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging
>>> >> > to
>>> >> > >>>> be done before Aug 2018.
>>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>>> Challenging as more discussions are on-going.
>>> >> > >>>>
>>> >> > >>>> *Summary of 3.2.0 issues status:*
>>> >> > >>>>
>>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following up
>>> with
>>> >> > >>>> owners to get status on each of them to get in by Code Freeze
>>> date.
>>> >> > >>>>
>>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>>> >> > Version/s" =
>>> >> > >>>> 3.2.0 ORDER BY priority DESC
>>> >> > >>>>
>>> >> > >>>> Thanks,
>>> >> > >>>> Sunil
>>> >> > >>>>
>>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org>
>>> >> wrote:
>>> >> > >>>>
>>> >> > >>>>> Hi All,
>>> >> > >>>>>
>>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
>>> dated
>>> >> > 17th
>>> >> > >>>>> July 2018, I would like to provide some updates.
>>> >> > >>>>>
>>> >> > >>>>> We are approaching previously proposed feature freeze date
>>> (Aug
>>> >> 21,
>>> >> > >>>>> about 7 days from today).
>>> >> > >>>>> If any features in branch which are targeted to 3.2.0, please
>>> >> reply
>>> >> > to
>>> >> > >>>>> this email thread.
>>> >> > >>>>> Steve has mentioned about the s3 features which will come
>>> close to
>>> >> > >>>>> Code Freeze Date (Aug 31st).
>>> >> > >>>>>
>>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>>> >> > >>>>>
>>> >> > >>>>> 1. Merged & Completed features:
>>> >> > >>>>>
>>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for
>>> DeepLearning
>>> >> > >>>>> workloads Initial cut.
>>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> >> > >>>>>
>>> >> > >>>>> 2. Features close to finish:
>>> >> > >>>>>
>>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> Major
>>> >> > >>>>> patches are all in, only one last
>>> >> > >>>>> patch is in review state.
>>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> >> > >>>>> Scheduler. Close to commit.
>>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>>> >> Service
>>> >> > >>>>> API and CLI. 2 patches are pending
>>> >> > >>>>> which will be closed by Feature freeze date.
>>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>>> >> from
>>> >> > >>>>> ATSv2. Patch in progress.
>>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>>> Improvement
>>> >> > >>>>> works.
>>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows
>>> Azure
>>> >> > >>>>> Storage. In progress.
>>> >> > >>>>>
>>> >> > >>>>> 3. Tentative features:
>>> >> > >>>>>
>>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>>> >> challenging
>>> >> > >>>>> to be done before Aug 2018.
>>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> >> > >>>>> Challenging as more discussions are on-going.
>>> >> > >>>>>
>>> >> > >>>>> *Summary of 3.2.0 issues status:*
>>> >> > >>>>>
>>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking
>>> with
>>> >> > owners
>>> >> > >>>>> to get status on each of them to get in by Code Freeze date.
>>> >> > >>>>>
>>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>>> >> > Version/s" =
>>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>>> >> > >>>>>
>>> >> > >>>>> Thanks,
>>> >> > >>>>> Sunil
>>> >> > >>>>>
>>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org>
>>> >> wrote:
>>> >> > >>>>>
>>> >> > >>>>>> Thanks Subru for the thoughts.
>>> >> > >>>>>> One of the main reason for a major release is to push out
>>> >> critical
>>> >> > >>>>>> features with a faster cadence to the users. If we are
>>> pulling
>>> >> more
>>> >> > and
>>> >> > >>>>>> more different types of features to a minor release, that
>>> branch
>>> >> > will
>>> >> > >>>>>> become more destabilized and it may be tough to say that
>>> 3.1.2 is
>>> >> > stable
>>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
>>> >> features
>>> >> > in
>>> >> > >>>>>> subsequent minor release.
>>> >> > >>>>>> For few companies, it makes sense to push out these new
>>> features
>>> >> > >>>>>> faster to make a reach to the users. Adding to the point to
>>> the
>>> >> > backporting
>>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
>>> with
>>> >> > some git
>>> >> > >>>>>> scripts. If we can make such scripts available to committers,
>>> >> > backport will
>>> >> > >>>>>> be seem-less across branches and we can achieve the faster
>>> >> release
>>> >> > cadence
>>> >> > >>>>>> also.
>>> >> > >>>>>>
>>> >> > >>>>>> Thoughts?
>>> >> > >>>>>>
>>> >> > >>>>>> - Sunil
>>> >> > >>>>>>
>>> >> > >>>>>>
>>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>>> subru@apache.org
>>> >> >
>>> >> > >>>>>> wrote:
>>> >> > >>>>>>
>>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release effort. I
>>> am
>>> >> > >>>>>>> generally
>>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as
>>> >> feel we
>>> >> > >>>>>>> already
>>> >> > >>>>>>> have too many branches to be maintained. I already see many
>>> >> commits
>>> >> > >>>>>>> are in
>>> >> > >>>>>>> different branches with no apparent rationale, for e.g: 3.1
>>> has
>>> >> > >>>>>>> commits
>>> >> > >>>>>>> which are absent in 3.0 etc.
>>> >> > >>>>>>>
>>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
>>> >> > production
>>> >> > >>>>>>> setting so the cost of adding features should be minimal.
>>> >> > >>>>>>>
>>> >> > >>>>>>> Thoughts?
>>> >> > >>>>>>>
>>> >> > >>>>>>> -Subru
>>> >> > >>>>>>>
>>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <
>>> sunilg@apache.org>
>>> >> > wrote:
>>> >> > >>>>>>>
>>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Yes, important changes and features are much needed,
>>> hence we
>>> >> > will
>>> >> > >>>>>>> be
>>> >> > >>>>>>> > keeping the door open for them as possible. Also
>>> considering
>>> >> few
>>> >> > >>>>>>> more
>>> >> > >>>>>>> > offline requests from other folks, I think extending the
>>> >> > timeframe
>>> >> > >>>>>>> by
>>> >> > >>>>>>> > couple of weeks makes sense (including a second RC
>>> buffer) and
>>> >> > >>>>>>> this should
>>> >> > >>>>>>> > ideally help us to ship this by September itself.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as
>>> >> well)
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Feature freeze date : all features to merge by August
>>> 21,
>>> >> 2018.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>>> improvements
>>> >> and
>>> >> > >>>>>>> non
>>> >> > >>>>>>> > blocker/critical
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > bug-fixes August 31, 2018.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > - Release date: September 15, 2018
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered
>>> your
>>> >> > >>>>>>> questions.
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > Thanks
>>> >> > >>>>>>> > Sunil
>>> >> > >>>>>>> >
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>>> >> wheeleast@gmail.com
>>> >> > >
>>> >> > >>>>>>> wrote:
>>> >> > >>>>>>> >
>>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release,
>>> +1
>>> >> for
>>> >> > >>>>>>> that.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > To concerns from Steve,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > It is a good idea to keep the door open to get important
>>> >> > changes
>>> >> > >>>>>>> /
>>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
>>> >> proposed
>>> >> > >>>>>>> release
>>> >> > >>>>>>> > date
>>> >> > >>>>>>> > > to make sure things can happen earlier instead of last
>>> >> minute
>>> >> > >>>>>>> and we all
>>> >> > >>>>>>> > > know that releases are always get delayed :). I'm also
>>> fine
>>> >> if
>>> >> > >>>>>>> we want
>>> >> > >>>>>>> > get
>>> >> > >>>>>>> > > another several weeks time.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
>>> before
>>> >> > >>>>>>> thanksgiving.
>>> >> > >>>>>>> > > Do you think is it good or too early / late?
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Eric,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if
>>> YARN-8135
>>> >> can
>>> >> > >>>>>>> get merged
>>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Sunil,
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Could u update
>>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>>> >> > >>>>>>> > Roadmap
>>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
>>> first
>>> >> > >>>>>>> before getting
>>> >> > >>>>>>> > > consensus of time.
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > Thanks,
>>> >> > >>>>>>> > > Wangda
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>> >> > >>>>>>> <fabbri@cloudera.com.invalid
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > > wrote:
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>> >> > >>>>>>> stevel@hortonworks.com>
>>> >> > >>>>>>> > >> wrote:
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org
>>> >> > <mailto:
>>> >> > >>>>>>> > >> > sunilg@apache.org>> wrote:
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I would also would like to take this opportunity to
>>> come
>>> >> up
>>> >> > >>>>>>> with a
>>> >> > >>>>>>> > >> detailed
>>> >> > >>>>>>> > >> > plan.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > - Feature freeze date : all features should be
>>> merged by
>>> >> > >>>>>>> August 10,
>>> >> > >>>>>>> > >> 2018.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > <snip>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > Please let me know if I missed any features targeted
>>> to
>>> >> 3.2
>>> >> > >>>>>>> per this
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
>>> Datalake
>>> >> > Gen
>>> >> > >>>>>>> 2
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been
>>> doing
>>> >> work
>>> >> > >>>>>>> on it, but
>>> >> > >>>>>>> > >> if
>>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
>>> >> timely
>>> >> > >>>>>>> reviews on
>>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will review
>>> the
>>> >> S3 &
>>> >> > >>>>>>> S3Guard
>>> >> > >>>>>>> > >> JIRAs
>>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
>>> some
>>> >> > >>>>>>> pressing
>>> >> > >>>>>>> > "guave,
>>> >> > >>>>>>> > >> > java 9 prep"
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> I can help with this part if you like.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as release
>>> >> > manager
>>> >> > >>>>>>> of 3.2.0
>>> >> > >>>>>>> > >> > release.
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > well volunteered!
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > I think this raises a good q: what timetable should
>>> we
>>> >> have
>>> >> > >>>>>>> for the
>>> >> > >>>>>>> > >> 3.2. &
>>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
>>> having
>>> >> > the
>>> >> > >>>>>>> outline
>>> >> > >>>>>>> > >> time
>>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
>>> less
>>> >> > >>>>>>> concern about
>>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> > -Steve
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> >
>>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >> -AF
>>> >> > >>>>>>> > >>
>>> >> > >>>>>>> > >
>>> >> > >>>>>>> >
>>> >> > >>>>>>>
>>> >> > >>>>>>
>>> >> >
>>> >>
>>> >
>>>
>>
>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi Folks,
We are now close to RC as other blocker issues are now merged to trunk and
branch-3.2. Last 2 critical issues are closer to merge and will be
committed in few hours.
With this, I will be creating 3.2.0 branch today and will go ahead with RC
related process.
- Sunil
On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jo...@stripe.com>
wrote:
> Hello, were there any updates around the 3.2.0 RC timing? All I see in the
> current blockers are related to the new Submarine subproject, wasn't sure
> if that is what is holding things up.
>
> Cheers,
> Jon
>
> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <su...@apache.org> wrote:
>
>> Thanks Robert and Haibo for quickly correcting same.
>> Sigh, I somehow missed one file while committing the change. Sorry for the
>> trouble.
>>
>> - Sunil
>>
>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rk...@cloudera.com>
>> wrote:
>>
>> > Looks like there's two that weren't updated:
>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
>> > --include=pom.xml
>> > ./hadoop-project/pom.xml:
>> > <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>> >
>> > I've just pushed in an addendum commit to fix those.
>> > In the future, please make sure to do a sanity compile when updating
>> poms.
>> >
>> > thanks
>> > - Robert
>> >
>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>> <fa...@cloudera.com.invalid>
>> > wrote:
>> >
>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
>> >> top-level pom.xml?
>> >>
>> >>
>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
>> >>
>> >> > Hi All
>> >> >
>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset trunk
>> to
>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary
>> >> > patches are pulled into branch-3.2.
>> >> >
>> >> > Thank You
>> >> > - Sunil
>> >> >
>> >> >
>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
>> >> >
>> >> > > Hi All
>> >> > >
>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged to
>> >> trunk.
>> >> > > Thanks for the support.
>> >> > >
>> >> > > *Plan for RC*
>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>> >> > > RC0 for 3.2: *28th Friday*
>> >> > >
>> >> > > Thank You
>> >> > > Sunil
>> >> > >
>> >> > >
>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
>> >> > >
>> >> > >> Hi All
>> >> > >>
>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you
>> for
>> >> > >> helping in this. I am following up on these tickets, once its
>> closed
>> >> we
>> >> > >> will cut the 3.2 branch.
>> >> > >>
>> >> > >> Thanks
>> >> > >> Sunil Govindan
>> >> > >>
>> >> > >>
>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org>
>> wrote:
>> >> > >>
>> >> > >>> Hi All,
>> >> > >>>
>> >> > >>> Inline with the original 3.2 communication proposal dated 17th
>> July
>> >> > >>> 2018, I would like to provide more updates.
>> >> > >>>
>> >> > >>> We are approaching previously proposed code freeze date
>> (September
>> >> 14,
>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
>> >> > existing
>> >> > >>> trunk to 3.3 if there are no issues.
>> >> > >>>
>> >> > >>> *Current Release Plan:*
>> >> > >>> Feature freeze date : all features to merge by September 7, 2018.
>> >> > >>> Code freeze date : blockers/critical only, no improvements and
>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>> >> > >>> Release date: September 28, 2018
>> >> > >>>
>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0, we
>> >> need to
>> >> > >>> backport to 3.2 post branch cut.
>> >> > >>>
>> >> > >>> Here's an updated 3.2.0 feature status:
>> >> > >>>
>> >> > >>> 1. Merged & Completed features:
>> >> > >>>
>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> >> > >>> workloads Initial cut.
>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> >> > >>> Scheduler.
>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> Service
>> >> > API
>> >> > >>> and CLI.
>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement
>> >> works.
>> >> > >>>
>> >> > >>> 2. Features close to finish:
>> >> > >>>
>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>> >> > >>> - (Steve) S3a phase V. Close to commit.
>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>> >> > >>>
>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>> from
>> >> > >>> ATSv2. Patch in progress.
>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >> challenging to
>> >> > >>> be done before Aug 2018.
>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> >> > >>> Challenging as more discussions are on-going.
>> >> > >>>
>> >> > >>> *Summary of 3.2.0 issues status:*
>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following up
>> with
>> >> > >>> owners to get status on each of them to get in by Code Freeze
>> date.
>> >> > >>>
>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> >> > (Blocker,
>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" =
>> 3.2.0
>> >> > ORDER
>> >> > >>> BY priority DESC
>> >> > >>>
>> >> > >>> Thanks,
>> >> > >>> Sunil
>> >> > >>>
>> >> > >>>
>> >> > >>>
>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org>
>> wrote:
>> >> > >>>
>> >> > >>>> Hi All,
>> >> > >>>>
>> >> > >>>> Inline with earlier communication dated 17th July 2018, I would
>> >> like
>> >> > to
>> >> > >>>> provide some updates.
>> >> > >>>>
>> >> > >>>> We are approaching previously proposed code freeze date (Aug
>> 31).
>> >> > >>>>
>> >> > >>>> One of the critical feature Node Attributes feature merge
>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need a
>> bit
>> >> > more
>> >> > >>>> time. With regard to this, suggesting to push the feature/code
>> >> freeze
>> >> > for 2
>> >> > >>>> more weeks to accommodate these jiras too.
>> >> > >>>>
>> >> > >>>> Proposing Updated changes in plan inline with this:
>> >> > >>>> Feature freeze date : all features to merge by September 7,
>> 2018.
>> >> > >>>> Code freeze date : blockers/critical only, no improvements and
>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>> >> > >>>> Release date: September 28, 2018
>> >> > >>>>
>> >> > >>>> If any features in branch which are targeted to 3.2.0, please
>> >> reply to
>> >> > >>>> this email thread.
>> >> > >>>>
>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>> >> > >>>>
>> >> > >>>> 1. Merged & Completed features:
>> >> > >>>>
>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> >> > >>>> workloads Initial cut.
>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> >> > >>>> Scheduler.
>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> >> Service
>> >> > >>>> API and CLI.
>> >> > >>>>
>> >> > >>>> 2. Features close to finish:
>> >> > >>>>
>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> >> Merge/Vote
>> >> > >>>> Ongoing.
>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>> from
>> >> > >>>> ATSv2. Patch in progress.
>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> Improvement
>> >> > works.
>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>> >> > >>>> Storage. In progress.
>> >> > >>>>
>> >> > >>>> 3. Tentative features:
>> >> > >>>>
>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >> challenging
>> >> > to
>> >> > >>>> be done before Aug 2018.
>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> >> > >>>> Challenging as more discussions are on-going.
>> >> > >>>>
>> >> > >>>> *Summary of 3.2.0 issues status:*
>> >> > >>>>
>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following up
>> with
>> >> > >>>> owners to get status on each of them to get in by Code Freeze
>> date.
>> >> > >>>>
>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> >> > Version/s" =
>> >> > >>>> 3.2.0 ORDER BY priority DESC
>> >> > >>>>
>> >> > >>>> Thanks,
>> >> > >>>> Sunil
>> >> > >>>>
>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org>
>> >> wrote:
>> >> > >>>>
>> >> > >>>>> Hi All,
>> >> > >>>>>
>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
>> dated
>> >> > 17th
>> >> > >>>>> July 2018, I would like to provide some updates.
>> >> > >>>>>
>> >> > >>>>> We are approaching previously proposed feature freeze date (Aug
>> >> 21,
>> >> > >>>>> about 7 days from today).
>> >> > >>>>> If any features in branch which are targeted to 3.2.0, please
>> >> reply
>> >> > to
>> >> > >>>>> this email thread.
>> >> > >>>>> Steve has mentioned about the s3 features which will come
>> close to
>> >> > >>>>> Code Freeze Date (Aug 31st).
>> >> > >>>>>
>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>> >> > >>>>>
>> >> > >>>>> 1. Merged & Completed features:
>> >> > >>>>>
>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> >> > >>>>> workloads Initial cut.
>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >> > >>>>>
>> >> > >>>>> 2. Features close to finish:
>> >> > >>>>>
>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> Major
>> >> > >>>>> patches are all in, only one last
>> >> > >>>>> patch is in review state.
>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> >> > >>>>> Scheduler. Close to commit.
>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> >> Service
>> >> > >>>>> API and CLI. 2 patches are pending
>> >> > >>>>> which will be closed by Feature freeze date.
>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>> >> from
>> >> > >>>>> ATSv2. Patch in progress.
>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> Improvement
>> >> > >>>>> works.
>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>> >> > >>>>> Storage. In progress.
>> >> > >>>>>
>> >> > >>>>> 3. Tentative features:
>> >> > >>>>>
>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >> challenging
>> >> > >>>>> to be done before Aug 2018.
>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> >> > >>>>> Challenging as more discussions are on-going.
>> >> > >>>>>
>> >> > >>>>> *Summary of 3.2.0 issues status:*
>> >> > >>>>>
>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking with
>> >> > owners
>> >> > >>>>> to get status on each of them to get in by Code Freeze date.
>> >> > >>>>>
>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> >> > Version/s" =
>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>> >> > >>>>>
>> >> > >>>>> Thanks,
>> >> > >>>>> Sunil
>> >> > >>>>>
>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org>
>> >> wrote:
>> >> > >>>>>
>> >> > >>>>>> Thanks Subru for the thoughts.
>> >> > >>>>>> One of the main reason for a major release is to push out
>> >> critical
>> >> > >>>>>> features with a faster cadence to the users. If we are pulling
>> >> more
>> >> > and
>> >> > >>>>>> more different types of features to a minor release, that
>> branch
>> >> > will
>> >> > >>>>>> become more destabilized and it may be tough to say that
>> 3.1.2 is
>> >> > stable
>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
>> >> features
>> >> > in
>> >> > >>>>>> subsequent minor release.
>> >> > >>>>>> For few companies, it makes sense to push out these new
>> features
>> >> > >>>>>> faster to make a reach to the users. Adding to the point to
>> the
>> >> > backporting
>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
>> with
>> >> > some git
>> >> > >>>>>> scripts. If we can make such scripts available to committers,
>> >> > backport will
>> >> > >>>>>> be seem-less across branches and we can achieve the faster
>> >> release
>> >> > cadence
>> >> > >>>>>> also.
>> >> > >>>>>>
>> >> > >>>>>> Thoughts?
>> >> > >>>>>>
>> >> > >>>>>> - Sunil
>> >> > >>>>>>
>> >> > >>>>>>
>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>> subru@apache.org
>> >> >
>> >> > >>>>>> wrote:
>> >> > >>>>>>
>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release effort. I
>> am
>> >> > >>>>>>> generally
>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as
>> >> feel we
>> >> > >>>>>>> already
>> >> > >>>>>>> have too many branches to be maintained. I already see many
>> >> commits
>> >> > >>>>>>> are in
>> >> > >>>>>>> different branches with no apparent rationale, for e.g: 3.1
>> has
>> >> > >>>>>>> commits
>> >> > >>>>>>> which are absent in 3.0 etc.
>> >> > >>>>>>>
>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
>> >> > production
>> >> > >>>>>>> setting so the cost of adding features should be minimal.
>> >> > >>>>>>>
>> >> > >>>>>>> Thoughts?
>> >> > >>>>>>>
>> >> > >>>>>>> -Subru
>> >> > >>>>>>>
>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <sunilg@apache.org
>> >
>> >> > wrote:
>> >> > >>>>>>>
>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>> >> > >>>>>>> >
>> >> > >>>>>>> > Yes, important changes and features are much needed, hence
>> we
>> >> > will
>> >> > >>>>>>> be
>> >> > >>>>>>> > keeping the door open for them as possible. Also
>> considering
>> >> few
>> >> > >>>>>>> more
>> >> > >>>>>>> > offline requests from other folks, I think extending the
>> >> > timeframe
>> >> > >>>>>>> by
>> >> > >>>>>>> > couple of weeks makes sense (including a second RC buffer)
>> and
>> >> > >>>>>>> this should
>> >> > >>>>>>> > ideally help us to ship this by September itself.
>> >> > >>>>>>> >
>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as
>> >> well)
>> >> > >>>>>>> >
>> >> > >>>>>>> > - Feature freeze date : all features to merge by August 21,
>> >> 2018.
>> >> > >>>>>>> >
>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>> improvements
>> >> and
>> >> > >>>>>>> non
>> >> > >>>>>>> > blocker/critical
>> >> > >>>>>>> >
>> >> > >>>>>>> > bug-fixes August 31, 2018.
>> >> > >>>>>>> >
>> >> > >>>>>>> > - Release date: September 15, 2018
>> >> > >>>>>>> >
>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered
>> your
>> >> > >>>>>>> questions.
>> >> > >>>>>>> >
>> >> > >>>>>>> > Thanks
>> >> > >>>>>>> > Sunil
>> >> > >>>>>>> >
>> >> > >>>>>>> >
>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>> >> wheeleast@gmail.com
>> >> > >
>> >> > >>>>>>> wrote:
>> >> > >>>>>>> >
>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1
>> >> for
>> >> > >>>>>>> that.
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > To concerns from Steve,
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > It is a good idea to keep the door open to get important
>> >> > changes
>> >> > >>>>>>> /
>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
>> >> proposed
>> >> > >>>>>>> release
>> >> > >>>>>>> > date
>> >> > >>>>>>> > > to make sure things can happen earlier instead of last
>> >> minute
>> >> > >>>>>>> and we all
>> >> > >>>>>>> > > know that releases are always get delayed :). I'm also
>> fine
>> >> if
>> >> > >>>>>>> we want
>> >> > >>>>>>> > get
>> >> > >>>>>>> > > another several weeks time.
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
>> before
>> >> > >>>>>>> thanksgiving.
>> >> > >>>>>>> > > Do you think is it good or too early / late?
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > Eric,
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135
>> >> can
>> >> > >>>>>>> get merged
>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > Sunil,
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > Could u update
>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>> >> > >>>>>>> > Roadmap
>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
>> first
>> >> > >>>>>>> before getting
>> >> > >>>>>>> > > consensus of time.
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > Thanks,
>> >> > >>>>>>> > > Wangda
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>> >> > >>>>>>> <fabbri@cloudera.com.invalid
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > wrote:
>> >> > >>>>>>> > >
>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>> >> > >>>>>>> stevel@hortonworks.com>
>> >> > >>>>>>> > >> wrote:
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org
>> >> > <mailto:
>> >> > >>>>>>> > >> > sunilg@apache.org>> wrote:
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > I would also would like to take this opportunity to
>> come
>> >> up
>> >> > >>>>>>> with a
>> >> > >>>>>>> > >> detailed
>> >> > >>>>>>> > >> > plan.
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > - Feature freeze date : all features should be merged
>> by
>> >> > >>>>>>> August 10,
>> >> > >>>>>>> > >> 2018.
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > <snip>
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > Please let me know if I missed any features targeted
>> to
>> >> 3.2
>> >> > >>>>>>> per this
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
>> Datalake
>> >> > Gen
>> >> > >>>>>>> 2
>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been doing
>> >> work
>> >> > >>>>>>> on it, but
>> >> > >>>>>>> > >> if
>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
>> >> timely
>> >> > >>>>>>> reviews on
>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will review
>> the
>> >> S3 &
>> >> > >>>>>>> S3Guard
>> >> > >>>>>>> > >> JIRAs
>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
>> some
>> >> > >>>>>>> pressing
>> >> > >>>>>>> > "guave,
>> >> > >>>>>>> > >> > java 9 prep"
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> I can help with this part if you like.
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as release
>> >> > manager
>> >> > >>>>>>> of 3.2.0
>> >> > >>>>>>> > >> > release.
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > well volunteered!
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > I think this raises a good q: what timetable should we
>> >> have
>> >> > >>>>>>> for the
>> >> > >>>>>>> > >> 3.2. &
>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
>> having
>> >> > the
>> >> > >>>>>>> outline
>> >> > >>>>>>> > >> time
>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
>> less
>> >> > >>>>>>> concern about
>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > -Steve
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >> -AF
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >
>> >> > >>>>>>> >
>> >> > >>>>>>>
>> >> > >>>>>>
>> >> >
>> >>
>> >
>>
>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi Folks,
We are now close to RC as other blocker issues are now merged to trunk and
branch-3.2. Last 2 critical issues are closer to merge and will be
committed in few hours.
With this, I will be creating 3.2.0 branch today and will go ahead with RC
related process.
- Sunil
On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jo...@stripe.com>
wrote:
> Hello, were there any updates around the 3.2.0 RC timing? All I see in the
> current blockers are related to the new Submarine subproject, wasn't sure
> if that is what is holding things up.
>
> Cheers,
> Jon
>
> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <su...@apache.org> wrote:
>
>> Thanks Robert and Haibo for quickly correcting same.
>> Sigh, I somehow missed one file while committing the change. Sorry for the
>> trouble.
>>
>> - Sunil
>>
>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rk...@cloudera.com>
>> wrote:
>>
>> > Looks like there's two that weren't updated:
>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
>> > --include=pom.xml
>> > ./hadoop-project/pom.xml:
>> > <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>> >
>> > I've just pushed in an addendum commit to fix those.
>> > In the future, please make sure to do a sanity compile when updating
>> poms.
>> >
>> > thanks
>> > - Robert
>> >
>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>> <fa...@cloudera.com.invalid>
>> > wrote:
>> >
>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
>> >> top-level pom.xml?
>> >>
>> >>
>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
>> >>
>> >> > Hi All
>> >> >
>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset trunk
>> to
>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary
>> >> > patches are pulled into branch-3.2.
>> >> >
>> >> > Thank You
>> >> > - Sunil
>> >> >
>> >> >
>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
>> >> >
>> >> > > Hi All
>> >> > >
>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged to
>> >> trunk.
>> >> > > Thanks for the support.
>> >> > >
>> >> > > *Plan for RC*
>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>> >> > > RC0 for 3.2: *28th Friday*
>> >> > >
>> >> > > Thank You
>> >> > > Sunil
>> >> > >
>> >> > >
>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
>> >> > >
>> >> > >> Hi All
>> >> > >>
>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you
>> for
>> >> > >> helping in this. I am following up on these tickets, once its
>> closed
>> >> we
>> >> > >> will cut the 3.2 branch.
>> >> > >>
>> >> > >> Thanks
>> >> > >> Sunil Govindan
>> >> > >>
>> >> > >>
>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org>
>> wrote:
>> >> > >>
>> >> > >>> Hi All,
>> >> > >>>
>> >> > >>> Inline with the original 3.2 communication proposal dated 17th
>> July
>> >> > >>> 2018, I would like to provide more updates.
>> >> > >>>
>> >> > >>> We are approaching previously proposed code freeze date
>> (September
>> >> 14,
>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
>> >> > existing
>> >> > >>> trunk to 3.3 if there are no issues.
>> >> > >>>
>> >> > >>> *Current Release Plan:*
>> >> > >>> Feature freeze date : all features to merge by September 7, 2018.
>> >> > >>> Code freeze date : blockers/critical only, no improvements and
>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>> >> > >>> Release date: September 28, 2018
>> >> > >>>
>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0, we
>> >> need to
>> >> > >>> backport to 3.2 post branch cut.
>> >> > >>>
>> >> > >>> Here's an updated 3.2.0 feature status:
>> >> > >>>
>> >> > >>> 1. Merged & Completed features:
>> >> > >>>
>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> >> > >>> workloads Initial cut.
>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> >> > >>> Scheduler.
>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> Service
>> >> > API
>> >> > >>> and CLI.
>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement
>> >> works.
>> >> > >>>
>> >> > >>> 2. Features close to finish:
>> >> > >>>
>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>> >> > >>> - (Steve) S3a phase V. Close to commit.
>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>> >> > >>>
>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>> from
>> >> > >>> ATSv2. Patch in progress.
>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >> challenging to
>> >> > >>> be done before Aug 2018.
>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> >> > >>> Challenging as more discussions are on-going.
>> >> > >>>
>> >> > >>> *Summary of 3.2.0 issues status:*
>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following up
>> with
>> >> > >>> owners to get status on each of them to get in by Code Freeze
>> date.
>> >> > >>>
>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> >> > (Blocker,
>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" =
>> 3.2.0
>> >> > ORDER
>> >> > >>> BY priority DESC
>> >> > >>>
>> >> > >>> Thanks,
>> >> > >>> Sunil
>> >> > >>>
>> >> > >>>
>> >> > >>>
>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org>
>> wrote:
>> >> > >>>
>> >> > >>>> Hi All,
>> >> > >>>>
>> >> > >>>> Inline with earlier communication dated 17th July 2018, I would
>> >> like
>> >> > to
>> >> > >>>> provide some updates.
>> >> > >>>>
>> >> > >>>> We are approaching previously proposed code freeze date (Aug
>> 31).
>> >> > >>>>
>> >> > >>>> One of the critical feature Node Attributes feature merge
>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need a
>> bit
>> >> > more
>> >> > >>>> time. With regard to this, suggesting to push the feature/code
>> >> freeze
>> >> > for 2
>> >> > >>>> more weeks to accommodate these jiras too.
>> >> > >>>>
>> >> > >>>> Proposing Updated changes in plan inline with this:
>> >> > >>>> Feature freeze date : all features to merge by September 7,
>> 2018.
>> >> > >>>> Code freeze date : blockers/critical only, no improvements and
>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>> >> > >>>> Release date: September 28, 2018
>> >> > >>>>
>> >> > >>>> If any features in branch which are targeted to 3.2.0, please
>> >> reply to
>> >> > >>>> this email thread.
>> >> > >>>>
>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>> >> > >>>>
>> >> > >>>> 1. Merged & Completed features:
>> >> > >>>>
>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> >> > >>>> workloads Initial cut.
>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> >> > >>>> Scheduler.
>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> >> Service
>> >> > >>>> API and CLI.
>> >> > >>>>
>> >> > >>>> 2. Features close to finish:
>> >> > >>>>
>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> >> Merge/Vote
>> >> > >>>> Ongoing.
>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>> from
>> >> > >>>> ATSv2. Patch in progress.
>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> Improvement
>> >> > works.
>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>> >> > >>>> Storage. In progress.
>> >> > >>>>
>> >> > >>>> 3. Tentative features:
>> >> > >>>>
>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >> challenging
>> >> > to
>> >> > >>>> be done before Aug 2018.
>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> >> > >>>> Challenging as more discussions are on-going.
>> >> > >>>>
>> >> > >>>> *Summary of 3.2.0 issues status:*
>> >> > >>>>
>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following up
>> with
>> >> > >>>> owners to get status on each of them to get in by Code Freeze
>> date.
>> >> > >>>>
>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> >> > Version/s" =
>> >> > >>>> 3.2.0 ORDER BY priority DESC
>> >> > >>>>
>> >> > >>>> Thanks,
>> >> > >>>> Sunil
>> >> > >>>>
>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org>
>> >> wrote:
>> >> > >>>>
>> >> > >>>>> Hi All,
>> >> > >>>>>
>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
>> dated
>> >> > 17th
>> >> > >>>>> July 2018, I would like to provide some updates.
>> >> > >>>>>
>> >> > >>>>> We are approaching previously proposed feature freeze date (Aug
>> >> 21,
>> >> > >>>>> about 7 days from today).
>> >> > >>>>> If any features in branch which are targeted to 3.2.0, please
>> >> reply
>> >> > to
>> >> > >>>>> this email thread.
>> >> > >>>>> Steve has mentioned about the s3 features which will come
>> close to
>> >> > >>>>> Code Freeze Date (Aug 31st).
>> >> > >>>>>
>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>> >> > >>>>>
>> >> > >>>>> 1. Merged & Completed features:
>> >> > >>>>>
>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> >> > >>>>> workloads Initial cut.
>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >> > >>>>>
>> >> > >>>>> 2. Features close to finish:
>> >> > >>>>>
>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> Major
>> >> > >>>>> patches are all in, only one last
>> >> > >>>>> patch is in review state.
>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> >> > >>>>> Scheduler. Close to commit.
>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> >> Service
>> >> > >>>>> API and CLI. 2 patches are pending
>> >> > >>>>> which will be closed by Feature freeze date.
>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>> >> from
>> >> > >>>>> ATSv2. Patch in progress.
>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> Improvement
>> >> > >>>>> works.
>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>> >> > >>>>> Storage. In progress.
>> >> > >>>>>
>> >> > >>>>> 3. Tentative features:
>> >> > >>>>>
>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >> challenging
>> >> > >>>>> to be done before Aug 2018.
>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> >> > >>>>> Challenging as more discussions are on-going.
>> >> > >>>>>
>> >> > >>>>> *Summary of 3.2.0 issues status:*
>> >> > >>>>>
>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking with
>> >> > owners
>> >> > >>>>> to get status on each of them to get in by Code Freeze date.
>> >> > >>>>>
>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> >> > Version/s" =
>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>> >> > >>>>>
>> >> > >>>>> Thanks,
>> >> > >>>>> Sunil
>> >> > >>>>>
>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org>
>> >> wrote:
>> >> > >>>>>
>> >> > >>>>>> Thanks Subru for the thoughts.
>> >> > >>>>>> One of the main reason for a major release is to push out
>> >> critical
>> >> > >>>>>> features with a faster cadence to the users. If we are pulling
>> >> more
>> >> > and
>> >> > >>>>>> more different types of features to a minor release, that
>> branch
>> >> > will
>> >> > >>>>>> become more destabilized and it may be tough to say that
>> 3.1.2 is
>> >> > stable
>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
>> >> features
>> >> > in
>> >> > >>>>>> subsequent minor release.
>> >> > >>>>>> For few companies, it makes sense to push out these new
>> features
>> >> > >>>>>> faster to make a reach to the users. Adding to the point to
>> the
>> >> > backporting
>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
>> with
>> >> > some git
>> >> > >>>>>> scripts. If we can make such scripts available to committers,
>> >> > backport will
>> >> > >>>>>> be seem-less across branches and we can achieve the faster
>> >> release
>> >> > cadence
>> >> > >>>>>> also.
>> >> > >>>>>>
>> >> > >>>>>> Thoughts?
>> >> > >>>>>>
>> >> > >>>>>> - Sunil
>> >> > >>>>>>
>> >> > >>>>>>
>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>> subru@apache.org
>> >> >
>> >> > >>>>>> wrote:
>> >> > >>>>>>
>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release effort. I
>> am
>> >> > >>>>>>> generally
>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as
>> >> feel we
>> >> > >>>>>>> already
>> >> > >>>>>>> have too many branches to be maintained. I already see many
>> >> commits
>> >> > >>>>>>> are in
>> >> > >>>>>>> different branches with no apparent rationale, for e.g: 3.1
>> has
>> >> > >>>>>>> commits
>> >> > >>>>>>> which are absent in 3.0 etc.
>> >> > >>>>>>>
>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
>> >> > production
>> >> > >>>>>>> setting so the cost of adding features should be minimal.
>> >> > >>>>>>>
>> >> > >>>>>>> Thoughts?
>> >> > >>>>>>>
>> >> > >>>>>>> -Subru
>> >> > >>>>>>>
>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <sunilg@apache.org
>> >
>> >> > wrote:
>> >> > >>>>>>>
>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>> >> > >>>>>>> >
>> >> > >>>>>>> > Yes, important changes and features are much needed, hence
>> we
>> >> > will
>> >> > >>>>>>> be
>> >> > >>>>>>> > keeping the door open for them as possible. Also
>> considering
>> >> few
>> >> > >>>>>>> more
>> >> > >>>>>>> > offline requests from other folks, I think extending the
>> >> > timeframe
>> >> > >>>>>>> by
>> >> > >>>>>>> > couple of weeks makes sense (including a second RC buffer)
>> and
>> >> > >>>>>>> this should
>> >> > >>>>>>> > ideally help us to ship this by September itself.
>> >> > >>>>>>> >
>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as
>> >> well)
>> >> > >>>>>>> >
>> >> > >>>>>>> > - Feature freeze date : all features to merge by August 21,
>> >> 2018.
>> >> > >>>>>>> >
>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>> improvements
>> >> and
>> >> > >>>>>>> non
>> >> > >>>>>>> > blocker/critical
>> >> > >>>>>>> >
>> >> > >>>>>>> > bug-fixes August 31, 2018.
>> >> > >>>>>>> >
>> >> > >>>>>>> > - Release date: September 15, 2018
>> >> > >>>>>>> >
>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered
>> your
>> >> > >>>>>>> questions.
>> >> > >>>>>>> >
>> >> > >>>>>>> > Thanks
>> >> > >>>>>>> > Sunil
>> >> > >>>>>>> >
>> >> > >>>>>>> >
>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>> >> wheeleast@gmail.com
>> >> > >
>> >> > >>>>>>> wrote:
>> >> > >>>>>>> >
>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1
>> >> for
>> >> > >>>>>>> that.
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > To concerns from Steve,
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > It is a good idea to keep the door open to get important
>> >> > changes
>> >> > >>>>>>> /
>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
>> >> proposed
>> >> > >>>>>>> release
>> >> > >>>>>>> > date
>> >> > >>>>>>> > > to make sure things can happen earlier instead of last
>> >> minute
>> >> > >>>>>>> and we all
>> >> > >>>>>>> > > know that releases are always get delayed :). I'm also
>> fine
>> >> if
>> >> > >>>>>>> we want
>> >> > >>>>>>> > get
>> >> > >>>>>>> > > another several weeks time.
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
>> before
>> >> > >>>>>>> thanksgiving.
>> >> > >>>>>>> > > Do you think is it good or too early / late?
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > Eric,
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135
>> >> can
>> >> > >>>>>>> get merged
>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > Sunil,
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > Could u update
>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>> >> > >>>>>>> > Roadmap
>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
>> first
>> >> > >>>>>>> before getting
>> >> > >>>>>>> > > consensus of time.
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > Thanks,
>> >> > >>>>>>> > > Wangda
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>> >> > >>>>>>> <fabbri@cloudera.com.invalid
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > wrote:
>> >> > >>>>>>> > >
>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>> >> > >>>>>>> stevel@hortonworks.com>
>> >> > >>>>>>> > >> wrote:
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org
>> >> > <mailto:
>> >> > >>>>>>> > >> > sunilg@apache.org>> wrote:
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > I would also would like to take this opportunity to
>> come
>> >> up
>> >> > >>>>>>> with a
>> >> > >>>>>>> > >> detailed
>> >> > >>>>>>> > >> > plan.
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > - Feature freeze date : all features should be merged
>> by
>> >> > >>>>>>> August 10,
>> >> > >>>>>>> > >> 2018.
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > <snip>
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > Please let me know if I missed any features targeted
>> to
>> >> 3.2
>> >> > >>>>>>> per this
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
>> Datalake
>> >> > Gen
>> >> > >>>>>>> 2
>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been doing
>> >> work
>> >> > >>>>>>> on it, but
>> >> > >>>>>>> > >> if
>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
>> >> timely
>> >> > >>>>>>> reviews on
>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will review
>> the
>> >> S3 &
>> >> > >>>>>>> S3Guard
>> >> > >>>>>>> > >> JIRAs
>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
>> some
>> >> > >>>>>>> pressing
>> >> > >>>>>>> > "guave,
>> >> > >>>>>>> > >> > java 9 prep"
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> I can help with this part if you like.
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as release
>> >> > manager
>> >> > >>>>>>> of 3.2.0
>> >> > >>>>>>> > >> > release.
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > well volunteered!
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > I think this raises a good q: what timetable should we
>> >> have
>> >> > >>>>>>> for the
>> >> > >>>>>>> > >> 3.2. &
>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
>> having
>> >> > the
>> >> > >>>>>>> outline
>> >> > >>>>>>> > >> time
>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
>> less
>> >> > >>>>>>> concern about
>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > -Steve
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >> -AF
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >
>> >> > >>>>>>> >
>> >> > >>>>>>>
>> >> > >>>>>>
>> >> >
>> >>
>> >
>>
>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi Folks,
We are now close to RC as other blocker issues are now merged to trunk and
branch-3.2. Last 2 critical issues are closer to merge and will be
committed in few hours.
With this, I will be creating 3.2.0 branch today and will go ahead with RC
related process.
- Sunil
On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jo...@stripe.com>
wrote:
> Hello, were there any updates around the 3.2.0 RC timing? All I see in the
> current blockers are related to the new Submarine subproject, wasn't sure
> if that is what is holding things up.
>
> Cheers,
> Jon
>
> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <su...@apache.org> wrote:
>
>> Thanks Robert and Haibo for quickly correcting same.
>> Sigh, I somehow missed one file while committing the change. Sorry for the
>> trouble.
>>
>> - Sunil
>>
>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rk...@cloudera.com>
>> wrote:
>>
>> > Looks like there's two that weren't updated:
>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
>> > --include=pom.xml
>> > ./hadoop-project/pom.xml:
>> > <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>> >
>> > I've just pushed in an addendum commit to fix those.
>> > In the future, please make sure to do a sanity compile when updating
>> poms.
>> >
>> > thanks
>> > - Robert
>> >
>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>> <fa...@cloudera.com.invalid>
>> > wrote:
>> >
>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
>> >> top-level pom.xml?
>> >>
>> >>
>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
>> >>
>> >> > Hi All
>> >> >
>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset trunk
>> to
>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary
>> >> > patches are pulled into branch-3.2.
>> >> >
>> >> > Thank You
>> >> > - Sunil
>> >> >
>> >> >
>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
>> >> >
>> >> > > Hi All
>> >> > >
>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged to
>> >> trunk.
>> >> > > Thanks for the support.
>> >> > >
>> >> > > *Plan for RC*
>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>> >> > > RC0 for 3.2: *28th Friday*
>> >> > >
>> >> > > Thank You
>> >> > > Sunil
>> >> > >
>> >> > >
>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
>> >> > >
>> >> > >> Hi All
>> >> > >>
>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you
>> for
>> >> > >> helping in this. I am following up on these tickets, once its
>> closed
>> >> we
>> >> > >> will cut the 3.2 branch.
>> >> > >>
>> >> > >> Thanks
>> >> > >> Sunil Govindan
>> >> > >>
>> >> > >>
>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org>
>> wrote:
>> >> > >>
>> >> > >>> Hi All,
>> >> > >>>
>> >> > >>> Inline with the original 3.2 communication proposal dated 17th
>> July
>> >> > >>> 2018, I would like to provide more updates.
>> >> > >>>
>> >> > >>> We are approaching previously proposed code freeze date
>> (September
>> >> 14,
>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
>> >> > existing
>> >> > >>> trunk to 3.3 if there are no issues.
>> >> > >>>
>> >> > >>> *Current Release Plan:*
>> >> > >>> Feature freeze date : all features to merge by September 7, 2018.
>> >> > >>> Code freeze date : blockers/critical only, no improvements and
>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>> >> > >>> Release date: September 28, 2018
>> >> > >>>
>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0, we
>> >> need to
>> >> > >>> backport to 3.2 post branch cut.
>> >> > >>>
>> >> > >>> Here's an updated 3.2.0 feature status:
>> >> > >>>
>> >> > >>> 1. Merged & Completed features:
>> >> > >>>
>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> >> > >>> workloads Initial cut.
>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> >> > >>> Scheduler.
>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> Service
>> >> > API
>> >> > >>> and CLI.
>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement
>> >> works.
>> >> > >>>
>> >> > >>> 2. Features close to finish:
>> >> > >>>
>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>> >> > >>> - (Steve) S3a phase V. Close to commit.
>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>> >> > >>>
>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>> from
>> >> > >>> ATSv2. Patch in progress.
>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >> challenging to
>> >> > >>> be done before Aug 2018.
>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> >> > >>> Challenging as more discussions are on-going.
>> >> > >>>
>> >> > >>> *Summary of 3.2.0 issues status:*
>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following up
>> with
>> >> > >>> owners to get status on each of them to get in by Code Freeze
>> date.
>> >> > >>>
>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> >> > (Blocker,
>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" =
>> 3.2.0
>> >> > ORDER
>> >> > >>> BY priority DESC
>> >> > >>>
>> >> > >>> Thanks,
>> >> > >>> Sunil
>> >> > >>>
>> >> > >>>
>> >> > >>>
>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org>
>> wrote:
>> >> > >>>
>> >> > >>>> Hi All,
>> >> > >>>>
>> >> > >>>> Inline with earlier communication dated 17th July 2018, I would
>> >> like
>> >> > to
>> >> > >>>> provide some updates.
>> >> > >>>>
>> >> > >>>> We are approaching previously proposed code freeze date (Aug
>> 31).
>> >> > >>>>
>> >> > >>>> One of the critical feature Node Attributes feature merge
>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need a
>> bit
>> >> > more
>> >> > >>>> time. With regard to this, suggesting to push the feature/code
>> >> freeze
>> >> > for 2
>> >> > >>>> more weeks to accommodate these jiras too.
>> >> > >>>>
>> >> > >>>> Proposing Updated changes in plan inline with this:
>> >> > >>>> Feature freeze date : all features to merge by September 7,
>> 2018.
>> >> > >>>> Code freeze date : blockers/critical only, no improvements and
>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>> >> > >>>> Release date: September 28, 2018
>> >> > >>>>
>> >> > >>>> If any features in branch which are targeted to 3.2.0, please
>> >> reply to
>> >> > >>>> this email thread.
>> >> > >>>>
>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>> >> > >>>>
>> >> > >>>> 1. Merged & Completed features:
>> >> > >>>>
>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> >> > >>>> workloads Initial cut.
>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> >> > >>>> Scheduler.
>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> >> Service
>> >> > >>>> API and CLI.
>> >> > >>>>
>> >> > >>>> 2. Features close to finish:
>> >> > >>>>
>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> >> Merge/Vote
>> >> > >>>> Ongoing.
>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>> from
>> >> > >>>> ATSv2. Patch in progress.
>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> Improvement
>> >> > works.
>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>> >> > >>>> Storage. In progress.
>> >> > >>>>
>> >> > >>>> 3. Tentative features:
>> >> > >>>>
>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >> challenging
>> >> > to
>> >> > >>>> be done before Aug 2018.
>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> >> > >>>> Challenging as more discussions are on-going.
>> >> > >>>>
>> >> > >>>> *Summary of 3.2.0 issues status:*
>> >> > >>>>
>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following up
>> with
>> >> > >>>> owners to get status on each of them to get in by Code Freeze
>> date.
>> >> > >>>>
>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> >> > Version/s" =
>> >> > >>>> 3.2.0 ORDER BY priority DESC
>> >> > >>>>
>> >> > >>>> Thanks,
>> >> > >>>> Sunil
>> >> > >>>>
>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org>
>> >> wrote:
>> >> > >>>>
>> >> > >>>>> Hi All,
>> >> > >>>>>
>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
>> dated
>> >> > 17th
>> >> > >>>>> July 2018, I would like to provide some updates.
>> >> > >>>>>
>> >> > >>>>> We are approaching previously proposed feature freeze date (Aug
>> >> 21,
>> >> > >>>>> about 7 days from today).
>> >> > >>>>> If any features in branch which are targeted to 3.2.0, please
>> >> reply
>> >> > to
>> >> > >>>>> this email thread.
>> >> > >>>>> Steve has mentioned about the s3 features which will come
>> close to
>> >> > >>>>> Code Freeze Date (Aug 31st).
>> >> > >>>>>
>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>> >> > >>>>>
>> >> > >>>>> 1. Merged & Completed features:
>> >> > >>>>>
>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> >> > >>>>> workloads Initial cut.
>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >> > >>>>>
>> >> > >>>>> 2. Features close to finish:
>> >> > >>>>>
>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> Major
>> >> > >>>>> patches are all in, only one last
>> >> > >>>>> patch is in review state.
>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> >> > >>>>> Scheduler. Close to commit.
>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> >> Service
>> >> > >>>>> API and CLI. 2 patches are pending
>> >> > >>>>> which will be closed by Feature freeze date.
>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>> >> from
>> >> > >>>>> ATSv2. Patch in progress.
>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> Improvement
>> >> > >>>>> works.
>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>> >> > >>>>> Storage. In progress.
>> >> > >>>>>
>> >> > >>>>> 3. Tentative features:
>> >> > >>>>>
>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >> challenging
>> >> > >>>>> to be done before Aug 2018.
>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> >> > >>>>> Challenging as more discussions are on-going.
>> >> > >>>>>
>> >> > >>>>> *Summary of 3.2.0 issues status:*
>> >> > >>>>>
>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking with
>> >> > owners
>> >> > >>>>> to get status on each of them to get in by Code Freeze date.
>> >> > >>>>>
>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> >> > Version/s" =
>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>> >> > >>>>>
>> >> > >>>>> Thanks,
>> >> > >>>>> Sunil
>> >> > >>>>>
>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org>
>> >> wrote:
>> >> > >>>>>
>> >> > >>>>>> Thanks Subru for the thoughts.
>> >> > >>>>>> One of the main reason for a major release is to push out
>> >> critical
>> >> > >>>>>> features with a faster cadence to the users. If we are pulling
>> >> more
>> >> > and
>> >> > >>>>>> more different types of features to a minor release, that
>> branch
>> >> > will
>> >> > >>>>>> become more destabilized and it may be tough to say that
>> 3.1.2 is
>> >> > stable
>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
>> >> features
>> >> > in
>> >> > >>>>>> subsequent minor release.
>> >> > >>>>>> For few companies, it makes sense to push out these new
>> features
>> >> > >>>>>> faster to make a reach to the users. Adding to the point to
>> the
>> >> > backporting
>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
>> with
>> >> > some git
>> >> > >>>>>> scripts. If we can make such scripts available to committers,
>> >> > backport will
>> >> > >>>>>> be seem-less across branches and we can achieve the faster
>> >> release
>> >> > cadence
>> >> > >>>>>> also.
>> >> > >>>>>>
>> >> > >>>>>> Thoughts?
>> >> > >>>>>>
>> >> > >>>>>> - Sunil
>> >> > >>>>>>
>> >> > >>>>>>
>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>> subru@apache.org
>> >> >
>> >> > >>>>>> wrote:
>> >> > >>>>>>
>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release effort. I
>> am
>> >> > >>>>>>> generally
>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as
>> >> feel we
>> >> > >>>>>>> already
>> >> > >>>>>>> have too many branches to be maintained. I already see many
>> >> commits
>> >> > >>>>>>> are in
>> >> > >>>>>>> different branches with no apparent rationale, for e.g: 3.1
>> has
>> >> > >>>>>>> commits
>> >> > >>>>>>> which are absent in 3.0 etc.
>> >> > >>>>>>>
>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
>> >> > production
>> >> > >>>>>>> setting so the cost of adding features should be minimal.
>> >> > >>>>>>>
>> >> > >>>>>>> Thoughts?
>> >> > >>>>>>>
>> >> > >>>>>>> -Subru
>> >> > >>>>>>>
>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <sunilg@apache.org
>> >
>> >> > wrote:
>> >> > >>>>>>>
>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>> >> > >>>>>>> >
>> >> > >>>>>>> > Yes, important changes and features are much needed, hence
>> we
>> >> > will
>> >> > >>>>>>> be
>> >> > >>>>>>> > keeping the door open for them as possible. Also
>> considering
>> >> few
>> >> > >>>>>>> more
>> >> > >>>>>>> > offline requests from other folks, I think extending the
>> >> > timeframe
>> >> > >>>>>>> by
>> >> > >>>>>>> > couple of weeks makes sense (including a second RC buffer)
>> and
>> >> > >>>>>>> this should
>> >> > >>>>>>> > ideally help us to ship this by September itself.
>> >> > >>>>>>> >
>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as
>> >> well)
>> >> > >>>>>>> >
>> >> > >>>>>>> > - Feature freeze date : all features to merge by August 21,
>> >> 2018.
>> >> > >>>>>>> >
>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>> improvements
>> >> and
>> >> > >>>>>>> non
>> >> > >>>>>>> > blocker/critical
>> >> > >>>>>>> >
>> >> > >>>>>>> > bug-fixes August 31, 2018.
>> >> > >>>>>>> >
>> >> > >>>>>>> > - Release date: September 15, 2018
>> >> > >>>>>>> >
>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered
>> your
>> >> > >>>>>>> questions.
>> >> > >>>>>>> >
>> >> > >>>>>>> > Thanks
>> >> > >>>>>>> > Sunil
>> >> > >>>>>>> >
>> >> > >>>>>>> >
>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>> >> wheeleast@gmail.com
>> >> > >
>> >> > >>>>>>> wrote:
>> >> > >>>>>>> >
>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1
>> >> for
>> >> > >>>>>>> that.
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > To concerns from Steve,
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > It is a good idea to keep the door open to get important
>> >> > changes
>> >> > >>>>>>> /
>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
>> >> proposed
>> >> > >>>>>>> release
>> >> > >>>>>>> > date
>> >> > >>>>>>> > > to make sure things can happen earlier instead of last
>> >> minute
>> >> > >>>>>>> and we all
>> >> > >>>>>>> > > know that releases are always get delayed :). I'm also
>> fine
>> >> if
>> >> > >>>>>>> we want
>> >> > >>>>>>> > get
>> >> > >>>>>>> > > another several weeks time.
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
>> before
>> >> > >>>>>>> thanksgiving.
>> >> > >>>>>>> > > Do you think is it good or too early / late?
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > Eric,
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135
>> >> can
>> >> > >>>>>>> get merged
>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > Sunil,
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > Could u update
>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>> >> > >>>>>>> > Roadmap
>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
>> first
>> >> > >>>>>>> before getting
>> >> > >>>>>>> > > consensus of time.
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > Thanks,
>> >> > >>>>>>> > > Wangda
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>> >> > >>>>>>> <fabbri@cloudera.com.invalid
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > wrote:
>> >> > >>>>>>> > >
>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>> >> > >>>>>>> stevel@hortonworks.com>
>> >> > >>>>>>> > >> wrote:
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org
>> >> > <mailto:
>> >> > >>>>>>> > >> > sunilg@apache.org>> wrote:
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > I would also would like to take this opportunity to
>> come
>> >> up
>> >> > >>>>>>> with a
>> >> > >>>>>>> > >> detailed
>> >> > >>>>>>> > >> > plan.
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > - Feature freeze date : all features should be merged
>> by
>> >> > >>>>>>> August 10,
>> >> > >>>>>>> > >> 2018.
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > <snip>
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > Please let me know if I missed any features targeted
>> to
>> >> 3.2
>> >> > >>>>>>> per this
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
>> Datalake
>> >> > Gen
>> >> > >>>>>>> 2
>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been doing
>> >> work
>> >> > >>>>>>> on it, but
>> >> > >>>>>>> > >> if
>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
>> >> timely
>> >> > >>>>>>> reviews on
>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will review
>> the
>> >> S3 &
>> >> > >>>>>>> S3Guard
>> >> > >>>>>>> > >> JIRAs
>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
>> some
>> >> > >>>>>>> pressing
>> >> > >>>>>>> > "guave,
>> >> > >>>>>>> > >> > java 9 prep"
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> I can help with this part if you like.
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as release
>> >> > manager
>> >> > >>>>>>> of 3.2.0
>> >> > >>>>>>> > >> > release.
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > well volunteered!
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > I think this raises a good q: what timetable should we
>> >> have
>> >> > >>>>>>> for the
>> >> > >>>>>>> > >> 3.2. &
>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
>> having
>> >> > the
>> >> > >>>>>>> outline
>> >> > >>>>>>> > >> time
>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
>> less
>> >> > >>>>>>> concern about
>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > -Steve
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >> -AF
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >
>> >> > >>>>>>> >
>> >> > >>>>>>>
>> >> > >>>>>>
>> >> >
>> >>
>> >
>>
>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi Folks,
We are now close to RC as other blocker issues are now merged to trunk and
branch-3.2. Last 2 critical issues are closer to merge and will be
committed in few hours.
With this, I will be creating 3.2.0 branch today and will go ahead with RC
related process.
- Sunil
On Mon, Oct 15, 2018 at 11:43 PM Jonathan Bender <jo...@stripe.com>
wrote:
> Hello, were there any updates around the 3.2.0 RC timing? All I see in the
> current blockers are related to the new Submarine subproject, wasn't sure
> if that is what is holding things up.
>
> Cheers,
> Jon
>
> On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <su...@apache.org> wrote:
>
>> Thanks Robert and Haibo for quickly correcting same.
>> Sigh, I somehow missed one file while committing the change. Sorry for the
>> trouble.
>>
>> - Sunil
>>
>> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rk...@cloudera.com>
>> wrote:
>>
>> > Looks like there's two that weren't updated:
>> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
>> > --include=pom.xml
>> > ./hadoop-project/pom.xml:
>> > <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
>> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>> >
>> > I've just pushed in an addendum commit to fix those.
>> > In the future, please make sure to do a sanity compile when updating
>> poms.
>> >
>> > thanks
>> > - Robert
>> >
>> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri
>> <fa...@cloudera.com.invalid>
>> > wrote:
>> >
>> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
>> >> top-level pom.xml?
>> >>
>> >>
>> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
>> >>
>> >> > Hi All
>> >> >
>> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset trunk
>> to
>> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary
>> >> > patches are pulled into branch-3.2.
>> >> >
>> >> > Thank You
>> >> > - Sunil
>> >> >
>> >> >
>> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
>> >> >
>> >> > > Hi All
>> >> > >
>> >> > > We are now down to the last Blocker and HADOOP-15407 is merged to
>> >> trunk.
>> >> > > Thanks for the support.
>> >> > >
>> >> > > *Plan for RC*
>> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>> >> > > RC0 for 3.2: *28th Friday*
>> >> > >
>> >> > > Thank You
>> >> > > Sunil
>> >> > >
>> >> > >
>> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
>> >> > >
>> >> > >> Hi All
>> >> > >>
>> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you
>> for
>> >> > >> helping in this. I am following up on these tickets, once its
>> closed
>> >> we
>> >> > >> will cut the 3.2 branch.
>> >> > >>
>> >> > >> Thanks
>> >> > >> Sunil Govindan
>> >> > >>
>> >> > >>
>> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org>
>> wrote:
>> >> > >>
>> >> > >>> Hi All,
>> >> > >>>
>> >> > >>> Inline with the original 3.2 communication proposal dated 17th
>> July
>> >> > >>> 2018, I would like to provide more updates.
>> >> > >>>
>> >> > >>> We are approaching previously proposed code freeze date
>> (September
>> >> 14,
>> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
>> >> > existing
>> >> > >>> trunk to 3.3 if there are no issues.
>> >> > >>>
>> >> > >>> *Current Release Plan:*
>> >> > >>> Feature freeze date : all features to merge by September 7, 2018.
>> >> > >>> Code freeze date : blockers/critical only, no improvements and
>> >> > >>> blocker/critical bug-fixes September 14, 2018.
>> >> > >>> Release date: September 28, 2018
>> >> > >>>
>> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0, we
>> >> need to
>> >> > >>> backport to 3.2 post branch cut.
>> >> > >>>
>> >> > >>> Here's an updated 3.2.0 feature status:
>> >> > >>>
>> >> > >>> 1. Merged & Completed features:
>> >> > >>>
>> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> >> > >>> workloads Initial cut.
>> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> >> > >>> Scheduler.
>> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> Service
>> >> > API
>> >> > >>> and CLI.
>> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement
>> >> works.
>> >> > >>>
>> >> > >>> 2. Features close to finish:
>> >> > >>>
>> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
>> >> > >>> - (Steve) S3a phase V. Close to commit.
>> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>> >> > >>>
>> >> > >>> 3. Tentative/Cancelled features for 3.2:
>> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>> from
>> >> > >>> ATSv2. Patch in progress.
>> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >> challenging to
>> >> > >>> be done before Aug 2018.
>> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> >> > >>> Challenging as more discussions are on-going.
>> >> > >>>
>> >> > >>> *Summary of 3.2.0 issues status:*
>> >> > >>> 19 Blocker and Critical issues [1] are open, I am following up
>> with
>> >> > >>> owners to get status on each of them to get in by Code Freeze
>> date.
>> >> > >>>
>> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> >> > (Blocker,
>> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" =
>> 3.2.0
>> >> > ORDER
>> >> > >>> BY priority DESC
>> >> > >>>
>> >> > >>> Thanks,
>> >> > >>> Sunil
>> >> > >>>
>> >> > >>>
>> >> > >>>
>> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org>
>> wrote:
>> >> > >>>
>> >> > >>>> Hi All,
>> >> > >>>>
>> >> > >>>> Inline with earlier communication dated 17th July 2018, I would
>> >> like
>> >> > to
>> >> > >>>> provide some updates.
>> >> > >>>>
>> >> > >>>> We are approaching previously proposed code freeze date (Aug
>> 31).
>> >> > >>>>
>> >> > >>>> One of the critical feature Node Attributes feature merge
>> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need a
>> bit
>> >> > more
>> >> > >>>> time. With regard to this, suggesting to push the feature/code
>> >> freeze
>> >> > for 2
>> >> > >>>> more weeks to accommodate these jiras too.
>> >> > >>>>
>> >> > >>>> Proposing Updated changes in plan inline with this:
>> >> > >>>> Feature freeze date : all features to merge by September 7,
>> 2018.
>> >> > >>>> Code freeze date : blockers/critical only, no improvements and
>> >> > >>>> blocker/critical bug-fixes September 14, 2018.
>> >> > >>>> Release date: September 28, 2018
>> >> > >>>>
>> >> > >>>> If any features in branch which are targeted to 3.2.0, please
>> >> reply to
>> >> > >>>> this email thread.
>> >> > >>>>
>> >> > >>>> *Here's an updated 3.2.0 feature status:*
>> >> > >>>>
>> >> > >>>> 1. Merged & Completed features:
>> >> > >>>>
>> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> >> > >>>> workloads Initial cut.
>> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> >> > >>>> Scheduler.
>> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> >> Service
>> >> > >>>> API and CLI.
>> >> > >>>>
>> >> > >>>> 2. Features close to finish:
>> >> > >>>>
>> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> >> Merge/Vote
>> >> > >>>> Ongoing.
>> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>> from
>> >> > >>>> ATSv2. Patch in progress.
>> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> Improvement
>> >> > works.
>> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>> >> > >>>> Storage. In progress.
>> >> > >>>>
>> >> > >>>> 3. Tentative features:
>> >> > >>>>
>> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >> challenging
>> >> > to
>> >> > >>>> be done before Aug 2018.
>> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> >> > >>>> Challenging as more discussions are on-going.
>> >> > >>>>
>> >> > >>>> *Summary of 3.2.0 issues status:*
>> >> > >>>>
>> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following up
>> with
>> >> > >>>> owners to get status on each of them to get in by Code Freeze
>> date.
>> >> > >>>>
>> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> >> > Version/s" =
>> >> > >>>> 3.2.0 ORDER BY priority DESC
>> >> > >>>>
>> >> > >>>> Thanks,
>> >> > >>>> Sunil
>> >> > >>>>
>> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org>
>> >> wrote:
>> >> > >>>>
>> >> > >>>>> Hi All,
>> >> > >>>>>
>> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
>> dated
>> >> > 17th
>> >> > >>>>> July 2018, I would like to provide some updates.
>> >> > >>>>>
>> >> > >>>>> We are approaching previously proposed feature freeze date (Aug
>> >> 21,
>> >> > >>>>> about 7 days from today).
>> >> > >>>>> If any features in branch which are targeted to 3.2.0, please
>> >> reply
>> >> > to
>> >> > >>>>> this email thread.
>> >> > >>>>> Steve has mentioned about the s3 features which will come
>> close to
>> >> > >>>>> Code Freeze Date (Aug 31st).
>> >> > >>>>>
>> >> > >>>>> *Here's an updated 3.2.0 feature status:*
>> >> > >>>>>
>> >> > >>>>> 1. Merged & Completed features:
>> >> > >>>>>
>> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> >> > >>>>> workloads Initial cut.
>> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> >> > >>>>>
>> >> > >>>>> 2. Features close to finish:
>> >> > >>>>>
>> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> Major
>> >> > >>>>> patches are all in, only one last
>> >> > >>>>> patch is in review state.
>> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> >> > >>>>> Scheduler. Close to commit.
>> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> >> Service
>> >> > >>>>> API and CLI. 2 patches are pending
>> >> > >>>>> which will be closed by Feature freeze date.
>> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>> >> from
>> >> > >>>>> ATSv2. Patch in progress.
>> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
>> Improvement
>> >> > >>>>> works.
>> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>> >> > >>>>> Storage. In progress.
>> >> > >>>>>
>> >> > >>>>> 3. Tentative features:
>> >> > >>>>>
>> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> >> challenging
>> >> > >>>>> to be done before Aug 2018.
>> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> >> > >>>>> Challenging as more discussions are on-going.
>> >> > >>>>>
>> >> > >>>>> *Summary of 3.2.0 issues status:*
>> >> > >>>>>
>> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking with
>> >> > owners
>> >> > >>>>> to get status on each of them to get in by Code Freeze date.
>> >> > >>>>>
>> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> >> > Version/s" =
>> >> > >>>>> 3.2.0 ORDER BY priority DESC
>> >> > >>>>>
>> >> > >>>>> Thanks,
>> >> > >>>>> Sunil
>> >> > >>>>>
>> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org>
>> >> wrote:
>> >> > >>>>>
>> >> > >>>>>> Thanks Subru for the thoughts.
>> >> > >>>>>> One of the main reason for a major release is to push out
>> >> critical
>> >> > >>>>>> features with a faster cadence to the users. If we are pulling
>> >> more
>> >> > and
>> >> > >>>>>> more different types of features to a minor release, that
>> branch
>> >> > will
>> >> > >>>>>> become more destabilized and it may be tough to say that
>> 3.1.2 is
>> >> > stable
>> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
>> >> features
>> >> > in
>> >> > >>>>>> subsequent minor release.
>> >> > >>>>>> For few companies, it makes sense to push out these new
>> features
>> >> > >>>>>> faster to make a reach to the users. Adding to the point to
>> the
>> >> > backporting
>> >> > >>>>>> issues, I agree that its a pain and we can workaround that
>> with
>> >> > some git
>> >> > >>>>>> scripts. If we can make such scripts available to committers,
>> >> > backport will
>> >> > >>>>>> be seem-less across branches and we can achieve the faster
>> >> release
>> >> > cadence
>> >> > >>>>>> also.
>> >> > >>>>>>
>> >> > >>>>>> Thoughts?
>> >> > >>>>>>
>> >> > >>>>>> - Sunil
>> >> > >>>>>>
>> >> > >>>>>>
>> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
>> subru@apache.org
>> >> >
>> >> > >>>>>> wrote:
>> >> > >>>>>>
>> >> > >>>>>>> Thanks Sunil for volunteering to lead the release effort. I
>> am
>> >> > >>>>>>> generally
>> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as
>> >> feel we
>> >> > >>>>>>> already
>> >> > >>>>>>> have too many branches to be maintained. I already see many
>> >> commits
>> >> > >>>>>>> are in
>> >> > >>>>>>> different branches with no apparent rationale, for e.g: 3.1
>> has
>> >> > >>>>>>> commits
>> >> > >>>>>>> which are absent in 3.0 etc.
>> >> > >>>>>>>
>> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
>> >> > production
>> >> > >>>>>>> setting so the cost of adding features should be minimal.
>> >> > >>>>>>>
>> >> > >>>>>>> Thoughts?
>> >> > >>>>>>>
>> >> > >>>>>>> -Subru
>> >> > >>>>>>>
>> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <sunilg@apache.org
>> >
>> >> > wrote:
>> >> > >>>>>>>
>> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>> >> > >>>>>>> >
>> >> > >>>>>>> > Yes, important changes and features are much needed, hence
>> we
>> >> > will
>> >> > >>>>>>> be
>> >> > >>>>>>> > keeping the door open for them as possible. Also
>> considering
>> >> few
>> >> > >>>>>>> more
>> >> > >>>>>>> > offline requests from other folks, I think extending the
>> >> > timeframe
>> >> > >>>>>>> by
>> >> > >>>>>>> > couple of weeks makes sense (including a second RC buffer)
>> and
>> >> > >>>>>>> this should
>> >> > >>>>>>> > ideally help us to ship this by September itself.
>> >> > >>>>>>> >
>> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as
>> >> well)
>> >> > >>>>>>> >
>> >> > >>>>>>> > - Feature freeze date : all features to merge by August 21,
>> >> 2018.
>> >> > >>>>>>> >
>> >> > >>>>>>> > - Code freeze date : blockers/critical only, no
>> improvements
>> >> and
>> >> > >>>>>>> non
>> >> > >>>>>>> > blocker/critical
>> >> > >>>>>>> >
>> >> > >>>>>>> > bug-fixes August 31, 2018.
>> >> > >>>>>>> >
>> >> > >>>>>>> > - Release date: September 15, 2018
>> >> > >>>>>>> >
>> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered
>> your
>> >> > >>>>>>> questions.
>> >> > >>>>>>> >
>> >> > >>>>>>> > Thanks
>> >> > >>>>>>> > Sunil
>> >> > >>>>>>> >
>> >> > >>>>>>> >
>> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>> >> wheeleast@gmail.com
>> >> > >
>> >> > >>>>>>> wrote:
>> >> > >>>>>>> >
>> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1
>> >> for
>> >> > >>>>>>> that.
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > To concerns from Steve,
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > It is a good idea to keep the door open to get important
>> >> > changes
>> >> > >>>>>>> /
>> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
>> >> proposed
>> >> > >>>>>>> release
>> >> > >>>>>>> > date
>> >> > >>>>>>> > > to make sure things can happen earlier instead of last
>> >> minute
>> >> > >>>>>>> and we all
>> >> > >>>>>>> > > know that releases are always get delayed :). I'm also
>> fine
>> >> if
>> >> > >>>>>>> we want
>> >> > >>>>>>> > get
>> >> > >>>>>>> > > another several weeks time.
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
>> before
>> >> > >>>>>>> thanksgiving.
>> >> > >>>>>>> > > Do you think is it good or too early / late?
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > Eric,
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135
>> >> can
>> >> > >>>>>>> get merged
>> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > Sunil,
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > Could u update
>> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>> >> > >>>>>>> > Roadmap
>> >> > >>>>>>> > > with proposed plan as well? We can fill feature list
>> first
>> >> > >>>>>>> before getting
>> >> > >>>>>>> > > consensus of time.
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > Thanks,
>> >> > >>>>>>> > > Wangda
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>> >> > >>>>>>> <fabbri@cloudera.com.invalid
>> >> > >>>>>>> > >
>> >> > >>>>>>> > > wrote:
>> >> > >>>>>>> > >
>> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>> >> > >>>>>>> stevel@hortonworks.com>
>> >> > >>>>>>> > >> wrote:
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org
>> >> > <mailto:
>> >> > >>>>>>> > >> > sunilg@apache.org>> wrote:
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > I would also would like to take this opportunity to
>> come
>> >> up
>> >> > >>>>>>> with a
>> >> > >>>>>>> > >> detailed
>> >> > >>>>>>> > >> > plan.
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > - Feature freeze date : all features should be merged
>> by
>> >> > >>>>>>> August 10,
>> >> > >>>>>>> > >> 2018.
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > <snip>
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > Please let me know if I missed any features targeted
>> to
>> >> 3.2
>> >> > >>>>>>> per this
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
>> Datalake
>> >> > Gen
>> >> > >>>>>>> 2
>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been doing
>> >> work
>> >> > >>>>>>> on it, but
>> >> > >>>>>>> > >> if
>> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
>> >> timely
>> >> > >>>>>>> reviews on
>> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will review
>> the
>> >> S3 &
>> >> > >>>>>>> S3Guard
>> >> > >>>>>>> > >> JIRAs
>> >> > >>>>>>> > >> > to see which of those are blocking. Then there are
>> some
>> >> > >>>>>>> pressing
>> >> > >>>>>>> > "guave,
>> >> > >>>>>>> > >> > java 9 prep"
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> I can help with this part if you like.
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as release
>> >> > manager
>> >> > >>>>>>> of 3.2.0
>> >> > >>>>>>> > >> > release.
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > well volunteered!
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> Yes, thank you for stepping up.
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > I think this raises a good q: what timetable should we
>> >> have
>> >> > >>>>>>> for the
>> >> > >>>>>>> > >> 3.2. &
>> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
>> having
>> >> > the
>> >> > >>>>>>> outline
>> >> > >>>>>>> > >> time
>> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's
>> less
>> >> > >>>>>>> concern about
>> >> > >>>>>>> > >> > things not making the 3.2 dealine
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> > -Steve
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> >
>> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >> -AF
>> >> > >>>>>>> > >>
>> >> > >>>>>>> > >
>> >> > >>>>>>> >
>> >> > >>>>>>>
>> >> > >>>>>>
>> >> >
>> >>
>> >
>>
>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Jonathan Bender <jo...@stripe.com.INVALID>.
Hello, were there any updates around the 3.2.0 RC timing? All I see in the
current blockers are related to the new Submarine subproject, wasn't sure
if that is what is holding things up.
Cheers,
Jon
On Tue, Oct 2, 2018 at 7:13 PM, Sunil G <su...@apache.org> wrote:
> Thanks Robert and Haibo for quickly correcting same.
> Sigh, I somehow missed one file while committing the change. Sorry for the
> trouble.
>
> - Sunil
>
> On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rk...@cloudera.com> wrote:
>
> > Looks like there's two that weren't updated:
> > >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
> > --include=pom.xml
> > ./hadoop-project/pom.xml:
> > <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
> > ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
> >
> > I've just pushed in an addendum commit to fix those.
> > In the future, please make sure to do a sanity compile when updating
> poms.
> >
> > thanks
> > - Robert
> >
> > On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri <fabbri@cloudera.com.invalid
> >
> > wrote:
> >
> >> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
> >> top-level pom.xml?
> >>
> >>
> >> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
> >>
> >> > Hi All
> >> >
> >> > As mentioned in earlier mail, I have cut branch-3.2 and reset trunk to
> >> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary
> >> > patches are pulled into branch-3.2.
> >> >
> >> > Thank You
> >> > - Sunil
> >> >
> >> >
> >> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
> >> >
> >> > > Hi All
> >> > >
> >> > > We are now down to the last Blocker and HADOOP-15407 is merged to
> >> trunk.
> >> > > Thanks for the support.
> >> > >
> >> > > *Plan for RC*
> >> > > 3.2 branch cut and reset trunk : *25th Tuesday*
> >> > > RC0 for 3.2: *28th Friday*
> >> > >
> >> > > Thank You
> >> > > Sunil
> >> > >
> >> > >
> >> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
> >> > >
> >> > >> Hi All
> >> > >>
> >> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you for
> >> > >> helping in this. I am following up on these tickets, once its
> closed
> >> we
> >> > >> will cut the 3.2 branch.
> >> > >>
> >> > >> Thanks
> >> > >> Sunil Govindan
> >> > >>
> >> > >>
> >> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
> >> > >>
> >> > >>> Hi All,
> >> > >>>
> >> > >>> Inline with the original 3.2 communication proposal dated 17th
> July
> >> > >>> 2018, I would like to provide more updates.
> >> > >>>
> >> > >>> We are approaching previously proposed code freeze date (September
> >> 14,
> >> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
> >> > existing
> >> > >>> trunk to 3.3 if there are no issues.
> >> > >>>
> >> > >>> *Current Release Plan:*
> >> > >>> Feature freeze date : all features to merge by September 7, 2018.
> >> > >>> Code freeze date : blockers/critical only, no improvements and
> >> > >>> blocker/critical bug-fixes September 14, 2018.
> >> > >>> Release date: September 28, 2018
> >> > >>>
> >> > >>> If any critical/blocker tickets which are targeted to 3.2.0, we
> >> need to
> >> > >>> backport to 3.2 post branch cut.
> >> > >>>
> >> > >>> Here's an updated 3.2.0 feature status:
> >> > >>>
> >> > >>> 1. Merged & Completed features:
> >> > >>>
> >> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> >> > >>> workloads Initial cut.
> >> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >> > >>> Scheduler.
> >> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> Service
> >> > API
> >> > >>> and CLI.
> >> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement
> >> works.
> >> > >>>
> >> > >>> 2. Features close to finish:
> >> > >>>
> >> > >>> - (Steve) S3Guard Phase III. Close to commit.
> >> > >>> - (Steve) S3a phase V. Close to commit.
> >> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
> >> > >>>
> >> > >>> 3. Tentative/Cancelled features for 3.2:
> >> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
> from
> >> > >>> ATSv2. Patch in progress.
> >> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >> challenging to
> >> > >>> be done before Aug 2018.
> >> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >> > >>> Challenging as more discussions are on-going.
> >> > >>>
> >> > >>> *Summary of 3.2.0 issues status:*
> >> > >>> 19 Blocker and Critical issues [1] are open, I am following up
> with
> >> > >>> owners to get status on each of them to get in by Code Freeze
> date.
> >> > >>>
> >> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> >> > (Blocker,
> >> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" =
> 3.2.0
> >> > ORDER
> >> > >>> BY priority DESC
> >> > >>>
> >> > >>> Thanks,
> >> > >>> Sunil
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org>
> wrote:
> >> > >>>
> >> > >>>> Hi All,
> >> > >>>>
> >> > >>>> Inline with earlier communication dated 17th July 2018, I would
> >> like
> >> > to
> >> > >>>> provide some updates.
> >> > >>>>
> >> > >>>> We are approaching previously proposed code freeze date (Aug 31).
> >> > >>>>
> >> > >>>> One of the critical feature Node Attributes feature merge
> >> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need a
> bit
> >> > more
> >> > >>>> time. With regard to this, suggesting to push the feature/code
> >> freeze
> >> > for 2
> >> > >>>> more weeks to accommodate these jiras too.
> >> > >>>>
> >> > >>>> Proposing Updated changes in plan inline with this:
> >> > >>>> Feature freeze date : all features to merge by September 7, 2018.
> >> > >>>> Code freeze date : blockers/critical only, no improvements and
> >> > >>>> blocker/critical bug-fixes September 14, 2018.
> >> > >>>> Release date: September 28, 2018
> >> > >>>>
> >> > >>>> If any features in branch which are targeted to 3.2.0, please
> >> reply to
> >> > >>>> this email thread.
> >> > >>>>
> >> > >>>> *Here's an updated 3.2.0 feature status:*
> >> > >>>>
> >> > >>>> 1. Merged & Completed features:
> >> > >>>>
> >> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> >> > >>>> workloads Initial cut.
> >> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >> > >>>> Scheduler.
> >> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >> Service
> >> > >>>> API and CLI.
> >> > >>>>
> >> > >>>> 2. Features close to finish:
> >> > >>>>
> >> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >> Merge/Vote
> >> > >>>> Ongoing.
> >> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
> from
> >> > >>>> ATSv2. Patch in progress.
> >> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
> >> > works.
> >> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> >> > >>>> Storage. In progress.
> >> > >>>>
> >> > >>>> 3. Tentative features:
> >> > >>>>
> >> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >> challenging
> >> > to
> >> > >>>> be done before Aug 2018.
> >> > >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >> > >>>> Challenging as more discussions are on-going.
> >> > >>>>
> >> > >>>> *Summary of 3.2.0 issues status:*
> >> > >>>>
> >> > >>>> 26 Blocker and Critical issues [1] are open, I am following up
> with
> >> > >>>> owners to get status on each of them to get in by Code Freeze
> date.
> >> > >>>>
> >> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> >> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> >> > Version/s" =
> >> > >>>> 3.2.0 ORDER BY priority DESC
> >> > >>>>
> >> > >>>> Thanks,
> >> > >>>> Sunil
> >> > >>>>
> >> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org>
> >> wrote:
> >> > >>>>
> >> > >>>>> Hi All,
> >> > >>>>>
> >> > >>>>> Thanks for the feedbacks. Inline with earlier communication
> dated
> >> > 17th
> >> > >>>>> July 2018, I would like to provide some updates.
> >> > >>>>>
> >> > >>>>> We are approaching previously proposed feature freeze date (Aug
> >> 21,
> >> > >>>>> about 7 days from today).
> >> > >>>>> If any features in branch which are targeted to 3.2.0, please
> >> reply
> >> > to
> >> > >>>>> this email thread.
> >> > >>>>> Steve has mentioned about the s3 features which will come close
> to
> >> > >>>>> Code Freeze Date (Aug 31st).
> >> > >>>>>
> >> > >>>>> *Here's an updated 3.2.0 feature status:*
> >> > >>>>>
> >> > >>>>> 1. Merged & Completed features:
> >> > >>>>>
> >> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> >> > >>>>> workloads Initial cut.
> >> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >> > >>>>>
> >> > >>>>> 2. Features close to finish:
> >> > >>>>>
> >> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
> >> > >>>>> patches are all in, only one last
> >> > >>>>> patch is in review state.
> >> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >> > >>>>> Scheduler. Close to commit.
> >> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> >> Service
> >> > >>>>> API and CLI. 2 patches are pending
> >> > >>>>> which will be closed by Feature freeze date.
> >> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
> >> from
> >> > >>>>> ATSv2. Patch in progress.
> >> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation.
> Improvement
> >> > >>>>> works.
> >> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> >> > >>>>> Storage. In progress.
> >> > >>>>>
> >> > >>>>> 3. Tentative features:
> >> > >>>>>
> >> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> >> challenging
> >> > >>>>> to be done before Aug 2018.
> >> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >> > >>>>> Challenging as more discussions are on-going.
> >> > >>>>>
> >> > >>>>> *Summary of 3.2.0 issues status:*
> >> > >>>>>
> >> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking with
> >> > owners
> >> > >>>>> to get status on each of them to get in by Code Freeze date.
> >> > >>>>>
> >> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> >> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> >> > Version/s" =
> >> > >>>>> 3.2.0 ORDER BY priority DESC
> >> > >>>>>
> >> > >>>>> Thanks,
> >> > >>>>> Sunil
> >> > >>>>>
> >> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org>
> >> wrote:
> >> > >>>>>
> >> > >>>>>> Thanks Subru for the thoughts.
> >> > >>>>>> One of the main reason for a major release is to push out
> >> critical
> >> > >>>>>> features with a faster cadence to the users. If we are pulling
> >> more
> >> > and
> >> > >>>>>> more different types of features to a minor release, that
> branch
> >> > will
> >> > >>>>>> become more destabilized and it may be tough to say that 3.1.2
> is
> >> > stable
> >> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
> >> features
> >> > in
> >> > >>>>>> subsequent minor release.
> >> > >>>>>> For few companies, it makes sense to push out these new
> features
> >> > >>>>>> faster to make a reach to the users. Adding to the point to the
> >> > backporting
> >> > >>>>>> issues, I agree that its a pain and we can workaround that with
> >> > some git
> >> > >>>>>> scripts. If we can make such scripts available to committers,
> >> > backport will
> >> > >>>>>> be seem-less across branches and we can achieve the faster
> >> release
> >> > cadence
> >> > >>>>>> also.
> >> > >>>>>>
> >> > >>>>>> Thoughts?
> >> > >>>>>>
> >> > >>>>>> - Sunil
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <
> subru@apache.org
> >> >
> >> > >>>>>> wrote:
> >> > >>>>>>
> >> > >>>>>>> Thanks Sunil for volunteering to lead the release effort. I am
> >> > >>>>>>> generally
> >> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as
> >> feel we
> >> > >>>>>>> already
> >> > >>>>>>> have too many branches to be maintained. I already see many
> >> commits
> >> > >>>>>>> are in
> >> > >>>>>>> different branches with no apparent rationale, for e.g: 3.1
> has
> >> > >>>>>>> commits
> >> > >>>>>>> which are absent in 3.0 etc.
> >> > >>>>>>>
> >> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
> >> > production
> >> > >>>>>>> setting so the cost of adding features should be minimal.
> >> > >>>>>>>
> >> > >>>>>>> Thoughts?
> >> > >>>>>>>
> >> > >>>>>>> -Subru
> >> > >>>>>>>
> >> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org>
> >> > wrote:
> >> > >>>>>>>
> >> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> >> > >>>>>>> >
> >> > >>>>>>> > Yes, important changes and features are much needed, hence
> we
> >> > will
> >> > >>>>>>> be
> >> > >>>>>>> > keeping the door open for them as possible. Also considering
> >> few
> >> > >>>>>>> more
> >> > >>>>>>> > offline requests from other folks, I think extending the
> >> > timeframe
> >> > >>>>>>> by
> >> > >>>>>>> > couple of weeks makes sense (including a second RC buffer)
> and
> >> > >>>>>>> this should
> >> > >>>>>>> > ideally help us to ship this by September itself.
> >> > >>>>>>> >
> >> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as
> >> well)
> >> > >>>>>>> >
> >> > >>>>>>> > - Feature freeze date : all features to merge by August 21,
> >> 2018.
> >> > >>>>>>> >
> >> > >>>>>>> > - Code freeze date : blockers/critical only, no improvements
> >> and
> >> > >>>>>>> non
> >> > >>>>>>> > blocker/critical
> >> > >>>>>>> >
> >> > >>>>>>> > bug-fixes August 31, 2018.
> >> > >>>>>>> >
> >> > >>>>>>> > - Release date: September 15, 2018
> >> > >>>>>>> >
> >> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered
> your
> >> > >>>>>>> questions.
> >> > >>>>>>> >
> >> > >>>>>>> > Thanks
> >> > >>>>>>> > Sunil
> >> > >>>>>>> >
> >> > >>>>>>> >
> >> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
> >> wheeleast@gmail.com
> >> > >
> >> > >>>>>>> wrote:
> >> > >>>>>>> >
> >> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1
> >> for
> >> > >>>>>>> that.
> >> > >>>>>>> > >
> >> > >>>>>>> > > To concerns from Steve,
> >> > >>>>>>> > >
> >> > >>>>>>> > > It is a good idea to keep the door open to get important
> >> > changes
> >> > >>>>>>> /
> >> > >>>>>>> > > features in before cutoff. I would prefer to keep the
> >> proposed
> >> > >>>>>>> release
> >> > >>>>>>> > date
> >> > >>>>>>> > > to make sure things can happen earlier instead of last
> >> minute
> >> > >>>>>>> and we all
> >> > >>>>>>> > > know that releases are always get delayed :). I'm also
> fine
> >> if
> >> > >>>>>>> we want
> >> > >>>>>>> > get
> >> > >>>>>>> > > another several weeks time.
> >> > >>>>>>> > >
> >> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that
> before
> >> > >>>>>>> thanksgiving.
> >> > >>>>>>> > > Do you think is it good or too early / late?
> >> > >>>>>>> > >
> >> > >>>>>>> > > Eric,
> >> > >>>>>>> > >
> >> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135
> >> can
> >> > >>>>>>> get merged
> >> > >>>>>>> > > in time, we probably not need the YARN-8220.
> >> > >>>>>>> > >
> >> > >>>>>>> > > Sunil,
> >> > >>>>>>> > >
> >> > >>>>>>> > > Could u update
> >> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
> >> > >>>>>>> > Roadmap
> >> > >>>>>>> > > with proposed plan as well? We can fill feature list first
> >> > >>>>>>> before getting
> >> > >>>>>>> > > consensus of time.
> >> > >>>>>>> > >
> >> > >>>>>>> > > Thanks,
> >> > >>>>>>> > > Wangda
> >> > >>>>>>> > >
> >> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> >> > >>>>>>> <fabbri@cloudera.com.invalid
> >> > >>>>>>> > >
> >> > >>>>>>> > > wrote:
> >> > >>>>>>> > >
> >> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> >> > >>>>>>> stevel@hortonworks.com>
> >> > >>>>>>> > >> wrote:
> >> > >>>>>>> > >>
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org
> >> > <mailto:
> >> > >>>>>>> > >> > sunilg@apache.org>> wrote:
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> > I would also would like to take this opportunity to
> come
> >> up
> >> > >>>>>>> with a
> >> > >>>>>>> > >> detailed
> >> > >>>>>>> > >> > plan.
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> > - Feature freeze date : all features should be merged
> by
> >> > >>>>>>> August 10,
> >> > >>>>>>> > >> 2018.
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> > <snip>
> >> > >>>>>>> > >>
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> > Please let me know if I missed any features targeted to
> >> 3.2
> >> > >>>>>>> per this
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure
> Datalake
> >> > Gen
> >> > >>>>>>> 2
> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> > I don't think this is quite ready yet, I've been doing
> >> work
> >> > >>>>>>> on it, but
> >> > >>>>>>> > >> if
> >> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
> >> timely
> >> > >>>>>>> reviews on
> >> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> > I've uprated that to a blocker feature; will review the
> >> S3 &
> >> > >>>>>>> S3Guard
> >> > >>>>>>> > >> JIRAs
> >> > >>>>>>> > >> > to see which of those are blocking. Then there are some
> >> > >>>>>>> pressing
> >> > >>>>>>> > "guave,
> >> > >>>>>>> > >> > java 9 prep"
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> I can help with this part if you like.
> >> > >>>>>>> > >>
> >> > >>>>>>> > >>
> >> > >>>>>>> > >>
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> > timeline. I would like to volunteer myself as release
> >> > manager
> >> > >>>>>>> of 3.2.0
> >> > >>>>>>> > >> > release.
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> > well volunteered!
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> Yes, thank you for stepping up.
> >> > >>>>>>> > >>
> >> > >>>>>>> > >>
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> > I think this raises a good q: what timetable should we
> >> have
> >> > >>>>>>> for the
> >> > >>>>>>> > >> 3.2. &
> >> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then
> having
> >> > the
> >> > >>>>>>> outline
> >> > >>>>>>> > >> time
> >> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's less
> >> > >>>>>>> concern about
> >> > >>>>>>> > >> > things not making the 3.2 dealine
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> > -Steve
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> >
> >> > >>>>>>> > >> Good idea to mitigate the short deadline.
> >> > >>>>>>> > >>
> >> > >>>>>>> > >> -AF
> >> > >>>>>>> > >>
> >> > >>>>>>> > >
> >> > >>>>>>> >
> >> > >>>>>>>
> >> > >>>>>>
> >> >
> >>
> >
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Thanks Robert and Haibo for quickly correcting same.
Sigh, I somehow missed one file while committing the change. Sorry for the
trouble.
- Sunil
On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rk...@cloudera.com> wrote:
> Looks like there's two that weren't updated:
> >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
> --include=pom.xml
> ./hadoop-project/pom.xml:
> <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
> ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>
> I've just pushed in an addendum commit to fix those.
> In the future, please make sure to do a sanity compile when updating poms.
>
> thanks
> - Robert
>
> On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri <fa...@cloudera.com.invalid>
> wrote:
>
>> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
>> top-level pom.xml?
>>
>>
>> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
>>
>> > Hi All
>> >
>> > As mentioned in earlier mail, I have cut branch-3.2 and reset trunk to
>> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary
>> > patches are pulled into branch-3.2.
>> >
>> > Thank You
>> > - Sunil
>> >
>> >
>> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
>> >
>> > > Hi All
>> > >
>> > > We are now down to the last Blocker and HADOOP-15407 is merged to
>> trunk.
>> > > Thanks for the support.
>> > >
>> > > *Plan for RC*
>> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>> > > RC0 for 3.2: *28th Friday*
>> > >
>> > > Thank You
>> > > Sunil
>> > >
>> > >
>> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
>> > >
>> > >> Hi All
>> > >>
>> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you for
>> > >> helping in this. I am following up on these tickets, once its closed
>> we
>> > >> will cut the 3.2 branch.
>> > >>
>> > >> Thanks
>> > >> Sunil Govindan
>> > >>
>> > >>
>> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
>> > >>
>> > >>> Hi All,
>> > >>>
>> > >>> Inline with the original 3.2 communication proposal dated 17th July
>> > >>> 2018, I would like to provide more updates.
>> > >>>
>> > >>> We are approaching previously proposed code freeze date (September
>> 14,
>> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
>> > existing
>> > >>> trunk to 3.3 if there are no issues.
>> > >>>
>> > >>> *Current Release Plan:*
>> > >>> Feature freeze date : all features to merge by September 7, 2018.
>> > >>> Code freeze date : blockers/critical only, no improvements and
>> > >>> blocker/critical bug-fixes September 14, 2018.
>> > >>> Release date: September 28, 2018
>> > >>>
>> > >>> If any critical/blocker tickets which are targeted to 3.2.0, we
>> need to
>> > >>> backport to 3.2 post branch cut.
>> > >>>
>> > >>> Here's an updated 3.2.0 feature status:
>> > >>>
>> > >>> 1. Merged & Completed features:
>> > >>>
>> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> > >>> workloads Initial cut.
>> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> > >>> Scheduler.
>> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
>> > API
>> > >>> and CLI.
>> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement
>> works.
>> > >>>
>> > >>> 2. Features close to finish:
>> > >>>
>> > >>> - (Steve) S3Guard Phase III. Close to commit.
>> > >>> - (Steve) S3a phase V. Close to commit.
>> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>> > >>>
>> > >>> 3. Tentative/Cancelled features for 3.2:
>> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>> > >>> ATSv2. Patch in progress.
>> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> challenging to
>> > >>> be done before Aug 2018.
>> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> > >>> Challenging as more discussions are on-going.
>> > >>>
>> > >>> *Summary of 3.2.0 issues status:*
>> > >>> 19 Blocker and Critical issues [1] are open, I am following up with
>> > >>> owners to get status on each of them to get in by Code Freeze date.
>> > >>>
>> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> > (Blocker,
>> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
>> > ORDER
>> > >>> BY priority DESC
>> > >>>
>> > >>> Thanks,
>> > >>> Sunil
>> > >>>
>> > >>>
>> > >>>
>> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
>> > >>>
>> > >>>> Hi All,
>> > >>>>
>> > >>>> Inline with earlier communication dated 17th July 2018, I would
>> like
>> > to
>> > >>>> provide some updates.
>> > >>>>
>> > >>>> We are approaching previously proposed code freeze date (Aug 31).
>> > >>>>
>> > >>>> One of the critical feature Node Attributes feature merge
>> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need a bit
>> > more
>> > >>>> time. With regard to this, suggesting to push the feature/code
>> freeze
>> > for 2
>> > >>>> more weeks to accommodate these jiras too.
>> > >>>>
>> > >>>> Proposing Updated changes in plan inline with this:
>> > >>>> Feature freeze date : all features to merge by September 7, 2018.
>> > >>>> Code freeze date : blockers/critical only, no improvements and
>> > >>>> blocker/critical bug-fixes September 14, 2018.
>> > >>>> Release date: September 28, 2018
>> > >>>>
>> > >>>> If any features in branch which are targeted to 3.2.0, please
>> reply to
>> > >>>> this email thread.
>> > >>>>
>> > >>>> *Here's an updated 3.2.0 feature status:*
>> > >>>>
>> > >>>> 1. Merged & Completed features:
>> > >>>>
>> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> > >>>> workloads Initial cut.
>> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> > >>>> Scheduler.
>> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> Service
>> > >>>> API and CLI.
>> > >>>>
>> > >>>> 2. Features close to finish:
>> > >>>>
>> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> Merge/Vote
>> > >>>> Ongoing.
>> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>> > >>>> ATSv2. Patch in progress.
>> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
>> > works.
>> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>> > >>>> Storage. In progress.
>> > >>>>
>> > >>>> 3. Tentative features:
>> > >>>>
>> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> challenging
>> > to
>> > >>>> be done before Aug 2018.
>> > >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> > >>>> Challenging as more discussions are on-going.
>> > >>>>
>> > >>>> *Summary of 3.2.0 issues status:*
>> > >>>>
>> > >>>> 26 Blocker and Critical issues [1] are open, I am following up with
>> > >>>> owners to get status on each of them to get in by Code Freeze date.
>> > >>>>
>> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> > Version/s" =
>> > >>>> 3.2.0 ORDER BY priority DESC
>> > >>>>
>> > >>>> Thanks,
>> > >>>> Sunil
>> > >>>>
>> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org>
>> wrote:
>> > >>>>
>> > >>>>> Hi All,
>> > >>>>>
>> > >>>>> Thanks for the feedbacks. Inline with earlier communication dated
>> > 17th
>> > >>>>> July 2018, I would like to provide some updates.
>> > >>>>>
>> > >>>>> We are approaching previously proposed feature freeze date (Aug
>> 21,
>> > >>>>> about 7 days from today).
>> > >>>>> If any features in branch which are targeted to 3.2.0, please
>> reply
>> > to
>> > >>>>> this email thread.
>> > >>>>> Steve has mentioned about the s3 features which will come close to
>> > >>>>> Code Freeze Date (Aug 31st).
>> > >>>>>
>> > >>>>> *Here's an updated 3.2.0 feature status:*
>> > >>>>>
>> > >>>>> 1. Merged & Completed features:
>> > >>>>>
>> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> > >>>>> workloads Initial cut.
>> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> > >>>>>
>> > >>>>> 2. Features close to finish:
>> > >>>>>
>> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
>> > >>>>> patches are all in, only one last
>> > >>>>> patch is in review state.
>> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> > >>>>> Scheduler. Close to commit.
>> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> Service
>> > >>>>> API and CLI. 2 patches are pending
>> > >>>>> which will be closed by Feature freeze date.
>> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>> from
>> > >>>>> ATSv2. Patch in progress.
>> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
>> > >>>>> works.
>> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>> > >>>>> Storage. In progress.
>> > >>>>>
>> > >>>>> 3. Tentative features:
>> > >>>>>
>> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> challenging
>> > >>>>> to be done before Aug 2018.
>> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> > >>>>> Challenging as more discussions are on-going.
>> > >>>>>
>> > >>>>> *Summary of 3.2.0 issues status:*
>> > >>>>>
>> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking with
>> > owners
>> > >>>>> to get status on each of them to get in by Code Freeze date.
>> > >>>>>
>> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> > Version/s" =
>> > >>>>> 3.2.0 ORDER BY priority DESC
>> > >>>>>
>> > >>>>> Thanks,
>> > >>>>> Sunil
>> > >>>>>
>> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org>
>> wrote:
>> > >>>>>
>> > >>>>>> Thanks Subru for the thoughts.
>> > >>>>>> One of the main reason for a major release is to push out
>> critical
>> > >>>>>> features with a faster cadence to the users. If we are pulling
>> more
>> > and
>> > >>>>>> more different types of features to a minor release, that branch
>> > will
>> > >>>>>> become more destabilized and it may be tough to say that 3.1.2 is
>> > stable
>> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
>> features
>> > in
>> > >>>>>> subsequent minor release.
>> > >>>>>> For few companies, it makes sense to push out these new features
>> > >>>>>> faster to make a reach to the users. Adding to the point to the
>> > backporting
>> > >>>>>> issues, I agree that its a pain and we can workaround that with
>> > some git
>> > >>>>>> scripts. If we can make such scripts available to committers,
>> > backport will
>> > >>>>>> be seem-less across branches and we can achieve the faster
>> release
>> > cadence
>> > >>>>>> also.
>> > >>>>>>
>> > >>>>>> Thoughts?
>> > >>>>>>
>> > >>>>>> - Sunil
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <subru@apache.org
>> >
>> > >>>>>> wrote:
>> > >>>>>>
>> > >>>>>>> Thanks Sunil for volunteering to lead the release effort. I am
>> > >>>>>>> generally
>> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as
>> feel we
>> > >>>>>>> already
>> > >>>>>>> have too many branches to be maintained. I already see many
>> commits
>> > >>>>>>> are in
>> > >>>>>>> different branches with no apparent rationale, for e.g: 3.1 has
>> > >>>>>>> commits
>> > >>>>>>> which are absent in 3.0 etc.
>> > >>>>>>>
>> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
>> > production
>> > >>>>>>> setting so the cost of adding features should be minimal.
>> > >>>>>>>
>> > >>>>>>> Thoughts?
>> > >>>>>>>
>> > >>>>>>> -Subru
>> > >>>>>>>
>> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org>
>> > wrote:
>> > >>>>>>>
>> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>> > >>>>>>> >
>> > >>>>>>> > Yes, important changes and features are much needed, hence we
>> > will
>> > >>>>>>> be
>> > >>>>>>> > keeping the door open for them as possible. Also considering
>> few
>> > >>>>>>> more
>> > >>>>>>> > offline requests from other folks, I think extending the
>> > timeframe
>> > >>>>>>> by
>> > >>>>>>> > couple of weeks makes sense (including a second RC buffer) and
>> > >>>>>>> this should
>> > >>>>>>> > ideally help us to ship this by September itself.
>> > >>>>>>> >
>> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as
>> well)
>> > >>>>>>> >
>> > >>>>>>> > - Feature freeze date : all features to merge by August 21,
>> 2018.
>> > >>>>>>> >
>> > >>>>>>> > - Code freeze date : blockers/critical only, no improvements
>> and
>> > >>>>>>> non
>> > >>>>>>> > blocker/critical
>> > >>>>>>> >
>> > >>>>>>> > bug-fixes August 31, 2018.
>> > >>>>>>> >
>> > >>>>>>> > - Release date: September 15, 2018
>> > >>>>>>> >
>> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered your
>> > >>>>>>> questions.
>> > >>>>>>> >
>> > >>>>>>> > Thanks
>> > >>>>>>> > Sunil
>> > >>>>>>> >
>> > >>>>>>> >
>> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>> wheeleast@gmail.com
>> > >
>> > >>>>>>> wrote:
>> > >>>>>>> >
>> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1
>> for
>> > >>>>>>> that.
>> > >>>>>>> > >
>> > >>>>>>> > > To concerns from Steve,
>> > >>>>>>> > >
>> > >>>>>>> > > It is a good idea to keep the door open to get important
>> > changes
>> > >>>>>>> /
>> > >>>>>>> > > features in before cutoff. I would prefer to keep the
>> proposed
>> > >>>>>>> release
>> > >>>>>>> > date
>> > >>>>>>> > > to make sure things can happen earlier instead of last
>> minute
>> > >>>>>>> and we all
>> > >>>>>>> > > know that releases are always get delayed :). I'm also fine
>> if
>> > >>>>>>> we want
>> > >>>>>>> > get
>> > >>>>>>> > > another several weeks time.
>> > >>>>>>> > >
>> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that before
>> > >>>>>>> thanksgiving.
>> > >>>>>>> > > Do you think is it good or too early / late?
>> > >>>>>>> > >
>> > >>>>>>> > > Eric,
>> > >>>>>>> > >
>> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135
>> can
>> > >>>>>>> get merged
>> > >>>>>>> > > in time, we probably not need the YARN-8220.
>> > >>>>>>> > >
>> > >>>>>>> > > Sunil,
>> > >>>>>>> > >
>> > >>>>>>> > > Could u update
>> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>> > >>>>>>> > Roadmap
>> > >>>>>>> > > with proposed plan as well? We can fill feature list first
>> > >>>>>>> before getting
>> > >>>>>>> > > consensus of time.
>> > >>>>>>> > >
>> > >>>>>>> > > Thanks,
>> > >>>>>>> > > Wangda
>> > >>>>>>> > >
>> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>> > >>>>>>> <fabbri@cloudera.com.invalid
>> > >>>>>>> > >
>> > >>>>>>> > > wrote:
>> > >>>>>>> > >
>> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>> > >>>>>>> stevel@hortonworks.com>
>> > >>>>>>> > >> wrote:
>> > >>>>>>> > >>
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org
>> > <mailto:
>> > >>>>>>> > >> > sunilg@apache.org>> wrote:
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > I would also would like to take this opportunity to come
>> up
>> > >>>>>>> with a
>> > >>>>>>> > >> detailed
>> > >>>>>>> > >> > plan.
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > - Feature freeze date : all features should be merged by
>> > >>>>>>> August 10,
>> > >>>>>>> > >> 2018.
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > <snip>
>> > >>>>>>> > >>
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > Please let me know if I missed any features targeted to
>> 3.2
>> > >>>>>>> per this
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake
>> > Gen
>> > >>>>>>> 2
>> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > I don't think this is quite ready yet, I've been doing
>> work
>> > >>>>>>> on it, but
>> > >>>>>>> > >> if
>> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
>> timely
>> > >>>>>>> reviews on
>> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > I've uprated that to a blocker feature; will review the
>> S3 &
>> > >>>>>>> S3Guard
>> > >>>>>>> > >> JIRAs
>> > >>>>>>> > >> > to see which of those are blocking. Then there are some
>> > >>>>>>> pressing
>> > >>>>>>> > "guave,
>> > >>>>>>> > >> > java 9 prep"
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> I can help with this part if you like.
>> > >>>>>>> > >>
>> > >>>>>>> > >>
>> > >>>>>>> > >>
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > timeline. I would like to volunteer myself as release
>> > manager
>> > >>>>>>> of 3.2.0
>> > >>>>>>> > >> > release.
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > well volunteered!
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> Yes, thank you for stepping up.
>> > >>>>>>> > >>
>> > >>>>>>> > >>
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > I think this raises a good q: what timetable should we
>> have
>> > >>>>>>> for the
>> > >>>>>>> > >> 3.2. &
>> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having
>> > the
>> > >>>>>>> outline
>> > >>>>>>> > >> time
>> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's less
>> > >>>>>>> concern about
>> > >>>>>>> > >> > things not making the 3.2 dealine
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > -Steve
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> Good idea to mitigate the short deadline.
>> > >>>>>>> > >>
>> > >>>>>>> > >> -AF
>> > >>>>>>> > >>
>> > >>>>>>> > >
>> > >>>>>>> >
>> > >>>>>>>
>> > >>>>>>
>> >
>>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Thanks Robert and Haibo for quickly correcting same.
Sigh, I somehow missed one file while committing the change. Sorry for the
trouble.
- Sunil
On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rk...@cloudera.com> wrote:
> Looks like there's two that weren't updated:
> >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
> --include=pom.xml
> ./hadoop-project/pom.xml:
> <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
> ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>
> I've just pushed in an addendum commit to fix those.
> In the future, please make sure to do a sanity compile when updating poms.
>
> thanks
> - Robert
>
> On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri <fa...@cloudera.com.invalid>
> wrote:
>
>> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
>> top-level pom.xml?
>>
>>
>> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
>>
>> > Hi All
>> >
>> > As mentioned in earlier mail, I have cut branch-3.2 and reset trunk to
>> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary
>> > patches are pulled into branch-3.2.
>> >
>> > Thank You
>> > - Sunil
>> >
>> >
>> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
>> >
>> > > Hi All
>> > >
>> > > We are now down to the last Blocker and HADOOP-15407 is merged to
>> trunk.
>> > > Thanks for the support.
>> > >
>> > > *Plan for RC*
>> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>> > > RC0 for 3.2: *28th Friday*
>> > >
>> > > Thank You
>> > > Sunil
>> > >
>> > >
>> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
>> > >
>> > >> Hi All
>> > >>
>> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you for
>> > >> helping in this. I am following up on these tickets, once its closed
>> we
>> > >> will cut the 3.2 branch.
>> > >>
>> > >> Thanks
>> > >> Sunil Govindan
>> > >>
>> > >>
>> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
>> > >>
>> > >>> Hi All,
>> > >>>
>> > >>> Inline with the original 3.2 communication proposal dated 17th July
>> > >>> 2018, I would like to provide more updates.
>> > >>>
>> > >>> We are approaching previously proposed code freeze date (September
>> 14,
>> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
>> > existing
>> > >>> trunk to 3.3 if there are no issues.
>> > >>>
>> > >>> *Current Release Plan:*
>> > >>> Feature freeze date : all features to merge by September 7, 2018.
>> > >>> Code freeze date : blockers/critical only, no improvements and
>> > >>> blocker/critical bug-fixes September 14, 2018.
>> > >>> Release date: September 28, 2018
>> > >>>
>> > >>> If any critical/blocker tickets which are targeted to 3.2.0, we
>> need to
>> > >>> backport to 3.2 post branch cut.
>> > >>>
>> > >>> Here's an updated 3.2.0 feature status:
>> > >>>
>> > >>> 1. Merged & Completed features:
>> > >>>
>> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> > >>> workloads Initial cut.
>> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> > >>> Scheduler.
>> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
>> > API
>> > >>> and CLI.
>> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement
>> works.
>> > >>>
>> > >>> 2. Features close to finish:
>> > >>>
>> > >>> - (Steve) S3Guard Phase III. Close to commit.
>> > >>> - (Steve) S3a phase V. Close to commit.
>> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>> > >>>
>> > >>> 3. Tentative/Cancelled features for 3.2:
>> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>> > >>> ATSv2. Patch in progress.
>> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> challenging to
>> > >>> be done before Aug 2018.
>> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> > >>> Challenging as more discussions are on-going.
>> > >>>
>> > >>> *Summary of 3.2.0 issues status:*
>> > >>> 19 Blocker and Critical issues [1] are open, I am following up with
>> > >>> owners to get status on each of them to get in by Code Freeze date.
>> > >>>
>> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> > (Blocker,
>> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
>> > ORDER
>> > >>> BY priority DESC
>> > >>>
>> > >>> Thanks,
>> > >>> Sunil
>> > >>>
>> > >>>
>> > >>>
>> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
>> > >>>
>> > >>>> Hi All,
>> > >>>>
>> > >>>> Inline with earlier communication dated 17th July 2018, I would
>> like
>> > to
>> > >>>> provide some updates.
>> > >>>>
>> > >>>> We are approaching previously proposed code freeze date (Aug 31).
>> > >>>>
>> > >>>> One of the critical feature Node Attributes feature merge
>> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need a bit
>> > more
>> > >>>> time. With regard to this, suggesting to push the feature/code
>> freeze
>> > for 2
>> > >>>> more weeks to accommodate these jiras too.
>> > >>>>
>> > >>>> Proposing Updated changes in plan inline with this:
>> > >>>> Feature freeze date : all features to merge by September 7, 2018.
>> > >>>> Code freeze date : blockers/critical only, no improvements and
>> > >>>> blocker/critical bug-fixes September 14, 2018.
>> > >>>> Release date: September 28, 2018
>> > >>>>
>> > >>>> If any features in branch which are targeted to 3.2.0, please
>> reply to
>> > >>>> this email thread.
>> > >>>>
>> > >>>> *Here's an updated 3.2.0 feature status:*
>> > >>>>
>> > >>>> 1. Merged & Completed features:
>> > >>>>
>> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> > >>>> workloads Initial cut.
>> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> > >>>> Scheduler.
>> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> Service
>> > >>>> API and CLI.
>> > >>>>
>> > >>>> 2. Features close to finish:
>> > >>>>
>> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> Merge/Vote
>> > >>>> Ongoing.
>> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>> > >>>> ATSv2. Patch in progress.
>> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
>> > works.
>> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>> > >>>> Storage. In progress.
>> > >>>>
>> > >>>> 3. Tentative features:
>> > >>>>
>> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> challenging
>> > to
>> > >>>> be done before Aug 2018.
>> > >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> > >>>> Challenging as more discussions are on-going.
>> > >>>>
>> > >>>> *Summary of 3.2.0 issues status:*
>> > >>>>
>> > >>>> 26 Blocker and Critical issues [1] are open, I am following up with
>> > >>>> owners to get status on each of them to get in by Code Freeze date.
>> > >>>>
>> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> > Version/s" =
>> > >>>> 3.2.0 ORDER BY priority DESC
>> > >>>>
>> > >>>> Thanks,
>> > >>>> Sunil
>> > >>>>
>> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org>
>> wrote:
>> > >>>>
>> > >>>>> Hi All,
>> > >>>>>
>> > >>>>> Thanks for the feedbacks. Inline with earlier communication dated
>> > 17th
>> > >>>>> July 2018, I would like to provide some updates.
>> > >>>>>
>> > >>>>> We are approaching previously proposed feature freeze date (Aug
>> 21,
>> > >>>>> about 7 days from today).
>> > >>>>> If any features in branch which are targeted to 3.2.0, please
>> reply
>> > to
>> > >>>>> this email thread.
>> > >>>>> Steve has mentioned about the s3 features which will come close to
>> > >>>>> Code Freeze Date (Aug 31st).
>> > >>>>>
>> > >>>>> *Here's an updated 3.2.0 feature status:*
>> > >>>>>
>> > >>>>> 1. Merged & Completed features:
>> > >>>>>
>> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> > >>>>> workloads Initial cut.
>> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> > >>>>>
>> > >>>>> 2. Features close to finish:
>> > >>>>>
>> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
>> > >>>>> patches are all in, only one last
>> > >>>>> patch is in review state.
>> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> > >>>>> Scheduler. Close to commit.
>> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> Service
>> > >>>>> API and CLI. 2 patches are pending
>> > >>>>> which will be closed by Feature freeze date.
>> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>> from
>> > >>>>> ATSv2. Patch in progress.
>> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
>> > >>>>> works.
>> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>> > >>>>> Storage. In progress.
>> > >>>>>
>> > >>>>> 3. Tentative features:
>> > >>>>>
>> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> challenging
>> > >>>>> to be done before Aug 2018.
>> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> > >>>>> Challenging as more discussions are on-going.
>> > >>>>>
>> > >>>>> *Summary of 3.2.0 issues status:*
>> > >>>>>
>> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking with
>> > owners
>> > >>>>> to get status on each of them to get in by Code Freeze date.
>> > >>>>>
>> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> > Version/s" =
>> > >>>>> 3.2.0 ORDER BY priority DESC
>> > >>>>>
>> > >>>>> Thanks,
>> > >>>>> Sunil
>> > >>>>>
>> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org>
>> wrote:
>> > >>>>>
>> > >>>>>> Thanks Subru for the thoughts.
>> > >>>>>> One of the main reason for a major release is to push out
>> critical
>> > >>>>>> features with a faster cadence to the users. If we are pulling
>> more
>> > and
>> > >>>>>> more different types of features to a minor release, that branch
>> > will
>> > >>>>>> become more destabilized and it may be tough to say that 3.1.2 is
>> > stable
>> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
>> features
>> > in
>> > >>>>>> subsequent minor release.
>> > >>>>>> For few companies, it makes sense to push out these new features
>> > >>>>>> faster to make a reach to the users. Adding to the point to the
>> > backporting
>> > >>>>>> issues, I agree that its a pain and we can workaround that with
>> > some git
>> > >>>>>> scripts. If we can make such scripts available to committers,
>> > backport will
>> > >>>>>> be seem-less across branches and we can achieve the faster
>> release
>> > cadence
>> > >>>>>> also.
>> > >>>>>>
>> > >>>>>> Thoughts?
>> > >>>>>>
>> > >>>>>> - Sunil
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <subru@apache.org
>> >
>> > >>>>>> wrote:
>> > >>>>>>
>> > >>>>>>> Thanks Sunil for volunteering to lead the release effort. I am
>> > >>>>>>> generally
>> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as
>> feel we
>> > >>>>>>> already
>> > >>>>>>> have too many branches to be maintained. I already see many
>> commits
>> > >>>>>>> are in
>> > >>>>>>> different branches with no apparent rationale, for e.g: 3.1 has
>> > >>>>>>> commits
>> > >>>>>>> which are absent in 3.0 etc.
>> > >>>>>>>
>> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
>> > production
>> > >>>>>>> setting so the cost of adding features should be minimal.
>> > >>>>>>>
>> > >>>>>>> Thoughts?
>> > >>>>>>>
>> > >>>>>>> -Subru
>> > >>>>>>>
>> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org>
>> > wrote:
>> > >>>>>>>
>> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>> > >>>>>>> >
>> > >>>>>>> > Yes, important changes and features are much needed, hence we
>> > will
>> > >>>>>>> be
>> > >>>>>>> > keeping the door open for them as possible. Also considering
>> few
>> > >>>>>>> more
>> > >>>>>>> > offline requests from other folks, I think extending the
>> > timeframe
>> > >>>>>>> by
>> > >>>>>>> > couple of weeks makes sense (including a second RC buffer) and
>> > >>>>>>> this should
>> > >>>>>>> > ideally help us to ship this by September itself.
>> > >>>>>>> >
>> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as
>> well)
>> > >>>>>>> >
>> > >>>>>>> > - Feature freeze date : all features to merge by August 21,
>> 2018.
>> > >>>>>>> >
>> > >>>>>>> > - Code freeze date : blockers/critical only, no improvements
>> and
>> > >>>>>>> non
>> > >>>>>>> > blocker/critical
>> > >>>>>>> >
>> > >>>>>>> > bug-fixes August 31, 2018.
>> > >>>>>>> >
>> > >>>>>>> > - Release date: September 15, 2018
>> > >>>>>>> >
>> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered your
>> > >>>>>>> questions.
>> > >>>>>>> >
>> > >>>>>>> > Thanks
>> > >>>>>>> > Sunil
>> > >>>>>>> >
>> > >>>>>>> >
>> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>> wheeleast@gmail.com
>> > >
>> > >>>>>>> wrote:
>> > >>>>>>> >
>> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1
>> for
>> > >>>>>>> that.
>> > >>>>>>> > >
>> > >>>>>>> > > To concerns from Steve,
>> > >>>>>>> > >
>> > >>>>>>> > > It is a good idea to keep the door open to get important
>> > changes
>> > >>>>>>> /
>> > >>>>>>> > > features in before cutoff. I would prefer to keep the
>> proposed
>> > >>>>>>> release
>> > >>>>>>> > date
>> > >>>>>>> > > to make sure things can happen earlier instead of last
>> minute
>> > >>>>>>> and we all
>> > >>>>>>> > > know that releases are always get delayed :). I'm also fine
>> if
>> > >>>>>>> we want
>> > >>>>>>> > get
>> > >>>>>>> > > another several weeks time.
>> > >>>>>>> > >
>> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that before
>> > >>>>>>> thanksgiving.
>> > >>>>>>> > > Do you think is it good or too early / late?
>> > >>>>>>> > >
>> > >>>>>>> > > Eric,
>> > >>>>>>> > >
>> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135
>> can
>> > >>>>>>> get merged
>> > >>>>>>> > > in time, we probably not need the YARN-8220.
>> > >>>>>>> > >
>> > >>>>>>> > > Sunil,
>> > >>>>>>> > >
>> > >>>>>>> > > Could u update
>> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>> > >>>>>>> > Roadmap
>> > >>>>>>> > > with proposed plan as well? We can fill feature list first
>> > >>>>>>> before getting
>> > >>>>>>> > > consensus of time.
>> > >>>>>>> > >
>> > >>>>>>> > > Thanks,
>> > >>>>>>> > > Wangda
>> > >>>>>>> > >
>> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>> > >>>>>>> <fabbri@cloudera.com.invalid
>> > >>>>>>> > >
>> > >>>>>>> > > wrote:
>> > >>>>>>> > >
>> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>> > >>>>>>> stevel@hortonworks.com>
>> > >>>>>>> > >> wrote:
>> > >>>>>>> > >>
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org
>> > <mailto:
>> > >>>>>>> > >> > sunilg@apache.org>> wrote:
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > I would also would like to take this opportunity to come
>> up
>> > >>>>>>> with a
>> > >>>>>>> > >> detailed
>> > >>>>>>> > >> > plan.
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > - Feature freeze date : all features should be merged by
>> > >>>>>>> August 10,
>> > >>>>>>> > >> 2018.
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > <snip>
>> > >>>>>>> > >>
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > Please let me know if I missed any features targeted to
>> 3.2
>> > >>>>>>> per this
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake
>> > Gen
>> > >>>>>>> 2
>> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > I don't think this is quite ready yet, I've been doing
>> work
>> > >>>>>>> on it, but
>> > >>>>>>> > >> if
>> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
>> timely
>> > >>>>>>> reviews on
>> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > I've uprated that to a blocker feature; will review the
>> S3 &
>> > >>>>>>> S3Guard
>> > >>>>>>> > >> JIRAs
>> > >>>>>>> > >> > to see which of those are blocking. Then there are some
>> > >>>>>>> pressing
>> > >>>>>>> > "guave,
>> > >>>>>>> > >> > java 9 prep"
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> I can help with this part if you like.
>> > >>>>>>> > >>
>> > >>>>>>> > >>
>> > >>>>>>> > >>
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > timeline. I would like to volunteer myself as release
>> > manager
>> > >>>>>>> of 3.2.0
>> > >>>>>>> > >> > release.
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > well volunteered!
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> Yes, thank you for stepping up.
>> > >>>>>>> > >>
>> > >>>>>>> > >>
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > I think this raises a good q: what timetable should we
>> have
>> > >>>>>>> for the
>> > >>>>>>> > >> 3.2. &
>> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having
>> > the
>> > >>>>>>> outline
>> > >>>>>>> > >> time
>> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's less
>> > >>>>>>> concern about
>> > >>>>>>> > >> > things not making the 3.2 dealine
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > -Steve
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> Good idea to mitigate the short deadline.
>> > >>>>>>> > >>
>> > >>>>>>> > >> -AF
>> > >>>>>>> > >>
>> > >>>>>>> > >
>> > >>>>>>> >
>> > >>>>>>>
>> > >>>>>>
>> >
>>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Thanks Robert and Haibo for quickly correcting same.
Sigh, I somehow missed one file while committing the change. Sorry for the
trouble.
- Sunil
On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rk...@cloudera.com> wrote:
> Looks like there's two that weren't updated:
> >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
> --include=pom.xml
> ./hadoop-project/pom.xml:
> <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
> ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>
> I've just pushed in an addendum commit to fix those.
> In the future, please make sure to do a sanity compile when updating poms.
>
> thanks
> - Robert
>
> On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri <fa...@cloudera.com.invalid>
> wrote:
>
>> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
>> top-level pom.xml?
>>
>>
>> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
>>
>> > Hi All
>> >
>> > As mentioned in earlier mail, I have cut branch-3.2 and reset trunk to
>> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary
>> > patches are pulled into branch-3.2.
>> >
>> > Thank You
>> > - Sunil
>> >
>> >
>> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
>> >
>> > > Hi All
>> > >
>> > > We are now down to the last Blocker and HADOOP-15407 is merged to
>> trunk.
>> > > Thanks for the support.
>> > >
>> > > *Plan for RC*
>> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>> > > RC0 for 3.2: *28th Friday*
>> > >
>> > > Thank You
>> > > Sunil
>> > >
>> > >
>> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
>> > >
>> > >> Hi All
>> > >>
>> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you for
>> > >> helping in this. I am following up on these tickets, once its closed
>> we
>> > >> will cut the 3.2 branch.
>> > >>
>> > >> Thanks
>> > >> Sunil Govindan
>> > >>
>> > >>
>> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
>> > >>
>> > >>> Hi All,
>> > >>>
>> > >>> Inline with the original 3.2 communication proposal dated 17th July
>> > >>> 2018, I would like to provide more updates.
>> > >>>
>> > >>> We are approaching previously proposed code freeze date (September
>> 14,
>> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
>> > existing
>> > >>> trunk to 3.3 if there are no issues.
>> > >>>
>> > >>> *Current Release Plan:*
>> > >>> Feature freeze date : all features to merge by September 7, 2018.
>> > >>> Code freeze date : blockers/critical only, no improvements and
>> > >>> blocker/critical bug-fixes September 14, 2018.
>> > >>> Release date: September 28, 2018
>> > >>>
>> > >>> If any critical/blocker tickets which are targeted to 3.2.0, we
>> need to
>> > >>> backport to 3.2 post branch cut.
>> > >>>
>> > >>> Here's an updated 3.2.0 feature status:
>> > >>>
>> > >>> 1. Merged & Completed features:
>> > >>>
>> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> > >>> workloads Initial cut.
>> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> > >>> Scheduler.
>> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
>> > API
>> > >>> and CLI.
>> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement
>> works.
>> > >>>
>> > >>> 2. Features close to finish:
>> > >>>
>> > >>> - (Steve) S3Guard Phase III. Close to commit.
>> > >>> - (Steve) S3a phase V. Close to commit.
>> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>> > >>>
>> > >>> 3. Tentative/Cancelled features for 3.2:
>> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>> > >>> ATSv2. Patch in progress.
>> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> challenging to
>> > >>> be done before Aug 2018.
>> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> > >>> Challenging as more discussions are on-going.
>> > >>>
>> > >>> *Summary of 3.2.0 issues status:*
>> > >>> 19 Blocker and Critical issues [1] are open, I am following up with
>> > >>> owners to get status on each of them to get in by Code Freeze date.
>> > >>>
>> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> > (Blocker,
>> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
>> > ORDER
>> > >>> BY priority DESC
>> > >>>
>> > >>> Thanks,
>> > >>> Sunil
>> > >>>
>> > >>>
>> > >>>
>> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
>> > >>>
>> > >>>> Hi All,
>> > >>>>
>> > >>>> Inline with earlier communication dated 17th July 2018, I would
>> like
>> > to
>> > >>>> provide some updates.
>> > >>>>
>> > >>>> We are approaching previously proposed code freeze date (Aug 31).
>> > >>>>
>> > >>>> One of the critical feature Node Attributes feature merge
>> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need a bit
>> > more
>> > >>>> time. With regard to this, suggesting to push the feature/code
>> freeze
>> > for 2
>> > >>>> more weeks to accommodate these jiras too.
>> > >>>>
>> > >>>> Proposing Updated changes in plan inline with this:
>> > >>>> Feature freeze date : all features to merge by September 7, 2018.
>> > >>>> Code freeze date : blockers/critical only, no improvements and
>> > >>>> blocker/critical bug-fixes September 14, 2018.
>> > >>>> Release date: September 28, 2018
>> > >>>>
>> > >>>> If any features in branch which are targeted to 3.2.0, please
>> reply to
>> > >>>> this email thread.
>> > >>>>
>> > >>>> *Here's an updated 3.2.0 feature status:*
>> > >>>>
>> > >>>> 1. Merged & Completed features:
>> > >>>>
>> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> > >>>> workloads Initial cut.
>> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> > >>>> Scheduler.
>> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> Service
>> > >>>> API and CLI.
>> > >>>>
>> > >>>> 2. Features close to finish:
>> > >>>>
>> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> Merge/Vote
>> > >>>> Ongoing.
>> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>> > >>>> ATSv2. Patch in progress.
>> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
>> > works.
>> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>> > >>>> Storage. In progress.
>> > >>>>
>> > >>>> 3. Tentative features:
>> > >>>>
>> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> challenging
>> > to
>> > >>>> be done before Aug 2018.
>> > >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> > >>>> Challenging as more discussions are on-going.
>> > >>>>
>> > >>>> *Summary of 3.2.0 issues status:*
>> > >>>>
>> > >>>> 26 Blocker and Critical issues [1] are open, I am following up with
>> > >>>> owners to get status on each of them to get in by Code Freeze date.
>> > >>>>
>> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> > Version/s" =
>> > >>>> 3.2.0 ORDER BY priority DESC
>> > >>>>
>> > >>>> Thanks,
>> > >>>> Sunil
>> > >>>>
>> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org>
>> wrote:
>> > >>>>
>> > >>>>> Hi All,
>> > >>>>>
>> > >>>>> Thanks for the feedbacks. Inline with earlier communication dated
>> > 17th
>> > >>>>> July 2018, I would like to provide some updates.
>> > >>>>>
>> > >>>>> We are approaching previously proposed feature freeze date (Aug
>> 21,
>> > >>>>> about 7 days from today).
>> > >>>>> If any features in branch which are targeted to 3.2.0, please
>> reply
>> > to
>> > >>>>> this email thread.
>> > >>>>> Steve has mentioned about the s3 features which will come close to
>> > >>>>> Code Freeze Date (Aug 31st).
>> > >>>>>
>> > >>>>> *Here's an updated 3.2.0 feature status:*
>> > >>>>>
>> > >>>>> 1. Merged & Completed features:
>> > >>>>>
>> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> > >>>>> workloads Initial cut.
>> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> > >>>>>
>> > >>>>> 2. Features close to finish:
>> > >>>>>
>> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
>> > >>>>> patches are all in, only one last
>> > >>>>> patch is in review state.
>> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> > >>>>> Scheduler. Close to commit.
>> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> Service
>> > >>>>> API and CLI. 2 patches are pending
>> > >>>>> which will be closed by Feature freeze date.
>> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>> from
>> > >>>>> ATSv2. Patch in progress.
>> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
>> > >>>>> works.
>> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>> > >>>>> Storage. In progress.
>> > >>>>>
>> > >>>>> 3. Tentative features:
>> > >>>>>
>> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> challenging
>> > >>>>> to be done before Aug 2018.
>> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> > >>>>> Challenging as more discussions are on-going.
>> > >>>>>
>> > >>>>> *Summary of 3.2.0 issues status:*
>> > >>>>>
>> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking with
>> > owners
>> > >>>>> to get status on each of them to get in by Code Freeze date.
>> > >>>>>
>> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> > Version/s" =
>> > >>>>> 3.2.0 ORDER BY priority DESC
>> > >>>>>
>> > >>>>> Thanks,
>> > >>>>> Sunil
>> > >>>>>
>> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org>
>> wrote:
>> > >>>>>
>> > >>>>>> Thanks Subru for the thoughts.
>> > >>>>>> One of the main reason for a major release is to push out
>> critical
>> > >>>>>> features with a faster cadence to the users. If we are pulling
>> more
>> > and
>> > >>>>>> more different types of features to a minor release, that branch
>> > will
>> > >>>>>> become more destabilized and it may be tough to say that 3.1.2 is
>> > stable
>> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
>> features
>> > in
>> > >>>>>> subsequent minor release.
>> > >>>>>> For few companies, it makes sense to push out these new features
>> > >>>>>> faster to make a reach to the users. Adding to the point to the
>> > backporting
>> > >>>>>> issues, I agree that its a pain and we can workaround that with
>> > some git
>> > >>>>>> scripts. If we can make such scripts available to committers,
>> > backport will
>> > >>>>>> be seem-less across branches and we can achieve the faster
>> release
>> > cadence
>> > >>>>>> also.
>> > >>>>>>
>> > >>>>>> Thoughts?
>> > >>>>>>
>> > >>>>>> - Sunil
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <subru@apache.org
>> >
>> > >>>>>> wrote:
>> > >>>>>>
>> > >>>>>>> Thanks Sunil for volunteering to lead the release effort. I am
>> > >>>>>>> generally
>> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as
>> feel we
>> > >>>>>>> already
>> > >>>>>>> have too many branches to be maintained. I already see many
>> commits
>> > >>>>>>> are in
>> > >>>>>>> different branches with no apparent rationale, for e.g: 3.1 has
>> > >>>>>>> commits
>> > >>>>>>> which are absent in 3.0 etc.
>> > >>>>>>>
>> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
>> > production
>> > >>>>>>> setting so the cost of adding features should be minimal.
>> > >>>>>>>
>> > >>>>>>> Thoughts?
>> > >>>>>>>
>> > >>>>>>> -Subru
>> > >>>>>>>
>> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org>
>> > wrote:
>> > >>>>>>>
>> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>> > >>>>>>> >
>> > >>>>>>> > Yes, important changes and features are much needed, hence we
>> > will
>> > >>>>>>> be
>> > >>>>>>> > keeping the door open for them as possible. Also considering
>> few
>> > >>>>>>> more
>> > >>>>>>> > offline requests from other folks, I think extending the
>> > timeframe
>> > >>>>>>> by
>> > >>>>>>> > couple of weeks makes sense (including a second RC buffer) and
>> > >>>>>>> this should
>> > >>>>>>> > ideally help us to ship this by September itself.
>> > >>>>>>> >
>> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as
>> well)
>> > >>>>>>> >
>> > >>>>>>> > - Feature freeze date : all features to merge by August 21,
>> 2018.
>> > >>>>>>> >
>> > >>>>>>> > - Code freeze date : blockers/critical only, no improvements
>> and
>> > >>>>>>> non
>> > >>>>>>> > blocker/critical
>> > >>>>>>> >
>> > >>>>>>> > bug-fixes August 31, 2018.
>> > >>>>>>> >
>> > >>>>>>> > - Release date: September 15, 2018
>> > >>>>>>> >
>> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered your
>> > >>>>>>> questions.
>> > >>>>>>> >
>> > >>>>>>> > Thanks
>> > >>>>>>> > Sunil
>> > >>>>>>> >
>> > >>>>>>> >
>> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>> wheeleast@gmail.com
>> > >
>> > >>>>>>> wrote:
>> > >>>>>>> >
>> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1
>> for
>> > >>>>>>> that.
>> > >>>>>>> > >
>> > >>>>>>> > > To concerns from Steve,
>> > >>>>>>> > >
>> > >>>>>>> > > It is a good idea to keep the door open to get important
>> > changes
>> > >>>>>>> /
>> > >>>>>>> > > features in before cutoff. I would prefer to keep the
>> proposed
>> > >>>>>>> release
>> > >>>>>>> > date
>> > >>>>>>> > > to make sure things can happen earlier instead of last
>> minute
>> > >>>>>>> and we all
>> > >>>>>>> > > know that releases are always get delayed :). I'm also fine
>> if
>> > >>>>>>> we want
>> > >>>>>>> > get
>> > >>>>>>> > > another several weeks time.
>> > >>>>>>> > >
>> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that before
>> > >>>>>>> thanksgiving.
>> > >>>>>>> > > Do you think is it good or too early / late?
>> > >>>>>>> > >
>> > >>>>>>> > > Eric,
>> > >>>>>>> > >
>> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135
>> can
>> > >>>>>>> get merged
>> > >>>>>>> > > in time, we probably not need the YARN-8220.
>> > >>>>>>> > >
>> > >>>>>>> > > Sunil,
>> > >>>>>>> > >
>> > >>>>>>> > > Could u update
>> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>> > >>>>>>> > Roadmap
>> > >>>>>>> > > with proposed plan as well? We can fill feature list first
>> > >>>>>>> before getting
>> > >>>>>>> > > consensus of time.
>> > >>>>>>> > >
>> > >>>>>>> > > Thanks,
>> > >>>>>>> > > Wangda
>> > >>>>>>> > >
>> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>> > >>>>>>> <fabbri@cloudera.com.invalid
>> > >>>>>>> > >
>> > >>>>>>> > > wrote:
>> > >>>>>>> > >
>> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>> > >>>>>>> stevel@hortonworks.com>
>> > >>>>>>> > >> wrote:
>> > >>>>>>> > >>
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org
>> > <mailto:
>> > >>>>>>> > >> > sunilg@apache.org>> wrote:
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > I would also would like to take this opportunity to come
>> up
>> > >>>>>>> with a
>> > >>>>>>> > >> detailed
>> > >>>>>>> > >> > plan.
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > - Feature freeze date : all features should be merged by
>> > >>>>>>> August 10,
>> > >>>>>>> > >> 2018.
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > <snip>
>> > >>>>>>> > >>
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > Please let me know if I missed any features targeted to
>> 3.2
>> > >>>>>>> per this
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake
>> > Gen
>> > >>>>>>> 2
>> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > I don't think this is quite ready yet, I've been doing
>> work
>> > >>>>>>> on it, but
>> > >>>>>>> > >> if
>> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
>> timely
>> > >>>>>>> reviews on
>> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > I've uprated that to a blocker feature; will review the
>> S3 &
>> > >>>>>>> S3Guard
>> > >>>>>>> > >> JIRAs
>> > >>>>>>> > >> > to see which of those are blocking. Then there are some
>> > >>>>>>> pressing
>> > >>>>>>> > "guave,
>> > >>>>>>> > >> > java 9 prep"
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> I can help with this part if you like.
>> > >>>>>>> > >>
>> > >>>>>>> > >>
>> > >>>>>>> > >>
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > timeline. I would like to volunteer myself as release
>> > manager
>> > >>>>>>> of 3.2.0
>> > >>>>>>> > >> > release.
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > well volunteered!
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> Yes, thank you for stepping up.
>> > >>>>>>> > >>
>> > >>>>>>> > >>
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > I think this raises a good q: what timetable should we
>> have
>> > >>>>>>> for the
>> > >>>>>>> > >> 3.2. &
>> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having
>> > the
>> > >>>>>>> outline
>> > >>>>>>> > >> time
>> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's less
>> > >>>>>>> concern about
>> > >>>>>>> > >> > things not making the 3.2 dealine
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > -Steve
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> Good idea to mitigate the short deadline.
>> > >>>>>>> > >>
>> > >>>>>>> > >> -AF
>> > >>>>>>> > >>
>> > >>>>>>> > >
>> > >>>>>>> >
>> > >>>>>>>
>> > >>>>>>
>> >
>>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Thanks Robert and Haibo for quickly correcting same.
Sigh, I somehow missed one file while committing the change. Sorry for the
trouble.
- Sunil
On Wed, Oct 3, 2018 at 5:22 AM Robert Kanter <rk...@cloudera.com> wrote:
> Looks like there's two that weren't updated:
> >> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
> --include=pom.xml
> ./hadoop-project/pom.xml:
> <hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
> ./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
>
> I've just pushed in an addendum commit to fix those.
> In the future, please make sure to do a sanity compile when updating poms.
>
> thanks
> - Robert
>
> On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri <fa...@cloudera.com.invalid>
> wrote:
>
>> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
>> top-level pom.xml?
>>
>>
>> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
>>
>> > Hi All
>> >
>> > As mentioned in earlier mail, I have cut branch-3.2 and reset trunk to
>> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary
>> > patches are pulled into branch-3.2.
>> >
>> > Thank You
>> > - Sunil
>> >
>> >
>> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
>> >
>> > > Hi All
>> > >
>> > > We are now down to the last Blocker and HADOOP-15407 is merged to
>> trunk.
>> > > Thanks for the support.
>> > >
>> > > *Plan for RC*
>> > > 3.2 branch cut and reset trunk : *25th Tuesday*
>> > > RC0 for 3.2: *28th Friday*
>> > >
>> > > Thank You
>> > > Sunil
>> > >
>> > >
>> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
>> > >
>> > >> Hi All
>> > >>
>> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you for
>> > >> helping in this. I am following up on these tickets, once its closed
>> we
>> > >> will cut the 3.2 branch.
>> > >>
>> > >> Thanks
>> > >> Sunil Govindan
>> > >>
>> > >>
>> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
>> > >>
>> > >>> Hi All,
>> > >>>
>> > >>> Inline with the original 3.2 communication proposal dated 17th July
>> > >>> 2018, I would like to provide more updates.
>> > >>>
>> > >>> We are approaching previously proposed code freeze date (September
>> 14,
>> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
>> > existing
>> > >>> trunk to 3.3 if there are no issues.
>> > >>>
>> > >>> *Current Release Plan:*
>> > >>> Feature freeze date : all features to merge by September 7, 2018.
>> > >>> Code freeze date : blockers/critical only, no improvements and
>> > >>> blocker/critical bug-fixes September 14, 2018.
>> > >>> Release date: September 28, 2018
>> > >>>
>> > >>> If any critical/blocker tickets which are targeted to 3.2.0, we
>> need to
>> > >>> backport to 3.2 post branch cut.
>> > >>>
>> > >>> Here's an updated 3.2.0 feature status:
>> > >>>
>> > >>> 1. Merged & Completed features:
>> > >>>
>> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> > >>> workloads Initial cut.
>> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> > >>> Scheduler.
>> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
>> > API
>> > >>> and CLI.
>> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement
>> works.
>> > >>>
>> > >>> 2. Features close to finish:
>> > >>>
>> > >>> - (Steve) S3Guard Phase III. Close to commit.
>> > >>> - (Steve) S3a phase V. Close to commit.
>> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
>> > >>>
>> > >>> 3. Tentative/Cancelled features for 3.2:
>> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>> > >>> ATSv2. Patch in progress.
>> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> challenging to
>> > >>> be done before Aug 2018.
>> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> > >>> Challenging as more discussions are on-going.
>> > >>>
>> > >>> *Summary of 3.2.0 issues status:*
>> > >>> 19 Blocker and Critical issues [1] are open, I am following up with
>> > >>> owners to get status on each of them to get in by Code Freeze date.
>> > >>>
>> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> > (Blocker,
>> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
>> > ORDER
>> > >>> BY priority DESC
>> > >>>
>> > >>> Thanks,
>> > >>> Sunil
>> > >>>
>> > >>>
>> > >>>
>> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
>> > >>>
>> > >>>> Hi All,
>> > >>>>
>> > >>>> Inline with earlier communication dated 17th July 2018, I would
>> like
>> > to
>> > >>>> provide some updates.
>> > >>>>
>> > >>>> We are approaching previously proposed code freeze date (Aug 31).
>> > >>>>
>> > >>>> One of the critical feature Node Attributes feature merge
>> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need a bit
>> > more
>> > >>>> time. With regard to this, suggesting to push the feature/code
>> freeze
>> > for 2
>> > >>>> more weeks to accommodate these jiras too.
>> > >>>>
>> > >>>> Proposing Updated changes in plan inline with this:
>> > >>>> Feature freeze date : all features to merge by September 7, 2018.
>> > >>>> Code freeze date : blockers/critical only, no improvements and
>> > >>>> blocker/critical bug-fixes September 14, 2018.
>> > >>>> Release date: September 28, 2018
>> > >>>>
>> > >>>> If any features in branch which are targeted to 3.2.0, please
>> reply to
>> > >>>> this email thread.
>> > >>>>
>> > >>>> *Here's an updated 3.2.0 feature status:*
>> > >>>>
>> > >>>> 1. Merged & Completed features:
>> > >>>>
>> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> > >>>> workloads Initial cut.
>> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> > >>>> Scheduler.
>> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> Service
>> > >>>> API and CLI.
>> > >>>>
>> > >>>> 2. Features close to finish:
>> > >>>>
>> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> Merge/Vote
>> > >>>> Ongoing.
>> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>> > >>>> ATSv2. Patch in progress.
>> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
>> > works.
>> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>> > >>>> Storage. In progress.
>> > >>>>
>> > >>>> 3. Tentative features:
>> > >>>>
>> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> challenging
>> > to
>> > >>>> be done before Aug 2018.
>> > >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> > >>>> Challenging as more discussions are on-going.
>> > >>>>
>> > >>>> *Summary of 3.2.0 issues status:*
>> > >>>>
>> > >>>> 26 Blocker and Critical issues [1] are open, I am following up with
>> > >>>> owners to get status on each of them to get in by Code Freeze date.
>> > >>>>
>> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> > Version/s" =
>> > >>>> 3.2.0 ORDER BY priority DESC
>> > >>>>
>> > >>>> Thanks,
>> > >>>> Sunil
>> > >>>>
>> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org>
>> wrote:
>> > >>>>
>> > >>>>> Hi All,
>> > >>>>>
>> > >>>>> Thanks for the feedbacks. Inline with earlier communication dated
>> > 17th
>> > >>>>> July 2018, I would like to provide some updates.
>> > >>>>>
>> > >>>>> We are approaching previously proposed feature freeze date (Aug
>> 21,
>> > >>>>> about 7 days from today).
>> > >>>>> If any features in branch which are targeted to 3.2.0, please
>> reply
>> > to
>> > >>>>> this email thread.
>> > >>>>> Steve has mentioned about the s3 features which will come close to
>> > >>>>> Code Freeze Date (Aug 31st).
>> > >>>>>
>> > >>>>> *Here's an updated 3.2.0 feature status:*
>> > >>>>>
>> > >>>>> 1. Merged & Completed features:
>> > >>>>>
>> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>> > >>>>> workloads Initial cut.
>> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> > >>>>>
>> > >>>>> 2. Features close to finish:
>> > >>>>>
>> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
>> > >>>>> patches are all in, only one last
>> > >>>>> patch is in review state.
>> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>> > >>>>> Scheduler. Close to commit.
>> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
>> Service
>> > >>>>> API and CLI. 2 patches are pending
>> > >>>>> which will be closed by Feature freeze date.
>> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps
>> from
>> > >>>>> ATSv2. Patch in progress.
>> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
>> > >>>>> works.
>> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>> > >>>>> Storage. In progress.
>> > >>>>>
>> > >>>>> 3. Tentative features:
>> > >>>>>
>> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
>> challenging
>> > >>>>> to be done before Aug 2018.
>> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> > >>>>> Challenging as more discussions are on-going.
>> > >>>>>
>> > >>>>> *Summary of 3.2.0 issues status:*
>> > >>>>>
>> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking with
>> > owners
>> > >>>>> to get status on each of them to get in by Code Freeze date.
>> > >>>>>
>> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
>> > Version/s" =
>> > >>>>> 3.2.0 ORDER BY priority DESC
>> > >>>>>
>> > >>>>> Thanks,
>> > >>>>> Sunil
>> > >>>>>
>> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org>
>> wrote:
>> > >>>>>
>> > >>>>>> Thanks Subru for the thoughts.
>> > >>>>>> One of the main reason for a major release is to push out
>> critical
>> > >>>>>> features with a faster cadence to the users. If we are pulling
>> more
>> > and
>> > >>>>>> more different types of features to a minor release, that branch
>> > will
>> > >>>>>> become more destabilized and it may be tough to say that 3.1.2 is
>> > stable
>> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
>> features
>> > in
>> > >>>>>> subsequent minor release.
>> > >>>>>> For few companies, it makes sense to push out these new features
>> > >>>>>> faster to make a reach to the users. Adding to the point to the
>> > backporting
>> > >>>>>> issues, I agree that its a pain and we can workaround that with
>> > some git
>> > >>>>>> scripts. If we can make such scripts available to committers,
>> > backport will
>> > >>>>>> be seem-less across branches and we can achieve the faster
>> release
>> > cadence
>> > >>>>>> also.
>> > >>>>>>
>> > >>>>>> Thoughts?
>> > >>>>>>
>> > >>>>>> - Sunil
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <subru@apache.org
>> >
>> > >>>>>> wrote:
>> > >>>>>>
>> > >>>>>>> Thanks Sunil for volunteering to lead the release effort. I am
>> > >>>>>>> generally
>> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as
>> feel we
>> > >>>>>>> already
>> > >>>>>>> have too many branches to be maintained. I already see many
>> commits
>> > >>>>>>> are in
>> > >>>>>>> different branches with no apparent rationale, for e.g: 3.1 has
>> > >>>>>>> commits
>> > >>>>>>> which are absent in 3.0 etc.
>> > >>>>>>>
>> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
>> > production
>> > >>>>>>> setting so the cost of adding features should be minimal.
>> > >>>>>>>
>> > >>>>>>> Thoughts?
>> > >>>>>>>
>> > >>>>>>> -Subru
>> > >>>>>>>
>> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org>
>> > wrote:
>> > >>>>>>>
>> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>> > >>>>>>> >
>> > >>>>>>> > Yes, important changes and features are much needed, hence we
>> > will
>> > >>>>>>> be
>> > >>>>>>> > keeping the door open for them as possible. Also considering
>> few
>> > >>>>>>> more
>> > >>>>>>> > offline requests from other folks, I think extending the
>> > timeframe
>> > >>>>>>> by
>> > >>>>>>> > couple of weeks makes sense (including a second RC buffer) and
>> > >>>>>>> this should
>> > >>>>>>> > ideally help us to ship this by September itself.
>> > >>>>>>> >
>> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as
>> well)
>> > >>>>>>> >
>> > >>>>>>> > - Feature freeze date : all features to merge by August 21,
>> 2018.
>> > >>>>>>> >
>> > >>>>>>> > - Code freeze date : blockers/critical only, no improvements
>> and
>> > >>>>>>> non
>> > >>>>>>> > blocker/critical
>> > >>>>>>> >
>> > >>>>>>> > bug-fixes August 31, 2018.
>> > >>>>>>> >
>> > >>>>>>> > - Release date: September 15, 2018
>> > >>>>>>> >
>> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered your
>> > >>>>>>> questions.
>> > >>>>>>> >
>> > >>>>>>> > Thanks
>> > >>>>>>> > Sunil
>> > >>>>>>> >
>> > >>>>>>> >
>> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
>> wheeleast@gmail.com
>> > >
>> > >>>>>>> wrote:
>> > >>>>>>> >
>> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1
>> for
>> > >>>>>>> that.
>> > >>>>>>> > >
>> > >>>>>>> > > To concerns from Steve,
>> > >>>>>>> > >
>> > >>>>>>> > > It is a good idea to keep the door open to get important
>> > changes
>> > >>>>>>> /
>> > >>>>>>> > > features in before cutoff. I would prefer to keep the
>> proposed
>> > >>>>>>> release
>> > >>>>>>> > date
>> > >>>>>>> > > to make sure things can happen earlier instead of last
>> minute
>> > >>>>>>> and we all
>> > >>>>>>> > > know that releases are always get delayed :). I'm also fine
>> if
>> > >>>>>>> we want
>> > >>>>>>> > get
>> > >>>>>>> > > another several weeks time.
>> > >>>>>>> > >
>> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that before
>> > >>>>>>> thanksgiving.
>> > >>>>>>> > > Do you think is it good or too early / late?
>> > >>>>>>> > >
>> > >>>>>>> > > Eric,
>> > >>>>>>> > >
>> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135
>> can
>> > >>>>>>> get merged
>> > >>>>>>> > > in time, we probably not need the YARN-8220.
>> > >>>>>>> > >
>> > >>>>>>> > > Sunil,
>> > >>>>>>> > >
>> > >>>>>>> > > Could u update
>> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>> > >>>>>>> > Roadmap
>> > >>>>>>> > > with proposed plan as well? We can fill feature list first
>> > >>>>>>> before getting
>> > >>>>>>> > > consensus of time.
>> > >>>>>>> > >
>> > >>>>>>> > > Thanks,
>> > >>>>>>> > > Wangda
>> > >>>>>>> > >
>> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>> > >>>>>>> <fabbri@cloudera.com.invalid
>> > >>>>>>> > >
>> > >>>>>>> > > wrote:
>> > >>>>>>> > >
>> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>> > >>>>>>> stevel@hortonworks.com>
>> > >>>>>>> > >> wrote:
>> > >>>>>>> > >>
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org
>> > <mailto:
>> > >>>>>>> > >> > sunilg@apache.org>> wrote:
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > I would also would like to take this opportunity to come
>> up
>> > >>>>>>> with a
>> > >>>>>>> > >> detailed
>> > >>>>>>> > >> > plan.
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > - Feature freeze date : all features should be merged by
>> > >>>>>>> August 10,
>> > >>>>>>> > >> 2018.
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > <snip>
>> > >>>>>>> > >>
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > Please let me know if I missed any features targeted to
>> 3.2
>> > >>>>>>> per this
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake
>> > Gen
>> > >>>>>>> 2
>> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > I don't think this is quite ready yet, I've been doing
>> work
>> > >>>>>>> on it, but
>> > >>>>>>> > >> if
>> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some
>> timely
>> > >>>>>>> reviews on
>> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > I've uprated that to a blocker feature; will review the
>> S3 &
>> > >>>>>>> S3Guard
>> > >>>>>>> > >> JIRAs
>> > >>>>>>> > >> > to see which of those are blocking. Then there are some
>> > >>>>>>> pressing
>> > >>>>>>> > "guave,
>> > >>>>>>> > >> > java 9 prep"
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> I can help with this part if you like.
>> > >>>>>>> > >>
>> > >>>>>>> > >>
>> > >>>>>>> > >>
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > timeline. I would like to volunteer myself as release
>> > manager
>> > >>>>>>> of 3.2.0
>> > >>>>>>> > >> > release.
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > well volunteered!
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> Yes, thank you for stepping up.
>> > >>>>>>> > >>
>> > >>>>>>> > >>
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > I think this raises a good q: what timetable should we
>> have
>> > >>>>>>> for the
>> > >>>>>>> > >> 3.2. &
>> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having
>> > the
>> > >>>>>>> outline
>> > >>>>>>> > >> time
>> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's less
>> > >>>>>>> concern about
>> > >>>>>>> > >> > things not making the 3.2 dealine
>> > >>>>>>> > >> >
>> > >>>>>>> > >> > -Steve
>> > >>>>>>> > >> >
>> > >>>>>>> > >> >
>> > >>>>>>> > >> Good idea to mitigate the short deadline.
>> > >>>>>>> > >>
>> > >>>>>>> > >> -AF
>> > >>>>>>> > >>
>> > >>>>>>> > >
>> > >>>>>>> >
>> > >>>>>>>
>> > >>>>>>
>> >
>>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Robert Kanter <rk...@cloudera.com.INVALID>.
Looks like there's two that weren't updated:
>> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
--include=pom.xml
./hadoop-project/pom.xml:
<hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
I've just pushed in an addendum commit to fix those.
In the future, please make sure to do a sanity compile when updating poms.
thanks
- Robert
On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri <fa...@cloudera.com.invalid>
wrote:
> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
> top-level pom.xml?
>
>
> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
>
> > Hi All
> >
> > As mentioned in earlier mail, I have cut branch-3.2 and reset trunk to
> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary
> > patches are pulled into branch-3.2.
> >
> > Thank You
> > - Sunil
> >
> >
> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
> >
> > > Hi All
> > >
> > > We are now down to the last Blocker and HADOOP-15407 is merged to
> trunk.
> > > Thanks for the support.
> > >
> > > *Plan for RC*
> > > 3.2 branch cut and reset trunk : *25th Tuesday*
> > > RC0 for 3.2: *28th Friday*
> > >
> > > Thank You
> > > Sunil
> > >
> > >
> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
> > >
> > >> Hi All
> > >>
> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you for
> > >> helping in this. I am following up on these tickets, once its closed
> we
> > >> will cut the 3.2 branch.
> > >>
> > >> Thanks
> > >> Sunil Govindan
> > >>
> > >>
> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
> > >>
> > >>> Hi All,
> > >>>
> > >>> Inline with the original 3.2 communication proposal dated 17th July
> > >>> 2018, I would like to provide more updates.
> > >>>
> > >>> We are approaching previously proposed code freeze date (September
> 14,
> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
> > existing
> > >>> trunk to 3.3 if there are no issues.
> > >>>
> > >>> *Current Release Plan:*
> > >>> Feature freeze date : all features to merge by September 7, 2018.
> > >>> Code freeze date : blockers/critical only, no improvements and
> > >>> blocker/critical bug-fixes September 14, 2018.
> > >>> Release date: September 28, 2018
> > >>>
> > >>> If any critical/blocker tickets which are targeted to 3.2.0, we need
> to
> > >>> backport to 3.2 post branch cut.
> > >>>
> > >>> Here's an updated 3.2.0 feature status:
> > >>>
> > >>> 1. Merged & Completed features:
> > >>>
> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> > >>> workloads Initial cut.
> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> > >>> Scheduler.
> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> > API
> > >>> and CLI.
> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement
> works.
> > >>>
> > >>> 2. Features close to finish:
> > >>>
> > >>> - (Steve) S3Guard Phase III. Close to commit.
> > >>> - (Steve) S3a phase V. Close to commit.
> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
> > >>>
> > >>> 3. Tentative/Cancelled features for 3.2:
> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> > >>> ATSv2. Patch in progress.
> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging
> to
> > >>> be done before Aug 2018.
> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> > >>> Challenging as more discussions are on-going.
> > >>>
> > >>> *Summary of 3.2.0 issues status:*
> > >>> 19 Blocker and Critical issues [1] are open, I am following up with
> > >>> owners to get status on each of them to get in by Code Freeze date.
> > >>>
> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> > (Blocker,
> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
> > ORDER
> > >>> BY priority DESC
> > >>>
> > >>> Thanks,
> > >>> Sunil
> > >>>
> > >>>
> > >>>
> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
> > >>>
> > >>>> Hi All,
> > >>>>
> > >>>> Inline with earlier communication dated 17th July 2018, I would like
> > to
> > >>>> provide some updates.
> > >>>>
> > >>>> We are approaching previously proposed code freeze date (Aug 31).
> > >>>>
> > >>>> One of the critical feature Node Attributes feature merge
> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need a bit
> > more
> > >>>> time. With regard to this, suggesting to push the feature/code
> freeze
> > for 2
> > >>>> more weeks to accommodate these jiras too.
> > >>>>
> > >>>> Proposing Updated changes in plan inline with this:
> > >>>> Feature freeze date : all features to merge by September 7, 2018.
> > >>>> Code freeze date : blockers/critical only, no improvements and
> > >>>> blocker/critical bug-fixes September 14, 2018.
> > >>>> Release date: September 28, 2018
> > >>>>
> > >>>> If any features in branch which are targeted to 3.2.0, please reply
> to
> > >>>> this email thread.
> > >>>>
> > >>>> *Here's an updated 3.2.0 feature status:*
> > >>>>
> > >>>> 1. Merged & Completed features:
> > >>>>
> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> > >>>> workloads Initial cut.
> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> > >>>> Scheduler.
> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> > >>>> API and CLI.
> > >>>>
> > >>>> 2. Features close to finish:
> > >>>>
> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> Merge/Vote
> > >>>> Ongoing.
> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> > >>>> ATSv2. Patch in progress.
> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
> > works.
> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> > >>>> Storage. In progress.
> > >>>>
> > >>>> 3. Tentative features:
> > >>>>
> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging
> > to
> > >>>> be done before Aug 2018.
> > >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> > >>>> Challenging as more discussions are on-going.
> > >>>>
> > >>>> *Summary of 3.2.0 issues status:*
> > >>>>
> > >>>> 26 Blocker and Critical issues [1] are open, I am following up with
> > >>>> owners to get status on each of them to get in by Code Freeze date.
> > >>>>
> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> > Version/s" =
> > >>>> 3.2.0 ORDER BY priority DESC
> > >>>>
> > >>>> Thanks,
> > >>>> Sunil
> > >>>>
> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
> > >>>>
> > >>>>> Hi All,
> > >>>>>
> > >>>>> Thanks for the feedbacks. Inline with earlier communication dated
> > 17th
> > >>>>> July 2018, I would like to provide some updates.
> > >>>>>
> > >>>>> We are approaching previously proposed feature freeze date (Aug 21,
> > >>>>> about 7 days from today).
> > >>>>> If any features in branch which are targeted to 3.2.0, please reply
> > to
> > >>>>> this email thread.
> > >>>>> Steve has mentioned about the s3 features which will come close to
> > >>>>> Code Freeze Date (Aug 31st).
> > >>>>>
> > >>>>> *Here's an updated 3.2.0 feature status:*
> > >>>>>
> > >>>>> 1. Merged & Completed features:
> > >>>>>
> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> > >>>>> workloads Initial cut.
> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> > >>>>>
> > >>>>> 2. Features close to finish:
> > >>>>>
> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
> > >>>>> patches are all in, only one last
> > >>>>> patch is in review state.
> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> > >>>>> Scheduler. Close to commit.
> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> Service
> > >>>>> API and CLI. 2 patches are pending
> > >>>>> which will be closed by Feature freeze date.
> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> > >>>>> ATSv2. Patch in progress.
> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
> > >>>>> works.
> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> > >>>>> Storage. In progress.
> > >>>>>
> > >>>>> 3. Tentative features:
> > >>>>>
> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> challenging
> > >>>>> to be done before Aug 2018.
> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> > >>>>> Challenging as more discussions are on-going.
> > >>>>>
> > >>>>> *Summary of 3.2.0 issues status:*
> > >>>>>
> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking with
> > owners
> > >>>>> to get status on each of them to get in by Code Freeze date.
> > >>>>>
> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> > Version/s" =
> > >>>>> 3.2.0 ORDER BY priority DESC
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Sunil
> > >>>>>
> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
> > >>>>>
> > >>>>>> Thanks Subru for the thoughts.
> > >>>>>> One of the main reason for a major release is to push out critical
> > >>>>>> features with a faster cadence to the users. If we are pulling
> more
> > and
> > >>>>>> more different types of features to a minor release, that branch
> > will
> > >>>>>> become more destabilized and it may be tough to say that 3.1.2 is
> > stable
> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
> features
> > in
> > >>>>>> subsequent minor release.
> > >>>>>> For few companies, it makes sense to push out these new features
> > >>>>>> faster to make a reach to the users. Adding to the point to the
> > backporting
> > >>>>>> issues, I agree that its a pain and we can workaround that with
> > some git
> > >>>>>> scripts. If we can make such scripts available to committers,
> > backport will
> > >>>>>> be seem-less across branches and we can achieve the faster release
> > cadence
> > >>>>>> also.
> > >>>>>>
> > >>>>>> Thoughts?
> > >>>>>>
> > >>>>>> - Sunil
> > >>>>>>
> > >>>>>>
> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Thanks Sunil for volunteering to lead the release effort. I am
> > >>>>>>> generally
> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel
> we
> > >>>>>>> already
> > >>>>>>> have too many branches to be maintained. I already see many
> commits
> > >>>>>>> are in
> > >>>>>>> different branches with no apparent rationale, for e.g: 3.1 has
> > >>>>>>> commits
> > >>>>>>> which are absent in 3.0 etc.
> > >>>>>>>
> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
> > production
> > >>>>>>> setting so the cost of adding features should be minimal.
> > >>>>>>>
> > >>>>>>> Thoughts?
> > >>>>>>>
> > >>>>>>> -Subru
> > >>>>>>>
> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org>
> > wrote:
> > >>>>>>>
> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> > >>>>>>> >
> > >>>>>>> > Yes, important changes and features are much needed, hence we
> > will
> > >>>>>>> be
> > >>>>>>> > keeping the door open for them as possible. Also considering
> few
> > >>>>>>> more
> > >>>>>>> > offline requests from other folks, I think extending the
> > timeframe
> > >>>>>>> by
> > >>>>>>> > couple of weeks makes sense (including a second RC buffer) and
> > >>>>>>> this should
> > >>>>>>> > ideally help us to ship this by September itself.
> > >>>>>>> >
> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
> > >>>>>>> >
> > >>>>>>> > - Feature freeze date : all features to merge by August 21,
> 2018.
> > >>>>>>> >
> > >>>>>>> > - Code freeze date : blockers/critical only, no improvements
> and
> > >>>>>>> non
> > >>>>>>> > blocker/critical
> > >>>>>>> >
> > >>>>>>> > bug-fixes August 31, 2018.
> > >>>>>>> >
> > >>>>>>> > - Release date: September 15, 2018
> > >>>>>>> >
> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered your
> > >>>>>>> questions.
> > >>>>>>> >
> > >>>>>>> > Thanks
> > >>>>>>> > Sunil
> > >>>>>>> >
> > >>>>>>> >
> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
> wheeleast@gmail.com
> > >
> > >>>>>>> wrote:
> > >>>>>>> >
> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for
> > >>>>>>> that.
> > >>>>>>> > >
> > >>>>>>> > > To concerns from Steve,
> > >>>>>>> > >
> > >>>>>>> > > It is a good idea to keep the door open to get important
> > changes
> > >>>>>>> /
> > >>>>>>> > > features in before cutoff. I would prefer to keep the
> proposed
> > >>>>>>> release
> > >>>>>>> > date
> > >>>>>>> > > to make sure things can happen earlier instead of last minute
> > >>>>>>> and we all
> > >>>>>>> > > know that releases are always get delayed :). I'm also fine
> if
> > >>>>>>> we want
> > >>>>>>> > get
> > >>>>>>> > > another several weeks time.
> > >>>>>>> > >
> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that before
> > >>>>>>> thanksgiving.
> > >>>>>>> > > Do you think is it good or too early / late?
> > >>>>>>> > >
> > >>>>>>> > > Eric,
> > >>>>>>> > >
> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can
> > >>>>>>> get merged
> > >>>>>>> > > in time, we probably not need the YARN-8220.
> > >>>>>>> > >
> > >>>>>>> > > Sunil,
> > >>>>>>> > >
> > >>>>>>> > > Could u update
> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
> > >>>>>>> > Roadmap
> > >>>>>>> > > with proposed plan as well? We can fill feature list first
> > >>>>>>> before getting
> > >>>>>>> > > consensus of time.
> > >>>>>>> > >
> > >>>>>>> > > Thanks,
> > >>>>>>> > > Wangda
> > >>>>>>> > >
> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> > >>>>>>> <fabbri@cloudera.com.invalid
> > >>>>>>> > >
> > >>>>>>> > > wrote:
> > >>>>>>> > >
> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> > >>>>>>> stevel@hortonworks.com>
> > >>>>>>> > >> wrote:
> > >>>>>>> > >>
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org
> > <mailto:
> > >>>>>>> > >> > sunilg@apache.org>> wrote:
> > >>>>>>> > >> >
> > >>>>>>> > >> > I would also would like to take this opportunity to come
> up
> > >>>>>>> with a
> > >>>>>>> > >> detailed
> > >>>>>>> > >> > plan.
> > >>>>>>> > >> >
> > >>>>>>> > >> > - Feature freeze date : all features should be merged by
> > >>>>>>> August 10,
> > >>>>>>> > >> 2018.
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > <snip>
> > >>>>>>> > >>
> > >>>>>>> > >> >
> > >>>>>>> > >> > Please let me know if I missed any features targeted to
> 3.2
> > >>>>>>> per this
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
> > >>>>>>> > >> >
> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake
> > Gen
> > >>>>>>> 2
> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> > >>>>>>> > >> >
> > >>>>>>> > >> > I don't think this is quite ready yet, I've been doing
> work
> > >>>>>>> on it, but
> > >>>>>>> > >> if
> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
> > >>>>>>> reviews on
> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> > >>>>>>> > >> >
> > >>>>>>> > >> > I've uprated that to a blocker feature; will review the
> S3 &
> > >>>>>>> S3Guard
> > >>>>>>> > >> JIRAs
> > >>>>>>> > >> > to see which of those are blocking. Then there are some
> > >>>>>>> pressing
> > >>>>>>> > "guave,
> > >>>>>>> > >> > java 9 prep"
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> I can help with this part if you like.
> > >>>>>>> > >>
> > >>>>>>> > >>
> > >>>>>>> > >>
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > timeline. I would like to volunteer myself as release
> > manager
> > >>>>>>> of 3.2.0
> > >>>>>>> > >> > release.
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > well volunteered!
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> Yes, thank you for stepping up.
> > >>>>>>> > >>
> > >>>>>>> > >>
> > >>>>>>> > >> >
> > >>>>>>> > >> > I think this raises a good q: what timetable should we
> have
> > >>>>>>> for the
> > >>>>>>> > >> 3.2. &
> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having
> > the
> > >>>>>>> outline
> > >>>>>>> > >> time
> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's less
> > >>>>>>> concern about
> > >>>>>>> > >> > things not making the 3.2 dealine
> > >>>>>>> > >> >
> > >>>>>>> > >> > -Steve
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> Good idea to mitigate the short deadline.
> > >>>>>>> > >>
> > >>>>>>> > >> -AF
> > >>>>>>> > >>
> > >>>>>>> > >
> > >>>>>>> >
> > >>>>>>>
> > >>>>>>
> >
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Robert Kanter <rk...@cloudera.com.INVALID>.
Looks like there's two that weren't updated:
>> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
--include=pom.xml
./hadoop-project/pom.xml:
<hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
I've just pushed in an addendum commit to fix those.
In the future, please make sure to do a sanity compile when updating poms.
thanks
- Robert
On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri <fa...@cloudera.com.invalid>
wrote:
> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
> top-level pom.xml?
>
>
> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
>
> > Hi All
> >
> > As mentioned in earlier mail, I have cut branch-3.2 and reset trunk to
> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary
> > patches are pulled into branch-3.2.
> >
> > Thank You
> > - Sunil
> >
> >
> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
> >
> > > Hi All
> > >
> > > We are now down to the last Blocker and HADOOP-15407 is merged to
> trunk.
> > > Thanks for the support.
> > >
> > > *Plan for RC*
> > > 3.2 branch cut and reset trunk : *25th Tuesday*
> > > RC0 for 3.2: *28th Friday*
> > >
> > > Thank You
> > > Sunil
> > >
> > >
> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
> > >
> > >> Hi All
> > >>
> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you for
> > >> helping in this. I am following up on these tickets, once its closed
> we
> > >> will cut the 3.2 branch.
> > >>
> > >> Thanks
> > >> Sunil Govindan
> > >>
> > >>
> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
> > >>
> > >>> Hi All,
> > >>>
> > >>> Inline with the original 3.2 communication proposal dated 17th July
> > >>> 2018, I would like to provide more updates.
> > >>>
> > >>> We are approaching previously proposed code freeze date (September
> 14,
> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
> > existing
> > >>> trunk to 3.3 if there are no issues.
> > >>>
> > >>> *Current Release Plan:*
> > >>> Feature freeze date : all features to merge by September 7, 2018.
> > >>> Code freeze date : blockers/critical only, no improvements and
> > >>> blocker/critical bug-fixes September 14, 2018.
> > >>> Release date: September 28, 2018
> > >>>
> > >>> If any critical/blocker tickets which are targeted to 3.2.0, we need
> to
> > >>> backport to 3.2 post branch cut.
> > >>>
> > >>> Here's an updated 3.2.0 feature status:
> > >>>
> > >>> 1. Merged & Completed features:
> > >>>
> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> > >>> workloads Initial cut.
> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> > >>> Scheduler.
> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> > API
> > >>> and CLI.
> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement
> works.
> > >>>
> > >>> 2. Features close to finish:
> > >>>
> > >>> - (Steve) S3Guard Phase III. Close to commit.
> > >>> - (Steve) S3a phase V. Close to commit.
> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
> > >>>
> > >>> 3. Tentative/Cancelled features for 3.2:
> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> > >>> ATSv2. Patch in progress.
> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging
> to
> > >>> be done before Aug 2018.
> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> > >>> Challenging as more discussions are on-going.
> > >>>
> > >>> *Summary of 3.2.0 issues status:*
> > >>> 19 Blocker and Critical issues [1] are open, I am following up with
> > >>> owners to get status on each of them to get in by Code Freeze date.
> > >>>
> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> > (Blocker,
> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
> > ORDER
> > >>> BY priority DESC
> > >>>
> > >>> Thanks,
> > >>> Sunil
> > >>>
> > >>>
> > >>>
> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
> > >>>
> > >>>> Hi All,
> > >>>>
> > >>>> Inline with earlier communication dated 17th July 2018, I would like
> > to
> > >>>> provide some updates.
> > >>>>
> > >>>> We are approaching previously proposed code freeze date (Aug 31).
> > >>>>
> > >>>> One of the critical feature Node Attributes feature merge
> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need a bit
> > more
> > >>>> time. With regard to this, suggesting to push the feature/code
> freeze
> > for 2
> > >>>> more weeks to accommodate these jiras too.
> > >>>>
> > >>>> Proposing Updated changes in plan inline with this:
> > >>>> Feature freeze date : all features to merge by September 7, 2018.
> > >>>> Code freeze date : blockers/critical only, no improvements and
> > >>>> blocker/critical bug-fixes September 14, 2018.
> > >>>> Release date: September 28, 2018
> > >>>>
> > >>>> If any features in branch which are targeted to 3.2.0, please reply
> to
> > >>>> this email thread.
> > >>>>
> > >>>> *Here's an updated 3.2.0 feature status:*
> > >>>>
> > >>>> 1. Merged & Completed features:
> > >>>>
> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> > >>>> workloads Initial cut.
> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> > >>>> Scheduler.
> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> > >>>> API and CLI.
> > >>>>
> > >>>> 2. Features close to finish:
> > >>>>
> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> Merge/Vote
> > >>>> Ongoing.
> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> > >>>> ATSv2. Patch in progress.
> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
> > works.
> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> > >>>> Storage. In progress.
> > >>>>
> > >>>> 3. Tentative features:
> > >>>>
> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging
> > to
> > >>>> be done before Aug 2018.
> > >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> > >>>> Challenging as more discussions are on-going.
> > >>>>
> > >>>> *Summary of 3.2.0 issues status:*
> > >>>>
> > >>>> 26 Blocker and Critical issues [1] are open, I am following up with
> > >>>> owners to get status on each of them to get in by Code Freeze date.
> > >>>>
> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> > Version/s" =
> > >>>> 3.2.0 ORDER BY priority DESC
> > >>>>
> > >>>> Thanks,
> > >>>> Sunil
> > >>>>
> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
> > >>>>
> > >>>>> Hi All,
> > >>>>>
> > >>>>> Thanks for the feedbacks. Inline with earlier communication dated
> > 17th
> > >>>>> July 2018, I would like to provide some updates.
> > >>>>>
> > >>>>> We are approaching previously proposed feature freeze date (Aug 21,
> > >>>>> about 7 days from today).
> > >>>>> If any features in branch which are targeted to 3.2.0, please reply
> > to
> > >>>>> this email thread.
> > >>>>> Steve has mentioned about the s3 features which will come close to
> > >>>>> Code Freeze Date (Aug 31st).
> > >>>>>
> > >>>>> *Here's an updated 3.2.0 feature status:*
> > >>>>>
> > >>>>> 1. Merged & Completed features:
> > >>>>>
> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> > >>>>> workloads Initial cut.
> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> > >>>>>
> > >>>>> 2. Features close to finish:
> > >>>>>
> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
> > >>>>> patches are all in, only one last
> > >>>>> patch is in review state.
> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> > >>>>> Scheduler. Close to commit.
> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> Service
> > >>>>> API and CLI. 2 patches are pending
> > >>>>> which will be closed by Feature freeze date.
> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> > >>>>> ATSv2. Patch in progress.
> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
> > >>>>> works.
> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> > >>>>> Storage. In progress.
> > >>>>>
> > >>>>> 3. Tentative features:
> > >>>>>
> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> challenging
> > >>>>> to be done before Aug 2018.
> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> > >>>>> Challenging as more discussions are on-going.
> > >>>>>
> > >>>>> *Summary of 3.2.0 issues status:*
> > >>>>>
> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking with
> > owners
> > >>>>> to get status on each of them to get in by Code Freeze date.
> > >>>>>
> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> > Version/s" =
> > >>>>> 3.2.0 ORDER BY priority DESC
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Sunil
> > >>>>>
> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
> > >>>>>
> > >>>>>> Thanks Subru for the thoughts.
> > >>>>>> One of the main reason for a major release is to push out critical
> > >>>>>> features with a faster cadence to the users. If we are pulling
> more
> > and
> > >>>>>> more different types of features to a minor release, that branch
> > will
> > >>>>>> become more destabilized and it may be tough to say that 3.1.2 is
> > stable
> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
> features
> > in
> > >>>>>> subsequent minor release.
> > >>>>>> For few companies, it makes sense to push out these new features
> > >>>>>> faster to make a reach to the users. Adding to the point to the
> > backporting
> > >>>>>> issues, I agree that its a pain and we can workaround that with
> > some git
> > >>>>>> scripts. If we can make such scripts available to committers,
> > backport will
> > >>>>>> be seem-less across branches and we can achieve the faster release
> > cadence
> > >>>>>> also.
> > >>>>>>
> > >>>>>> Thoughts?
> > >>>>>>
> > >>>>>> - Sunil
> > >>>>>>
> > >>>>>>
> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Thanks Sunil for volunteering to lead the release effort. I am
> > >>>>>>> generally
> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel
> we
> > >>>>>>> already
> > >>>>>>> have too many branches to be maintained. I already see many
> commits
> > >>>>>>> are in
> > >>>>>>> different branches with no apparent rationale, for e.g: 3.1 has
> > >>>>>>> commits
> > >>>>>>> which are absent in 3.0 etc.
> > >>>>>>>
> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
> > production
> > >>>>>>> setting so the cost of adding features should be minimal.
> > >>>>>>>
> > >>>>>>> Thoughts?
> > >>>>>>>
> > >>>>>>> -Subru
> > >>>>>>>
> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org>
> > wrote:
> > >>>>>>>
> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> > >>>>>>> >
> > >>>>>>> > Yes, important changes and features are much needed, hence we
> > will
> > >>>>>>> be
> > >>>>>>> > keeping the door open for them as possible. Also considering
> few
> > >>>>>>> more
> > >>>>>>> > offline requests from other folks, I think extending the
> > timeframe
> > >>>>>>> by
> > >>>>>>> > couple of weeks makes sense (including a second RC buffer) and
> > >>>>>>> this should
> > >>>>>>> > ideally help us to ship this by September itself.
> > >>>>>>> >
> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
> > >>>>>>> >
> > >>>>>>> > - Feature freeze date : all features to merge by August 21,
> 2018.
> > >>>>>>> >
> > >>>>>>> > - Code freeze date : blockers/critical only, no improvements
> and
> > >>>>>>> non
> > >>>>>>> > blocker/critical
> > >>>>>>> >
> > >>>>>>> > bug-fixes August 31, 2018.
> > >>>>>>> >
> > >>>>>>> > - Release date: September 15, 2018
> > >>>>>>> >
> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered your
> > >>>>>>> questions.
> > >>>>>>> >
> > >>>>>>> > Thanks
> > >>>>>>> > Sunil
> > >>>>>>> >
> > >>>>>>> >
> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
> wheeleast@gmail.com
> > >
> > >>>>>>> wrote:
> > >>>>>>> >
> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for
> > >>>>>>> that.
> > >>>>>>> > >
> > >>>>>>> > > To concerns from Steve,
> > >>>>>>> > >
> > >>>>>>> > > It is a good idea to keep the door open to get important
> > changes
> > >>>>>>> /
> > >>>>>>> > > features in before cutoff. I would prefer to keep the
> proposed
> > >>>>>>> release
> > >>>>>>> > date
> > >>>>>>> > > to make sure things can happen earlier instead of last minute
> > >>>>>>> and we all
> > >>>>>>> > > know that releases are always get delayed :). I'm also fine
> if
> > >>>>>>> we want
> > >>>>>>> > get
> > >>>>>>> > > another several weeks time.
> > >>>>>>> > >
> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that before
> > >>>>>>> thanksgiving.
> > >>>>>>> > > Do you think is it good or too early / late?
> > >>>>>>> > >
> > >>>>>>> > > Eric,
> > >>>>>>> > >
> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can
> > >>>>>>> get merged
> > >>>>>>> > > in time, we probably not need the YARN-8220.
> > >>>>>>> > >
> > >>>>>>> > > Sunil,
> > >>>>>>> > >
> > >>>>>>> > > Could u update
> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
> > >>>>>>> > Roadmap
> > >>>>>>> > > with proposed plan as well? We can fill feature list first
> > >>>>>>> before getting
> > >>>>>>> > > consensus of time.
> > >>>>>>> > >
> > >>>>>>> > > Thanks,
> > >>>>>>> > > Wangda
> > >>>>>>> > >
> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> > >>>>>>> <fabbri@cloudera.com.invalid
> > >>>>>>> > >
> > >>>>>>> > > wrote:
> > >>>>>>> > >
> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> > >>>>>>> stevel@hortonworks.com>
> > >>>>>>> > >> wrote:
> > >>>>>>> > >>
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org
> > <mailto:
> > >>>>>>> > >> > sunilg@apache.org>> wrote:
> > >>>>>>> > >> >
> > >>>>>>> > >> > I would also would like to take this opportunity to come
> up
> > >>>>>>> with a
> > >>>>>>> > >> detailed
> > >>>>>>> > >> > plan.
> > >>>>>>> > >> >
> > >>>>>>> > >> > - Feature freeze date : all features should be merged by
> > >>>>>>> August 10,
> > >>>>>>> > >> 2018.
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > <snip>
> > >>>>>>> > >>
> > >>>>>>> > >> >
> > >>>>>>> > >> > Please let me know if I missed any features targeted to
> 3.2
> > >>>>>>> per this
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
> > >>>>>>> > >> >
> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake
> > Gen
> > >>>>>>> 2
> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> > >>>>>>> > >> >
> > >>>>>>> > >> > I don't think this is quite ready yet, I've been doing
> work
> > >>>>>>> on it, but
> > >>>>>>> > >> if
> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
> > >>>>>>> reviews on
> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> > >>>>>>> > >> >
> > >>>>>>> > >> > I've uprated that to a blocker feature; will review the
> S3 &
> > >>>>>>> S3Guard
> > >>>>>>> > >> JIRAs
> > >>>>>>> > >> > to see which of those are blocking. Then there are some
> > >>>>>>> pressing
> > >>>>>>> > "guave,
> > >>>>>>> > >> > java 9 prep"
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> I can help with this part if you like.
> > >>>>>>> > >>
> > >>>>>>> > >>
> > >>>>>>> > >>
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > timeline. I would like to volunteer myself as release
> > manager
> > >>>>>>> of 3.2.0
> > >>>>>>> > >> > release.
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > well volunteered!
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> Yes, thank you for stepping up.
> > >>>>>>> > >>
> > >>>>>>> > >>
> > >>>>>>> > >> >
> > >>>>>>> > >> > I think this raises a good q: what timetable should we
> have
> > >>>>>>> for the
> > >>>>>>> > >> 3.2. &
> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having
> > the
> > >>>>>>> outline
> > >>>>>>> > >> time
> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's less
> > >>>>>>> concern about
> > >>>>>>> > >> > things not making the 3.2 dealine
> > >>>>>>> > >> >
> > >>>>>>> > >> > -Steve
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> Good idea to mitigate the short deadline.
> > >>>>>>> > >>
> > >>>>>>> > >> -AF
> > >>>>>>> > >>
> > >>>>>>> > >
> > >>>>>>> >
> > >>>>>>>
> > >>>>>>
> >
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Robert Kanter <rk...@cloudera.com.INVALID>.
Looks like there's two that weren't updated:
>> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
--include=pom.xml
./hadoop-project/pom.xml:
<hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
I've just pushed in an addendum commit to fix those.
In the future, please make sure to do a sanity compile when updating poms.
thanks
- Robert
On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri <fa...@cloudera.com.invalid>
wrote:
> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
> top-level pom.xml?
>
>
> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
>
> > Hi All
> >
> > As mentioned in earlier mail, I have cut branch-3.2 and reset trunk to
> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary
> > patches are pulled into branch-3.2.
> >
> > Thank You
> > - Sunil
> >
> >
> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
> >
> > > Hi All
> > >
> > > We are now down to the last Blocker and HADOOP-15407 is merged to
> trunk.
> > > Thanks for the support.
> > >
> > > *Plan for RC*
> > > 3.2 branch cut and reset trunk : *25th Tuesday*
> > > RC0 for 3.2: *28th Friday*
> > >
> > > Thank You
> > > Sunil
> > >
> > >
> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
> > >
> > >> Hi All
> > >>
> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you for
> > >> helping in this. I am following up on these tickets, once its closed
> we
> > >> will cut the 3.2 branch.
> > >>
> > >> Thanks
> > >> Sunil Govindan
> > >>
> > >>
> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
> > >>
> > >>> Hi All,
> > >>>
> > >>> Inline with the original 3.2 communication proposal dated 17th July
> > >>> 2018, I would like to provide more updates.
> > >>>
> > >>> We are approaching previously proposed code freeze date (September
> 14,
> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
> > existing
> > >>> trunk to 3.3 if there are no issues.
> > >>>
> > >>> *Current Release Plan:*
> > >>> Feature freeze date : all features to merge by September 7, 2018.
> > >>> Code freeze date : blockers/critical only, no improvements and
> > >>> blocker/critical bug-fixes September 14, 2018.
> > >>> Release date: September 28, 2018
> > >>>
> > >>> If any critical/blocker tickets which are targeted to 3.2.0, we need
> to
> > >>> backport to 3.2 post branch cut.
> > >>>
> > >>> Here's an updated 3.2.0 feature status:
> > >>>
> > >>> 1. Merged & Completed features:
> > >>>
> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> > >>> workloads Initial cut.
> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> > >>> Scheduler.
> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> > API
> > >>> and CLI.
> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement
> works.
> > >>>
> > >>> 2. Features close to finish:
> > >>>
> > >>> - (Steve) S3Guard Phase III. Close to commit.
> > >>> - (Steve) S3a phase V. Close to commit.
> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
> > >>>
> > >>> 3. Tentative/Cancelled features for 3.2:
> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> > >>> ATSv2. Patch in progress.
> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging
> to
> > >>> be done before Aug 2018.
> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> > >>> Challenging as more discussions are on-going.
> > >>>
> > >>> *Summary of 3.2.0 issues status:*
> > >>> 19 Blocker and Critical issues [1] are open, I am following up with
> > >>> owners to get status on each of them to get in by Code Freeze date.
> > >>>
> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> > (Blocker,
> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
> > ORDER
> > >>> BY priority DESC
> > >>>
> > >>> Thanks,
> > >>> Sunil
> > >>>
> > >>>
> > >>>
> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
> > >>>
> > >>>> Hi All,
> > >>>>
> > >>>> Inline with earlier communication dated 17th July 2018, I would like
> > to
> > >>>> provide some updates.
> > >>>>
> > >>>> We are approaching previously proposed code freeze date (Aug 31).
> > >>>>
> > >>>> One of the critical feature Node Attributes feature merge
> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need a bit
> > more
> > >>>> time. With regard to this, suggesting to push the feature/code
> freeze
> > for 2
> > >>>> more weeks to accommodate these jiras too.
> > >>>>
> > >>>> Proposing Updated changes in plan inline with this:
> > >>>> Feature freeze date : all features to merge by September 7, 2018.
> > >>>> Code freeze date : blockers/critical only, no improvements and
> > >>>> blocker/critical bug-fixes September 14, 2018.
> > >>>> Release date: September 28, 2018
> > >>>>
> > >>>> If any features in branch which are targeted to 3.2.0, please reply
> to
> > >>>> this email thread.
> > >>>>
> > >>>> *Here's an updated 3.2.0 feature status:*
> > >>>>
> > >>>> 1. Merged & Completed features:
> > >>>>
> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> > >>>> workloads Initial cut.
> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> > >>>> Scheduler.
> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> > >>>> API and CLI.
> > >>>>
> > >>>> 2. Features close to finish:
> > >>>>
> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> Merge/Vote
> > >>>> Ongoing.
> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> > >>>> ATSv2. Patch in progress.
> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
> > works.
> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> > >>>> Storage. In progress.
> > >>>>
> > >>>> 3. Tentative features:
> > >>>>
> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging
> > to
> > >>>> be done before Aug 2018.
> > >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> > >>>> Challenging as more discussions are on-going.
> > >>>>
> > >>>> *Summary of 3.2.0 issues status:*
> > >>>>
> > >>>> 26 Blocker and Critical issues [1] are open, I am following up with
> > >>>> owners to get status on each of them to get in by Code Freeze date.
> > >>>>
> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> > Version/s" =
> > >>>> 3.2.0 ORDER BY priority DESC
> > >>>>
> > >>>> Thanks,
> > >>>> Sunil
> > >>>>
> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
> > >>>>
> > >>>>> Hi All,
> > >>>>>
> > >>>>> Thanks for the feedbacks. Inline with earlier communication dated
> > 17th
> > >>>>> July 2018, I would like to provide some updates.
> > >>>>>
> > >>>>> We are approaching previously proposed feature freeze date (Aug 21,
> > >>>>> about 7 days from today).
> > >>>>> If any features in branch which are targeted to 3.2.0, please reply
> > to
> > >>>>> this email thread.
> > >>>>> Steve has mentioned about the s3 features which will come close to
> > >>>>> Code Freeze Date (Aug 31st).
> > >>>>>
> > >>>>> *Here's an updated 3.2.0 feature status:*
> > >>>>>
> > >>>>> 1. Merged & Completed features:
> > >>>>>
> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> > >>>>> workloads Initial cut.
> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> > >>>>>
> > >>>>> 2. Features close to finish:
> > >>>>>
> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
> > >>>>> patches are all in, only one last
> > >>>>> patch is in review state.
> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> > >>>>> Scheduler. Close to commit.
> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> Service
> > >>>>> API and CLI. 2 patches are pending
> > >>>>> which will be closed by Feature freeze date.
> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> > >>>>> ATSv2. Patch in progress.
> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
> > >>>>> works.
> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> > >>>>> Storage. In progress.
> > >>>>>
> > >>>>> 3. Tentative features:
> > >>>>>
> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> challenging
> > >>>>> to be done before Aug 2018.
> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> > >>>>> Challenging as more discussions are on-going.
> > >>>>>
> > >>>>> *Summary of 3.2.0 issues status:*
> > >>>>>
> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking with
> > owners
> > >>>>> to get status on each of them to get in by Code Freeze date.
> > >>>>>
> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> > Version/s" =
> > >>>>> 3.2.0 ORDER BY priority DESC
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Sunil
> > >>>>>
> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
> > >>>>>
> > >>>>>> Thanks Subru for the thoughts.
> > >>>>>> One of the main reason for a major release is to push out critical
> > >>>>>> features with a faster cadence to the users. If we are pulling
> more
> > and
> > >>>>>> more different types of features to a minor release, that branch
> > will
> > >>>>>> become more destabilized and it may be tough to say that 3.1.2 is
> > stable
> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
> features
> > in
> > >>>>>> subsequent minor release.
> > >>>>>> For few companies, it makes sense to push out these new features
> > >>>>>> faster to make a reach to the users. Adding to the point to the
> > backporting
> > >>>>>> issues, I agree that its a pain and we can workaround that with
> > some git
> > >>>>>> scripts. If we can make such scripts available to committers,
> > backport will
> > >>>>>> be seem-less across branches and we can achieve the faster release
> > cadence
> > >>>>>> also.
> > >>>>>>
> > >>>>>> Thoughts?
> > >>>>>>
> > >>>>>> - Sunil
> > >>>>>>
> > >>>>>>
> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Thanks Sunil for volunteering to lead the release effort. I am
> > >>>>>>> generally
> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel
> we
> > >>>>>>> already
> > >>>>>>> have too many branches to be maintained. I already see many
> commits
> > >>>>>>> are in
> > >>>>>>> different branches with no apparent rationale, for e.g: 3.1 has
> > >>>>>>> commits
> > >>>>>>> which are absent in 3.0 etc.
> > >>>>>>>
> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
> > production
> > >>>>>>> setting so the cost of adding features should be minimal.
> > >>>>>>>
> > >>>>>>> Thoughts?
> > >>>>>>>
> > >>>>>>> -Subru
> > >>>>>>>
> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org>
> > wrote:
> > >>>>>>>
> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> > >>>>>>> >
> > >>>>>>> > Yes, important changes and features are much needed, hence we
> > will
> > >>>>>>> be
> > >>>>>>> > keeping the door open for them as possible. Also considering
> few
> > >>>>>>> more
> > >>>>>>> > offline requests from other folks, I think extending the
> > timeframe
> > >>>>>>> by
> > >>>>>>> > couple of weeks makes sense (including a second RC buffer) and
> > >>>>>>> this should
> > >>>>>>> > ideally help us to ship this by September itself.
> > >>>>>>> >
> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
> > >>>>>>> >
> > >>>>>>> > - Feature freeze date : all features to merge by August 21,
> 2018.
> > >>>>>>> >
> > >>>>>>> > - Code freeze date : blockers/critical only, no improvements
> and
> > >>>>>>> non
> > >>>>>>> > blocker/critical
> > >>>>>>> >
> > >>>>>>> > bug-fixes August 31, 2018.
> > >>>>>>> >
> > >>>>>>> > - Release date: September 15, 2018
> > >>>>>>> >
> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered your
> > >>>>>>> questions.
> > >>>>>>> >
> > >>>>>>> > Thanks
> > >>>>>>> > Sunil
> > >>>>>>> >
> > >>>>>>> >
> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
> wheeleast@gmail.com
> > >
> > >>>>>>> wrote:
> > >>>>>>> >
> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for
> > >>>>>>> that.
> > >>>>>>> > >
> > >>>>>>> > > To concerns from Steve,
> > >>>>>>> > >
> > >>>>>>> > > It is a good idea to keep the door open to get important
> > changes
> > >>>>>>> /
> > >>>>>>> > > features in before cutoff. I would prefer to keep the
> proposed
> > >>>>>>> release
> > >>>>>>> > date
> > >>>>>>> > > to make sure things can happen earlier instead of last minute
> > >>>>>>> and we all
> > >>>>>>> > > know that releases are always get delayed :). I'm also fine
> if
> > >>>>>>> we want
> > >>>>>>> > get
> > >>>>>>> > > another several weeks time.
> > >>>>>>> > >
> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that before
> > >>>>>>> thanksgiving.
> > >>>>>>> > > Do you think is it good or too early / late?
> > >>>>>>> > >
> > >>>>>>> > > Eric,
> > >>>>>>> > >
> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can
> > >>>>>>> get merged
> > >>>>>>> > > in time, we probably not need the YARN-8220.
> > >>>>>>> > >
> > >>>>>>> > > Sunil,
> > >>>>>>> > >
> > >>>>>>> > > Could u update
> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
> > >>>>>>> > Roadmap
> > >>>>>>> > > with proposed plan as well? We can fill feature list first
> > >>>>>>> before getting
> > >>>>>>> > > consensus of time.
> > >>>>>>> > >
> > >>>>>>> > > Thanks,
> > >>>>>>> > > Wangda
> > >>>>>>> > >
> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> > >>>>>>> <fabbri@cloudera.com.invalid
> > >>>>>>> > >
> > >>>>>>> > > wrote:
> > >>>>>>> > >
> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> > >>>>>>> stevel@hortonworks.com>
> > >>>>>>> > >> wrote:
> > >>>>>>> > >>
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org
> > <mailto:
> > >>>>>>> > >> > sunilg@apache.org>> wrote:
> > >>>>>>> > >> >
> > >>>>>>> > >> > I would also would like to take this opportunity to come
> up
> > >>>>>>> with a
> > >>>>>>> > >> detailed
> > >>>>>>> > >> > plan.
> > >>>>>>> > >> >
> > >>>>>>> > >> > - Feature freeze date : all features should be merged by
> > >>>>>>> August 10,
> > >>>>>>> > >> 2018.
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > <snip>
> > >>>>>>> > >>
> > >>>>>>> > >> >
> > >>>>>>> > >> > Please let me know if I missed any features targeted to
> 3.2
> > >>>>>>> per this
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
> > >>>>>>> > >> >
> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake
> > Gen
> > >>>>>>> 2
> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> > >>>>>>> > >> >
> > >>>>>>> > >> > I don't think this is quite ready yet, I've been doing
> work
> > >>>>>>> on it, but
> > >>>>>>> > >> if
> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
> > >>>>>>> reviews on
> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> > >>>>>>> > >> >
> > >>>>>>> > >> > I've uprated that to a blocker feature; will review the
> S3 &
> > >>>>>>> S3Guard
> > >>>>>>> > >> JIRAs
> > >>>>>>> > >> > to see which of those are blocking. Then there are some
> > >>>>>>> pressing
> > >>>>>>> > "guave,
> > >>>>>>> > >> > java 9 prep"
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> I can help with this part if you like.
> > >>>>>>> > >>
> > >>>>>>> > >>
> > >>>>>>> > >>
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > timeline. I would like to volunteer myself as release
> > manager
> > >>>>>>> of 3.2.0
> > >>>>>>> > >> > release.
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > well volunteered!
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> Yes, thank you for stepping up.
> > >>>>>>> > >>
> > >>>>>>> > >>
> > >>>>>>> > >> >
> > >>>>>>> > >> > I think this raises a good q: what timetable should we
> have
> > >>>>>>> for the
> > >>>>>>> > >> 3.2. &
> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having
> > the
> > >>>>>>> outline
> > >>>>>>> > >> time
> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's less
> > >>>>>>> concern about
> > >>>>>>> > >> > things not making the 3.2 dealine
> > >>>>>>> > >> >
> > >>>>>>> > >> > -Steve
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> Good idea to mitigate the short deadline.
> > >>>>>>> > >>
> > >>>>>>> > >> -AF
> > >>>>>>> > >>
> > >>>>>>> > >
> > >>>>>>> >
> > >>>>>>>
> > >>>>>>
> >
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Robert Kanter <rk...@cloudera.com.INVALID>.
Looks like there's two that weren't updated:
>> [115] 16:32 : hadoop-common (trunk) :: grep "3.2.0-SNAPSHOT" . -r
--include=pom.xml
./hadoop-project/pom.xml:
<hadoop.assemblies.version>3.2.0-SNAPSHOT</hadoop.assemblies.version>
./pom.xml: <hadoop.version>3.2.0-SNAPSHOT</hadoop.version>
I've just pushed in an addendum commit to fix those.
In the future, please make sure to do a sanity compile when updating poms.
thanks
- Robert
On Tue, Oct 2, 2018 at 11:44 AM Aaron Fabbri <fa...@cloudera.com.invalid>
wrote:
> Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
> top-level pom.xml?
>
>
> On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
>
> > Hi All
> >
> > As mentioned in earlier mail, I have cut branch-3.2 and reset trunk to
> > 3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary
> > patches are pulled into branch-3.2.
> >
> > Thank You
> > - Sunil
> >
> >
> > On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
> >
> > > Hi All
> > >
> > > We are now down to the last Blocker and HADOOP-15407 is merged to
> trunk.
> > > Thanks for the support.
> > >
> > > *Plan for RC*
> > > 3.2 branch cut and reset trunk : *25th Tuesday*
> > > RC0 for 3.2: *28th Friday*
> > >
> > > Thank You
> > > Sunil
> > >
> > >
> > > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
> > >
> > >> Hi All
> > >>
> > >> We are down to 3 Blockers and 4 Critical now. Thanks all of you for
> > >> helping in this. I am following up on these tickets, once its closed
> we
> > >> will cut the 3.2 branch.
> > >>
> > >> Thanks
> > >> Sunil Govindan
> > >>
> > >>
> > >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
> > >>
> > >>> Hi All,
> > >>>
> > >>> Inline with the original 3.2 communication proposal dated 17th July
> > >>> 2018, I would like to provide more updates.
> > >>>
> > >>> We are approaching previously proposed code freeze date (September
> 14,
> > >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
> > existing
> > >>> trunk to 3.3 if there are no issues.
> > >>>
> > >>> *Current Release Plan:*
> > >>> Feature freeze date : all features to merge by September 7, 2018.
> > >>> Code freeze date : blockers/critical only, no improvements and
> > >>> blocker/critical bug-fixes September 14, 2018.
> > >>> Release date: September 28, 2018
> > >>>
> > >>> If any critical/blocker tickets which are targeted to 3.2.0, we need
> to
> > >>> backport to 3.2 post branch cut.
> > >>>
> > >>> Here's an updated 3.2.0 feature status:
> > >>>
> > >>> 1. Merged & Completed features:
> > >>>
> > >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> > >>> workloads Initial cut.
> > >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> > >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> > >>> Scheduler.
> > >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> > API
> > >>> and CLI.
> > >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> > >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement
> works.
> > >>>
> > >>> 2. Features close to finish:
> > >>>
> > >>> - (Steve) S3Guard Phase III. Close to commit.
> > >>> - (Steve) S3a phase V. Close to commit.
> > >>> - (Steve) Support Windows Azure Storage. Close to commit.
> > >>>
> > >>> 3. Tentative/Cancelled features for 3.2:
> > >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> > >>> ATSv2. Patch in progress.
> > >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging
> to
> > >>> be done before Aug 2018.
> > >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> > >>> Challenging as more discussions are on-going.
> > >>>
> > >>> *Summary of 3.2.0 issues status:*
> > >>> 19 Blocker and Critical issues [1] are open, I am following up with
> > >>> owners to get status on each of them to get in by Code Freeze date.
> > >>>
> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> > (Blocker,
> > >>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
> > ORDER
> > >>> BY priority DESC
> > >>>
> > >>> Thanks,
> > >>> Sunil
> > >>>
> > >>>
> > >>>
> > >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
> > >>>
> > >>>> Hi All,
> > >>>>
> > >>>> Inline with earlier communication dated 17th July 2018, I would like
> > to
> > >>>> provide some updates.
> > >>>>
> > >>>> We are approaching previously proposed code freeze date (Aug 31).
> > >>>>
> > >>>> One of the critical feature Node Attributes feature merge
> > >>>> discussion/vote is ongoing. Also few other Blocker bugs need a bit
> > more
> > >>>> time. With regard to this, suggesting to push the feature/code
> freeze
> > for 2
> > >>>> more weeks to accommodate these jiras too.
> > >>>>
> > >>>> Proposing Updated changes in plan inline with this:
> > >>>> Feature freeze date : all features to merge by September 7, 2018.
> > >>>> Code freeze date : blockers/critical only, no improvements and
> > >>>> blocker/critical bug-fixes September 14, 2018.
> > >>>> Release date: September 28, 2018
> > >>>>
> > >>>> If any features in branch which are targeted to 3.2.0, please reply
> to
> > >>>> this email thread.
> > >>>>
> > >>>> *Here's an updated 3.2.0 feature status:*
> > >>>>
> > >>>> 1. Merged & Completed features:
> > >>>>
> > >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> > >>>> workloads Initial cut.
> > >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> > >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> > >>>> Scheduler.
> > >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> > >>>> API and CLI.
> > >>>>
> > >>>> 2. Features close to finish:
> > >>>>
> > >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> Merge/Vote
> > >>>> Ongoing.
> > >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> > >>>> ATSv2. Patch in progress.
> > >>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
> > works.
> > >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> > >>>> Storage. In progress.
> > >>>>
> > >>>> 3. Tentative features:
> > >>>>
> > >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging
> > to
> > >>>> be done before Aug 2018.
> > >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> > >>>> Challenging as more discussions are on-going.
> > >>>>
> > >>>> *Summary of 3.2.0 issues status:*
> > >>>>
> > >>>> 26 Blocker and Critical issues [1] are open, I am following up with
> > >>>> owners to get status on each of them to get in by Code Freeze date.
> > >>>>
> > >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> > >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> > Version/s" =
> > >>>> 3.2.0 ORDER BY priority DESC
> > >>>>
> > >>>> Thanks,
> > >>>> Sunil
> > >>>>
> > >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
> > >>>>
> > >>>>> Hi All,
> > >>>>>
> > >>>>> Thanks for the feedbacks. Inline with earlier communication dated
> > 17th
> > >>>>> July 2018, I would like to provide some updates.
> > >>>>>
> > >>>>> We are approaching previously proposed feature freeze date (Aug 21,
> > >>>>> about 7 days from today).
> > >>>>> If any features in branch which are targeted to 3.2.0, please reply
> > to
> > >>>>> this email thread.
> > >>>>> Steve has mentioned about the s3 features which will come close to
> > >>>>> Code Freeze Date (Aug 31st).
> > >>>>>
> > >>>>> *Here's an updated 3.2.0 feature status:*
> > >>>>>
> > >>>>> 1. Merged & Completed features:
> > >>>>>
> > >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> > >>>>> workloads Initial cut.
> > >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> > >>>>>
> > >>>>> 2. Features close to finish:
> > >>>>>
> > >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
> > >>>>> patches are all in, only one last
> > >>>>> patch is in review state.
> > >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> > >>>>> Scheduler. Close to commit.
> > >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN
> Service
> > >>>>> API and CLI. 2 patches are pending
> > >>>>> which will be closed by Feature freeze date.
> > >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> > >>>>> ATSv2. Patch in progress.
> > >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
> > >>>>> works.
> > >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> > >>>>> Storage. In progress.
> > >>>>>
> > >>>>> 3. Tentative features:
> > >>>>>
> > >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks
> challenging
> > >>>>> to be done before Aug 2018.
> > >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> > >>>>> Challenging as more discussions are on-going.
> > >>>>>
> > >>>>> *Summary of 3.2.0 issues status:*
> > >>>>>
> > >>>>> 39 Blocker and Critical issues [1] are open, I am checking with
> > owners
> > >>>>> to get status on each of them to get in by Code Freeze date.
> > >>>>>
> > >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> > >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> > Version/s" =
> > >>>>> 3.2.0 ORDER BY priority DESC
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Sunil
> > >>>>>
> > >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
> > >>>>>
> > >>>>>> Thanks Subru for the thoughts.
> > >>>>>> One of the main reason for a major release is to push out critical
> > >>>>>> features with a faster cadence to the users. If we are pulling
> more
> > and
> > >>>>>> more different types of features to a minor release, that branch
> > will
> > >>>>>> become more destabilized and it may be tough to say that 3.1.2 is
> > stable
> > >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize
> features
> > in
> > >>>>>> subsequent minor release.
> > >>>>>> For few companies, it makes sense to push out these new features
> > >>>>>> faster to make a reach to the users. Adding to the point to the
> > backporting
> > >>>>>> issues, I agree that its a pain and we can workaround that with
> > some git
> > >>>>>> scripts. If we can make such scripts available to committers,
> > backport will
> > >>>>>> be seem-less across branches and we can achieve the faster release
> > cadence
> > >>>>>> also.
> > >>>>>>
> > >>>>>> Thoughts?
> > >>>>>>
> > >>>>>> - Sunil
> > >>>>>>
> > >>>>>>
> > >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Thanks Sunil for volunteering to lead the release effort. I am
> > >>>>>>> generally
> > >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel
> we
> > >>>>>>> already
> > >>>>>>> have too many branches to be maintained. I already see many
> commits
> > >>>>>>> are in
> > >>>>>>> different branches with no apparent rationale, for e.g: 3.1 has
> > >>>>>>> commits
> > >>>>>>> which are absent in 3.0 etc.
> > >>>>>>>
> > >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
> > production
> > >>>>>>> setting so the cost of adding features should be minimal.
> > >>>>>>>
> > >>>>>>> Thoughts?
> > >>>>>>>
> > >>>>>>> -Subru
> > >>>>>>>
> > >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org>
> > wrote:
> > >>>>>>>
> > >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> > >>>>>>> >
> > >>>>>>> > Yes, important changes and features are much needed, hence we
> > will
> > >>>>>>> be
> > >>>>>>> > keeping the door open for them as possible. Also considering
> few
> > >>>>>>> more
> > >>>>>>> > offline requests from other folks, I think extending the
> > timeframe
> > >>>>>>> by
> > >>>>>>> > couple of weeks makes sense (including a second RC buffer) and
> > >>>>>>> this should
> > >>>>>>> > ideally help us to ship this by September itself.
> > >>>>>>> >
> > >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
> > >>>>>>> >
> > >>>>>>> > - Feature freeze date : all features to merge by August 21,
> 2018.
> > >>>>>>> >
> > >>>>>>> > - Code freeze date : blockers/critical only, no improvements
> and
> > >>>>>>> non
> > >>>>>>> > blocker/critical
> > >>>>>>> >
> > >>>>>>> > bug-fixes August 31, 2018.
> > >>>>>>> >
> > >>>>>>> > - Release date: September 15, 2018
> > >>>>>>> >
> > >>>>>>> > Thank Eric and Zian, I think Wangda has already answered your
> > >>>>>>> questions.
> > >>>>>>> >
> > >>>>>>> > Thanks
> > >>>>>>> > Sunil
> > >>>>>>> >
> > >>>>>>> >
> > >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <
> wheeleast@gmail.com
> > >
> > >>>>>>> wrote:
> > >>>>>>> >
> > >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for
> > >>>>>>> that.
> > >>>>>>> > >
> > >>>>>>> > > To concerns from Steve,
> > >>>>>>> > >
> > >>>>>>> > > It is a good idea to keep the door open to get important
> > changes
> > >>>>>>> /
> > >>>>>>> > > features in before cutoff. I would prefer to keep the
> proposed
> > >>>>>>> release
> > >>>>>>> > date
> > >>>>>>> > > to make sure things can happen earlier instead of last minute
> > >>>>>>> and we all
> > >>>>>>> > > know that releases are always get delayed :). I'm also fine
> if
> > >>>>>>> we want
> > >>>>>>> > get
> > >>>>>>> > > another several weeks time.
> > >>>>>>> > >
> > >>>>>>> > > Regarding of 3.3 release, I would suggest doing that before
> > >>>>>>> thanksgiving.
> > >>>>>>> > > Do you think is it good or too early / late?
> > >>>>>>> > >
> > >>>>>>> > > Eric,
> > >>>>>>> > >
> > >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can
> > >>>>>>> get merged
> > >>>>>>> > > in time, we probably not need the YARN-8220.
> > >>>>>>> > >
> > >>>>>>> > > Sunil,
> > >>>>>>> > >
> > >>>>>>> > > Could u update
> > >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
> > >>>>>>> > Roadmap
> > >>>>>>> > > with proposed plan as well? We can fill feature list first
> > >>>>>>> before getting
> > >>>>>>> > > consensus of time.
> > >>>>>>> > >
> > >>>>>>> > > Thanks,
> > >>>>>>> > > Wangda
> > >>>>>>> > >
> > >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> > >>>>>>> <fabbri@cloudera.com.invalid
> > >>>>>>> > >
> > >>>>>>> > > wrote:
> > >>>>>>> > >
> > >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> > >>>>>>> stevel@hortonworks.com>
> > >>>>>>> > >> wrote:
> > >>>>>>> > >>
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org
> > <mailto:
> > >>>>>>> > >> > sunilg@apache.org>> wrote:
> > >>>>>>> > >> >
> > >>>>>>> > >> > I would also would like to take this opportunity to come
> up
> > >>>>>>> with a
> > >>>>>>> > >> detailed
> > >>>>>>> > >> > plan.
> > >>>>>>> > >> >
> > >>>>>>> > >> > - Feature freeze date : all features should be merged by
> > >>>>>>> August 10,
> > >>>>>>> > >> 2018.
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > <snip>
> > >>>>>>> > >>
> > >>>>>>> > >> >
> > >>>>>>> > >> > Please let me know if I missed any features targeted to
> 3.2
> > >>>>>>> per this
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
> > >>>>>>> > >> >
> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake
> > Gen
> > >>>>>>> 2
> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> > >>>>>>> > >> >
> > >>>>>>> > >> > I don't think this is quite ready yet, I've been doing
> work
> > >>>>>>> on it, but
> > >>>>>>> > >> if
> > >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
> > >>>>>>> reviews on
> > >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> > >>>>>>> > >> >
> > >>>>>>> > >> > I've uprated that to a blocker feature; will review the
> S3 &
> > >>>>>>> S3Guard
> > >>>>>>> > >> JIRAs
> > >>>>>>> > >> > to see which of those are blocking. Then there are some
> > >>>>>>> pressing
> > >>>>>>> > "guave,
> > >>>>>>> > >> > java 9 prep"
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> I can help with this part if you like.
> > >>>>>>> > >>
> > >>>>>>> > >>
> > >>>>>>> > >>
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > timeline. I would like to volunteer myself as release
> > manager
> > >>>>>>> of 3.2.0
> > >>>>>>> > >> > release.
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> > well volunteered!
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> Yes, thank you for stepping up.
> > >>>>>>> > >>
> > >>>>>>> > >>
> > >>>>>>> > >> >
> > >>>>>>> > >> > I think this raises a good q: what timetable should we
> have
> > >>>>>>> for the
> > >>>>>>> > >> 3.2. &
> > >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having
> > the
> > >>>>>>> outline
> > >>>>>>> > >> time
> > >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's less
> > >>>>>>> concern about
> > >>>>>>> > >> > things not making the 3.2 dealine
> > >>>>>>> > >> >
> > >>>>>>> > >> > -Steve
> > >>>>>>> > >> >
> > >>>>>>> > >> >
> > >>>>>>> > >> Good idea to mitigate the short deadline.
> > >>>>>>> > >>
> > >>>>>>> > >> -AF
> > >>>>>>> > >>
> > >>>>>>> > >
> > >>>>>>> >
> > >>>>>>>
> > >>>>>>
> >
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Aaron Fabbri <fa...@cloudera.com.INVALID>.
Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
top-level pom.xml?
On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
> Hi All
>
> As mentioned in earlier mail, I have cut branch-3.2 and reset trunk to
> 3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary
> patches are pulled into branch-3.2.
>
> Thank You
> - Sunil
>
>
> On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
>
> > Hi All
> >
> > We are now down to the last Blocker and HADOOP-15407 is merged to trunk.
> > Thanks for the support.
> >
> > *Plan for RC*
> > 3.2 branch cut and reset trunk : *25th Tuesday*
> > RC0 for 3.2: *28th Friday*
> >
> > Thank You
> > Sunil
> >
> >
> > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
> >
> >> Hi All
> >>
> >> We are down to 3 Blockers and 4 Critical now. Thanks all of you for
> >> helping in this. I am following up on these tickets, once its closed we
> >> will cut the 3.2 branch.
> >>
> >> Thanks
> >> Sunil Govindan
> >>
> >>
> >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
> >>
> >>> Hi All,
> >>>
> >>> Inline with the original 3.2 communication proposal dated 17th July
> >>> 2018, I would like to provide more updates.
> >>>
> >>> We are approaching previously proposed code freeze date (September 14,
> >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
> existing
> >>> trunk to 3.3 if there are no issues.
> >>>
> >>> *Current Release Plan:*
> >>> Feature freeze date : all features to merge by September 7, 2018.
> >>> Code freeze date : blockers/critical only, no improvements and
> >>> blocker/critical bug-fixes September 14, 2018.
> >>> Release date: September 28, 2018
> >>>
> >>> If any critical/blocker tickets which are targeted to 3.2.0, we need to
> >>> backport to 3.2 post branch cut.
> >>>
> >>> Here's an updated 3.2.0 feature status:
> >>>
> >>> 1. Merged & Completed features:
> >>>
> >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> >>> workloads Initial cut.
> >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >>> Scheduler.
> >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> API
> >>> and CLI.
> >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.
> >>>
> >>> 2. Features close to finish:
> >>>
> >>> - (Steve) S3Guard Phase III. Close to commit.
> >>> - (Steve) S3a phase V. Close to commit.
> >>> - (Steve) Support Windows Azure Storage. Close to commit.
> >>>
> >>> 3. Tentative/Cancelled features for 3.2:
> >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> >>> ATSv2. Patch in progress.
> >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
> >>> be done before Aug 2018.
> >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >>> Challenging as more discussions are on-going.
> >>>
> >>> *Summary of 3.2.0 issues status:*
> >>> 19 Blocker and Critical issues [1] are open, I am following up with
> >>> owners to get status on each of them to get in by Code Freeze date.
> >>>
> >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> (Blocker,
> >>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
> ORDER
> >>> BY priority DESC
> >>>
> >>> Thanks,
> >>> Sunil
> >>>
> >>>
> >>>
> >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
> >>>
> >>>> Hi All,
> >>>>
> >>>> Inline with earlier communication dated 17th July 2018, I would like
> to
> >>>> provide some updates.
> >>>>
> >>>> We are approaching previously proposed code freeze date (Aug 31).
> >>>>
> >>>> One of the critical feature Node Attributes feature merge
> >>>> discussion/vote is ongoing. Also few other Blocker bugs need a bit
> more
> >>>> time. With regard to this, suggesting to push the feature/code freeze
> for 2
> >>>> more weeks to accommodate these jiras too.
> >>>>
> >>>> Proposing Updated changes in plan inline with this:
> >>>> Feature freeze date : all features to merge by September 7, 2018.
> >>>> Code freeze date : blockers/critical only, no improvements and
> >>>> blocker/critical bug-fixes September 14, 2018.
> >>>> Release date: September 28, 2018
> >>>>
> >>>> If any features in branch which are targeted to 3.2.0, please reply to
> >>>> this email thread.
> >>>>
> >>>> *Here's an updated 3.2.0 feature status:*
> >>>>
> >>>> 1. Merged & Completed features:
> >>>>
> >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> >>>> workloads Initial cut.
> >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >>>> Scheduler.
> >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> >>>> API and CLI.
> >>>>
> >>>> 2. Features close to finish:
> >>>>
> >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
> >>>> Ongoing.
> >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> >>>> ATSv2. Patch in progress.
> >>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
> works.
> >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> >>>> Storage. In progress.
> >>>>
> >>>> 3. Tentative features:
> >>>>
> >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging
> to
> >>>> be done before Aug 2018.
> >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >>>> Challenging as more discussions are on-going.
> >>>>
> >>>> *Summary of 3.2.0 issues status:*
> >>>>
> >>>> 26 Blocker and Critical issues [1] are open, I am following up with
> >>>> owners to get status on each of them to get in by Code Freeze date.
> >>>>
> >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> Version/s" =
> >>>> 3.2.0 ORDER BY priority DESC
> >>>>
> >>>> Thanks,
> >>>> Sunil
> >>>>
> >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
> >>>>
> >>>>> Hi All,
> >>>>>
> >>>>> Thanks for the feedbacks. Inline with earlier communication dated
> 17th
> >>>>> July 2018, I would like to provide some updates.
> >>>>>
> >>>>> We are approaching previously proposed feature freeze date (Aug 21,
> >>>>> about 7 days from today).
> >>>>> If any features in branch which are targeted to 3.2.0, please reply
> to
> >>>>> this email thread.
> >>>>> Steve has mentioned about the s3 features which will come close to
> >>>>> Code Freeze Date (Aug 31st).
> >>>>>
> >>>>> *Here's an updated 3.2.0 feature status:*
> >>>>>
> >>>>> 1. Merged & Completed features:
> >>>>>
> >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> >>>>> workloads Initial cut.
> >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>>>>
> >>>>> 2. Features close to finish:
> >>>>>
> >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
> >>>>> patches are all in, only one last
> >>>>> patch is in review state.
> >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >>>>> Scheduler. Close to commit.
> >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> >>>>> API and CLI. 2 patches are pending
> >>>>> which will be closed by Feature freeze date.
> >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> >>>>> ATSv2. Patch in progress.
> >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
> >>>>> works.
> >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> >>>>> Storage. In progress.
> >>>>>
> >>>>> 3. Tentative features:
> >>>>>
> >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging
> >>>>> to be done before Aug 2018.
> >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >>>>> Challenging as more discussions are on-going.
> >>>>>
> >>>>> *Summary of 3.2.0 issues status:*
> >>>>>
> >>>>> 39 Blocker and Critical issues [1] are open, I am checking with
> owners
> >>>>> to get status on each of them to get in by Code Freeze date.
> >>>>>
> >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> Version/s" =
> >>>>> 3.2.0 ORDER BY priority DESC
> >>>>>
> >>>>> Thanks,
> >>>>> Sunil
> >>>>>
> >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
> >>>>>
> >>>>>> Thanks Subru for the thoughts.
> >>>>>> One of the main reason for a major release is to push out critical
> >>>>>> features with a faster cadence to the users. If we are pulling more
> and
> >>>>>> more different types of features to a minor release, that branch
> will
> >>>>>> become more destabilized and it may be tough to say that 3.1.2 is
> stable
> >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize features
> in
> >>>>>> subsequent minor release.
> >>>>>> For few companies, it makes sense to push out these new features
> >>>>>> faster to make a reach to the users. Adding to the point to the
> backporting
> >>>>>> issues, I agree that its a pain and we can workaround that with
> some git
> >>>>>> scripts. If we can make such scripts available to committers,
> backport will
> >>>>>> be seem-less across branches and we can achieve the faster release
> cadence
> >>>>>> also.
> >>>>>>
> >>>>>> Thoughts?
> >>>>>>
> >>>>>> - Sunil
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Thanks Sunil for volunteering to lead the release effort. I am
> >>>>>>> generally
> >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
> >>>>>>> already
> >>>>>>> have too many branches to be maintained. I already see many commits
> >>>>>>> are in
> >>>>>>> different branches with no apparent rationale, for e.g: 3.1 has
> >>>>>>> commits
> >>>>>>> which are absent in 3.0 etc.
> >>>>>>>
> >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
> production
> >>>>>>> setting so the cost of adding features should be minimal.
> >>>>>>>
> >>>>>>> Thoughts?
> >>>>>>>
> >>>>>>> -Subru
> >>>>>>>
> >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org>
> wrote:
> >>>>>>>
> >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> >>>>>>> >
> >>>>>>> > Yes, important changes and features are much needed, hence we
> will
> >>>>>>> be
> >>>>>>> > keeping the door open for them as possible. Also considering few
> >>>>>>> more
> >>>>>>> > offline requests from other folks, I think extending the
> timeframe
> >>>>>>> by
> >>>>>>> > couple of weeks makes sense (including a second RC buffer) and
> >>>>>>> this should
> >>>>>>> > ideally help us to ship this by September itself.
> >>>>>>> >
> >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
> >>>>>>> >
> >>>>>>> > - Feature freeze date : all features to merge by August 21, 2018.
> >>>>>>> >
> >>>>>>> > - Code freeze date : blockers/critical only, no improvements and
> >>>>>>> non
> >>>>>>> > blocker/critical
> >>>>>>> >
> >>>>>>> > bug-fixes August 31, 2018.
> >>>>>>> >
> >>>>>>> > - Release date: September 15, 2018
> >>>>>>> >
> >>>>>>> > Thank Eric and Zian, I think Wangda has already answered your
> >>>>>>> questions.
> >>>>>>> >
> >>>>>>> > Thanks
> >>>>>>> > Sunil
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wheeleast@gmail.com
> >
> >>>>>>> wrote:
> >>>>>>> >
> >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for
> >>>>>>> that.
> >>>>>>> > >
> >>>>>>> > > To concerns from Steve,
> >>>>>>> > >
> >>>>>>> > > It is a good idea to keep the door open to get important
> changes
> >>>>>>> /
> >>>>>>> > > features in before cutoff. I would prefer to keep the proposed
> >>>>>>> release
> >>>>>>> > date
> >>>>>>> > > to make sure things can happen earlier instead of last minute
> >>>>>>> and we all
> >>>>>>> > > know that releases are always get delayed :). I'm also fine if
> >>>>>>> we want
> >>>>>>> > get
> >>>>>>> > > another several weeks time.
> >>>>>>> > >
> >>>>>>> > > Regarding of 3.3 release, I would suggest doing that before
> >>>>>>> thanksgiving.
> >>>>>>> > > Do you think is it good or too early / late?
> >>>>>>> > >
> >>>>>>> > > Eric,
> >>>>>>> > >
> >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can
> >>>>>>> get merged
> >>>>>>> > > in time, we probably not need the YARN-8220.
> >>>>>>> > >
> >>>>>>> > > Sunil,
> >>>>>>> > >
> >>>>>>> > > Could u update
> >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
> >>>>>>> > Roadmap
> >>>>>>> > > with proposed plan as well? We can fill feature list first
> >>>>>>> before getting
> >>>>>>> > > consensus of time.
> >>>>>>> > >
> >>>>>>> > > Thanks,
> >>>>>>> > > Wangda
> >>>>>>> > >
> >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> >>>>>>> <fabbri@cloudera.com.invalid
> >>>>>>> > >
> >>>>>>> > > wrote:
> >>>>>>> > >
> >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> >>>>>>> stevel@hortonworks.com>
> >>>>>>> > >> wrote:
> >>>>>>> > >>
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org
> <mailto:
> >>>>>>> > >> > sunilg@apache.org>> wrote:
> >>>>>>> > >> >
> >>>>>>> > >> > I would also would like to take this opportunity to come up
> >>>>>>> with a
> >>>>>>> > >> detailed
> >>>>>>> > >> > plan.
> >>>>>>> > >> >
> >>>>>>> > >> > - Feature freeze date : all features should be merged by
> >>>>>>> August 10,
> >>>>>>> > >> 2018.
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > <snip>
> >>>>>>> > >>
> >>>>>>> > >> >
> >>>>>>> > >> > Please let me know if I missed any features targeted to 3.2
> >>>>>>> per this
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
> >>>>>>> > >> >
> >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake
> Gen
> >>>>>>> 2
> >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> >>>>>>> > >> >
> >>>>>>> > >> > I don't think this is quite ready yet, I've been doing work
> >>>>>>> on it, but
> >>>>>>> > >> if
> >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
> >>>>>>> reviews on
> >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> >>>>>>> > >> >
> >>>>>>> > >> > I've uprated that to a blocker feature; will review the S3 &
> >>>>>>> S3Guard
> >>>>>>> > >> JIRAs
> >>>>>>> > >> > to see which of those are blocking. Then there are some
> >>>>>>> pressing
> >>>>>>> > "guave,
> >>>>>>> > >> > java 9 prep"
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> I can help with this part if you like.
> >>>>>>> > >>
> >>>>>>> > >>
> >>>>>>> > >>
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > timeline. I would like to volunteer myself as release
> manager
> >>>>>>> of 3.2.0
> >>>>>>> > >> > release.
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > well volunteered!
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> Yes, thank you for stepping up.
> >>>>>>> > >>
> >>>>>>> > >>
> >>>>>>> > >> >
> >>>>>>> > >> > I think this raises a good q: what timetable should we have
> >>>>>>> for the
> >>>>>>> > >> 3.2. &
> >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having
> the
> >>>>>>> outline
> >>>>>>> > >> time
> >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's less
> >>>>>>> concern about
> >>>>>>> > >> > things not making the 3.2 dealine
> >>>>>>> > >> >
> >>>>>>> > >> > -Steve
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> Good idea to mitigate the short deadline.
> >>>>>>> > >>
> >>>>>>> > >> -AF
> >>>>>>> > >>
> >>>>>>> > >
> >>>>>>> >
> >>>>>>>
> >>>>>>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Aaron Fabbri <fa...@cloudera.com.INVALID>.
Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
top-level pom.xml?
On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
> Hi All
>
> As mentioned in earlier mail, I have cut branch-3.2 and reset trunk to
> 3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary
> patches are pulled into branch-3.2.
>
> Thank You
> - Sunil
>
>
> On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
>
> > Hi All
> >
> > We are now down to the last Blocker and HADOOP-15407 is merged to trunk.
> > Thanks for the support.
> >
> > *Plan for RC*
> > 3.2 branch cut and reset trunk : *25th Tuesday*
> > RC0 for 3.2: *28th Friday*
> >
> > Thank You
> > Sunil
> >
> >
> > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
> >
> >> Hi All
> >>
> >> We are down to 3 Blockers and 4 Critical now. Thanks all of you for
> >> helping in this. I am following up on these tickets, once its closed we
> >> will cut the 3.2 branch.
> >>
> >> Thanks
> >> Sunil Govindan
> >>
> >>
> >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
> >>
> >>> Hi All,
> >>>
> >>> Inline with the original 3.2 communication proposal dated 17th July
> >>> 2018, I would like to provide more updates.
> >>>
> >>> We are approaching previously proposed code freeze date (September 14,
> >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
> existing
> >>> trunk to 3.3 if there are no issues.
> >>>
> >>> *Current Release Plan:*
> >>> Feature freeze date : all features to merge by September 7, 2018.
> >>> Code freeze date : blockers/critical only, no improvements and
> >>> blocker/critical bug-fixes September 14, 2018.
> >>> Release date: September 28, 2018
> >>>
> >>> If any critical/blocker tickets which are targeted to 3.2.0, we need to
> >>> backport to 3.2 post branch cut.
> >>>
> >>> Here's an updated 3.2.0 feature status:
> >>>
> >>> 1. Merged & Completed features:
> >>>
> >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> >>> workloads Initial cut.
> >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >>> Scheduler.
> >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> API
> >>> and CLI.
> >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.
> >>>
> >>> 2. Features close to finish:
> >>>
> >>> - (Steve) S3Guard Phase III. Close to commit.
> >>> - (Steve) S3a phase V. Close to commit.
> >>> - (Steve) Support Windows Azure Storage. Close to commit.
> >>>
> >>> 3. Tentative/Cancelled features for 3.2:
> >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> >>> ATSv2. Patch in progress.
> >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
> >>> be done before Aug 2018.
> >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >>> Challenging as more discussions are on-going.
> >>>
> >>> *Summary of 3.2.0 issues status:*
> >>> 19 Blocker and Critical issues [1] are open, I am following up with
> >>> owners to get status on each of them to get in by Code Freeze date.
> >>>
> >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> (Blocker,
> >>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
> ORDER
> >>> BY priority DESC
> >>>
> >>> Thanks,
> >>> Sunil
> >>>
> >>>
> >>>
> >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
> >>>
> >>>> Hi All,
> >>>>
> >>>> Inline with earlier communication dated 17th July 2018, I would like
> to
> >>>> provide some updates.
> >>>>
> >>>> We are approaching previously proposed code freeze date (Aug 31).
> >>>>
> >>>> One of the critical feature Node Attributes feature merge
> >>>> discussion/vote is ongoing. Also few other Blocker bugs need a bit
> more
> >>>> time. With regard to this, suggesting to push the feature/code freeze
> for 2
> >>>> more weeks to accommodate these jiras too.
> >>>>
> >>>> Proposing Updated changes in plan inline with this:
> >>>> Feature freeze date : all features to merge by September 7, 2018.
> >>>> Code freeze date : blockers/critical only, no improvements and
> >>>> blocker/critical bug-fixes September 14, 2018.
> >>>> Release date: September 28, 2018
> >>>>
> >>>> If any features in branch which are targeted to 3.2.0, please reply to
> >>>> this email thread.
> >>>>
> >>>> *Here's an updated 3.2.0 feature status:*
> >>>>
> >>>> 1. Merged & Completed features:
> >>>>
> >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> >>>> workloads Initial cut.
> >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >>>> Scheduler.
> >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> >>>> API and CLI.
> >>>>
> >>>> 2. Features close to finish:
> >>>>
> >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
> >>>> Ongoing.
> >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> >>>> ATSv2. Patch in progress.
> >>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
> works.
> >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> >>>> Storage. In progress.
> >>>>
> >>>> 3. Tentative features:
> >>>>
> >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging
> to
> >>>> be done before Aug 2018.
> >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >>>> Challenging as more discussions are on-going.
> >>>>
> >>>> *Summary of 3.2.0 issues status:*
> >>>>
> >>>> 26 Blocker and Critical issues [1] are open, I am following up with
> >>>> owners to get status on each of them to get in by Code Freeze date.
> >>>>
> >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> Version/s" =
> >>>> 3.2.0 ORDER BY priority DESC
> >>>>
> >>>> Thanks,
> >>>> Sunil
> >>>>
> >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
> >>>>
> >>>>> Hi All,
> >>>>>
> >>>>> Thanks for the feedbacks. Inline with earlier communication dated
> 17th
> >>>>> July 2018, I would like to provide some updates.
> >>>>>
> >>>>> We are approaching previously proposed feature freeze date (Aug 21,
> >>>>> about 7 days from today).
> >>>>> If any features in branch which are targeted to 3.2.0, please reply
> to
> >>>>> this email thread.
> >>>>> Steve has mentioned about the s3 features which will come close to
> >>>>> Code Freeze Date (Aug 31st).
> >>>>>
> >>>>> *Here's an updated 3.2.0 feature status:*
> >>>>>
> >>>>> 1. Merged & Completed features:
> >>>>>
> >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> >>>>> workloads Initial cut.
> >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>>>>
> >>>>> 2. Features close to finish:
> >>>>>
> >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
> >>>>> patches are all in, only one last
> >>>>> patch is in review state.
> >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >>>>> Scheduler. Close to commit.
> >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> >>>>> API and CLI. 2 patches are pending
> >>>>> which will be closed by Feature freeze date.
> >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> >>>>> ATSv2. Patch in progress.
> >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
> >>>>> works.
> >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> >>>>> Storage. In progress.
> >>>>>
> >>>>> 3. Tentative features:
> >>>>>
> >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging
> >>>>> to be done before Aug 2018.
> >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >>>>> Challenging as more discussions are on-going.
> >>>>>
> >>>>> *Summary of 3.2.0 issues status:*
> >>>>>
> >>>>> 39 Blocker and Critical issues [1] are open, I am checking with
> owners
> >>>>> to get status on each of them to get in by Code Freeze date.
> >>>>>
> >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> Version/s" =
> >>>>> 3.2.0 ORDER BY priority DESC
> >>>>>
> >>>>> Thanks,
> >>>>> Sunil
> >>>>>
> >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
> >>>>>
> >>>>>> Thanks Subru for the thoughts.
> >>>>>> One of the main reason for a major release is to push out critical
> >>>>>> features with a faster cadence to the users. If we are pulling more
> and
> >>>>>> more different types of features to a minor release, that branch
> will
> >>>>>> become more destabilized and it may be tough to say that 3.1.2 is
> stable
> >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize features
> in
> >>>>>> subsequent minor release.
> >>>>>> For few companies, it makes sense to push out these new features
> >>>>>> faster to make a reach to the users. Adding to the point to the
> backporting
> >>>>>> issues, I agree that its a pain and we can workaround that with
> some git
> >>>>>> scripts. If we can make such scripts available to committers,
> backport will
> >>>>>> be seem-less across branches and we can achieve the faster release
> cadence
> >>>>>> also.
> >>>>>>
> >>>>>> Thoughts?
> >>>>>>
> >>>>>> - Sunil
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Thanks Sunil for volunteering to lead the release effort. I am
> >>>>>>> generally
> >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
> >>>>>>> already
> >>>>>>> have too many branches to be maintained. I already see many commits
> >>>>>>> are in
> >>>>>>> different branches with no apparent rationale, for e.g: 3.1 has
> >>>>>>> commits
> >>>>>>> which are absent in 3.0 etc.
> >>>>>>>
> >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
> production
> >>>>>>> setting so the cost of adding features should be minimal.
> >>>>>>>
> >>>>>>> Thoughts?
> >>>>>>>
> >>>>>>> -Subru
> >>>>>>>
> >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org>
> wrote:
> >>>>>>>
> >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> >>>>>>> >
> >>>>>>> > Yes, important changes and features are much needed, hence we
> will
> >>>>>>> be
> >>>>>>> > keeping the door open for them as possible. Also considering few
> >>>>>>> more
> >>>>>>> > offline requests from other folks, I think extending the
> timeframe
> >>>>>>> by
> >>>>>>> > couple of weeks makes sense (including a second RC buffer) and
> >>>>>>> this should
> >>>>>>> > ideally help us to ship this by September itself.
> >>>>>>> >
> >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
> >>>>>>> >
> >>>>>>> > - Feature freeze date : all features to merge by August 21, 2018.
> >>>>>>> >
> >>>>>>> > - Code freeze date : blockers/critical only, no improvements and
> >>>>>>> non
> >>>>>>> > blocker/critical
> >>>>>>> >
> >>>>>>> > bug-fixes August 31, 2018.
> >>>>>>> >
> >>>>>>> > - Release date: September 15, 2018
> >>>>>>> >
> >>>>>>> > Thank Eric and Zian, I think Wangda has already answered your
> >>>>>>> questions.
> >>>>>>> >
> >>>>>>> > Thanks
> >>>>>>> > Sunil
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wheeleast@gmail.com
> >
> >>>>>>> wrote:
> >>>>>>> >
> >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for
> >>>>>>> that.
> >>>>>>> > >
> >>>>>>> > > To concerns from Steve,
> >>>>>>> > >
> >>>>>>> > > It is a good idea to keep the door open to get important
> changes
> >>>>>>> /
> >>>>>>> > > features in before cutoff. I would prefer to keep the proposed
> >>>>>>> release
> >>>>>>> > date
> >>>>>>> > > to make sure things can happen earlier instead of last minute
> >>>>>>> and we all
> >>>>>>> > > know that releases are always get delayed :). I'm also fine if
> >>>>>>> we want
> >>>>>>> > get
> >>>>>>> > > another several weeks time.
> >>>>>>> > >
> >>>>>>> > > Regarding of 3.3 release, I would suggest doing that before
> >>>>>>> thanksgiving.
> >>>>>>> > > Do you think is it good or too early / late?
> >>>>>>> > >
> >>>>>>> > > Eric,
> >>>>>>> > >
> >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can
> >>>>>>> get merged
> >>>>>>> > > in time, we probably not need the YARN-8220.
> >>>>>>> > >
> >>>>>>> > > Sunil,
> >>>>>>> > >
> >>>>>>> > > Could u update
> >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
> >>>>>>> > Roadmap
> >>>>>>> > > with proposed plan as well? We can fill feature list first
> >>>>>>> before getting
> >>>>>>> > > consensus of time.
> >>>>>>> > >
> >>>>>>> > > Thanks,
> >>>>>>> > > Wangda
> >>>>>>> > >
> >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> >>>>>>> <fabbri@cloudera.com.invalid
> >>>>>>> > >
> >>>>>>> > > wrote:
> >>>>>>> > >
> >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> >>>>>>> stevel@hortonworks.com>
> >>>>>>> > >> wrote:
> >>>>>>> > >>
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org
> <mailto:
> >>>>>>> > >> > sunilg@apache.org>> wrote:
> >>>>>>> > >> >
> >>>>>>> > >> > I would also would like to take this opportunity to come up
> >>>>>>> with a
> >>>>>>> > >> detailed
> >>>>>>> > >> > plan.
> >>>>>>> > >> >
> >>>>>>> > >> > - Feature freeze date : all features should be merged by
> >>>>>>> August 10,
> >>>>>>> > >> 2018.
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > <snip>
> >>>>>>> > >>
> >>>>>>> > >> >
> >>>>>>> > >> > Please let me know if I missed any features targeted to 3.2
> >>>>>>> per this
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
> >>>>>>> > >> >
> >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake
> Gen
> >>>>>>> 2
> >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> >>>>>>> > >> >
> >>>>>>> > >> > I don't think this is quite ready yet, I've been doing work
> >>>>>>> on it, but
> >>>>>>> > >> if
> >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
> >>>>>>> reviews on
> >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> >>>>>>> > >> >
> >>>>>>> > >> > I've uprated that to a blocker feature; will review the S3 &
> >>>>>>> S3Guard
> >>>>>>> > >> JIRAs
> >>>>>>> > >> > to see which of those are blocking. Then there are some
> >>>>>>> pressing
> >>>>>>> > "guave,
> >>>>>>> > >> > java 9 prep"
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> I can help with this part if you like.
> >>>>>>> > >>
> >>>>>>> > >>
> >>>>>>> > >>
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > timeline. I would like to volunteer myself as release
> manager
> >>>>>>> of 3.2.0
> >>>>>>> > >> > release.
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > well volunteered!
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> Yes, thank you for stepping up.
> >>>>>>> > >>
> >>>>>>> > >>
> >>>>>>> > >> >
> >>>>>>> > >> > I think this raises a good q: what timetable should we have
> >>>>>>> for the
> >>>>>>> > >> 3.2. &
> >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having
> the
> >>>>>>> outline
> >>>>>>> > >> time
> >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's less
> >>>>>>> concern about
> >>>>>>> > >> > things not making the 3.2 dealine
> >>>>>>> > >> >
> >>>>>>> > >> > -Steve
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> Good idea to mitigate the short deadline.
> >>>>>>> > >>
> >>>>>>> > >> -AF
> >>>>>>> > >>
> >>>>>>> > >
> >>>>>>> >
> >>>>>>>
> >>>>>>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Aaron Fabbri <fa...@cloudera.com.INVALID>.
Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
top-level pom.xml?
On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
> Hi All
>
> As mentioned in earlier mail, I have cut branch-3.2 and reset trunk to
> 3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary
> patches are pulled into branch-3.2.
>
> Thank You
> - Sunil
>
>
> On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
>
> > Hi All
> >
> > We are now down to the last Blocker and HADOOP-15407 is merged to trunk.
> > Thanks for the support.
> >
> > *Plan for RC*
> > 3.2 branch cut and reset trunk : *25th Tuesday*
> > RC0 for 3.2: *28th Friday*
> >
> > Thank You
> > Sunil
> >
> >
> > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
> >
> >> Hi All
> >>
> >> We are down to 3 Blockers and 4 Critical now. Thanks all of you for
> >> helping in this. I am following up on these tickets, once its closed we
> >> will cut the 3.2 branch.
> >>
> >> Thanks
> >> Sunil Govindan
> >>
> >>
> >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
> >>
> >>> Hi All,
> >>>
> >>> Inline with the original 3.2 communication proposal dated 17th July
> >>> 2018, I would like to provide more updates.
> >>>
> >>> We are approaching previously proposed code freeze date (September 14,
> >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
> existing
> >>> trunk to 3.3 if there are no issues.
> >>>
> >>> *Current Release Plan:*
> >>> Feature freeze date : all features to merge by September 7, 2018.
> >>> Code freeze date : blockers/critical only, no improvements and
> >>> blocker/critical bug-fixes September 14, 2018.
> >>> Release date: September 28, 2018
> >>>
> >>> If any critical/blocker tickets which are targeted to 3.2.0, we need to
> >>> backport to 3.2 post branch cut.
> >>>
> >>> Here's an updated 3.2.0 feature status:
> >>>
> >>> 1. Merged & Completed features:
> >>>
> >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> >>> workloads Initial cut.
> >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >>> Scheduler.
> >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> API
> >>> and CLI.
> >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.
> >>>
> >>> 2. Features close to finish:
> >>>
> >>> - (Steve) S3Guard Phase III. Close to commit.
> >>> - (Steve) S3a phase V. Close to commit.
> >>> - (Steve) Support Windows Azure Storage. Close to commit.
> >>>
> >>> 3. Tentative/Cancelled features for 3.2:
> >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> >>> ATSv2. Patch in progress.
> >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
> >>> be done before Aug 2018.
> >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >>> Challenging as more discussions are on-going.
> >>>
> >>> *Summary of 3.2.0 issues status:*
> >>> 19 Blocker and Critical issues [1] are open, I am following up with
> >>> owners to get status on each of them to get in by Code Freeze date.
> >>>
> >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> (Blocker,
> >>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
> ORDER
> >>> BY priority DESC
> >>>
> >>> Thanks,
> >>> Sunil
> >>>
> >>>
> >>>
> >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
> >>>
> >>>> Hi All,
> >>>>
> >>>> Inline with earlier communication dated 17th July 2018, I would like
> to
> >>>> provide some updates.
> >>>>
> >>>> We are approaching previously proposed code freeze date (Aug 31).
> >>>>
> >>>> One of the critical feature Node Attributes feature merge
> >>>> discussion/vote is ongoing. Also few other Blocker bugs need a bit
> more
> >>>> time. With regard to this, suggesting to push the feature/code freeze
> for 2
> >>>> more weeks to accommodate these jiras too.
> >>>>
> >>>> Proposing Updated changes in plan inline with this:
> >>>> Feature freeze date : all features to merge by September 7, 2018.
> >>>> Code freeze date : blockers/critical only, no improvements and
> >>>> blocker/critical bug-fixes September 14, 2018.
> >>>> Release date: September 28, 2018
> >>>>
> >>>> If any features in branch which are targeted to 3.2.0, please reply to
> >>>> this email thread.
> >>>>
> >>>> *Here's an updated 3.2.0 feature status:*
> >>>>
> >>>> 1. Merged & Completed features:
> >>>>
> >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> >>>> workloads Initial cut.
> >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >>>> Scheduler.
> >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> >>>> API and CLI.
> >>>>
> >>>> 2. Features close to finish:
> >>>>
> >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
> >>>> Ongoing.
> >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> >>>> ATSv2. Patch in progress.
> >>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
> works.
> >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> >>>> Storage. In progress.
> >>>>
> >>>> 3. Tentative features:
> >>>>
> >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging
> to
> >>>> be done before Aug 2018.
> >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >>>> Challenging as more discussions are on-going.
> >>>>
> >>>> *Summary of 3.2.0 issues status:*
> >>>>
> >>>> 26 Blocker and Critical issues [1] are open, I am following up with
> >>>> owners to get status on each of them to get in by Code Freeze date.
> >>>>
> >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> Version/s" =
> >>>> 3.2.0 ORDER BY priority DESC
> >>>>
> >>>> Thanks,
> >>>> Sunil
> >>>>
> >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
> >>>>
> >>>>> Hi All,
> >>>>>
> >>>>> Thanks for the feedbacks. Inline with earlier communication dated
> 17th
> >>>>> July 2018, I would like to provide some updates.
> >>>>>
> >>>>> We are approaching previously proposed feature freeze date (Aug 21,
> >>>>> about 7 days from today).
> >>>>> If any features in branch which are targeted to 3.2.0, please reply
> to
> >>>>> this email thread.
> >>>>> Steve has mentioned about the s3 features which will come close to
> >>>>> Code Freeze Date (Aug 31st).
> >>>>>
> >>>>> *Here's an updated 3.2.0 feature status:*
> >>>>>
> >>>>> 1. Merged & Completed features:
> >>>>>
> >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> >>>>> workloads Initial cut.
> >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>>>>
> >>>>> 2. Features close to finish:
> >>>>>
> >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
> >>>>> patches are all in, only one last
> >>>>> patch is in review state.
> >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >>>>> Scheduler. Close to commit.
> >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> >>>>> API and CLI. 2 patches are pending
> >>>>> which will be closed by Feature freeze date.
> >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> >>>>> ATSv2. Patch in progress.
> >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
> >>>>> works.
> >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> >>>>> Storage. In progress.
> >>>>>
> >>>>> 3. Tentative features:
> >>>>>
> >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging
> >>>>> to be done before Aug 2018.
> >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >>>>> Challenging as more discussions are on-going.
> >>>>>
> >>>>> *Summary of 3.2.0 issues status:*
> >>>>>
> >>>>> 39 Blocker and Critical issues [1] are open, I am checking with
> owners
> >>>>> to get status on each of them to get in by Code Freeze date.
> >>>>>
> >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> Version/s" =
> >>>>> 3.2.0 ORDER BY priority DESC
> >>>>>
> >>>>> Thanks,
> >>>>> Sunil
> >>>>>
> >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
> >>>>>
> >>>>>> Thanks Subru for the thoughts.
> >>>>>> One of the main reason for a major release is to push out critical
> >>>>>> features with a faster cadence to the users. If we are pulling more
> and
> >>>>>> more different types of features to a minor release, that branch
> will
> >>>>>> become more destabilized and it may be tough to say that 3.1.2 is
> stable
> >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize features
> in
> >>>>>> subsequent minor release.
> >>>>>> For few companies, it makes sense to push out these new features
> >>>>>> faster to make a reach to the users. Adding to the point to the
> backporting
> >>>>>> issues, I agree that its a pain and we can workaround that with
> some git
> >>>>>> scripts. If we can make such scripts available to committers,
> backport will
> >>>>>> be seem-less across branches and we can achieve the faster release
> cadence
> >>>>>> also.
> >>>>>>
> >>>>>> Thoughts?
> >>>>>>
> >>>>>> - Sunil
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Thanks Sunil for volunteering to lead the release effort. I am
> >>>>>>> generally
> >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
> >>>>>>> already
> >>>>>>> have too many branches to be maintained. I already see many commits
> >>>>>>> are in
> >>>>>>> different branches with no apparent rationale, for e.g: 3.1 has
> >>>>>>> commits
> >>>>>>> which are absent in 3.0 etc.
> >>>>>>>
> >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
> production
> >>>>>>> setting so the cost of adding features should be minimal.
> >>>>>>>
> >>>>>>> Thoughts?
> >>>>>>>
> >>>>>>> -Subru
> >>>>>>>
> >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org>
> wrote:
> >>>>>>>
> >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> >>>>>>> >
> >>>>>>> > Yes, important changes and features are much needed, hence we
> will
> >>>>>>> be
> >>>>>>> > keeping the door open for them as possible. Also considering few
> >>>>>>> more
> >>>>>>> > offline requests from other folks, I think extending the
> timeframe
> >>>>>>> by
> >>>>>>> > couple of weeks makes sense (including a second RC buffer) and
> >>>>>>> this should
> >>>>>>> > ideally help us to ship this by September itself.
> >>>>>>> >
> >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
> >>>>>>> >
> >>>>>>> > - Feature freeze date : all features to merge by August 21, 2018.
> >>>>>>> >
> >>>>>>> > - Code freeze date : blockers/critical only, no improvements and
> >>>>>>> non
> >>>>>>> > blocker/critical
> >>>>>>> >
> >>>>>>> > bug-fixes August 31, 2018.
> >>>>>>> >
> >>>>>>> > - Release date: September 15, 2018
> >>>>>>> >
> >>>>>>> > Thank Eric and Zian, I think Wangda has already answered your
> >>>>>>> questions.
> >>>>>>> >
> >>>>>>> > Thanks
> >>>>>>> > Sunil
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wheeleast@gmail.com
> >
> >>>>>>> wrote:
> >>>>>>> >
> >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for
> >>>>>>> that.
> >>>>>>> > >
> >>>>>>> > > To concerns from Steve,
> >>>>>>> > >
> >>>>>>> > > It is a good idea to keep the door open to get important
> changes
> >>>>>>> /
> >>>>>>> > > features in before cutoff. I would prefer to keep the proposed
> >>>>>>> release
> >>>>>>> > date
> >>>>>>> > > to make sure things can happen earlier instead of last minute
> >>>>>>> and we all
> >>>>>>> > > know that releases are always get delayed :). I'm also fine if
> >>>>>>> we want
> >>>>>>> > get
> >>>>>>> > > another several weeks time.
> >>>>>>> > >
> >>>>>>> > > Regarding of 3.3 release, I would suggest doing that before
> >>>>>>> thanksgiving.
> >>>>>>> > > Do you think is it good or too early / late?
> >>>>>>> > >
> >>>>>>> > > Eric,
> >>>>>>> > >
> >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can
> >>>>>>> get merged
> >>>>>>> > > in time, we probably not need the YARN-8220.
> >>>>>>> > >
> >>>>>>> > > Sunil,
> >>>>>>> > >
> >>>>>>> > > Could u update
> >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
> >>>>>>> > Roadmap
> >>>>>>> > > with proposed plan as well? We can fill feature list first
> >>>>>>> before getting
> >>>>>>> > > consensus of time.
> >>>>>>> > >
> >>>>>>> > > Thanks,
> >>>>>>> > > Wangda
> >>>>>>> > >
> >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> >>>>>>> <fabbri@cloudera.com.invalid
> >>>>>>> > >
> >>>>>>> > > wrote:
> >>>>>>> > >
> >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> >>>>>>> stevel@hortonworks.com>
> >>>>>>> > >> wrote:
> >>>>>>> > >>
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org
> <mailto:
> >>>>>>> > >> > sunilg@apache.org>> wrote:
> >>>>>>> > >> >
> >>>>>>> > >> > I would also would like to take this opportunity to come up
> >>>>>>> with a
> >>>>>>> > >> detailed
> >>>>>>> > >> > plan.
> >>>>>>> > >> >
> >>>>>>> > >> > - Feature freeze date : all features should be merged by
> >>>>>>> August 10,
> >>>>>>> > >> 2018.
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > <snip>
> >>>>>>> > >>
> >>>>>>> > >> >
> >>>>>>> > >> > Please let me know if I missed any features targeted to 3.2
> >>>>>>> per this
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
> >>>>>>> > >> >
> >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake
> Gen
> >>>>>>> 2
> >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> >>>>>>> > >> >
> >>>>>>> > >> > I don't think this is quite ready yet, I've been doing work
> >>>>>>> on it, but
> >>>>>>> > >> if
> >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
> >>>>>>> reviews on
> >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> >>>>>>> > >> >
> >>>>>>> > >> > I've uprated that to a blocker feature; will review the S3 &
> >>>>>>> S3Guard
> >>>>>>> > >> JIRAs
> >>>>>>> > >> > to see which of those are blocking. Then there are some
> >>>>>>> pressing
> >>>>>>> > "guave,
> >>>>>>> > >> > java 9 prep"
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> I can help with this part if you like.
> >>>>>>> > >>
> >>>>>>> > >>
> >>>>>>> > >>
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > timeline. I would like to volunteer myself as release
> manager
> >>>>>>> of 3.2.0
> >>>>>>> > >> > release.
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > well volunteered!
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> Yes, thank you for stepping up.
> >>>>>>> > >>
> >>>>>>> > >>
> >>>>>>> > >> >
> >>>>>>> > >> > I think this raises a good q: what timetable should we have
> >>>>>>> for the
> >>>>>>> > >> 3.2. &
> >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having
> the
> >>>>>>> outline
> >>>>>>> > >> time
> >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's less
> >>>>>>> concern about
> >>>>>>> > >> > things not making the 3.2 dealine
> >>>>>>> > >> >
> >>>>>>> > >> > -Steve
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> Good idea to mitigate the short deadline.
> >>>>>>> > >>
> >>>>>>> > >> -AF
> >>>>>>> > >>
> >>>>>>> > >
> >>>>>>> >
> >>>>>>>
> >>>>>>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Aaron Fabbri <fa...@cloudera.com.INVALID>.
Trunk is not building for me.. Did you miss a 3.2.0-SNAPSHOT in the
top-level pom.xml?
On Tue, Oct 2, 2018 at 10:16 AM Sunil G <su...@apache.org> wrote:
> Hi All
>
> As mentioned in earlier mail, I have cut branch-3.2 and reset trunk to
> 3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary
> patches are pulled into branch-3.2.
>
> Thank You
> - Sunil
>
>
> On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
>
> > Hi All
> >
> > We are now down to the last Blocker and HADOOP-15407 is merged to trunk.
> > Thanks for the support.
> >
> > *Plan for RC*
> > 3.2 branch cut and reset trunk : *25th Tuesday*
> > RC0 for 3.2: *28th Friday*
> >
> > Thank You
> > Sunil
> >
> >
> > On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
> >
> >> Hi All
> >>
> >> We are down to 3 Blockers and 4 Critical now. Thanks all of you for
> >> helping in this. I am following up on these tickets, once its closed we
> >> will cut the 3.2 branch.
> >>
> >> Thanks
> >> Sunil Govindan
> >>
> >>
> >> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
> >>
> >>> Hi All,
> >>>
> >>> Inline with the original 3.2 communication proposal dated 17th July
> >>> 2018, I would like to provide more updates.
> >>>
> >>> We are approaching previously proposed code freeze date (September 14,
> >>> 2018). So I would like to cut 3.2 branch on 17th Sept and point
> existing
> >>> trunk to 3.3 if there are no issues.
> >>>
> >>> *Current Release Plan:*
> >>> Feature freeze date : all features to merge by September 7, 2018.
> >>> Code freeze date : blockers/critical only, no improvements and
> >>> blocker/critical bug-fixes September 14, 2018.
> >>> Release date: September 28, 2018
> >>>
> >>> If any critical/blocker tickets which are targeted to 3.2.0, we need to
> >>> backport to 3.2 post branch cut.
> >>>
> >>> Here's an updated 3.2.0 feature status:
> >>>
> >>> 1. Merged & Completed features:
> >>>
> >>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> >>> workloads Initial cut.
> >>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >>> Scheduler.
> >>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> API
> >>> and CLI.
> >>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> >>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.
> >>>
> >>> 2. Features close to finish:
> >>>
> >>> - (Steve) S3Guard Phase III. Close to commit.
> >>> - (Steve) S3a phase V. Close to commit.
> >>> - (Steve) Support Windows Azure Storage. Close to commit.
> >>>
> >>> 3. Tentative/Cancelled features for 3.2:
> >>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> >>> ATSv2. Patch in progress.
> >>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
> >>> be done before Aug 2018.
> >>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >>> Challenging as more discussions are on-going.
> >>>
> >>> *Summary of 3.2.0 issues status:*
> >>> 19 Blocker and Critical issues [1] are open, I am following up with
> >>> owners to get status on each of them to get in by Code Freeze date.
> >>>
> >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> (Blocker,
> >>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
> ORDER
> >>> BY priority DESC
> >>>
> >>> Thanks,
> >>> Sunil
> >>>
> >>>
> >>>
> >>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
> >>>
> >>>> Hi All,
> >>>>
> >>>> Inline with earlier communication dated 17th July 2018, I would like
> to
> >>>> provide some updates.
> >>>>
> >>>> We are approaching previously proposed code freeze date (Aug 31).
> >>>>
> >>>> One of the critical feature Node Attributes feature merge
> >>>> discussion/vote is ongoing. Also few other Blocker bugs need a bit
> more
> >>>> time. With regard to this, suggesting to push the feature/code freeze
> for 2
> >>>> more weeks to accommodate these jiras too.
> >>>>
> >>>> Proposing Updated changes in plan inline with this:
> >>>> Feature freeze date : all features to merge by September 7, 2018.
> >>>> Code freeze date : blockers/critical only, no improvements and
> >>>> blocker/critical bug-fixes September 14, 2018.
> >>>> Release date: September 28, 2018
> >>>>
> >>>> If any features in branch which are targeted to 3.2.0, please reply to
> >>>> this email thread.
> >>>>
> >>>> *Here's an updated 3.2.0 feature status:*
> >>>>
> >>>> 1. Merged & Completed features:
> >>>>
> >>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> >>>> workloads Initial cut.
> >>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >>>> Scheduler.
> >>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> >>>> API and CLI.
> >>>>
> >>>> 2. Features close to finish:
> >>>>
> >>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
> >>>> Ongoing.
> >>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> >>>> ATSv2. Patch in progress.
> >>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
> works.
> >>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> >>>> Storage. In progress.
> >>>>
> >>>> 3. Tentative features:
> >>>>
> >>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging
> to
> >>>> be done before Aug 2018.
> >>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >>>> Challenging as more discussions are on-going.
> >>>>
> >>>> *Summary of 3.2.0 issues status:*
> >>>>
> >>>> 26 Blocker and Critical issues [1] are open, I am following up with
> >>>> owners to get status on each of them to get in by Code Freeze date.
> >>>>
> >>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> >>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> Version/s" =
> >>>> 3.2.0 ORDER BY priority DESC
> >>>>
> >>>> Thanks,
> >>>> Sunil
> >>>>
> >>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
> >>>>
> >>>>> Hi All,
> >>>>>
> >>>>> Thanks for the feedbacks. Inline with earlier communication dated
> 17th
> >>>>> July 2018, I would like to provide some updates.
> >>>>>
> >>>>> We are approaching previously proposed feature freeze date (Aug 21,
> >>>>> about 7 days from today).
> >>>>> If any features in branch which are targeted to 3.2.0, please reply
> to
> >>>>> this email thread.
> >>>>> Steve has mentioned about the s3 features which will come close to
> >>>>> Code Freeze Date (Aug 31st).
> >>>>>
> >>>>> *Here's an updated 3.2.0 feature status:*
> >>>>>
> >>>>> 1. Merged & Completed features:
> >>>>>
> >>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
> >>>>> workloads Initial cut.
> >>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >>>>>
> >>>>> 2. Features close to finish:
> >>>>>
> >>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
> >>>>> patches are all in, only one last
> >>>>> patch is in review state.
> >>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
> >>>>> Scheduler. Close to commit.
> >>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
> >>>>> API and CLI. 2 patches are pending
> >>>>> which will be closed by Feature freeze date.
> >>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> >>>>> ATSv2. Patch in progress.
> >>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
> >>>>> works.
> >>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
> >>>>> Storage. In progress.
> >>>>>
> >>>>> 3. Tentative features:
> >>>>>
> >>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging
> >>>>> to be done before Aug 2018.
> >>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
> >>>>> Challenging as more discussions are on-going.
> >>>>>
> >>>>> *Summary of 3.2.0 issues status:*
> >>>>>
> >>>>> 39 Blocker and Critical issues [1] are open, I am checking with
> owners
> >>>>> to get status on each of them to get in by Code Freeze date.
> >>>>>
> >>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> >>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target
> Version/s" =
> >>>>> 3.2.0 ORDER BY priority DESC
> >>>>>
> >>>>> Thanks,
> >>>>> Sunil
> >>>>>
> >>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
> >>>>>
> >>>>>> Thanks Subru for the thoughts.
> >>>>>> One of the main reason for a major release is to push out critical
> >>>>>> features with a faster cadence to the users. If we are pulling more
> and
> >>>>>> more different types of features to a minor release, that branch
> will
> >>>>>> become more destabilized and it may be tough to say that 3.1.2 is
> stable
> >>>>>> that 3.1.1 for eg. We always tend to improve and stabilize features
> in
> >>>>>> subsequent minor release.
> >>>>>> For few companies, it makes sense to push out these new features
> >>>>>> faster to make a reach to the users. Adding to the point to the
> backporting
> >>>>>> issues, I agree that its a pain and we can workaround that with
> some git
> >>>>>> scripts. If we can make such scripts available to committers,
> backport will
> >>>>>> be seem-less across branches and we can achieve the faster release
> cadence
> >>>>>> also.
> >>>>>>
> >>>>>> Thoughts?
> >>>>>>
> >>>>>> - Sunil
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Thanks Sunil for volunteering to lead the release effort. I am
> >>>>>>> generally
> >>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
> >>>>>>> already
> >>>>>>> have too many branches to be maintained. I already see many commits
> >>>>>>> are in
> >>>>>>> different branches with no apparent rationale, for e.g: 3.1 has
> >>>>>>> commits
> >>>>>>> which are absent in 3.0 etc.
> >>>>>>>
> >>>>>>> Additionally AFAIK 3.x has not been deployed in any major
> production
> >>>>>>> setting so the cost of adding features should be minimal.
> >>>>>>>
> >>>>>>> Thoughts?
> >>>>>>>
> >>>>>>> -Subru
> >>>>>>>
> >>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org>
> wrote:
> >>>>>>>
> >>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> >>>>>>> >
> >>>>>>> > Yes, important changes and features are much needed, hence we
> will
> >>>>>>> be
> >>>>>>> > keeping the door open for them as possible. Also considering few
> >>>>>>> more
> >>>>>>> > offline requests from other folks, I think extending the
> timeframe
> >>>>>>> by
> >>>>>>> > couple of weeks makes sense (including a second RC buffer) and
> >>>>>>> this should
> >>>>>>> > ideally help us to ship this by September itself.
> >>>>>>> >
> >>>>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
> >>>>>>> >
> >>>>>>> > - Feature freeze date : all features to merge by August 21, 2018.
> >>>>>>> >
> >>>>>>> > - Code freeze date : blockers/critical only, no improvements and
> >>>>>>> non
> >>>>>>> > blocker/critical
> >>>>>>> >
> >>>>>>> > bug-fixes August 31, 2018.
> >>>>>>> >
> >>>>>>> > - Release date: September 15, 2018
> >>>>>>> >
> >>>>>>> > Thank Eric and Zian, I think Wangda has already answered your
> >>>>>>> questions.
> >>>>>>> >
> >>>>>>> > Thanks
> >>>>>>> > Sunil
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wheeleast@gmail.com
> >
> >>>>>>> wrote:
> >>>>>>> >
> >>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for
> >>>>>>> that.
> >>>>>>> > >
> >>>>>>> > > To concerns from Steve,
> >>>>>>> > >
> >>>>>>> > > It is a good idea to keep the door open to get important
> changes
> >>>>>>> /
> >>>>>>> > > features in before cutoff. I would prefer to keep the proposed
> >>>>>>> release
> >>>>>>> > date
> >>>>>>> > > to make sure things can happen earlier instead of last minute
> >>>>>>> and we all
> >>>>>>> > > know that releases are always get delayed :). I'm also fine if
> >>>>>>> we want
> >>>>>>> > get
> >>>>>>> > > another several weeks time.
> >>>>>>> > >
> >>>>>>> > > Regarding of 3.3 release, I would suggest doing that before
> >>>>>>> thanksgiving.
> >>>>>>> > > Do you think is it good or too early / late?
> >>>>>>> > >
> >>>>>>> > > Eric,
> >>>>>>> > >
> >>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can
> >>>>>>> get merged
> >>>>>>> > > in time, we probably not need the YARN-8220.
> >>>>>>> > >
> >>>>>>> > > Sunil,
> >>>>>>> > >
> >>>>>>> > > Could u update
> >>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
> >>>>>>> > Roadmap
> >>>>>>> > > with proposed plan as well? We can fill feature list first
> >>>>>>> before getting
> >>>>>>> > > consensus of time.
> >>>>>>> > >
> >>>>>>> > > Thanks,
> >>>>>>> > > Wangda
> >>>>>>> > >
> >>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> >>>>>>> <fabbri@cloudera.com.invalid
> >>>>>>> > >
> >>>>>>> > > wrote:
> >>>>>>> > >
> >>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> >>>>>>> stevel@hortonworks.com>
> >>>>>>> > >> wrote:
> >>>>>>> > >>
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org
> <mailto:
> >>>>>>> > >> > sunilg@apache.org>> wrote:
> >>>>>>> > >> >
> >>>>>>> > >> > I would also would like to take this opportunity to come up
> >>>>>>> with a
> >>>>>>> > >> detailed
> >>>>>>> > >> > plan.
> >>>>>>> > >> >
> >>>>>>> > >> > - Feature freeze date : all features should be merged by
> >>>>>>> August 10,
> >>>>>>> > >> 2018.
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > <snip>
> >>>>>>> > >>
> >>>>>>> > >> >
> >>>>>>> > >> > Please let me know if I missed any features targeted to 3.2
> >>>>>>> per this
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
> >>>>>>> > >> >
> >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake
> Gen
> >>>>>>> 2
> >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> >>>>>>> > >> >
> >>>>>>> > >> > I don't think this is quite ready yet, I've been doing work
> >>>>>>> on it, but
> >>>>>>> > >> if
> >>>>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
> >>>>>>> reviews on
> >>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> >>>>>>> > >> >
> >>>>>>> > >> > I've uprated that to a blocker feature; will review the S3 &
> >>>>>>> S3Guard
> >>>>>>> > >> JIRAs
> >>>>>>> > >> > to see which of those are blocking. Then there are some
> >>>>>>> pressing
> >>>>>>> > "guave,
> >>>>>>> > >> > java 9 prep"
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> I can help with this part if you like.
> >>>>>>> > >>
> >>>>>>> > >>
> >>>>>>> > >>
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > timeline. I would like to volunteer myself as release
> manager
> >>>>>>> of 3.2.0
> >>>>>>> > >> > release.
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> > well volunteered!
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> Yes, thank you for stepping up.
> >>>>>>> > >>
> >>>>>>> > >>
> >>>>>>> > >> >
> >>>>>>> > >> > I think this raises a good q: what timetable should we have
> >>>>>>> for the
> >>>>>>> > >> 3.2. &
> >>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having
> the
> >>>>>>> outline
> >>>>>>> > >> time
> >>>>>>> > >> > from the 3.2 to the 3.3 release means that there's less
> >>>>>>> concern about
> >>>>>>> > >> > things not making the 3.2 dealine
> >>>>>>> > >> >
> >>>>>>> > >> > -Steve
> >>>>>>> > >> >
> >>>>>>> > >> >
> >>>>>>> > >> Good idea to mitigate the short deadline.
> >>>>>>> > >>
> >>>>>>> > >> -AF
> >>>>>>> > >>
> >>>>>>> > >
> >>>>>>> >
> >>>>>>>
> >>>>>>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi All
As mentioned in earlier mail, I have cut branch-3.2 and reset trunk to
3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary
patches are pulled into branch-3.2.
Thank You
- Sunil
On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
> Hi All
>
> We are now down to the last Blocker and HADOOP-15407 is merged to trunk.
> Thanks for the support.
>
> *Plan for RC*
> 3.2 branch cut and reset trunk : *25th Tuesday*
> RC0 for 3.2: *28th Friday*
>
> Thank You
> Sunil
>
>
> On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
>
>> Hi All
>>
>> We are down to 3 Blockers and 4 Critical now. Thanks all of you for
>> helping in this. I am following up on these tickets, once its closed we
>> will cut the 3.2 branch.
>>
>> Thanks
>> Sunil Govindan
>>
>>
>> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
>>
>>> Hi All,
>>>
>>> Inline with the original 3.2 communication proposal dated 17th July
>>> 2018, I would like to provide more updates.
>>>
>>> We are approaching previously proposed code freeze date (September 14,
>>> 2018). So I would like to cut 3.2 branch on 17th Sept and point existing
>>> trunk to 3.3 if there are no issues.
>>>
>>> *Current Release Plan:*
>>> Feature freeze date : all features to merge by September 7, 2018.
>>> Code freeze date : blockers/critical only, no improvements and
>>> blocker/critical bug-fixes September 14, 2018.
>>> Release date: September 28, 2018
>>>
>>> If any critical/blocker tickets which are targeted to 3.2.0, we need to
>>> backport to 3.2 post branch cut.
>>>
>>> Here's an updated 3.2.0 feature status:
>>>
>>> 1. Merged & Completed features:
>>>
>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> workloads Initial cut.
>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> Scheduler.
>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>>> and CLI.
>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.
>>>
>>> 2. Features close to finish:
>>>
>>> - (Steve) S3Guard Phase III. Close to commit.
>>> - (Steve) S3a phase V. Close to commit.
>>> - (Steve) Support Windows Azure Storage. Close to commit.
>>>
>>> 3. Tentative/Cancelled features for 3.2:
>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>> ATSv2. Patch in progress.
>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>>> be done before Aug 2018.
>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> Challenging as more discussions are on-going.
>>>
>>> *Summary of 3.2.0 issues status:*
>>> 19 Blocker and Critical issues [1] are open, I am following up with
>>> owners to get status on each of them to get in by Code Freeze date.
>>>
>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>>> BY priority DESC
>>>
>>> Thanks,
>>> Sunil
>>>
>>>
>>>
>>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
>>>
>>>> Hi All,
>>>>
>>>> Inline with earlier communication dated 17th July 2018, I would like to
>>>> provide some updates.
>>>>
>>>> We are approaching previously proposed code freeze date (Aug 31).
>>>>
>>>> One of the critical feature Node Attributes feature merge
>>>> discussion/vote is ongoing. Also few other Blocker bugs need a bit more
>>>> time. With regard to this, suggesting to push the feature/code freeze for 2
>>>> more weeks to accommodate these jiras too.
>>>>
>>>> Proposing Updated changes in plan inline with this:
>>>> Feature freeze date : all features to merge by September 7, 2018.
>>>> Code freeze date : blockers/critical only, no improvements and
>>>> blocker/critical bug-fixes September 14, 2018.
>>>> Release date: September 28, 2018
>>>>
>>>> If any features in branch which are targeted to 3.2.0, please reply to
>>>> this email thread.
>>>>
>>>> *Here's an updated 3.2.0 feature status:*
>>>>
>>>> 1. Merged & Completed features:
>>>>
>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>>> workloads Initial cut.
>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>>> Scheduler.
>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
>>>> API and CLI.
>>>>
>>>> 2. Features close to finish:
>>>>
>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
>>>> Ongoing.
>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>>> ATSv2. Patch in progress.
>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>>>> Storage. In progress.
>>>>
>>>> 3. Tentative features:
>>>>
>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>>>> be done before Aug 2018.
>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>>> Challenging as more discussions are on-going.
>>>>
>>>> *Summary of 3.2.0 issues status:*
>>>>
>>>> 26 Blocker and Critical issues [1] are open, I am following up with
>>>> owners to get status on each of them to get in by Code Freeze date.
>>>>
>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target Version/s" =
>>>> 3.2.0 ORDER BY priority DESC
>>>>
>>>> Thanks,
>>>> Sunil
>>>>
>>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> Thanks for the feedbacks. Inline with earlier communication dated 17th
>>>>> July 2018, I would like to provide some updates.
>>>>>
>>>>> We are approaching previously proposed feature freeze date (Aug 21,
>>>>> about 7 days from today).
>>>>> If any features in branch which are targeted to 3.2.0, please reply to
>>>>> this email thread.
>>>>> Steve has mentioned about the s3 features which will come close to
>>>>> Code Freeze Date (Aug 31st).
>>>>>
>>>>> *Here's an updated 3.2.0 feature status:*
>>>>>
>>>>> 1. Merged & Completed features:
>>>>>
>>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>>>> workloads Initial cut.
>>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>>>>
>>>>> 2. Features close to finish:
>>>>>
>>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
>>>>> patches are all in, only one last
>>>>> patch is in review state.
>>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>>>> Scheduler. Close to commit.
>>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
>>>>> API and CLI. 2 patches are pending
>>>>> which will be closed by Feature freeze date.
>>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>>>> ATSv2. Patch in progress.
>>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
>>>>> works.
>>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>>>>> Storage. In progress.
>>>>>
>>>>> 3. Tentative features:
>>>>>
>>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging
>>>>> to be done before Aug 2018.
>>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>>>> Challenging as more discussions are on-going.
>>>>>
>>>>> *Summary of 3.2.0 issues status:*
>>>>>
>>>>> 39 Blocker and Critical issues [1] are open, I am checking with owners
>>>>> to get status on each of them to get in by Code Freeze date.
>>>>>
>>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target Version/s" =
>>>>> 3.2.0 ORDER BY priority DESC
>>>>>
>>>>> Thanks,
>>>>> Sunil
>>>>>
>>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
>>>>>
>>>>>> Thanks Subru for the thoughts.
>>>>>> One of the main reason for a major release is to push out critical
>>>>>> features with a faster cadence to the users. If we are pulling more and
>>>>>> more different types of features to a minor release, that branch will
>>>>>> become more destabilized and it may be tough to say that 3.1.2 is stable
>>>>>> that 3.1.1 for eg. We always tend to improve and stabilize features in
>>>>>> subsequent minor release.
>>>>>> For few companies, it makes sense to push out these new features
>>>>>> faster to make a reach to the users. Adding to the point to the backporting
>>>>>> issues, I agree that its a pain and we can workaround that with some git
>>>>>> scripts. If we can make such scripts available to committers, backport will
>>>>>> be seem-less across branches and we can achieve the faster release cadence
>>>>>> also.
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> - Sunil
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Sunil for volunteering to lead the release effort. I am
>>>>>>> generally
>>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
>>>>>>> already
>>>>>>> have too many branches to be maintained. I already see many commits
>>>>>>> are in
>>>>>>> different branches with no apparent rationale, for e.g: 3.1 has
>>>>>>> commits
>>>>>>> which are absent in 3.0 etc.
>>>>>>>
>>>>>>> Additionally AFAIK 3.x has not been deployed in any major production
>>>>>>> setting so the cost of adding features should be minimal.
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>> -Subru
>>>>>>>
>>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
>>>>>>>
>>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>>>>>> >
>>>>>>> > Yes, important changes and features are much needed, hence we will
>>>>>>> be
>>>>>>> > keeping the door open for them as possible. Also considering few
>>>>>>> more
>>>>>>> > offline requests from other folks, I think extending the timeframe
>>>>>>> by
>>>>>>> > couple of weeks makes sense (including a second RC buffer) and
>>>>>>> this should
>>>>>>> > ideally help us to ship this by September itself.
>>>>>>> >
>>>>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
>>>>>>> >
>>>>>>> > - Feature freeze date : all features to merge by August 21, 2018.
>>>>>>> >
>>>>>>> > - Code freeze date : blockers/critical only, no improvements and
>>>>>>> non
>>>>>>> > blocker/critical
>>>>>>> >
>>>>>>> > bug-fixes August 31, 2018.
>>>>>>> >
>>>>>>> > - Release date: September 15, 2018
>>>>>>> >
>>>>>>> > Thank Eric and Zian, I think Wangda has already answered your
>>>>>>> questions.
>>>>>>> >
>>>>>>> > Thanks
>>>>>>> > Sunil
>>>>>>> >
>>>>>>> >
>>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for
>>>>>>> that.
>>>>>>> > >
>>>>>>> > > To concerns from Steve,
>>>>>>> > >
>>>>>>> > > It is a good idea to keep the door open to get important changes
>>>>>>> /
>>>>>>> > > features in before cutoff. I would prefer to keep the proposed
>>>>>>> release
>>>>>>> > date
>>>>>>> > > to make sure things can happen earlier instead of last minute
>>>>>>> and we all
>>>>>>> > > know that releases are always get delayed :). I'm also fine if
>>>>>>> we want
>>>>>>> > get
>>>>>>> > > another several weeks time.
>>>>>>> > >
>>>>>>> > > Regarding of 3.3 release, I would suggest doing that before
>>>>>>> thanksgiving.
>>>>>>> > > Do you think is it good or too early / late?
>>>>>>> > >
>>>>>>> > > Eric,
>>>>>>> > >
>>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can
>>>>>>> get merged
>>>>>>> > > in time, we probably not need the YARN-8220.
>>>>>>> > >
>>>>>>> > > Sunil,
>>>>>>> > >
>>>>>>> > > Could u update
>>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>>>>>>> > Roadmap
>>>>>>> > > with proposed plan as well? We can fill feature list first
>>>>>>> before getting
>>>>>>> > > consensus of time.
>>>>>>> > >
>>>>>>> > > Thanks,
>>>>>>> > > Wangda
>>>>>>> > >
>>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>>>>>> <fabbri@cloudera.com.invalid
>>>>>>> > >
>>>>>>> > > wrote:
>>>>>>> > >
>>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>>>>>> stevel@hortonworks.com>
>>>>>>> > >> wrote:
>>>>>>> > >>
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
>>>>>>> > >> > sunilg@apache.org>> wrote:
>>>>>>> > >> >
>>>>>>> > >> > I would also would like to take this opportunity to come up
>>>>>>> with a
>>>>>>> > >> detailed
>>>>>>> > >> > plan.
>>>>>>> > >> >
>>>>>>> > >> > - Feature freeze date : all features should be merged by
>>>>>>> August 10,
>>>>>>> > >> 2018.
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > <snip>
>>>>>>> > >>
>>>>>>> > >> >
>>>>>>> > >> > Please let me know if I missed any features targeted to 3.2
>>>>>>> per this
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>>>>>> > >> >
>>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen
>>>>>>> 2
>>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>>>>>> > >> >
>>>>>>> > >> > I don't think this is quite ready yet, I've been doing work
>>>>>>> on it, but
>>>>>>> > >> if
>>>>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
>>>>>>> reviews on
>>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>>>>>> > >> >
>>>>>>> > >> > I've uprated that to a blocker feature; will review the S3 &
>>>>>>> S3Guard
>>>>>>> > >> JIRAs
>>>>>>> > >> > to see which of those are blocking. Then there are some
>>>>>>> pressing
>>>>>>> > "guave,
>>>>>>> > >> > java 9 prep"
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> I can help with this part if you like.
>>>>>>> > >>
>>>>>>> > >>
>>>>>>> > >>
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > timeline. I would like to volunteer myself as release manager
>>>>>>> of 3.2.0
>>>>>>> > >> > release.
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > well volunteered!
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> Yes, thank you for stepping up.
>>>>>>> > >>
>>>>>>> > >>
>>>>>>> > >> >
>>>>>>> > >> > I think this raises a good q: what timetable should we have
>>>>>>> for the
>>>>>>> > >> 3.2. &
>>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having the
>>>>>>> outline
>>>>>>> > >> time
>>>>>>> > >> > from the 3.2 to the 3.3 release means that there's less
>>>>>>> concern about
>>>>>>> > >> > things not making the 3.2 dealine
>>>>>>> > >> >
>>>>>>> > >> > -Steve
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> Good idea to mitigate the short deadline.
>>>>>>> > >>
>>>>>>> > >> -AF
>>>>>>> > >>
>>>>>>> > >
>>>>>>> >
>>>>>>>
>>>>>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi All
As mentioned in earlier mail, I have cut branch-3.2 and reset trunk to
3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary
patches are pulled into branch-3.2.
Thank You
- Sunil
On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
> Hi All
>
> We are now down to the last Blocker and HADOOP-15407 is merged to trunk.
> Thanks for the support.
>
> *Plan for RC*
> 3.2 branch cut and reset trunk : *25th Tuesday*
> RC0 for 3.2: *28th Friday*
>
> Thank You
> Sunil
>
>
> On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
>
>> Hi All
>>
>> We are down to 3 Blockers and 4 Critical now. Thanks all of you for
>> helping in this. I am following up on these tickets, once its closed we
>> will cut the 3.2 branch.
>>
>> Thanks
>> Sunil Govindan
>>
>>
>> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
>>
>>> Hi All,
>>>
>>> Inline with the original 3.2 communication proposal dated 17th July
>>> 2018, I would like to provide more updates.
>>>
>>> We are approaching previously proposed code freeze date (September 14,
>>> 2018). So I would like to cut 3.2 branch on 17th Sept and point existing
>>> trunk to 3.3 if there are no issues.
>>>
>>> *Current Release Plan:*
>>> Feature freeze date : all features to merge by September 7, 2018.
>>> Code freeze date : blockers/critical only, no improvements and
>>> blocker/critical bug-fixes September 14, 2018.
>>> Release date: September 28, 2018
>>>
>>> If any critical/blocker tickets which are targeted to 3.2.0, we need to
>>> backport to 3.2 post branch cut.
>>>
>>> Here's an updated 3.2.0 feature status:
>>>
>>> 1. Merged & Completed features:
>>>
>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> workloads Initial cut.
>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> Scheduler.
>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>>> and CLI.
>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.
>>>
>>> 2. Features close to finish:
>>>
>>> - (Steve) S3Guard Phase III. Close to commit.
>>> - (Steve) S3a phase V. Close to commit.
>>> - (Steve) Support Windows Azure Storage. Close to commit.
>>>
>>> 3. Tentative/Cancelled features for 3.2:
>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>> ATSv2. Patch in progress.
>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>>> be done before Aug 2018.
>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> Challenging as more discussions are on-going.
>>>
>>> *Summary of 3.2.0 issues status:*
>>> 19 Blocker and Critical issues [1] are open, I am following up with
>>> owners to get status on each of them to get in by Code Freeze date.
>>>
>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>>> BY priority DESC
>>>
>>> Thanks,
>>> Sunil
>>>
>>>
>>>
>>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
>>>
>>>> Hi All,
>>>>
>>>> Inline with earlier communication dated 17th July 2018, I would like to
>>>> provide some updates.
>>>>
>>>> We are approaching previously proposed code freeze date (Aug 31).
>>>>
>>>> One of the critical feature Node Attributes feature merge
>>>> discussion/vote is ongoing. Also few other Blocker bugs need a bit more
>>>> time. With regard to this, suggesting to push the feature/code freeze for 2
>>>> more weeks to accommodate these jiras too.
>>>>
>>>> Proposing Updated changes in plan inline with this:
>>>> Feature freeze date : all features to merge by September 7, 2018.
>>>> Code freeze date : blockers/critical only, no improvements and
>>>> blocker/critical bug-fixes September 14, 2018.
>>>> Release date: September 28, 2018
>>>>
>>>> If any features in branch which are targeted to 3.2.0, please reply to
>>>> this email thread.
>>>>
>>>> *Here's an updated 3.2.0 feature status:*
>>>>
>>>> 1. Merged & Completed features:
>>>>
>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>>> workloads Initial cut.
>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>>> Scheduler.
>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
>>>> API and CLI.
>>>>
>>>> 2. Features close to finish:
>>>>
>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
>>>> Ongoing.
>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>>> ATSv2. Patch in progress.
>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>>>> Storage. In progress.
>>>>
>>>> 3. Tentative features:
>>>>
>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>>>> be done before Aug 2018.
>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>>> Challenging as more discussions are on-going.
>>>>
>>>> *Summary of 3.2.0 issues status:*
>>>>
>>>> 26 Blocker and Critical issues [1] are open, I am following up with
>>>> owners to get status on each of them to get in by Code Freeze date.
>>>>
>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target Version/s" =
>>>> 3.2.0 ORDER BY priority DESC
>>>>
>>>> Thanks,
>>>> Sunil
>>>>
>>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> Thanks for the feedbacks. Inline with earlier communication dated 17th
>>>>> July 2018, I would like to provide some updates.
>>>>>
>>>>> We are approaching previously proposed feature freeze date (Aug 21,
>>>>> about 7 days from today).
>>>>> If any features in branch which are targeted to 3.2.0, please reply to
>>>>> this email thread.
>>>>> Steve has mentioned about the s3 features which will come close to
>>>>> Code Freeze Date (Aug 31st).
>>>>>
>>>>> *Here's an updated 3.2.0 feature status:*
>>>>>
>>>>> 1. Merged & Completed features:
>>>>>
>>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>>>> workloads Initial cut.
>>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>>>>
>>>>> 2. Features close to finish:
>>>>>
>>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
>>>>> patches are all in, only one last
>>>>> patch is in review state.
>>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>>>> Scheduler. Close to commit.
>>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
>>>>> API and CLI. 2 patches are pending
>>>>> which will be closed by Feature freeze date.
>>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>>>> ATSv2. Patch in progress.
>>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
>>>>> works.
>>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>>>>> Storage. In progress.
>>>>>
>>>>> 3. Tentative features:
>>>>>
>>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging
>>>>> to be done before Aug 2018.
>>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>>>> Challenging as more discussions are on-going.
>>>>>
>>>>> *Summary of 3.2.0 issues status:*
>>>>>
>>>>> 39 Blocker and Critical issues [1] are open, I am checking with owners
>>>>> to get status on each of them to get in by Code Freeze date.
>>>>>
>>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target Version/s" =
>>>>> 3.2.0 ORDER BY priority DESC
>>>>>
>>>>> Thanks,
>>>>> Sunil
>>>>>
>>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
>>>>>
>>>>>> Thanks Subru for the thoughts.
>>>>>> One of the main reason for a major release is to push out critical
>>>>>> features with a faster cadence to the users. If we are pulling more and
>>>>>> more different types of features to a minor release, that branch will
>>>>>> become more destabilized and it may be tough to say that 3.1.2 is stable
>>>>>> that 3.1.1 for eg. We always tend to improve and stabilize features in
>>>>>> subsequent minor release.
>>>>>> For few companies, it makes sense to push out these new features
>>>>>> faster to make a reach to the users. Adding to the point to the backporting
>>>>>> issues, I agree that its a pain and we can workaround that with some git
>>>>>> scripts. If we can make such scripts available to committers, backport will
>>>>>> be seem-less across branches and we can achieve the faster release cadence
>>>>>> also.
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> - Sunil
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Sunil for volunteering to lead the release effort. I am
>>>>>>> generally
>>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
>>>>>>> already
>>>>>>> have too many branches to be maintained. I already see many commits
>>>>>>> are in
>>>>>>> different branches with no apparent rationale, for e.g: 3.1 has
>>>>>>> commits
>>>>>>> which are absent in 3.0 etc.
>>>>>>>
>>>>>>> Additionally AFAIK 3.x has not been deployed in any major production
>>>>>>> setting so the cost of adding features should be minimal.
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>> -Subru
>>>>>>>
>>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
>>>>>>>
>>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>>>>>> >
>>>>>>> > Yes, important changes and features are much needed, hence we will
>>>>>>> be
>>>>>>> > keeping the door open for them as possible. Also considering few
>>>>>>> more
>>>>>>> > offline requests from other folks, I think extending the timeframe
>>>>>>> by
>>>>>>> > couple of weeks makes sense (including a second RC buffer) and
>>>>>>> this should
>>>>>>> > ideally help us to ship this by September itself.
>>>>>>> >
>>>>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
>>>>>>> >
>>>>>>> > - Feature freeze date : all features to merge by August 21, 2018.
>>>>>>> >
>>>>>>> > - Code freeze date : blockers/critical only, no improvements and
>>>>>>> non
>>>>>>> > blocker/critical
>>>>>>> >
>>>>>>> > bug-fixes August 31, 2018.
>>>>>>> >
>>>>>>> > - Release date: September 15, 2018
>>>>>>> >
>>>>>>> > Thank Eric and Zian, I think Wangda has already answered your
>>>>>>> questions.
>>>>>>> >
>>>>>>> > Thanks
>>>>>>> > Sunil
>>>>>>> >
>>>>>>> >
>>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for
>>>>>>> that.
>>>>>>> > >
>>>>>>> > > To concerns from Steve,
>>>>>>> > >
>>>>>>> > > It is a good idea to keep the door open to get important changes
>>>>>>> /
>>>>>>> > > features in before cutoff. I would prefer to keep the proposed
>>>>>>> release
>>>>>>> > date
>>>>>>> > > to make sure things can happen earlier instead of last minute
>>>>>>> and we all
>>>>>>> > > know that releases are always get delayed :). I'm also fine if
>>>>>>> we want
>>>>>>> > get
>>>>>>> > > another several weeks time.
>>>>>>> > >
>>>>>>> > > Regarding of 3.3 release, I would suggest doing that before
>>>>>>> thanksgiving.
>>>>>>> > > Do you think is it good or too early / late?
>>>>>>> > >
>>>>>>> > > Eric,
>>>>>>> > >
>>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can
>>>>>>> get merged
>>>>>>> > > in time, we probably not need the YARN-8220.
>>>>>>> > >
>>>>>>> > > Sunil,
>>>>>>> > >
>>>>>>> > > Could u update
>>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>>>>>>> > Roadmap
>>>>>>> > > with proposed plan as well? We can fill feature list first
>>>>>>> before getting
>>>>>>> > > consensus of time.
>>>>>>> > >
>>>>>>> > > Thanks,
>>>>>>> > > Wangda
>>>>>>> > >
>>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>>>>>> <fabbri@cloudera.com.invalid
>>>>>>> > >
>>>>>>> > > wrote:
>>>>>>> > >
>>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>>>>>> stevel@hortonworks.com>
>>>>>>> > >> wrote:
>>>>>>> > >>
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
>>>>>>> > >> > sunilg@apache.org>> wrote:
>>>>>>> > >> >
>>>>>>> > >> > I would also would like to take this opportunity to come up
>>>>>>> with a
>>>>>>> > >> detailed
>>>>>>> > >> > plan.
>>>>>>> > >> >
>>>>>>> > >> > - Feature freeze date : all features should be merged by
>>>>>>> August 10,
>>>>>>> > >> 2018.
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > <snip>
>>>>>>> > >>
>>>>>>> > >> >
>>>>>>> > >> > Please let me know if I missed any features targeted to 3.2
>>>>>>> per this
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>>>>>> > >> >
>>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen
>>>>>>> 2
>>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>>>>>> > >> >
>>>>>>> > >> > I don't think this is quite ready yet, I've been doing work
>>>>>>> on it, but
>>>>>>> > >> if
>>>>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
>>>>>>> reviews on
>>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>>>>>> > >> >
>>>>>>> > >> > I've uprated that to a blocker feature; will review the S3 &
>>>>>>> S3Guard
>>>>>>> > >> JIRAs
>>>>>>> > >> > to see which of those are blocking. Then there are some
>>>>>>> pressing
>>>>>>> > "guave,
>>>>>>> > >> > java 9 prep"
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> I can help with this part if you like.
>>>>>>> > >>
>>>>>>> > >>
>>>>>>> > >>
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > timeline. I would like to volunteer myself as release manager
>>>>>>> of 3.2.0
>>>>>>> > >> > release.
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > well volunteered!
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> Yes, thank you for stepping up.
>>>>>>> > >>
>>>>>>> > >>
>>>>>>> > >> >
>>>>>>> > >> > I think this raises a good q: what timetable should we have
>>>>>>> for the
>>>>>>> > >> 3.2. &
>>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having the
>>>>>>> outline
>>>>>>> > >> time
>>>>>>> > >> > from the 3.2 to the 3.3 release means that there's less
>>>>>>> concern about
>>>>>>> > >> > things not making the 3.2 dealine
>>>>>>> > >> >
>>>>>>> > >> > -Steve
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> Good idea to mitigate the short deadline.
>>>>>>> > >>
>>>>>>> > >> -AF
>>>>>>> > >>
>>>>>>> > >
>>>>>>> >
>>>>>>>
>>>>>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi All
As mentioned in earlier mail, I have cut branch-3.2 and reset trunk to
3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary
patches are pulled into branch-3.2.
Thank You
- Sunil
On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
> Hi All
>
> We are now down to the last Blocker and HADOOP-15407 is merged to trunk.
> Thanks for the support.
>
> *Plan for RC*
> 3.2 branch cut and reset trunk : *25th Tuesday*
> RC0 for 3.2: *28th Friday*
>
> Thank You
> Sunil
>
>
> On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
>
>> Hi All
>>
>> We are down to 3 Blockers and 4 Critical now. Thanks all of you for
>> helping in this. I am following up on these tickets, once its closed we
>> will cut the 3.2 branch.
>>
>> Thanks
>> Sunil Govindan
>>
>>
>> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
>>
>>> Hi All,
>>>
>>> Inline with the original 3.2 communication proposal dated 17th July
>>> 2018, I would like to provide more updates.
>>>
>>> We are approaching previously proposed code freeze date (September 14,
>>> 2018). So I would like to cut 3.2 branch on 17th Sept and point existing
>>> trunk to 3.3 if there are no issues.
>>>
>>> *Current Release Plan:*
>>> Feature freeze date : all features to merge by September 7, 2018.
>>> Code freeze date : blockers/critical only, no improvements and
>>> blocker/critical bug-fixes September 14, 2018.
>>> Release date: September 28, 2018
>>>
>>> If any critical/blocker tickets which are targeted to 3.2.0, we need to
>>> backport to 3.2 post branch cut.
>>>
>>> Here's an updated 3.2.0 feature status:
>>>
>>> 1. Merged & Completed features:
>>>
>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> workloads Initial cut.
>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> Scheduler.
>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>>> and CLI.
>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.
>>>
>>> 2. Features close to finish:
>>>
>>> - (Steve) S3Guard Phase III. Close to commit.
>>> - (Steve) S3a phase V. Close to commit.
>>> - (Steve) Support Windows Azure Storage. Close to commit.
>>>
>>> 3. Tentative/Cancelled features for 3.2:
>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>> ATSv2. Patch in progress.
>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>>> be done before Aug 2018.
>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> Challenging as more discussions are on-going.
>>>
>>> *Summary of 3.2.0 issues status:*
>>> 19 Blocker and Critical issues [1] are open, I am following up with
>>> owners to get status on each of them to get in by Code Freeze date.
>>>
>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>>> BY priority DESC
>>>
>>> Thanks,
>>> Sunil
>>>
>>>
>>>
>>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
>>>
>>>> Hi All,
>>>>
>>>> Inline with earlier communication dated 17th July 2018, I would like to
>>>> provide some updates.
>>>>
>>>> We are approaching previously proposed code freeze date (Aug 31).
>>>>
>>>> One of the critical feature Node Attributes feature merge
>>>> discussion/vote is ongoing. Also few other Blocker bugs need a bit more
>>>> time. With regard to this, suggesting to push the feature/code freeze for 2
>>>> more weeks to accommodate these jiras too.
>>>>
>>>> Proposing Updated changes in plan inline with this:
>>>> Feature freeze date : all features to merge by September 7, 2018.
>>>> Code freeze date : blockers/critical only, no improvements and
>>>> blocker/critical bug-fixes September 14, 2018.
>>>> Release date: September 28, 2018
>>>>
>>>> If any features in branch which are targeted to 3.2.0, please reply to
>>>> this email thread.
>>>>
>>>> *Here's an updated 3.2.0 feature status:*
>>>>
>>>> 1. Merged & Completed features:
>>>>
>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>>> workloads Initial cut.
>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>>> Scheduler.
>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
>>>> API and CLI.
>>>>
>>>> 2. Features close to finish:
>>>>
>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
>>>> Ongoing.
>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>>> ATSv2. Patch in progress.
>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>>>> Storage. In progress.
>>>>
>>>> 3. Tentative features:
>>>>
>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>>>> be done before Aug 2018.
>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>>> Challenging as more discussions are on-going.
>>>>
>>>> *Summary of 3.2.0 issues status:*
>>>>
>>>> 26 Blocker and Critical issues [1] are open, I am following up with
>>>> owners to get status on each of them to get in by Code Freeze date.
>>>>
>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target Version/s" =
>>>> 3.2.0 ORDER BY priority DESC
>>>>
>>>> Thanks,
>>>> Sunil
>>>>
>>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> Thanks for the feedbacks. Inline with earlier communication dated 17th
>>>>> July 2018, I would like to provide some updates.
>>>>>
>>>>> We are approaching previously proposed feature freeze date (Aug 21,
>>>>> about 7 days from today).
>>>>> If any features in branch which are targeted to 3.2.0, please reply to
>>>>> this email thread.
>>>>> Steve has mentioned about the s3 features which will come close to
>>>>> Code Freeze Date (Aug 31st).
>>>>>
>>>>> *Here's an updated 3.2.0 feature status:*
>>>>>
>>>>> 1. Merged & Completed features:
>>>>>
>>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>>>> workloads Initial cut.
>>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>>>>
>>>>> 2. Features close to finish:
>>>>>
>>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
>>>>> patches are all in, only one last
>>>>> patch is in review state.
>>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>>>> Scheduler. Close to commit.
>>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
>>>>> API and CLI. 2 patches are pending
>>>>> which will be closed by Feature freeze date.
>>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>>>> ATSv2. Patch in progress.
>>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
>>>>> works.
>>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>>>>> Storage. In progress.
>>>>>
>>>>> 3. Tentative features:
>>>>>
>>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging
>>>>> to be done before Aug 2018.
>>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>>>> Challenging as more discussions are on-going.
>>>>>
>>>>> *Summary of 3.2.0 issues status:*
>>>>>
>>>>> 39 Blocker and Critical issues [1] are open, I am checking with owners
>>>>> to get status on each of them to get in by Code Freeze date.
>>>>>
>>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target Version/s" =
>>>>> 3.2.0 ORDER BY priority DESC
>>>>>
>>>>> Thanks,
>>>>> Sunil
>>>>>
>>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
>>>>>
>>>>>> Thanks Subru for the thoughts.
>>>>>> One of the main reason for a major release is to push out critical
>>>>>> features with a faster cadence to the users. If we are pulling more and
>>>>>> more different types of features to a minor release, that branch will
>>>>>> become more destabilized and it may be tough to say that 3.1.2 is stable
>>>>>> that 3.1.1 for eg. We always tend to improve and stabilize features in
>>>>>> subsequent minor release.
>>>>>> For few companies, it makes sense to push out these new features
>>>>>> faster to make a reach to the users. Adding to the point to the backporting
>>>>>> issues, I agree that its a pain and we can workaround that with some git
>>>>>> scripts. If we can make such scripts available to committers, backport will
>>>>>> be seem-less across branches and we can achieve the faster release cadence
>>>>>> also.
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> - Sunil
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Sunil for volunteering to lead the release effort. I am
>>>>>>> generally
>>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
>>>>>>> already
>>>>>>> have too many branches to be maintained. I already see many commits
>>>>>>> are in
>>>>>>> different branches with no apparent rationale, for e.g: 3.1 has
>>>>>>> commits
>>>>>>> which are absent in 3.0 etc.
>>>>>>>
>>>>>>> Additionally AFAIK 3.x has not been deployed in any major production
>>>>>>> setting so the cost of adding features should be minimal.
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>> -Subru
>>>>>>>
>>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
>>>>>>>
>>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>>>>>> >
>>>>>>> > Yes, important changes and features are much needed, hence we will
>>>>>>> be
>>>>>>> > keeping the door open for them as possible. Also considering few
>>>>>>> more
>>>>>>> > offline requests from other folks, I think extending the timeframe
>>>>>>> by
>>>>>>> > couple of weeks makes sense (including a second RC buffer) and
>>>>>>> this should
>>>>>>> > ideally help us to ship this by September itself.
>>>>>>> >
>>>>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
>>>>>>> >
>>>>>>> > - Feature freeze date : all features to merge by August 21, 2018.
>>>>>>> >
>>>>>>> > - Code freeze date : blockers/critical only, no improvements and
>>>>>>> non
>>>>>>> > blocker/critical
>>>>>>> >
>>>>>>> > bug-fixes August 31, 2018.
>>>>>>> >
>>>>>>> > - Release date: September 15, 2018
>>>>>>> >
>>>>>>> > Thank Eric and Zian, I think Wangda has already answered your
>>>>>>> questions.
>>>>>>> >
>>>>>>> > Thanks
>>>>>>> > Sunil
>>>>>>> >
>>>>>>> >
>>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for
>>>>>>> that.
>>>>>>> > >
>>>>>>> > > To concerns from Steve,
>>>>>>> > >
>>>>>>> > > It is a good idea to keep the door open to get important changes
>>>>>>> /
>>>>>>> > > features in before cutoff. I would prefer to keep the proposed
>>>>>>> release
>>>>>>> > date
>>>>>>> > > to make sure things can happen earlier instead of last minute
>>>>>>> and we all
>>>>>>> > > know that releases are always get delayed :). I'm also fine if
>>>>>>> we want
>>>>>>> > get
>>>>>>> > > another several weeks time.
>>>>>>> > >
>>>>>>> > > Regarding of 3.3 release, I would suggest doing that before
>>>>>>> thanksgiving.
>>>>>>> > > Do you think is it good or too early / late?
>>>>>>> > >
>>>>>>> > > Eric,
>>>>>>> > >
>>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can
>>>>>>> get merged
>>>>>>> > > in time, we probably not need the YARN-8220.
>>>>>>> > >
>>>>>>> > > Sunil,
>>>>>>> > >
>>>>>>> > > Could u update
>>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>>>>>>> > Roadmap
>>>>>>> > > with proposed plan as well? We can fill feature list first
>>>>>>> before getting
>>>>>>> > > consensus of time.
>>>>>>> > >
>>>>>>> > > Thanks,
>>>>>>> > > Wangda
>>>>>>> > >
>>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>>>>>> <fabbri@cloudera.com.invalid
>>>>>>> > >
>>>>>>> > > wrote:
>>>>>>> > >
>>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>>>>>> stevel@hortonworks.com>
>>>>>>> > >> wrote:
>>>>>>> > >>
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
>>>>>>> > >> > sunilg@apache.org>> wrote:
>>>>>>> > >> >
>>>>>>> > >> > I would also would like to take this opportunity to come up
>>>>>>> with a
>>>>>>> > >> detailed
>>>>>>> > >> > plan.
>>>>>>> > >> >
>>>>>>> > >> > - Feature freeze date : all features should be merged by
>>>>>>> August 10,
>>>>>>> > >> 2018.
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > <snip>
>>>>>>> > >>
>>>>>>> > >> >
>>>>>>> > >> > Please let me know if I missed any features targeted to 3.2
>>>>>>> per this
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>>>>>> > >> >
>>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen
>>>>>>> 2
>>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>>>>>> > >> >
>>>>>>> > >> > I don't think this is quite ready yet, I've been doing work
>>>>>>> on it, but
>>>>>>> > >> if
>>>>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
>>>>>>> reviews on
>>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>>>>>> > >> >
>>>>>>> > >> > I've uprated that to a blocker feature; will review the S3 &
>>>>>>> S3Guard
>>>>>>> > >> JIRAs
>>>>>>> > >> > to see which of those are blocking. Then there are some
>>>>>>> pressing
>>>>>>> > "guave,
>>>>>>> > >> > java 9 prep"
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> I can help with this part if you like.
>>>>>>> > >>
>>>>>>> > >>
>>>>>>> > >>
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > timeline. I would like to volunteer myself as release manager
>>>>>>> of 3.2.0
>>>>>>> > >> > release.
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > well volunteered!
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> Yes, thank you for stepping up.
>>>>>>> > >>
>>>>>>> > >>
>>>>>>> > >> >
>>>>>>> > >> > I think this raises a good q: what timetable should we have
>>>>>>> for the
>>>>>>> > >> 3.2. &
>>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having the
>>>>>>> outline
>>>>>>> > >> time
>>>>>>> > >> > from the 3.2 to the 3.3 release means that there's less
>>>>>>> concern about
>>>>>>> > >> > things not making the 3.2 dealine
>>>>>>> > >> >
>>>>>>> > >> > -Steve
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> Good idea to mitigate the short deadline.
>>>>>>> > >>
>>>>>>> > >> -AF
>>>>>>> > >>
>>>>>>> > >
>>>>>>> >
>>>>>>>
>>>>>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi All
As mentioned in earlier mail, I have cut branch-3.2 and reset trunk to
3.3.0-SNAPSHOT. I will share the RC details sooner once all necessary
patches are pulled into branch-3.2.
Thank You
- Sunil
On Mon, Sep 24, 2018 at 2:00 PM Sunil G <su...@apache.org> wrote:
> Hi All
>
> We are now down to the last Blocker and HADOOP-15407 is merged to trunk.
> Thanks for the support.
>
> *Plan for RC*
> 3.2 branch cut and reset trunk : *25th Tuesday*
> RC0 for 3.2: *28th Friday*
>
> Thank You
> Sunil
>
>
> On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
>
>> Hi All
>>
>> We are down to 3 Blockers and 4 Critical now. Thanks all of you for
>> helping in this. I am following up on these tickets, once its closed we
>> will cut the 3.2 branch.
>>
>> Thanks
>> Sunil Govindan
>>
>>
>> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
>>
>>> Hi All,
>>>
>>> Inline with the original 3.2 communication proposal dated 17th July
>>> 2018, I would like to provide more updates.
>>>
>>> We are approaching previously proposed code freeze date (September 14,
>>> 2018). So I would like to cut 3.2 branch on 17th Sept and point existing
>>> trunk to 3.3 if there are no issues.
>>>
>>> *Current Release Plan:*
>>> Feature freeze date : all features to merge by September 7, 2018.
>>> Code freeze date : blockers/critical only, no improvements and
>>> blocker/critical bug-fixes September 14, 2018.
>>> Release date: September 28, 2018
>>>
>>> If any critical/blocker tickets which are targeted to 3.2.0, we need to
>>> backport to 3.2 post branch cut.
>>>
>>> Here's an updated 3.2.0 feature status:
>>>
>>> 1. Merged & Completed features:
>>>
>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> workloads Initial cut.
>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> Scheduler.
>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>>> and CLI.
>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.
>>>
>>> 2. Features close to finish:
>>>
>>> - (Steve) S3Guard Phase III. Close to commit.
>>> - (Steve) S3a phase V. Close to commit.
>>> - (Steve) Support Windows Azure Storage. Close to commit.
>>>
>>> 3. Tentative/Cancelled features for 3.2:
>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>> ATSv2. Patch in progress.
>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>>> be done before Aug 2018.
>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> Challenging as more discussions are on-going.
>>>
>>> *Summary of 3.2.0 issues status:*
>>> 19 Blocker and Critical issues [1] are open, I am following up with
>>> owners to get status on each of them to get in by Code Freeze date.
>>>
>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>>> BY priority DESC
>>>
>>> Thanks,
>>> Sunil
>>>
>>>
>>>
>>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
>>>
>>>> Hi All,
>>>>
>>>> Inline with earlier communication dated 17th July 2018, I would like to
>>>> provide some updates.
>>>>
>>>> We are approaching previously proposed code freeze date (Aug 31).
>>>>
>>>> One of the critical feature Node Attributes feature merge
>>>> discussion/vote is ongoing. Also few other Blocker bugs need a bit more
>>>> time. With regard to this, suggesting to push the feature/code freeze for 2
>>>> more weeks to accommodate these jiras too.
>>>>
>>>> Proposing Updated changes in plan inline with this:
>>>> Feature freeze date : all features to merge by September 7, 2018.
>>>> Code freeze date : blockers/critical only, no improvements and
>>>> blocker/critical bug-fixes September 14, 2018.
>>>> Release date: September 28, 2018
>>>>
>>>> If any features in branch which are targeted to 3.2.0, please reply to
>>>> this email thread.
>>>>
>>>> *Here's an updated 3.2.0 feature status:*
>>>>
>>>> 1. Merged & Completed features:
>>>>
>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>>> workloads Initial cut.
>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>>> Scheduler.
>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
>>>> API and CLI.
>>>>
>>>> 2. Features close to finish:
>>>>
>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
>>>> Ongoing.
>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>>> ATSv2. Patch in progress.
>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>>>> Storage. In progress.
>>>>
>>>> 3. Tentative features:
>>>>
>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>>>> be done before Aug 2018.
>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>>> Challenging as more discussions are on-going.
>>>>
>>>> *Summary of 3.2.0 issues status:*
>>>>
>>>> 26 Blocker and Critical issues [1] are open, I am following up with
>>>> owners to get status on each of them to get in by Code Freeze date.
>>>>
>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target Version/s" =
>>>> 3.2.0 ORDER BY priority DESC
>>>>
>>>> Thanks,
>>>> Sunil
>>>>
>>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> Thanks for the feedbacks. Inline with earlier communication dated 17th
>>>>> July 2018, I would like to provide some updates.
>>>>>
>>>>> We are approaching previously proposed feature freeze date (Aug 21,
>>>>> about 7 days from today).
>>>>> If any features in branch which are targeted to 3.2.0, please reply to
>>>>> this email thread.
>>>>> Steve has mentioned about the s3 features which will come close to
>>>>> Code Freeze Date (Aug 31st).
>>>>>
>>>>> *Here's an updated 3.2.0 feature status:*
>>>>>
>>>>> 1. Merged & Completed features:
>>>>>
>>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>>>> workloads Initial cut.
>>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>>>>
>>>>> 2. Features close to finish:
>>>>>
>>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
>>>>> patches are all in, only one last
>>>>> patch is in review state.
>>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>>>> Scheduler. Close to commit.
>>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
>>>>> API and CLI. 2 patches are pending
>>>>> which will be closed by Feature freeze date.
>>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>>>> ATSv2. Patch in progress.
>>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement
>>>>> works.
>>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>>>>> Storage. In progress.
>>>>>
>>>>> 3. Tentative features:
>>>>>
>>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging
>>>>> to be done before Aug 2018.
>>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>>>> Challenging as more discussions are on-going.
>>>>>
>>>>> *Summary of 3.2.0 issues status:*
>>>>>
>>>>> 39 Blocker and Critical issues [1] are open, I am checking with owners
>>>>> to get status on each of them to get in by Code Freeze date.
>>>>>
>>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target Version/s" =
>>>>> 3.2.0 ORDER BY priority DESC
>>>>>
>>>>> Thanks,
>>>>> Sunil
>>>>>
>>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
>>>>>
>>>>>> Thanks Subru for the thoughts.
>>>>>> One of the main reason for a major release is to push out critical
>>>>>> features with a faster cadence to the users. If we are pulling more and
>>>>>> more different types of features to a minor release, that branch will
>>>>>> become more destabilized and it may be tough to say that 3.1.2 is stable
>>>>>> that 3.1.1 for eg. We always tend to improve and stabilize features in
>>>>>> subsequent minor release.
>>>>>> For few companies, it makes sense to push out these new features
>>>>>> faster to make a reach to the users. Adding to the point to the backporting
>>>>>> issues, I agree that its a pain and we can workaround that with some git
>>>>>> scripts. If we can make such scripts available to committers, backport will
>>>>>> be seem-less across branches and we can achieve the faster release cadence
>>>>>> also.
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> - Sunil
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Sunil for volunteering to lead the release effort. I am
>>>>>>> generally
>>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
>>>>>>> already
>>>>>>> have too many branches to be maintained. I already see many commits
>>>>>>> are in
>>>>>>> different branches with no apparent rationale, for e.g: 3.1 has
>>>>>>> commits
>>>>>>> which are absent in 3.0 etc.
>>>>>>>
>>>>>>> Additionally AFAIK 3.x has not been deployed in any major production
>>>>>>> setting so the cost of adding features should be minimal.
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>> -Subru
>>>>>>>
>>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
>>>>>>>
>>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>>>>>> >
>>>>>>> > Yes, important changes and features are much needed, hence we will
>>>>>>> be
>>>>>>> > keeping the door open for them as possible. Also considering few
>>>>>>> more
>>>>>>> > offline requests from other folks, I think extending the timeframe
>>>>>>> by
>>>>>>> > couple of weeks makes sense (including a second RC buffer) and
>>>>>>> this should
>>>>>>> > ideally help us to ship this by September itself.
>>>>>>> >
>>>>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
>>>>>>> >
>>>>>>> > - Feature freeze date : all features to merge by August 21, 2018.
>>>>>>> >
>>>>>>> > - Code freeze date : blockers/critical only, no improvements and
>>>>>>> non
>>>>>>> > blocker/critical
>>>>>>> >
>>>>>>> > bug-fixes August 31, 2018.
>>>>>>> >
>>>>>>> > - Release date: September 15, 2018
>>>>>>> >
>>>>>>> > Thank Eric and Zian, I think Wangda has already answered your
>>>>>>> questions.
>>>>>>> >
>>>>>>> > Thanks
>>>>>>> > Sunil
>>>>>>> >
>>>>>>> >
>>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for
>>>>>>> that.
>>>>>>> > >
>>>>>>> > > To concerns from Steve,
>>>>>>> > >
>>>>>>> > > It is a good idea to keep the door open to get important changes
>>>>>>> /
>>>>>>> > > features in before cutoff. I would prefer to keep the proposed
>>>>>>> release
>>>>>>> > date
>>>>>>> > > to make sure things can happen earlier instead of last minute
>>>>>>> and we all
>>>>>>> > > know that releases are always get delayed :). I'm also fine if
>>>>>>> we want
>>>>>>> > get
>>>>>>> > > another several weeks time.
>>>>>>> > >
>>>>>>> > > Regarding of 3.3 release, I would suggest doing that before
>>>>>>> thanksgiving.
>>>>>>> > > Do you think is it good or too early / late?
>>>>>>> > >
>>>>>>> > > Eric,
>>>>>>> > >
>>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can
>>>>>>> get merged
>>>>>>> > > in time, we probably not need the YARN-8220.
>>>>>>> > >
>>>>>>> > > Sunil,
>>>>>>> > >
>>>>>>> > > Could u update
>>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>>>>>>> > Roadmap
>>>>>>> > > with proposed plan as well? We can fill feature list first
>>>>>>> before getting
>>>>>>> > > consensus of time.
>>>>>>> > >
>>>>>>> > > Thanks,
>>>>>>> > > Wangda
>>>>>>> > >
>>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>>>>>> <fabbri@cloudera.com.invalid
>>>>>>> > >
>>>>>>> > > wrote:
>>>>>>> > >
>>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>>>>>> stevel@hortonworks.com>
>>>>>>> > >> wrote:
>>>>>>> > >>
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
>>>>>>> > >> > sunilg@apache.org>> wrote:
>>>>>>> > >> >
>>>>>>> > >> > I would also would like to take this opportunity to come up
>>>>>>> with a
>>>>>>> > >> detailed
>>>>>>> > >> > plan.
>>>>>>> > >> >
>>>>>>> > >> > - Feature freeze date : all features should be merged by
>>>>>>> August 10,
>>>>>>> > >> 2018.
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > <snip>
>>>>>>> > >>
>>>>>>> > >> >
>>>>>>> > >> > Please let me know if I missed any features targeted to 3.2
>>>>>>> per this
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>>>>>> > >> >
>>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen
>>>>>>> 2
>>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>>>>>> > >> >
>>>>>>> > >> > I don't think this is quite ready yet, I've been doing work
>>>>>>> on it, but
>>>>>>> > >> if
>>>>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
>>>>>>> reviews on
>>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>>>>>> > >> >
>>>>>>> > >> > I've uprated that to a blocker feature; will review the S3 &
>>>>>>> S3Guard
>>>>>>> > >> JIRAs
>>>>>>> > >> > to see which of those are blocking. Then there are some
>>>>>>> pressing
>>>>>>> > "guave,
>>>>>>> > >> > java 9 prep"
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> I can help with this part if you like.
>>>>>>> > >>
>>>>>>> > >>
>>>>>>> > >>
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > timeline. I would like to volunteer myself as release manager
>>>>>>> of 3.2.0
>>>>>>> > >> > release.
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> > well volunteered!
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> Yes, thank you for stepping up.
>>>>>>> > >>
>>>>>>> > >>
>>>>>>> > >> >
>>>>>>> > >> > I think this raises a good q: what timetable should we have
>>>>>>> for the
>>>>>>> > >> 3.2. &
>>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having the
>>>>>>> outline
>>>>>>> > >> time
>>>>>>> > >> > from the 3.2 to the 3.3 release means that there's less
>>>>>>> concern about
>>>>>>> > >> > things not making the 3.2 dealine
>>>>>>> > >> >
>>>>>>> > >> > -Steve
>>>>>>> > >> >
>>>>>>> > >> >
>>>>>>> > >> Good idea to mitigate the short deadline.
>>>>>>> > >>
>>>>>>> > >> -AF
>>>>>>> > >>
>>>>>>> > >
>>>>>>> >
>>>>>>>
>>>>>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi All
We are now down to the last Blocker and HADOOP-15407 is merged to trunk.
Thanks for the support.
*Plan for RC*
3.2 branch cut and reset trunk : *25th Tuesday*
RC0 for 3.2: *28th Friday*
Thank You
Sunil
On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
> Hi All
>
> We are down to 3 Blockers and 4 Critical now. Thanks all of you for
> helping in this. I am following up on these tickets, once its closed we
> will cut the 3.2 branch.
>
> Thanks
> Sunil Govindan
>
>
> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
>
>> Hi All,
>>
>> Inline with the original 3.2 communication proposal dated 17th July 2018,
>> I would like to provide more updates.
>>
>> We are approaching previously proposed code freeze date (September 14,
>> 2018). So I would like to cut 3.2 branch on 17th Sept and point existing
>> trunk to 3.3 if there are no issues.
>>
>> *Current Release Plan:*
>> Feature freeze date : all features to merge by September 7, 2018.
>> Code freeze date : blockers/critical only, no improvements and
>> blocker/critical bug-fixes September 14, 2018.
>> Release date: September 28, 2018
>>
>> If any critical/blocker tickets which are targeted to 3.2.0, we need to
>> backport to 3.2 post branch cut.
>>
>> Here's an updated 3.2.0 feature status:
>>
>> 1. Merged & Completed features:
>>
>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
>> Initial cut.
>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>> and CLI.
>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.
>>
>> 2. Features close to finish:
>>
>> - (Steve) S3Guard Phase III. Close to commit.
>> - (Steve) S3a phase V. Close to commit.
>> - (Steve) Support Windows Azure Storage. Close to commit.
>>
>> 3. Tentative/Cancelled features for 3.2:
>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>> ATSv2. Patch in progress.
>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>> be done before Aug 2018.
>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> Challenging as more discussions are on-going.
>>
>> *Summary of 3.2.0 issues status:*
>> 19 Blocker and Critical issues [1] are open, I am following up with
>> owners to get status on each of them to get in by Code Freeze date.
>>
>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>> BY priority DESC
>>
>> Thanks,
>> Sunil
>>
>>
>>
>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
>>
>>> Hi All,
>>>
>>> Inline with earlier communication dated 17th July 2018, I would like to
>>> provide some updates.
>>>
>>> We are approaching previously proposed code freeze date (Aug 31).
>>>
>>> One of the critical feature Node Attributes feature merge
>>> discussion/vote is ongoing. Also few other Blocker bugs need a bit more
>>> time. With regard to this, suggesting to push the feature/code freeze for 2
>>> more weeks to accommodate these jiras too.
>>>
>>> Proposing Updated changes in plan inline with this:
>>> Feature freeze date : all features to merge by September 7, 2018.
>>> Code freeze date : blockers/critical only, no improvements and
>>> blocker/critical bug-fixes September 14, 2018.
>>> Release date: September 28, 2018
>>>
>>> If any features in branch which are targeted to 3.2.0, please reply to
>>> this email thread.
>>>
>>> *Here's an updated 3.2.0 feature status:*
>>>
>>> 1. Merged & Completed features:
>>>
>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> workloads Initial cut.
>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> Scheduler.
>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>>> and CLI.
>>>
>>> 2. Features close to finish:
>>>
>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
>>> Ongoing.
>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>> ATSv2. Patch in progress.
>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
>>> In progress.
>>>
>>> 3. Tentative features:
>>>
>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>>> be done before Aug 2018.
>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> Challenging as more discussions are on-going.
>>>
>>> *Summary of 3.2.0 issues status:*
>>>
>>> 26 Blocker and Critical issues [1] are open, I am following up with
>>> owners to get status on each of them to get in by Code Freeze date.
>>>
>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>>> BY priority DESC
>>>
>>> Thanks,
>>> Sunil
>>>
>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
>>>
>>>> Hi All,
>>>>
>>>> Thanks for the feedbacks. Inline with earlier communication dated 17th
>>>> July 2018, I would like to provide some updates.
>>>>
>>>> We are approaching previously proposed feature freeze date (Aug 21,
>>>> about 7 days from today).
>>>> If any features in branch which are targeted to 3.2.0, please reply to
>>>> this email thread.
>>>> Steve has mentioned about the s3 features which will come close to Code
>>>> Freeze Date (Aug 31st).
>>>>
>>>> *Here's an updated 3.2.0 feature status:*
>>>>
>>>> 1. Merged & Completed features:
>>>>
>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>>> workloads Initial cut.
>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>>>
>>>> 2. Features close to finish:
>>>>
>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
>>>> patches are all in, only one last
>>>> patch is in review state.
>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>>> Scheduler. Close to commit.
>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
>>>> API and CLI. 2 patches are pending
>>>> which will be closed by Feature freeze date.
>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>>> ATSv2. Patch in progress.
>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>>>> Storage. In progress.
>>>>
>>>> 3. Tentative features:
>>>>
>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>>>> be done before Aug 2018.
>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>>> Challenging as more discussions are on-going.
>>>>
>>>> *Summary of 3.2.0 issues status:*
>>>>
>>>> 39 Blocker and Critical issues [1] are open, I am checking with owners
>>>> to get status on each of them to get in by Code Freeze date.
>>>>
>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target Version/s" =
>>>> 3.2.0 ORDER BY priority DESC
>>>>
>>>> Thanks,
>>>> Sunil
>>>>
>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
>>>>
>>>>> Thanks Subru for the thoughts.
>>>>> One of the main reason for a major release is to push out critical
>>>>> features with a faster cadence to the users. If we are pulling more and
>>>>> more different types of features to a minor release, that branch will
>>>>> become more destabilized and it may be tough to say that 3.1.2 is stable
>>>>> that 3.1.1 for eg. We always tend to improve and stabilize features in
>>>>> subsequent minor release.
>>>>> For few companies, it makes sense to push out these new features
>>>>> faster to make a reach to the users. Adding to the point to the backporting
>>>>> issues, I agree that its a pain and we can workaround that with some git
>>>>> scripts. If we can make such scripts available to committers, backport will
>>>>> be seem-less across branches and we can achieve the faster release cadence
>>>>> also.
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> - Sunil
>>>>>
>>>>>
>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Thanks Sunil for volunteering to lead the release effort. I am
>>>>>> generally
>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
>>>>>> already
>>>>>> have too many branches to be maintained. I already see many commits
>>>>>> are in
>>>>>> different branches with no apparent rationale, for e.g: 3.1 has
>>>>>> commits
>>>>>> which are absent in 3.0 etc.
>>>>>>
>>>>>> Additionally AFAIK 3.x has not been deployed in any major production
>>>>>> setting so the cost of adding features should be minimal.
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> -Subru
>>>>>>
>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
>>>>>>
>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>>>>> >
>>>>>> > Yes, important changes and features are much needed, hence we will
>>>>>> be
>>>>>> > keeping the door open for them as possible. Also considering few
>>>>>> more
>>>>>> > offline requests from other folks, I think extending the timeframe
>>>>>> by
>>>>>> > couple of weeks makes sense (including a second RC buffer) and this
>>>>>> should
>>>>>> > ideally help us to ship this by September itself.
>>>>>> >
>>>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
>>>>>> >
>>>>>> > - Feature freeze date : all features to merge by August 21, 2018.
>>>>>> >
>>>>>> > - Code freeze date : blockers/critical only, no improvements and non
>>>>>> > blocker/critical
>>>>>> >
>>>>>> > bug-fixes August 31, 2018.
>>>>>> >
>>>>>> > - Release date: September 15, 2018
>>>>>> >
>>>>>> > Thank Eric and Zian, I think Wangda has already answered your
>>>>>> questions.
>>>>>> >
>>>>>> > Thanks
>>>>>> > Sunil
>>>>>> >
>>>>>> >
>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for
>>>>>> that.
>>>>>> > >
>>>>>> > > To concerns from Steve,
>>>>>> > >
>>>>>> > > It is a good idea to keep the door open to get important changes /
>>>>>> > > features in before cutoff. I would prefer to keep the proposed
>>>>>> release
>>>>>> > date
>>>>>> > > to make sure things can happen earlier instead of last minute and
>>>>>> we all
>>>>>> > > know that releases are always get delayed :). I'm also fine if we
>>>>>> want
>>>>>> > get
>>>>>> > > another several weeks time.
>>>>>> > >
>>>>>> > > Regarding of 3.3 release, I would suggest doing that before
>>>>>> thanksgiving.
>>>>>> > > Do you think is it good or too early / late?
>>>>>> > >
>>>>>> > > Eric,
>>>>>> > >
>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
>>>>>> merged
>>>>>> > > in time, we probably not need the YARN-8220.
>>>>>> > >
>>>>>> > > Sunil,
>>>>>> > >
>>>>>> > > Could u update
>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>>>>>> > Roadmap
>>>>>> > > with proposed plan as well? We can fill feature list first before
>>>>>> getting
>>>>>> > > consensus of time.
>>>>>> > >
>>>>>> > > Thanks,
>>>>>> > > Wangda
>>>>>> > >
>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>>>>> <fabbri@cloudera.com.invalid
>>>>>> > >
>>>>>> > > wrote:
>>>>>> > >
>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>>>>> stevel@hortonworks.com>
>>>>>> > >> wrote:
>>>>>> > >>
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
>>>>>> > >> > sunilg@apache.org>> wrote:
>>>>>> > >> >
>>>>>> > >> > I would also would like to take this opportunity to come up
>>>>>> with a
>>>>>> > >> detailed
>>>>>> > >> > plan.
>>>>>> > >> >
>>>>>> > >> > - Feature freeze date : all features should be merged by
>>>>>> August 10,
>>>>>> > >> 2018.
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > <snip>
>>>>>> > >>
>>>>>> > >> >
>>>>>> > >> > Please let me know if I missed any features targeted to 3.2
>>>>>> per this
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>>>>> > >> >
>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen 2
>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>>>>> > >> >
>>>>>> > >> > I don't think this is quite ready yet, I've been doing work on
>>>>>> it, but
>>>>>> > >> if
>>>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
>>>>>> reviews on
>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>>>>> > >> >
>>>>>> > >> > I've uprated that to a blocker feature; will review the S3 &
>>>>>> S3Guard
>>>>>> > >> JIRAs
>>>>>> > >> > to see which of those are blocking. Then there are some
>>>>>> pressing
>>>>>> > "guave,
>>>>>> > >> > java 9 prep"
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> I can help with this part if you like.
>>>>>> > >>
>>>>>> > >>
>>>>>> > >>
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > timeline. I would like to volunteer myself as release manager
>>>>>> of 3.2.0
>>>>>> > >> > release.
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > well volunteered!
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> Yes, thank you for stepping up.
>>>>>> > >>
>>>>>> > >>
>>>>>> > >> >
>>>>>> > >> > I think this raises a good q: what timetable should we have
>>>>>> for the
>>>>>> > >> 3.2. &
>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having the
>>>>>> outline
>>>>>> > >> time
>>>>>> > >> > from the 3.2 to the 3.3 release means that there's less
>>>>>> concern about
>>>>>> > >> > things not making the 3.2 dealine
>>>>>> > >> >
>>>>>> > >> > -Steve
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> Good idea to mitigate the short deadline.
>>>>>> > >>
>>>>>> > >> -AF
>>>>>> > >>
>>>>>> > >
>>>>>> >
>>>>>>
>>>>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi All
We are now down to the last Blocker and HADOOP-15407 is merged to trunk.
Thanks for the support.
*Plan for RC*
3.2 branch cut and reset trunk : *25th Tuesday*
RC0 for 3.2: *28th Friday*
Thank You
Sunil
On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
> Hi All
>
> We are down to 3 Blockers and 4 Critical now. Thanks all of you for
> helping in this. I am following up on these tickets, once its closed we
> will cut the 3.2 branch.
>
> Thanks
> Sunil Govindan
>
>
> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
>
>> Hi All,
>>
>> Inline with the original 3.2 communication proposal dated 17th July 2018,
>> I would like to provide more updates.
>>
>> We are approaching previously proposed code freeze date (September 14,
>> 2018). So I would like to cut 3.2 branch on 17th Sept and point existing
>> trunk to 3.3 if there are no issues.
>>
>> *Current Release Plan:*
>> Feature freeze date : all features to merge by September 7, 2018.
>> Code freeze date : blockers/critical only, no improvements and
>> blocker/critical bug-fixes September 14, 2018.
>> Release date: September 28, 2018
>>
>> If any critical/blocker tickets which are targeted to 3.2.0, we need to
>> backport to 3.2 post branch cut.
>>
>> Here's an updated 3.2.0 feature status:
>>
>> 1. Merged & Completed features:
>>
>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
>> Initial cut.
>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>> and CLI.
>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.
>>
>> 2. Features close to finish:
>>
>> - (Steve) S3Guard Phase III. Close to commit.
>> - (Steve) S3a phase V. Close to commit.
>> - (Steve) Support Windows Azure Storage. Close to commit.
>>
>> 3. Tentative/Cancelled features for 3.2:
>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>> ATSv2. Patch in progress.
>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>> be done before Aug 2018.
>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> Challenging as more discussions are on-going.
>>
>> *Summary of 3.2.0 issues status:*
>> 19 Blocker and Critical issues [1] are open, I am following up with
>> owners to get status on each of them to get in by Code Freeze date.
>>
>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>> BY priority DESC
>>
>> Thanks,
>> Sunil
>>
>>
>>
>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
>>
>>> Hi All,
>>>
>>> Inline with earlier communication dated 17th July 2018, I would like to
>>> provide some updates.
>>>
>>> We are approaching previously proposed code freeze date (Aug 31).
>>>
>>> One of the critical feature Node Attributes feature merge
>>> discussion/vote is ongoing. Also few other Blocker bugs need a bit more
>>> time. With regard to this, suggesting to push the feature/code freeze for 2
>>> more weeks to accommodate these jiras too.
>>>
>>> Proposing Updated changes in plan inline with this:
>>> Feature freeze date : all features to merge by September 7, 2018.
>>> Code freeze date : blockers/critical only, no improvements and
>>> blocker/critical bug-fixes September 14, 2018.
>>> Release date: September 28, 2018
>>>
>>> If any features in branch which are targeted to 3.2.0, please reply to
>>> this email thread.
>>>
>>> *Here's an updated 3.2.0 feature status:*
>>>
>>> 1. Merged & Completed features:
>>>
>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> workloads Initial cut.
>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> Scheduler.
>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>>> and CLI.
>>>
>>> 2. Features close to finish:
>>>
>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
>>> Ongoing.
>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>> ATSv2. Patch in progress.
>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
>>> In progress.
>>>
>>> 3. Tentative features:
>>>
>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>>> be done before Aug 2018.
>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> Challenging as more discussions are on-going.
>>>
>>> *Summary of 3.2.0 issues status:*
>>>
>>> 26 Blocker and Critical issues [1] are open, I am following up with
>>> owners to get status on each of them to get in by Code Freeze date.
>>>
>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>>> BY priority DESC
>>>
>>> Thanks,
>>> Sunil
>>>
>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
>>>
>>>> Hi All,
>>>>
>>>> Thanks for the feedbacks. Inline with earlier communication dated 17th
>>>> July 2018, I would like to provide some updates.
>>>>
>>>> We are approaching previously proposed feature freeze date (Aug 21,
>>>> about 7 days from today).
>>>> If any features in branch which are targeted to 3.2.0, please reply to
>>>> this email thread.
>>>> Steve has mentioned about the s3 features which will come close to Code
>>>> Freeze Date (Aug 31st).
>>>>
>>>> *Here's an updated 3.2.0 feature status:*
>>>>
>>>> 1. Merged & Completed features:
>>>>
>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>>> workloads Initial cut.
>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>>>
>>>> 2. Features close to finish:
>>>>
>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
>>>> patches are all in, only one last
>>>> patch is in review state.
>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>>> Scheduler. Close to commit.
>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
>>>> API and CLI. 2 patches are pending
>>>> which will be closed by Feature freeze date.
>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>>> ATSv2. Patch in progress.
>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>>>> Storage. In progress.
>>>>
>>>> 3. Tentative features:
>>>>
>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>>>> be done before Aug 2018.
>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>>> Challenging as more discussions are on-going.
>>>>
>>>> *Summary of 3.2.0 issues status:*
>>>>
>>>> 39 Blocker and Critical issues [1] are open, I am checking with owners
>>>> to get status on each of them to get in by Code Freeze date.
>>>>
>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target Version/s" =
>>>> 3.2.0 ORDER BY priority DESC
>>>>
>>>> Thanks,
>>>> Sunil
>>>>
>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
>>>>
>>>>> Thanks Subru for the thoughts.
>>>>> One of the main reason for a major release is to push out critical
>>>>> features with a faster cadence to the users. If we are pulling more and
>>>>> more different types of features to a minor release, that branch will
>>>>> become more destabilized and it may be tough to say that 3.1.2 is stable
>>>>> that 3.1.1 for eg. We always tend to improve and stabilize features in
>>>>> subsequent minor release.
>>>>> For few companies, it makes sense to push out these new features
>>>>> faster to make a reach to the users. Adding to the point to the backporting
>>>>> issues, I agree that its a pain and we can workaround that with some git
>>>>> scripts. If we can make such scripts available to committers, backport will
>>>>> be seem-less across branches and we can achieve the faster release cadence
>>>>> also.
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> - Sunil
>>>>>
>>>>>
>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Thanks Sunil for volunteering to lead the release effort. I am
>>>>>> generally
>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
>>>>>> already
>>>>>> have too many branches to be maintained. I already see many commits
>>>>>> are in
>>>>>> different branches with no apparent rationale, for e.g: 3.1 has
>>>>>> commits
>>>>>> which are absent in 3.0 etc.
>>>>>>
>>>>>> Additionally AFAIK 3.x has not been deployed in any major production
>>>>>> setting so the cost of adding features should be minimal.
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> -Subru
>>>>>>
>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
>>>>>>
>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>>>>> >
>>>>>> > Yes, important changes and features are much needed, hence we will
>>>>>> be
>>>>>> > keeping the door open for them as possible. Also considering few
>>>>>> more
>>>>>> > offline requests from other folks, I think extending the timeframe
>>>>>> by
>>>>>> > couple of weeks makes sense (including a second RC buffer) and this
>>>>>> should
>>>>>> > ideally help us to ship this by September itself.
>>>>>> >
>>>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
>>>>>> >
>>>>>> > - Feature freeze date : all features to merge by August 21, 2018.
>>>>>> >
>>>>>> > - Code freeze date : blockers/critical only, no improvements and non
>>>>>> > blocker/critical
>>>>>> >
>>>>>> > bug-fixes August 31, 2018.
>>>>>> >
>>>>>> > - Release date: September 15, 2018
>>>>>> >
>>>>>> > Thank Eric and Zian, I think Wangda has already answered your
>>>>>> questions.
>>>>>> >
>>>>>> > Thanks
>>>>>> > Sunil
>>>>>> >
>>>>>> >
>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for
>>>>>> that.
>>>>>> > >
>>>>>> > > To concerns from Steve,
>>>>>> > >
>>>>>> > > It is a good idea to keep the door open to get important changes /
>>>>>> > > features in before cutoff. I would prefer to keep the proposed
>>>>>> release
>>>>>> > date
>>>>>> > > to make sure things can happen earlier instead of last minute and
>>>>>> we all
>>>>>> > > know that releases are always get delayed :). I'm also fine if we
>>>>>> want
>>>>>> > get
>>>>>> > > another several weeks time.
>>>>>> > >
>>>>>> > > Regarding of 3.3 release, I would suggest doing that before
>>>>>> thanksgiving.
>>>>>> > > Do you think is it good or too early / late?
>>>>>> > >
>>>>>> > > Eric,
>>>>>> > >
>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
>>>>>> merged
>>>>>> > > in time, we probably not need the YARN-8220.
>>>>>> > >
>>>>>> > > Sunil,
>>>>>> > >
>>>>>> > > Could u update
>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>>>>>> > Roadmap
>>>>>> > > with proposed plan as well? We can fill feature list first before
>>>>>> getting
>>>>>> > > consensus of time.
>>>>>> > >
>>>>>> > > Thanks,
>>>>>> > > Wangda
>>>>>> > >
>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>>>>> <fabbri@cloudera.com.invalid
>>>>>> > >
>>>>>> > > wrote:
>>>>>> > >
>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>>>>> stevel@hortonworks.com>
>>>>>> > >> wrote:
>>>>>> > >>
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
>>>>>> > >> > sunilg@apache.org>> wrote:
>>>>>> > >> >
>>>>>> > >> > I would also would like to take this opportunity to come up
>>>>>> with a
>>>>>> > >> detailed
>>>>>> > >> > plan.
>>>>>> > >> >
>>>>>> > >> > - Feature freeze date : all features should be merged by
>>>>>> August 10,
>>>>>> > >> 2018.
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > <snip>
>>>>>> > >>
>>>>>> > >> >
>>>>>> > >> > Please let me know if I missed any features targeted to 3.2
>>>>>> per this
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>>>>> > >> >
>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen 2
>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>>>>> > >> >
>>>>>> > >> > I don't think this is quite ready yet, I've been doing work on
>>>>>> it, but
>>>>>> > >> if
>>>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
>>>>>> reviews on
>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>>>>> > >> >
>>>>>> > >> > I've uprated that to a blocker feature; will review the S3 &
>>>>>> S3Guard
>>>>>> > >> JIRAs
>>>>>> > >> > to see which of those are blocking. Then there are some
>>>>>> pressing
>>>>>> > "guave,
>>>>>> > >> > java 9 prep"
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> I can help with this part if you like.
>>>>>> > >>
>>>>>> > >>
>>>>>> > >>
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > timeline. I would like to volunteer myself as release manager
>>>>>> of 3.2.0
>>>>>> > >> > release.
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > well volunteered!
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> Yes, thank you for stepping up.
>>>>>> > >>
>>>>>> > >>
>>>>>> > >> >
>>>>>> > >> > I think this raises a good q: what timetable should we have
>>>>>> for the
>>>>>> > >> 3.2. &
>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having the
>>>>>> outline
>>>>>> > >> time
>>>>>> > >> > from the 3.2 to the 3.3 release means that there's less
>>>>>> concern about
>>>>>> > >> > things not making the 3.2 dealine
>>>>>> > >> >
>>>>>> > >> > -Steve
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> Good idea to mitigate the short deadline.
>>>>>> > >>
>>>>>> > >> -AF
>>>>>> > >>
>>>>>> > >
>>>>>> >
>>>>>>
>>>>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi All
We are now down to the last Blocker and HADOOP-15407 is merged to trunk.
Thanks for the support.
*Plan for RC*
3.2 branch cut and reset trunk : *25th Tuesday*
RC0 for 3.2: *28th Friday*
Thank You
Sunil
On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
> Hi All
>
> We are down to 3 Blockers and 4 Critical now. Thanks all of you for
> helping in this. I am following up on these tickets, once its closed we
> will cut the 3.2 branch.
>
> Thanks
> Sunil Govindan
>
>
> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
>
>> Hi All,
>>
>> Inline with the original 3.2 communication proposal dated 17th July 2018,
>> I would like to provide more updates.
>>
>> We are approaching previously proposed code freeze date (September 14,
>> 2018). So I would like to cut 3.2 branch on 17th Sept and point existing
>> trunk to 3.3 if there are no issues.
>>
>> *Current Release Plan:*
>> Feature freeze date : all features to merge by September 7, 2018.
>> Code freeze date : blockers/critical only, no improvements and
>> blocker/critical bug-fixes September 14, 2018.
>> Release date: September 28, 2018
>>
>> If any critical/blocker tickets which are targeted to 3.2.0, we need to
>> backport to 3.2 post branch cut.
>>
>> Here's an updated 3.2.0 feature status:
>>
>> 1. Merged & Completed features:
>>
>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
>> Initial cut.
>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>> and CLI.
>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.
>>
>> 2. Features close to finish:
>>
>> - (Steve) S3Guard Phase III. Close to commit.
>> - (Steve) S3a phase V. Close to commit.
>> - (Steve) Support Windows Azure Storage. Close to commit.
>>
>> 3. Tentative/Cancelled features for 3.2:
>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>> ATSv2. Patch in progress.
>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>> be done before Aug 2018.
>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> Challenging as more discussions are on-going.
>>
>> *Summary of 3.2.0 issues status:*
>> 19 Blocker and Critical issues [1] are open, I am following up with
>> owners to get status on each of them to get in by Code Freeze date.
>>
>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>> BY priority DESC
>>
>> Thanks,
>> Sunil
>>
>>
>>
>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
>>
>>> Hi All,
>>>
>>> Inline with earlier communication dated 17th July 2018, I would like to
>>> provide some updates.
>>>
>>> We are approaching previously proposed code freeze date (Aug 31).
>>>
>>> One of the critical feature Node Attributes feature merge
>>> discussion/vote is ongoing. Also few other Blocker bugs need a bit more
>>> time. With regard to this, suggesting to push the feature/code freeze for 2
>>> more weeks to accommodate these jiras too.
>>>
>>> Proposing Updated changes in plan inline with this:
>>> Feature freeze date : all features to merge by September 7, 2018.
>>> Code freeze date : blockers/critical only, no improvements and
>>> blocker/critical bug-fixes September 14, 2018.
>>> Release date: September 28, 2018
>>>
>>> If any features in branch which are targeted to 3.2.0, please reply to
>>> this email thread.
>>>
>>> *Here's an updated 3.2.0 feature status:*
>>>
>>> 1. Merged & Completed features:
>>>
>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> workloads Initial cut.
>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> Scheduler.
>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>>> and CLI.
>>>
>>> 2. Features close to finish:
>>>
>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
>>> Ongoing.
>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>> ATSv2. Patch in progress.
>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
>>> In progress.
>>>
>>> 3. Tentative features:
>>>
>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>>> be done before Aug 2018.
>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> Challenging as more discussions are on-going.
>>>
>>> *Summary of 3.2.0 issues status:*
>>>
>>> 26 Blocker and Critical issues [1] are open, I am following up with
>>> owners to get status on each of them to get in by Code Freeze date.
>>>
>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>>> BY priority DESC
>>>
>>> Thanks,
>>> Sunil
>>>
>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
>>>
>>>> Hi All,
>>>>
>>>> Thanks for the feedbacks. Inline with earlier communication dated 17th
>>>> July 2018, I would like to provide some updates.
>>>>
>>>> We are approaching previously proposed feature freeze date (Aug 21,
>>>> about 7 days from today).
>>>> If any features in branch which are targeted to 3.2.0, please reply to
>>>> this email thread.
>>>> Steve has mentioned about the s3 features which will come close to Code
>>>> Freeze Date (Aug 31st).
>>>>
>>>> *Here's an updated 3.2.0 feature status:*
>>>>
>>>> 1. Merged & Completed features:
>>>>
>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>>> workloads Initial cut.
>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>>>
>>>> 2. Features close to finish:
>>>>
>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
>>>> patches are all in, only one last
>>>> patch is in review state.
>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>>> Scheduler. Close to commit.
>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
>>>> API and CLI. 2 patches are pending
>>>> which will be closed by Feature freeze date.
>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>>> ATSv2. Patch in progress.
>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>>>> Storage. In progress.
>>>>
>>>> 3. Tentative features:
>>>>
>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>>>> be done before Aug 2018.
>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>>> Challenging as more discussions are on-going.
>>>>
>>>> *Summary of 3.2.0 issues status:*
>>>>
>>>> 39 Blocker and Critical issues [1] are open, I am checking with owners
>>>> to get status on each of them to get in by Code Freeze date.
>>>>
>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target Version/s" =
>>>> 3.2.0 ORDER BY priority DESC
>>>>
>>>> Thanks,
>>>> Sunil
>>>>
>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
>>>>
>>>>> Thanks Subru for the thoughts.
>>>>> One of the main reason for a major release is to push out critical
>>>>> features with a faster cadence to the users. If we are pulling more and
>>>>> more different types of features to a minor release, that branch will
>>>>> become more destabilized and it may be tough to say that 3.1.2 is stable
>>>>> that 3.1.1 for eg. We always tend to improve and stabilize features in
>>>>> subsequent minor release.
>>>>> For few companies, it makes sense to push out these new features
>>>>> faster to make a reach to the users. Adding to the point to the backporting
>>>>> issues, I agree that its a pain and we can workaround that with some git
>>>>> scripts. If we can make such scripts available to committers, backport will
>>>>> be seem-less across branches and we can achieve the faster release cadence
>>>>> also.
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> - Sunil
>>>>>
>>>>>
>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Thanks Sunil for volunteering to lead the release effort. I am
>>>>>> generally
>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
>>>>>> already
>>>>>> have too many branches to be maintained. I already see many commits
>>>>>> are in
>>>>>> different branches with no apparent rationale, for e.g: 3.1 has
>>>>>> commits
>>>>>> which are absent in 3.0 etc.
>>>>>>
>>>>>> Additionally AFAIK 3.x has not been deployed in any major production
>>>>>> setting so the cost of adding features should be minimal.
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> -Subru
>>>>>>
>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
>>>>>>
>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>>>>> >
>>>>>> > Yes, important changes and features are much needed, hence we will
>>>>>> be
>>>>>> > keeping the door open for them as possible. Also considering few
>>>>>> more
>>>>>> > offline requests from other folks, I think extending the timeframe
>>>>>> by
>>>>>> > couple of weeks makes sense (including a second RC buffer) and this
>>>>>> should
>>>>>> > ideally help us to ship this by September itself.
>>>>>> >
>>>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
>>>>>> >
>>>>>> > - Feature freeze date : all features to merge by August 21, 2018.
>>>>>> >
>>>>>> > - Code freeze date : blockers/critical only, no improvements and non
>>>>>> > blocker/critical
>>>>>> >
>>>>>> > bug-fixes August 31, 2018.
>>>>>> >
>>>>>> > - Release date: September 15, 2018
>>>>>> >
>>>>>> > Thank Eric and Zian, I think Wangda has already answered your
>>>>>> questions.
>>>>>> >
>>>>>> > Thanks
>>>>>> > Sunil
>>>>>> >
>>>>>> >
>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for
>>>>>> that.
>>>>>> > >
>>>>>> > > To concerns from Steve,
>>>>>> > >
>>>>>> > > It is a good idea to keep the door open to get important changes /
>>>>>> > > features in before cutoff. I would prefer to keep the proposed
>>>>>> release
>>>>>> > date
>>>>>> > > to make sure things can happen earlier instead of last minute and
>>>>>> we all
>>>>>> > > know that releases are always get delayed :). I'm also fine if we
>>>>>> want
>>>>>> > get
>>>>>> > > another several weeks time.
>>>>>> > >
>>>>>> > > Regarding of 3.3 release, I would suggest doing that before
>>>>>> thanksgiving.
>>>>>> > > Do you think is it good or too early / late?
>>>>>> > >
>>>>>> > > Eric,
>>>>>> > >
>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
>>>>>> merged
>>>>>> > > in time, we probably not need the YARN-8220.
>>>>>> > >
>>>>>> > > Sunil,
>>>>>> > >
>>>>>> > > Could u update
>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>>>>>> > Roadmap
>>>>>> > > with proposed plan as well? We can fill feature list first before
>>>>>> getting
>>>>>> > > consensus of time.
>>>>>> > >
>>>>>> > > Thanks,
>>>>>> > > Wangda
>>>>>> > >
>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>>>>> <fabbri@cloudera.com.invalid
>>>>>> > >
>>>>>> > > wrote:
>>>>>> > >
>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>>>>> stevel@hortonworks.com>
>>>>>> > >> wrote:
>>>>>> > >>
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
>>>>>> > >> > sunilg@apache.org>> wrote:
>>>>>> > >> >
>>>>>> > >> > I would also would like to take this opportunity to come up
>>>>>> with a
>>>>>> > >> detailed
>>>>>> > >> > plan.
>>>>>> > >> >
>>>>>> > >> > - Feature freeze date : all features should be merged by
>>>>>> August 10,
>>>>>> > >> 2018.
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > <snip>
>>>>>> > >>
>>>>>> > >> >
>>>>>> > >> > Please let me know if I missed any features targeted to 3.2
>>>>>> per this
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>>>>> > >> >
>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen 2
>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>>>>> > >> >
>>>>>> > >> > I don't think this is quite ready yet, I've been doing work on
>>>>>> it, but
>>>>>> > >> if
>>>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
>>>>>> reviews on
>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>>>>> > >> >
>>>>>> > >> > I've uprated that to a blocker feature; will review the S3 &
>>>>>> S3Guard
>>>>>> > >> JIRAs
>>>>>> > >> > to see which of those are blocking. Then there are some
>>>>>> pressing
>>>>>> > "guave,
>>>>>> > >> > java 9 prep"
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> I can help with this part if you like.
>>>>>> > >>
>>>>>> > >>
>>>>>> > >>
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > timeline. I would like to volunteer myself as release manager
>>>>>> of 3.2.0
>>>>>> > >> > release.
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > well volunteered!
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> Yes, thank you for stepping up.
>>>>>> > >>
>>>>>> > >>
>>>>>> > >> >
>>>>>> > >> > I think this raises a good q: what timetable should we have
>>>>>> for the
>>>>>> > >> 3.2. &
>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having the
>>>>>> outline
>>>>>> > >> time
>>>>>> > >> > from the 3.2 to the 3.3 release means that there's less
>>>>>> concern about
>>>>>> > >> > things not making the 3.2 dealine
>>>>>> > >> >
>>>>>> > >> > -Steve
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> Good idea to mitigate the short deadline.
>>>>>> > >>
>>>>>> > >> -AF
>>>>>> > >>
>>>>>> > >
>>>>>> >
>>>>>>
>>>>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi All
We are now down to the last Blocker and HADOOP-15407 is merged to trunk.
Thanks for the support.
*Plan for RC*
3.2 branch cut and reset trunk : *25th Tuesday*
RC0 for 3.2: *28th Friday*
Thank You
Sunil
On Mon, Sep 17, 2018 at 3:21 PM Sunil G <su...@apache.org> wrote:
> Hi All
>
> We are down to 3 Blockers and 4 Critical now. Thanks all of you for
> helping in this. I am following up on these tickets, once its closed we
> will cut the 3.2 branch.
>
> Thanks
> Sunil Govindan
>
>
> On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
>
>> Hi All,
>>
>> Inline with the original 3.2 communication proposal dated 17th July 2018,
>> I would like to provide more updates.
>>
>> We are approaching previously proposed code freeze date (September 14,
>> 2018). So I would like to cut 3.2 branch on 17th Sept and point existing
>> trunk to 3.3 if there are no issues.
>>
>> *Current Release Plan:*
>> Feature freeze date : all features to merge by September 7, 2018.
>> Code freeze date : blockers/critical only, no improvements and
>> blocker/critical bug-fixes September 14, 2018.
>> Release date: September 28, 2018
>>
>> If any critical/blocker tickets which are targeted to 3.2.0, we need to
>> backport to 3.2 post branch cut.
>>
>> Here's an updated 3.2.0 feature status:
>>
>> 1. Merged & Completed features:
>>
>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
>> Initial cut.
>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>> and CLI.
>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
>> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.
>>
>> 2. Features close to finish:
>>
>> - (Steve) S3Guard Phase III. Close to commit.
>> - (Steve) S3a phase V. Close to commit.
>> - (Steve) Support Windows Azure Storage. Close to commit.
>>
>> 3. Tentative/Cancelled features for 3.2:
>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>> ATSv2. Patch in progress.
>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>> be done before Aug 2018.
>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> Challenging as more discussions are on-going.
>>
>> *Summary of 3.2.0 issues status:*
>> 19 Blocker and Critical issues [1] are open, I am following up with
>> owners to get status on each of them to get in by Code Freeze date.
>>
>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>> BY priority DESC
>>
>> Thanks,
>> Sunil
>>
>>
>>
>> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
>>
>>> Hi All,
>>>
>>> Inline with earlier communication dated 17th July 2018, I would like to
>>> provide some updates.
>>>
>>> We are approaching previously proposed code freeze date (Aug 31).
>>>
>>> One of the critical feature Node Attributes feature merge
>>> discussion/vote is ongoing. Also few other Blocker bugs need a bit more
>>> time. With regard to this, suggesting to push the feature/code freeze for 2
>>> more weeks to accommodate these jiras too.
>>>
>>> Proposing Updated changes in plan inline with this:
>>> Feature freeze date : all features to merge by September 7, 2018.
>>> Code freeze date : blockers/critical only, no improvements and
>>> blocker/critical bug-fixes September 14, 2018.
>>> Release date: September 28, 2018
>>>
>>> If any features in branch which are targeted to 3.2.0, please reply to
>>> this email thread.
>>>
>>> *Here's an updated 3.2.0 feature status:*
>>>
>>> 1. Merged & Completed features:
>>>
>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> workloads Initial cut.
>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> Scheduler.
>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>>> and CLI.
>>>
>>> 2. Features close to finish:
>>>
>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
>>> Ongoing.
>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>> ATSv2. Patch in progress.
>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
>>> In progress.
>>>
>>> 3. Tentative features:
>>>
>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>>> be done before Aug 2018.
>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> Challenging as more discussions are on-going.
>>>
>>> *Summary of 3.2.0 issues status:*
>>>
>>> 26 Blocker and Critical issues [1] are open, I am following up with
>>> owners to get status on each of them to get in by Code Freeze date.
>>>
>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>>> BY priority DESC
>>>
>>> Thanks,
>>> Sunil
>>>
>>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
>>>
>>>> Hi All,
>>>>
>>>> Thanks for the feedbacks. Inline with earlier communication dated 17th
>>>> July 2018, I would like to provide some updates.
>>>>
>>>> We are approaching previously proposed feature freeze date (Aug 21,
>>>> about 7 days from today).
>>>> If any features in branch which are targeted to 3.2.0, please reply to
>>>> this email thread.
>>>> Steve has mentioned about the s3 features which will come close to Code
>>>> Freeze Date (Aug 31st).
>>>>
>>>> *Here's an updated 3.2.0 feature status:*
>>>>
>>>> 1. Merged & Completed features:
>>>>
>>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>>> workloads Initial cut.
>>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>>>
>>>> 2. Features close to finish:
>>>>
>>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major
>>>> patches are all in, only one last
>>>> patch is in review state.
>>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>>> Scheduler. Close to commit.
>>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service
>>>> API and CLI. 2 patches are pending
>>>> which will be closed by Feature freeze date.
>>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>>> ATSv2. Patch in progress.
>>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure
>>>> Storage. In progress.
>>>>
>>>> 3. Tentative features:
>>>>
>>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>>>> be done before Aug 2018.
>>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>>> Challenging as more discussions are on-going.
>>>>
>>>> *Summary of 3.2.0 issues status:*
>>>>
>>>> 39 Blocker and Critical issues [1] are open, I am checking with owners
>>>> to get status on each of them to get in by Code Freeze date.
>>>>
>>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
>>>> (Blocker, Critical) AND resolution = Unresolved AND "Target Version/s" =
>>>> 3.2.0 ORDER BY priority DESC
>>>>
>>>> Thanks,
>>>> Sunil
>>>>
>>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
>>>>
>>>>> Thanks Subru for the thoughts.
>>>>> One of the main reason for a major release is to push out critical
>>>>> features with a faster cadence to the users. If we are pulling more and
>>>>> more different types of features to a minor release, that branch will
>>>>> become more destabilized and it may be tough to say that 3.1.2 is stable
>>>>> that 3.1.1 for eg. We always tend to improve and stabilize features in
>>>>> subsequent minor release.
>>>>> For few companies, it makes sense to push out these new features
>>>>> faster to make a reach to the users. Adding to the point to the backporting
>>>>> issues, I agree that its a pain and we can workaround that with some git
>>>>> scripts. If we can make such scripts available to committers, backport will
>>>>> be seem-less across branches and we can achieve the faster release cadence
>>>>> also.
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> - Sunil
>>>>>
>>>>>
>>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Thanks Sunil for volunteering to lead the release effort. I am
>>>>>> generally
>>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
>>>>>> already
>>>>>> have too many branches to be maintained. I already see many commits
>>>>>> are in
>>>>>> different branches with no apparent rationale, for e.g: 3.1 has
>>>>>> commits
>>>>>> which are absent in 3.0 etc.
>>>>>>
>>>>>> Additionally AFAIK 3.x has not been deployed in any major production
>>>>>> setting so the cost of adding features should be minimal.
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> -Subru
>>>>>>
>>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
>>>>>>
>>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>>>>> >
>>>>>> > Yes, important changes and features are much needed, hence we will
>>>>>> be
>>>>>> > keeping the door open for them as possible. Also considering few
>>>>>> more
>>>>>> > offline requests from other folks, I think extending the timeframe
>>>>>> by
>>>>>> > couple of weeks makes sense (including a second RC buffer) and this
>>>>>> should
>>>>>> > ideally help us to ship this by September itself.
>>>>>> >
>>>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
>>>>>> >
>>>>>> > - Feature freeze date : all features to merge by August 21, 2018.
>>>>>> >
>>>>>> > - Code freeze date : blockers/critical only, no improvements and non
>>>>>> > blocker/critical
>>>>>> >
>>>>>> > bug-fixes August 31, 2018.
>>>>>> >
>>>>>> > - Release date: September 15, 2018
>>>>>> >
>>>>>> > Thank Eric and Zian, I think Wangda has already answered your
>>>>>> questions.
>>>>>> >
>>>>>> > Thanks
>>>>>> > Sunil
>>>>>> >
>>>>>> >
>>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for
>>>>>> that.
>>>>>> > >
>>>>>> > > To concerns from Steve,
>>>>>> > >
>>>>>> > > It is a good idea to keep the door open to get important changes /
>>>>>> > > features in before cutoff. I would prefer to keep the proposed
>>>>>> release
>>>>>> > date
>>>>>> > > to make sure things can happen earlier instead of last minute and
>>>>>> we all
>>>>>> > > know that releases are always get delayed :). I'm also fine if we
>>>>>> want
>>>>>> > get
>>>>>> > > another several weeks time.
>>>>>> > >
>>>>>> > > Regarding of 3.3 release, I would suggest doing that before
>>>>>> thanksgiving.
>>>>>> > > Do you think is it good or too early / late?
>>>>>> > >
>>>>>> > > Eric,
>>>>>> > >
>>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
>>>>>> merged
>>>>>> > > in time, we probably not need the YARN-8220.
>>>>>> > >
>>>>>> > > Sunil,
>>>>>> > >
>>>>>> > > Could u update
>>>>>> https://cwiki.apache.org/confluence/display/HADOOP/
>>>>>> > Roadmap
>>>>>> > > with proposed plan as well? We can fill feature list first before
>>>>>> getting
>>>>>> > > consensus of time.
>>>>>> > >
>>>>>> > > Thanks,
>>>>>> > > Wangda
>>>>>> > >
>>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>>>>> <fabbri@cloudera.com.invalid
>>>>>> > >
>>>>>> > > wrote:
>>>>>> > >
>>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>>>>> stevel@hortonworks.com>
>>>>>> > >> wrote:
>>>>>> > >>
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
>>>>>> > >> > sunilg@apache.org>> wrote:
>>>>>> > >> >
>>>>>> > >> > I would also would like to take this opportunity to come up
>>>>>> with a
>>>>>> > >> detailed
>>>>>> > >> > plan.
>>>>>> > >> >
>>>>>> > >> > - Feature freeze date : all features should be merged by
>>>>>> August 10,
>>>>>> > >> 2018.
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > <snip>
>>>>>> > >>
>>>>>> > >> >
>>>>>> > >> > Please let me know if I missed any features targeted to 3.2
>>>>>> per this
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>>>>> > >> >
>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen 2
>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>>>>> > >> >
>>>>>> > >> > I don't think this is quite ready yet, I've been doing work on
>>>>>> it, but
>>>>>> > >> if
>>>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
>>>>>> reviews on
>>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>>>>> > >> >
>>>>>> > >> > I've uprated that to a blocker feature; will review the S3 &
>>>>>> S3Guard
>>>>>> > >> JIRAs
>>>>>> > >> > to see which of those are blocking. Then there are some
>>>>>> pressing
>>>>>> > "guave,
>>>>>> > >> > java 9 prep"
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> I can help with this part if you like.
>>>>>> > >>
>>>>>> > >>
>>>>>> > >>
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > timeline. I would like to volunteer myself as release manager
>>>>>> of 3.2.0
>>>>>> > >> > release.
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> > well volunteered!
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> Yes, thank you for stepping up.
>>>>>> > >>
>>>>>> > >>
>>>>>> > >> >
>>>>>> > >> > I think this raises a good q: what timetable should we have
>>>>>> for the
>>>>>> > >> 3.2. &
>>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having the
>>>>>> outline
>>>>>> > >> time
>>>>>> > >> > from the 3.2 to the 3.3 release means that there's less
>>>>>> concern about
>>>>>> > >> > things not making the 3.2 dealine
>>>>>> > >> >
>>>>>> > >> > -Steve
>>>>>> > >> >
>>>>>> > >> >
>>>>>> > >> Good idea to mitigate the short deadline.
>>>>>> > >>
>>>>>> > >> -AF
>>>>>> > >>
>>>>>> > >
>>>>>> >
>>>>>>
>>>>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi All
We are down to 3 Blockers and 4 Critical now. Thanks all of you for helping
in this. I am following up on these tickets, once its closed we will cut
the 3.2 branch.
Thanks
Sunil Govindan
On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
> Hi All,
>
> Inline with the original 3.2 communication proposal dated 17th July 2018,
> I would like to provide more updates.
>
> We are approaching previously proposed code freeze date (September 14,
> 2018). So I would like to cut 3.2 branch on 17th Sept and point existing
> trunk to 3.3 if there are no issues.
>
> *Current Release Plan:*
> Feature freeze date : all features to merge by September 7, 2018.
> Code freeze date : blockers/critical only, no improvements and
> blocker/critical bug-fixes September 14, 2018.
> Release date: September 28, 2018
>
> If any critical/blocker tickets which are targeted to 3.2.0, we need to
> backport to 3.2 post branch cut.
>
> Here's an updated 3.2.0 feature status:
>
> 1. Merged & Completed features:
>
> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> Initial cut.
> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> and CLI.
> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.
>
> 2. Features close to finish:
>
> - (Steve) S3Guard Phase III. Close to commit.
> - (Steve) S3a phase V. Close to commit.
> - (Steve) Support Windows Azure Storage. Close to commit.
>
> 3. Tentative/Cancelled features for 3.2:
> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
> Patch in progress.
> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
> done before Aug 2018.
> - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
> as more discussions are on-going.
>
> *Summary of 3.2.0 issues status:*
> 19 Blocker and Critical issues [1] are open, I am following up with owners
> to get status on each of them to get in by Code Freeze date.
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
> BY priority DESC
>
> Thanks,
> Sunil
>
>
>
> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
>
>> Hi All,
>>
>> Inline with earlier communication dated 17th July 2018, I would like to
>> provide some updates.
>>
>> We are approaching previously proposed code freeze date (Aug 31).
>>
>> One of the critical feature Node Attributes feature merge discussion/vote
>> is ongoing. Also few other Blocker bugs need a bit more time. With regard
>> to this, suggesting to push the feature/code freeze for 2 more weeks to
>> accommodate these jiras too.
>>
>> Proposing Updated changes in plan inline with this:
>> Feature freeze date : all features to merge by September 7, 2018.
>> Code freeze date : blockers/critical only, no improvements and
>> blocker/critical bug-fixes September 14, 2018.
>> Release date: September 28, 2018
>>
>> If any features in branch which are targeted to 3.2.0, please reply to
>> this email thread.
>>
>> *Here's an updated 3.2.0 feature status:*
>>
>> 1. Merged & Completed features:
>>
>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
>> Initial cut.
>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>> and CLI.
>>
>> 2. Features close to finish:
>>
>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
>> Ongoing.
>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>> ATSv2. Patch in progress.
>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
>> In progress.
>>
>> 3. Tentative features:
>>
>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>> be done before Aug 2018.
>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> Challenging as more discussions are on-going.
>>
>> *Summary of 3.2.0 issues status:*
>>
>> 26 Blocker and Critical issues [1] are open, I am following up with
>> owners to get status on each of them to get in by Code Freeze date.
>>
>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>> BY priority DESC
>>
>> Thanks,
>> Sunil
>>
>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
>>
>>> Hi All,
>>>
>>> Thanks for the feedbacks. Inline with earlier communication dated 17th
>>> July 2018, I would like to provide some updates.
>>>
>>> We are approaching previously proposed feature freeze date (Aug 21,
>>> about 7 days from today).
>>> If any features in branch which are targeted to 3.2.0, please reply to
>>> this email thread.
>>> Steve has mentioned about the s3 features which will come close to Code
>>> Freeze Date (Aug 31st).
>>>
>>> *Here's an updated 3.2.0 feature status:*
>>>
>>> 1. Merged & Completed features:
>>>
>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> workloads Initial cut.
>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>>
>>> 2. Features close to finish:
>>>
>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major patches
>>> are all in, only one last
>>> patch is in review state.
>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> Scheduler. Close to commit.
>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>>> and CLI. 2 patches are pending
>>> which will be closed by Feature freeze date.
>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>> ATSv2. Patch in progress.
>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
>>> In progress.
>>>
>>> 3. Tentative features:
>>>
>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>>> be done before Aug 2018.
>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> Challenging as more discussions are on-going.
>>>
>>> *Summary of 3.2.0 issues status:*
>>>
>>> 39 Blocker and Critical issues [1] are open, I am checking with owners
>>> to get status on each of them to get in by Code Freeze date.
>>>
>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>>> BY priority DESC
>>>
>>> Thanks,
>>> Sunil
>>>
>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
>>>
>>>> Thanks Subru for the thoughts.
>>>> One of the main reason for a major release is to push out critical
>>>> features with a faster cadence to the users. If we are pulling more and
>>>> more different types of features to a minor release, that branch will
>>>> become more destabilized and it may be tough to say that 3.1.2 is stable
>>>> that 3.1.1 for eg. We always tend to improve and stabilize features in
>>>> subsequent minor release.
>>>> For few companies, it makes sense to push out these new features faster
>>>> to make a reach to the users. Adding to the point to the backporting
>>>> issues, I agree that its a pain and we can workaround that with some git
>>>> scripts. If we can make such scripts available to committers, backport will
>>>> be seem-less across branches and we can achieve the faster release cadence
>>>> also.
>>>>
>>>> Thoughts?
>>>>
>>>> - Sunil
>>>>
>>>>
>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
>>>> wrote:
>>>>
>>>>> Thanks Sunil for volunteering to lead the release effort. I am
>>>>> generally
>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
>>>>> already
>>>>> have too many branches to be maintained. I already see many commits
>>>>> are in
>>>>> different branches with no apparent rationale, for e.g: 3.1 has commits
>>>>> which are absent in 3.0 etc.
>>>>>
>>>>> Additionally AFAIK 3.x has not been deployed in any major production
>>>>> setting so the cost of adding features should be minimal.
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> -Subru
>>>>>
>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
>>>>>
>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>>>> >
>>>>> > Yes, important changes and features are much needed, hence we will be
>>>>> > keeping the door open for them as possible. Also considering few more
>>>>> > offline requests from other folks, I think extending the timeframe by
>>>>> > couple of weeks makes sense (including a second RC buffer) and this
>>>>> should
>>>>> > ideally help us to ship this by September itself.
>>>>> >
>>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
>>>>> >
>>>>> > - Feature freeze date : all features to merge by August 21, 2018.
>>>>> >
>>>>> > - Code freeze date : blockers/critical only, no improvements and non
>>>>> > blocker/critical
>>>>> >
>>>>> > bug-fixes August 31, 2018.
>>>>> >
>>>>> > - Release date: September 15, 2018
>>>>> >
>>>>> > Thank Eric and Zian, I think Wangda has already answered your
>>>>> questions.
>>>>> >
>>>>> > Thanks
>>>>> > Sunil
>>>>> >
>>>>> >
>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for that.
>>>>> > >
>>>>> > > To concerns from Steve,
>>>>> > >
>>>>> > > It is a good idea to keep the door open to get important changes /
>>>>> > > features in before cutoff. I would prefer to keep the proposed
>>>>> release
>>>>> > date
>>>>> > > to make sure things can happen earlier instead of last minute and
>>>>> we all
>>>>> > > know that releases are always get delayed :). I'm also fine if we
>>>>> want
>>>>> > get
>>>>> > > another several weeks time.
>>>>> > >
>>>>> > > Regarding of 3.3 release, I would suggest doing that before
>>>>> thanksgiving.
>>>>> > > Do you think is it good or too early / late?
>>>>> > >
>>>>> > > Eric,
>>>>> > >
>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
>>>>> merged
>>>>> > > in time, we probably not need the YARN-8220.
>>>>> > >
>>>>> > > Sunil,
>>>>> > >
>>>>> > > Could u update https://cwiki.apache.org/confluence/display/HADOOP/
>>>>> > Roadmap
>>>>> > > with proposed plan as well? We can fill feature list first before
>>>>> getting
>>>>> > > consensus of time.
>>>>> > >
>>>>> > > Thanks,
>>>>> > > Wangda
>>>>> > >
>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>>>> <fabbri@cloudera.com.invalid
>>>>> > >
>>>>> > > wrote:
>>>>> > >
>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>>>> stevel@hortonworks.com>
>>>>> > >> wrote:
>>>>> > >>
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
>>>>> > >> > sunilg@apache.org>> wrote:
>>>>> > >> >
>>>>> > >> > I would also would like to take this opportunity to come up
>>>>> with a
>>>>> > >> detailed
>>>>> > >> > plan.
>>>>> > >> >
>>>>> > >> > - Feature freeze date : all features should be merged by August
>>>>> 10,
>>>>> > >> 2018.
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > <snip>
>>>>> > >>
>>>>> > >> >
>>>>> > >> > Please let me know if I missed any features targeted to 3.2 per
>>>>> this
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>>>> > >> >
>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen 2
>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>>>> > >> >
>>>>> > >> > I don't think this is quite ready yet, I've been doing work on
>>>>> it, but
>>>>> > >> if
>>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
>>>>> reviews on
>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>>>> > >> >
>>>>> > >> > I've uprated that to a blocker feature; will review the S3 &
>>>>> S3Guard
>>>>> > >> JIRAs
>>>>> > >> > to see which of those are blocking. Then there are some pressing
>>>>> > "guave,
>>>>> > >> > java 9 prep"
>>>>> > >> >
>>>>> > >> >
>>>>> > >> I can help with this part if you like.
>>>>> > >>
>>>>> > >>
>>>>> > >>
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > timeline. I would like to volunteer myself as release manager
>>>>> of 3.2.0
>>>>> > >> > release.
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > well volunteered!
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> Yes, thank you for stepping up.
>>>>> > >>
>>>>> > >>
>>>>> > >> >
>>>>> > >> > I think this raises a good q: what timetable should we have for
>>>>> the
>>>>> > >> 3.2. &
>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having the
>>>>> outline
>>>>> > >> time
>>>>> > >> > from the 3.2 to the 3.3 release means that there's less concern
>>>>> about
>>>>> > >> > things not making the 3.2 dealine
>>>>> > >> >
>>>>> > >> > -Steve
>>>>> > >> >
>>>>> > >> >
>>>>> > >> Good idea to mitigate the short deadline.
>>>>> > >>
>>>>> > >> -AF
>>>>> > >>
>>>>> > >
>>>>> >
>>>>>
>>>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi All
We are down to 3 Blockers and 4 Critical now. Thanks all of you for helping
in this. I am following up on these tickets, once its closed we will cut
the 3.2 branch.
Thanks
Sunil Govindan
On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
> Hi All,
>
> Inline with the original 3.2 communication proposal dated 17th July 2018,
> I would like to provide more updates.
>
> We are approaching previously proposed code freeze date (September 14,
> 2018). So I would like to cut 3.2 branch on 17th Sept and point existing
> trunk to 3.3 if there are no issues.
>
> *Current Release Plan:*
> Feature freeze date : all features to merge by September 7, 2018.
> Code freeze date : blockers/critical only, no improvements and
> blocker/critical bug-fixes September 14, 2018.
> Release date: September 28, 2018
>
> If any critical/blocker tickets which are targeted to 3.2.0, we need to
> backport to 3.2 post branch cut.
>
> Here's an updated 3.2.0 feature status:
>
> 1. Merged & Completed features:
>
> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> Initial cut.
> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> and CLI.
> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.
>
> 2. Features close to finish:
>
> - (Steve) S3Guard Phase III. Close to commit.
> - (Steve) S3a phase V. Close to commit.
> - (Steve) Support Windows Azure Storage. Close to commit.
>
> 3. Tentative/Cancelled features for 3.2:
> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
> Patch in progress.
> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
> done before Aug 2018.
> - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
> as more discussions are on-going.
>
> *Summary of 3.2.0 issues status:*
> 19 Blocker and Critical issues [1] are open, I am following up with owners
> to get status on each of them to get in by Code Freeze date.
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
> BY priority DESC
>
> Thanks,
> Sunil
>
>
>
> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
>
>> Hi All,
>>
>> Inline with earlier communication dated 17th July 2018, I would like to
>> provide some updates.
>>
>> We are approaching previously proposed code freeze date (Aug 31).
>>
>> One of the critical feature Node Attributes feature merge discussion/vote
>> is ongoing. Also few other Blocker bugs need a bit more time. With regard
>> to this, suggesting to push the feature/code freeze for 2 more weeks to
>> accommodate these jiras too.
>>
>> Proposing Updated changes in plan inline with this:
>> Feature freeze date : all features to merge by September 7, 2018.
>> Code freeze date : blockers/critical only, no improvements and
>> blocker/critical bug-fixes September 14, 2018.
>> Release date: September 28, 2018
>>
>> If any features in branch which are targeted to 3.2.0, please reply to
>> this email thread.
>>
>> *Here's an updated 3.2.0 feature status:*
>>
>> 1. Merged & Completed features:
>>
>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
>> Initial cut.
>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>> and CLI.
>>
>> 2. Features close to finish:
>>
>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
>> Ongoing.
>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>> ATSv2. Patch in progress.
>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
>> In progress.
>>
>> 3. Tentative features:
>>
>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>> be done before Aug 2018.
>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> Challenging as more discussions are on-going.
>>
>> *Summary of 3.2.0 issues status:*
>>
>> 26 Blocker and Critical issues [1] are open, I am following up with
>> owners to get status on each of them to get in by Code Freeze date.
>>
>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>> BY priority DESC
>>
>> Thanks,
>> Sunil
>>
>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
>>
>>> Hi All,
>>>
>>> Thanks for the feedbacks. Inline with earlier communication dated 17th
>>> July 2018, I would like to provide some updates.
>>>
>>> We are approaching previously proposed feature freeze date (Aug 21,
>>> about 7 days from today).
>>> If any features in branch which are targeted to 3.2.0, please reply to
>>> this email thread.
>>> Steve has mentioned about the s3 features which will come close to Code
>>> Freeze Date (Aug 31st).
>>>
>>> *Here's an updated 3.2.0 feature status:*
>>>
>>> 1. Merged & Completed features:
>>>
>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> workloads Initial cut.
>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>>
>>> 2. Features close to finish:
>>>
>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major patches
>>> are all in, only one last
>>> patch is in review state.
>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> Scheduler. Close to commit.
>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>>> and CLI. 2 patches are pending
>>> which will be closed by Feature freeze date.
>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>> ATSv2. Patch in progress.
>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
>>> In progress.
>>>
>>> 3. Tentative features:
>>>
>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>>> be done before Aug 2018.
>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> Challenging as more discussions are on-going.
>>>
>>> *Summary of 3.2.0 issues status:*
>>>
>>> 39 Blocker and Critical issues [1] are open, I am checking with owners
>>> to get status on each of them to get in by Code Freeze date.
>>>
>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>>> BY priority DESC
>>>
>>> Thanks,
>>> Sunil
>>>
>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
>>>
>>>> Thanks Subru for the thoughts.
>>>> One of the main reason for a major release is to push out critical
>>>> features with a faster cadence to the users. If we are pulling more and
>>>> more different types of features to a minor release, that branch will
>>>> become more destabilized and it may be tough to say that 3.1.2 is stable
>>>> that 3.1.1 for eg. We always tend to improve and stabilize features in
>>>> subsequent minor release.
>>>> For few companies, it makes sense to push out these new features faster
>>>> to make a reach to the users. Adding to the point to the backporting
>>>> issues, I agree that its a pain and we can workaround that with some git
>>>> scripts. If we can make such scripts available to committers, backport will
>>>> be seem-less across branches and we can achieve the faster release cadence
>>>> also.
>>>>
>>>> Thoughts?
>>>>
>>>> - Sunil
>>>>
>>>>
>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
>>>> wrote:
>>>>
>>>>> Thanks Sunil for volunteering to lead the release effort. I am
>>>>> generally
>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
>>>>> already
>>>>> have too many branches to be maintained. I already see many commits
>>>>> are in
>>>>> different branches with no apparent rationale, for e.g: 3.1 has commits
>>>>> which are absent in 3.0 etc.
>>>>>
>>>>> Additionally AFAIK 3.x has not been deployed in any major production
>>>>> setting so the cost of adding features should be minimal.
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> -Subru
>>>>>
>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
>>>>>
>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>>>> >
>>>>> > Yes, important changes and features are much needed, hence we will be
>>>>> > keeping the door open for them as possible. Also considering few more
>>>>> > offline requests from other folks, I think extending the timeframe by
>>>>> > couple of weeks makes sense (including a second RC buffer) and this
>>>>> should
>>>>> > ideally help us to ship this by September itself.
>>>>> >
>>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
>>>>> >
>>>>> > - Feature freeze date : all features to merge by August 21, 2018.
>>>>> >
>>>>> > - Code freeze date : blockers/critical only, no improvements and non
>>>>> > blocker/critical
>>>>> >
>>>>> > bug-fixes August 31, 2018.
>>>>> >
>>>>> > - Release date: September 15, 2018
>>>>> >
>>>>> > Thank Eric and Zian, I think Wangda has already answered your
>>>>> questions.
>>>>> >
>>>>> > Thanks
>>>>> > Sunil
>>>>> >
>>>>> >
>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for that.
>>>>> > >
>>>>> > > To concerns from Steve,
>>>>> > >
>>>>> > > It is a good idea to keep the door open to get important changes /
>>>>> > > features in before cutoff. I would prefer to keep the proposed
>>>>> release
>>>>> > date
>>>>> > > to make sure things can happen earlier instead of last minute and
>>>>> we all
>>>>> > > know that releases are always get delayed :). I'm also fine if we
>>>>> want
>>>>> > get
>>>>> > > another several weeks time.
>>>>> > >
>>>>> > > Regarding of 3.3 release, I would suggest doing that before
>>>>> thanksgiving.
>>>>> > > Do you think is it good or too early / late?
>>>>> > >
>>>>> > > Eric,
>>>>> > >
>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
>>>>> merged
>>>>> > > in time, we probably not need the YARN-8220.
>>>>> > >
>>>>> > > Sunil,
>>>>> > >
>>>>> > > Could u update https://cwiki.apache.org/confluence/display/HADOOP/
>>>>> > Roadmap
>>>>> > > with proposed plan as well? We can fill feature list first before
>>>>> getting
>>>>> > > consensus of time.
>>>>> > >
>>>>> > > Thanks,
>>>>> > > Wangda
>>>>> > >
>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>>>> <fabbri@cloudera.com.invalid
>>>>> > >
>>>>> > > wrote:
>>>>> > >
>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>>>> stevel@hortonworks.com>
>>>>> > >> wrote:
>>>>> > >>
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
>>>>> > >> > sunilg@apache.org>> wrote:
>>>>> > >> >
>>>>> > >> > I would also would like to take this opportunity to come up
>>>>> with a
>>>>> > >> detailed
>>>>> > >> > plan.
>>>>> > >> >
>>>>> > >> > - Feature freeze date : all features should be merged by August
>>>>> 10,
>>>>> > >> 2018.
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > <snip>
>>>>> > >>
>>>>> > >> >
>>>>> > >> > Please let me know if I missed any features targeted to 3.2 per
>>>>> this
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>>>> > >> >
>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen 2
>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>>>> > >> >
>>>>> > >> > I don't think this is quite ready yet, I've been doing work on
>>>>> it, but
>>>>> > >> if
>>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
>>>>> reviews on
>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>>>> > >> >
>>>>> > >> > I've uprated that to a blocker feature; will review the S3 &
>>>>> S3Guard
>>>>> > >> JIRAs
>>>>> > >> > to see which of those are blocking. Then there are some pressing
>>>>> > "guave,
>>>>> > >> > java 9 prep"
>>>>> > >> >
>>>>> > >> >
>>>>> > >> I can help with this part if you like.
>>>>> > >>
>>>>> > >>
>>>>> > >>
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > timeline. I would like to volunteer myself as release manager
>>>>> of 3.2.0
>>>>> > >> > release.
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > well volunteered!
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> Yes, thank you for stepping up.
>>>>> > >>
>>>>> > >>
>>>>> > >> >
>>>>> > >> > I think this raises a good q: what timetable should we have for
>>>>> the
>>>>> > >> 3.2. &
>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having the
>>>>> outline
>>>>> > >> time
>>>>> > >> > from the 3.2 to the 3.3 release means that there's less concern
>>>>> about
>>>>> > >> > things not making the 3.2 dealine
>>>>> > >> >
>>>>> > >> > -Steve
>>>>> > >> >
>>>>> > >> >
>>>>> > >> Good idea to mitigate the short deadline.
>>>>> > >>
>>>>> > >> -AF
>>>>> > >>
>>>>> > >
>>>>> >
>>>>>
>>>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi All
We are down to 3 Blockers and 4 Critical now. Thanks all of you for helping
in this. I am following up on these tickets, once its closed we will cut
the 3.2 branch.
Thanks
Sunil Govindan
On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
> Hi All,
>
> Inline with the original 3.2 communication proposal dated 17th July 2018,
> I would like to provide more updates.
>
> We are approaching previously proposed code freeze date (September 14,
> 2018). So I would like to cut 3.2 branch on 17th Sept and point existing
> trunk to 3.3 if there are no issues.
>
> *Current Release Plan:*
> Feature freeze date : all features to merge by September 7, 2018.
> Code freeze date : blockers/critical only, no improvements and
> blocker/critical bug-fixes September 14, 2018.
> Release date: September 28, 2018
>
> If any critical/blocker tickets which are targeted to 3.2.0, we need to
> backport to 3.2 post branch cut.
>
> Here's an updated 3.2.0 feature status:
>
> 1. Merged & Completed features:
>
> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> Initial cut.
> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> and CLI.
> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.
>
> 2. Features close to finish:
>
> - (Steve) S3Guard Phase III. Close to commit.
> - (Steve) S3a phase V. Close to commit.
> - (Steve) Support Windows Azure Storage. Close to commit.
>
> 3. Tentative/Cancelled features for 3.2:
> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
> Patch in progress.
> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
> done before Aug 2018.
> - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
> as more discussions are on-going.
>
> *Summary of 3.2.0 issues status:*
> 19 Blocker and Critical issues [1] are open, I am following up with owners
> to get status on each of them to get in by Code Freeze date.
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
> BY priority DESC
>
> Thanks,
> Sunil
>
>
>
> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
>
>> Hi All,
>>
>> Inline with earlier communication dated 17th July 2018, I would like to
>> provide some updates.
>>
>> We are approaching previously proposed code freeze date (Aug 31).
>>
>> One of the critical feature Node Attributes feature merge discussion/vote
>> is ongoing. Also few other Blocker bugs need a bit more time. With regard
>> to this, suggesting to push the feature/code freeze for 2 more weeks to
>> accommodate these jiras too.
>>
>> Proposing Updated changes in plan inline with this:
>> Feature freeze date : all features to merge by September 7, 2018.
>> Code freeze date : blockers/critical only, no improvements and
>> blocker/critical bug-fixes September 14, 2018.
>> Release date: September 28, 2018
>>
>> If any features in branch which are targeted to 3.2.0, please reply to
>> this email thread.
>>
>> *Here's an updated 3.2.0 feature status:*
>>
>> 1. Merged & Completed features:
>>
>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
>> Initial cut.
>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>> and CLI.
>>
>> 2. Features close to finish:
>>
>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
>> Ongoing.
>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>> ATSv2. Patch in progress.
>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
>> In progress.
>>
>> 3. Tentative features:
>>
>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>> be done before Aug 2018.
>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> Challenging as more discussions are on-going.
>>
>> *Summary of 3.2.0 issues status:*
>>
>> 26 Blocker and Critical issues [1] are open, I am following up with
>> owners to get status on each of them to get in by Code Freeze date.
>>
>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>> BY priority DESC
>>
>> Thanks,
>> Sunil
>>
>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
>>
>>> Hi All,
>>>
>>> Thanks for the feedbacks. Inline with earlier communication dated 17th
>>> July 2018, I would like to provide some updates.
>>>
>>> We are approaching previously proposed feature freeze date (Aug 21,
>>> about 7 days from today).
>>> If any features in branch which are targeted to 3.2.0, please reply to
>>> this email thread.
>>> Steve has mentioned about the s3 features which will come close to Code
>>> Freeze Date (Aug 31st).
>>>
>>> *Here's an updated 3.2.0 feature status:*
>>>
>>> 1. Merged & Completed features:
>>>
>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> workloads Initial cut.
>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>>
>>> 2. Features close to finish:
>>>
>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major patches
>>> are all in, only one last
>>> patch is in review state.
>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> Scheduler. Close to commit.
>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>>> and CLI. 2 patches are pending
>>> which will be closed by Feature freeze date.
>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>> ATSv2. Patch in progress.
>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
>>> In progress.
>>>
>>> 3. Tentative features:
>>>
>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>>> be done before Aug 2018.
>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> Challenging as more discussions are on-going.
>>>
>>> *Summary of 3.2.0 issues status:*
>>>
>>> 39 Blocker and Critical issues [1] are open, I am checking with owners
>>> to get status on each of them to get in by Code Freeze date.
>>>
>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>>> BY priority DESC
>>>
>>> Thanks,
>>> Sunil
>>>
>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
>>>
>>>> Thanks Subru for the thoughts.
>>>> One of the main reason for a major release is to push out critical
>>>> features with a faster cadence to the users. If we are pulling more and
>>>> more different types of features to a minor release, that branch will
>>>> become more destabilized and it may be tough to say that 3.1.2 is stable
>>>> that 3.1.1 for eg. We always tend to improve and stabilize features in
>>>> subsequent minor release.
>>>> For few companies, it makes sense to push out these new features faster
>>>> to make a reach to the users. Adding to the point to the backporting
>>>> issues, I agree that its a pain and we can workaround that with some git
>>>> scripts. If we can make such scripts available to committers, backport will
>>>> be seem-less across branches and we can achieve the faster release cadence
>>>> also.
>>>>
>>>> Thoughts?
>>>>
>>>> - Sunil
>>>>
>>>>
>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
>>>> wrote:
>>>>
>>>>> Thanks Sunil for volunteering to lead the release effort. I am
>>>>> generally
>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
>>>>> already
>>>>> have too many branches to be maintained. I already see many commits
>>>>> are in
>>>>> different branches with no apparent rationale, for e.g: 3.1 has commits
>>>>> which are absent in 3.0 etc.
>>>>>
>>>>> Additionally AFAIK 3.x has not been deployed in any major production
>>>>> setting so the cost of adding features should be minimal.
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> -Subru
>>>>>
>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
>>>>>
>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>>>> >
>>>>> > Yes, important changes and features are much needed, hence we will be
>>>>> > keeping the door open for them as possible. Also considering few more
>>>>> > offline requests from other folks, I think extending the timeframe by
>>>>> > couple of weeks makes sense (including a second RC buffer) and this
>>>>> should
>>>>> > ideally help us to ship this by September itself.
>>>>> >
>>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
>>>>> >
>>>>> > - Feature freeze date : all features to merge by August 21, 2018.
>>>>> >
>>>>> > - Code freeze date : blockers/critical only, no improvements and non
>>>>> > blocker/critical
>>>>> >
>>>>> > bug-fixes August 31, 2018.
>>>>> >
>>>>> > - Release date: September 15, 2018
>>>>> >
>>>>> > Thank Eric and Zian, I think Wangda has already answered your
>>>>> questions.
>>>>> >
>>>>> > Thanks
>>>>> > Sunil
>>>>> >
>>>>> >
>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for that.
>>>>> > >
>>>>> > > To concerns from Steve,
>>>>> > >
>>>>> > > It is a good idea to keep the door open to get important changes /
>>>>> > > features in before cutoff. I would prefer to keep the proposed
>>>>> release
>>>>> > date
>>>>> > > to make sure things can happen earlier instead of last minute and
>>>>> we all
>>>>> > > know that releases are always get delayed :). I'm also fine if we
>>>>> want
>>>>> > get
>>>>> > > another several weeks time.
>>>>> > >
>>>>> > > Regarding of 3.3 release, I would suggest doing that before
>>>>> thanksgiving.
>>>>> > > Do you think is it good or too early / late?
>>>>> > >
>>>>> > > Eric,
>>>>> > >
>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
>>>>> merged
>>>>> > > in time, we probably not need the YARN-8220.
>>>>> > >
>>>>> > > Sunil,
>>>>> > >
>>>>> > > Could u update https://cwiki.apache.org/confluence/display/HADOOP/
>>>>> > Roadmap
>>>>> > > with proposed plan as well? We can fill feature list first before
>>>>> getting
>>>>> > > consensus of time.
>>>>> > >
>>>>> > > Thanks,
>>>>> > > Wangda
>>>>> > >
>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>>>> <fabbri@cloudera.com.invalid
>>>>> > >
>>>>> > > wrote:
>>>>> > >
>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>>>> stevel@hortonworks.com>
>>>>> > >> wrote:
>>>>> > >>
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
>>>>> > >> > sunilg@apache.org>> wrote:
>>>>> > >> >
>>>>> > >> > I would also would like to take this opportunity to come up
>>>>> with a
>>>>> > >> detailed
>>>>> > >> > plan.
>>>>> > >> >
>>>>> > >> > - Feature freeze date : all features should be merged by August
>>>>> 10,
>>>>> > >> 2018.
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > <snip>
>>>>> > >>
>>>>> > >> >
>>>>> > >> > Please let me know if I missed any features targeted to 3.2 per
>>>>> this
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>>>> > >> >
>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen 2
>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>>>> > >> >
>>>>> > >> > I don't think this is quite ready yet, I've been doing work on
>>>>> it, but
>>>>> > >> if
>>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
>>>>> reviews on
>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>>>> > >> >
>>>>> > >> > I've uprated that to a blocker feature; will review the S3 &
>>>>> S3Guard
>>>>> > >> JIRAs
>>>>> > >> > to see which of those are blocking. Then there are some pressing
>>>>> > "guave,
>>>>> > >> > java 9 prep"
>>>>> > >> >
>>>>> > >> >
>>>>> > >> I can help with this part if you like.
>>>>> > >>
>>>>> > >>
>>>>> > >>
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > timeline. I would like to volunteer myself as release manager
>>>>> of 3.2.0
>>>>> > >> > release.
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > well volunteered!
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> Yes, thank you for stepping up.
>>>>> > >>
>>>>> > >>
>>>>> > >> >
>>>>> > >> > I think this raises a good q: what timetable should we have for
>>>>> the
>>>>> > >> 3.2. &
>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having the
>>>>> outline
>>>>> > >> time
>>>>> > >> > from the 3.2 to the 3.3 release means that there's less concern
>>>>> about
>>>>> > >> > things not making the 3.2 dealine
>>>>> > >> >
>>>>> > >> > -Steve
>>>>> > >> >
>>>>> > >> >
>>>>> > >> Good idea to mitigate the short deadline.
>>>>> > >>
>>>>> > >> -AF
>>>>> > >>
>>>>> > >
>>>>> >
>>>>>
>>>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi All
We are down to 3 Blockers and 4 Critical now. Thanks all of you for helping
in this. I am following up on these tickets, once its closed we will cut
the 3.2 branch.
Thanks
Sunil Govindan
On Wed, Sep 12, 2018 at 5:10 PM Sunil G <su...@apache.org> wrote:
> Hi All,
>
> Inline with the original 3.2 communication proposal dated 17th July 2018,
> I would like to provide more updates.
>
> We are approaching previously proposed code freeze date (September 14,
> 2018). So I would like to cut 3.2 branch on 17th Sept and point existing
> trunk to 3.3 if there are no issues.
>
> *Current Release Plan:*
> Feature freeze date : all features to merge by September 7, 2018.
> Code freeze date : blockers/critical only, no improvements and
> blocker/critical bug-fixes September 14, 2018.
> Release date: September 28, 2018
>
> If any critical/blocker tickets which are targeted to 3.2.0, we need to
> backport to 3.2 post branch cut.
>
> Here's an updated 3.2.0 feature status:
>
> 1. Merged & Completed features:
>
> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> Initial cut.
> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> and CLI.
> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
> - (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.
>
> 2. Features close to finish:
>
> - (Steve) S3Guard Phase III. Close to commit.
> - (Steve) S3a phase V. Close to commit.
> - (Steve) Support Windows Azure Storage. Close to commit.
>
> 3. Tentative/Cancelled features for 3.2:
> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
> Patch in progress.
> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
> done before Aug 2018.
> - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
> as more discussions are on-going.
>
> *Summary of 3.2.0 issues status:*
> 19 Blocker and Critical issues [1] are open, I am following up with owners
> to get status on each of them to get in by Code Freeze date.
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
> BY priority DESC
>
> Thanks,
> Sunil
>
>
>
> On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
>
>> Hi All,
>>
>> Inline with earlier communication dated 17th July 2018, I would like to
>> provide some updates.
>>
>> We are approaching previously proposed code freeze date (Aug 31).
>>
>> One of the critical feature Node Attributes feature merge discussion/vote
>> is ongoing. Also few other Blocker bugs need a bit more time. With regard
>> to this, suggesting to push the feature/code freeze for 2 more weeks to
>> accommodate these jiras too.
>>
>> Proposing Updated changes in plan inline with this:
>> Feature freeze date : all features to merge by September 7, 2018.
>> Code freeze date : blockers/critical only, no improvements and
>> blocker/critical bug-fixes September 14, 2018.
>> Release date: September 28, 2018
>>
>> If any features in branch which are targeted to 3.2.0, please reply to
>> this email thread.
>>
>> *Here's an updated 3.2.0 feature status:*
>>
>> 1. Merged & Completed features:
>>
>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
>> Initial cut.
>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>> and CLI.
>>
>> 2. Features close to finish:
>>
>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
>> Ongoing.
>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>> ATSv2. Patch in progress.
>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
>> In progress.
>>
>> 3. Tentative features:
>>
>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>> be done before Aug 2018.
>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> Challenging as more discussions are on-going.
>>
>> *Summary of 3.2.0 issues status:*
>>
>> 26 Blocker and Critical issues [1] are open, I am following up with
>> owners to get status on each of them to get in by Code Freeze date.
>>
>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>> BY priority DESC
>>
>> Thanks,
>> Sunil
>>
>> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
>>
>>> Hi All,
>>>
>>> Thanks for the feedbacks. Inline with earlier communication dated 17th
>>> July 2018, I would like to provide some updates.
>>>
>>> We are approaching previously proposed feature freeze date (Aug 21,
>>> about 7 days from today).
>>> If any features in branch which are targeted to 3.2.0, please reply to
>>> this email thread.
>>> Steve has mentioned about the s3 features which will come close to Code
>>> Freeze Date (Aug 31st).
>>>
>>> *Here's an updated 3.2.0 feature status:*
>>>
>>> 1. Merged & Completed features:
>>>
>>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning
>>> workloads Initial cut.
>>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>>
>>> 2. Features close to finish:
>>>
>>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major patches
>>> are all in, only one last
>>> patch is in review state.
>>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity
>>> Scheduler. Close to commit.
>>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>>> and CLI. 2 patches are pending
>>> which will be closed by Feature freeze date.
>>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>>> ATSv2. Patch in progress.
>>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
>>> In progress.
>>>
>>> 3. Tentative features:
>>>
>>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>>> be done before Aug 2018.
>>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>>> Challenging as more discussions are on-going.
>>>
>>> *Summary of 3.2.0 issues status:*
>>>
>>> 39 Blocker and Critical issues [1] are open, I am checking with owners
>>> to get status on each of them to get in by Code Freeze date.
>>>
>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>>> BY priority DESC
>>>
>>> Thanks,
>>> Sunil
>>>
>>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
>>>
>>>> Thanks Subru for the thoughts.
>>>> One of the main reason for a major release is to push out critical
>>>> features with a faster cadence to the users. If we are pulling more and
>>>> more different types of features to a minor release, that branch will
>>>> become more destabilized and it may be tough to say that 3.1.2 is stable
>>>> that 3.1.1 for eg. We always tend to improve and stabilize features in
>>>> subsequent minor release.
>>>> For few companies, it makes sense to push out these new features faster
>>>> to make a reach to the users. Adding to the point to the backporting
>>>> issues, I agree that its a pain and we can workaround that with some git
>>>> scripts. If we can make such scripts available to committers, backport will
>>>> be seem-less across branches and we can achieve the faster release cadence
>>>> also.
>>>>
>>>> Thoughts?
>>>>
>>>> - Sunil
>>>>
>>>>
>>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
>>>> wrote:
>>>>
>>>>> Thanks Sunil for volunteering to lead the release effort. I am
>>>>> generally
>>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
>>>>> already
>>>>> have too many branches to be maintained. I already see many commits
>>>>> are in
>>>>> different branches with no apparent rationale, for e.g: 3.1 has commits
>>>>> which are absent in 3.0 etc.
>>>>>
>>>>> Additionally AFAIK 3.x has not been deployed in any major production
>>>>> setting so the cost of adding features should be minimal.
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> -Subru
>>>>>
>>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
>>>>>
>>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>>>> >
>>>>> > Yes, important changes and features are much needed, hence we will be
>>>>> > keeping the door open for them as possible. Also considering few more
>>>>> > offline requests from other folks, I think extending the timeframe by
>>>>> > couple of weeks makes sense (including a second RC buffer) and this
>>>>> should
>>>>> > ideally help us to ship this by September itself.
>>>>> >
>>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
>>>>> >
>>>>> > - Feature freeze date : all features to merge by August 21, 2018.
>>>>> >
>>>>> > - Code freeze date : blockers/critical only, no improvements and non
>>>>> > blocker/critical
>>>>> >
>>>>> > bug-fixes August 31, 2018.
>>>>> >
>>>>> > - Release date: September 15, 2018
>>>>> >
>>>>> > Thank Eric and Zian, I think Wangda has already answered your
>>>>> questions.
>>>>> >
>>>>> > Thanks
>>>>> > Sunil
>>>>> >
>>>>> >
>>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for that.
>>>>> > >
>>>>> > > To concerns from Steve,
>>>>> > >
>>>>> > > It is a good idea to keep the door open to get important changes /
>>>>> > > features in before cutoff. I would prefer to keep the proposed
>>>>> release
>>>>> > date
>>>>> > > to make sure things can happen earlier instead of last minute and
>>>>> we all
>>>>> > > know that releases are always get delayed :). I'm also fine if we
>>>>> want
>>>>> > get
>>>>> > > another several weeks time.
>>>>> > >
>>>>> > > Regarding of 3.3 release, I would suggest doing that before
>>>>> thanksgiving.
>>>>> > > Do you think is it good or too early / late?
>>>>> > >
>>>>> > > Eric,
>>>>> > >
>>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
>>>>> merged
>>>>> > > in time, we probably not need the YARN-8220.
>>>>> > >
>>>>> > > Sunil,
>>>>> > >
>>>>> > > Could u update https://cwiki.apache.org/confluence/display/HADOOP/
>>>>> > Roadmap
>>>>> > > with proposed plan as well? We can fill feature list first before
>>>>> getting
>>>>> > > consensus of time.
>>>>> > >
>>>>> > > Thanks,
>>>>> > > Wangda
>>>>> > >
>>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>>>> <fabbri@cloudera.com.invalid
>>>>> > >
>>>>> > > wrote:
>>>>> > >
>>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>>>> stevel@hortonworks.com>
>>>>> > >> wrote:
>>>>> > >>
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
>>>>> > >> > sunilg@apache.org>> wrote:
>>>>> > >> >
>>>>> > >> > I would also would like to take this opportunity to come up
>>>>> with a
>>>>> > >> detailed
>>>>> > >> > plan.
>>>>> > >> >
>>>>> > >> > - Feature freeze date : all features should be merged by August
>>>>> 10,
>>>>> > >> 2018.
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > <snip>
>>>>> > >>
>>>>> > >> >
>>>>> > >> > Please let me know if I missed any features targeted to 3.2 per
>>>>> this
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>>>> > >> >
>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen 2
>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>>>> > >> >
>>>>> > >> > I don't think this is quite ready yet, I've been doing work on
>>>>> it, but
>>>>> > >> if
>>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
>>>>> reviews on
>>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>>>> > >> >
>>>>> > >> > I've uprated that to a blocker feature; will review the S3 &
>>>>> S3Guard
>>>>> > >> JIRAs
>>>>> > >> > to see which of those are blocking. Then there are some pressing
>>>>> > "guave,
>>>>> > >> > java 9 prep"
>>>>> > >> >
>>>>> > >> >
>>>>> > >> I can help with this part if you like.
>>>>> > >>
>>>>> > >>
>>>>> > >>
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > timeline. I would like to volunteer myself as release manager
>>>>> of 3.2.0
>>>>> > >> > release.
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > well volunteered!
>>>>> > >> >
>>>>> > >> >
>>>>> > >> >
>>>>> > >> Yes, thank you for stepping up.
>>>>> > >>
>>>>> > >>
>>>>> > >> >
>>>>> > >> > I think this raises a good q: what timetable should we have for
>>>>> the
>>>>> > >> 3.2. &
>>>>> > >> > 3.3 releases; if we do want a faster cadence, then having the
>>>>> outline
>>>>> > >> time
>>>>> > >> > from the 3.2 to the 3.3 release means that there's less concern
>>>>> about
>>>>> > >> > things not making the 3.2 dealine
>>>>> > >> >
>>>>> > >> > -Steve
>>>>> > >> >
>>>>> > >> >
>>>>> > >> Good idea to mitigate the short deadline.
>>>>> > >>
>>>>> > >> -AF
>>>>> > >>
>>>>> > >
>>>>> >
>>>>>
>>>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi All,
Inline with the original 3.2 communication proposal dated 17th July 2018, I
would like to provide more updates.
We are approaching previously proposed code freeze date (September 14,
2018). So I would like to cut 3.2 branch on 17th Sept and point existing
trunk to 3.3 if there are no issues.
*Current Release Plan:*
Feature freeze date : all features to merge by September 7, 2018.
Code freeze date : blockers/critical only, no improvements and
blocker/critical bug-fixes September 14, 2018.
Release date: September 28, 2018
If any critical/blocker tickets which are targeted to 3.2.0, we need to
backport to 3.2 post branch cut.
Here's an updated 3.2.0 feature status:
1. Merged & Completed features:
- (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
Initial cut.
- (Uma) HDFS-10285: HDFS Storage Policy Satisfier
- (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
- (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
and CLI.
- (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
- (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.
2. Features close to finish:
- (Steve) S3Guard Phase III. Close to commit.
- (Steve) S3a phase V. Close to commit.
- (Steve) Support Windows Azure Storage. Close to commit.
3. Tentative/Cancelled features for 3.2:
- (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
Patch in progress.
- (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
done before Aug 2018.
- (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
as more discussions are on-going.
*Summary of 3.2.0 issues status:*
19 Blocker and Critical issues [1] are open, I am following up with owners
to get status on each of them to get in by Code Freeze date.
[1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
BY priority DESC
Thanks,
Sunil
On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
> Hi All,
>
> Inline with earlier communication dated 17th July 2018, I would like to
> provide some updates.
>
> We are approaching previously proposed code freeze date (Aug 31).
>
> One of the critical feature Node Attributes feature merge discussion/vote
> is ongoing. Also few other Blocker bugs need a bit more time. With regard
> to this, suggesting to push the feature/code freeze for 2 more weeks to
> accommodate these jiras too.
>
> Proposing Updated changes in plan inline with this:
> Feature freeze date : all features to merge by September 7, 2018.
> Code freeze date : blockers/critical only, no improvements and
> blocker/critical bug-fixes September 14, 2018.
> Release date: September 28, 2018
>
> If any features in branch which are targeted to 3.2.0, please reply to
> this email thread.
>
> *Here's an updated 3.2.0 feature status:*
>
> 1. Merged & Completed features:
>
> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> Initial cut.
> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> and CLI.
>
> 2. Features close to finish:
>
> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
> Ongoing.
> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
> Patch in progress.
> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
> In progress.
>
> 3. Tentative features:
>
> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
> done before Aug 2018.
> - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
> as more discussions are on-going.
>
> *Summary of 3.2.0 issues status:*
>
> 26 Blocker and Critical issues [1] are open, I am following up with
> owners to get status on each of them to get in by Code Freeze date.
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
> BY priority DESC
>
> Thanks,
> Sunil
>
> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
>
>> Hi All,
>>
>> Thanks for the feedbacks. Inline with earlier communication dated 17th
>> July 2018, I would like to provide some updates.
>>
>> We are approaching previously proposed feature freeze date (Aug 21, about
>> 7 days from today).
>> If any features in branch which are targeted to 3.2.0, please reply to
>> this email thread.
>> Steve has mentioned about the s3 features which will come close to Code
>> Freeze Date (Aug 31st).
>>
>> *Here's an updated 3.2.0 feature status:*
>>
>> 1. Merged & Completed features:
>>
>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
>> Initial cut.
>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>
>> 2. Features close to finish:
>>
>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major patches
>> are all in, only one last
>> patch is in review state.
>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
>> Close to commit.
>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>> and CLI. 2 patches are pending
>> which will be closed by Feature freeze date.
>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>> ATSv2. Patch in progress.
>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
>> In progress.
>>
>> 3. Tentative features:
>>
>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>> be done before Aug 2018.
>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> Challenging as more discussions are on-going.
>>
>> *Summary of 3.2.0 issues status:*
>>
>> 39 Blocker and Critical issues [1] are open, I am checking with owners to
>> get status on each of them to get in by Code Freeze date.
>>
>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>> BY priority DESC
>>
>> Thanks,
>> Sunil
>>
>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
>>
>>> Thanks Subru for the thoughts.
>>> One of the main reason for a major release is to push out critical
>>> features with a faster cadence to the users. If we are pulling more and
>>> more different types of features to a minor release, that branch will
>>> become more destabilized and it may be tough to say that 3.1.2 is stable
>>> that 3.1.1 for eg. We always tend to improve and stabilize features in
>>> subsequent minor release.
>>> For few companies, it makes sense to push out these new features faster
>>> to make a reach to the users. Adding to the point to the backporting
>>> issues, I agree that its a pain and we can workaround that with some git
>>> scripts. If we can make such scripts available to committers, backport will
>>> be seem-less across branches and we can achieve the faster release cadence
>>> also.
>>>
>>> Thoughts?
>>>
>>> - Sunil
>>>
>>>
>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org> wrote:
>>>
>>>> Thanks Sunil for volunteering to lead the release effort. I am generally
>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
>>>> already
>>>> have too many branches to be maintained. I already see many commits are
>>>> in
>>>> different branches with no apparent rationale, for e.g: 3.1 has commits
>>>> which are absent in 3.0 etc.
>>>>
>>>> Additionally AFAIK 3.x has not been deployed in any major production
>>>> setting so the cost of adding features should be minimal.
>>>>
>>>> Thoughts?
>>>>
>>>> -Subru
>>>>
>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
>>>>
>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>>> >
>>>> > Yes, important changes and features are much needed, hence we will be
>>>> > keeping the door open for them as possible. Also considering few more
>>>> > offline requests from other folks, I think extending the timeframe by
>>>> > couple of weeks makes sense (including a second RC buffer) and this
>>>> should
>>>> > ideally help us to ship this by September itself.
>>>> >
>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
>>>> >
>>>> > - Feature freeze date : all features to merge by August 21, 2018.
>>>> >
>>>> > - Code freeze date : blockers/critical only, no improvements and non
>>>> > blocker/critical
>>>> >
>>>> > bug-fixes August 31, 2018.
>>>> >
>>>> > - Release date: September 15, 2018
>>>> >
>>>> > Thank Eric and Zian, I think Wangda has already answered your
>>>> questions.
>>>> >
>>>> > Thanks
>>>> > Sunil
>>>> >
>>>> >
>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
>>>> wrote:
>>>> >
>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for that.
>>>> > >
>>>> > > To concerns from Steve,
>>>> > >
>>>> > > It is a good idea to keep the door open to get important changes /
>>>> > > features in before cutoff. I would prefer to keep the proposed
>>>> release
>>>> > date
>>>> > > to make sure things can happen earlier instead of last minute and
>>>> we all
>>>> > > know that releases are always get delayed :). I'm also fine if we
>>>> want
>>>> > get
>>>> > > another several weeks time.
>>>> > >
>>>> > > Regarding of 3.3 release, I would suggest doing that before
>>>> thanksgiving.
>>>> > > Do you think is it good or too early / late?
>>>> > >
>>>> > > Eric,
>>>> > >
>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
>>>> merged
>>>> > > in time, we probably not need the YARN-8220.
>>>> > >
>>>> > > Sunil,
>>>> > >
>>>> > > Could u update https://cwiki.apache.org/confluence/display/HADOOP/
>>>> > Roadmap
>>>> > > with proposed plan as well? We can fill feature list first before
>>>> getting
>>>> > > consensus of time.
>>>> > >
>>>> > > Thanks,
>>>> > > Wangda
>>>> > >
>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>>> <fabbri@cloudera.com.invalid
>>>> > >
>>>> > > wrote:
>>>> > >
>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>>> stevel@hortonworks.com>
>>>> > >> wrote:
>>>> > >>
>>>> > >> >
>>>> > >> >
>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
>>>> > >> > sunilg@apache.org>> wrote:
>>>> > >> >
>>>> > >> > I would also would like to take this opportunity to come up with
>>>> a
>>>> > >> detailed
>>>> > >> > plan.
>>>> > >> >
>>>> > >> > - Feature freeze date : all features should be merged by August
>>>> 10,
>>>> > >> 2018.
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> > <snip>
>>>> > >>
>>>> > >> >
>>>> > >> > Please let me know if I missed any features targeted to 3.2 per
>>>> this
>>>> > >> >
>>>> > >> >
>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>>> > >> >
>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>>> > >> >
>>>> > >> >
>>>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen 2
>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>>> > >> >
>>>> > >> > I don't think this is quite ready yet, I've been doing work on
>>>> it, but
>>>> > >> if
>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
>>>> reviews on
>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>>> > >> >
>>>> > >> > I've uprated that to a blocker feature; will review the S3 &
>>>> S3Guard
>>>> > >> JIRAs
>>>> > >> > to see which of those are blocking. Then there are some pressing
>>>> > "guave,
>>>> > >> > java 9 prep"
>>>> > >> >
>>>> > >> >
>>>> > >> I can help with this part if you like.
>>>> > >>
>>>> > >>
>>>> > >>
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> > timeline. I would like to volunteer myself as release manager of
>>>> 3.2.0
>>>> > >> > release.
>>>> > >> >
>>>> > >> >
>>>> > >> > well volunteered!
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> Yes, thank you for stepping up.
>>>> > >>
>>>> > >>
>>>> > >> >
>>>> > >> > I think this raises a good q: what timetable should we have for
>>>> the
>>>> > >> 3.2. &
>>>> > >> > 3.3 releases; if we do want a faster cadence, then having the
>>>> outline
>>>> > >> time
>>>> > >> > from the 3.2 to the 3.3 release means that there's less concern
>>>> about
>>>> > >> > things not making the 3.2 dealine
>>>> > >> >
>>>> > >> > -Steve
>>>> > >> >
>>>> > >> >
>>>> > >> Good idea to mitigate the short deadline.
>>>> > >>
>>>> > >> -AF
>>>> > >>
>>>> > >
>>>> >
>>>>
>>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi All,
Inline with the original 3.2 communication proposal dated 17th July 2018, I
would like to provide more updates.
We are approaching previously proposed code freeze date (September 14,
2018). So I would like to cut 3.2 branch on 17th Sept and point existing
trunk to 3.3 if there are no issues.
*Current Release Plan:*
Feature freeze date : all features to merge by September 7, 2018.
Code freeze date : blockers/critical only, no improvements and
blocker/critical bug-fixes September 14, 2018.
Release date: September 28, 2018
If any critical/blocker tickets which are targeted to 3.2.0, we need to
backport to 3.2 post branch cut.
Here's an updated 3.2.0 feature status:
1. Merged & Completed features:
- (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
Initial cut.
- (Uma) HDFS-10285: HDFS Storage Policy Satisfier
- (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
- (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
and CLI.
- (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
- (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.
2. Features close to finish:
- (Steve) S3Guard Phase III. Close to commit.
- (Steve) S3a phase V. Close to commit.
- (Steve) Support Windows Azure Storage. Close to commit.
3. Tentative/Cancelled features for 3.2:
- (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
Patch in progress.
- (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
done before Aug 2018.
- (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
as more discussions are on-going.
*Summary of 3.2.0 issues status:*
19 Blocker and Critical issues [1] are open, I am following up with owners
to get status on each of them to get in by Code Freeze date.
[1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
BY priority DESC
Thanks,
Sunil
On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
> Hi All,
>
> Inline with earlier communication dated 17th July 2018, I would like to
> provide some updates.
>
> We are approaching previously proposed code freeze date (Aug 31).
>
> One of the critical feature Node Attributes feature merge discussion/vote
> is ongoing. Also few other Blocker bugs need a bit more time. With regard
> to this, suggesting to push the feature/code freeze for 2 more weeks to
> accommodate these jiras too.
>
> Proposing Updated changes in plan inline with this:
> Feature freeze date : all features to merge by September 7, 2018.
> Code freeze date : blockers/critical only, no improvements and
> blocker/critical bug-fixes September 14, 2018.
> Release date: September 28, 2018
>
> If any features in branch which are targeted to 3.2.0, please reply to
> this email thread.
>
> *Here's an updated 3.2.0 feature status:*
>
> 1. Merged & Completed features:
>
> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> Initial cut.
> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> and CLI.
>
> 2. Features close to finish:
>
> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
> Ongoing.
> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
> Patch in progress.
> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
> In progress.
>
> 3. Tentative features:
>
> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
> done before Aug 2018.
> - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
> as more discussions are on-going.
>
> *Summary of 3.2.0 issues status:*
>
> 26 Blocker and Critical issues [1] are open, I am following up with
> owners to get status on each of them to get in by Code Freeze date.
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
> BY priority DESC
>
> Thanks,
> Sunil
>
> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
>
>> Hi All,
>>
>> Thanks for the feedbacks. Inline with earlier communication dated 17th
>> July 2018, I would like to provide some updates.
>>
>> We are approaching previously proposed feature freeze date (Aug 21, about
>> 7 days from today).
>> If any features in branch which are targeted to 3.2.0, please reply to
>> this email thread.
>> Steve has mentioned about the s3 features which will come close to Code
>> Freeze Date (Aug 31st).
>>
>> *Here's an updated 3.2.0 feature status:*
>>
>> 1. Merged & Completed features:
>>
>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
>> Initial cut.
>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>
>> 2. Features close to finish:
>>
>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major patches
>> are all in, only one last
>> patch is in review state.
>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
>> Close to commit.
>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>> and CLI. 2 patches are pending
>> which will be closed by Feature freeze date.
>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>> ATSv2. Patch in progress.
>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
>> In progress.
>>
>> 3. Tentative features:
>>
>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>> be done before Aug 2018.
>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> Challenging as more discussions are on-going.
>>
>> *Summary of 3.2.0 issues status:*
>>
>> 39 Blocker and Critical issues [1] are open, I am checking with owners to
>> get status on each of them to get in by Code Freeze date.
>>
>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>> BY priority DESC
>>
>> Thanks,
>> Sunil
>>
>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
>>
>>> Thanks Subru for the thoughts.
>>> One of the main reason for a major release is to push out critical
>>> features with a faster cadence to the users. If we are pulling more and
>>> more different types of features to a minor release, that branch will
>>> become more destabilized and it may be tough to say that 3.1.2 is stable
>>> that 3.1.1 for eg. We always tend to improve and stabilize features in
>>> subsequent minor release.
>>> For few companies, it makes sense to push out these new features faster
>>> to make a reach to the users. Adding to the point to the backporting
>>> issues, I agree that its a pain and we can workaround that with some git
>>> scripts. If we can make such scripts available to committers, backport will
>>> be seem-less across branches and we can achieve the faster release cadence
>>> also.
>>>
>>> Thoughts?
>>>
>>> - Sunil
>>>
>>>
>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org> wrote:
>>>
>>>> Thanks Sunil for volunteering to lead the release effort. I am generally
>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
>>>> already
>>>> have too many branches to be maintained. I already see many commits are
>>>> in
>>>> different branches with no apparent rationale, for e.g: 3.1 has commits
>>>> which are absent in 3.0 etc.
>>>>
>>>> Additionally AFAIK 3.x has not been deployed in any major production
>>>> setting so the cost of adding features should be minimal.
>>>>
>>>> Thoughts?
>>>>
>>>> -Subru
>>>>
>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
>>>>
>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>>> >
>>>> > Yes, important changes and features are much needed, hence we will be
>>>> > keeping the door open for them as possible. Also considering few more
>>>> > offline requests from other folks, I think extending the timeframe by
>>>> > couple of weeks makes sense (including a second RC buffer) and this
>>>> should
>>>> > ideally help us to ship this by September itself.
>>>> >
>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
>>>> >
>>>> > - Feature freeze date : all features to merge by August 21, 2018.
>>>> >
>>>> > - Code freeze date : blockers/critical only, no improvements and non
>>>> > blocker/critical
>>>> >
>>>> > bug-fixes August 31, 2018.
>>>> >
>>>> > - Release date: September 15, 2018
>>>> >
>>>> > Thank Eric and Zian, I think Wangda has already answered your
>>>> questions.
>>>> >
>>>> > Thanks
>>>> > Sunil
>>>> >
>>>> >
>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
>>>> wrote:
>>>> >
>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for that.
>>>> > >
>>>> > > To concerns from Steve,
>>>> > >
>>>> > > It is a good idea to keep the door open to get important changes /
>>>> > > features in before cutoff. I would prefer to keep the proposed
>>>> release
>>>> > date
>>>> > > to make sure things can happen earlier instead of last minute and
>>>> we all
>>>> > > know that releases are always get delayed :). I'm also fine if we
>>>> want
>>>> > get
>>>> > > another several weeks time.
>>>> > >
>>>> > > Regarding of 3.3 release, I would suggest doing that before
>>>> thanksgiving.
>>>> > > Do you think is it good or too early / late?
>>>> > >
>>>> > > Eric,
>>>> > >
>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
>>>> merged
>>>> > > in time, we probably not need the YARN-8220.
>>>> > >
>>>> > > Sunil,
>>>> > >
>>>> > > Could u update https://cwiki.apache.org/confluence/display/HADOOP/
>>>> > Roadmap
>>>> > > with proposed plan as well? We can fill feature list first before
>>>> getting
>>>> > > consensus of time.
>>>> > >
>>>> > > Thanks,
>>>> > > Wangda
>>>> > >
>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>>> <fabbri@cloudera.com.invalid
>>>> > >
>>>> > > wrote:
>>>> > >
>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>>> stevel@hortonworks.com>
>>>> > >> wrote:
>>>> > >>
>>>> > >> >
>>>> > >> >
>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
>>>> > >> > sunilg@apache.org>> wrote:
>>>> > >> >
>>>> > >> > I would also would like to take this opportunity to come up with
>>>> a
>>>> > >> detailed
>>>> > >> > plan.
>>>> > >> >
>>>> > >> > - Feature freeze date : all features should be merged by August
>>>> 10,
>>>> > >> 2018.
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> > <snip>
>>>> > >>
>>>> > >> >
>>>> > >> > Please let me know if I missed any features targeted to 3.2 per
>>>> this
>>>> > >> >
>>>> > >> >
>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>>> > >> >
>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>>> > >> >
>>>> > >> >
>>>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen 2
>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>>> > >> >
>>>> > >> > I don't think this is quite ready yet, I've been doing work on
>>>> it, but
>>>> > >> if
>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
>>>> reviews on
>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>>> > >> >
>>>> > >> > I've uprated that to a blocker feature; will review the S3 &
>>>> S3Guard
>>>> > >> JIRAs
>>>> > >> > to see which of those are blocking. Then there are some pressing
>>>> > "guave,
>>>> > >> > java 9 prep"
>>>> > >> >
>>>> > >> >
>>>> > >> I can help with this part if you like.
>>>> > >>
>>>> > >>
>>>> > >>
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> > timeline. I would like to volunteer myself as release manager of
>>>> 3.2.0
>>>> > >> > release.
>>>> > >> >
>>>> > >> >
>>>> > >> > well volunteered!
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> Yes, thank you for stepping up.
>>>> > >>
>>>> > >>
>>>> > >> >
>>>> > >> > I think this raises a good q: what timetable should we have for
>>>> the
>>>> > >> 3.2. &
>>>> > >> > 3.3 releases; if we do want a faster cadence, then having the
>>>> outline
>>>> > >> time
>>>> > >> > from the 3.2 to the 3.3 release means that there's less concern
>>>> about
>>>> > >> > things not making the 3.2 dealine
>>>> > >> >
>>>> > >> > -Steve
>>>> > >> >
>>>> > >> >
>>>> > >> Good idea to mitigate the short deadline.
>>>> > >>
>>>> > >> -AF
>>>> > >>
>>>> > >
>>>> >
>>>>
>>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Virajith Jalaparti <vi...@gmail.com>.
Hi Sunil,
Quick correction on the task list (missed this earlier) -- HDFS-12615 is
being done by Inigo Goiri
-Virajith
On Thu, Aug 30, 2018 at 9:30 AM Sunil G <su...@apache.org> wrote:
> Hi All,
>
> Inline with earlier communication dated 17th July 2018, I would like to
> provide some updates.
>
> We are approaching previously proposed code freeze date (Aug 31).
>
> One of the critical feature Node Attributes feature merge discussion/vote
> is ongoing. Also few other Blocker bugs need a bit more time. With regard
> to this, suggesting to push the feature/code freeze for 2 more weeks to
> accommodate these jiras too.
>
> Proposing Updated changes in plan inline with this:
> Feature freeze date : all features to merge by September 7, 2018.
> Code freeze date : blockers/critical only, no improvements and
> blocker/critical bug-fixes September 14, 2018.
> Release date: September 28, 2018
>
> If any features in branch which are targeted to 3.2.0, please reply to this
> email thread.
>
> *Here's an updated 3.2.0 feature status:*
>
> 1. Merged & Completed features:
>
> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> Initial cut.
> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> and CLI.
>
> 2. Features close to finish:
>
> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
> Ongoing.
> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
> Patch in progress.
> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage. In
> progress.
>
> 3. Tentative features:
>
> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
> done before Aug 2018.
> - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
> as more discussions are on-going.
>
> *Summary of 3.2.0 issues status:*
>
> 26 Blocker and Critical issues [1] are open, I am following up with owners
> to get status on each of them to get in by Code Freeze date.
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
> BY priority DESC
>
> Thanks,
> Sunil
>
> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
>
> > Hi All,
> >
> > Thanks for the feedbacks. Inline with earlier communication dated 17th
> > July 2018, I would like to provide some updates.
> >
> > We are approaching previously proposed feature freeze date (Aug 21, about
> > 7 days from today).
> > If any features in branch which are targeted to 3.2.0, please reply to
> > this email thread.
> > Steve has mentioned about the s3 features which will come close to Code
> > Freeze Date (Aug 31st).
> >
> > *Here's an updated 3.2.0 feature status:*
> >
> > 1. Merged & Completed features:
> >
> > - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> > Initial cut.
> > - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >
> > 2. Features close to finish:
> >
> > - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major patches
> > are all in, only one last
> > patch is in review state.
> > - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> > Close to commit.
> > - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> > and CLI. 2 patches are pending
> > which will be closed by Feature freeze date.
> > - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> ATSv2.
> > Patch in progress.
> > - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> > - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
> > In progress.
> >
> > 3. Tentative features:
> >
> > - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
> be
> > done before Aug 2018.
> > - (Eric) YARN-7129: Application Catalog for YARN applications.
> Challenging
> > as more discussions are on-going.
> >
> > *Summary of 3.2.0 issues status:*
> >
> > 39 Blocker and Critical issues [1] are open, I am checking with owners to
> > get status on each of them to get in by Code Freeze date.
> >
> > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> > Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
> ORDER
> > BY priority DESC
> >
> > Thanks,
> > Sunil
> >
> > On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
> >
> >> Thanks Subru for the thoughts.
> >> One of the main reason for a major release is to push out critical
> >> features with a faster cadence to the users. If we are pulling more and
> >> more different types of features to a minor release, that branch will
> >> become more destabilized and it may be tough to say that 3.1.2 is stable
> >> that 3.1.1 for eg. We always tend to improve and stabilize features in
> >> subsequent minor release.
> >> For few companies, it makes sense to push out these new features faster
> >> to make a reach to the users. Adding to the point to the backporting
> >> issues, I agree that its a pain and we can workaround that with some git
> >> scripts. If we can make such scripts available to committers, backport
> will
> >> be seem-less across branches and we can achieve the faster release
> cadence
> >> also.
> >>
> >> Thoughts?
> >>
> >> - Sunil
> >>
> >>
> >> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
> wrote:
> >>
> >>> Thanks Sunil for volunteering to lead the release effort. I am
> generally
> >>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
> >>> already
> >>> have too many branches to be maintained. I already see many commits are
> >>> in
> >>> different branches with no apparent rationale, for e.g: 3.1 has commits
> >>> which are absent in 3.0 etc.
> >>>
> >>> Additionally AFAIK 3.x has not been deployed in any major production
> >>> setting so the cost of adding features should be minimal.
> >>>
> >>> Thoughts?
> >>>
> >>> -Subru
> >>>
> >>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
> >>>
> >>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> >>> >
> >>> > Yes, important changes and features are much needed, hence we will be
> >>> > keeping the door open for them as possible. Also considering few more
> >>> > offline requests from other folks, I think extending the timeframe by
> >>> > couple of weeks makes sense (including a second RC buffer) and this
> >>> should
> >>> > ideally help us to ship this by September itself.
> >>> >
> >>> > Revised dates (I will be updating same in Roadmap wiki as well)
> >>> >
> >>> > - Feature freeze date : all features to merge by August 21, 2018.
> >>> >
> >>> > - Code freeze date : blockers/critical only, no improvements and non
> >>> > blocker/critical
> >>> >
> >>> > bug-fixes August 31, 2018.
> >>> >
> >>> > - Release date: September 15, 2018
> >>> >
> >>> > Thank Eric and Zian, I think Wangda has already answered your
> >>> questions.
> >>> >
> >>> > Thanks
> >>> > Sunil
> >>> >
> >>> >
> >>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
> >>> wrote:
> >>> >
> >>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for that.
> >>> > >
> >>> > > To concerns from Steve,
> >>> > >
> >>> > > It is a good idea to keep the door open to get important changes /
> >>> > > features in before cutoff. I would prefer to keep the proposed
> >>> release
> >>> > date
> >>> > > to make sure things can happen earlier instead of last minute and
> we
> >>> all
> >>> > > know that releases are always get delayed :). I'm also fine if we
> >>> want
> >>> > get
> >>> > > another several weeks time.
> >>> > >
> >>> > > Regarding of 3.3 release, I would suggest doing that before
> >>> thanksgiving.
> >>> > > Do you think is it good or too early / late?
> >>> > >
> >>> > > Eric,
> >>> > >
> >>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
> >>> merged
> >>> > > in time, we probably not need the YARN-8220.
> >>> > >
> >>> > > Sunil,
> >>> > >
> >>> > > Could u update https://cwiki.apache.org/confluence/display/HADOOP/
> >>> > Roadmap
> >>> > > with proposed plan as well? We can fill feature list first before
> >>> getting
> >>> > > consensus of time.
> >>> > >
> >>> > > Thanks,
> >>> > > Wangda
> >>> > >
> >>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> >>> <fabbri@cloudera.com.invalid
> >>> > >
> >>> > > wrote:
> >>> > >
> >>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> >>> stevel@hortonworks.com>
> >>> > >> wrote:
> >>> > >>
> >>> > >> >
> >>> > >> >
> >>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
> >>> > >> > sunilg@apache.org>> wrote:
> >>> > >> >
> >>> > >> > I would also would like to take this opportunity to come up
> with a
> >>> > >> detailed
> >>> > >> > plan.
> >>> > >> >
> >>> > >> > - Feature freeze date : all features should be merged by August
> >>> 10,
> >>> > >> 2018.
> >>> > >> >
> >>> > >> >
> >>> > >> >
> >>> > >> > <snip>
> >>> > >>
> >>> > >> >
> >>> > >> > Please let me know if I missed any features targeted to 3.2 per
> >>> this
> >>> > >> >
> >>> > >> >
> >>> > >> > Well there these big todo lists for S3 & S3Guard.
> >>> > >> >
> >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> >>> > >> >
> >>> > >> >
> >>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen 2
> >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> >>> > >> >
> >>> > >> > I don't think this is quite ready yet, I've been doing work on
> >>> it, but
> >>> > >> if
> >>> > >> > we have a 3 week deadline, I'm going to expect some timely
> >>> reviews on
> >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> >>> > >> >
> >>> > >> > I've uprated that to a blocker feature; will review the S3 &
> >>> S3Guard
> >>> > >> JIRAs
> >>> > >> > to see which of those are blocking. Then there are some pressing
> >>> > "guave,
> >>> > >> > java 9 prep"
> >>> > >> >
> >>> > >> >
> >>> > >> I can help with this part if you like.
> >>> > >>
> >>> > >>
> >>> > >>
> >>> > >> >
> >>> > >> >
> >>> > >> >
> >>> > >> > timeline. I would like to volunteer myself as release manager of
> >>> 3.2.0
> >>> > >> > release.
> >>> > >> >
> >>> > >> >
> >>> > >> > well volunteered!
> >>> > >> >
> >>> > >> >
> >>> > >> >
> >>> > >> Yes, thank you for stepping up.
> >>> > >>
> >>> > >>
> >>> > >> >
> >>> > >> > I think this raises a good q: what timetable should we have for
> >>> the
> >>> > >> 3.2. &
> >>> > >> > 3.3 releases; if we do want a faster cadence, then having the
> >>> outline
> >>> > >> time
> >>> > >> > from the 3.2 to the 3.3 release means that there's less concern
> >>> about
> >>> > >> > things not making the 3.2 dealine
> >>> > >> >
> >>> > >> > -Steve
> >>> > >> >
> >>> > >> >
> >>> > >> Good idea to mitigate the short deadline.
> >>> > >>
> >>> > >> -AF
> >>> > >>
> >>> > >
> >>> >
> >>>
> >>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi All,
Inline with the original 3.2 communication proposal dated 17th July 2018, I
would like to provide more updates.
We are approaching previously proposed code freeze date (September 14,
2018). So I would like to cut 3.2 branch on 17th Sept and point existing
trunk to 3.3 if there are no issues.
*Current Release Plan:*
Feature freeze date : all features to merge by September 7, 2018.
Code freeze date : blockers/critical only, no improvements and
blocker/critical bug-fixes September 14, 2018.
Release date: September 28, 2018
If any critical/blocker tickets which are targeted to 3.2.0, we need to
backport to 3.2 post branch cut.
Here's an updated 3.2.0 feature status:
1. Merged & Completed features:
- (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
Initial cut.
- (Uma) HDFS-10285: HDFS Storage Policy Satisfier
- (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
- (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
and CLI.
- (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
- (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.
2. Features close to finish:
- (Steve) S3Guard Phase III. Close to commit.
- (Steve) S3a phase V. Close to commit.
- (Steve) Support Windows Azure Storage. Close to commit.
3. Tentative/Cancelled features for 3.2:
- (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
Patch in progress.
- (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
done before Aug 2018.
- (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
as more discussions are on-going.
*Summary of 3.2.0 issues status:*
19 Blocker and Critical issues [1] are open, I am following up with owners
to get status on each of them to get in by Code Freeze date.
[1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
BY priority DESC
Thanks,
Sunil
On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
> Hi All,
>
> Inline with earlier communication dated 17th July 2018, I would like to
> provide some updates.
>
> We are approaching previously proposed code freeze date (Aug 31).
>
> One of the critical feature Node Attributes feature merge discussion/vote
> is ongoing. Also few other Blocker bugs need a bit more time. With regard
> to this, suggesting to push the feature/code freeze for 2 more weeks to
> accommodate these jiras too.
>
> Proposing Updated changes in plan inline with this:
> Feature freeze date : all features to merge by September 7, 2018.
> Code freeze date : blockers/critical only, no improvements and
> blocker/critical bug-fixes September 14, 2018.
> Release date: September 28, 2018
>
> If any features in branch which are targeted to 3.2.0, please reply to
> this email thread.
>
> *Here's an updated 3.2.0 feature status:*
>
> 1. Merged & Completed features:
>
> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> Initial cut.
> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> and CLI.
>
> 2. Features close to finish:
>
> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
> Ongoing.
> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
> Patch in progress.
> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
> In progress.
>
> 3. Tentative features:
>
> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
> done before Aug 2018.
> - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
> as more discussions are on-going.
>
> *Summary of 3.2.0 issues status:*
>
> 26 Blocker and Critical issues [1] are open, I am following up with
> owners to get status on each of them to get in by Code Freeze date.
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
> BY priority DESC
>
> Thanks,
> Sunil
>
> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
>
>> Hi All,
>>
>> Thanks for the feedbacks. Inline with earlier communication dated 17th
>> July 2018, I would like to provide some updates.
>>
>> We are approaching previously proposed feature freeze date (Aug 21, about
>> 7 days from today).
>> If any features in branch which are targeted to 3.2.0, please reply to
>> this email thread.
>> Steve has mentioned about the s3 features which will come close to Code
>> Freeze Date (Aug 31st).
>>
>> *Here's an updated 3.2.0 feature status:*
>>
>> 1. Merged & Completed features:
>>
>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
>> Initial cut.
>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>
>> 2. Features close to finish:
>>
>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major patches
>> are all in, only one last
>> patch is in review state.
>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
>> Close to commit.
>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>> and CLI. 2 patches are pending
>> which will be closed by Feature freeze date.
>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>> ATSv2. Patch in progress.
>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
>> In progress.
>>
>> 3. Tentative features:
>>
>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>> be done before Aug 2018.
>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> Challenging as more discussions are on-going.
>>
>> *Summary of 3.2.0 issues status:*
>>
>> 39 Blocker and Critical issues [1] are open, I am checking with owners to
>> get status on each of them to get in by Code Freeze date.
>>
>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>> BY priority DESC
>>
>> Thanks,
>> Sunil
>>
>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
>>
>>> Thanks Subru for the thoughts.
>>> One of the main reason for a major release is to push out critical
>>> features with a faster cadence to the users. If we are pulling more and
>>> more different types of features to a minor release, that branch will
>>> become more destabilized and it may be tough to say that 3.1.2 is stable
>>> that 3.1.1 for eg. We always tend to improve and stabilize features in
>>> subsequent minor release.
>>> For few companies, it makes sense to push out these new features faster
>>> to make a reach to the users. Adding to the point to the backporting
>>> issues, I agree that its a pain and we can workaround that with some git
>>> scripts. If we can make such scripts available to committers, backport will
>>> be seem-less across branches and we can achieve the faster release cadence
>>> also.
>>>
>>> Thoughts?
>>>
>>> - Sunil
>>>
>>>
>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org> wrote:
>>>
>>>> Thanks Sunil for volunteering to lead the release effort. I am generally
>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
>>>> already
>>>> have too many branches to be maintained. I already see many commits are
>>>> in
>>>> different branches with no apparent rationale, for e.g: 3.1 has commits
>>>> which are absent in 3.0 etc.
>>>>
>>>> Additionally AFAIK 3.x has not been deployed in any major production
>>>> setting so the cost of adding features should be minimal.
>>>>
>>>> Thoughts?
>>>>
>>>> -Subru
>>>>
>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
>>>>
>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>>> >
>>>> > Yes, important changes and features are much needed, hence we will be
>>>> > keeping the door open for them as possible. Also considering few more
>>>> > offline requests from other folks, I think extending the timeframe by
>>>> > couple of weeks makes sense (including a second RC buffer) and this
>>>> should
>>>> > ideally help us to ship this by September itself.
>>>> >
>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
>>>> >
>>>> > - Feature freeze date : all features to merge by August 21, 2018.
>>>> >
>>>> > - Code freeze date : blockers/critical only, no improvements and non
>>>> > blocker/critical
>>>> >
>>>> > bug-fixes August 31, 2018.
>>>> >
>>>> > - Release date: September 15, 2018
>>>> >
>>>> > Thank Eric and Zian, I think Wangda has already answered your
>>>> questions.
>>>> >
>>>> > Thanks
>>>> > Sunil
>>>> >
>>>> >
>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
>>>> wrote:
>>>> >
>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for that.
>>>> > >
>>>> > > To concerns from Steve,
>>>> > >
>>>> > > It is a good idea to keep the door open to get important changes /
>>>> > > features in before cutoff. I would prefer to keep the proposed
>>>> release
>>>> > date
>>>> > > to make sure things can happen earlier instead of last minute and
>>>> we all
>>>> > > know that releases are always get delayed :). I'm also fine if we
>>>> want
>>>> > get
>>>> > > another several weeks time.
>>>> > >
>>>> > > Regarding of 3.3 release, I would suggest doing that before
>>>> thanksgiving.
>>>> > > Do you think is it good or too early / late?
>>>> > >
>>>> > > Eric,
>>>> > >
>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
>>>> merged
>>>> > > in time, we probably not need the YARN-8220.
>>>> > >
>>>> > > Sunil,
>>>> > >
>>>> > > Could u update https://cwiki.apache.org/confluence/display/HADOOP/
>>>> > Roadmap
>>>> > > with proposed plan as well? We can fill feature list first before
>>>> getting
>>>> > > consensus of time.
>>>> > >
>>>> > > Thanks,
>>>> > > Wangda
>>>> > >
>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>>> <fabbri@cloudera.com.invalid
>>>> > >
>>>> > > wrote:
>>>> > >
>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>>> stevel@hortonworks.com>
>>>> > >> wrote:
>>>> > >>
>>>> > >> >
>>>> > >> >
>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
>>>> > >> > sunilg@apache.org>> wrote:
>>>> > >> >
>>>> > >> > I would also would like to take this opportunity to come up with
>>>> a
>>>> > >> detailed
>>>> > >> > plan.
>>>> > >> >
>>>> > >> > - Feature freeze date : all features should be merged by August
>>>> 10,
>>>> > >> 2018.
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> > <snip>
>>>> > >>
>>>> > >> >
>>>> > >> > Please let me know if I missed any features targeted to 3.2 per
>>>> this
>>>> > >> >
>>>> > >> >
>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>>> > >> >
>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>>> > >> >
>>>> > >> >
>>>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen 2
>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>>> > >> >
>>>> > >> > I don't think this is quite ready yet, I've been doing work on
>>>> it, but
>>>> > >> if
>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
>>>> reviews on
>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>>> > >> >
>>>> > >> > I've uprated that to a blocker feature; will review the S3 &
>>>> S3Guard
>>>> > >> JIRAs
>>>> > >> > to see which of those are blocking. Then there are some pressing
>>>> > "guave,
>>>> > >> > java 9 prep"
>>>> > >> >
>>>> > >> >
>>>> > >> I can help with this part if you like.
>>>> > >>
>>>> > >>
>>>> > >>
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> > timeline. I would like to volunteer myself as release manager of
>>>> 3.2.0
>>>> > >> > release.
>>>> > >> >
>>>> > >> >
>>>> > >> > well volunteered!
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> Yes, thank you for stepping up.
>>>> > >>
>>>> > >>
>>>> > >> >
>>>> > >> > I think this raises a good q: what timetable should we have for
>>>> the
>>>> > >> 3.2. &
>>>> > >> > 3.3 releases; if we do want a faster cadence, then having the
>>>> outline
>>>> > >> time
>>>> > >> > from the 3.2 to the 3.3 release means that there's less concern
>>>> about
>>>> > >> > things not making the 3.2 dealine
>>>> > >> >
>>>> > >> > -Steve
>>>> > >> >
>>>> > >> >
>>>> > >> Good idea to mitigate the short deadline.
>>>> > >>
>>>> > >> -AF
>>>> > >>
>>>> > >
>>>> >
>>>>
>>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Virajith Jalaparti <vi...@gmail.com>.
Hi Sunil,
Quick correction on the task list (missed this earlier) -- HDFS-12615 is
being done by Inigo Goiri
-Virajith
On Thu, Aug 30, 2018 at 9:30 AM Sunil G <su...@apache.org> wrote:
> Hi All,
>
> Inline with earlier communication dated 17th July 2018, I would like to
> provide some updates.
>
> We are approaching previously proposed code freeze date (Aug 31).
>
> One of the critical feature Node Attributes feature merge discussion/vote
> is ongoing. Also few other Blocker bugs need a bit more time. With regard
> to this, suggesting to push the feature/code freeze for 2 more weeks to
> accommodate these jiras too.
>
> Proposing Updated changes in plan inline with this:
> Feature freeze date : all features to merge by September 7, 2018.
> Code freeze date : blockers/critical only, no improvements and
> blocker/critical bug-fixes September 14, 2018.
> Release date: September 28, 2018
>
> If any features in branch which are targeted to 3.2.0, please reply to this
> email thread.
>
> *Here's an updated 3.2.0 feature status:*
>
> 1. Merged & Completed features:
>
> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> Initial cut.
> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> and CLI.
>
> 2. Features close to finish:
>
> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
> Ongoing.
> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
> Patch in progress.
> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage. In
> progress.
>
> 3. Tentative features:
>
> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
> done before Aug 2018.
> - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
> as more discussions are on-going.
>
> *Summary of 3.2.0 issues status:*
>
> 26 Blocker and Critical issues [1] are open, I am following up with owners
> to get status on each of them to get in by Code Freeze date.
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
> BY priority DESC
>
> Thanks,
> Sunil
>
> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
>
> > Hi All,
> >
> > Thanks for the feedbacks. Inline with earlier communication dated 17th
> > July 2018, I would like to provide some updates.
> >
> > We are approaching previously proposed feature freeze date (Aug 21, about
> > 7 days from today).
> > If any features in branch which are targeted to 3.2.0, please reply to
> > this email thread.
> > Steve has mentioned about the s3 features which will come close to Code
> > Freeze Date (Aug 31st).
> >
> > *Here's an updated 3.2.0 feature status:*
> >
> > 1. Merged & Completed features:
> >
> > - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> > Initial cut.
> > - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> >
> > 2. Features close to finish:
> >
> > - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major patches
> > are all in, only one last
> > patch is in review state.
> > - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> > Close to commit.
> > - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> > and CLI. 2 patches are pending
> > which will be closed by Feature freeze date.
> > - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
> ATSv2.
> > Patch in progress.
> > - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> > - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
> > In progress.
> >
> > 3. Tentative features:
> >
> > - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
> be
> > done before Aug 2018.
> > - (Eric) YARN-7129: Application Catalog for YARN applications.
> Challenging
> > as more discussions are on-going.
> >
> > *Summary of 3.2.0 issues status:*
> >
> > 39 Blocker and Critical issues [1] are open, I am checking with owners to
> > get status on each of them to get in by Code Freeze date.
> >
> > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> > Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0
> ORDER
> > BY priority DESC
> >
> > Thanks,
> > Sunil
> >
> > On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
> >
> >> Thanks Subru for the thoughts.
> >> One of the main reason for a major release is to push out critical
> >> features with a faster cadence to the users. If we are pulling more and
> >> more different types of features to a minor release, that branch will
> >> become more destabilized and it may be tough to say that 3.1.2 is stable
> >> that 3.1.1 for eg. We always tend to improve and stabilize features in
> >> subsequent minor release.
> >> For few companies, it makes sense to push out these new features faster
> >> to make a reach to the users. Adding to the point to the backporting
> >> issues, I agree that its a pain and we can workaround that with some git
> >> scripts. If we can make such scripts available to committers, backport
> will
> >> be seem-less across branches and we can achieve the faster release
> cadence
> >> also.
> >>
> >> Thoughts?
> >>
> >> - Sunil
> >>
> >>
> >> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org>
> wrote:
> >>
> >>> Thanks Sunil for volunteering to lead the release effort. I am
> generally
> >>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
> >>> already
> >>> have too many branches to be maintained. I already see many commits are
> >>> in
> >>> different branches with no apparent rationale, for e.g: 3.1 has commits
> >>> which are absent in 3.0 etc.
> >>>
> >>> Additionally AFAIK 3.x has not been deployed in any major production
> >>> setting so the cost of adding features should be minimal.
> >>>
> >>> Thoughts?
> >>>
> >>> -Subru
> >>>
> >>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
> >>>
> >>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
> >>> >
> >>> > Yes, important changes and features are much needed, hence we will be
> >>> > keeping the door open for them as possible. Also considering few more
> >>> > offline requests from other folks, I think extending the timeframe by
> >>> > couple of weeks makes sense (including a second RC buffer) and this
> >>> should
> >>> > ideally help us to ship this by September itself.
> >>> >
> >>> > Revised dates (I will be updating same in Roadmap wiki as well)
> >>> >
> >>> > - Feature freeze date : all features to merge by August 21, 2018.
> >>> >
> >>> > - Code freeze date : blockers/critical only, no improvements and non
> >>> > blocker/critical
> >>> >
> >>> > bug-fixes August 31, 2018.
> >>> >
> >>> > - Release date: September 15, 2018
> >>> >
> >>> > Thank Eric and Zian, I think Wangda has already answered your
> >>> questions.
> >>> >
> >>> > Thanks
> >>> > Sunil
> >>> >
> >>> >
> >>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
> >>> wrote:
> >>> >
> >>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for that.
> >>> > >
> >>> > > To concerns from Steve,
> >>> > >
> >>> > > It is a good idea to keep the door open to get important changes /
> >>> > > features in before cutoff. I would prefer to keep the proposed
> >>> release
> >>> > date
> >>> > > to make sure things can happen earlier instead of last minute and
> we
> >>> all
> >>> > > know that releases are always get delayed :). I'm also fine if we
> >>> want
> >>> > get
> >>> > > another several weeks time.
> >>> > >
> >>> > > Regarding of 3.3 release, I would suggest doing that before
> >>> thanksgiving.
> >>> > > Do you think is it good or too early / late?
> >>> > >
> >>> > > Eric,
> >>> > >
> >>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
> >>> merged
> >>> > > in time, we probably not need the YARN-8220.
> >>> > >
> >>> > > Sunil,
> >>> > >
> >>> > > Could u update https://cwiki.apache.org/confluence/display/HADOOP/
> >>> > Roadmap
> >>> > > with proposed plan as well? We can fill feature list first before
> >>> getting
> >>> > > consensus of time.
> >>> > >
> >>> > > Thanks,
> >>> > > Wangda
> >>> > >
> >>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
> >>> <fabbri@cloudera.com.invalid
> >>> > >
> >>> > > wrote:
> >>> > >
> >>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
> >>> stevel@hortonworks.com>
> >>> > >> wrote:
> >>> > >>
> >>> > >> >
> >>> > >> >
> >>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
> >>> > >> > sunilg@apache.org>> wrote:
> >>> > >> >
> >>> > >> > I would also would like to take this opportunity to come up
> with a
> >>> > >> detailed
> >>> > >> > plan.
> >>> > >> >
> >>> > >> > - Feature freeze date : all features should be merged by August
> >>> 10,
> >>> > >> 2018.
> >>> > >> >
> >>> > >> >
> >>> > >> >
> >>> > >> > <snip>
> >>> > >>
> >>> > >> >
> >>> > >> > Please let me know if I missed any features targeted to 3.2 per
> >>> this
> >>> > >> >
> >>> > >> >
> >>> > >> > Well there these big todo lists for S3 & S3Guard.
> >>> > >> >
> >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
> >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
> >>> > >> >
> >>> > >> >
> >>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen 2
> >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
> >>> > >> >
> >>> > >> > I don't think this is quite ready yet, I've been doing work on
> >>> it, but
> >>> > >> if
> >>> > >> > we have a 3 week deadline, I'm going to expect some timely
> >>> reviews on
> >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
> >>> > >> >
> >>> > >> > I've uprated that to a blocker feature; will review the S3 &
> >>> S3Guard
> >>> > >> JIRAs
> >>> > >> > to see which of those are blocking. Then there are some pressing
> >>> > "guave,
> >>> > >> > java 9 prep"
> >>> > >> >
> >>> > >> >
> >>> > >> I can help with this part if you like.
> >>> > >>
> >>> > >>
> >>> > >>
> >>> > >> >
> >>> > >> >
> >>> > >> >
> >>> > >> > timeline. I would like to volunteer myself as release manager of
> >>> 3.2.0
> >>> > >> > release.
> >>> > >> >
> >>> > >> >
> >>> > >> > well volunteered!
> >>> > >> >
> >>> > >> >
> >>> > >> >
> >>> > >> Yes, thank you for stepping up.
> >>> > >>
> >>> > >>
> >>> > >> >
> >>> > >> > I think this raises a good q: what timetable should we have for
> >>> the
> >>> > >> 3.2. &
> >>> > >> > 3.3 releases; if we do want a faster cadence, then having the
> >>> outline
> >>> > >> time
> >>> > >> > from the 3.2 to the 3.3 release means that there's less concern
> >>> about
> >>> > >> > things not making the 3.2 dealine
> >>> > >> >
> >>> > >> > -Steve
> >>> > >> >
> >>> > >> >
> >>> > >> Good idea to mitigate the short deadline.
> >>> > >>
> >>> > >> -AF
> >>> > >>
> >>> > >
> >>> >
> >>>
> >>
>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi All,
Inline with the original 3.2 communication proposal dated 17th July 2018, I
would like to provide more updates.
We are approaching previously proposed code freeze date (September 14,
2018). So I would like to cut 3.2 branch on 17th Sept and point existing
trunk to 3.3 if there are no issues.
*Current Release Plan:*
Feature freeze date : all features to merge by September 7, 2018.
Code freeze date : blockers/critical only, no improvements and
blocker/critical bug-fixes September 14, 2018.
Release date: September 28, 2018
If any critical/blocker tickets which are targeted to 3.2.0, we need to
backport to 3.2 post branch cut.
Here's an updated 3.2.0 feature status:
1. Merged & Completed features:
- (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
Initial cut.
- (Uma) HDFS-10285: HDFS Storage Policy Satisfier
- (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
- (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
and CLI.
- (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
- (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.
2. Features close to finish:
- (Steve) S3Guard Phase III. Close to commit.
- (Steve) S3a phase V. Close to commit.
- (Steve) Support Windows Azure Storage. Close to commit.
3. Tentative/Cancelled features for 3.2:
- (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
Patch in progress.
- (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
done before Aug 2018.
- (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
as more discussions are on-going.
*Summary of 3.2.0 issues status:*
19 Blocker and Critical issues [1] are open, I am following up with owners
to get status on each of them to get in by Code Freeze date.
[1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
BY priority DESC
Thanks,
Sunil
On Thu, Aug 30, 2018 at 9:59 PM Sunil G <su...@apache.org> wrote:
> Hi All,
>
> Inline with earlier communication dated 17th July 2018, I would like to
> provide some updates.
>
> We are approaching previously proposed code freeze date (Aug 31).
>
> One of the critical feature Node Attributes feature merge discussion/vote
> is ongoing. Also few other Blocker bugs need a bit more time. With regard
> to this, suggesting to push the feature/code freeze for 2 more weeks to
> accommodate these jiras too.
>
> Proposing Updated changes in plan inline with this:
> Feature freeze date : all features to merge by September 7, 2018.
> Code freeze date : blockers/critical only, no improvements and
> blocker/critical bug-fixes September 14, 2018.
> Release date: September 28, 2018
>
> If any features in branch which are targeted to 3.2.0, please reply to
> this email thread.
>
> *Here's an updated 3.2.0 feature status:*
>
> 1. Merged & Completed features:
>
> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> Initial cut.
> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> and CLI.
>
> 2. Features close to finish:
>
> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
> Ongoing.
> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
> Patch in progress.
> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
> In progress.
>
> 3. Tentative features:
>
> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
> done before Aug 2018.
> - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
> as more discussions are on-going.
>
> *Summary of 3.2.0 issues status:*
>
> 26 Blocker and Critical issues [1] are open, I am following up with
> owners to get status on each of them to get in by Code Freeze date.
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
> BY priority DESC
>
> Thanks,
> Sunil
>
> On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
>
>> Hi All,
>>
>> Thanks for the feedbacks. Inline with earlier communication dated 17th
>> July 2018, I would like to provide some updates.
>>
>> We are approaching previously proposed feature freeze date (Aug 21, about
>> 7 days from today).
>> If any features in branch which are targeted to 3.2.0, please reply to
>> this email thread.
>> Steve has mentioned about the s3 features which will come close to Code
>> Freeze Date (Aug 31st).
>>
>> *Here's an updated 3.2.0 feature status:*
>>
>> 1. Merged & Completed features:
>>
>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
>> Initial cut.
>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>
>> 2. Features close to finish:
>>
>> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major patches
>> are all in, only one last
>> patch is in review state.
>> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
>> Close to commit.
>> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
>> and CLI. 2 patches are pending
>> which will be closed by Feature freeze date.
>> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from
>> ATSv2. Patch in progress.
>> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
>> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
>> In progress.
>>
>> 3. Tentative features:
>>
>> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to
>> be done before Aug 2018.
>> - (Eric) YARN-7129: Application Catalog for YARN applications.
>> Challenging as more discussions are on-going.
>>
>> *Summary of 3.2.0 issues status:*
>>
>> 39 Blocker and Critical issues [1] are open, I am checking with owners to
>> get status on each of them to get in by Code Freeze date.
>>
>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
>> BY priority DESC
>>
>> Thanks,
>> Sunil
>>
>> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
>>
>>> Thanks Subru for the thoughts.
>>> One of the main reason for a major release is to push out critical
>>> features with a faster cadence to the users. If we are pulling more and
>>> more different types of features to a minor release, that branch will
>>> become more destabilized and it may be tough to say that 3.1.2 is stable
>>> that 3.1.1 for eg. We always tend to improve and stabilize features in
>>> subsequent minor release.
>>> For few companies, it makes sense to push out these new features faster
>>> to make a reach to the users. Adding to the point to the backporting
>>> issues, I agree that its a pain and we can workaround that with some git
>>> scripts. If we can make such scripts available to committers, backport will
>>> be seem-less across branches and we can achieve the faster release cadence
>>> also.
>>>
>>> Thoughts?
>>>
>>> - Sunil
>>>
>>>
>>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org> wrote:
>>>
>>>> Thanks Sunil for volunteering to lead the release effort. I am generally
>>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
>>>> already
>>>> have too many branches to be maintained. I already see many commits are
>>>> in
>>>> different branches with no apparent rationale, for e.g: 3.1 has commits
>>>> which are absent in 3.0 etc.
>>>>
>>>> Additionally AFAIK 3.x has not been deployed in any major production
>>>> setting so the cost of adding features should be minimal.
>>>>
>>>> Thoughts?
>>>>
>>>> -Subru
>>>>
>>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
>>>>
>>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>>> >
>>>> > Yes, important changes and features are much needed, hence we will be
>>>> > keeping the door open for them as possible. Also considering few more
>>>> > offline requests from other folks, I think extending the timeframe by
>>>> > couple of weeks makes sense (including a second RC buffer) and this
>>>> should
>>>> > ideally help us to ship this by September itself.
>>>> >
>>>> > Revised dates (I will be updating same in Roadmap wiki as well)
>>>> >
>>>> > - Feature freeze date : all features to merge by August 21, 2018.
>>>> >
>>>> > - Code freeze date : blockers/critical only, no improvements and non
>>>> > blocker/critical
>>>> >
>>>> > bug-fixes August 31, 2018.
>>>> >
>>>> > - Release date: September 15, 2018
>>>> >
>>>> > Thank Eric and Zian, I think Wangda has already answered your
>>>> questions.
>>>> >
>>>> > Thanks
>>>> > Sunil
>>>> >
>>>> >
>>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
>>>> wrote:
>>>> >
>>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for that.
>>>> > >
>>>> > > To concerns from Steve,
>>>> > >
>>>> > > It is a good idea to keep the door open to get important changes /
>>>> > > features in before cutoff. I would prefer to keep the proposed
>>>> release
>>>> > date
>>>> > > to make sure things can happen earlier instead of last minute and
>>>> we all
>>>> > > know that releases are always get delayed :). I'm also fine if we
>>>> want
>>>> > get
>>>> > > another several weeks time.
>>>> > >
>>>> > > Regarding of 3.3 release, I would suggest doing that before
>>>> thanksgiving.
>>>> > > Do you think is it good or too early / late?
>>>> > >
>>>> > > Eric,
>>>> > >
>>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
>>>> merged
>>>> > > in time, we probably not need the YARN-8220.
>>>> > >
>>>> > > Sunil,
>>>> > >
>>>> > > Could u update https://cwiki.apache.org/confluence/display/HADOOP/
>>>> > Roadmap
>>>> > > with proposed plan as well? We can fill feature list first before
>>>> getting
>>>> > > consensus of time.
>>>> > >
>>>> > > Thanks,
>>>> > > Wangda
>>>> > >
>>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>>> <fabbri@cloudera.com.invalid
>>>> > >
>>>> > > wrote:
>>>> > >
>>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>>> stevel@hortonworks.com>
>>>> > >> wrote:
>>>> > >>
>>>> > >> >
>>>> > >> >
>>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
>>>> > >> > sunilg@apache.org>> wrote:
>>>> > >> >
>>>> > >> > I would also would like to take this opportunity to come up with
>>>> a
>>>> > >> detailed
>>>> > >> > plan.
>>>> > >> >
>>>> > >> > - Feature freeze date : all features should be merged by August
>>>> 10,
>>>> > >> 2018.
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> > <snip>
>>>> > >>
>>>> > >> >
>>>> > >> > Please let me know if I missed any features targeted to 3.2 per
>>>> this
>>>> > >> >
>>>> > >> >
>>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>>> > >> >
>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>>> > >> >
>>>> > >> >
>>>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen 2
>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>>> > >> >
>>>> > >> > I don't think this is quite ready yet, I've been doing work on
>>>> it, but
>>>> > >> if
>>>> > >> > we have a 3 week deadline, I'm going to expect some timely
>>>> reviews on
>>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>>> > >> >
>>>> > >> > I've uprated that to a blocker feature; will review the S3 &
>>>> S3Guard
>>>> > >> JIRAs
>>>> > >> > to see which of those are blocking. Then there are some pressing
>>>> > "guave,
>>>> > >> > java 9 prep"
>>>> > >> >
>>>> > >> >
>>>> > >> I can help with this part if you like.
>>>> > >>
>>>> > >>
>>>> > >>
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> > timeline. I would like to volunteer myself as release manager of
>>>> 3.2.0
>>>> > >> > release.
>>>> > >> >
>>>> > >> >
>>>> > >> > well volunteered!
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> Yes, thank you for stepping up.
>>>> > >>
>>>> > >>
>>>> > >> >
>>>> > >> > I think this raises a good q: what timetable should we have for
>>>> the
>>>> > >> 3.2. &
>>>> > >> > 3.3 releases; if we do want a faster cadence, then having the
>>>> outline
>>>> > >> time
>>>> > >> > from the 3.2 to the 3.3 release means that there's less concern
>>>> about
>>>> > >> > things not making the 3.2 dealine
>>>> > >> >
>>>> > >> > -Steve
>>>> > >> >
>>>> > >> >
>>>> > >> Good idea to mitigate the short deadline.
>>>> > >>
>>>> > >> -AF
>>>> > >>
>>>> > >
>>>> >
>>>>
>>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi All,
Inline with earlier communication dated 17th July 2018, I would like to
provide some updates.
We are approaching previously proposed code freeze date (Aug 31).
One of the critical feature Node Attributes feature merge discussion/vote
is ongoing. Also few other Blocker bugs need a bit more time. With regard
to this, suggesting to push the feature/code freeze for 2 more weeks to
accommodate these jiras too.
Proposing Updated changes in plan inline with this:
Feature freeze date : all features to merge by September 7, 2018.
Code freeze date : blockers/critical only, no improvements and
blocker/critical bug-fixes September 14, 2018.
Release date: September 28, 2018
If any features in branch which are targeted to 3.2.0, please reply to this
email thread.
*Here's an updated 3.2.0 feature status:*
1. Merged & Completed features:
- (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
Initial cut.
- (Uma) HDFS-10285: HDFS Storage Policy Satisfier
- (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
- (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
and CLI.
2. Features close to finish:
- (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
Ongoing.
- (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
Patch in progress.
- (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
- (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage. In
progress.
3. Tentative features:
- (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
done before Aug 2018.
- (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
as more discussions are on-going.
*Summary of 3.2.0 issues status:*
26 Blocker and Critical issues [1] are open, I am following up with owners
to get status on each of them to get in by Code Freeze date.
[1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
BY priority DESC
Thanks,
Sunil
On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
> Hi All,
>
> Thanks for the feedbacks. Inline with earlier communication dated 17th
> July 2018, I would like to provide some updates.
>
> We are approaching previously proposed feature freeze date (Aug 21, about
> 7 days from today).
> If any features in branch which are targeted to 3.2.0, please reply to
> this email thread.
> Steve has mentioned about the s3 features which will come close to Code
> Freeze Date (Aug 31st).
>
> *Here's an updated 3.2.0 feature status:*
>
> 1. Merged & Completed features:
>
> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> Initial cut.
> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>
> 2. Features close to finish:
>
> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major patches
> are all in, only one last
> patch is in review state.
> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> Close to commit.
> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> and CLI. 2 patches are pending
> which will be closed by Feature freeze date.
> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
> Patch in progress.
> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
> In progress.
>
> 3. Tentative features:
>
> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
> done before Aug 2018.
> - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
> as more discussions are on-going.
>
> *Summary of 3.2.0 issues status:*
>
> 39 Blocker and Critical issues [1] are open, I am checking with owners to
> get status on each of them to get in by Code Freeze date.
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
> BY priority DESC
>
> Thanks,
> Sunil
>
> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
>
>> Thanks Subru for the thoughts.
>> One of the main reason for a major release is to push out critical
>> features with a faster cadence to the users. If we are pulling more and
>> more different types of features to a minor release, that branch will
>> become more destabilized and it may be tough to say that 3.1.2 is stable
>> that 3.1.1 for eg. We always tend to improve and stabilize features in
>> subsequent minor release.
>> For few companies, it makes sense to push out these new features faster
>> to make a reach to the users. Adding to the point to the backporting
>> issues, I agree that its a pain and we can workaround that with some git
>> scripts. If we can make such scripts available to committers, backport will
>> be seem-less across branches and we can achieve the faster release cadence
>> also.
>>
>> Thoughts?
>>
>> - Sunil
>>
>>
>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org> wrote:
>>
>>> Thanks Sunil for volunteering to lead the release effort. I am generally
>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
>>> already
>>> have too many branches to be maintained. I already see many commits are
>>> in
>>> different branches with no apparent rationale, for e.g: 3.1 has commits
>>> which are absent in 3.0 etc.
>>>
>>> Additionally AFAIK 3.x has not been deployed in any major production
>>> setting so the cost of adding features should be minimal.
>>>
>>> Thoughts?
>>>
>>> -Subru
>>>
>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
>>>
>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>> >
>>> > Yes, important changes and features are much needed, hence we will be
>>> > keeping the door open for them as possible. Also considering few more
>>> > offline requests from other folks, I think extending the timeframe by
>>> > couple of weeks makes sense (including a second RC buffer) and this
>>> should
>>> > ideally help us to ship this by September itself.
>>> >
>>> > Revised dates (I will be updating same in Roadmap wiki as well)
>>> >
>>> > - Feature freeze date : all features to merge by August 21, 2018.
>>> >
>>> > - Code freeze date : blockers/critical only, no improvements and non
>>> > blocker/critical
>>> >
>>> > bug-fixes August 31, 2018.
>>> >
>>> > - Release date: September 15, 2018
>>> >
>>> > Thank Eric and Zian, I think Wangda has already answered your
>>> questions.
>>> >
>>> > Thanks
>>> > Sunil
>>> >
>>> >
>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
>>> wrote:
>>> >
>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for that.
>>> > >
>>> > > To concerns from Steve,
>>> > >
>>> > > It is a good idea to keep the door open to get important changes /
>>> > > features in before cutoff. I would prefer to keep the proposed
>>> release
>>> > date
>>> > > to make sure things can happen earlier instead of last minute and we
>>> all
>>> > > know that releases are always get delayed :). I'm also fine if we
>>> want
>>> > get
>>> > > another several weeks time.
>>> > >
>>> > > Regarding of 3.3 release, I would suggest doing that before
>>> thanksgiving.
>>> > > Do you think is it good or too early / late?
>>> > >
>>> > > Eric,
>>> > >
>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
>>> merged
>>> > > in time, we probably not need the YARN-8220.
>>> > >
>>> > > Sunil,
>>> > >
>>> > > Could u update https://cwiki.apache.org/confluence/display/HADOOP/
>>> > Roadmap
>>> > > with proposed plan as well? We can fill feature list first before
>>> getting
>>> > > consensus of time.
>>> > >
>>> > > Thanks,
>>> > > Wangda
>>> > >
>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>> <fabbri@cloudera.com.invalid
>>> > >
>>> > > wrote:
>>> > >
>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>> stevel@hortonworks.com>
>>> > >> wrote:
>>> > >>
>>> > >> >
>>> > >> >
>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
>>> > >> > sunilg@apache.org>> wrote:
>>> > >> >
>>> > >> > I would also would like to take this opportunity to come up with a
>>> > >> detailed
>>> > >> > plan.
>>> > >> >
>>> > >> > - Feature freeze date : all features should be merged by August
>>> 10,
>>> > >> 2018.
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > <snip>
>>> > >>
>>> > >> >
>>> > >> > Please let me know if I missed any features targeted to 3.2 per
>>> this
>>> > >> >
>>> > >> >
>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>> > >> >
>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>> > >> >
>>> > >> >
>>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen 2
>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>> > >> >
>>> > >> > I don't think this is quite ready yet, I've been doing work on
>>> it, but
>>> > >> if
>>> > >> > we have a 3 week deadline, I'm going to expect some timely
>>> reviews on
>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>> > >> >
>>> > >> > I've uprated that to a blocker feature; will review the S3 &
>>> S3Guard
>>> > >> JIRAs
>>> > >> > to see which of those are blocking. Then there are some pressing
>>> > "guave,
>>> > >> > java 9 prep"
>>> > >> >
>>> > >> >
>>> > >> I can help with this part if you like.
>>> > >>
>>> > >>
>>> > >>
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > timeline. I would like to volunteer myself as release manager of
>>> 3.2.0
>>> > >> > release.
>>> > >> >
>>> > >> >
>>> > >> > well volunteered!
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> Yes, thank you for stepping up.
>>> > >>
>>> > >>
>>> > >> >
>>> > >> > I think this raises a good q: what timetable should we have for
>>> the
>>> > >> 3.2. &
>>> > >> > 3.3 releases; if we do want a faster cadence, then having the
>>> outline
>>> > >> time
>>> > >> > from the 3.2 to the 3.3 release means that there's less concern
>>> about
>>> > >> > things not making the 3.2 dealine
>>> > >> >
>>> > >> > -Steve
>>> > >> >
>>> > >> >
>>> > >> Good idea to mitigate the short deadline.
>>> > >>
>>> > >> -AF
>>> > >>
>>> > >
>>> >
>>>
>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi All,
Inline with earlier communication dated 17th July 2018, I would like to
provide some updates.
We are approaching previously proposed code freeze date (Aug 31).
One of the critical feature Node Attributes feature merge discussion/vote
is ongoing. Also few other Blocker bugs need a bit more time. With regard
to this, suggesting to push the feature/code freeze for 2 more weeks to
accommodate these jiras too.
Proposing Updated changes in plan inline with this:
Feature freeze date : all features to merge by September 7, 2018.
Code freeze date : blockers/critical only, no improvements and
blocker/critical bug-fixes September 14, 2018.
Release date: September 28, 2018
If any features in branch which are targeted to 3.2.0, please reply to this
email thread.
*Here's an updated 3.2.0 feature status:*
1. Merged & Completed features:
- (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
Initial cut.
- (Uma) HDFS-10285: HDFS Storage Policy Satisfier
- (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
- (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
and CLI.
2. Features close to finish:
- (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
Ongoing.
- (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
Patch in progress.
- (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
- (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage. In
progress.
3. Tentative features:
- (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
done before Aug 2018.
- (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
as more discussions are on-going.
*Summary of 3.2.0 issues status:*
26 Blocker and Critical issues [1] are open, I am following up with owners
to get status on each of them to get in by Code Freeze date.
[1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
BY priority DESC
Thanks,
Sunil
On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
> Hi All,
>
> Thanks for the feedbacks. Inline with earlier communication dated 17th
> July 2018, I would like to provide some updates.
>
> We are approaching previously proposed feature freeze date (Aug 21, about
> 7 days from today).
> If any features in branch which are targeted to 3.2.0, please reply to
> this email thread.
> Steve has mentioned about the s3 features which will come close to Code
> Freeze Date (Aug 31st).
>
> *Here's an updated 3.2.0 feature status:*
>
> 1. Merged & Completed features:
>
> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> Initial cut.
> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>
> 2. Features close to finish:
>
> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major patches
> are all in, only one last
> patch is in review state.
> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> Close to commit.
> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> and CLI. 2 patches are pending
> which will be closed by Feature freeze date.
> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
> Patch in progress.
> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
> In progress.
>
> 3. Tentative features:
>
> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
> done before Aug 2018.
> - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
> as more discussions are on-going.
>
> *Summary of 3.2.0 issues status:*
>
> 39 Blocker and Critical issues [1] are open, I am checking with owners to
> get status on each of them to get in by Code Freeze date.
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
> BY priority DESC
>
> Thanks,
> Sunil
>
> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
>
>> Thanks Subru for the thoughts.
>> One of the main reason for a major release is to push out critical
>> features with a faster cadence to the users. If we are pulling more and
>> more different types of features to a minor release, that branch will
>> become more destabilized and it may be tough to say that 3.1.2 is stable
>> that 3.1.1 for eg. We always tend to improve and stabilize features in
>> subsequent minor release.
>> For few companies, it makes sense to push out these new features faster
>> to make a reach to the users. Adding to the point to the backporting
>> issues, I agree that its a pain and we can workaround that with some git
>> scripts. If we can make such scripts available to committers, backport will
>> be seem-less across branches and we can achieve the faster release cadence
>> also.
>>
>> Thoughts?
>>
>> - Sunil
>>
>>
>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org> wrote:
>>
>>> Thanks Sunil for volunteering to lead the release effort. I am generally
>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
>>> already
>>> have too many branches to be maintained. I already see many commits are
>>> in
>>> different branches with no apparent rationale, for e.g: 3.1 has commits
>>> which are absent in 3.0 etc.
>>>
>>> Additionally AFAIK 3.x has not been deployed in any major production
>>> setting so the cost of adding features should be minimal.
>>>
>>> Thoughts?
>>>
>>> -Subru
>>>
>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
>>>
>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>> >
>>> > Yes, important changes and features are much needed, hence we will be
>>> > keeping the door open for them as possible. Also considering few more
>>> > offline requests from other folks, I think extending the timeframe by
>>> > couple of weeks makes sense (including a second RC buffer) and this
>>> should
>>> > ideally help us to ship this by September itself.
>>> >
>>> > Revised dates (I will be updating same in Roadmap wiki as well)
>>> >
>>> > - Feature freeze date : all features to merge by August 21, 2018.
>>> >
>>> > - Code freeze date : blockers/critical only, no improvements and non
>>> > blocker/critical
>>> >
>>> > bug-fixes August 31, 2018.
>>> >
>>> > - Release date: September 15, 2018
>>> >
>>> > Thank Eric and Zian, I think Wangda has already answered your
>>> questions.
>>> >
>>> > Thanks
>>> > Sunil
>>> >
>>> >
>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
>>> wrote:
>>> >
>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for that.
>>> > >
>>> > > To concerns from Steve,
>>> > >
>>> > > It is a good idea to keep the door open to get important changes /
>>> > > features in before cutoff. I would prefer to keep the proposed
>>> release
>>> > date
>>> > > to make sure things can happen earlier instead of last minute and we
>>> all
>>> > > know that releases are always get delayed :). I'm also fine if we
>>> want
>>> > get
>>> > > another several weeks time.
>>> > >
>>> > > Regarding of 3.3 release, I would suggest doing that before
>>> thanksgiving.
>>> > > Do you think is it good or too early / late?
>>> > >
>>> > > Eric,
>>> > >
>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
>>> merged
>>> > > in time, we probably not need the YARN-8220.
>>> > >
>>> > > Sunil,
>>> > >
>>> > > Could u update https://cwiki.apache.org/confluence/display/HADOOP/
>>> > Roadmap
>>> > > with proposed plan as well? We can fill feature list first before
>>> getting
>>> > > consensus of time.
>>> > >
>>> > > Thanks,
>>> > > Wangda
>>> > >
>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>> <fabbri@cloudera.com.invalid
>>> > >
>>> > > wrote:
>>> > >
>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>> stevel@hortonworks.com>
>>> > >> wrote:
>>> > >>
>>> > >> >
>>> > >> >
>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
>>> > >> > sunilg@apache.org>> wrote:
>>> > >> >
>>> > >> > I would also would like to take this opportunity to come up with a
>>> > >> detailed
>>> > >> > plan.
>>> > >> >
>>> > >> > - Feature freeze date : all features should be merged by August
>>> 10,
>>> > >> 2018.
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > <snip>
>>> > >>
>>> > >> >
>>> > >> > Please let me know if I missed any features targeted to 3.2 per
>>> this
>>> > >> >
>>> > >> >
>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>> > >> >
>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>> > >> >
>>> > >> >
>>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen 2
>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>> > >> >
>>> > >> > I don't think this is quite ready yet, I've been doing work on
>>> it, but
>>> > >> if
>>> > >> > we have a 3 week deadline, I'm going to expect some timely
>>> reviews on
>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>> > >> >
>>> > >> > I've uprated that to a blocker feature; will review the S3 &
>>> S3Guard
>>> > >> JIRAs
>>> > >> > to see which of those are blocking. Then there are some pressing
>>> > "guave,
>>> > >> > java 9 prep"
>>> > >> >
>>> > >> >
>>> > >> I can help with this part if you like.
>>> > >>
>>> > >>
>>> > >>
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > timeline. I would like to volunteer myself as release manager of
>>> 3.2.0
>>> > >> > release.
>>> > >> >
>>> > >> >
>>> > >> > well volunteered!
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> Yes, thank you for stepping up.
>>> > >>
>>> > >>
>>> > >> >
>>> > >> > I think this raises a good q: what timetable should we have for
>>> the
>>> > >> 3.2. &
>>> > >> > 3.3 releases; if we do want a faster cadence, then having the
>>> outline
>>> > >> time
>>> > >> > from the 3.2 to the 3.3 release means that there's less concern
>>> about
>>> > >> > things not making the 3.2 dealine
>>> > >> >
>>> > >> > -Steve
>>> > >> >
>>> > >> >
>>> > >> Good idea to mitigate the short deadline.
>>> > >>
>>> > >> -AF
>>> > >>
>>> > >
>>> >
>>>
>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi All,
Inline with earlier communication dated 17th July 2018, I would like to
provide some updates.
We are approaching previously proposed code freeze date (Aug 31).
One of the critical feature Node Attributes feature merge discussion/vote
is ongoing. Also few other Blocker bugs need a bit more time. With regard
to this, suggesting to push the feature/code freeze for 2 more weeks to
accommodate these jiras too.
Proposing Updated changes in plan inline with this:
Feature freeze date : all features to merge by September 7, 2018.
Code freeze date : blockers/critical only, no improvements and
blocker/critical bug-fixes September 14, 2018.
Release date: September 28, 2018
If any features in branch which are targeted to 3.2.0, please reply to this
email thread.
*Here's an updated 3.2.0 feature status:*
1. Merged & Completed features:
- (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
Initial cut.
- (Uma) HDFS-10285: HDFS Storage Policy Satisfier
- (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
- (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
and CLI.
2. Features close to finish:
- (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
Ongoing.
- (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
Patch in progress.
- (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
- (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage. In
progress.
3. Tentative features:
- (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
done before Aug 2018.
- (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
as more discussions are on-going.
*Summary of 3.2.0 issues status:*
26 Blocker and Critical issues [1] are open, I am following up with owners
to get status on each of them to get in by Code Freeze date.
[1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
BY priority DESC
Thanks,
Sunil
On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
> Hi All,
>
> Thanks for the feedbacks. Inline with earlier communication dated 17th
> July 2018, I would like to provide some updates.
>
> We are approaching previously proposed feature freeze date (Aug 21, about
> 7 days from today).
> If any features in branch which are targeted to 3.2.0, please reply to
> this email thread.
> Steve has mentioned about the s3 features which will come close to Code
> Freeze Date (Aug 31st).
>
> *Here's an updated 3.2.0 feature status:*
>
> 1. Merged & Completed features:
>
> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> Initial cut.
> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>
> 2. Features close to finish:
>
> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major patches
> are all in, only one last
> patch is in review state.
> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> Close to commit.
> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> and CLI. 2 patches are pending
> which will be closed by Feature freeze date.
> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
> Patch in progress.
> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
> In progress.
>
> 3. Tentative features:
>
> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
> done before Aug 2018.
> - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
> as more discussions are on-going.
>
> *Summary of 3.2.0 issues status:*
>
> 39 Blocker and Critical issues [1] are open, I am checking with owners to
> get status on each of them to get in by Code Freeze date.
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
> BY priority DESC
>
> Thanks,
> Sunil
>
> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
>
>> Thanks Subru for the thoughts.
>> One of the main reason for a major release is to push out critical
>> features with a faster cadence to the users. If we are pulling more and
>> more different types of features to a minor release, that branch will
>> become more destabilized and it may be tough to say that 3.1.2 is stable
>> that 3.1.1 for eg. We always tend to improve and stabilize features in
>> subsequent minor release.
>> For few companies, it makes sense to push out these new features faster
>> to make a reach to the users. Adding to the point to the backporting
>> issues, I agree that its a pain and we can workaround that with some git
>> scripts. If we can make such scripts available to committers, backport will
>> be seem-less across branches and we can achieve the faster release cadence
>> also.
>>
>> Thoughts?
>>
>> - Sunil
>>
>>
>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org> wrote:
>>
>>> Thanks Sunil for volunteering to lead the release effort. I am generally
>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
>>> already
>>> have too many branches to be maintained. I already see many commits are
>>> in
>>> different branches with no apparent rationale, for e.g: 3.1 has commits
>>> which are absent in 3.0 etc.
>>>
>>> Additionally AFAIK 3.x has not been deployed in any major production
>>> setting so the cost of adding features should be minimal.
>>>
>>> Thoughts?
>>>
>>> -Subru
>>>
>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
>>>
>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>> >
>>> > Yes, important changes and features are much needed, hence we will be
>>> > keeping the door open for them as possible. Also considering few more
>>> > offline requests from other folks, I think extending the timeframe by
>>> > couple of weeks makes sense (including a second RC buffer) and this
>>> should
>>> > ideally help us to ship this by September itself.
>>> >
>>> > Revised dates (I will be updating same in Roadmap wiki as well)
>>> >
>>> > - Feature freeze date : all features to merge by August 21, 2018.
>>> >
>>> > - Code freeze date : blockers/critical only, no improvements and non
>>> > blocker/critical
>>> >
>>> > bug-fixes August 31, 2018.
>>> >
>>> > - Release date: September 15, 2018
>>> >
>>> > Thank Eric and Zian, I think Wangda has already answered your
>>> questions.
>>> >
>>> > Thanks
>>> > Sunil
>>> >
>>> >
>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
>>> wrote:
>>> >
>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for that.
>>> > >
>>> > > To concerns from Steve,
>>> > >
>>> > > It is a good idea to keep the door open to get important changes /
>>> > > features in before cutoff. I would prefer to keep the proposed
>>> release
>>> > date
>>> > > to make sure things can happen earlier instead of last minute and we
>>> all
>>> > > know that releases are always get delayed :). I'm also fine if we
>>> want
>>> > get
>>> > > another several weeks time.
>>> > >
>>> > > Regarding of 3.3 release, I would suggest doing that before
>>> thanksgiving.
>>> > > Do you think is it good or too early / late?
>>> > >
>>> > > Eric,
>>> > >
>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
>>> merged
>>> > > in time, we probably not need the YARN-8220.
>>> > >
>>> > > Sunil,
>>> > >
>>> > > Could u update https://cwiki.apache.org/confluence/display/HADOOP/
>>> > Roadmap
>>> > > with proposed plan as well? We can fill feature list first before
>>> getting
>>> > > consensus of time.
>>> > >
>>> > > Thanks,
>>> > > Wangda
>>> > >
>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>> <fabbri@cloudera.com.invalid
>>> > >
>>> > > wrote:
>>> > >
>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>> stevel@hortonworks.com>
>>> > >> wrote:
>>> > >>
>>> > >> >
>>> > >> >
>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
>>> > >> > sunilg@apache.org>> wrote:
>>> > >> >
>>> > >> > I would also would like to take this opportunity to come up with a
>>> > >> detailed
>>> > >> > plan.
>>> > >> >
>>> > >> > - Feature freeze date : all features should be merged by August
>>> 10,
>>> > >> 2018.
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > <snip>
>>> > >>
>>> > >> >
>>> > >> > Please let me know if I missed any features targeted to 3.2 per
>>> this
>>> > >> >
>>> > >> >
>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>> > >> >
>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>> > >> >
>>> > >> >
>>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen 2
>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>> > >> >
>>> > >> > I don't think this is quite ready yet, I've been doing work on
>>> it, but
>>> > >> if
>>> > >> > we have a 3 week deadline, I'm going to expect some timely
>>> reviews on
>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>> > >> >
>>> > >> > I've uprated that to a blocker feature; will review the S3 &
>>> S3Guard
>>> > >> JIRAs
>>> > >> > to see which of those are blocking. Then there are some pressing
>>> > "guave,
>>> > >> > java 9 prep"
>>> > >> >
>>> > >> >
>>> > >> I can help with this part if you like.
>>> > >>
>>> > >>
>>> > >>
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > timeline. I would like to volunteer myself as release manager of
>>> 3.2.0
>>> > >> > release.
>>> > >> >
>>> > >> >
>>> > >> > well volunteered!
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> Yes, thank you for stepping up.
>>> > >>
>>> > >>
>>> > >> >
>>> > >> > I think this raises a good q: what timetable should we have for
>>> the
>>> > >> 3.2. &
>>> > >> > 3.3 releases; if we do want a faster cadence, then having the
>>> outline
>>> > >> time
>>> > >> > from the 3.2 to the 3.3 release means that there's less concern
>>> about
>>> > >> > things not making the 3.2 dealine
>>> > >> >
>>> > >> > -Steve
>>> > >> >
>>> > >> >
>>> > >> Good idea to mitigate the short deadline.
>>> > >>
>>> > >> -AF
>>> > >>
>>> > >
>>> >
>>>
>>
Re: Hadoop 3.2 Release Plan proposal
Posted by Sunil G <su...@apache.org>.
Hi All,
Inline with earlier communication dated 17th July 2018, I would like to
provide some updates.
We are approaching previously proposed code freeze date (Aug 31).
One of the critical feature Node Attributes feature merge discussion/vote
is ongoing. Also few other Blocker bugs need a bit more time. With regard
to this, suggesting to push the feature/code freeze for 2 more weeks to
accommodate these jiras too.
Proposing Updated changes in plan inline with this:
Feature freeze date : all features to merge by September 7, 2018.
Code freeze date : blockers/critical only, no improvements and
blocker/critical bug-fixes September 14, 2018.
Release date: September 28, 2018
If any features in branch which are targeted to 3.2.0, please reply to this
email thread.
*Here's an updated 3.2.0 feature status:*
1. Merged & Completed features:
- (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
Initial cut.
- (Uma) HDFS-10285: HDFS Storage Policy Satisfier
- (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
- (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
and CLI.
2. Features close to finish:
- (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
Ongoing.
- (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
Patch in progress.
- (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
- (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage. In
progress.
3. Tentative features:
- (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
done before Aug 2018.
- (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
as more discussions are on-going.
*Summary of 3.2.0 issues status:*
26 Blocker and Critical issues [1] are open, I am following up with owners
to get status on each of them to get in by Code Freeze date.
[1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
BY priority DESC
Thanks,
Sunil
On Tue, Aug 14, 2018 at 10:30 PM Sunil G <su...@apache.org> wrote:
> Hi All,
>
> Thanks for the feedbacks. Inline with earlier communication dated 17th
> July 2018, I would like to provide some updates.
>
> We are approaching previously proposed feature freeze date (Aug 21, about
> 7 days from today).
> If any features in branch which are targeted to 3.2.0, please reply to
> this email thread.
> Steve has mentioned about the s3 features which will come close to Code
> Freeze Date (Aug 31st).
>
> *Here's an updated 3.2.0 feature status:*
>
> 1. Merged & Completed features:
>
> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> Initial cut.
> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>
> 2. Features close to finish:
>
> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major patches
> are all in, only one last
> patch is in review state.
> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> Close to commit.
> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> and CLI. 2 patches are pending
> which will be closed by Feature freeze date.
> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
> Patch in progress.
> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
> In progress.
>
> 3. Tentative features:
>
> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
> done before Aug 2018.
> - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
> as more discussions are on-going.
>
> *Summary of 3.2.0 issues status:*
>
> 39 Blocker and Critical issues [1] are open, I am checking with owners to
> get status on each of them to get in by Code Freeze date.
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
> BY priority DESC
>
> Thanks,
> Sunil
>
> On Fri, Jul 20, 2018 at 8:03 AM Sunil G <su...@apache.org> wrote:
>
>> Thanks Subru for the thoughts.
>> One of the main reason for a major release is to push out critical
>> features with a faster cadence to the users. If we are pulling more and
>> more different types of features to a minor release, that branch will
>> become more destabilized and it may be tough to say that 3.1.2 is stable
>> that 3.1.1 for eg. We always tend to improve and stabilize features in
>> subsequent minor release.
>> For few companies, it makes sense to push out these new features faster
>> to make a reach to the users. Adding to the point to the backporting
>> issues, I agree that its a pain and we can workaround that with some git
>> scripts. If we can make such scripts available to committers, backport will
>> be seem-less across branches and we can achieve the faster release cadence
>> also.
>>
>> Thoughts?
>>
>> - Sunil
>>
>>
>> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org> wrote:
>>
>>> Thanks Sunil for volunteering to lead the release effort. I am generally
>>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we
>>> already
>>> have too many branches to be maintained. I already see many commits are
>>> in
>>> different branches with no apparent rationale, for e.g: 3.1 has commits
>>> which are absent in 3.0 etc.
>>>
>>> Additionally AFAIK 3.x has not been deployed in any major production
>>> setting so the cost of adding features should be minimal.
>>>
>>> Thoughts?
>>>
>>> -Subru
>>>
>>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <su...@apache.org> wrote:
>>>
>>> > Thanks Steve, Aaron, Wangda for sharing thoughts.
>>> >
>>> > Yes, important changes and features are much needed, hence we will be
>>> > keeping the door open for them as possible. Also considering few more
>>> > offline requests from other folks, I think extending the timeframe by
>>> > couple of weeks makes sense (including a second RC buffer) and this
>>> should
>>> > ideally help us to ship this by September itself.
>>> >
>>> > Revised dates (I will be updating same in Roadmap wiki as well)
>>> >
>>> > - Feature freeze date : all features to merge by August 21, 2018.
>>> >
>>> > - Code freeze date : blockers/critical only, no improvements and non
>>> > blocker/critical
>>> >
>>> > bug-fixes August 31, 2018.
>>> >
>>> > - Release date: September 15, 2018
>>> >
>>> > Thank Eric and Zian, I think Wangda has already answered your
>>> questions.
>>> >
>>> > Thanks
>>> > Sunil
>>> >
>>> >
>>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wh...@gmail.com>
>>> wrote:
>>> >
>>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for that.
>>> > >
>>> > > To concerns from Steve,
>>> > >
>>> > > It is a good idea to keep the door open to get important changes /
>>> > > features in before cutoff. I would prefer to keep the proposed
>>> release
>>> > date
>>> > > to make sure things can happen earlier instead of last minute and we
>>> all
>>> > > know that releases are always get delayed :). I'm also fine if we
>>> want
>>> > get
>>> > > another several weeks time.
>>> > >
>>> > > Regarding of 3.3 release, I would suggest doing that before
>>> thanksgiving.
>>> > > Do you think is it good or too early / late?
>>> > >
>>> > > Eric,
>>> > >
>>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get
>>> merged
>>> > > in time, we probably not need the YARN-8220.
>>> > >
>>> > > Sunil,
>>> > >
>>> > > Could u update https://cwiki.apache.org/confluence/display/HADOOP/
>>> > Roadmap
>>> > > with proposed plan as well? We can fill feature list first before
>>> getting
>>> > > consensus of time.
>>> > >
>>> > > Thanks,
>>> > > Wangda
>>> > >
>>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri
>>> <fabbri@cloudera.com.invalid
>>> > >
>>> > > wrote:
>>> > >
>>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran <
>>> stevel@hortonworks.com>
>>> > >> wrote:
>>> > >>
>>> > >> >
>>> > >> >
>>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sunilg@apache.org<mailto:
>>> > >> > sunilg@apache.org>> wrote:
>>> > >> >
>>> > >> > I would also would like to take this opportunity to come up with a
>>> > >> detailed
>>> > >> > plan.
>>> > >> >
>>> > >> > - Feature freeze date : all features should be merged by August
>>> 10,
>>> > >> 2018.
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > <snip>
>>> > >>
>>> > >> >
>>> > >> > Please let me know if I missed any features targeted to 3.2 per
>>> this
>>> > >> >
>>> > >> >
>>> > >> > Well there these big todo lists for S3 & S3Guard.
>>> > >> >
>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226
>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220
>>> > >> >
>>> > >> >
>>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen 2
>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407
>>> > >> >
>>> > >> > I don't think this is quite ready yet, I've been doing work on
>>> it, but
>>> > >> if
>>> > >> > we have a 3 week deadline, I'm going to expect some timely
>>> reviews on
>>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546
>>> > >> >
>>> > >> > I've uprated that to a blocker feature; will review the S3 &
>>> S3Guard
>>> > >> JIRAs
>>> > >> > to see which of those are blocking. Then there are some pressing
>>> > "guave,
>>> > >> > java 9 prep"
>>> > >> >
>>> > >> >
>>> > >> I can help with this part if you like.
>>> > >>
>>> > >>
>>> > >>
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > timeline. I would like to volunteer myself as release manager of
>>> 3.2.0
>>> > >> > release.
>>> > >> >
>>> > >> >
>>> > >> > well volunteered!
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> Yes, thank you for stepping up.
>>> > >>
>>> > >>
>>> > >> >
>>> > >> > I think this raises a good q: what timetable should we have for
>>> the
>>> > >> 3.2. &
>>> > >> > 3.3 releases; if we do want a faster cadence, then having the
>>> outline
>>> > >> time
>>> > >> > from the 3.2 to the 3.3 release means that there's less concern
>>> about
>>> > >> > things not making the 3.2 dealine
>>> > >> >
>>> > >> > -Steve
>>> > >> >
>>> > >> >
>>> > >> Good idea to mitigate the short deadline.
>>> > >>
>>> > >> -AF
>>> > >>
>>> > >
>>> >
>>>
>>