Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Hi All,

Since we're fast approaching previously proposed feature freeze date (Jan
30, about 13 days from today). If you've any features which live in a
branch and targeted to 3.1.0, please reply this email thread. Ideally, we
should finish branch merging before feature freeze date.

Here's an updated 3.1.0 feature status:

1. Merged & Completed features:
* (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
* (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
end-to-end.
* (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
* (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename commits.
* (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
Queues While Doing Queue Mapping.
* (Chris Douglas) HDFS-9806: HDFS Tiered Storage.

2. Features close to finish:
* (Zhankun) YARN-5983: FPGA support. Majority implementations completed and
merged to trunk. Except for UI/documentation.
* (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
discussions going on about implementation.
* (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
anti-affinity support. Close to finish, on track to be merged before Jan 30.

3. Tentative features:
* (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
containers. Only one pending patch. Plan to finish before Jan 7th.
* (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to be
done before Jan 2018.
* (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
challenging to be done before Jan 2018.
* (Varun V) YARN-5673: container-executor write. Given security refactoring
of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to 3.2.

Thanks,
Wangda




On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com> wrote:

> Hi all,
>
> Congratulations on the 3.0.0-GA release!
>
> As we discussed in the previous email thread [1], I'd like to restart
> 3.1.0 release plans.
>
> a) Quick summary:
> a.1 Release status
> We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
> there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open blockers and
> 22 open critical issues.
>
> a.2 Release date update
> Considering delays of 3.0-GA release by month-and-a-half, I propose to
> move the dates as follows
>  - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last date for
> any branches to get merged too;
>  - code freeze (blockers & critical only) date to Feb 08, 2018;
>  - release voting start by Feb 18, 2018, leaving time for at least two RCx
>  - release date from Jan 15, 2018, to Feb 28, 2018;
>
> Unlike before, I added an additional milestone for release-vote-start so
> that we can account for voting time-period also.
>
> This overall is still 5 1/2 months of release-timeline unlike the faster
> cadence we hoped for, but this, in my opinion, is the best-updated timeline
> given the delays of the final release of 3.0-GA.
>
> b) Individual feature status:
> I spoke to several feature owners and checked the status of un-finished
> features, following are status of features planned to 3.1.0:
>
> b.1 Merged & Completed features:
> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
> end-to-end.
> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
> commits.
> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
> Queues While Doing Queue Mapping.
>
> b.2 Features close to finish:
> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
> and merged to trunk. Except for UI/documentation.
> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
> discussions going on about implementation.
>
> b.3 Tentative features:
> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
> containers. Only one pending patch. Plan to finish before Jan 7th.
> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
> be done before Jan 2018.
> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
> anti-affinity support. Tentative will figure out by Jan 1st.
> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
> challenging to be done before Jan 2018.
> * (Varun V) YARN-5673: container-executor write. Given security
> refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may be
> moved to 3.2.
>
> b.4 Additional release drivers
> * More exhaustive upgrade testing from 2.x to 3.x.
>
> c) Regarding branch cut:
>
> We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
> feature planned to 3.2 has to be landed on trunk or B. After feature freeze
> date, whichever comes first.
>
> I've also talked offline with Vinod to get help on release-management
> given this is my first release. He agreed to help do this release jointly.
>
> Thoughts?
>
> Thanks,
> Wangda Tan
>
> [1] https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0
> 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
> [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>
>

Re: Hadoop 3.1.0 release discussion

[Arun Suresh <as...@apache.org>]

Hey Wangda

Update: YARN-6592 has been merged to trunk.

Cheers
-Arun

On Wed, Jan 31, 2018 at 12:58 AM, Wangda Tan <wh...@gmail.com> wrote:

> Hi Daniel,
>
> I agree, let's move the discussion to YARN-7292 and get it resolved before
> 3.1.0
>
> Thanks,
> Wangda
>
> On Wed, Jan 31, 2018 at 4:35 PM, Daniel Templeton <da...@cloudera.com>
> wrote:
>
> > I added my comments on that JIRA.  Looks like YARN-7292 is marked as a
> > blocker for 3.1, and I would tend to agree with that.  Let's see what we
> > can to do get profiles nailed down so that 3.1 can go forward.
> >
> > Daniel
> >
> >
> > On 1/18/18 10:25 AM, Wangda Tan wrote:
> >
> > Thanks Daniel,
> >
> > We need to make a decision about this: https://issues.apache.
> > org/jira/browse/YARN-7292, I believe this is the JIRA you mentioned
> > correct? Please let me know if there's anything else. And let's move the
> > discussion on JIRA.
> >
> > The good news is that resource profile is merged to trunk already so we
> > can finish that before code freeze date (Feb 08).
> >
> > + Sunil as well.
> >
> > Thanks,
> > Wangda
> >
> >
> >
> > On Wed, Jan 17, 2018 at 4:31 PM, Daniel Templeton <da...@cloudera.com>
> > wrote:
> >
> >> What's the status on resource profiles?  I believe there are still a
> >> couple of open JIRAs to rethink some of the design choices.
> >>
> >> Daniel
> >>
> >>
> >> On 1/17/18 11:33 AM, Wangda Tan wrote:
> >>
> >>> Hi All,
> >>>
> >>> Since we're fast approaching previously proposed feature freeze date
> (Jan
> >>> 30, about 13 days from today). If you've any features which live in a
> >>> branch and targeted to 3.1.0, please reply this email thread. Ideally,
> we
> >>> should finish branch merging before feature freeze date.
> >>>
> >>> Here's an updated 3.1.0 feature status:
> >>>
> >>> 1. Merged & Completed features:
> >>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
> >>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
> >>> end-to-end.
> >>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
> >>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
> >>> commits.
> >>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
> >>> Queues While Doing Queue Mapping.
> >>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
> >>>
> >>> 2. Features close to finish:
> >>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
> >>> and
> >>> merged to trunk. Except for UI/documentation.
> >>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
> >>> discussions going on about implementation.
> >>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
> >>> anti-affinity support. Close to finish, on track to be merged before
> Jan
> >>> 30.
> >>>
> >>> 3. Tentative features:
> >>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
> >>> containers. Only one pending patch. Plan to finish before Jan 7th.
> >>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging
> to
> >>> be
> >>> done before Jan 2018.
> >>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
> >>> challenging to be done before Jan 2018.
> >>> * (Varun V) YARN-5673: container-executor write. Given security
> >>> refactoring
> >>> of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to
> >>> 3.2.
> >>>
> >>> Thanks,
> >>> Wangda
> >>>
> >>>
> >>>
> >>>
> >>> On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com>
> wrote:
> >>>
> >>> Hi all,
> >>>>
> >>>> Congratulations on the 3.0.0-GA release!
> >>>>
> >>>> As we discussed in the previous email thread [1], I'd like to restart
> >>>> 3.1.0 release plans.
> >>>>
> >>>> a) Quick summary:
> >>>> a.1 Release status
> >>>> We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
> >>>> there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
> blockers
> >>>> and
> >>>> 22 open critical issues.
> >>>>
> >>>> a.2 Release date update
> >>>> Considering delays of 3.0-GA release by month-and-a-half, I propose to
> >>>> move the dates as follows
> >>>>   - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last date
> >>>> for
> >>>> any branches to get merged too;
> >>>>   - code freeze (blockers & critical only) date to Feb 08, 2018;
> >>>>   - release voting start by Feb 18, 2018, leaving time for at least
> two
> >>>> RCx
> >>>>   - release date from Jan 15, 2018, to Feb 28, 2018;
> >>>>
> >>>> Unlike before, I added an additional milestone for release-vote-start
> so
> >>>> that we can account for voting time-period also.
> >>>>
> >>>> This overall is still 5 1/2 months of release-timeline unlike the
> faster
> >>>> cadence we hoped for, but this, in my opinion, is the best-updated
> >>>> timeline
> >>>> given the delays of the final release of 3.0-GA.
> >>>>
> >>>> b) Individual feature status:
> >>>> I spoke to several feature owners and checked the status of
> un-finished
> >>>> features, following are status of features planned to 3.1.0:
> >>>>
> >>>> b.1 Merged & Completed features:
> >>>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
> >>>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
> >>>> end-to-end.
> >>>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
> >>>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
> >>>> commits.
> >>>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
> >>>> Queues While Doing Queue Mapping.
> >>>>
> >>>> b.2 Features close to finish:
> >>>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
> >>>> * (Zhankun) YARN-5983: FPGA support. Majority implementations
> completed
> >>>> and merged to trunk. Except for UI/documentation.
> >>>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
> >>>> discussions going on about implementation.
> >>>>
> >>>> b.3 Tentative features:
> >>>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
> >>>> containers. Only one pending patch. Plan to finish before Jan 7th.
> >>>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging
> to
> >>>> be done before Jan 2018.
> >>>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest
> and
> >>>> anti-affinity support. Tentative will figure out by Jan 1st.
> >>>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
> >>>> challenging to be done before Jan 2018.
> >>>> * (Varun V) YARN-5673: container-executor write. Given security
> >>>> refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may
> >>>> be
> >>>> moved to 3.2.
> >>>>
> >>>> b.4 Additional release drivers
> >>>> * More exhaustive upgrade testing from 2.x to 3.x.
> >>>>
> >>>> c) Regarding branch cut:
> >>>>
> >>>> We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
> >>>> feature planned to 3.2 has to be landed on trunk or B. After feature
> >>>> freeze
> >>>> date, whichever comes first.
> >>>>
> >>>> I've also talked offline with Vinod to get help on release-management
> >>>> given this is my first release. He agreed to help do this release
> >>>> jointly.
> >>>>
> >>>> Thoughts?
> >>>>
> >>>> Thanks,
> >>>> Wangda Tan
> >>>>
> >>>> [1] https://lists.apache.org/thread.html/
> c11506c3250c9481852130616b3cb0
> >>>> 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
> >>>> [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> >>>> (3.1.0)
> >>>> AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
> >>>>
> >>>>
> >>>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>
> >>
> >
> >
>

Re: Hadoop 3.1.0 release discussion

[Arun Suresh <as...@apache.org>]

Hey Wangda

Update: YARN-6592 has been merged to trunk.

Cheers
-Arun

On Wed, Jan 31, 2018 at 12:58 AM, Wangda Tan <wh...@gmail.com> wrote:

> Hi Daniel,
>
> I agree, let's move the discussion to YARN-7292 and get it resolved before
> 3.1.0
>
> Thanks,
> Wangda
>
> On Wed, Jan 31, 2018 at 4:35 PM, Daniel Templeton <da...@cloudera.com>
> wrote:
>
> > I added my comments on that JIRA.  Looks like YARN-7292 is marked as a
> > blocker for 3.1, and I would tend to agree with that.  Let's see what we
> > can to do get profiles nailed down so that 3.1 can go forward.
> >
> > Daniel
> >
> >
> > On 1/18/18 10:25 AM, Wangda Tan wrote:
> >
> > Thanks Daniel,
> >
> > We need to make a decision about this: https://issues.apache.
> > org/jira/browse/YARN-7292, I believe this is the JIRA you mentioned
> > correct? Please let me know if there's anything else. And let's move the
> > discussion on JIRA.
> >
> > The good news is that resource profile is merged to trunk already so we
> > can finish that before code freeze date (Feb 08).
> >
> > + Sunil as well.
> >
> > Thanks,
> > Wangda
> >
> >
> >
> > On Wed, Jan 17, 2018 at 4:31 PM, Daniel Templeton <da...@cloudera.com>
> > wrote:
> >
> >> What's the status on resource profiles?  I believe there are still a
> >> couple of open JIRAs to rethink some of the design choices.
> >>
> >> Daniel
> >>
> >>
> >> On 1/17/18 11:33 AM, Wangda Tan wrote:
> >>
> >>> Hi All,
> >>>
> >>> Since we're fast approaching previously proposed feature freeze date
> (Jan
> >>> 30, about 13 days from today). If you've any features which live in a
> >>> branch and targeted to 3.1.0, please reply this email thread. Ideally,
> we
> >>> should finish branch merging before feature freeze date.
> >>>
> >>> Here's an updated 3.1.0 feature status:
> >>>
> >>> 1. Merged & Completed features:
> >>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
> >>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
> >>> end-to-end.
> >>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
> >>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
> >>> commits.
> >>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
> >>> Queues While Doing Queue Mapping.
> >>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
> >>>
> >>> 2. Features close to finish:
> >>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
> >>> and
> >>> merged to trunk. Except for UI/documentation.
> >>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
> >>> discussions going on about implementation.
> >>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
> >>> anti-affinity support. Close to finish, on track to be merged before
> Jan
> >>> 30.
> >>>
> >>> 3. Tentative features:
> >>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
> >>> containers. Only one pending patch. Plan to finish before Jan 7th.
> >>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging
> to
> >>> be
> >>> done before Jan 2018.
> >>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
> >>> challenging to be done before Jan 2018.
> >>> * (Varun V) YARN-5673: container-executor write. Given security
> >>> refactoring
> >>> of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to
> >>> 3.2.
> >>>
> >>> Thanks,
> >>> Wangda
> >>>
> >>>
> >>>
> >>>
> >>> On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com>
> wrote:
> >>>
> >>> Hi all,
> >>>>
> >>>> Congratulations on the 3.0.0-GA release!
> >>>>
> >>>> As we discussed in the previous email thread [1], I'd like to restart
> >>>> 3.1.0 release plans.
> >>>>
> >>>> a) Quick summary:
> >>>> a.1 Release status
> >>>> We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
> >>>> there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
> blockers
> >>>> and
> >>>> 22 open critical issues.
> >>>>
> >>>> a.2 Release date update
> >>>> Considering delays of 3.0-GA release by month-and-a-half, I propose to
> >>>> move the dates as follows
> >>>>   - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last date
> >>>> for
> >>>> any branches to get merged too;
> >>>>   - code freeze (blockers & critical only) date to Feb 08, 2018;
> >>>>   - release voting start by Feb 18, 2018, leaving time for at least
> two
> >>>> RCx
> >>>>   - release date from Jan 15, 2018, to Feb 28, 2018;
> >>>>
> >>>> Unlike before, I added an additional milestone for release-vote-start
> so
> >>>> that we can account for voting time-period also.
> >>>>
> >>>> This overall is still 5 1/2 months of release-timeline unlike the
> faster
> >>>> cadence we hoped for, but this, in my opinion, is the best-updated
> >>>> timeline
> >>>> given the delays of the final release of 3.0-GA.
> >>>>
> >>>> b) Individual feature status:
> >>>> I spoke to several feature owners and checked the status of
> un-finished
> >>>> features, following are status of features planned to 3.1.0:
> >>>>
> >>>> b.1 Merged & Completed features:
> >>>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
> >>>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
> >>>> end-to-end.
> >>>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
> >>>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
> >>>> commits.
> >>>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
> >>>> Queues While Doing Queue Mapping.
> >>>>
> >>>> b.2 Features close to finish:
> >>>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
> >>>> * (Zhankun) YARN-5983: FPGA support. Majority implementations
> completed
> >>>> and merged to trunk. Except for UI/documentation.
> >>>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
> >>>> discussions going on about implementation.
> >>>>
> >>>> b.3 Tentative features:
> >>>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
> >>>> containers. Only one pending patch. Plan to finish before Jan 7th.
> >>>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging
> to
> >>>> be done before Jan 2018.
> >>>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest
> and
> >>>> anti-affinity support. Tentative will figure out by Jan 1st.
> >>>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
> >>>> challenging to be done before Jan 2018.
> >>>> * (Varun V) YARN-5673: container-executor write. Given security
> >>>> refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may
> >>>> be
> >>>> moved to 3.2.
> >>>>
> >>>> b.4 Additional release drivers
> >>>> * More exhaustive upgrade testing from 2.x to 3.x.
> >>>>
> >>>> c) Regarding branch cut:
> >>>>
> >>>> We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
> >>>> feature planned to 3.2 has to be landed on trunk or B. After feature
> >>>> freeze
> >>>> date, whichever comes first.
> >>>>
> >>>> I've also talked offline with Vinod to get help on release-management
> >>>> given this is my first release. He agreed to help do this release
> >>>> jointly.
> >>>>
> >>>> Thoughts?
> >>>>
> >>>> Thanks,
> >>>> Wangda Tan
> >>>>
> >>>> [1] https://lists.apache.org/thread.html/
> c11506c3250c9481852130616b3cb0
> >>>> 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
> >>>> [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> >>>> (3.1.0)
> >>>> AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
> >>>>
> >>>>
> >>>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>
> >>
> >
> >
>

Re: Hadoop 3.1.0 release discussion

[Arun Suresh <as...@apache.org>]

Hey Wangda

Update: YARN-6592 has been merged to trunk.

Cheers
-Arun

On Wed, Jan 31, 2018 at 12:58 AM, Wangda Tan <wh...@gmail.com> wrote:

> Hi Daniel,
>
> I agree, let's move the discussion to YARN-7292 and get it resolved before
> 3.1.0
>
> Thanks,
> Wangda
>
> On Wed, Jan 31, 2018 at 4:35 PM, Daniel Templeton <da...@cloudera.com>
> wrote:
>
> > I added my comments on that JIRA.  Looks like YARN-7292 is marked as a
> > blocker for 3.1, and I would tend to agree with that.  Let's see what we
> > can to do get profiles nailed down so that 3.1 can go forward.
> >
> > Daniel
> >
> >
> > On 1/18/18 10:25 AM, Wangda Tan wrote:
> >
> > Thanks Daniel,
> >
> > We need to make a decision about this: https://issues.apache.
> > org/jira/browse/YARN-7292, I believe this is the JIRA you mentioned
> > correct? Please let me know if there's anything else. And let's move the
> > discussion on JIRA.
> >
> > The good news is that resource profile is merged to trunk already so we
> > can finish that before code freeze date (Feb 08).
> >
> > + Sunil as well.
> >
> > Thanks,
> > Wangda
> >
> >
> >
> > On Wed, Jan 17, 2018 at 4:31 PM, Daniel Templeton <da...@cloudera.com>
> > wrote:
> >
> >> What's the status on resource profiles?  I believe there are still a
> >> couple of open JIRAs to rethink some of the design choices.
> >>
> >> Daniel
> >>
> >>
> >> On 1/17/18 11:33 AM, Wangda Tan wrote:
> >>
> >>> Hi All,
> >>>
> >>> Since we're fast approaching previously proposed feature freeze date
> (Jan
> >>> 30, about 13 days from today). If you've any features which live in a
> >>> branch and targeted to 3.1.0, please reply this email thread. Ideally,
> we
> >>> should finish branch merging before feature freeze date.
> >>>
> >>> Here's an updated 3.1.0 feature status:
> >>>
> >>> 1. Merged & Completed features:
> >>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
> >>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
> >>> end-to-end.
> >>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
> >>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
> >>> commits.
> >>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
> >>> Queues While Doing Queue Mapping.
> >>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
> >>>
> >>> 2. Features close to finish:
> >>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
> >>> and
> >>> merged to trunk. Except for UI/documentation.
> >>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
> >>> discussions going on about implementation.
> >>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
> >>> anti-affinity support. Close to finish, on track to be merged before
> Jan
> >>> 30.
> >>>
> >>> 3. Tentative features:
> >>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
> >>> containers. Only one pending patch. Plan to finish before Jan 7th.
> >>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging
> to
> >>> be
> >>> done before Jan 2018.
> >>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
> >>> challenging to be done before Jan 2018.
> >>> * (Varun V) YARN-5673: container-executor write. Given security
> >>> refactoring
> >>> of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to
> >>> 3.2.
> >>>
> >>> Thanks,
> >>> Wangda
> >>>
> >>>
> >>>
> >>>
> >>> On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com>
> wrote:
> >>>
> >>> Hi all,
> >>>>
> >>>> Congratulations on the 3.0.0-GA release!
> >>>>
> >>>> As we discussed in the previous email thread [1], I'd like to restart
> >>>> 3.1.0 release plans.
> >>>>
> >>>> a) Quick summary:
> >>>> a.1 Release status
> >>>> We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
> >>>> there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
> blockers
> >>>> and
> >>>> 22 open critical issues.
> >>>>
> >>>> a.2 Release date update
> >>>> Considering delays of 3.0-GA release by month-and-a-half, I propose to
> >>>> move the dates as follows
> >>>>   - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last date
> >>>> for
> >>>> any branches to get merged too;
> >>>>   - code freeze (blockers & critical only) date to Feb 08, 2018;
> >>>>   - release voting start by Feb 18, 2018, leaving time for at least
> two
> >>>> RCx
> >>>>   - release date from Jan 15, 2018, to Feb 28, 2018;
> >>>>
> >>>> Unlike before, I added an additional milestone for release-vote-start
> so
> >>>> that we can account for voting time-period also.
> >>>>
> >>>> This overall is still 5 1/2 months of release-timeline unlike the
> faster
> >>>> cadence we hoped for, but this, in my opinion, is the best-updated
> >>>> timeline
> >>>> given the delays of the final release of 3.0-GA.
> >>>>
> >>>> b) Individual feature status:
> >>>> I spoke to several feature owners and checked the status of
> un-finished
> >>>> features, following are status of features planned to 3.1.0:
> >>>>
> >>>> b.1 Merged & Completed features:
> >>>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
> >>>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
> >>>> end-to-end.
> >>>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
> >>>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
> >>>> commits.
> >>>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
> >>>> Queues While Doing Queue Mapping.
> >>>>
> >>>> b.2 Features close to finish:
> >>>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
> >>>> * (Zhankun) YARN-5983: FPGA support. Majority implementations
> completed
> >>>> and merged to trunk. Except for UI/documentation.
> >>>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
> >>>> discussions going on about implementation.
> >>>>
> >>>> b.3 Tentative features:
> >>>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
> >>>> containers. Only one pending patch. Plan to finish before Jan 7th.
> >>>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging
> to
> >>>> be done before Jan 2018.
> >>>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest
> and
> >>>> anti-affinity support. Tentative will figure out by Jan 1st.
> >>>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
> >>>> challenging to be done before Jan 2018.
> >>>> * (Varun V) YARN-5673: container-executor write. Given security
> >>>> refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may
> >>>> be
> >>>> moved to 3.2.
> >>>>
> >>>> b.4 Additional release drivers
> >>>> * More exhaustive upgrade testing from 2.x to 3.x.
> >>>>
> >>>> c) Regarding branch cut:
> >>>>
> >>>> We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
> >>>> feature planned to 3.2 has to be landed on trunk or B. After feature
> >>>> freeze
> >>>> date, whichever comes first.
> >>>>
> >>>> I've also talked offline with Vinod to get help on release-management
> >>>> given this is my first release. He agreed to help do this release
> >>>> jointly.
> >>>>
> >>>> Thoughts?
> >>>>
> >>>> Thanks,
> >>>> Wangda Tan
> >>>>
> >>>> [1] https://lists.apache.org/thread.html/
> c11506c3250c9481852130616b3cb0
> >>>> 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
> >>>> [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> >>>> (3.1.0)
> >>>> AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
> >>>>
> >>>>
> >>>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>
> >>
> >
> >
>

Re: Hadoop 3.1.0 release discussion

[Arun Suresh <as...@apache.org>]

Hey Wangda

Update: YARN-6592 has been merged to trunk.

Cheers
-Arun

On Wed, Jan 31, 2018 at 12:58 AM, Wangda Tan <wh...@gmail.com> wrote:

> Hi Daniel,
>
> I agree, let's move the discussion to YARN-7292 and get it resolved before
> 3.1.0
>
> Thanks,
> Wangda
>
> On Wed, Jan 31, 2018 at 4:35 PM, Daniel Templeton <da...@cloudera.com>
> wrote:
>
> > I added my comments on that JIRA.  Looks like YARN-7292 is marked as a
> > blocker for 3.1, and I would tend to agree with that.  Let's see what we
> > can to do get profiles nailed down so that 3.1 can go forward.
> >
> > Daniel
> >
> >
> > On 1/18/18 10:25 AM, Wangda Tan wrote:
> >
> > Thanks Daniel,
> >
> > We need to make a decision about this: https://issues.apache.
> > org/jira/browse/YARN-7292, I believe this is the JIRA you mentioned
> > correct? Please let me know if there's anything else. And let's move the
> > discussion on JIRA.
> >
> > The good news is that resource profile is merged to trunk already so we
> > can finish that before code freeze date (Feb 08).
> >
> > + Sunil as well.
> >
> > Thanks,
> > Wangda
> >
> >
> >
> > On Wed, Jan 17, 2018 at 4:31 PM, Daniel Templeton <da...@cloudera.com>
> > wrote:
> >
> >> What's the status on resource profiles?  I believe there are still a
> >> couple of open JIRAs to rethink some of the design choices.
> >>
> >> Daniel
> >>
> >>
> >> On 1/17/18 11:33 AM, Wangda Tan wrote:
> >>
> >>> Hi All,
> >>>
> >>> Since we're fast approaching previously proposed feature freeze date
> (Jan
> >>> 30, about 13 days from today). If you've any features which live in a
> >>> branch and targeted to 3.1.0, please reply this email thread. Ideally,
> we
> >>> should finish branch merging before feature freeze date.
> >>>
> >>> Here's an updated 3.1.0 feature status:
> >>>
> >>> 1. Merged & Completed features:
> >>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
> >>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
> >>> end-to-end.
> >>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
> >>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
> >>> commits.
> >>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
> >>> Queues While Doing Queue Mapping.
> >>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
> >>>
> >>> 2. Features close to finish:
> >>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
> >>> and
> >>> merged to trunk. Except for UI/documentation.
> >>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
> >>> discussions going on about implementation.
> >>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
> >>> anti-affinity support. Close to finish, on track to be merged before
> Jan
> >>> 30.
> >>>
> >>> 3. Tentative features:
> >>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
> >>> containers. Only one pending patch. Plan to finish before Jan 7th.
> >>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging
> to
> >>> be
> >>> done before Jan 2018.
> >>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
> >>> challenging to be done before Jan 2018.
> >>> * (Varun V) YARN-5673: container-executor write. Given security
> >>> refactoring
> >>> of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to
> >>> 3.2.
> >>>
> >>> Thanks,
> >>> Wangda
> >>>
> >>>
> >>>
> >>>
> >>> On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com>
> wrote:
> >>>
> >>> Hi all,
> >>>>
> >>>> Congratulations on the 3.0.0-GA release!
> >>>>
> >>>> As we discussed in the previous email thread [1], I'd like to restart
> >>>> 3.1.0 release plans.
> >>>>
> >>>> a) Quick summary:
> >>>> a.1 Release status
> >>>> We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
> >>>> there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
> blockers
> >>>> and
> >>>> 22 open critical issues.
> >>>>
> >>>> a.2 Release date update
> >>>> Considering delays of 3.0-GA release by month-and-a-half, I propose to
> >>>> move the dates as follows
> >>>>   - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last date
> >>>> for
> >>>> any branches to get merged too;
> >>>>   - code freeze (blockers & critical only) date to Feb 08, 2018;
> >>>>   - release voting start by Feb 18, 2018, leaving time for at least
> two
> >>>> RCx
> >>>>   - release date from Jan 15, 2018, to Feb 28, 2018;
> >>>>
> >>>> Unlike before, I added an additional milestone for release-vote-start
> so
> >>>> that we can account for voting time-period also.
> >>>>
> >>>> This overall is still 5 1/2 months of release-timeline unlike the
> faster
> >>>> cadence we hoped for, but this, in my opinion, is the best-updated
> >>>> timeline
> >>>> given the delays of the final release of 3.0-GA.
> >>>>
> >>>> b) Individual feature status:
> >>>> I spoke to several feature owners and checked the status of
> un-finished
> >>>> features, following are status of features planned to 3.1.0:
> >>>>
> >>>> b.1 Merged & Completed features:
> >>>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
> >>>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
> >>>> end-to-end.
> >>>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
> >>>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
> >>>> commits.
> >>>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
> >>>> Queues While Doing Queue Mapping.
> >>>>
> >>>> b.2 Features close to finish:
> >>>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
> >>>> * (Zhankun) YARN-5983: FPGA support. Majority implementations
> completed
> >>>> and merged to trunk. Except for UI/documentation.
> >>>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
> >>>> discussions going on about implementation.
> >>>>
> >>>> b.3 Tentative features:
> >>>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
> >>>> containers. Only one pending patch. Plan to finish before Jan 7th.
> >>>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging
> to
> >>>> be done before Jan 2018.
> >>>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest
> and
> >>>> anti-affinity support. Tentative will figure out by Jan 1st.
> >>>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
> >>>> challenging to be done before Jan 2018.
> >>>> * (Varun V) YARN-5673: container-executor write. Given security
> >>>> refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may
> >>>> be
> >>>> moved to 3.2.
> >>>>
> >>>> b.4 Additional release drivers
> >>>> * More exhaustive upgrade testing from 2.x to 3.x.
> >>>>
> >>>> c) Regarding branch cut:
> >>>>
> >>>> We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
> >>>> feature planned to 3.2 has to be landed on trunk or B. After feature
> >>>> freeze
> >>>> date, whichever comes first.
> >>>>
> >>>> I've also talked offline with Vinod to get help on release-management
> >>>> given this is my first release. He agreed to help do this release
> >>>> jointly.
> >>>>
> >>>> Thoughts?
> >>>>
> >>>> Thanks,
> >>>> Wangda Tan
> >>>>
> >>>> [1] https://lists.apache.org/thread.html/
> c11506c3250c9481852130616b3cb0
> >>>> 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
> >>>> [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> >>>> (3.1.0)
> >>>> AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
> >>>>
> >>>>
> >>>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> >> For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >>
> >>
> >
> >
>

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Hi Daniel,

I agree, let's move the discussion to YARN-7292 and get it resolved before
3.1.0

Thanks,
Wangda

On Wed, Jan 31, 2018 at 4:35 PM, Daniel Templeton <da...@cloudera.com>
wrote:

> I added my comments on that JIRA.  Looks like YARN-7292 is marked as a
> blocker for 3.1, and I would tend to agree with that.  Let's see what we
> can to do get profiles nailed down so that 3.1 can go forward.
>
> Daniel
>
>
> On 1/18/18 10:25 AM, Wangda Tan wrote:
>
> Thanks Daniel,
>
> We need to make a decision about this: https://issues.apache.
> org/jira/browse/YARN-7292, I believe this is the JIRA you mentioned
> correct? Please let me know if there's anything else. And let's move the
> discussion on JIRA.
>
> The good news is that resource profile is merged to trunk already so we
> can finish that before code freeze date (Feb 08).
>
> + Sunil as well.
>
> Thanks,
> Wangda
>
>
>
> On Wed, Jan 17, 2018 at 4:31 PM, Daniel Templeton <da...@cloudera.com>
> wrote:
>
>> What's the status on resource profiles?  I believe there are still a
>> couple of open JIRAs to rethink some of the design choices.
>>
>> Daniel
>>
>>
>> On 1/17/18 11:33 AM, Wangda Tan wrote:
>>
>>> Hi All,
>>>
>>> Since we're fast approaching previously proposed feature freeze date (Jan
>>> 30, about 13 days from today). If you've any features which live in a
>>> branch and targeted to 3.1.0, please reply this email thread. Ideally, we
>>> should finish branch merging before feature freeze date.
>>>
>>> Here's an updated 3.1.0 feature status:
>>>
>>> 1. Merged & Completed features:
>>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>>> end-to-end.
>>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>>> commits.
>>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>>> Queues While Doing Queue Mapping.
>>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>>>
>>> 2. Features close to finish:
>>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
>>> and
>>> merged to trunk. Except for UI/documentation.
>>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>>> discussions going on about implementation.
>>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>>> anti-affinity support. Close to finish, on track to be merged before Jan
>>> 30.
>>>
>>> 3. Tentative features:
>>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>>> containers. Only one pending patch. Plan to finish before Jan 7th.
>>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
>>> be
>>> done before Jan 2018.
>>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>>> challenging to be done before Jan 2018.
>>> * (Varun V) YARN-5673: container-executor write. Given security
>>> refactoring
>>> of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to
>>> 3.2.
>>>
>>> Thanks,
>>> Wangda
>>>
>>>
>>>
>>>
>>> On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com> wrote:
>>>
>>> Hi all,
>>>>
>>>> Congratulations on the 3.0.0-GA release!
>>>>
>>>> As we discussed in the previous email thread [1], I'd like to restart
>>>> 3.1.0 release plans.
>>>>
>>>> a) Quick summary:
>>>> a.1 Release status
>>>> We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
>>>> there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open blockers
>>>> and
>>>> 22 open critical issues.
>>>>
>>>> a.2 Release date update
>>>> Considering delays of 3.0-GA release by month-and-a-half, I propose to
>>>> move the dates as follows
>>>>   - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last date
>>>> for
>>>> any branches to get merged too;
>>>>   - code freeze (blockers & critical only) date to Feb 08, 2018;
>>>>   - release voting start by Feb 18, 2018, leaving time for at least two
>>>> RCx
>>>>   - release date from Jan 15, 2018, to Feb 28, 2018;
>>>>
>>>> Unlike before, I added an additional milestone for release-vote-start so
>>>> that we can account for voting time-period also.
>>>>
>>>> This overall is still 5 1/2 months of release-timeline unlike the faster
>>>> cadence we hoped for, but this, in my opinion, is the best-updated
>>>> timeline
>>>> given the delays of the final release of 3.0-GA.
>>>>
>>>> b) Individual feature status:
>>>> I spoke to several feature owners and checked the status of un-finished
>>>> features, following are status of features planned to 3.1.0:
>>>>
>>>> b.1 Merged & Completed features:
>>>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>>>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>>>> end-to-end.
>>>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>>>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>>>> commits.
>>>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>>>> Queues While Doing Queue Mapping.
>>>>
>>>> b.2 Features close to finish:
>>>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
>>>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
>>>> and merged to trunk. Except for UI/documentation.
>>>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>>>> discussions going on about implementation.
>>>>
>>>> b.3 Tentative features:
>>>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>>>> containers. Only one pending patch. Plan to finish before Jan 7th.
>>>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
>>>> be done before Jan 2018.
>>>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>>>> anti-affinity support. Tentative will figure out by Jan 1st.
>>>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>>>> challenging to be done before Jan 2018.
>>>> * (Varun V) YARN-5673: container-executor write. Given security
>>>> refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may
>>>> be
>>>> moved to 3.2.
>>>>
>>>> b.4 Additional release drivers
>>>> * More exhaustive upgrade testing from 2.x to 3.x.
>>>>
>>>> c) Regarding branch cut:
>>>>
>>>> We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
>>>> feature planned to 3.2 has to be landed on trunk or B. After feature
>>>> freeze
>>>> date, whichever comes first.
>>>>
>>>> I've also talked offline with Vinod to get help on release-management
>>>> given this is my first release. He agreed to help do this release
>>>> jointly.
>>>>
>>>> Thoughts?
>>>>
>>>> Thanks,
>>>> Wangda Tan
>>>>
>>>> [1] https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0
>>>> 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>>>> [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
>>>> (3.1.0)
>>>> AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>>>>
>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>
>>
>
>

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Hi Daniel,

I agree, let's move the discussion to YARN-7292 and get it resolved before
3.1.0

Thanks,
Wangda

On Wed, Jan 31, 2018 at 4:35 PM, Daniel Templeton <da...@cloudera.com>
wrote:

> I added my comments on that JIRA.  Looks like YARN-7292 is marked as a
> blocker for 3.1, and I would tend to agree with that.  Let's see what we
> can to do get profiles nailed down so that 3.1 can go forward.
>
> Daniel
>
>
> On 1/18/18 10:25 AM, Wangda Tan wrote:
>
> Thanks Daniel,
>
> We need to make a decision about this: https://issues.apache.
> org/jira/browse/YARN-7292, I believe this is the JIRA you mentioned
> correct? Please let me know if there's anything else. And let's move the
> discussion on JIRA.
>
> The good news is that resource profile is merged to trunk already so we
> can finish that before code freeze date (Feb 08).
>
> + Sunil as well.
>
> Thanks,
> Wangda
>
>
>
> On Wed, Jan 17, 2018 at 4:31 PM, Daniel Templeton <da...@cloudera.com>
> wrote:
>
>> What's the status on resource profiles?  I believe there are still a
>> couple of open JIRAs to rethink some of the design choices.
>>
>> Daniel
>>
>>
>> On 1/17/18 11:33 AM, Wangda Tan wrote:
>>
>>> Hi All,
>>>
>>> Since we're fast approaching previously proposed feature freeze date (Jan
>>> 30, about 13 days from today). If you've any features which live in a
>>> branch and targeted to 3.1.0, please reply this email thread. Ideally, we
>>> should finish branch merging before feature freeze date.
>>>
>>> Here's an updated 3.1.0 feature status:
>>>
>>> 1. Merged & Completed features:
>>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>>> end-to-end.
>>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>>> commits.
>>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>>> Queues While Doing Queue Mapping.
>>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>>>
>>> 2. Features close to finish:
>>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
>>> and
>>> merged to trunk. Except for UI/documentation.
>>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>>> discussions going on about implementation.
>>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>>> anti-affinity support. Close to finish, on track to be merged before Jan
>>> 30.
>>>
>>> 3. Tentative features:
>>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>>> containers. Only one pending patch. Plan to finish before Jan 7th.
>>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
>>> be
>>> done before Jan 2018.
>>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>>> challenging to be done before Jan 2018.
>>> * (Varun V) YARN-5673: container-executor write. Given security
>>> refactoring
>>> of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to
>>> 3.2.
>>>
>>> Thanks,
>>> Wangda
>>>
>>>
>>>
>>>
>>> On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com> wrote:
>>>
>>> Hi all,
>>>>
>>>> Congratulations on the 3.0.0-GA release!
>>>>
>>>> As we discussed in the previous email thread [1], I'd like to restart
>>>> 3.1.0 release plans.
>>>>
>>>> a) Quick summary:
>>>> a.1 Release status
>>>> We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
>>>> there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open blockers
>>>> and
>>>> 22 open critical issues.
>>>>
>>>> a.2 Release date update
>>>> Considering delays of 3.0-GA release by month-and-a-half, I propose to
>>>> move the dates as follows
>>>>   - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last date
>>>> for
>>>> any branches to get merged too;
>>>>   - code freeze (blockers & critical only) date to Feb 08, 2018;
>>>>   - release voting start by Feb 18, 2018, leaving time for at least two
>>>> RCx
>>>>   - release date from Jan 15, 2018, to Feb 28, 2018;
>>>>
>>>> Unlike before, I added an additional milestone for release-vote-start so
>>>> that we can account for voting time-period also.
>>>>
>>>> This overall is still 5 1/2 months of release-timeline unlike the faster
>>>> cadence we hoped for, but this, in my opinion, is the best-updated
>>>> timeline
>>>> given the delays of the final release of 3.0-GA.
>>>>
>>>> b) Individual feature status:
>>>> I spoke to several feature owners and checked the status of un-finished
>>>> features, following are status of features planned to 3.1.0:
>>>>
>>>> b.1 Merged & Completed features:
>>>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>>>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>>>> end-to-end.
>>>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>>>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>>>> commits.
>>>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>>>> Queues While Doing Queue Mapping.
>>>>
>>>> b.2 Features close to finish:
>>>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
>>>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
>>>> and merged to trunk. Except for UI/documentation.
>>>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>>>> discussions going on about implementation.
>>>>
>>>> b.3 Tentative features:
>>>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>>>> containers. Only one pending patch. Plan to finish before Jan 7th.
>>>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
>>>> be done before Jan 2018.
>>>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>>>> anti-affinity support. Tentative will figure out by Jan 1st.
>>>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>>>> challenging to be done before Jan 2018.
>>>> * (Varun V) YARN-5673: container-executor write. Given security
>>>> refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may
>>>> be
>>>> moved to 3.2.
>>>>
>>>> b.4 Additional release drivers
>>>> * More exhaustive upgrade testing from 2.x to 3.x.
>>>>
>>>> c) Regarding branch cut:
>>>>
>>>> We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
>>>> feature planned to 3.2 has to be landed on trunk or B. After feature
>>>> freeze
>>>> date, whichever comes first.
>>>>
>>>> I've also talked offline with Vinod to get help on release-management
>>>> given this is my first release. He agreed to help do this release
>>>> jointly.
>>>>
>>>> Thoughts?
>>>>
>>>> Thanks,
>>>> Wangda Tan
>>>>
>>>> [1] https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0
>>>> 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>>>> [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
>>>> (3.1.0)
>>>> AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>>>>
>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>
>>
>
>

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Hi Daniel,

I agree, let's move the discussion to YARN-7292 and get it resolved before
3.1.0

Thanks,
Wangda

On Wed, Jan 31, 2018 at 4:35 PM, Daniel Templeton <da...@cloudera.com>
wrote:

> I added my comments on that JIRA.  Looks like YARN-7292 is marked as a
> blocker for 3.1, and I would tend to agree with that.  Let's see what we
> can to do get profiles nailed down so that 3.1 can go forward.
>
> Daniel
>
>
> On 1/18/18 10:25 AM, Wangda Tan wrote:
>
> Thanks Daniel,
>
> We need to make a decision about this: https://issues.apache.
> org/jira/browse/YARN-7292, I believe this is the JIRA you mentioned
> correct? Please let me know if there's anything else. And let's move the
> discussion on JIRA.
>
> The good news is that resource profile is merged to trunk already so we
> can finish that before code freeze date (Feb 08).
>
> + Sunil as well.
>
> Thanks,
> Wangda
>
>
>
> On Wed, Jan 17, 2018 at 4:31 PM, Daniel Templeton <da...@cloudera.com>
> wrote:
>
>> What's the status on resource profiles?  I believe there are still a
>> couple of open JIRAs to rethink some of the design choices.
>>
>> Daniel
>>
>>
>> On 1/17/18 11:33 AM, Wangda Tan wrote:
>>
>>> Hi All,
>>>
>>> Since we're fast approaching previously proposed feature freeze date (Jan
>>> 30, about 13 days from today). If you've any features which live in a
>>> branch and targeted to 3.1.0, please reply this email thread. Ideally, we
>>> should finish branch merging before feature freeze date.
>>>
>>> Here's an updated 3.1.0 feature status:
>>>
>>> 1. Merged & Completed features:
>>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>>> end-to-end.
>>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>>> commits.
>>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>>> Queues While Doing Queue Mapping.
>>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>>>
>>> 2. Features close to finish:
>>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
>>> and
>>> merged to trunk. Except for UI/documentation.
>>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>>> discussions going on about implementation.
>>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>>> anti-affinity support. Close to finish, on track to be merged before Jan
>>> 30.
>>>
>>> 3. Tentative features:
>>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>>> containers. Only one pending patch. Plan to finish before Jan 7th.
>>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
>>> be
>>> done before Jan 2018.
>>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>>> challenging to be done before Jan 2018.
>>> * (Varun V) YARN-5673: container-executor write. Given security
>>> refactoring
>>> of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to
>>> 3.2.
>>>
>>> Thanks,
>>> Wangda
>>>
>>>
>>>
>>>
>>> On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com> wrote:
>>>
>>> Hi all,
>>>>
>>>> Congratulations on the 3.0.0-GA release!
>>>>
>>>> As we discussed in the previous email thread [1], I'd like to restart
>>>> 3.1.0 release plans.
>>>>
>>>> a) Quick summary:
>>>> a.1 Release status
>>>> We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
>>>> there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open blockers
>>>> and
>>>> 22 open critical issues.
>>>>
>>>> a.2 Release date update
>>>> Considering delays of 3.0-GA release by month-and-a-half, I propose to
>>>> move the dates as follows
>>>>   - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last date
>>>> for
>>>> any branches to get merged too;
>>>>   - code freeze (blockers & critical only) date to Feb 08, 2018;
>>>>   - release voting start by Feb 18, 2018, leaving time for at least two
>>>> RCx
>>>>   - release date from Jan 15, 2018, to Feb 28, 2018;
>>>>
>>>> Unlike before, I added an additional milestone for release-vote-start so
>>>> that we can account for voting time-period also.
>>>>
>>>> This overall is still 5 1/2 months of release-timeline unlike the faster
>>>> cadence we hoped for, but this, in my opinion, is the best-updated
>>>> timeline
>>>> given the delays of the final release of 3.0-GA.
>>>>
>>>> b) Individual feature status:
>>>> I spoke to several feature owners and checked the status of un-finished
>>>> features, following are status of features planned to 3.1.0:
>>>>
>>>> b.1 Merged & Completed features:
>>>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>>>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>>>> end-to-end.
>>>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>>>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>>>> commits.
>>>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>>>> Queues While Doing Queue Mapping.
>>>>
>>>> b.2 Features close to finish:
>>>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
>>>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
>>>> and merged to trunk. Except for UI/documentation.
>>>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>>>> discussions going on about implementation.
>>>>
>>>> b.3 Tentative features:
>>>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>>>> containers. Only one pending patch. Plan to finish before Jan 7th.
>>>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
>>>> be done before Jan 2018.
>>>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>>>> anti-affinity support. Tentative will figure out by Jan 1st.
>>>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>>>> challenging to be done before Jan 2018.
>>>> * (Varun V) YARN-5673: container-executor write. Given security
>>>> refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may
>>>> be
>>>> moved to 3.2.
>>>>
>>>> b.4 Additional release drivers
>>>> * More exhaustive upgrade testing from 2.x to 3.x.
>>>>
>>>> c) Regarding branch cut:
>>>>
>>>> We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
>>>> feature planned to 3.2 has to be landed on trunk or B. After feature
>>>> freeze
>>>> date, whichever comes first.
>>>>
>>>> I've also talked offline with Vinod to get help on release-management
>>>> given this is my first release. He agreed to help do this release
>>>> jointly.
>>>>
>>>> Thoughts?
>>>>
>>>> Thanks,
>>>> Wangda Tan
>>>>
>>>> [1] https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0
>>>> 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>>>> [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
>>>> (3.1.0)
>>>> AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>>>>
>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>
>>
>
>

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Hi Daniel,

I agree, let's move the discussion to YARN-7292 and get it resolved before
3.1.0

Thanks,
Wangda

On Wed, Jan 31, 2018 at 4:35 PM, Daniel Templeton <da...@cloudera.com>
wrote:

> I added my comments on that JIRA.  Looks like YARN-7292 is marked as a
> blocker for 3.1, and I would tend to agree with that.  Let's see what we
> can to do get profiles nailed down so that 3.1 can go forward.
>
> Daniel
>
>
> On 1/18/18 10:25 AM, Wangda Tan wrote:
>
> Thanks Daniel,
>
> We need to make a decision about this: https://issues.apache.
> org/jira/browse/YARN-7292, I believe this is the JIRA you mentioned
> correct? Please let me know if there's anything else. And let's move the
> discussion on JIRA.
>
> The good news is that resource profile is merged to trunk already so we
> can finish that before code freeze date (Feb 08).
>
> + Sunil as well.
>
> Thanks,
> Wangda
>
>
>
> On Wed, Jan 17, 2018 at 4:31 PM, Daniel Templeton <da...@cloudera.com>
> wrote:
>
>> What's the status on resource profiles?  I believe there are still a
>> couple of open JIRAs to rethink some of the design choices.
>>
>> Daniel
>>
>>
>> On 1/17/18 11:33 AM, Wangda Tan wrote:
>>
>>> Hi All,
>>>
>>> Since we're fast approaching previously proposed feature freeze date (Jan
>>> 30, about 13 days from today). If you've any features which live in a
>>> branch and targeted to 3.1.0, please reply this email thread. Ideally, we
>>> should finish branch merging before feature freeze date.
>>>
>>> Here's an updated 3.1.0 feature status:
>>>
>>> 1. Merged & Completed features:
>>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>>> end-to-end.
>>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>>> commits.
>>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>>> Queues While Doing Queue Mapping.
>>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>>>
>>> 2. Features close to finish:
>>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
>>> and
>>> merged to trunk. Except for UI/documentation.
>>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>>> discussions going on about implementation.
>>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>>> anti-affinity support. Close to finish, on track to be merged before Jan
>>> 30.
>>>
>>> 3. Tentative features:
>>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>>> containers. Only one pending patch. Plan to finish before Jan 7th.
>>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
>>> be
>>> done before Jan 2018.
>>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>>> challenging to be done before Jan 2018.
>>> * (Varun V) YARN-5673: container-executor write. Given security
>>> refactoring
>>> of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to
>>> 3.2.
>>>
>>> Thanks,
>>> Wangda
>>>
>>>
>>>
>>>
>>> On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com> wrote:
>>>
>>> Hi all,
>>>>
>>>> Congratulations on the 3.0.0-GA release!
>>>>
>>>> As we discussed in the previous email thread [1], I'd like to restart
>>>> 3.1.0 release plans.
>>>>
>>>> a) Quick summary:
>>>> a.1 Release status
>>>> We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
>>>> there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open blockers
>>>> and
>>>> 22 open critical issues.
>>>>
>>>> a.2 Release date update
>>>> Considering delays of 3.0-GA release by month-and-a-half, I propose to
>>>> move the dates as follows
>>>>   - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last date
>>>> for
>>>> any branches to get merged too;
>>>>   - code freeze (blockers & critical only) date to Feb 08, 2018;
>>>>   - release voting start by Feb 18, 2018, leaving time for at least two
>>>> RCx
>>>>   - release date from Jan 15, 2018, to Feb 28, 2018;
>>>>
>>>> Unlike before, I added an additional milestone for release-vote-start so
>>>> that we can account for voting time-period also.
>>>>
>>>> This overall is still 5 1/2 months of release-timeline unlike the faster
>>>> cadence we hoped for, but this, in my opinion, is the best-updated
>>>> timeline
>>>> given the delays of the final release of 3.0-GA.
>>>>
>>>> b) Individual feature status:
>>>> I spoke to several feature owners and checked the status of un-finished
>>>> features, following are status of features planned to 3.1.0:
>>>>
>>>> b.1 Merged & Completed features:
>>>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>>>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>>>> end-to-end.
>>>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>>>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>>>> commits.
>>>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>>>> Queues While Doing Queue Mapping.
>>>>
>>>> b.2 Features close to finish:
>>>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
>>>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
>>>> and merged to trunk. Except for UI/documentation.
>>>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>>>> discussions going on about implementation.
>>>>
>>>> b.3 Tentative features:
>>>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>>>> containers. Only one pending patch. Plan to finish before Jan 7th.
>>>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
>>>> be done before Jan 2018.
>>>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>>>> anti-affinity support. Tentative will figure out by Jan 1st.
>>>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>>>> challenging to be done before Jan 2018.
>>>> * (Varun V) YARN-5673: container-executor write. Given security
>>>> refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may
>>>> be
>>>> moved to 3.2.
>>>>
>>>> b.4 Additional release drivers
>>>> * More exhaustive upgrade testing from 2.x to 3.x.
>>>>
>>>> c) Regarding branch cut:
>>>>
>>>> We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
>>>> feature planned to 3.2 has to be landed on trunk or B. After feature
>>>> freeze
>>>> date, whichever comes first.
>>>>
>>>> I've also talked offline with Vinod to get help on release-management
>>>> given this is my first release. He agreed to help do this release
>>>> jointly.
>>>>
>>>> Thoughts?
>>>>
>>>> Thanks,
>>>> Wangda Tan
>>>>
>>>> [1] https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0
>>>> 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>>>> [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
>>>> (3.1.0)
>>>> AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>>>>
>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>
>>
>
>

Re: Hadoop 3.1.0 release discussion

[Daniel Templeton <da...@cloudera.com>]

I added my comments on that JIRA.  Looks like YARN-7292 is marked as a 
blocker for 3.1, and I would tend to agree with that.  Let's see what we 
can to do get profiles nailed down so that 3.1 can go forward.

Daniel

On 1/18/18 10:25 AM, Wangda Tan wrote:
> Thanks Daniel,
>
> We need to make a decision about this: 
> https://issues.apache.org/jira/browse/YARN-7292, I believe this is the 
> JIRA you mentioned correct? Please let me know if there's anything 
> else. And let's move the discussion on JIRA.
>
> The good news is that resource profile is merged to trunk already so 
> we can finish that before code freeze date (Feb 08).
>
> + Sunil as well.
>
> Thanks,
> Wangda
>
>
>
> On Wed, Jan 17, 2018 at 4:31 PM, Daniel Templeton <daniel@cloudera.com 
> <ma...@cloudera.com>> wrote:
>
>     What's the status on resource profiles?  I believe there are still
>     a couple of open JIRAs to rethink some of the design choices.
>
>     Daniel
>
>
>     On 1/17/18 11:33 AM, Wangda Tan wrote:
>
>         Hi All,
>
>         Since we're fast approaching previously proposed feature
>         freeze date (Jan
>         30, about 13 days from today). If you've any features which
>         live in a
>         branch and targeted to 3.1.0, please reply this email thread.
>         Ideally, we
>         should finish branch merging before feature freeze date.
>
>         Here's an updated 3.1.0 feature status:
>
>         1. Merged & Completed features:
>         * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>         * (Wangda) YARN-6223: GPU support on YARN. Features in trunk
>         and works
>         end-to-end.
>         * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
>         services.
>         * (Steve Loughran): HADOOP-13786: S3Guard committer for
>         zero-rename commits.
>         * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
>         of Leaf
>         Queues While Doing Queue Mapping.
>         * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>
>         2. Features close to finish:
>         * (Zhankun) YARN-5983: FPGA support. Majority implementations
>         completed and
>         merged to trunk. Except for UI/documentation.
>         * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
>         done, some
>         discussions going on about implementation.
>         * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>         SchedulingRequest and
>         anti-affinity support. Close to finish, on track to be merged
>         before Jan 30.
>
>         3. Tentative features:
>         * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>         containers. Only one pending patch. Plan to finish before Jan 7th.
>         * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>         challenging to be
>         done before Jan 2018.
>         * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
>         Looks
>         challenging to be done before Jan 2018.
>         * (Varun V) YARN-5673: container-executor write. Given
>         security refactoring
>         of c-e (YARN-6623) is already landed, IMHO other stuff may be
>         moved to 3.2.
>
>         Thanks,
>         Wangda
>
>
>
>
>         On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan
>         <wheeleast@gmail.com <ma...@gmail.com>> wrote:
>
>             Hi all,
>
>             Congratulations on the 3.0.0-GA release!
>
>             As we discussed in the previous email thread [1], I'd like
>             to restart
>             3.1.0 release plans.
>
>             a) Quick summary:
>             a.1 Release status
>             We started 3.1 release discussion on Sep 6, 2017 [1]. As
>             of today,
>             there’re 232 patches loaded on 3.1.0 alone [2], besides 6
>             open blockers and
>             22 open critical issues.
>
>             a.2 Release date update
>             Considering delays of 3.0-GA release by month-and-a-half,
>             I propose to
>             move the dates as follows
>               - feature freeze date from Dec 15, 2017, to Jan 30, 2018
>             - last date for
>             any branches to get merged too;
>               - code freeze (blockers & critical only) date to Feb 08,
>             2018;
>               - release voting start by Feb 18, 2018, leaving time for
>             at least two RCx
>               - release date from Jan 15, 2018, to Feb 28, 2018;
>
>             Unlike before, I added an additional milestone for
>             release-vote-start so
>             that we can account for voting time-period also.
>
>             This overall is still 5 1/2 months of release-timeline
>             unlike the faster
>             cadence we hoped for, but this, in my opinion, is the
>             best-updated timeline
>             given the delays of the final release of 3.0-GA.
>
>             b) Individual feature status:
>             I spoke to several feature owners and checked the status
>             of un-finished
>             features, following are status of features planned to 3.1.0:
>
>             b.1 Merged & Completed features:
>             * (Sunil) YARN-5881: Support absolute value in
>             CapacityScheduler.
>             * (Wangda) YARN-6223: GPU support on YARN. Features in
>             trunk and works
>             end-to-end.
>             * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN
>             native services.
>             * (Steve Loughran): HADOOP-13786: S3Guard committer for
>             zero-rename
>             commits.
>             * (Suma): YARN-7117: Capacity Scheduler: Support Auto
>             Creation of Leaf
>             Queues While Doing Queue Mapping.
>
>             b.2 Features close to finish:
>             * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being
>             voting now.
>             * (Zhankun) YARN-5983: FPGA support. Majority
>             implementations completed
>             and merged to trunk. Except for UI/documentation.
>             * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
>             done, some
>             discussions going on about implementation.
>
>             b.3 Tentative features:
>             * (Arun Suresh). YARN-5972: Support pausing/freezing
>             opportunistic
>             containers. Only one pending patch. Plan to finish before
>             Jan 7th.
>             * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>             challenging to
>             be done before Jan 2018.
>             * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>             SchedulingRequest and
>             anti-affinity support. Tentative will figure out by Jan 1st.
>             * (Anu): HDFS-7240: Ozone. Given the discussion on
>             HDFS-7240. Looks
>             challenging to be done before Jan 2018.
>             * (Varun V) YARN-5673: container-executor write. Given
>             security
>             refactoring of c-e (YARN-6623) is already landed, IMHO
>             other stuff may be
>             moved to 3.2.
>
>             b.4 Additional release drivers
>             * More exhaustive upgrade testing from 2.x to 3.x.
>
>             c) Regarding branch cut:
>
>             We will keep pointing trunk to 3.1 and cut branch-3.1
>             until: A. some
>             feature planned to 3.2 has to be landed on trunk or B.
>             After feature freeze
>             date, whichever comes first.
>
>             I've also talked offline with Vinod to get help on
>             release-management
>             given this is my first release. He agreed to help do this
>             release jointly.
>
>             Thoughts?
>
>             Thanks,
>             Wangda Tan
>
>             [1]
>             https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0
>             <https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0>
>             9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org
>             <http://3Cyarn-dev.hadoop.apache.org>%3E
>             [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>             fixVersion in (3.1.0)
>             AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>
>
>
>
>     ---------------------------------------------------------------------
>     To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>     <ma...@hadoop.apache.org>
>     For additional commands, e-mail: common-dev-help@hadoop.apache.org
>     <ma...@hadoop.apache.org>
>
>

Re: Hadoop 3.1.0 release discussion

[Daniel Templeton <da...@cloudera.com>]

I added my comments on that JIRA.  Looks like YARN-7292 is marked as a 
blocker for 3.1, and I would tend to agree with that.  Let's see what we 
can to do get profiles nailed down so that 3.1 can go forward.

Daniel

On 1/18/18 10:25 AM, Wangda Tan wrote:
> Thanks Daniel,
>
> We need to make a decision about this: 
> https://issues.apache.org/jira/browse/YARN-7292, I believe this is the 
> JIRA you mentioned correct? Please let me know if there's anything 
> else. And let's move the discussion on JIRA.
>
> The good news is that resource profile is merged to trunk already so 
> we can finish that before code freeze date (Feb 08).
>
> + Sunil as well.
>
> Thanks,
> Wangda
>
>
>
> On Wed, Jan 17, 2018 at 4:31 PM, Daniel Templeton <daniel@cloudera.com 
> <ma...@cloudera.com>> wrote:
>
>     What's the status on resource profiles?  I believe there are still
>     a couple of open JIRAs to rethink some of the design choices.
>
>     Daniel
>
>
>     On 1/17/18 11:33 AM, Wangda Tan wrote:
>
>         Hi All,
>
>         Since we're fast approaching previously proposed feature
>         freeze date (Jan
>         30, about 13 days from today). If you've any features which
>         live in a
>         branch and targeted to 3.1.0, please reply this email thread.
>         Ideally, we
>         should finish branch merging before feature freeze date.
>
>         Here's an updated 3.1.0 feature status:
>
>         1. Merged & Completed features:
>         * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>         * (Wangda) YARN-6223: GPU support on YARN. Features in trunk
>         and works
>         end-to-end.
>         * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
>         services.
>         * (Steve Loughran): HADOOP-13786: S3Guard committer for
>         zero-rename commits.
>         * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
>         of Leaf
>         Queues While Doing Queue Mapping.
>         * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>
>         2. Features close to finish:
>         * (Zhankun) YARN-5983: FPGA support. Majority implementations
>         completed and
>         merged to trunk. Except for UI/documentation.
>         * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
>         done, some
>         discussions going on about implementation.
>         * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>         SchedulingRequest and
>         anti-affinity support. Close to finish, on track to be merged
>         before Jan 30.
>
>         3. Tentative features:
>         * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>         containers. Only one pending patch. Plan to finish before Jan 7th.
>         * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>         challenging to be
>         done before Jan 2018.
>         * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
>         Looks
>         challenging to be done before Jan 2018.
>         * (Varun V) YARN-5673: container-executor write. Given
>         security refactoring
>         of c-e (YARN-6623) is already landed, IMHO other stuff may be
>         moved to 3.2.
>
>         Thanks,
>         Wangda
>
>
>
>
>         On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan
>         <wheeleast@gmail.com <ma...@gmail.com>> wrote:
>
>             Hi all,
>
>             Congratulations on the 3.0.0-GA release!
>
>             As we discussed in the previous email thread [1], I'd like
>             to restart
>             3.1.0 release plans.
>
>             a) Quick summary:
>             a.1 Release status
>             We started 3.1 release discussion on Sep 6, 2017 [1]. As
>             of today,
>             there’re 232 patches loaded on 3.1.0 alone [2], besides 6
>             open blockers and
>             22 open critical issues.
>
>             a.2 Release date update
>             Considering delays of 3.0-GA release by month-and-a-half,
>             I propose to
>             move the dates as follows
>               - feature freeze date from Dec 15, 2017, to Jan 30, 2018
>             - last date for
>             any branches to get merged too;
>               - code freeze (blockers & critical only) date to Feb 08,
>             2018;
>               - release voting start by Feb 18, 2018, leaving time for
>             at least two RCx
>               - release date from Jan 15, 2018, to Feb 28, 2018;
>
>             Unlike before, I added an additional milestone for
>             release-vote-start so
>             that we can account for voting time-period also.
>
>             This overall is still 5 1/2 months of release-timeline
>             unlike the faster
>             cadence we hoped for, but this, in my opinion, is the
>             best-updated timeline
>             given the delays of the final release of 3.0-GA.
>
>             b) Individual feature status:
>             I spoke to several feature owners and checked the status
>             of un-finished
>             features, following are status of features planned to 3.1.0:
>
>             b.1 Merged & Completed features:
>             * (Sunil) YARN-5881: Support absolute value in
>             CapacityScheduler.
>             * (Wangda) YARN-6223: GPU support on YARN. Features in
>             trunk and works
>             end-to-end.
>             * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN
>             native services.
>             * (Steve Loughran): HADOOP-13786: S3Guard committer for
>             zero-rename
>             commits.
>             * (Suma): YARN-7117: Capacity Scheduler: Support Auto
>             Creation of Leaf
>             Queues While Doing Queue Mapping.
>
>             b.2 Features close to finish:
>             * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being
>             voting now.
>             * (Zhankun) YARN-5983: FPGA support. Majority
>             implementations completed
>             and merged to trunk. Except for UI/documentation.
>             * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
>             done, some
>             discussions going on about implementation.
>
>             b.3 Tentative features:
>             * (Arun Suresh). YARN-5972: Support pausing/freezing
>             opportunistic
>             containers. Only one pending patch. Plan to finish before
>             Jan 7th.
>             * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>             challenging to
>             be done before Jan 2018.
>             * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>             SchedulingRequest and
>             anti-affinity support. Tentative will figure out by Jan 1st.
>             * (Anu): HDFS-7240: Ozone. Given the discussion on
>             HDFS-7240. Looks
>             challenging to be done before Jan 2018.
>             * (Varun V) YARN-5673: container-executor write. Given
>             security
>             refactoring of c-e (YARN-6623) is already landed, IMHO
>             other stuff may be
>             moved to 3.2.
>
>             b.4 Additional release drivers
>             * More exhaustive upgrade testing from 2.x to 3.x.
>
>             c) Regarding branch cut:
>
>             We will keep pointing trunk to 3.1 and cut branch-3.1
>             until: A. some
>             feature planned to 3.2 has to be landed on trunk or B.
>             After feature freeze
>             date, whichever comes first.
>
>             I've also talked offline with Vinod to get help on
>             release-management
>             given this is my first release. He agreed to help do this
>             release jointly.
>
>             Thoughts?
>
>             Thanks,
>             Wangda Tan
>
>             [1]
>             https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0
>             <https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0>
>             9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org
>             <http://3Cyarn-dev.hadoop.apache.org>%3E
>             [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>             fixVersion in (3.1.0)
>             AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>
>
>
>
>     ---------------------------------------------------------------------
>     To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>     <ma...@hadoop.apache.org>
>     For additional commands, e-mail: common-dev-help@hadoop.apache.org
>     <ma...@hadoop.apache.org>
>
>

Re: Hadoop 3.1.0 release discussion

[Daniel Templeton <da...@cloudera.com>]

I added my comments on that JIRA.  Looks like YARN-7292 is marked as a 
blocker for 3.1, and I would tend to agree with that.  Let's see what we 
can to do get profiles nailed down so that 3.1 can go forward.

Daniel

On 1/18/18 10:25 AM, Wangda Tan wrote:
> Thanks Daniel,
>
> We need to make a decision about this: 
> https://issues.apache.org/jira/browse/YARN-7292, I believe this is the 
> JIRA you mentioned correct? Please let me know if there's anything 
> else. And let's move the discussion on JIRA.
>
> The good news is that resource profile is merged to trunk already so 
> we can finish that before code freeze date (Feb 08).
>
> + Sunil as well.
>
> Thanks,
> Wangda
>
>
>
> On Wed, Jan 17, 2018 at 4:31 PM, Daniel Templeton <daniel@cloudera.com 
> <ma...@cloudera.com>> wrote:
>
>     What's the status on resource profiles?  I believe there are still
>     a couple of open JIRAs to rethink some of the design choices.
>
>     Daniel
>
>
>     On 1/17/18 11:33 AM, Wangda Tan wrote:
>
>         Hi All,
>
>         Since we're fast approaching previously proposed feature
>         freeze date (Jan
>         30, about 13 days from today). If you've any features which
>         live in a
>         branch and targeted to 3.1.0, please reply this email thread.
>         Ideally, we
>         should finish branch merging before feature freeze date.
>
>         Here's an updated 3.1.0 feature status:
>
>         1. Merged & Completed features:
>         * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>         * (Wangda) YARN-6223: GPU support on YARN. Features in trunk
>         and works
>         end-to-end.
>         * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
>         services.
>         * (Steve Loughran): HADOOP-13786: S3Guard committer for
>         zero-rename commits.
>         * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
>         of Leaf
>         Queues While Doing Queue Mapping.
>         * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>
>         2. Features close to finish:
>         * (Zhankun) YARN-5983: FPGA support. Majority implementations
>         completed and
>         merged to trunk. Except for UI/documentation.
>         * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
>         done, some
>         discussions going on about implementation.
>         * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>         SchedulingRequest and
>         anti-affinity support. Close to finish, on track to be merged
>         before Jan 30.
>
>         3. Tentative features:
>         * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>         containers. Only one pending patch. Plan to finish before Jan 7th.
>         * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>         challenging to be
>         done before Jan 2018.
>         * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
>         Looks
>         challenging to be done before Jan 2018.
>         * (Varun V) YARN-5673: container-executor write. Given
>         security refactoring
>         of c-e (YARN-6623) is already landed, IMHO other stuff may be
>         moved to 3.2.
>
>         Thanks,
>         Wangda
>
>
>
>
>         On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan
>         <wheeleast@gmail.com <ma...@gmail.com>> wrote:
>
>             Hi all,
>
>             Congratulations on the 3.0.0-GA release!
>
>             As we discussed in the previous email thread [1], I'd like
>             to restart
>             3.1.0 release plans.
>
>             a) Quick summary:
>             a.1 Release status
>             We started 3.1 release discussion on Sep 6, 2017 [1]. As
>             of today,
>             there’re 232 patches loaded on 3.1.0 alone [2], besides 6
>             open blockers and
>             22 open critical issues.
>
>             a.2 Release date update
>             Considering delays of 3.0-GA release by month-and-a-half,
>             I propose to
>             move the dates as follows
>               - feature freeze date from Dec 15, 2017, to Jan 30, 2018
>             - last date for
>             any branches to get merged too;
>               - code freeze (blockers & critical only) date to Feb 08,
>             2018;
>               - release voting start by Feb 18, 2018, leaving time for
>             at least two RCx
>               - release date from Jan 15, 2018, to Feb 28, 2018;
>
>             Unlike before, I added an additional milestone for
>             release-vote-start so
>             that we can account for voting time-period also.
>
>             This overall is still 5 1/2 months of release-timeline
>             unlike the faster
>             cadence we hoped for, but this, in my opinion, is the
>             best-updated timeline
>             given the delays of the final release of 3.0-GA.
>
>             b) Individual feature status:
>             I spoke to several feature owners and checked the status
>             of un-finished
>             features, following are status of features planned to 3.1.0:
>
>             b.1 Merged & Completed features:
>             * (Sunil) YARN-5881: Support absolute value in
>             CapacityScheduler.
>             * (Wangda) YARN-6223: GPU support on YARN. Features in
>             trunk and works
>             end-to-end.
>             * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN
>             native services.
>             * (Steve Loughran): HADOOP-13786: S3Guard committer for
>             zero-rename
>             commits.
>             * (Suma): YARN-7117: Capacity Scheduler: Support Auto
>             Creation of Leaf
>             Queues While Doing Queue Mapping.
>
>             b.2 Features close to finish:
>             * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being
>             voting now.
>             * (Zhankun) YARN-5983: FPGA support. Majority
>             implementations completed
>             and merged to trunk. Except for UI/documentation.
>             * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
>             done, some
>             discussions going on about implementation.
>
>             b.3 Tentative features:
>             * (Arun Suresh). YARN-5972: Support pausing/freezing
>             opportunistic
>             containers. Only one pending patch. Plan to finish before
>             Jan 7th.
>             * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>             challenging to
>             be done before Jan 2018.
>             * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>             SchedulingRequest and
>             anti-affinity support. Tentative will figure out by Jan 1st.
>             * (Anu): HDFS-7240: Ozone. Given the discussion on
>             HDFS-7240. Looks
>             challenging to be done before Jan 2018.
>             * (Varun V) YARN-5673: container-executor write. Given
>             security
>             refactoring of c-e (YARN-6623) is already landed, IMHO
>             other stuff may be
>             moved to 3.2.
>
>             b.4 Additional release drivers
>             * More exhaustive upgrade testing from 2.x to 3.x.
>
>             c) Regarding branch cut:
>
>             We will keep pointing trunk to 3.1 and cut branch-3.1
>             until: A. some
>             feature planned to 3.2 has to be landed on trunk or B.
>             After feature freeze
>             date, whichever comes first.
>
>             I've also talked offline with Vinod to get help on
>             release-management
>             given this is my first release. He agreed to help do this
>             release jointly.
>
>             Thoughts?
>
>             Thanks,
>             Wangda Tan
>
>             [1]
>             https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0
>             <https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0>
>             9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org
>             <http://3Cyarn-dev.hadoop.apache.org>%3E
>             [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>             fixVersion in (3.1.0)
>             AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>
>
>
>
>     ---------------------------------------------------------------------
>     To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>     <ma...@hadoop.apache.org>
>     For additional commands, e-mail: common-dev-help@hadoop.apache.org
>     <ma...@hadoop.apache.org>
>
>

Re: Hadoop 3.1.0 release discussion

[Daniel Templeton <da...@cloudera.com>]

I added my comments on that JIRA.  Looks like YARN-7292 is marked as a 
blocker for 3.1, and I would tend to agree with that.  Let's see what we 
can to do get profiles nailed down so that 3.1 can go forward.

Daniel

On 1/18/18 10:25 AM, Wangda Tan wrote:
> Thanks Daniel,
>
> We need to make a decision about this: 
> https://issues.apache.org/jira/browse/YARN-7292, I believe this is the 
> JIRA you mentioned correct? Please let me know if there's anything 
> else. And let's move the discussion on JIRA.
>
> The good news is that resource profile is merged to trunk already so 
> we can finish that before code freeze date (Feb 08).
>
> + Sunil as well.
>
> Thanks,
> Wangda
>
>
>
> On Wed, Jan 17, 2018 at 4:31 PM, Daniel Templeton <daniel@cloudera.com 
> <ma...@cloudera.com>> wrote:
>
>     What's the status on resource profiles?  I believe there are still
>     a couple of open JIRAs to rethink some of the design choices.
>
>     Daniel
>
>
>     On 1/17/18 11:33 AM, Wangda Tan wrote:
>
>         Hi All,
>
>         Since we're fast approaching previously proposed feature
>         freeze date (Jan
>         30, about 13 days from today). If you've any features which
>         live in a
>         branch and targeted to 3.1.0, please reply this email thread.
>         Ideally, we
>         should finish branch merging before feature freeze date.
>
>         Here's an updated 3.1.0 feature status:
>
>         1. Merged & Completed features:
>         * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>         * (Wangda) YARN-6223: GPU support on YARN. Features in trunk
>         and works
>         end-to-end.
>         * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
>         services.
>         * (Steve Loughran): HADOOP-13786: S3Guard committer for
>         zero-rename commits.
>         * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
>         of Leaf
>         Queues While Doing Queue Mapping.
>         * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>
>         2. Features close to finish:
>         * (Zhankun) YARN-5983: FPGA support. Majority implementations
>         completed and
>         merged to trunk. Except for UI/documentation.
>         * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
>         done, some
>         discussions going on about implementation.
>         * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>         SchedulingRequest and
>         anti-affinity support. Close to finish, on track to be merged
>         before Jan 30.
>
>         3. Tentative features:
>         * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>         containers. Only one pending patch. Plan to finish before Jan 7th.
>         * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>         challenging to be
>         done before Jan 2018.
>         * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
>         Looks
>         challenging to be done before Jan 2018.
>         * (Varun V) YARN-5673: container-executor write. Given
>         security refactoring
>         of c-e (YARN-6623) is already landed, IMHO other stuff may be
>         moved to 3.2.
>
>         Thanks,
>         Wangda
>
>
>
>
>         On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan
>         <wheeleast@gmail.com <ma...@gmail.com>> wrote:
>
>             Hi all,
>
>             Congratulations on the 3.0.0-GA release!
>
>             As we discussed in the previous email thread [1], I'd like
>             to restart
>             3.1.0 release plans.
>
>             a) Quick summary:
>             a.1 Release status
>             We started 3.1 release discussion on Sep 6, 2017 [1]. As
>             of today,
>             there’re 232 patches loaded on 3.1.0 alone [2], besides 6
>             open blockers and
>             22 open critical issues.
>
>             a.2 Release date update
>             Considering delays of 3.0-GA release by month-and-a-half,
>             I propose to
>             move the dates as follows
>               - feature freeze date from Dec 15, 2017, to Jan 30, 2018
>             - last date for
>             any branches to get merged too;
>               - code freeze (blockers & critical only) date to Feb 08,
>             2018;
>               - release voting start by Feb 18, 2018, leaving time for
>             at least two RCx
>               - release date from Jan 15, 2018, to Feb 28, 2018;
>
>             Unlike before, I added an additional milestone for
>             release-vote-start so
>             that we can account for voting time-period also.
>
>             This overall is still 5 1/2 months of release-timeline
>             unlike the faster
>             cadence we hoped for, but this, in my opinion, is the
>             best-updated timeline
>             given the delays of the final release of 3.0-GA.
>
>             b) Individual feature status:
>             I spoke to several feature owners and checked the status
>             of un-finished
>             features, following are status of features planned to 3.1.0:
>
>             b.1 Merged & Completed features:
>             * (Sunil) YARN-5881: Support absolute value in
>             CapacityScheduler.
>             * (Wangda) YARN-6223: GPU support on YARN. Features in
>             trunk and works
>             end-to-end.
>             * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN
>             native services.
>             * (Steve Loughran): HADOOP-13786: S3Guard committer for
>             zero-rename
>             commits.
>             * (Suma): YARN-7117: Capacity Scheduler: Support Auto
>             Creation of Leaf
>             Queues While Doing Queue Mapping.
>
>             b.2 Features close to finish:
>             * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being
>             voting now.
>             * (Zhankun) YARN-5983: FPGA support. Majority
>             implementations completed
>             and merged to trunk. Except for UI/documentation.
>             * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
>             done, some
>             discussions going on about implementation.
>
>             b.3 Tentative features:
>             * (Arun Suresh). YARN-5972: Support pausing/freezing
>             opportunistic
>             containers. Only one pending patch. Plan to finish before
>             Jan 7th.
>             * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>             challenging to
>             be done before Jan 2018.
>             * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>             SchedulingRequest and
>             anti-affinity support. Tentative will figure out by Jan 1st.
>             * (Anu): HDFS-7240: Ozone. Given the discussion on
>             HDFS-7240. Looks
>             challenging to be done before Jan 2018.
>             * (Varun V) YARN-5673: container-executor write. Given
>             security
>             refactoring of c-e (YARN-6623) is already landed, IMHO
>             other stuff may be
>             moved to 3.2.
>
>             b.4 Additional release drivers
>             * More exhaustive upgrade testing from 2.x to 3.x.
>
>             c) Regarding branch cut:
>
>             We will keep pointing trunk to 3.1 and cut branch-3.1
>             until: A. some
>             feature planned to 3.2 has to be landed on trunk or B.
>             After feature freeze
>             date, whichever comes first.
>
>             I've also talked offline with Vinod to get help on
>             release-management
>             given this is my first release. He agreed to help do this
>             release jointly.
>
>             Thoughts?
>
>             Thanks,
>             Wangda Tan
>
>             [1]
>             https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0
>             <https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0>
>             9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org
>             <http://3Cyarn-dev.hadoop.apache.org>%3E
>             [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
>             fixVersion in (3.1.0)
>             AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>
>
>
>
>     ---------------------------------------------------------------------
>     To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>     <ma...@hadoop.apache.org>
>     For additional commands, e-mail: common-dev-help@hadoop.apache.org
>     <ma...@hadoop.apache.org>
>
>

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Thanks Daniel,

We need to make a decision about this:
https://issues.apache.org/jira/browse/YARN-7292, I believe this is the JIRA
you mentioned correct? Please let me know if there's anything else. And
let's move the discussion on JIRA.

The good news is that resource profile is merged to trunk already so we can
finish that before code freeze date (Feb 08).

+ Sunil as well.

Thanks,
Wangda



On Wed, Jan 17, 2018 at 4:31 PM, Daniel Templeton <da...@cloudera.com>
wrote:

> What's the status on resource profiles?  I believe there are still a
> couple of open JIRAs to rethink some of the design choices.
>
> Daniel
>
>
> On 1/17/18 11:33 AM, Wangda Tan wrote:
>
>> Hi All,
>>
>> Since we're fast approaching previously proposed feature freeze date (Jan
>> 30, about 13 days from today). If you've any features which live in a
>> branch and targeted to 3.1.0, please reply this email thread. Ideally, we
>> should finish branch merging before feature freeze date.
>>
>> Here's an updated 3.1.0 feature status:
>>
>> 1. Merged & Completed features:
>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>> end-to-end.
>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>> commits.
>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>> Queues While Doing Queue Mapping.
>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>>
>> 2. Features close to finish:
>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
>> and
>> merged to trunk. Except for UI/documentation.
>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>> discussions going on about implementation.
>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>> anti-affinity support. Close to finish, on track to be merged before Jan
>> 30.
>>
>> 3. Tentative features:
>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>> containers. Only one pending patch. Plan to finish before Jan 7th.
>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
>> be
>> done before Jan 2018.
>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>> challenging to be done before Jan 2018.
>> * (Varun V) YARN-5673: container-executor write. Given security
>> refactoring
>> of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to
>> 3.2.
>>
>> Thanks,
>> Wangda
>>
>>
>>
>>
>> On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com> wrote:
>>
>> Hi all,
>>>
>>> Congratulations on the 3.0.0-GA release!
>>>
>>> As we discussed in the previous email thread [1], I'd like to restart
>>> 3.1.0 release plans.
>>>
>>> a) Quick summary:
>>> a.1 Release status
>>> We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
>>> there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open blockers
>>> and
>>> 22 open critical issues.
>>>
>>> a.2 Release date update
>>> Considering delays of 3.0-GA release by month-and-a-half, I propose to
>>> move the dates as follows
>>>   - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last date
>>> for
>>> any branches to get merged too;
>>>   - code freeze (blockers & critical only) date to Feb 08, 2018;
>>>   - release voting start by Feb 18, 2018, leaving time for at least two
>>> RCx
>>>   - release date from Jan 15, 2018, to Feb 28, 2018;
>>>
>>> Unlike before, I added an additional milestone for release-vote-start so
>>> that we can account for voting time-period also.
>>>
>>> This overall is still 5 1/2 months of release-timeline unlike the faster
>>> cadence we hoped for, but this, in my opinion, is the best-updated
>>> timeline
>>> given the delays of the final release of 3.0-GA.
>>>
>>> b) Individual feature status:
>>> I spoke to several feature owners and checked the status of un-finished
>>> features, following are status of features planned to 3.1.0:
>>>
>>> b.1 Merged & Completed features:
>>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>>> end-to-end.
>>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>>> commits.
>>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>>> Queues While Doing Queue Mapping.
>>>
>>> b.2 Features close to finish:
>>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
>>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
>>> and merged to trunk. Except for UI/documentation.
>>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>>> discussions going on about implementation.
>>>
>>> b.3 Tentative features:
>>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>>> containers. Only one pending patch. Plan to finish before Jan 7th.
>>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
>>> be done before Jan 2018.
>>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>>> anti-affinity support. Tentative will figure out by Jan 1st.
>>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>>> challenging to be done before Jan 2018.
>>> * (Varun V) YARN-5673: container-executor write. Given security
>>> refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may be
>>> moved to 3.2.
>>>
>>> b.4 Additional release drivers
>>> * More exhaustive upgrade testing from 2.x to 3.x.
>>>
>>> c) Regarding branch cut:
>>>
>>> We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
>>> feature planned to 3.2 has to be landed on trunk or B. After feature
>>> freeze
>>> date, whichever comes first.
>>>
>>> I've also talked offline with Vinod to get help on release-management
>>> given this is my first release. He agreed to help do this release
>>> jointly.
>>>
>>> Thoughts?
>>>
>>> Thanks,
>>> Wangda Tan
>>>
>>> [1] https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0
>>> 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>>> [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>>> AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>>>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Thanks Daniel,

We need to make a decision about this:
https://issues.apache.org/jira/browse/YARN-7292, I believe this is the JIRA
you mentioned correct? Please let me know if there's anything else. And
let's move the discussion on JIRA.

The good news is that resource profile is merged to trunk already so we can
finish that before code freeze date (Feb 08).

+ Sunil as well.

Thanks,
Wangda



On Wed, Jan 17, 2018 at 4:31 PM, Daniel Templeton <da...@cloudera.com>
wrote:

> What's the status on resource profiles?  I believe there are still a
> couple of open JIRAs to rethink some of the design choices.
>
> Daniel
>
>
> On 1/17/18 11:33 AM, Wangda Tan wrote:
>
>> Hi All,
>>
>> Since we're fast approaching previously proposed feature freeze date (Jan
>> 30, about 13 days from today). If you've any features which live in a
>> branch and targeted to 3.1.0, please reply this email thread. Ideally, we
>> should finish branch merging before feature freeze date.
>>
>> Here's an updated 3.1.0 feature status:
>>
>> 1. Merged & Completed features:
>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>> end-to-end.
>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>> commits.
>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>> Queues While Doing Queue Mapping.
>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>>
>> 2. Features close to finish:
>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
>> and
>> merged to trunk. Except for UI/documentation.
>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>> discussions going on about implementation.
>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>> anti-affinity support. Close to finish, on track to be merged before Jan
>> 30.
>>
>> 3. Tentative features:
>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>> containers. Only one pending patch. Plan to finish before Jan 7th.
>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
>> be
>> done before Jan 2018.
>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>> challenging to be done before Jan 2018.
>> * (Varun V) YARN-5673: container-executor write. Given security
>> refactoring
>> of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to
>> 3.2.
>>
>> Thanks,
>> Wangda
>>
>>
>>
>>
>> On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com> wrote:
>>
>> Hi all,
>>>
>>> Congratulations on the 3.0.0-GA release!
>>>
>>> As we discussed in the previous email thread [1], I'd like to restart
>>> 3.1.0 release plans.
>>>
>>> a) Quick summary:
>>> a.1 Release status
>>> We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
>>> there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open blockers
>>> and
>>> 22 open critical issues.
>>>
>>> a.2 Release date update
>>> Considering delays of 3.0-GA release by month-and-a-half, I propose to
>>> move the dates as follows
>>>   - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last date
>>> for
>>> any branches to get merged too;
>>>   - code freeze (blockers & critical only) date to Feb 08, 2018;
>>>   - release voting start by Feb 18, 2018, leaving time for at least two
>>> RCx
>>>   - release date from Jan 15, 2018, to Feb 28, 2018;
>>>
>>> Unlike before, I added an additional milestone for release-vote-start so
>>> that we can account for voting time-period also.
>>>
>>> This overall is still 5 1/2 months of release-timeline unlike the faster
>>> cadence we hoped for, but this, in my opinion, is the best-updated
>>> timeline
>>> given the delays of the final release of 3.0-GA.
>>>
>>> b) Individual feature status:
>>> I spoke to several feature owners and checked the status of un-finished
>>> features, following are status of features planned to 3.1.0:
>>>
>>> b.1 Merged & Completed features:
>>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>>> end-to-end.
>>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>>> commits.
>>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>>> Queues While Doing Queue Mapping.
>>>
>>> b.2 Features close to finish:
>>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
>>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
>>> and merged to trunk. Except for UI/documentation.
>>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>>> discussions going on about implementation.
>>>
>>> b.3 Tentative features:
>>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>>> containers. Only one pending patch. Plan to finish before Jan 7th.
>>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
>>> be done before Jan 2018.
>>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>>> anti-affinity support. Tentative will figure out by Jan 1st.
>>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>>> challenging to be done before Jan 2018.
>>> * (Varun V) YARN-5673: container-executor write. Given security
>>> refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may be
>>> moved to 3.2.
>>>
>>> b.4 Additional release drivers
>>> * More exhaustive upgrade testing from 2.x to 3.x.
>>>
>>> c) Regarding branch cut:
>>>
>>> We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
>>> feature planned to 3.2 has to be landed on trunk or B. After feature
>>> freeze
>>> date, whichever comes first.
>>>
>>> I've also talked offline with Vinod to get help on release-management
>>> given this is my first release. He agreed to help do this release
>>> jointly.
>>>
>>> Thoughts?
>>>
>>> Thanks,
>>> Wangda Tan
>>>
>>> [1] https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0
>>> 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>>> [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>>> AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>>>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Thanks Daniel,

We need to make a decision about this:
https://issues.apache.org/jira/browse/YARN-7292, I believe this is the JIRA
you mentioned correct? Please let me know if there's anything else. And
let's move the discussion on JIRA.

The good news is that resource profile is merged to trunk already so we can
finish that before code freeze date (Feb 08).

+ Sunil as well.

Thanks,
Wangda



On Wed, Jan 17, 2018 at 4:31 PM, Daniel Templeton <da...@cloudera.com>
wrote:

> What's the status on resource profiles?  I believe there are still a
> couple of open JIRAs to rethink some of the design choices.
>
> Daniel
>
>
> On 1/17/18 11:33 AM, Wangda Tan wrote:
>
>> Hi All,
>>
>> Since we're fast approaching previously proposed feature freeze date (Jan
>> 30, about 13 days from today). If you've any features which live in a
>> branch and targeted to 3.1.0, please reply this email thread. Ideally, we
>> should finish branch merging before feature freeze date.
>>
>> Here's an updated 3.1.0 feature status:
>>
>> 1. Merged & Completed features:
>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>> end-to-end.
>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>> commits.
>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>> Queues While Doing Queue Mapping.
>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>>
>> 2. Features close to finish:
>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
>> and
>> merged to trunk. Except for UI/documentation.
>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>> discussions going on about implementation.
>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>> anti-affinity support. Close to finish, on track to be merged before Jan
>> 30.
>>
>> 3. Tentative features:
>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>> containers. Only one pending patch. Plan to finish before Jan 7th.
>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
>> be
>> done before Jan 2018.
>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>> challenging to be done before Jan 2018.
>> * (Varun V) YARN-5673: container-executor write. Given security
>> refactoring
>> of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to
>> 3.2.
>>
>> Thanks,
>> Wangda
>>
>>
>>
>>
>> On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com> wrote:
>>
>> Hi all,
>>>
>>> Congratulations on the 3.0.0-GA release!
>>>
>>> As we discussed in the previous email thread [1], I'd like to restart
>>> 3.1.0 release plans.
>>>
>>> a) Quick summary:
>>> a.1 Release status
>>> We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
>>> there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open blockers
>>> and
>>> 22 open critical issues.
>>>
>>> a.2 Release date update
>>> Considering delays of 3.0-GA release by month-and-a-half, I propose to
>>> move the dates as follows
>>>   - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last date
>>> for
>>> any branches to get merged too;
>>>   - code freeze (blockers & critical only) date to Feb 08, 2018;
>>>   - release voting start by Feb 18, 2018, leaving time for at least two
>>> RCx
>>>   - release date from Jan 15, 2018, to Feb 28, 2018;
>>>
>>> Unlike before, I added an additional milestone for release-vote-start so
>>> that we can account for voting time-period also.
>>>
>>> This overall is still 5 1/2 months of release-timeline unlike the faster
>>> cadence we hoped for, but this, in my opinion, is the best-updated
>>> timeline
>>> given the delays of the final release of 3.0-GA.
>>>
>>> b) Individual feature status:
>>> I spoke to several feature owners and checked the status of un-finished
>>> features, following are status of features planned to 3.1.0:
>>>
>>> b.1 Merged & Completed features:
>>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>>> end-to-end.
>>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>>> commits.
>>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>>> Queues While Doing Queue Mapping.
>>>
>>> b.2 Features close to finish:
>>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
>>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
>>> and merged to trunk. Except for UI/documentation.
>>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>>> discussions going on about implementation.
>>>
>>> b.3 Tentative features:
>>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>>> containers. Only one pending patch. Plan to finish before Jan 7th.
>>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
>>> be done before Jan 2018.
>>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>>> anti-affinity support. Tentative will figure out by Jan 1st.
>>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>>> challenging to be done before Jan 2018.
>>> * (Varun V) YARN-5673: container-executor write. Given security
>>> refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may be
>>> moved to 3.2.
>>>
>>> b.4 Additional release drivers
>>> * More exhaustive upgrade testing from 2.x to 3.x.
>>>
>>> c) Regarding branch cut:
>>>
>>> We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
>>> feature planned to 3.2 has to be landed on trunk or B. After feature
>>> freeze
>>> date, whichever comes first.
>>>
>>> I've also talked offline with Vinod to get help on release-management
>>> given this is my first release. He agreed to help do this release
>>> jointly.
>>>
>>> Thoughts?
>>>
>>> Thanks,
>>> Wangda Tan
>>>
>>> [1] https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0
>>> 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>>> [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>>> AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>>>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Thanks Daniel,

We need to make a decision about this:
https://issues.apache.org/jira/browse/YARN-7292, I believe this is the JIRA
you mentioned correct? Please let me know if there's anything else. And
let's move the discussion on JIRA.

The good news is that resource profile is merged to trunk already so we can
finish that before code freeze date (Feb 08).

+ Sunil as well.

Thanks,
Wangda



On Wed, Jan 17, 2018 at 4:31 PM, Daniel Templeton <da...@cloudera.com>
wrote:

> What's the status on resource profiles?  I believe there are still a
> couple of open JIRAs to rethink some of the design choices.
>
> Daniel
>
>
> On 1/17/18 11:33 AM, Wangda Tan wrote:
>
>> Hi All,
>>
>> Since we're fast approaching previously proposed feature freeze date (Jan
>> 30, about 13 days from today). If you've any features which live in a
>> branch and targeted to 3.1.0, please reply this email thread. Ideally, we
>> should finish branch merging before feature freeze date.
>>
>> Here's an updated 3.1.0 feature status:
>>
>> 1. Merged & Completed features:
>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>> end-to-end.
>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>> commits.
>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>> Queues While Doing Queue Mapping.
>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>>
>> 2. Features close to finish:
>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
>> and
>> merged to trunk. Except for UI/documentation.
>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>> discussions going on about implementation.
>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>> anti-affinity support. Close to finish, on track to be merged before Jan
>> 30.
>>
>> 3. Tentative features:
>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>> containers. Only one pending patch. Plan to finish before Jan 7th.
>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
>> be
>> done before Jan 2018.
>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>> challenging to be done before Jan 2018.
>> * (Varun V) YARN-5673: container-executor write. Given security
>> refactoring
>> of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to
>> 3.2.
>>
>> Thanks,
>> Wangda
>>
>>
>>
>>
>> On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com> wrote:
>>
>> Hi all,
>>>
>>> Congratulations on the 3.0.0-GA release!
>>>
>>> As we discussed in the previous email thread [1], I'd like to restart
>>> 3.1.0 release plans.
>>>
>>> a) Quick summary:
>>> a.1 Release status
>>> We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
>>> there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open blockers
>>> and
>>> 22 open critical issues.
>>>
>>> a.2 Release date update
>>> Considering delays of 3.0-GA release by month-and-a-half, I propose to
>>> move the dates as follows
>>>   - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last date
>>> for
>>> any branches to get merged too;
>>>   - code freeze (blockers & critical only) date to Feb 08, 2018;
>>>   - release voting start by Feb 18, 2018, leaving time for at least two
>>> RCx
>>>   - release date from Jan 15, 2018, to Feb 28, 2018;
>>>
>>> Unlike before, I added an additional milestone for release-vote-start so
>>> that we can account for voting time-period also.
>>>
>>> This overall is still 5 1/2 months of release-timeline unlike the faster
>>> cadence we hoped for, but this, in my opinion, is the best-updated
>>> timeline
>>> given the delays of the final release of 3.0-GA.
>>>
>>> b) Individual feature status:
>>> I spoke to several feature owners and checked the status of un-finished
>>> features, following are status of features planned to 3.1.0:
>>>
>>> b.1 Merged & Completed features:
>>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>>> end-to-end.
>>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>>> commits.
>>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>>> Queues While Doing Queue Mapping.
>>>
>>> b.2 Features close to finish:
>>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
>>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
>>> and merged to trunk. Except for UI/documentation.
>>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>>> discussions going on about implementation.
>>>
>>> b.3 Tentative features:
>>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>>> containers. Only one pending patch. Plan to finish before Jan 7th.
>>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
>>> be done before Jan 2018.
>>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>>> anti-affinity support. Tentative will figure out by Jan 1st.
>>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>>> challenging to be done before Jan 2018.
>>> * (Varun V) YARN-5673: container-executor write. Given security
>>> refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may be
>>> moved to 3.2.
>>>
>>> b.4 Additional release drivers
>>> * More exhaustive upgrade testing from 2.x to 3.x.
>>>
>>> c) Regarding branch cut:
>>>
>>> We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
>>> feature planned to 3.2 has to be landed on trunk or B. After feature
>>> freeze
>>> date, whichever comes first.
>>>
>>> I've also talked offline with Vinod to get help on release-management
>>> given this is my first release. He agreed to help do this release
>>> jointly.
>>>
>>> Thoughts?
>>>
>>> Thanks,
>>> Wangda Tan
>>>
>>> [1] https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0
>>> 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>>> [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>>> AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>>>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>

Re: Hadoop 3.1.0 release discussion

[Daniel Templeton <da...@cloudera.com>]

What's the status on resource profiles?  I believe there are still a 
couple of open JIRAs to rethink some of the design choices.

Daniel

On 1/17/18 11:33 AM, Wangda Tan wrote:
> Hi All,
>
> Since we're fast approaching previously proposed feature freeze date (Jan
> 30, about 13 days from today). If you've any features which live in a
> branch and targeted to 3.1.0, please reply this email thread. Ideally, we
> should finish branch merging before feature freeze date.
>
> Here's an updated 3.1.0 feature status:
>
> 1. Merged & Completed features:
> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
> end-to-end.
> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename commits.
> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
> Queues While Doing Queue Mapping.
> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>
> 2. Features close to finish:
> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed and
> merged to trunk. Except for UI/documentation.
> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
> discussions going on about implementation.
> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
> anti-affinity support. Close to finish, on track to be merged before Jan 30.
>
> 3. Tentative features:
> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
> containers. Only one pending patch. Plan to finish before Jan 7th.
> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to be
> done before Jan 2018.
> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
> challenging to be done before Jan 2018.
> * (Varun V) YARN-5673: container-executor write. Given security refactoring
> of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to 3.2.
>
> Thanks,
> Wangda
>
>
>
>
> On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com> wrote:
>
>> Hi all,
>>
>> Congratulations on the 3.0.0-GA release!
>>
>> As we discussed in the previous email thread [1], I'd like to restart
>> 3.1.0 release plans.
>>
>> a) Quick summary:
>> a.1 Release status
>> We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
>> there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open blockers and
>> 22 open critical issues.
>>
>> a.2 Release date update
>> Considering delays of 3.0-GA release by month-and-a-half, I propose to
>> move the dates as follows
>>   - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last date for
>> any branches to get merged too;
>>   - code freeze (blockers & critical only) date to Feb 08, 2018;
>>   - release voting start by Feb 18, 2018, leaving time for at least two RCx
>>   - release date from Jan 15, 2018, to Feb 28, 2018;
>>
>> Unlike before, I added an additional milestone for release-vote-start so
>> that we can account for voting time-period also.
>>
>> This overall is still 5 1/2 months of release-timeline unlike the faster
>> cadence we hoped for, but this, in my opinion, is the best-updated timeline
>> given the delays of the final release of 3.0-GA.
>>
>> b) Individual feature status:
>> I spoke to several feature owners and checked the status of un-finished
>> features, following are status of features planned to 3.1.0:
>>
>> b.1 Merged & Completed features:
>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>> end-to-end.
>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>> commits.
>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>> Queues While Doing Queue Mapping.
>>
>> b.2 Features close to finish:
>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
>> and merged to trunk. Except for UI/documentation.
>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>> discussions going on about implementation.
>>
>> b.3 Tentative features:
>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>> containers. Only one pending patch. Plan to finish before Jan 7th.
>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
>> be done before Jan 2018.
>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>> anti-affinity support. Tentative will figure out by Jan 1st.
>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>> challenging to be done before Jan 2018.
>> * (Varun V) YARN-5673: container-executor write. Given security
>> refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may be
>> moved to 3.2.
>>
>> b.4 Additional release drivers
>> * More exhaustive upgrade testing from 2.x to 3.x.
>>
>> c) Regarding branch cut:
>>
>> We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
>> feature planned to 3.2 has to be landed on trunk or B. After feature freeze
>> date, whichever comes first.
>>
>> I've also talked offline with Vinod to get help on release-management
>> given this is my first release. He agreed to help do this release jointly.
>>
>> Thoughts?
>>
>> Thanks,
>> Wangda Tan
>>
>> [1] https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0
>> 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>> [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

Re: Hadoop 3.1.0 release discussion

[Daniel Templeton <da...@cloudera.com>]

What's the status on resource profiles?  I believe there are still a 
couple of open JIRAs to rethink some of the design choices.

Daniel

On 1/17/18 11:33 AM, Wangda Tan wrote:
> Hi All,
>
> Since we're fast approaching previously proposed feature freeze date (Jan
> 30, about 13 days from today). If you've any features which live in a
> branch and targeted to 3.1.0, please reply this email thread. Ideally, we
> should finish branch merging before feature freeze date.
>
> Here's an updated 3.1.0 feature status:
>
> 1. Merged & Completed features:
> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
> end-to-end.
> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename commits.
> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
> Queues While Doing Queue Mapping.
> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>
> 2. Features close to finish:
> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed and
> merged to trunk. Except for UI/documentation.
> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
> discussions going on about implementation.
> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
> anti-affinity support. Close to finish, on track to be merged before Jan 30.
>
> 3. Tentative features:
> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
> containers. Only one pending patch. Plan to finish before Jan 7th.
> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to be
> done before Jan 2018.
> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
> challenging to be done before Jan 2018.
> * (Varun V) YARN-5673: container-executor write. Given security refactoring
> of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to 3.2.
>
> Thanks,
> Wangda
>
>
>
>
> On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com> wrote:
>
>> Hi all,
>>
>> Congratulations on the 3.0.0-GA release!
>>
>> As we discussed in the previous email thread [1], I'd like to restart
>> 3.1.0 release plans.
>>
>> a) Quick summary:
>> a.1 Release status
>> We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
>> there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open blockers and
>> 22 open critical issues.
>>
>> a.2 Release date update
>> Considering delays of 3.0-GA release by month-and-a-half, I propose to
>> move the dates as follows
>>   - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last date for
>> any branches to get merged too;
>>   - code freeze (blockers & critical only) date to Feb 08, 2018;
>>   - release voting start by Feb 18, 2018, leaving time for at least two RCx
>>   - release date from Jan 15, 2018, to Feb 28, 2018;
>>
>> Unlike before, I added an additional milestone for release-vote-start so
>> that we can account for voting time-period also.
>>
>> This overall is still 5 1/2 months of release-timeline unlike the faster
>> cadence we hoped for, but this, in my opinion, is the best-updated timeline
>> given the delays of the final release of 3.0-GA.
>>
>> b) Individual feature status:
>> I spoke to several feature owners and checked the status of un-finished
>> features, following are status of features planned to 3.1.0:
>>
>> b.1 Merged & Completed features:
>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>> end-to-end.
>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>> commits.
>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>> Queues While Doing Queue Mapping.
>>
>> b.2 Features close to finish:
>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
>> and merged to trunk. Except for UI/documentation.
>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>> discussions going on about implementation.
>>
>> b.3 Tentative features:
>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>> containers. Only one pending patch. Plan to finish before Jan 7th.
>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
>> be done before Jan 2018.
>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>> anti-affinity support. Tentative will figure out by Jan 1st.
>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>> challenging to be done before Jan 2018.
>> * (Varun V) YARN-5673: container-executor write. Given security
>> refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may be
>> moved to 3.2.
>>
>> b.4 Additional release drivers
>> * More exhaustive upgrade testing from 2.x to 3.x.
>>
>> c) Regarding branch cut:
>>
>> We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
>> feature planned to 3.2 has to be landed on trunk or B. After feature freeze
>> date, whichever comes first.
>>
>> I've also talked offline with Vinod to get help on release-management
>> given this is my first release. He agreed to help do this release jointly.
>>
>> Thoughts?
>>
>> Thanks,
>> Wangda Tan
>>
>> [1] https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0
>> 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>> [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Hi Uma,

Thanks for the update,

As we discussed before, we will delay cutting branch-3.1 until commit has
to go to 3.2 to avoid maintaining two branches. Please let me know when the
merge vote of HDFS-10285 started, I will cut branch-3.1 after that.

Best,
Wangda

On Wed, Jan 31, 2018 at 3:58 AM, Gangumalla, Uma <um...@intel.com>
wrote:

> Hi Wangda,
>
>
>
> Daryn has provided his feedback and we would like to handle some of his
> feedbacks before merge.
>
> That will take couple of more days for us to go through review cycles etc.
>
> Next couple of days I will be in travel, I would not be able to put my
> efforts in it. Meanwhile Rakesh/Surendra will update if any changes in the
> plan.
>
>
>
> So, to conclude, you can proceed cutting the branch and we will merge
> after reviews closed, that may go into 3.2.
>
>
>
> Regards,
>
> Uma
>
>
>
> *From: *Wangda Tan <wh...@gmail.com>
> *Date: *Tuesday, January 30, 2018 at 6:24 AM
> *To: *Uma Gangumalla <um...@intel.com>
> *Cc: *"hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "
> common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod
> Kumar Vavilapalli <vi...@hortonworks.com>
> *Subject: *Re: Hadoop 3.1.0 release discussion
>
>
>
> Thanks for the update.
>
>
>
> On Tue, Jan 30, 2018 at 4:51 PM, Gangumalla, Uma <um...@intel.com>
> wrote:
>
> Hi Wangda,
>
> Sorry that we have not started vote on 29th. Daryn is reviewing the branch
> and he needs a day for his finalized review, then we have to prioritize the
> comments and decide.
> Will keep updated here.
>
> Regards,
> Uma
>
>
> On 1/28/18, 5:31 PM, "Wangda Tan" <wh...@gmail.com> wrote:
>
>     Hi Uma,
>
>     Thanks, I saw HDFS-13050 has been resolved 4 hours ago, I don't see any
>     other blockers under HDFS-10285. I think you guys should be able to
> start
>     voting thread in time for merging to trunk.
>
>     - Wangda
>
>     On Mon, Jan 29, 2018 at 3:12 AM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     wrote:
>
>     > Hi Wangda,
>     >
>     >
>     >
>     > Sorry for the delay.
>     >
>     >
>     >
>     > >>* (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers:
>     > HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the
> two
>     > JIRAs?
>     >
>     > We have only one blocker now that is HDFS-13050 and we finished the
> key
>     > implementation from HDFS-12995, by HDFS-13075.
>     >
>     > We are planning to start vote by tomorrow (29th PST time). So, we
> request
>     > you to give us time for running vote. We will keep SPS off by
> default. So,
>     > interested users only can enable explicitly.
>     >
>     >
>     >
>     > Regards,
>     >
>     > Uma
>     >
>     >
>     >
>     > *From: *Wangda Tan <wh...@gmail.com>
>     > *Date: *Friday, January 26, 2018 at 5:21 PM
>     > *To: *Uma Gangumalla <um...@intel.com>
>     > *Cc: *"hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
>     > yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "
>     > common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
>     > mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod
>     > Kumar Vavilapalli <vi...@hortonworks.com>
>     > *Subject: *Re: Hadoop 3.1.0 release discussion
>
>     >
>     >
>     >
>     > Hi All,
>     >
>     >
>     >
>     > Just a reminder about feature freeze date.
>     >
>     >
>     >
>     > Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days
> from
>     > today), If you've any features which live in a branch and targeted to
>     > 3.1.0, please reply this email thread. Ideally, we should finish
> branch
>     > merging before feature freeze date.
>     >
>     >
>     >
>     > Here's an updated 3.1.0 feature status:
>     >
>     >
>     >
>     > 1. Merged features:
>     >
>     > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     >
>     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
> works
>     > end-to-end.
>     >
>     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
>     >
>     > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>     > commits.
>     >
>     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of
> Leaf
>     > Queues While Doing Queue Mapping.
>     >
>     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >
>     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
> completed
>     > and merged to trunk. Except for UI/documentation.
>     >
>     >
>     >
>     > 2. Features close to finish:
>     >
>     > * (Uma) HDFS-10285: HDFS SPS. There're two remaining
>     > blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the
> ETA of
>     > the two JIRAs?
>     >
>     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest
> and
>     > anti-affinity support. (Voting thread started).
>     >
>     >
>     >
>     > 3. Tentative features:
>     >
>     > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>     > containers. Only one pending patch. Plan to finish before Jan 7th.
>     >
>     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
> challenging to
>     > be done before Jan 2018.
>     >
>     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>     > challenging to be done before Jan 2018.
>     >
>     > * (Varun V) YARN-5673: container-executor write. Given security
>     > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff
> may be
>     > moved to 3.2.
>     >
>     >
>     >
>     > Thanks,
>     >
>     > Wangda
>     >
>     >
>     >
>     >
>     >
>     > On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     > wrote:
>     >
>     > Sure, Wangda.
>     >
>     > Regards,
>     > Uma
>     >
>     >
>     > On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>     >
>     >     Thanks Uma,
>     >
>     >     Could you update this thread once the merge vote started?
>     >
>     >     Best,
>     >     Wangda
>     >
>     >     On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <
>     > uma.gangumalla@intel.com>
>     >     wrote:
>     >
>     >     > HI Wangda,
>     >     >
>     >     >  Thank you for the head-up mail.
>     >     >  We are in the branch (HDFS-10285) and trying to push the tasks
>     > sooner
>     >     > before the deadline.
>     >     >
>     >     > Regards,
>     >     > Uma
>     >     >
>     >     > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com>
> wrote:
>     >     >
>     >     >     Hi All,
>     >     >
>     >     >     Since we're fast approaching previously proposed feature
> freeze
>     > date
>     >     > (Jan
>     >     >     30, about 13 days from today). If you've any features
> which live
>     > in a
>     >     >     branch and targeted to 3.1.0, please reply this email
> thread.
>     > Ideally,
>     >     > we
>     >     >     should finish branch merging before feature freeze date.
>     >     >
>     >     >     Here's an updated 3.1.0 feature status:
>     >     >
>     >     >     1. Merged & Completed features:
>     >     >     * (Sunil) YARN-5881: Support absolute value in
> CapacityScheduler.
>     >     >     * (Wangda) YARN-6223: GPU support on YARN. Features in
> trunk and
>     > works
>     >     >     end-to-end.
>     >     >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN
> native
>     > services.
>     >     >     * (Steve Loughran): HADOOP-13786: S3Guard committer for
>     > zero-rename
>     >     > commits.
>     >     >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto
> Creation
>     > of Leaf
>     >     >     Queues While Doing Queue Mapping.
>     >     >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >     >
>     >     >     2. Features close to finish:
>     >     >     * (Zhankun) YARN-5983: FPGA support. Majority
> implementations
>     >     > completed and
>     >     >     merged to trunk. Except for UI/documentation.
>     >     >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
> done,
>     > some
>     >     >     discussions going on about implementation.
>     >     >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>     > SchedulingRequest and
>     >     >     anti-affinity support. Close to finish, on track to be
> merged
>     > before
>     >     > Jan 30.
>     >     >
>     >     >     3. Tentative features:
>     >     >     * (Arun Suresh). YARN-5972: Support pausing/freezing
>     > opportunistic
>     >     >     containers. Only one pending patch. Plan to finish before
> Jan
>     > 7th.
>     >     >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     > challenging
>     >     > to be
>     >     >     done before Jan 2018.
>     >     >     * (Anu): HDFS-7240: Ozone. Given the discussion on
> HDFS-7240.
>     > Looks
>     >     >     challenging to be done before Jan 2018.
>     >     >     * (Varun V) YARN-5673: container-executor write. Given
> security
>     >     > refactoring
>     >     >     of c-e (YARN-6623) is already landed, IMHO other stuff may
> be
>     > moved to
>     >     > 3.2.
>     >     >
>     >     >     Thanks,
>     >     >     Wangda
>     >     >
>     >     >
>     >     >
>     >     >
>     >     >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <
> wheeleast@gmail.com
>     > >
>     >     > wrote:
>     >     >
>     >     >     > Hi all,
>     >     >     >
>     >     >     > Congratulations on the 3.0.0-GA release!
>     >     >     >
>     >     >     > As we discussed in the previous email thread [1], I'd
> like to
>     > restart
>     >     >     > 3.1.0 release plans.
>     >     >     >
>     >     >     > a) Quick summary:
>     >     >     > a.1 Release status
>     >     >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As
> of
>     > today,
>     >     >     > there’re 232 patches loaded on 3.1.0 alone [2], besides
> 6 open
>     >     > blockers and
>     >     >     > 22 open critical issues.
>     >     >     >
>     >     >     > a.2 Release date update
>     >     >     > Considering delays of 3.0-GA release by
> month-and-a-half, I
>     > propose
>     >     > to
>     >     >     > move the dates as follows
>     >     >     >  - feature freeze date from Dec 15, 2017, to Jan 30,
> 2018 -
>     > last
>     >     > date for
>     >     >     > any branches to get merged too;
>     >     >     >  - code freeze (blockers & critical only) date to Feb
> 08, 2018;
>     >     >     >  - release voting start by Feb 18, 2018, leaving time
> for at
>     > least
>     >     > two RCx
>     >     >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
>     >     >     >
>     >     >     > Unlike before, I added an additional milestone for
>     >     > release-vote-start so
>     >     >     > that we can account for voting time-period also.
>     >     >     >
>     >     >     > This overall is still 5 1/2 months of release-timeline
> unlike
>     > the
>     >     > faster
>     >     >     > cadence we hoped for, but this, in my opinion, is the
>     > best-updated
>     >     > timeline
>     >     >     > given the delays of the final release of 3.0-GA.
>     >     >     >
>     >     >     > b) Individual feature status:
>     >     >     > I spoke to several feature owners and checked the status
> of
>     >     > un-finished
>     >     >     > features, following are status of features planned to
> 3.1.0:
>     >     >     >
>     >     >     > b.1 Merged & Completed features:
>     >     >     > * (Sunil) YARN-5881: Support absolute value in
>     > CapacityScheduler.
>     >     >     > * (Wangda) YARN-6223: GPU support on YARN. Features in
> trunk
>     > and
>     >     > works
>     >     >     > end-to-end.
>     >     >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN
> native
>     >     > services.
>     >     >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for
>     > zero-rename
>     >     >     > commits.
>     >     >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto
> Creation
>     > of
>     >     > Leaf
>     >     >     > Queues While Doing Queue Mapping.
>     >     >     >
>     >     >     > b.2 Features close to finish:
>     >     >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being
> voting
>     > now.
>     >     >     > * (Zhankun) YARN-5983: FPGA support. Majority
> implementations
>     >     > completed
>     >     >     > and merged to trunk. Except for UI/documentation.
>     >     >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations
> are
>     > done, some
>     >     >     > discussions going on about implementation.
>     >     >     >
>     >     >     > b.3 Tentative features:
>     >     >     > * (Arun Suresh). YARN-5972: Support pausing/freezing
>     > opportunistic
>     >     >     > containers. Only one pending patch. Plan to finish
> before Jan
>     > 7th.
>     >     >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     >     > challenging to
>     >     >     > be done before Jan 2018.
>     >     >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>     > SchedulingRequest
>     >     > and
>     >     >     > anti-affinity support. Tentative will figure out by Jan
> 1st.
>     >     >     > * (Anu): HDFS-7240: Ozone. Given the discussion on
> HDFS-7240.
>     > Looks
>     >     >     > challenging to be done before Jan 2018.
>     >     >     > * (Varun V) YARN-5673: container-executor write. Given
> security
>     >     >     > refactoring of c-e (YARN-6623) is already landed, IMHO
> other
>     > stuff
>     >     > may be
>     >     >     > moved to 3.2.
>     >     >     >
>     >     >     > b.4 Additional release drivers
>     >     >     > * More exhaustive upgrade testing from 2.x to 3.x.
>     >     >     >
>     >     >     > c) Regarding branch cut:
>     >     >     >
>     >     >     > We will keep pointing trunk to 3.1 and cut branch-3.1
> until:
>     > A. some
>     >     >     > feature planned to 3.2 has to be landed on trunk or B.
> After
>     > feature
>     >     > freeze
>     >     >     > date, whichever comes first.
>     >     >     >
>     >     >     > I've also talked offline with Vinod to get help on
>     > release-management
>     >     >     > given this is my first release. He agreed to help do this
>     > release
>     >     > jointly.
>     >     >     >
>     >     >     > Thoughts?
>     >     >     >
>     >     >     > Thanks,
>     >     >     > Wangda Tan
>     >     >     >
>     >     >     > [1] https://lists.apache.org/thread.html/
>     >     > c11506c3250c9481852130616b3cb0
>     >     >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org
> %3E
>     >     >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
> fixVersion
>     > in
>     >     > (3.1.0)
>     >     >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority
> DESC”
>     >     >     >
>     >     >     >
>     >     >
>     >     >
>     >     >
>     >
>     >
>     >
>
>
>

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Hi Uma,

Thanks for the update,

As we discussed before, we will delay cutting branch-3.1 until commit has
to go to 3.2 to avoid maintaining two branches. Please let me know when the
merge vote of HDFS-10285 started, I will cut branch-3.1 after that.

Best,
Wangda

On Wed, Jan 31, 2018 at 3:58 AM, Gangumalla, Uma <um...@intel.com>
wrote:

> Hi Wangda,
>
>
>
> Daryn has provided his feedback and we would like to handle some of his
> feedbacks before merge.
>
> That will take couple of more days for us to go through review cycles etc.
>
> Next couple of days I will be in travel, I would not be able to put my
> efforts in it. Meanwhile Rakesh/Surendra will update if any changes in the
> plan.
>
>
>
> So, to conclude, you can proceed cutting the branch and we will merge
> after reviews closed, that may go into 3.2.
>
>
>
> Regards,
>
> Uma
>
>
>
> *From: *Wangda Tan <wh...@gmail.com>
> *Date: *Tuesday, January 30, 2018 at 6:24 AM
> *To: *Uma Gangumalla <um...@intel.com>
> *Cc: *"hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "
> common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod
> Kumar Vavilapalli <vi...@hortonworks.com>
> *Subject: *Re: Hadoop 3.1.0 release discussion
>
>
>
> Thanks for the update.
>
>
>
> On Tue, Jan 30, 2018 at 4:51 PM, Gangumalla, Uma <um...@intel.com>
> wrote:
>
> Hi Wangda,
>
> Sorry that we have not started vote on 29th. Daryn is reviewing the branch
> and he needs a day for his finalized review, then we have to prioritize the
> comments and decide.
> Will keep updated here.
>
> Regards,
> Uma
>
>
> On 1/28/18, 5:31 PM, "Wangda Tan" <wh...@gmail.com> wrote:
>
>     Hi Uma,
>
>     Thanks, I saw HDFS-13050 has been resolved 4 hours ago, I don't see any
>     other blockers under HDFS-10285. I think you guys should be able to
> start
>     voting thread in time for merging to trunk.
>
>     - Wangda
>
>     On Mon, Jan 29, 2018 at 3:12 AM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     wrote:
>
>     > Hi Wangda,
>     >
>     >
>     >
>     > Sorry for the delay.
>     >
>     >
>     >
>     > >>* (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers:
>     > HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the
> two
>     > JIRAs?
>     >
>     > We have only one blocker now that is HDFS-13050 and we finished the
> key
>     > implementation from HDFS-12995, by HDFS-13075.
>     >
>     > We are planning to start vote by tomorrow (29th PST time). So, we
> request
>     > you to give us time for running vote. We will keep SPS off by
> default. So,
>     > interested users only can enable explicitly.
>     >
>     >
>     >
>     > Regards,
>     >
>     > Uma
>     >
>     >
>     >
>     > *From: *Wangda Tan <wh...@gmail.com>
>     > *Date: *Friday, January 26, 2018 at 5:21 PM
>     > *To: *Uma Gangumalla <um...@intel.com>
>     > *Cc: *"hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
>     > yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "
>     > common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
>     > mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod
>     > Kumar Vavilapalli <vi...@hortonworks.com>
>     > *Subject: *Re: Hadoop 3.1.0 release discussion
>
>     >
>     >
>     >
>     > Hi All,
>     >
>     >
>     >
>     > Just a reminder about feature freeze date.
>     >
>     >
>     >
>     > Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days
> from
>     > today), If you've any features which live in a branch and targeted to
>     > 3.1.0, please reply this email thread. Ideally, we should finish
> branch
>     > merging before feature freeze date.
>     >
>     >
>     >
>     > Here's an updated 3.1.0 feature status:
>     >
>     >
>     >
>     > 1. Merged features:
>     >
>     > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     >
>     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
> works
>     > end-to-end.
>     >
>     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
>     >
>     > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>     > commits.
>     >
>     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of
> Leaf
>     > Queues While Doing Queue Mapping.
>     >
>     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >
>     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
> completed
>     > and merged to trunk. Except for UI/documentation.
>     >
>     >
>     >
>     > 2. Features close to finish:
>     >
>     > * (Uma) HDFS-10285: HDFS SPS. There're two remaining
>     > blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the
> ETA of
>     > the two JIRAs?
>     >
>     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest
> and
>     > anti-affinity support. (Voting thread started).
>     >
>     >
>     >
>     > 3. Tentative features:
>     >
>     > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>     > containers. Only one pending patch. Plan to finish before Jan 7th.
>     >
>     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
> challenging to
>     > be done before Jan 2018.
>     >
>     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>     > challenging to be done before Jan 2018.
>     >
>     > * (Varun V) YARN-5673: container-executor write. Given security
>     > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff
> may be
>     > moved to 3.2.
>     >
>     >
>     >
>     > Thanks,
>     >
>     > Wangda
>     >
>     >
>     >
>     >
>     >
>     > On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     > wrote:
>     >
>     > Sure, Wangda.
>     >
>     > Regards,
>     > Uma
>     >
>     >
>     > On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>     >
>     >     Thanks Uma,
>     >
>     >     Could you update this thread once the merge vote started?
>     >
>     >     Best,
>     >     Wangda
>     >
>     >     On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <
>     > uma.gangumalla@intel.com>
>     >     wrote:
>     >
>     >     > HI Wangda,
>     >     >
>     >     >  Thank you for the head-up mail.
>     >     >  We are in the branch (HDFS-10285) and trying to push the tasks
>     > sooner
>     >     > before the deadline.
>     >     >
>     >     > Regards,
>     >     > Uma
>     >     >
>     >     > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com>
> wrote:
>     >     >
>     >     >     Hi All,
>     >     >
>     >     >     Since we're fast approaching previously proposed feature
> freeze
>     > date
>     >     > (Jan
>     >     >     30, about 13 days from today). If you've any features
> which live
>     > in a
>     >     >     branch and targeted to 3.1.0, please reply this email
> thread.
>     > Ideally,
>     >     > we
>     >     >     should finish branch merging before feature freeze date.
>     >     >
>     >     >     Here's an updated 3.1.0 feature status:
>     >     >
>     >     >     1. Merged & Completed features:
>     >     >     * (Sunil) YARN-5881: Support absolute value in
> CapacityScheduler.
>     >     >     * (Wangda) YARN-6223: GPU support on YARN. Features in
> trunk and
>     > works
>     >     >     end-to-end.
>     >     >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN
> native
>     > services.
>     >     >     * (Steve Loughran): HADOOP-13786: S3Guard committer for
>     > zero-rename
>     >     > commits.
>     >     >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto
> Creation
>     > of Leaf
>     >     >     Queues While Doing Queue Mapping.
>     >     >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >     >
>     >     >     2. Features close to finish:
>     >     >     * (Zhankun) YARN-5983: FPGA support. Majority
> implementations
>     >     > completed and
>     >     >     merged to trunk. Except for UI/documentation.
>     >     >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
> done,
>     > some
>     >     >     discussions going on about implementation.
>     >     >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>     > SchedulingRequest and
>     >     >     anti-affinity support. Close to finish, on track to be
> merged
>     > before
>     >     > Jan 30.
>     >     >
>     >     >     3. Tentative features:
>     >     >     * (Arun Suresh). YARN-5972: Support pausing/freezing
>     > opportunistic
>     >     >     containers. Only one pending patch. Plan to finish before
> Jan
>     > 7th.
>     >     >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     > challenging
>     >     > to be
>     >     >     done before Jan 2018.
>     >     >     * (Anu): HDFS-7240: Ozone. Given the discussion on
> HDFS-7240.
>     > Looks
>     >     >     challenging to be done before Jan 2018.
>     >     >     * (Varun V) YARN-5673: container-executor write. Given
> security
>     >     > refactoring
>     >     >     of c-e (YARN-6623) is already landed, IMHO other stuff may
> be
>     > moved to
>     >     > 3.2.
>     >     >
>     >     >     Thanks,
>     >     >     Wangda
>     >     >
>     >     >
>     >     >
>     >     >
>     >     >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <
> wheeleast@gmail.com
>     > >
>     >     > wrote:
>     >     >
>     >     >     > Hi all,
>     >     >     >
>     >     >     > Congratulations on the 3.0.0-GA release!
>     >     >     >
>     >     >     > As we discussed in the previous email thread [1], I'd
> like to
>     > restart
>     >     >     > 3.1.0 release plans.
>     >     >     >
>     >     >     > a) Quick summary:
>     >     >     > a.1 Release status
>     >     >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As
> of
>     > today,
>     >     >     > there’re 232 patches loaded on 3.1.0 alone [2], besides
> 6 open
>     >     > blockers and
>     >     >     > 22 open critical issues.
>     >     >     >
>     >     >     > a.2 Release date update
>     >     >     > Considering delays of 3.0-GA release by
> month-and-a-half, I
>     > propose
>     >     > to
>     >     >     > move the dates as follows
>     >     >     >  - feature freeze date from Dec 15, 2017, to Jan 30,
> 2018 -
>     > last
>     >     > date for
>     >     >     > any branches to get merged too;
>     >     >     >  - code freeze (blockers & critical only) date to Feb
> 08, 2018;
>     >     >     >  - release voting start by Feb 18, 2018, leaving time
> for at
>     > least
>     >     > two RCx
>     >     >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
>     >     >     >
>     >     >     > Unlike before, I added an additional milestone for
>     >     > release-vote-start so
>     >     >     > that we can account for voting time-period also.
>     >     >     >
>     >     >     > This overall is still 5 1/2 months of release-timeline
> unlike
>     > the
>     >     > faster
>     >     >     > cadence we hoped for, but this, in my opinion, is the
>     > best-updated
>     >     > timeline
>     >     >     > given the delays of the final release of 3.0-GA.
>     >     >     >
>     >     >     > b) Individual feature status:
>     >     >     > I spoke to several feature owners and checked the status
> of
>     >     > un-finished
>     >     >     > features, following are status of features planned to
> 3.1.0:
>     >     >     >
>     >     >     > b.1 Merged & Completed features:
>     >     >     > * (Sunil) YARN-5881: Support absolute value in
>     > CapacityScheduler.
>     >     >     > * (Wangda) YARN-6223: GPU support on YARN. Features in
> trunk
>     > and
>     >     > works
>     >     >     > end-to-end.
>     >     >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN
> native
>     >     > services.
>     >     >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for
>     > zero-rename
>     >     >     > commits.
>     >     >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto
> Creation
>     > of
>     >     > Leaf
>     >     >     > Queues While Doing Queue Mapping.
>     >     >     >
>     >     >     > b.2 Features close to finish:
>     >     >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being
> voting
>     > now.
>     >     >     > * (Zhankun) YARN-5983: FPGA support. Majority
> implementations
>     >     > completed
>     >     >     > and merged to trunk. Except for UI/documentation.
>     >     >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations
> are
>     > done, some
>     >     >     > discussions going on about implementation.
>     >     >     >
>     >     >     > b.3 Tentative features:
>     >     >     > * (Arun Suresh). YARN-5972: Support pausing/freezing
>     > opportunistic
>     >     >     > containers. Only one pending patch. Plan to finish
> before Jan
>     > 7th.
>     >     >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     >     > challenging to
>     >     >     > be done before Jan 2018.
>     >     >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>     > SchedulingRequest
>     >     > and
>     >     >     > anti-affinity support. Tentative will figure out by Jan
> 1st.
>     >     >     > * (Anu): HDFS-7240: Ozone. Given the discussion on
> HDFS-7240.
>     > Looks
>     >     >     > challenging to be done before Jan 2018.
>     >     >     > * (Varun V) YARN-5673: container-executor write. Given
> security
>     >     >     > refactoring of c-e (YARN-6623) is already landed, IMHO
> other
>     > stuff
>     >     > may be
>     >     >     > moved to 3.2.
>     >     >     >
>     >     >     > b.4 Additional release drivers
>     >     >     > * More exhaustive upgrade testing from 2.x to 3.x.
>     >     >     >
>     >     >     > c) Regarding branch cut:
>     >     >     >
>     >     >     > We will keep pointing trunk to 3.1 and cut branch-3.1
> until:
>     > A. some
>     >     >     > feature planned to 3.2 has to be landed on trunk or B.
> After
>     > feature
>     >     > freeze
>     >     >     > date, whichever comes first.
>     >     >     >
>     >     >     > I've also talked offline with Vinod to get help on
>     > release-management
>     >     >     > given this is my first release. He agreed to help do this
>     > release
>     >     > jointly.
>     >     >     >
>     >     >     > Thoughts?
>     >     >     >
>     >     >     > Thanks,
>     >     >     > Wangda Tan
>     >     >     >
>     >     >     > [1] https://lists.apache.org/thread.html/
>     >     > c11506c3250c9481852130616b3cb0
>     >     >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org
> %3E
>     >     >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
> fixVersion
>     > in
>     >     > (3.1.0)
>     >     >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority
> DESC”
>     >     >     >
>     >     >     >
>     >     >
>     >     >
>     >     >
>     >
>     >
>     >
>
>
>

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Hi Uma,

Thanks for the update,

As we discussed before, we will delay cutting branch-3.1 until commit has
to go to 3.2 to avoid maintaining two branches. Please let me know when the
merge vote of HDFS-10285 started, I will cut branch-3.1 after that.

Best,
Wangda

On Wed, Jan 31, 2018 at 3:58 AM, Gangumalla, Uma <um...@intel.com>
wrote:

> Hi Wangda,
>
>
>
> Daryn has provided his feedback and we would like to handle some of his
> feedbacks before merge.
>
> That will take couple of more days for us to go through review cycles etc.
>
> Next couple of days I will be in travel, I would not be able to put my
> efforts in it. Meanwhile Rakesh/Surendra will update if any changes in the
> plan.
>
>
>
> So, to conclude, you can proceed cutting the branch and we will merge
> after reviews closed, that may go into 3.2.
>
>
>
> Regards,
>
> Uma
>
>
>
> *From: *Wangda Tan <wh...@gmail.com>
> *Date: *Tuesday, January 30, 2018 at 6:24 AM
> *To: *Uma Gangumalla <um...@intel.com>
> *Cc: *"hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "
> common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod
> Kumar Vavilapalli <vi...@hortonworks.com>
> *Subject: *Re: Hadoop 3.1.0 release discussion
>
>
>
> Thanks for the update.
>
>
>
> On Tue, Jan 30, 2018 at 4:51 PM, Gangumalla, Uma <um...@intel.com>
> wrote:
>
> Hi Wangda,
>
> Sorry that we have not started vote on 29th. Daryn is reviewing the branch
> and he needs a day for his finalized review, then we have to prioritize the
> comments and decide.
> Will keep updated here.
>
> Regards,
> Uma
>
>
> On 1/28/18, 5:31 PM, "Wangda Tan" <wh...@gmail.com> wrote:
>
>     Hi Uma,
>
>     Thanks, I saw HDFS-13050 has been resolved 4 hours ago, I don't see any
>     other blockers under HDFS-10285. I think you guys should be able to
> start
>     voting thread in time for merging to trunk.
>
>     - Wangda
>
>     On Mon, Jan 29, 2018 at 3:12 AM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     wrote:
>
>     > Hi Wangda,
>     >
>     >
>     >
>     > Sorry for the delay.
>     >
>     >
>     >
>     > >>* (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers:
>     > HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the
> two
>     > JIRAs?
>     >
>     > We have only one blocker now that is HDFS-13050 and we finished the
> key
>     > implementation from HDFS-12995, by HDFS-13075.
>     >
>     > We are planning to start vote by tomorrow (29th PST time). So, we
> request
>     > you to give us time for running vote. We will keep SPS off by
> default. So,
>     > interested users only can enable explicitly.
>     >
>     >
>     >
>     > Regards,
>     >
>     > Uma
>     >
>     >
>     >
>     > *From: *Wangda Tan <wh...@gmail.com>
>     > *Date: *Friday, January 26, 2018 at 5:21 PM
>     > *To: *Uma Gangumalla <um...@intel.com>
>     > *Cc: *"hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
>     > yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "
>     > common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
>     > mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod
>     > Kumar Vavilapalli <vi...@hortonworks.com>
>     > *Subject: *Re: Hadoop 3.1.0 release discussion
>
>     >
>     >
>     >
>     > Hi All,
>     >
>     >
>     >
>     > Just a reminder about feature freeze date.
>     >
>     >
>     >
>     > Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days
> from
>     > today), If you've any features which live in a branch and targeted to
>     > 3.1.0, please reply this email thread. Ideally, we should finish
> branch
>     > merging before feature freeze date.
>     >
>     >
>     >
>     > Here's an updated 3.1.0 feature status:
>     >
>     >
>     >
>     > 1. Merged features:
>     >
>     > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     >
>     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
> works
>     > end-to-end.
>     >
>     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
>     >
>     > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>     > commits.
>     >
>     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of
> Leaf
>     > Queues While Doing Queue Mapping.
>     >
>     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >
>     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
> completed
>     > and merged to trunk. Except for UI/documentation.
>     >
>     >
>     >
>     > 2. Features close to finish:
>     >
>     > * (Uma) HDFS-10285: HDFS SPS. There're two remaining
>     > blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the
> ETA of
>     > the two JIRAs?
>     >
>     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest
> and
>     > anti-affinity support. (Voting thread started).
>     >
>     >
>     >
>     > 3. Tentative features:
>     >
>     > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>     > containers. Only one pending patch. Plan to finish before Jan 7th.
>     >
>     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
> challenging to
>     > be done before Jan 2018.
>     >
>     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>     > challenging to be done before Jan 2018.
>     >
>     > * (Varun V) YARN-5673: container-executor write. Given security
>     > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff
> may be
>     > moved to 3.2.
>     >
>     >
>     >
>     > Thanks,
>     >
>     > Wangda
>     >
>     >
>     >
>     >
>     >
>     > On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     > wrote:
>     >
>     > Sure, Wangda.
>     >
>     > Regards,
>     > Uma
>     >
>     >
>     > On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>     >
>     >     Thanks Uma,
>     >
>     >     Could you update this thread once the merge vote started?
>     >
>     >     Best,
>     >     Wangda
>     >
>     >     On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <
>     > uma.gangumalla@intel.com>
>     >     wrote:
>     >
>     >     > HI Wangda,
>     >     >
>     >     >  Thank you for the head-up mail.
>     >     >  We are in the branch (HDFS-10285) and trying to push the tasks
>     > sooner
>     >     > before the deadline.
>     >     >
>     >     > Regards,
>     >     > Uma
>     >     >
>     >     > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com>
> wrote:
>     >     >
>     >     >     Hi All,
>     >     >
>     >     >     Since we're fast approaching previously proposed feature
> freeze
>     > date
>     >     > (Jan
>     >     >     30, about 13 days from today). If you've any features
> which live
>     > in a
>     >     >     branch and targeted to 3.1.0, please reply this email
> thread.
>     > Ideally,
>     >     > we
>     >     >     should finish branch merging before feature freeze date.
>     >     >
>     >     >     Here's an updated 3.1.0 feature status:
>     >     >
>     >     >     1. Merged & Completed features:
>     >     >     * (Sunil) YARN-5881: Support absolute value in
> CapacityScheduler.
>     >     >     * (Wangda) YARN-6223: GPU support on YARN. Features in
> trunk and
>     > works
>     >     >     end-to-end.
>     >     >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN
> native
>     > services.
>     >     >     * (Steve Loughran): HADOOP-13786: S3Guard committer for
>     > zero-rename
>     >     > commits.
>     >     >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto
> Creation
>     > of Leaf
>     >     >     Queues While Doing Queue Mapping.
>     >     >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >     >
>     >     >     2. Features close to finish:
>     >     >     * (Zhankun) YARN-5983: FPGA support. Majority
> implementations
>     >     > completed and
>     >     >     merged to trunk. Except for UI/documentation.
>     >     >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
> done,
>     > some
>     >     >     discussions going on about implementation.
>     >     >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>     > SchedulingRequest and
>     >     >     anti-affinity support. Close to finish, on track to be
> merged
>     > before
>     >     > Jan 30.
>     >     >
>     >     >     3. Tentative features:
>     >     >     * (Arun Suresh). YARN-5972: Support pausing/freezing
>     > opportunistic
>     >     >     containers. Only one pending patch. Plan to finish before
> Jan
>     > 7th.
>     >     >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     > challenging
>     >     > to be
>     >     >     done before Jan 2018.
>     >     >     * (Anu): HDFS-7240: Ozone. Given the discussion on
> HDFS-7240.
>     > Looks
>     >     >     challenging to be done before Jan 2018.
>     >     >     * (Varun V) YARN-5673: container-executor write. Given
> security
>     >     > refactoring
>     >     >     of c-e (YARN-6623) is already landed, IMHO other stuff may
> be
>     > moved to
>     >     > 3.2.
>     >     >
>     >     >     Thanks,
>     >     >     Wangda
>     >     >
>     >     >
>     >     >
>     >     >
>     >     >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <
> wheeleast@gmail.com
>     > >
>     >     > wrote:
>     >     >
>     >     >     > Hi all,
>     >     >     >
>     >     >     > Congratulations on the 3.0.0-GA release!
>     >     >     >
>     >     >     > As we discussed in the previous email thread [1], I'd
> like to
>     > restart
>     >     >     > 3.1.0 release plans.
>     >     >     >
>     >     >     > a) Quick summary:
>     >     >     > a.1 Release status
>     >     >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As
> of
>     > today,
>     >     >     > there’re 232 patches loaded on 3.1.0 alone [2], besides
> 6 open
>     >     > blockers and
>     >     >     > 22 open critical issues.
>     >     >     >
>     >     >     > a.2 Release date update
>     >     >     > Considering delays of 3.0-GA release by
> month-and-a-half, I
>     > propose
>     >     > to
>     >     >     > move the dates as follows
>     >     >     >  - feature freeze date from Dec 15, 2017, to Jan 30,
> 2018 -
>     > last
>     >     > date for
>     >     >     > any branches to get merged too;
>     >     >     >  - code freeze (blockers & critical only) date to Feb
> 08, 2018;
>     >     >     >  - release voting start by Feb 18, 2018, leaving time
> for at
>     > least
>     >     > two RCx
>     >     >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
>     >     >     >
>     >     >     > Unlike before, I added an additional milestone for
>     >     > release-vote-start so
>     >     >     > that we can account for voting time-period also.
>     >     >     >
>     >     >     > This overall is still 5 1/2 months of release-timeline
> unlike
>     > the
>     >     > faster
>     >     >     > cadence we hoped for, but this, in my opinion, is the
>     > best-updated
>     >     > timeline
>     >     >     > given the delays of the final release of 3.0-GA.
>     >     >     >
>     >     >     > b) Individual feature status:
>     >     >     > I spoke to several feature owners and checked the status
> of
>     >     > un-finished
>     >     >     > features, following are status of features planned to
> 3.1.0:
>     >     >     >
>     >     >     > b.1 Merged & Completed features:
>     >     >     > * (Sunil) YARN-5881: Support absolute value in
>     > CapacityScheduler.
>     >     >     > * (Wangda) YARN-6223: GPU support on YARN. Features in
> trunk
>     > and
>     >     > works
>     >     >     > end-to-end.
>     >     >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN
> native
>     >     > services.
>     >     >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for
>     > zero-rename
>     >     >     > commits.
>     >     >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto
> Creation
>     > of
>     >     > Leaf
>     >     >     > Queues While Doing Queue Mapping.
>     >     >     >
>     >     >     > b.2 Features close to finish:
>     >     >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being
> voting
>     > now.
>     >     >     > * (Zhankun) YARN-5983: FPGA support. Majority
> implementations
>     >     > completed
>     >     >     > and merged to trunk. Except for UI/documentation.
>     >     >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations
> are
>     > done, some
>     >     >     > discussions going on about implementation.
>     >     >     >
>     >     >     > b.3 Tentative features:
>     >     >     > * (Arun Suresh). YARN-5972: Support pausing/freezing
>     > opportunistic
>     >     >     > containers. Only one pending patch. Plan to finish
> before Jan
>     > 7th.
>     >     >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     >     > challenging to
>     >     >     > be done before Jan 2018.
>     >     >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>     > SchedulingRequest
>     >     > and
>     >     >     > anti-affinity support. Tentative will figure out by Jan
> 1st.
>     >     >     > * (Anu): HDFS-7240: Ozone. Given the discussion on
> HDFS-7240.
>     > Looks
>     >     >     > challenging to be done before Jan 2018.
>     >     >     > * (Varun V) YARN-5673: container-executor write. Given
> security
>     >     >     > refactoring of c-e (YARN-6623) is already landed, IMHO
> other
>     > stuff
>     >     > may be
>     >     >     > moved to 3.2.
>     >     >     >
>     >     >     > b.4 Additional release drivers
>     >     >     > * More exhaustive upgrade testing from 2.x to 3.x.
>     >     >     >
>     >     >     > c) Regarding branch cut:
>     >     >     >
>     >     >     > We will keep pointing trunk to 3.1 and cut branch-3.1
> until:
>     > A. some
>     >     >     > feature planned to 3.2 has to be landed on trunk or B.
> After
>     > feature
>     >     > freeze
>     >     >     > date, whichever comes first.
>     >     >     >
>     >     >     > I've also talked offline with Vinod to get help on
>     > release-management
>     >     >     > given this is my first release. He agreed to help do this
>     > release
>     >     > jointly.
>     >     >     >
>     >     >     > Thoughts?
>     >     >     >
>     >     >     > Thanks,
>     >     >     > Wangda Tan
>     >     >     >
>     >     >     > [1] https://lists.apache.org/thread.html/
>     >     > c11506c3250c9481852130616b3cb0
>     >     >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org
> %3E
>     >     >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
> fixVersion
>     > in
>     >     > (3.1.0)
>     >     >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority
> DESC”
>     >     >     >
>     >     >     >
>     >     >
>     >     >
>     >     >
>     >
>     >
>     >
>
>
>

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Hi Uma,

Thanks for the update,

As we discussed before, we will delay cutting branch-3.1 until commit has
to go to 3.2 to avoid maintaining two branches. Please let me know when the
merge vote of HDFS-10285 started, I will cut branch-3.1 after that.

Best,
Wangda

On Wed, Jan 31, 2018 at 3:58 AM, Gangumalla, Uma <um...@intel.com>
wrote:

> Hi Wangda,
>
>
>
> Daryn has provided his feedback and we would like to handle some of his
> feedbacks before merge.
>
> That will take couple of more days for us to go through review cycles etc.
>
> Next couple of days I will be in travel, I would not be able to put my
> efforts in it. Meanwhile Rakesh/Surendra will update if any changes in the
> plan.
>
>
>
> So, to conclude, you can proceed cutting the branch and we will merge
> after reviews closed, that may go into 3.2.
>
>
>
> Regards,
>
> Uma
>
>
>
> *From: *Wangda Tan <wh...@gmail.com>
> *Date: *Tuesday, January 30, 2018 at 6:24 AM
> *To: *Uma Gangumalla <um...@intel.com>
> *Cc: *"hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "
> common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod
> Kumar Vavilapalli <vi...@hortonworks.com>
> *Subject: *Re: Hadoop 3.1.0 release discussion
>
>
>
> Thanks for the update.
>
>
>
> On Tue, Jan 30, 2018 at 4:51 PM, Gangumalla, Uma <um...@intel.com>
> wrote:
>
> Hi Wangda,
>
> Sorry that we have not started vote on 29th. Daryn is reviewing the branch
> and he needs a day for his finalized review, then we have to prioritize the
> comments and decide.
> Will keep updated here.
>
> Regards,
> Uma
>
>
> On 1/28/18, 5:31 PM, "Wangda Tan" <wh...@gmail.com> wrote:
>
>     Hi Uma,
>
>     Thanks, I saw HDFS-13050 has been resolved 4 hours ago, I don't see any
>     other blockers under HDFS-10285. I think you guys should be able to
> start
>     voting thread in time for merging to trunk.
>
>     - Wangda
>
>     On Mon, Jan 29, 2018 at 3:12 AM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     wrote:
>
>     > Hi Wangda,
>     >
>     >
>     >
>     > Sorry for the delay.
>     >
>     >
>     >
>     > >>* (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers:
>     > HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the
> two
>     > JIRAs?
>     >
>     > We have only one blocker now that is HDFS-13050 and we finished the
> key
>     > implementation from HDFS-12995, by HDFS-13075.
>     >
>     > We are planning to start vote by tomorrow (29th PST time). So, we
> request
>     > you to give us time for running vote. We will keep SPS off by
> default. So,
>     > interested users only can enable explicitly.
>     >
>     >
>     >
>     > Regards,
>     >
>     > Uma
>     >
>     >
>     >
>     > *From: *Wangda Tan <wh...@gmail.com>
>     > *Date: *Friday, January 26, 2018 at 5:21 PM
>     > *To: *Uma Gangumalla <um...@intel.com>
>     > *Cc: *"hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
>     > yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "
>     > common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
>     > mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod
>     > Kumar Vavilapalli <vi...@hortonworks.com>
>     > *Subject: *Re: Hadoop 3.1.0 release discussion
>
>     >
>     >
>     >
>     > Hi All,
>     >
>     >
>     >
>     > Just a reminder about feature freeze date.
>     >
>     >
>     >
>     > Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days
> from
>     > today), If you've any features which live in a branch and targeted to
>     > 3.1.0, please reply this email thread. Ideally, we should finish
> branch
>     > merging before feature freeze date.
>     >
>     >
>     >
>     > Here's an updated 3.1.0 feature status:
>     >
>     >
>     >
>     > 1. Merged features:
>     >
>     > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     >
>     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
> works
>     > end-to-end.
>     >
>     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
>     >
>     > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>     > commits.
>     >
>     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of
> Leaf
>     > Queues While Doing Queue Mapping.
>     >
>     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >
>     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
> completed
>     > and merged to trunk. Except for UI/documentation.
>     >
>     >
>     >
>     > 2. Features close to finish:
>     >
>     > * (Uma) HDFS-10285: HDFS SPS. There're two remaining
>     > blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the
> ETA of
>     > the two JIRAs?
>     >
>     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest
> and
>     > anti-affinity support. (Voting thread started).
>     >
>     >
>     >
>     > 3. Tentative features:
>     >
>     > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>     > containers. Only one pending patch. Plan to finish before Jan 7th.
>     >
>     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
> challenging to
>     > be done before Jan 2018.
>     >
>     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>     > challenging to be done before Jan 2018.
>     >
>     > * (Varun V) YARN-5673: container-executor write. Given security
>     > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff
> may be
>     > moved to 3.2.
>     >
>     >
>     >
>     > Thanks,
>     >
>     > Wangda
>     >
>     >
>     >
>     >
>     >
>     > On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     > wrote:
>     >
>     > Sure, Wangda.
>     >
>     > Regards,
>     > Uma
>     >
>     >
>     > On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>     >
>     >     Thanks Uma,
>     >
>     >     Could you update this thread once the merge vote started?
>     >
>     >     Best,
>     >     Wangda
>     >
>     >     On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <
>     > uma.gangumalla@intel.com>
>     >     wrote:
>     >
>     >     > HI Wangda,
>     >     >
>     >     >  Thank you for the head-up mail.
>     >     >  We are in the branch (HDFS-10285) and trying to push the tasks
>     > sooner
>     >     > before the deadline.
>     >     >
>     >     > Regards,
>     >     > Uma
>     >     >
>     >     > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com>
> wrote:
>     >     >
>     >     >     Hi All,
>     >     >
>     >     >     Since we're fast approaching previously proposed feature
> freeze
>     > date
>     >     > (Jan
>     >     >     30, about 13 days from today). If you've any features
> which live
>     > in a
>     >     >     branch and targeted to 3.1.0, please reply this email
> thread.
>     > Ideally,
>     >     > we
>     >     >     should finish branch merging before feature freeze date.
>     >     >
>     >     >     Here's an updated 3.1.0 feature status:
>     >     >
>     >     >     1. Merged & Completed features:
>     >     >     * (Sunil) YARN-5881: Support absolute value in
> CapacityScheduler.
>     >     >     * (Wangda) YARN-6223: GPU support on YARN. Features in
> trunk and
>     > works
>     >     >     end-to-end.
>     >     >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN
> native
>     > services.
>     >     >     * (Steve Loughran): HADOOP-13786: S3Guard committer for
>     > zero-rename
>     >     > commits.
>     >     >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto
> Creation
>     > of Leaf
>     >     >     Queues While Doing Queue Mapping.
>     >     >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >     >
>     >     >     2. Features close to finish:
>     >     >     * (Zhankun) YARN-5983: FPGA support. Majority
> implementations
>     >     > completed and
>     >     >     merged to trunk. Except for UI/documentation.
>     >     >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
> done,
>     > some
>     >     >     discussions going on about implementation.
>     >     >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>     > SchedulingRequest and
>     >     >     anti-affinity support. Close to finish, on track to be
> merged
>     > before
>     >     > Jan 30.
>     >     >
>     >     >     3. Tentative features:
>     >     >     * (Arun Suresh). YARN-5972: Support pausing/freezing
>     > opportunistic
>     >     >     containers. Only one pending patch. Plan to finish before
> Jan
>     > 7th.
>     >     >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     > challenging
>     >     > to be
>     >     >     done before Jan 2018.
>     >     >     * (Anu): HDFS-7240: Ozone. Given the discussion on
> HDFS-7240.
>     > Looks
>     >     >     challenging to be done before Jan 2018.
>     >     >     * (Varun V) YARN-5673: container-executor write. Given
> security
>     >     > refactoring
>     >     >     of c-e (YARN-6623) is already landed, IMHO other stuff may
> be
>     > moved to
>     >     > 3.2.
>     >     >
>     >     >     Thanks,
>     >     >     Wangda
>     >     >
>     >     >
>     >     >
>     >     >
>     >     >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <
> wheeleast@gmail.com
>     > >
>     >     > wrote:
>     >     >
>     >     >     > Hi all,
>     >     >     >
>     >     >     > Congratulations on the 3.0.0-GA release!
>     >     >     >
>     >     >     > As we discussed in the previous email thread [1], I'd
> like to
>     > restart
>     >     >     > 3.1.0 release plans.
>     >     >     >
>     >     >     > a) Quick summary:
>     >     >     > a.1 Release status
>     >     >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As
> of
>     > today,
>     >     >     > there’re 232 patches loaded on 3.1.0 alone [2], besides
> 6 open
>     >     > blockers and
>     >     >     > 22 open critical issues.
>     >     >     >
>     >     >     > a.2 Release date update
>     >     >     > Considering delays of 3.0-GA release by
> month-and-a-half, I
>     > propose
>     >     > to
>     >     >     > move the dates as follows
>     >     >     >  - feature freeze date from Dec 15, 2017, to Jan 30,
> 2018 -
>     > last
>     >     > date for
>     >     >     > any branches to get merged too;
>     >     >     >  - code freeze (blockers & critical only) date to Feb
> 08, 2018;
>     >     >     >  - release voting start by Feb 18, 2018, leaving time
> for at
>     > least
>     >     > two RCx
>     >     >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
>     >     >     >
>     >     >     > Unlike before, I added an additional milestone for
>     >     > release-vote-start so
>     >     >     > that we can account for voting time-period also.
>     >     >     >
>     >     >     > This overall is still 5 1/2 months of release-timeline
> unlike
>     > the
>     >     > faster
>     >     >     > cadence we hoped for, but this, in my opinion, is the
>     > best-updated
>     >     > timeline
>     >     >     > given the delays of the final release of 3.0-GA.
>     >     >     >
>     >     >     > b) Individual feature status:
>     >     >     > I spoke to several feature owners and checked the status
> of
>     >     > un-finished
>     >     >     > features, following are status of features planned to
> 3.1.0:
>     >     >     >
>     >     >     > b.1 Merged & Completed features:
>     >     >     > * (Sunil) YARN-5881: Support absolute value in
>     > CapacityScheduler.
>     >     >     > * (Wangda) YARN-6223: GPU support on YARN. Features in
> trunk
>     > and
>     >     > works
>     >     >     > end-to-end.
>     >     >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN
> native
>     >     > services.
>     >     >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for
>     > zero-rename
>     >     >     > commits.
>     >     >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto
> Creation
>     > of
>     >     > Leaf
>     >     >     > Queues While Doing Queue Mapping.
>     >     >     >
>     >     >     > b.2 Features close to finish:
>     >     >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being
> voting
>     > now.
>     >     >     > * (Zhankun) YARN-5983: FPGA support. Majority
> implementations
>     >     > completed
>     >     >     > and merged to trunk. Except for UI/documentation.
>     >     >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations
> are
>     > done, some
>     >     >     > discussions going on about implementation.
>     >     >     >
>     >     >     > b.3 Tentative features:
>     >     >     > * (Arun Suresh). YARN-5972: Support pausing/freezing
>     > opportunistic
>     >     >     > containers. Only one pending patch. Plan to finish
> before Jan
>     > 7th.
>     >     >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     >     > challenging to
>     >     >     > be done before Jan 2018.
>     >     >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>     > SchedulingRequest
>     >     > and
>     >     >     > anti-affinity support. Tentative will figure out by Jan
> 1st.
>     >     >     > * (Anu): HDFS-7240: Ozone. Given the discussion on
> HDFS-7240.
>     > Looks
>     >     >     > challenging to be done before Jan 2018.
>     >     >     > * (Varun V) YARN-5673: container-executor write. Given
> security
>     >     >     > refactoring of c-e (YARN-6623) is already landed, IMHO
> other
>     > stuff
>     >     > may be
>     >     >     > moved to 3.2.
>     >     >     >
>     >     >     > b.4 Additional release drivers
>     >     >     > * More exhaustive upgrade testing from 2.x to 3.x.
>     >     >     >
>     >     >     > c) Regarding branch cut:
>     >     >     >
>     >     >     > We will keep pointing trunk to 3.1 and cut branch-3.1
> until:
>     > A. some
>     >     >     > feature planned to 3.2 has to be landed on trunk or B.
> After
>     > feature
>     >     > freeze
>     >     >     > date, whichever comes first.
>     >     >     >
>     >     >     > I've also talked offline with Vinod to get help on
>     > release-management
>     >     >     > given this is my first release. He agreed to help do this
>     > release
>     >     > jointly.
>     >     >     >
>     >     >     > Thoughts?
>     >     >     >
>     >     >     > Thanks,
>     >     >     > Wangda Tan
>     >     >     >
>     >     >     > [1] https://lists.apache.org/thread.html/
>     >     > c11506c3250c9481852130616b3cb0
>     >     >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org
> %3E
>     >     >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
> fixVersion
>     > in
>     >     > (3.1.0)
>     >     >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority
> DESC”
>     >     >     >
>     >     >     >
>     >     >
>     >     >
>     >     >
>     >
>     >
>     >
>
>
>

Re: Hadoop 3.1.0 release discussion

["Gangumalla, Uma" <um...@intel.com>]

Hi Wangda,

Daryn has provided his feedback and we would like to handle some of his feedbacks before merge.
That will take couple of more days for us to go through review cycles etc.
Next couple of days I will be in travel, I would not be able to put my efforts in it. Meanwhile Rakesh/Surendra will update if any changes in the plan.

So, to conclude, you can proceed cutting the branch and we will merge after reviews closed, that may go into 3.2.

Regards,
Uma

From: Wangda Tan <wh...@gmail.com>
Date: Tuesday, January 30, 2018 at 6:24 AM
To: Uma Gangumalla <um...@intel.com>
Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod Kumar Vavilapalli <vi...@hortonworks.com>
Subject: Re: Hadoop 3.1.0 release discussion

Thanks for the update.

On Tue, Jan 30, 2018 at 4:51 PM, Gangumalla, Uma <um...@intel.com>> wrote:
Hi Wangda,

Sorry that we have not started vote on 29th. Daryn is reviewing the branch and he needs a day for his finalized review, then we have to prioritize the comments and decide.
Will keep updated here.

Regards,
Uma


On 1/28/18, 5:31 PM, "Wangda Tan" <wh...@gmail.com>> wrote:

    Hi Uma,

    Thanks, I saw HDFS-13050 has been resolved 4 hours ago, I don't see any
    other blockers under HDFS-10285. I think you guys should be able to start
    voting thread in time for merging to trunk.

    - Wangda

    On Mon, Jan 29, 2018 at 3:12 AM, Gangumalla, Uma <um...@intel.com>>
    wrote:

    > Hi Wangda,
    >
    >
    >
    > Sorry for the delay.
    >
    >
    >
    > >>* (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers:
    > HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the two
    > JIRAs?
    >
    > We have only one blocker now that is HDFS-13050 and we finished the key
    > implementation from HDFS-12995, by HDFS-13075.
    >
    > We are planning to start vote by tomorrow (29th PST time). So, we request
    > you to give us time for running vote. We will keep SPS off by default. So,
    > interested users only can enable explicitly.
    >
    >
    >
    > Regards,
    >
    > Uma
    >
    >
    >
    > *From: *Wangda Tan <wh...@gmail.com>>
    > *Date: *Friday, January 26, 2018 at 5:21 PM
    > *To: *Uma Gangumalla <um...@intel.com>>
    > *Cc: *"hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <hd...@hadoop.apache.org>>, "
    > yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <ya...@hadoop.apache.org>>, "
    > common-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <co...@hadoop.apache.org>>, "
    > mapreduce-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <ma...@hadoop.apache.org>>, Vinod
    > Kumar Vavilapalli <vi...@hortonworks.com>>
    > *Subject: *Re: Hadoop 3.1.0 release discussion
    >
    >
    >
    > Hi All,
    >
    >
    >
    > Just a reminder about feature freeze date.
    >
    >
    >
    > Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days from
    > today), If you've any features which live in a branch and targeted to
    > 3.1.0, please reply this email thread. Ideally, we should finish branch
    > merging before feature freeze date.
    >
    >
    >
    > Here's an updated 3.1.0 feature status:
    >
    >
    >
    > 1. Merged features:
    >
    > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
    >
    > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
    > end-to-end.
    >
    > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
    >
    > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
    > commits.
    >
    > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
    > Queues While Doing Queue Mapping.
    >
    > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
    >
    > * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
    > and merged to trunk. Except for UI/documentation.
    >
    >
    >
    > 2. Features close to finish:
    >
    > * (Uma) HDFS-10285: HDFS SPS. There're two remaining
    > blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of
    > the two JIRAs?
    >
    > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
    > anti-affinity support. (Voting thread started).
    >
    >
    >
    > 3. Tentative features:
    >
    > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
    > containers. Only one pending patch. Plan to finish before Jan 7th.
    >
    > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
    > be done before Jan 2018.
    >
    > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
    > challenging to be done before Jan 2018.
    >
    > * (Varun V) YARN-5673: container-executor write. Given security
    > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may be
    > moved to 3.2.
    >
    >
    >
    > Thanks,
    >
    > Wangda
    >
    >
    >
    >
    >
    > On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma <um...@intel.com>>
    > wrote:
    >
    > Sure, Wangda.
    >
    > Regards,
    > Uma
    >
    >
    > On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com>> wrote:
    >
    >     Thanks Uma,
    >
    >     Could you update this thread once the merge vote started?
    >
    >     Best,
    >     Wangda
    >
    >     On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <
    > uma.gangumalla@intel.com<ma...@intel.com>>
    >     wrote:
    >
    >     > HI Wangda,
    >     >
    >     >  Thank you for the head-up mail.
    >     >  We are in the branch (HDFS-10285) and trying to push the tasks
    > sooner
    >     > before the deadline.
    >     >
    >     > Regards,
    >     > Uma
    >     >
    >     > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com>> wrote:
    >     >
    >     >     Hi All,
    >     >
    >     >     Since we're fast approaching previously proposed feature freeze
    > date
    >     > (Jan
    >     >     30, about 13 days from today). If you've any features which live
    > in a
    >     >     branch and targeted to 3.1.0, please reply this email thread.
    > Ideally,
    >     > we
    >     >     should finish branch merging before feature freeze date.
    >     >
    >     >     Here's an updated 3.1.0 feature status:
    >     >
    >     >     1. Merged & Completed features:
    >     >     * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
    >     >     * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
    > works
    >     >     end-to-end.
    >     >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
    > services.
    >     >     * (Steve Loughran): HADOOP-13786: S3Guard committer for
    > zero-rename
    >     > commits.
    >     >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
    > of Leaf
    >     >     Queues While Doing Queue Mapping.
    >     >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
    >     >
    >     >     2. Features close to finish:
    >     >     * (Zhankun) YARN-5983: FPGA support. Majority implementations
    >     > completed and
    >     >     merged to trunk. Except for UI/documentation.
    >     >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done,
    > some
    >     >     discussions going on about implementation.
    >     >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New
    > SchedulingRequest and
    >     >     anti-affinity support. Close to finish, on track to be merged
    > before
    >     > Jan 30.
    >     >
    >     >     3. Tentative features:
    >     >     * (Arun Suresh). YARN-5972: Support pausing/freezing
    > opportunistic
    >     >     containers. Only one pending patch. Plan to finish before Jan
    > 7th.
    >     >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
    > challenging
    >     > to be
    >     >     done before Jan 2018.
    >     >     * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
    > Looks
    >     >     challenging to be done before Jan 2018.
    >     >     * (Varun V) YARN-5673: container-executor write. Given security
    >     > refactoring
    >     >     of c-e (YARN-6623) is already landed, IMHO other stuff may be
    > moved to
    >     > 3.2.
    >     >
    >     >     Thanks,
    >     >     Wangda
    >     >
    >     >
    >     >
    >     >
    >     >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com>
    > >
    >     > wrote:
    >     >
    >     >     > Hi all,
    >     >     >
    >     >     > Congratulations on the 3.0.0-GA release!
    >     >     >
    >     >     > As we discussed in the previous email thread [1], I'd like to
    > restart
    >     >     > 3.1.0 release plans.
    >     >     >
    >     >     > a) Quick summary:
    >     >     > a.1 Release status
    >     >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As of
    > today,
    >     >     > there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
    >     > blockers and
    >     >     > 22 open critical issues.
    >     >     >
    >     >     > a.2 Release date update
    >     >     > Considering delays of 3.0-GA release by month-and-a-half, I
    > propose
    >     > to
    >     >     > move the dates as follows
    >     >     >  - feature freeze date from Dec 15, 2017, to Jan 30, 2018 -
    > last
    >     > date for
    >     >     > any branches to get merged too;
    >     >     >  - code freeze (blockers & critical only) date to Feb 08, 2018;
    >     >     >  - release voting start by Feb 18, 2018, leaving time for at
    > least
    >     > two RCx
    >     >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
    >     >     >
    >     >     > Unlike before, I added an additional milestone for
    >     > release-vote-start so
    >     >     > that we can account for voting time-period also.
    >     >     >
    >     >     > This overall is still 5 1/2 months of release-timeline unlike
    > the
    >     > faster
    >     >     > cadence we hoped for, but this, in my opinion, is the
    > best-updated
    >     > timeline
    >     >     > given the delays of the final release of 3.0-GA.
    >     >     >
    >     >     > b) Individual feature status:
    >     >     > I spoke to several feature owners and checked the status of
    >     > un-finished
    >     >     > features, following are status of features planned to 3.1.0:
    >     >     >
    >     >     > b.1 Merged & Completed features:
    >     >     > * (Sunil) YARN-5881: Support absolute value in
    > CapacityScheduler.
    >     >     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk
    > and
    >     > works
    >     >     > end-to-end.
    >     >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
    >     > services.
    >     >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for
    > zero-rename
    >     >     > commits.
    >     >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
    > of
    >     > Leaf
    >     >     > Queues While Doing Queue Mapping.
    >     >     >
    >     >     > b.2 Features close to finish:
    >     >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting
    > now.
    >     >     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
    >     > completed
    >     >     > and merged to trunk. Except for UI/documentation.
    >     >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
    > done, some
    >     >     > discussions going on about implementation.
    >     >     >
    >     >     > b.3 Tentative features:
    >     >     > * (Arun Suresh). YARN-5972: Support pausing/freezing
    > opportunistic
    >     >     > containers. Only one pending patch. Plan to finish before Jan
    > 7th.
    >     >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
    >     > challenging to
    >     >     > be done before Jan 2018.
    >     >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New
    > SchedulingRequest
    >     > and
    >     >     > anti-affinity support. Tentative will figure out by Jan 1st.
    >     >     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
    > Looks
    >     >     > challenging to be done before Jan 2018.
    >     >     > * (Varun V) YARN-5673: container-executor write. Given security
    >     >     > refactoring of c-e (YARN-6623) is already landed, IMHO other
    > stuff
    >     > may be
    >     >     > moved to 3.2.
    >     >     >
    >     >     > b.4 Additional release drivers
    >     >     > * More exhaustive upgrade testing from 2.x to 3.x.
    >     >     >
    >     >     > c) Regarding branch cut:
    >     >     >
    >     >     > We will keep pointing trunk to 3.1 and cut branch-3.1 until:
    > A. some
    >     >     > feature planned to 3.2 has to be landed on trunk or B. After
    > feature
    >     > freeze
    >     >     > date, whichever comes first.
    >     >     >
    >     >     > I've also talked offline with Vinod to get help on
    > release-management
    >     >     > given this is my first release. He agreed to help do this
    > release
    >     > jointly.
    >     >     >
    >     >     > Thoughts?
    >     >     >
    >     >     > Thanks,
    >     >     > Wangda Tan
    >     >     >
    >     >     > [1] https://lists.apache.org/thread.html/
    >     > c11506c3250c9481852130616b3cb0
    >     >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org<http://3Cyarn-dev.hadoop.apache.org>%3E
    >     >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion
    > in
    >     > (3.1.0)
    >     >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
    >     >     >
    >     >     >
    >     >
    >     >
    >     >
    >
    >
    >

Re: Hadoop 3.1.0 release discussion

["Gangumalla, Uma" <um...@intel.com>]

Hi Wangda,

Daryn has provided his feedback and we would like to handle some of his feedbacks before merge.
That will take couple of more days for us to go through review cycles etc.
Next couple of days I will be in travel, I would not be able to put my efforts in it. Meanwhile Rakesh/Surendra will update if any changes in the plan.

So, to conclude, you can proceed cutting the branch and we will merge after reviews closed, that may go into 3.2.

Regards,
Uma

From: Wangda Tan <wh...@gmail.com>
Date: Tuesday, January 30, 2018 at 6:24 AM
To: Uma Gangumalla <um...@intel.com>
Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod Kumar Vavilapalli <vi...@hortonworks.com>
Subject: Re: Hadoop 3.1.0 release discussion

Thanks for the update.

On Tue, Jan 30, 2018 at 4:51 PM, Gangumalla, Uma <um...@intel.com>> wrote:
Hi Wangda,

Sorry that we have not started vote on 29th. Daryn is reviewing the branch and he needs a day for his finalized review, then we have to prioritize the comments and decide.
Will keep updated here.

Regards,
Uma


On 1/28/18, 5:31 PM, "Wangda Tan" <wh...@gmail.com>> wrote:

    Hi Uma,

    Thanks, I saw HDFS-13050 has been resolved 4 hours ago, I don't see any
    other blockers under HDFS-10285. I think you guys should be able to start
    voting thread in time for merging to trunk.

    - Wangda

    On Mon, Jan 29, 2018 at 3:12 AM, Gangumalla, Uma <um...@intel.com>>
    wrote:

    > Hi Wangda,
    >
    >
    >
    > Sorry for the delay.
    >
    >
    >
    > >>* (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers:
    > HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the two
    > JIRAs?
    >
    > We have only one blocker now that is HDFS-13050 and we finished the key
    > implementation from HDFS-12995, by HDFS-13075.
    >
    > We are planning to start vote by tomorrow (29th PST time). So, we request
    > you to give us time for running vote. We will keep SPS off by default. So,
    > interested users only can enable explicitly.
    >
    >
    >
    > Regards,
    >
    > Uma
    >
    >
    >
    > *From: *Wangda Tan <wh...@gmail.com>>
    > *Date: *Friday, January 26, 2018 at 5:21 PM
    > *To: *Uma Gangumalla <um...@intel.com>>
    > *Cc: *"hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <hd...@hadoop.apache.org>>, "
    > yarn-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <ya...@hadoop.apache.org>>, "
    > common-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <co...@hadoop.apache.org>>, "
    > mapreduce-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <ma...@hadoop.apache.org>>, Vinod
    > Kumar Vavilapalli <vi...@hortonworks.com>>
    > *Subject: *Re: Hadoop 3.1.0 release discussion
    >
    >
    >
    > Hi All,
    >
    >
    >
    > Just a reminder about feature freeze date.
    >
    >
    >
    > Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days from
    > today), If you've any features which live in a branch and targeted to
    > 3.1.0, please reply this email thread. Ideally, we should finish branch
    > merging before feature freeze date.
    >
    >
    >
    > Here's an updated 3.1.0 feature status:
    >
    >
    >
    > 1. Merged features:
    >
    > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
    >
    > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
    > end-to-end.
    >
    > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
    >
    > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
    > commits.
    >
    > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
    > Queues While Doing Queue Mapping.
    >
    > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
    >
    > * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
    > and merged to trunk. Except for UI/documentation.
    >
    >
    >
    > 2. Features close to finish:
    >
    > * (Uma) HDFS-10285: HDFS SPS. There're two remaining
    > blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of
    > the two JIRAs?
    >
    > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
    > anti-affinity support. (Voting thread started).
    >
    >
    >
    > 3. Tentative features:
    >
    > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
    > containers. Only one pending patch. Plan to finish before Jan 7th.
    >
    > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
    > be done before Jan 2018.
    >
    > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
    > challenging to be done before Jan 2018.
    >
    > * (Varun V) YARN-5673: container-executor write. Given security
    > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may be
    > moved to 3.2.
    >
    >
    >
    > Thanks,
    >
    > Wangda
    >
    >
    >
    >
    >
    > On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma <um...@intel.com>>
    > wrote:
    >
    > Sure, Wangda.
    >
    > Regards,
    > Uma
    >
    >
    > On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com>> wrote:
    >
    >     Thanks Uma,
    >
    >     Could you update this thread once the merge vote started?
    >
    >     Best,
    >     Wangda
    >
    >     On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <
    > uma.gangumalla@intel.com<ma...@intel.com>>
    >     wrote:
    >
    >     > HI Wangda,
    >     >
    >     >  Thank you for the head-up mail.
    >     >  We are in the branch (HDFS-10285) and trying to push the tasks
    > sooner
    >     > before the deadline.
    >     >
    >     > Regards,
    >     > Uma
    >     >
    >     > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com>> wrote:
    >     >
    >     >     Hi All,
    >     >
    >     >     Since we're fast approaching previously proposed feature freeze
    > date
    >     > (Jan
    >     >     30, about 13 days from today). If you've any features which live
    > in a
    >     >     branch and targeted to 3.1.0, please reply this email thread.
    > Ideally,
    >     > we
    >     >     should finish branch merging before feature freeze date.
    >     >
    >     >     Here's an updated 3.1.0 feature status:
    >     >
    >     >     1. Merged & Completed features:
    >     >     * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
    >     >     * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
    > works
    >     >     end-to-end.
    >     >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
    > services.
    >     >     * (Steve Loughran): HADOOP-13786: S3Guard committer for
    > zero-rename
    >     > commits.
    >     >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
    > of Leaf
    >     >     Queues While Doing Queue Mapping.
    >     >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
    >     >
    >     >     2. Features close to finish:
    >     >     * (Zhankun) YARN-5983: FPGA support. Majority implementations
    >     > completed and
    >     >     merged to trunk. Except for UI/documentation.
    >     >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done,
    > some
    >     >     discussions going on about implementation.
    >     >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New
    > SchedulingRequest and
    >     >     anti-affinity support. Close to finish, on track to be merged
    > before
    >     > Jan 30.
    >     >
    >     >     3. Tentative features:
    >     >     * (Arun Suresh). YARN-5972: Support pausing/freezing
    > opportunistic
    >     >     containers. Only one pending patch. Plan to finish before Jan
    > 7th.
    >     >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
    > challenging
    >     > to be
    >     >     done before Jan 2018.
    >     >     * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
    > Looks
    >     >     challenging to be done before Jan 2018.
    >     >     * (Varun V) YARN-5673: container-executor write. Given security
    >     > refactoring
    >     >     of c-e (YARN-6623) is already landed, IMHO other stuff may be
    > moved to
    >     > 3.2.
    >     >
    >     >     Thanks,
    >     >     Wangda
    >     >
    >     >
    >     >
    >     >
    >     >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com>
    > >
    >     > wrote:
    >     >
    >     >     > Hi all,
    >     >     >
    >     >     > Congratulations on the 3.0.0-GA release!
    >     >     >
    >     >     > As we discussed in the previous email thread [1], I'd like to
    > restart
    >     >     > 3.1.0 release plans.
    >     >     >
    >     >     > a) Quick summary:
    >     >     > a.1 Release status
    >     >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As of
    > today,
    >     >     > there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
    >     > blockers and
    >     >     > 22 open critical issues.
    >     >     >
    >     >     > a.2 Release date update
    >     >     > Considering delays of 3.0-GA release by month-and-a-half, I
    > propose
    >     > to
    >     >     > move the dates as follows
    >     >     >  - feature freeze date from Dec 15, 2017, to Jan 30, 2018 -
    > last
    >     > date for
    >     >     > any branches to get merged too;
    >     >     >  - code freeze (blockers & critical only) date to Feb 08, 2018;
    >     >     >  - release voting start by Feb 18, 2018, leaving time for at
    > least
    >     > two RCx
    >     >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
    >     >     >
    >     >     > Unlike before, I added an additional milestone for
    >     > release-vote-start so
    >     >     > that we can account for voting time-period also.
    >     >     >
    >     >     > This overall is still 5 1/2 months of release-timeline unlike
    > the
    >     > faster
    >     >     > cadence we hoped for, but this, in my opinion, is the
    > best-updated
    >     > timeline
    >     >     > given the delays of the final release of 3.0-GA.
    >     >     >
    >     >     > b) Individual feature status:
    >     >     > I spoke to several feature owners and checked the status of
    >     > un-finished
    >     >     > features, following are status of features planned to 3.1.0:
    >     >     >
    >     >     > b.1 Merged & Completed features:
    >     >     > * (Sunil) YARN-5881: Support absolute value in
    > CapacityScheduler.
    >     >     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk
    > and
    >     > works
    >     >     > end-to-end.
    >     >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
    >     > services.
    >     >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for
    > zero-rename
    >     >     > commits.
    >     >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
    > of
    >     > Leaf
    >     >     > Queues While Doing Queue Mapping.
    >     >     >
    >     >     > b.2 Features close to finish:
    >     >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting
    > now.
    >     >     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
    >     > completed
    >     >     > and merged to trunk. Except for UI/documentation.
    >     >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
    > done, some
    >     >     > discussions going on about implementation.
    >     >     >
    >     >     > b.3 Tentative features:
    >     >     > * (Arun Suresh). YARN-5972: Support pausing/freezing
    > opportunistic
    >     >     > containers. Only one pending patch. Plan to finish before Jan
    > 7th.
    >     >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
    >     > challenging to
    >     >     > be done before Jan 2018.
    >     >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New
    > SchedulingRequest
    >     > and
    >     >     > anti-affinity support. Tentative will figure out by Jan 1st.
    >     >     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
    > Looks
    >     >     > challenging to be done before Jan 2018.
    >     >     > * (Varun V) YARN-5673: container-executor write. Given security
    >     >     > refactoring of c-e (YARN-6623) is already landed, IMHO other
    > stuff
    >     > may be
    >     >     > moved to 3.2.
    >     >     >
    >     >     > b.4 Additional release drivers
    >     >     > * More exhaustive upgrade testing from 2.x to 3.x.
    >     >     >
    >     >     > c) Regarding branch cut:
    >     >     >
    >     >     > We will keep pointing trunk to 3.1 and cut branch-3.1 until:
    > A. some
    >     >     > feature planned to 3.2 has to be landed on trunk or B. After
    > feature
    >     > freeze
    >     >     > date, whichever comes first.
    >     >     >
    >     >     > I've also talked offline with Vinod to get help on
    > release-management
    >     >     > given this is my first release. He agreed to help do this
    > release
    >     > jointly.
    >     >     >
    >     >     > Thoughts?
    >     >     >
    >     >     > Thanks,
    >     >     > Wangda Tan
    >     >     >
    >     >     > [1] https://lists.apache.org/thread.html/
    >     > c11506c3250c9481852130616b3cb0
    >     >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org<http://3Cyarn-dev.hadoop.apache.org>%3E
    >     >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion
    > in
    >     > (3.1.0)
    >     >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
    >     >     >
    >     >     >
    >     >
    >     >
    >     >
    >
    >
    >

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Thanks for the update.

On Tue, Jan 30, 2018 at 4:51 PM, Gangumalla, Uma <um...@intel.com>
wrote:

> Hi Wangda,
>
> Sorry that we have not started vote on 29th. Daryn is reviewing the branch
> and he needs a day for his finalized review, then we have to prioritize the
> comments and decide.
> Will keep updated here.
>
> Regards,
> Uma
>
>
> On 1/28/18, 5:31 PM, "Wangda Tan" <wh...@gmail.com> wrote:
>
>     Hi Uma,
>
>     Thanks, I saw HDFS-13050 has been resolved 4 hours ago, I don't see any
>     other blockers under HDFS-10285. I think you guys should be able to
> start
>     voting thread in time for merging to trunk.
>
>     - Wangda
>
>     On Mon, Jan 29, 2018 at 3:12 AM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     wrote:
>
>     > Hi Wangda,
>     >
>     >
>     >
>     > Sorry for the delay.
>     >
>     >
>     >
>     > >>* (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers:
>     > HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the
> two
>     > JIRAs?
>     >
>     > We have only one blocker now that is HDFS-13050 and we finished the
> key
>     > implementation from HDFS-12995, by HDFS-13075.
>     >
>     > We are planning to start vote by tomorrow (29th PST time). So, we
> request
>     > you to give us time for running vote. We will keep SPS off by
> default. So,
>     > interested users only can enable explicitly.
>     >
>     >
>     >
>     > Regards,
>     >
>     > Uma
>     >
>     >
>     >
>     > *From: *Wangda Tan <wh...@gmail.com>
>     > *Date: *Friday, January 26, 2018 at 5:21 PM
>     > *To: *Uma Gangumalla <um...@intel.com>
>     > *Cc: *"hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
>     > yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "
>     > common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
>     > mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod
>     > Kumar Vavilapalli <vi...@hortonworks.com>
>     > *Subject: *Re: Hadoop 3.1.0 release discussion
>     >
>     >
>     >
>     > Hi All,
>     >
>     >
>     >
>     > Just a reminder about feature freeze date.
>     >
>     >
>     >
>     > Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days
> from
>     > today), If you've any features which live in a branch and targeted to
>     > 3.1.0, please reply this email thread. Ideally, we should finish
> branch
>     > merging before feature freeze date.
>     >
>     >
>     >
>     > Here's an updated 3.1.0 feature status:
>     >
>     >
>     >
>     > 1. Merged features:
>     >
>     > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     >
>     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
> works
>     > end-to-end.
>     >
>     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
>     >
>     > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>     > commits.
>     >
>     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of
> Leaf
>     > Queues While Doing Queue Mapping.
>     >
>     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >
>     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
> completed
>     > and merged to trunk. Except for UI/documentation.
>     >
>     >
>     >
>     > 2. Features close to finish:
>     >
>     > * (Uma) HDFS-10285: HDFS SPS. There're two remaining
>     > blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the
> ETA of
>     > the two JIRAs?
>     >
>     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest
> and
>     > anti-affinity support. (Voting thread started).
>     >
>     >
>     >
>     > 3. Tentative features:
>     >
>     > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>     > containers. Only one pending patch. Plan to finish before Jan 7th.
>     >
>     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
> challenging to
>     > be done before Jan 2018.
>     >
>     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>     > challenging to be done before Jan 2018.
>     >
>     > * (Varun V) YARN-5673: container-executor write. Given security
>     > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff
> may be
>     > moved to 3.2.
>     >
>     >
>     >
>     > Thanks,
>     >
>     > Wangda
>     >
>     >
>     >
>     >
>     >
>     > On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     > wrote:
>     >
>     > Sure, Wangda.
>     >
>     > Regards,
>     > Uma
>     >
>     >
>     > On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>     >
>     >     Thanks Uma,
>     >
>     >     Could you update this thread once the merge vote started?
>     >
>     >     Best,
>     >     Wangda
>     >
>     >     On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <
>     > uma.gangumalla@intel.com>
>     >     wrote:
>     >
>     >     > HI Wangda,
>     >     >
>     >     >  Thank you for the head-up mail.
>     >     >  We are in the branch (HDFS-10285) and trying to push the tasks
>     > sooner
>     >     > before the deadline.
>     >     >
>     >     > Regards,
>     >     > Uma
>     >     >
>     >     > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com>
> wrote:
>     >     >
>     >     >     Hi All,
>     >     >
>     >     >     Since we're fast approaching previously proposed feature
> freeze
>     > date
>     >     > (Jan
>     >     >     30, about 13 days from today). If you've any features
> which live
>     > in a
>     >     >     branch and targeted to 3.1.0, please reply this email
> thread.
>     > Ideally,
>     >     > we
>     >     >     should finish branch merging before feature freeze date.
>     >     >
>     >     >     Here's an updated 3.1.0 feature status:
>     >     >
>     >     >     1. Merged & Completed features:
>     >     >     * (Sunil) YARN-5881: Support absolute value in
> CapacityScheduler.
>     >     >     * (Wangda) YARN-6223: GPU support on YARN. Features in
> trunk and
>     > works
>     >     >     end-to-end.
>     >     >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN
> native
>     > services.
>     >     >     * (Steve Loughran): HADOOP-13786: S3Guard committer for
>     > zero-rename
>     >     > commits.
>     >     >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto
> Creation
>     > of Leaf
>     >     >     Queues While Doing Queue Mapping.
>     >     >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >     >
>     >     >     2. Features close to finish:
>     >     >     * (Zhankun) YARN-5983: FPGA support. Majority
> implementations
>     >     > completed and
>     >     >     merged to trunk. Except for UI/documentation.
>     >     >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
> done,
>     > some
>     >     >     discussions going on about implementation.
>     >     >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>     > SchedulingRequest and
>     >     >     anti-affinity support. Close to finish, on track to be
> merged
>     > before
>     >     > Jan 30.
>     >     >
>     >     >     3. Tentative features:
>     >     >     * (Arun Suresh). YARN-5972: Support pausing/freezing
>     > opportunistic
>     >     >     containers. Only one pending patch. Plan to finish before
> Jan
>     > 7th.
>     >     >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     > challenging
>     >     > to be
>     >     >     done before Jan 2018.
>     >     >     * (Anu): HDFS-7240: Ozone. Given the discussion on
> HDFS-7240.
>     > Looks
>     >     >     challenging to be done before Jan 2018.
>     >     >     * (Varun V) YARN-5673: container-executor write. Given
> security
>     >     > refactoring
>     >     >     of c-e (YARN-6623) is already landed, IMHO other stuff may
> be
>     > moved to
>     >     > 3.2.
>     >     >
>     >     >     Thanks,
>     >     >     Wangda
>     >     >
>     >     >
>     >     >
>     >     >
>     >     >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <
> wheeleast@gmail.com
>     > >
>     >     > wrote:
>     >     >
>     >     >     > Hi all,
>     >     >     >
>     >     >     > Congratulations on the 3.0.0-GA release!
>     >     >     >
>     >     >     > As we discussed in the previous email thread [1], I'd
> like to
>     > restart
>     >     >     > 3.1.0 release plans.
>     >     >     >
>     >     >     > a) Quick summary:
>     >     >     > a.1 Release status
>     >     >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As
> of
>     > today,
>     >     >     > there’re 232 patches loaded on 3.1.0 alone [2], besides
> 6 open
>     >     > blockers and
>     >     >     > 22 open critical issues.
>     >     >     >
>     >     >     > a.2 Release date update
>     >     >     > Considering delays of 3.0-GA release by
> month-and-a-half, I
>     > propose
>     >     > to
>     >     >     > move the dates as follows
>     >     >     >  - feature freeze date from Dec 15, 2017, to Jan 30,
> 2018 -
>     > last
>     >     > date for
>     >     >     > any branches to get merged too;
>     >     >     >  - code freeze (blockers & critical only) date to Feb
> 08, 2018;
>     >     >     >  - release voting start by Feb 18, 2018, leaving time
> for at
>     > least
>     >     > two RCx
>     >     >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
>     >     >     >
>     >     >     > Unlike before, I added an additional milestone for
>     >     > release-vote-start so
>     >     >     > that we can account for voting time-period also.
>     >     >     >
>     >     >     > This overall is still 5 1/2 months of release-timeline
> unlike
>     > the
>     >     > faster
>     >     >     > cadence we hoped for, but this, in my opinion, is the
>     > best-updated
>     >     > timeline
>     >     >     > given the delays of the final release of 3.0-GA.
>     >     >     >
>     >     >     > b) Individual feature status:
>     >     >     > I spoke to several feature owners and checked the status
> of
>     >     > un-finished
>     >     >     > features, following are status of features planned to
> 3.1.0:
>     >     >     >
>     >     >     > b.1 Merged & Completed features:
>     >     >     > * (Sunil) YARN-5881: Support absolute value in
>     > CapacityScheduler.
>     >     >     > * (Wangda) YARN-6223: GPU support on YARN. Features in
> trunk
>     > and
>     >     > works
>     >     >     > end-to-end.
>     >     >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN
> native
>     >     > services.
>     >     >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for
>     > zero-rename
>     >     >     > commits.
>     >     >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto
> Creation
>     > of
>     >     > Leaf
>     >     >     > Queues While Doing Queue Mapping.
>     >     >     >
>     >     >     > b.2 Features close to finish:
>     >     >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being
> voting
>     > now.
>     >     >     > * (Zhankun) YARN-5983: FPGA support. Majority
> implementations
>     >     > completed
>     >     >     > and merged to trunk. Except for UI/documentation.
>     >     >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations
> are
>     > done, some
>     >     >     > discussions going on about implementation.
>     >     >     >
>     >     >     > b.3 Tentative features:
>     >     >     > * (Arun Suresh). YARN-5972: Support pausing/freezing
>     > opportunistic
>     >     >     > containers. Only one pending patch. Plan to finish
> before Jan
>     > 7th.
>     >     >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     >     > challenging to
>     >     >     > be done before Jan 2018.
>     >     >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>     > SchedulingRequest
>     >     > and
>     >     >     > anti-affinity support. Tentative will figure out by Jan
> 1st.
>     >     >     > * (Anu): HDFS-7240: Ozone. Given the discussion on
> HDFS-7240.
>     > Looks
>     >     >     > challenging to be done before Jan 2018.
>     >     >     > * (Varun V) YARN-5673: container-executor write. Given
> security
>     >     >     > refactoring of c-e (YARN-6623) is already landed, IMHO
> other
>     > stuff
>     >     > may be
>     >     >     > moved to 3.2.
>     >     >     >
>     >     >     > b.4 Additional release drivers
>     >     >     > * More exhaustive upgrade testing from 2.x to 3.x.
>     >     >     >
>     >     >     > c) Regarding branch cut:
>     >     >     >
>     >     >     > We will keep pointing trunk to 3.1 and cut branch-3.1
> until:
>     > A. some
>     >     >     > feature planned to 3.2 has to be landed on trunk or B.
> After
>     > feature
>     >     > freeze
>     >     >     > date, whichever comes first.
>     >     >     >
>     >     >     > I've also talked offline with Vinod to get help on
>     > release-management
>     >     >     > given this is my first release. He agreed to help do this
>     > release
>     >     > jointly.
>     >     >     >
>     >     >     > Thoughts?
>     >     >     >
>     >     >     > Thanks,
>     >     >     > Wangda Tan
>     >     >     >
>     >     >     > [1] https://lists.apache.org/thread.html/
>     >     > c11506c3250c9481852130616b3cb0
>     >     >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org
> %3E
>     >     >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
> fixVersion
>     > in
>     >     > (3.1.0)
>     >     >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority
> DESC”
>     >     >     >
>     >     >     >
>     >     >
>     >     >
>     >     >
>     >
>     >
>     >
>
>
>

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Thanks for the update.

On Tue, Jan 30, 2018 at 4:51 PM, Gangumalla, Uma <um...@intel.com>
wrote:

> Hi Wangda,
>
> Sorry that we have not started vote on 29th. Daryn is reviewing the branch
> and he needs a day for his finalized review, then we have to prioritize the
> comments and decide.
> Will keep updated here.
>
> Regards,
> Uma
>
>
> On 1/28/18, 5:31 PM, "Wangda Tan" <wh...@gmail.com> wrote:
>
>     Hi Uma,
>
>     Thanks, I saw HDFS-13050 has been resolved 4 hours ago, I don't see any
>     other blockers under HDFS-10285. I think you guys should be able to
> start
>     voting thread in time for merging to trunk.
>
>     - Wangda
>
>     On Mon, Jan 29, 2018 at 3:12 AM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     wrote:
>
>     > Hi Wangda,
>     >
>     >
>     >
>     > Sorry for the delay.
>     >
>     >
>     >
>     > >>* (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers:
>     > HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the
> two
>     > JIRAs?
>     >
>     > We have only one blocker now that is HDFS-13050 and we finished the
> key
>     > implementation from HDFS-12995, by HDFS-13075.
>     >
>     > We are planning to start vote by tomorrow (29th PST time). So, we
> request
>     > you to give us time for running vote. We will keep SPS off by
> default. So,
>     > interested users only can enable explicitly.
>     >
>     >
>     >
>     > Regards,
>     >
>     > Uma
>     >
>     >
>     >
>     > *From: *Wangda Tan <wh...@gmail.com>
>     > *Date: *Friday, January 26, 2018 at 5:21 PM
>     > *To: *Uma Gangumalla <um...@intel.com>
>     > *Cc: *"hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
>     > yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "
>     > common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
>     > mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod
>     > Kumar Vavilapalli <vi...@hortonworks.com>
>     > *Subject: *Re: Hadoop 3.1.0 release discussion
>     >
>     >
>     >
>     > Hi All,
>     >
>     >
>     >
>     > Just a reminder about feature freeze date.
>     >
>     >
>     >
>     > Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days
> from
>     > today), If you've any features which live in a branch and targeted to
>     > 3.1.0, please reply this email thread. Ideally, we should finish
> branch
>     > merging before feature freeze date.
>     >
>     >
>     >
>     > Here's an updated 3.1.0 feature status:
>     >
>     >
>     >
>     > 1. Merged features:
>     >
>     > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     >
>     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
> works
>     > end-to-end.
>     >
>     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
>     >
>     > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>     > commits.
>     >
>     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of
> Leaf
>     > Queues While Doing Queue Mapping.
>     >
>     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >
>     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
> completed
>     > and merged to trunk. Except for UI/documentation.
>     >
>     >
>     >
>     > 2. Features close to finish:
>     >
>     > * (Uma) HDFS-10285: HDFS SPS. There're two remaining
>     > blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the
> ETA of
>     > the two JIRAs?
>     >
>     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest
> and
>     > anti-affinity support. (Voting thread started).
>     >
>     >
>     >
>     > 3. Tentative features:
>     >
>     > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>     > containers. Only one pending patch. Plan to finish before Jan 7th.
>     >
>     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
> challenging to
>     > be done before Jan 2018.
>     >
>     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>     > challenging to be done before Jan 2018.
>     >
>     > * (Varun V) YARN-5673: container-executor write. Given security
>     > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff
> may be
>     > moved to 3.2.
>     >
>     >
>     >
>     > Thanks,
>     >
>     > Wangda
>     >
>     >
>     >
>     >
>     >
>     > On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     > wrote:
>     >
>     > Sure, Wangda.
>     >
>     > Regards,
>     > Uma
>     >
>     >
>     > On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>     >
>     >     Thanks Uma,
>     >
>     >     Could you update this thread once the merge vote started?
>     >
>     >     Best,
>     >     Wangda
>     >
>     >     On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <
>     > uma.gangumalla@intel.com>
>     >     wrote:
>     >
>     >     > HI Wangda,
>     >     >
>     >     >  Thank you for the head-up mail.
>     >     >  We are in the branch (HDFS-10285) and trying to push the tasks
>     > sooner
>     >     > before the deadline.
>     >     >
>     >     > Regards,
>     >     > Uma
>     >     >
>     >     > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com>
> wrote:
>     >     >
>     >     >     Hi All,
>     >     >
>     >     >     Since we're fast approaching previously proposed feature
> freeze
>     > date
>     >     > (Jan
>     >     >     30, about 13 days from today). If you've any features
> which live
>     > in a
>     >     >     branch and targeted to 3.1.0, please reply this email
> thread.
>     > Ideally,
>     >     > we
>     >     >     should finish branch merging before feature freeze date.
>     >     >
>     >     >     Here's an updated 3.1.0 feature status:
>     >     >
>     >     >     1. Merged & Completed features:
>     >     >     * (Sunil) YARN-5881: Support absolute value in
> CapacityScheduler.
>     >     >     * (Wangda) YARN-6223: GPU support on YARN. Features in
> trunk and
>     > works
>     >     >     end-to-end.
>     >     >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN
> native
>     > services.
>     >     >     * (Steve Loughran): HADOOP-13786: S3Guard committer for
>     > zero-rename
>     >     > commits.
>     >     >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto
> Creation
>     > of Leaf
>     >     >     Queues While Doing Queue Mapping.
>     >     >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >     >
>     >     >     2. Features close to finish:
>     >     >     * (Zhankun) YARN-5983: FPGA support. Majority
> implementations
>     >     > completed and
>     >     >     merged to trunk. Except for UI/documentation.
>     >     >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
> done,
>     > some
>     >     >     discussions going on about implementation.
>     >     >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>     > SchedulingRequest and
>     >     >     anti-affinity support. Close to finish, on track to be
> merged
>     > before
>     >     > Jan 30.
>     >     >
>     >     >     3. Tentative features:
>     >     >     * (Arun Suresh). YARN-5972: Support pausing/freezing
>     > opportunistic
>     >     >     containers. Only one pending patch. Plan to finish before
> Jan
>     > 7th.
>     >     >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     > challenging
>     >     > to be
>     >     >     done before Jan 2018.
>     >     >     * (Anu): HDFS-7240: Ozone. Given the discussion on
> HDFS-7240.
>     > Looks
>     >     >     challenging to be done before Jan 2018.
>     >     >     * (Varun V) YARN-5673: container-executor write. Given
> security
>     >     > refactoring
>     >     >     of c-e (YARN-6623) is already landed, IMHO other stuff may
> be
>     > moved to
>     >     > 3.2.
>     >     >
>     >     >     Thanks,
>     >     >     Wangda
>     >     >
>     >     >
>     >     >
>     >     >
>     >     >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <
> wheeleast@gmail.com
>     > >
>     >     > wrote:
>     >     >
>     >     >     > Hi all,
>     >     >     >
>     >     >     > Congratulations on the 3.0.0-GA release!
>     >     >     >
>     >     >     > As we discussed in the previous email thread [1], I'd
> like to
>     > restart
>     >     >     > 3.1.0 release plans.
>     >     >     >
>     >     >     > a) Quick summary:
>     >     >     > a.1 Release status
>     >     >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As
> of
>     > today,
>     >     >     > there’re 232 patches loaded on 3.1.0 alone [2], besides
> 6 open
>     >     > blockers and
>     >     >     > 22 open critical issues.
>     >     >     >
>     >     >     > a.2 Release date update
>     >     >     > Considering delays of 3.0-GA release by
> month-and-a-half, I
>     > propose
>     >     > to
>     >     >     > move the dates as follows
>     >     >     >  - feature freeze date from Dec 15, 2017, to Jan 30,
> 2018 -
>     > last
>     >     > date for
>     >     >     > any branches to get merged too;
>     >     >     >  - code freeze (blockers & critical only) date to Feb
> 08, 2018;
>     >     >     >  - release voting start by Feb 18, 2018, leaving time
> for at
>     > least
>     >     > two RCx
>     >     >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
>     >     >     >
>     >     >     > Unlike before, I added an additional milestone for
>     >     > release-vote-start so
>     >     >     > that we can account for voting time-period also.
>     >     >     >
>     >     >     > This overall is still 5 1/2 months of release-timeline
> unlike
>     > the
>     >     > faster
>     >     >     > cadence we hoped for, but this, in my opinion, is the
>     > best-updated
>     >     > timeline
>     >     >     > given the delays of the final release of 3.0-GA.
>     >     >     >
>     >     >     > b) Individual feature status:
>     >     >     > I spoke to several feature owners and checked the status
> of
>     >     > un-finished
>     >     >     > features, following are status of features planned to
> 3.1.0:
>     >     >     >
>     >     >     > b.1 Merged & Completed features:
>     >     >     > * (Sunil) YARN-5881: Support absolute value in
>     > CapacityScheduler.
>     >     >     > * (Wangda) YARN-6223: GPU support on YARN. Features in
> trunk
>     > and
>     >     > works
>     >     >     > end-to-end.
>     >     >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN
> native
>     >     > services.
>     >     >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for
>     > zero-rename
>     >     >     > commits.
>     >     >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto
> Creation
>     > of
>     >     > Leaf
>     >     >     > Queues While Doing Queue Mapping.
>     >     >     >
>     >     >     > b.2 Features close to finish:
>     >     >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being
> voting
>     > now.
>     >     >     > * (Zhankun) YARN-5983: FPGA support. Majority
> implementations
>     >     > completed
>     >     >     > and merged to trunk. Except for UI/documentation.
>     >     >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations
> are
>     > done, some
>     >     >     > discussions going on about implementation.
>     >     >     >
>     >     >     > b.3 Tentative features:
>     >     >     > * (Arun Suresh). YARN-5972: Support pausing/freezing
>     > opportunistic
>     >     >     > containers. Only one pending patch. Plan to finish
> before Jan
>     > 7th.
>     >     >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     >     > challenging to
>     >     >     > be done before Jan 2018.
>     >     >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>     > SchedulingRequest
>     >     > and
>     >     >     > anti-affinity support. Tentative will figure out by Jan
> 1st.
>     >     >     > * (Anu): HDFS-7240: Ozone. Given the discussion on
> HDFS-7240.
>     > Looks
>     >     >     > challenging to be done before Jan 2018.
>     >     >     > * (Varun V) YARN-5673: container-executor write. Given
> security
>     >     >     > refactoring of c-e (YARN-6623) is already landed, IMHO
> other
>     > stuff
>     >     > may be
>     >     >     > moved to 3.2.
>     >     >     >
>     >     >     > b.4 Additional release drivers
>     >     >     > * More exhaustive upgrade testing from 2.x to 3.x.
>     >     >     >
>     >     >     > c) Regarding branch cut:
>     >     >     >
>     >     >     > We will keep pointing trunk to 3.1 and cut branch-3.1
> until:
>     > A. some
>     >     >     > feature planned to 3.2 has to be landed on trunk or B.
> After
>     > feature
>     >     > freeze
>     >     >     > date, whichever comes first.
>     >     >     >
>     >     >     > I've also talked offline with Vinod to get help on
>     > release-management
>     >     >     > given this is my first release. He agreed to help do this
>     > release
>     >     > jointly.
>     >     >     >
>     >     >     > Thoughts?
>     >     >     >
>     >     >     > Thanks,
>     >     >     > Wangda Tan
>     >     >     >
>     >     >     > [1] https://lists.apache.org/thread.html/
>     >     > c11506c3250c9481852130616b3cb0
>     >     >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org
> %3E
>     >     >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
> fixVersion
>     > in
>     >     > (3.1.0)
>     >     >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority
> DESC”
>     >     >     >
>     >     >     >
>     >     >
>     >     >
>     >     >
>     >
>     >
>     >
>
>
>

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Thanks for the update.

On Tue, Jan 30, 2018 at 4:51 PM, Gangumalla, Uma <um...@intel.com>
wrote:

> Hi Wangda,
>
> Sorry that we have not started vote on 29th. Daryn is reviewing the branch
> and he needs a day for his finalized review, then we have to prioritize the
> comments and decide.
> Will keep updated here.
>
> Regards,
> Uma
>
>
> On 1/28/18, 5:31 PM, "Wangda Tan" <wh...@gmail.com> wrote:
>
>     Hi Uma,
>
>     Thanks, I saw HDFS-13050 has been resolved 4 hours ago, I don't see any
>     other blockers under HDFS-10285. I think you guys should be able to
> start
>     voting thread in time for merging to trunk.
>
>     - Wangda
>
>     On Mon, Jan 29, 2018 at 3:12 AM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     wrote:
>
>     > Hi Wangda,
>     >
>     >
>     >
>     > Sorry for the delay.
>     >
>     >
>     >
>     > >>* (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers:
>     > HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the
> two
>     > JIRAs?
>     >
>     > We have only one blocker now that is HDFS-13050 and we finished the
> key
>     > implementation from HDFS-12995, by HDFS-13075.
>     >
>     > We are planning to start vote by tomorrow (29th PST time). So, we
> request
>     > you to give us time for running vote. We will keep SPS off by
> default. So,
>     > interested users only can enable explicitly.
>     >
>     >
>     >
>     > Regards,
>     >
>     > Uma
>     >
>     >
>     >
>     > *From: *Wangda Tan <wh...@gmail.com>
>     > *Date: *Friday, January 26, 2018 at 5:21 PM
>     > *To: *Uma Gangumalla <um...@intel.com>
>     > *Cc: *"hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
>     > yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "
>     > common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
>     > mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod
>     > Kumar Vavilapalli <vi...@hortonworks.com>
>     > *Subject: *Re: Hadoop 3.1.0 release discussion
>     >
>     >
>     >
>     > Hi All,
>     >
>     >
>     >
>     > Just a reminder about feature freeze date.
>     >
>     >
>     >
>     > Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days
> from
>     > today), If you've any features which live in a branch and targeted to
>     > 3.1.0, please reply this email thread. Ideally, we should finish
> branch
>     > merging before feature freeze date.
>     >
>     >
>     >
>     > Here's an updated 3.1.0 feature status:
>     >
>     >
>     >
>     > 1. Merged features:
>     >
>     > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     >
>     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
> works
>     > end-to-end.
>     >
>     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
>     >
>     > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>     > commits.
>     >
>     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of
> Leaf
>     > Queues While Doing Queue Mapping.
>     >
>     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >
>     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
> completed
>     > and merged to trunk. Except for UI/documentation.
>     >
>     >
>     >
>     > 2. Features close to finish:
>     >
>     > * (Uma) HDFS-10285: HDFS SPS. There're two remaining
>     > blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the
> ETA of
>     > the two JIRAs?
>     >
>     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest
> and
>     > anti-affinity support. (Voting thread started).
>     >
>     >
>     >
>     > 3. Tentative features:
>     >
>     > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>     > containers. Only one pending patch. Plan to finish before Jan 7th.
>     >
>     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
> challenging to
>     > be done before Jan 2018.
>     >
>     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>     > challenging to be done before Jan 2018.
>     >
>     > * (Varun V) YARN-5673: container-executor write. Given security
>     > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff
> may be
>     > moved to 3.2.
>     >
>     >
>     >
>     > Thanks,
>     >
>     > Wangda
>     >
>     >
>     >
>     >
>     >
>     > On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     > wrote:
>     >
>     > Sure, Wangda.
>     >
>     > Regards,
>     > Uma
>     >
>     >
>     > On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>     >
>     >     Thanks Uma,
>     >
>     >     Could you update this thread once the merge vote started?
>     >
>     >     Best,
>     >     Wangda
>     >
>     >     On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <
>     > uma.gangumalla@intel.com>
>     >     wrote:
>     >
>     >     > HI Wangda,
>     >     >
>     >     >  Thank you for the head-up mail.
>     >     >  We are in the branch (HDFS-10285) and trying to push the tasks
>     > sooner
>     >     > before the deadline.
>     >     >
>     >     > Regards,
>     >     > Uma
>     >     >
>     >     > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com>
> wrote:
>     >     >
>     >     >     Hi All,
>     >     >
>     >     >     Since we're fast approaching previously proposed feature
> freeze
>     > date
>     >     > (Jan
>     >     >     30, about 13 days from today). If you've any features
> which live
>     > in a
>     >     >     branch and targeted to 3.1.0, please reply this email
> thread.
>     > Ideally,
>     >     > we
>     >     >     should finish branch merging before feature freeze date.
>     >     >
>     >     >     Here's an updated 3.1.0 feature status:
>     >     >
>     >     >     1. Merged & Completed features:
>     >     >     * (Sunil) YARN-5881: Support absolute value in
> CapacityScheduler.
>     >     >     * (Wangda) YARN-6223: GPU support on YARN. Features in
> trunk and
>     > works
>     >     >     end-to-end.
>     >     >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN
> native
>     > services.
>     >     >     * (Steve Loughran): HADOOP-13786: S3Guard committer for
>     > zero-rename
>     >     > commits.
>     >     >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto
> Creation
>     > of Leaf
>     >     >     Queues While Doing Queue Mapping.
>     >     >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >     >
>     >     >     2. Features close to finish:
>     >     >     * (Zhankun) YARN-5983: FPGA support. Majority
> implementations
>     >     > completed and
>     >     >     merged to trunk. Except for UI/documentation.
>     >     >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
> done,
>     > some
>     >     >     discussions going on about implementation.
>     >     >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>     > SchedulingRequest and
>     >     >     anti-affinity support. Close to finish, on track to be
> merged
>     > before
>     >     > Jan 30.
>     >     >
>     >     >     3. Tentative features:
>     >     >     * (Arun Suresh). YARN-5972: Support pausing/freezing
>     > opportunistic
>     >     >     containers. Only one pending patch. Plan to finish before
> Jan
>     > 7th.
>     >     >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     > challenging
>     >     > to be
>     >     >     done before Jan 2018.
>     >     >     * (Anu): HDFS-7240: Ozone. Given the discussion on
> HDFS-7240.
>     > Looks
>     >     >     challenging to be done before Jan 2018.
>     >     >     * (Varun V) YARN-5673: container-executor write. Given
> security
>     >     > refactoring
>     >     >     of c-e (YARN-6623) is already landed, IMHO other stuff may
> be
>     > moved to
>     >     > 3.2.
>     >     >
>     >     >     Thanks,
>     >     >     Wangda
>     >     >
>     >     >
>     >     >
>     >     >
>     >     >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <
> wheeleast@gmail.com
>     > >
>     >     > wrote:
>     >     >
>     >     >     > Hi all,
>     >     >     >
>     >     >     > Congratulations on the 3.0.0-GA release!
>     >     >     >
>     >     >     > As we discussed in the previous email thread [1], I'd
> like to
>     > restart
>     >     >     > 3.1.0 release plans.
>     >     >     >
>     >     >     > a) Quick summary:
>     >     >     > a.1 Release status
>     >     >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As
> of
>     > today,
>     >     >     > there’re 232 patches loaded on 3.1.0 alone [2], besides
> 6 open
>     >     > blockers and
>     >     >     > 22 open critical issues.
>     >     >     >
>     >     >     > a.2 Release date update
>     >     >     > Considering delays of 3.0-GA release by
> month-and-a-half, I
>     > propose
>     >     > to
>     >     >     > move the dates as follows
>     >     >     >  - feature freeze date from Dec 15, 2017, to Jan 30,
> 2018 -
>     > last
>     >     > date for
>     >     >     > any branches to get merged too;
>     >     >     >  - code freeze (blockers & critical only) date to Feb
> 08, 2018;
>     >     >     >  - release voting start by Feb 18, 2018, leaving time
> for at
>     > least
>     >     > two RCx
>     >     >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
>     >     >     >
>     >     >     > Unlike before, I added an additional milestone for
>     >     > release-vote-start so
>     >     >     > that we can account for voting time-period also.
>     >     >     >
>     >     >     > This overall is still 5 1/2 months of release-timeline
> unlike
>     > the
>     >     > faster
>     >     >     > cadence we hoped for, but this, in my opinion, is the
>     > best-updated
>     >     > timeline
>     >     >     > given the delays of the final release of 3.0-GA.
>     >     >     >
>     >     >     > b) Individual feature status:
>     >     >     > I spoke to several feature owners and checked the status
> of
>     >     > un-finished
>     >     >     > features, following are status of features planned to
> 3.1.0:
>     >     >     >
>     >     >     > b.1 Merged & Completed features:
>     >     >     > * (Sunil) YARN-5881: Support absolute value in
>     > CapacityScheduler.
>     >     >     > * (Wangda) YARN-6223: GPU support on YARN. Features in
> trunk
>     > and
>     >     > works
>     >     >     > end-to-end.
>     >     >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN
> native
>     >     > services.
>     >     >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for
>     > zero-rename
>     >     >     > commits.
>     >     >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto
> Creation
>     > of
>     >     > Leaf
>     >     >     > Queues While Doing Queue Mapping.
>     >     >     >
>     >     >     > b.2 Features close to finish:
>     >     >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being
> voting
>     > now.
>     >     >     > * (Zhankun) YARN-5983: FPGA support. Majority
> implementations
>     >     > completed
>     >     >     > and merged to trunk. Except for UI/documentation.
>     >     >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations
> are
>     > done, some
>     >     >     > discussions going on about implementation.
>     >     >     >
>     >     >     > b.3 Tentative features:
>     >     >     > * (Arun Suresh). YARN-5972: Support pausing/freezing
>     > opportunistic
>     >     >     > containers. Only one pending patch. Plan to finish
> before Jan
>     > 7th.
>     >     >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     >     > challenging to
>     >     >     > be done before Jan 2018.
>     >     >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>     > SchedulingRequest
>     >     > and
>     >     >     > anti-affinity support. Tentative will figure out by Jan
> 1st.
>     >     >     > * (Anu): HDFS-7240: Ozone. Given the discussion on
> HDFS-7240.
>     > Looks
>     >     >     > challenging to be done before Jan 2018.
>     >     >     > * (Varun V) YARN-5673: container-executor write. Given
> security
>     >     >     > refactoring of c-e (YARN-6623) is already landed, IMHO
> other
>     > stuff
>     >     > may be
>     >     >     > moved to 3.2.
>     >     >     >
>     >     >     > b.4 Additional release drivers
>     >     >     > * More exhaustive upgrade testing from 2.x to 3.x.
>     >     >     >
>     >     >     > c) Regarding branch cut:
>     >     >     >
>     >     >     > We will keep pointing trunk to 3.1 and cut branch-3.1
> until:
>     > A. some
>     >     >     > feature planned to 3.2 has to be landed on trunk or B.
> After
>     > feature
>     >     > freeze
>     >     >     > date, whichever comes first.
>     >     >     >
>     >     >     > I've also talked offline with Vinod to get help on
>     > release-management
>     >     >     > given this is my first release. He agreed to help do this
>     > release
>     >     > jointly.
>     >     >     >
>     >     >     > Thoughts?
>     >     >     >
>     >     >     > Thanks,
>     >     >     > Wangda Tan
>     >     >     >
>     >     >     > [1] https://lists.apache.org/thread.html/
>     >     > c11506c3250c9481852130616b3cb0
>     >     >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org
> %3E
>     >     >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
> fixVersion
>     > in
>     >     > (3.1.0)
>     >     >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority
> DESC”
>     >     >     >
>     >     >     >
>     >     >
>     >     >
>     >     >
>     >
>     >
>     >
>
>
>

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Thanks for the update.

On Tue, Jan 30, 2018 at 4:51 PM, Gangumalla, Uma <um...@intel.com>
wrote:

> Hi Wangda,
>
> Sorry that we have not started vote on 29th. Daryn is reviewing the branch
> and he needs a day for his finalized review, then we have to prioritize the
> comments and decide.
> Will keep updated here.
>
> Regards,
> Uma
>
>
> On 1/28/18, 5:31 PM, "Wangda Tan" <wh...@gmail.com> wrote:
>
>     Hi Uma,
>
>     Thanks, I saw HDFS-13050 has been resolved 4 hours ago, I don't see any
>     other blockers under HDFS-10285. I think you guys should be able to
> start
>     voting thread in time for merging to trunk.
>
>     - Wangda
>
>     On Mon, Jan 29, 2018 at 3:12 AM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     wrote:
>
>     > Hi Wangda,
>     >
>     >
>     >
>     > Sorry for the delay.
>     >
>     >
>     >
>     > >>* (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers:
>     > HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the
> two
>     > JIRAs?
>     >
>     > We have only one blocker now that is HDFS-13050 and we finished the
> key
>     > implementation from HDFS-12995, by HDFS-13075.
>     >
>     > We are planning to start vote by tomorrow (29th PST time). So, we
> request
>     > you to give us time for running vote. We will keep SPS off by
> default. So,
>     > interested users only can enable explicitly.
>     >
>     >
>     >
>     > Regards,
>     >
>     > Uma
>     >
>     >
>     >
>     > *From: *Wangda Tan <wh...@gmail.com>
>     > *Date: *Friday, January 26, 2018 at 5:21 PM
>     > *To: *Uma Gangumalla <um...@intel.com>
>     > *Cc: *"hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
>     > yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "
>     > common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
>     > mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>,
> Vinod
>     > Kumar Vavilapalli <vi...@hortonworks.com>
>     > *Subject: *Re: Hadoop 3.1.0 release discussion
>     >
>     >
>     >
>     > Hi All,
>     >
>     >
>     >
>     > Just a reminder about feature freeze date.
>     >
>     >
>     >
>     > Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days
> from
>     > today), If you've any features which live in a branch and targeted to
>     > 3.1.0, please reply this email thread. Ideally, we should finish
> branch
>     > merging before feature freeze date.
>     >
>     >
>     >
>     > Here's an updated 3.1.0 feature status:
>     >
>     >
>     >
>     > 1. Merged features:
>     >
>     > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     >
>     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
> works
>     > end-to-end.
>     >
>     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
>     >
>     > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>     > commits.
>     >
>     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of
> Leaf
>     > Queues While Doing Queue Mapping.
>     >
>     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >
>     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
> completed
>     > and merged to trunk. Except for UI/documentation.
>     >
>     >
>     >
>     > 2. Features close to finish:
>     >
>     > * (Uma) HDFS-10285: HDFS SPS. There're two remaining
>     > blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the
> ETA of
>     > the two JIRAs?
>     >
>     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest
> and
>     > anti-affinity support. (Voting thread started).
>     >
>     >
>     >
>     > 3. Tentative features:
>     >
>     > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>     > containers. Only one pending patch. Plan to finish before Jan 7th.
>     >
>     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
> challenging to
>     > be done before Jan 2018.
>     >
>     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>     > challenging to be done before Jan 2018.
>     >
>     > * (Varun V) YARN-5673: container-executor write. Given security
>     > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff
> may be
>     > moved to 3.2.
>     >
>     >
>     >
>     > Thanks,
>     >
>     > Wangda
>     >
>     >
>     >
>     >
>     >
>     > On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     > wrote:
>     >
>     > Sure, Wangda.
>     >
>     > Regards,
>     > Uma
>     >
>     >
>     > On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>     >
>     >     Thanks Uma,
>     >
>     >     Could you update this thread once the merge vote started?
>     >
>     >     Best,
>     >     Wangda
>     >
>     >     On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <
>     > uma.gangumalla@intel.com>
>     >     wrote:
>     >
>     >     > HI Wangda,
>     >     >
>     >     >  Thank you for the head-up mail.
>     >     >  We are in the branch (HDFS-10285) and trying to push the tasks
>     > sooner
>     >     > before the deadline.
>     >     >
>     >     > Regards,
>     >     > Uma
>     >     >
>     >     > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com>
> wrote:
>     >     >
>     >     >     Hi All,
>     >     >
>     >     >     Since we're fast approaching previously proposed feature
> freeze
>     > date
>     >     > (Jan
>     >     >     30, about 13 days from today). If you've any features
> which live
>     > in a
>     >     >     branch and targeted to 3.1.0, please reply this email
> thread.
>     > Ideally,
>     >     > we
>     >     >     should finish branch merging before feature freeze date.
>     >     >
>     >     >     Here's an updated 3.1.0 feature status:
>     >     >
>     >     >     1. Merged & Completed features:
>     >     >     * (Sunil) YARN-5881: Support absolute value in
> CapacityScheduler.
>     >     >     * (Wangda) YARN-6223: GPU support on YARN. Features in
> trunk and
>     > works
>     >     >     end-to-end.
>     >     >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN
> native
>     > services.
>     >     >     * (Steve Loughran): HADOOP-13786: S3Guard committer for
>     > zero-rename
>     >     > commits.
>     >     >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto
> Creation
>     > of Leaf
>     >     >     Queues While Doing Queue Mapping.
>     >     >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >     >
>     >     >     2. Features close to finish:
>     >     >     * (Zhankun) YARN-5983: FPGA support. Majority
> implementations
>     >     > completed and
>     >     >     merged to trunk. Except for UI/documentation.
>     >     >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
> done,
>     > some
>     >     >     discussions going on about implementation.
>     >     >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>     > SchedulingRequest and
>     >     >     anti-affinity support. Close to finish, on track to be
> merged
>     > before
>     >     > Jan 30.
>     >     >
>     >     >     3. Tentative features:
>     >     >     * (Arun Suresh). YARN-5972: Support pausing/freezing
>     > opportunistic
>     >     >     containers. Only one pending patch. Plan to finish before
> Jan
>     > 7th.
>     >     >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     > challenging
>     >     > to be
>     >     >     done before Jan 2018.
>     >     >     * (Anu): HDFS-7240: Ozone. Given the discussion on
> HDFS-7240.
>     > Looks
>     >     >     challenging to be done before Jan 2018.
>     >     >     * (Varun V) YARN-5673: container-executor write. Given
> security
>     >     > refactoring
>     >     >     of c-e (YARN-6623) is already landed, IMHO other stuff may
> be
>     > moved to
>     >     > 3.2.
>     >     >
>     >     >     Thanks,
>     >     >     Wangda
>     >     >
>     >     >
>     >     >
>     >     >
>     >     >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <
> wheeleast@gmail.com
>     > >
>     >     > wrote:
>     >     >
>     >     >     > Hi all,
>     >     >     >
>     >     >     > Congratulations on the 3.0.0-GA release!
>     >     >     >
>     >     >     > As we discussed in the previous email thread [1], I'd
> like to
>     > restart
>     >     >     > 3.1.0 release plans.
>     >     >     >
>     >     >     > a) Quick summary:
>     >     >     > a.1 Release status
>     >     >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As
> of
>     > today,
>     >     >     > there’re 232 patches loaded on 3.1.0 alone [2], besides
> 6 open
>     >     > blockers and
>     >     >     > 22 open critical issues.
>     >     >     >
>     >     >     > a.2 Release date update
>     >     >     > Considering delays of 3.0-GA release by
> month-and-a-half, I
>     > propose
>     >     > to
>     >     >     > move the dates as follows
>     >     >     >  - feature freeze date from Dec 15, 2017, to Jan 30,
> 2018 -
>     > last
>     >     > date for
>     >     >     > any branches to get merged too;
>     >     >     >  - code freeze (blockers & critical only) date to Feb
> 08, 2018;
>     >     >     >  - release voting start by Feb 18, 2018, leaving time
> for at
>     > least
>     >     > two RCx
>     >     >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
>     >     >     >
>     >     >     > Unlike before, I added an additional milestone for
>     >     > release-vote-start so
>     >     >     > that we can account for voting time-period also.
>     >     >     >
>     >     >     > This overall is still 5 1/2 months of release-timeline
> unlike
>     > the
>     >     > faster
>     >     >     > cadence we hoped for, but this, in my opinion, is the
>     > best-updated
>     >     > timeline
>     >     >     > given the delays of the final release of 3.0-GA.
>     >     >     >
>     >     >     > b) Individual feature status:
>     >     >     > I spoke to several feature owners and checked the status
> of
>     >     > un-finished
>     >     >     > features, following are status of features planned to
> 3.1.0:
>     >     >     >
>     >     >     > b.1 Merged & Completed features:
>     >     >     > * (Sunil) YARN-5881: Support absolute value in
>     > CapacityScheduler.
>     >     >     > * (Wangda) YARN-6223: GPU support on YARN. Features in
> trunk
>     > and
>     >     > works
>     >     >     > end-to-end.
>     >     >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN
> native
>     >     > services.
>     >     >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for
>     > zero-rename
>     >     >     > commits.
>     >     >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto
> Creation
>     > of
>     >     > Leaf
>     >     >     > Queues While Doing Queue Mapping.
>     >     >     >
>     >     >     > b.2 Features close to finish:
>     >     >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being
> voting
>     > now.
>     >     >     > * (Zhankun) YARN-5983: FPGA support. Majority
> implementations
>     >     > completed
>     >     >     > and merged to trunk. Except for UI/documentation.
>     >     >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations
> are
>     > done, some
>     >     >     > discussions going on about implementation.
>     >     >     >
>     >     >     > b.3 Tentative features:
>     >     >     > * (Arun Suresh). YARN-5972: Support pausing/freezing
>     > opportunistic
>     >     >     > containers. Only one pending patch. Plan to finish
> before Jan
>     > 7th.
>     >     >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     >     > challenging to
>     >     >     > be done before Jan 2018.
>     >     >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New
>     > SchedulingRequest
>     >     > and
>     >     >     > anti-affinity support. Tentative will figure out by Jan
> 1st.
>     >     >     > * (Anu): HDFS-7240: Ozone. Given the discussion on
> HDFS-7240.
>     > Looks
>     >     >     > challenging to be done before Jan 2018.
>     >     >     > * (Varun V) YARN-5673: container-executor write. Given
> security
>     >     >     > refactoring of c-e (YARN-6623) is already landed, IMHO
> other
>     > stuff
>     >     > may be
>     >     >     > moved to 3.2.
>     >     >     >
>     >     >     > b.4 Additional release drivers
>     >     >     > * More exhaustive upgrade testing from 2.x to 3.x.
>     >     >     >
>     >     >     > c) Regarding branch cut:
>     >     >     >
>     >     >     > We will keep pointing trunk to 3.1 and cut branch-3.1
> until:
>     > A. some
>     >     >     > feature planned to 3.2 has to be landed on trunk or B.
> After
>     > feature
>     >     > freeze
>     >     >     > date, whichever comes first.
>     >     >     >
>     >     >     > I've also talked offline with Vinod to get help on
>     > release-management
>     >     >     > given this is my first release. He agreed to help do this
>     > release
>     >     > jointly.
>     >     >     >
>     >     >     > Thoughts?
>     >     >     >
>     >     >     > Thanks,
>     >     >     > Wangda Tan
>     >     >     >
>     >     >     > [1] https://lists.apache.org/thread.html/
>     >     > c11506c3250c9481852130616b3cb0
>     >     >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org
> %3E
>     >     >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND
> fixVersion
>     > in
>     >     > (3.1.0)
>     >     >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority
> DESC”
>     >     >     >
>     >     >     >
>     >     >
>     >     >
>     >     >
>     >
>     >
>     >
>
>
>

Re: Hadoop 3.1.0 release discussion

["Gangumalla, Uma" <um...@intel.com>]

Hi Wangda,

Sorry that we have not started vote on 29th. Daryn is reviewing the branch and he needs a day for his finalized review, then we have to prioritize the comments and decide.
Will keep updated here.

Regards,
Uma


On 1/28/18, 5:31 PM, "Wangda Tan" <wh...@gmail.com> wrote:

    Hi Uma,
    
    Thanks, I saw HDFS-13050 has been resolved 4 hours ago, I don't see any
    other blockers under HDFS-10285. I think you guys should be able to start
    voting thread in time for merging to trunk.
    
    - Wangda
    
    On Mon, Jan 29, 2018 at 3:12 AM, Gangumalla, Uma <um...@intel.com>
    wrote:
    
    > Hi Wangda,
    >
    >
    >
    > Sorry for the delay.
    >
    >
    >
    > >>* (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers:
    > HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the two
    > JIRAs?
    >
    > We have only one blocker now that is HDFS-13050 and we finished the key
    > implementation from HDFS-12995, by HDFS-13075.
    >
    > We are planning to start vote by tomorrow (29th PST time). So, we request
    > you to give us time for running vote. We will keep SPS off by default. So,
    > interested users only can enable explicitly.
    >
    >
    >
    > Regards,
    >
    > Uma
    >
    >
    >
    > *From: *Wangda Tan <wh...@gmail.com>
    > *Date: *Friday, January 26, 2018 at 5:21 PM
    > *To: *Uma Gangumalla <um...@intel.com>
    > *Cc: *"hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
    > yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "
    > common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
    > mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod
    > Kumar Vavilapalli <vi...@hortonworks.com>
    > *Subject: *Re: Hadoop 3.1.0 release discussion
    >
    >
    >
    > Hi All,
    >
    >
    >
    > Just a reminder about feature freeze date.
    >
    >
    >
    > Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days from
    > today), If you've any features which live in a branch and targeted to
    > 3.1.0, please reply this email thread. Ideally, we should finish branch
    > merging before feature freeze date.
    >
    >
    >
    > Here's an updated 3.1.0 feature status:
    >
    >
    >
    > 1. Merged features:
    >
    > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
    >
    > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
    > end-to-end.
    >
    > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
    >
    > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
    > commits.
    >
    > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
    > Queues While Doing Queue Mapping.
    >
    > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
    >
    > * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
    > and merged to trunk. Except for UI/documentation.
    >
    >
    >
    > 2. Features close to finish:
    >
    > * (Uma) HDFS-10285: HDFS SPS. There're two remaining
    > blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of
    > the two JIRAs?
    >
    > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
    > anti-affinity support. (Voting thread started).
    >
    >
    >
    > 3. Tentative features:
    >
    > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
    > containers. Only one pending patch. Plan to finish before Jan 7th.
    >
    > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
    > be done before Jan 2018.
    >
    > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
    > challenging to be done before Jan 2018.
    >
    > * (Varun V) YARN-5673: container-executor write. Given security
    > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may be
    > moved to 3.2.
    >
    >
    >
    > Thanks,
    >
    > Wangda
    >
    >
    >
    >
    >
    > On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma <um...@intel.com>
    > wrote:
    >
    > Sure, Wangda.
    >
    > Regards,
    > Uma
    >
    >
    > On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com> wrote:
    >
    >     Thanks Uma,
    >
    >     Could you update this thread once the merge vote started?
    >
    >     Best,
    >     Wangda
    >
    >     On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <
    > uma.gangumalla@intel.com>
    >     wrote:
    >
    >     > HI Wangda,
    >     >
    >     >  Thank you for the head-up mail.
    >     >  We are in the branch (HDFS-10285) and trying to push the tasks
    > sooner
    >     > before the deadline.
    >     >
    >     > Regards,
    >     > Uma
    >     >
    >     > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com> wrote:
    >     >
    >     >     Hi All,
    >     >
    >     >     Since we're fast approaching previously proposed feature freeze
    > date
    >     > (Jan
    >     >     30, about 13 days from today). If you've any features which live
    > in a
    >     >     branch and targeted to 3.1.0, please reply this email thread.
    > Ideally,
    >     > we
    >     >     should finish branch merging before feature freeze date.
    >     >
    >     >     Here's an updated 3.1.0 feature status:
    >     >
    >     >     1. Merged & Completed features:
    >     >     * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
    >     >     * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
    > works
    >     >     end-to-end.
    >     >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
    > services.
    >     >     * (Steve Loughran): HADOOP-13786: S3Guard committer for
    > zero-rename
    >     > commits.
    >     >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
    > of Leaf
    >     >     Queues While Doing Queue Mapping.
    >     >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
    >     >
    >     >     2. Features close to finish:
    >     >     * (Zhankun) YARN-5983: FPGA support. Majority implementations
    >     > completed and
    >     >     merged to trunk. Except for UI/documentation.
    >     >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done,
    > some
    >     >     discussions going on about implementation.
    >     >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New
    > SchedulingRequest and
    >     >     anti-affinity support. Close to finish, on track to be merged
    > before
    >     > Jan 30.
    >     >
    >     >     3. Tentative features:
    >     >     * (Arun Suresh). YARN-5972: Support pausing/freezing
    > opportunistic
    >     >     containers. Only one pending patch. Plan to finish before Jan
    > 7th.
    >     >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
    > challenging
    >     > to be
    >     >     done before Jan 2018.
    >     >     * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
    > Looks
    >     >     challenging to be done before Jan 2018.
    >     >     * (Varun V) YARN-5673: container-executor write. Given security
    >     > refactoring
    >     >     of c-e (YARN-6623) is already landed, IMHO other stuff may be
    > moved to
    >     > 3.2.
    >     >
    >     >     Thanks,
    >     >     Wangda
    >     >
    >     >
    >     >
    >     >
    >     >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wheeleast@gmail.com
    > >
    >     > wrote:
    >     >
    >     >     > Hi all,
    >     >     >
    >     >     > Congratulations on the 3.0.0-GA release!
    >     >     >
    >     >     > As we discussed in the previous email thread [1], I'd like to
    > restart
    >     >     > 3.1.0 release plans.
    >     >     >
    >     >     > a) Quick summary:
    >     >     > a.1 Release status
    >     >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As of
    > today,
    >     >     > there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
    >     > blockers and
    >     >     > 22 open critical issues.
    >     >     >
    >     >     > a.2 Release date update
    >     >     > Considering delays of 3.0-GA release by month-and-a-half, I
    > propose
    >     > to
    >     >     > move the dates as follows
    >     >     >  - feature freeze date from Dec 15, 2017, to Jan 30, 2018 -
    > last
    >     > date for
    >     >     > any branches to get merged too;
    >     >     >  - code freeze (blockers & critical only) date to Feb 08, 2018;
    >     >     >  - release voting start by Feb 18, 2018, leaving time for at
    > least
    >     > two RCx
    >     >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
    >     >     >
    >     >     > Unlike before, I added an additional milestone for
    >     > release-vote-start so
    >     >     > that we can account for voting time-period also.
    >     >     >
    >     >     > This overall is still 5 1/2 months of release-timeline unlike
    > the
    >     > faster
    >     >     > cadence we hoped for, but this, in my opinion, is the
    > best-updated
    >     > timeline
    >     >     > given the delays of the final release of 3.0-GA.
    >     >     >
    >     >     > b) Individual feature status:
    >     >     > I spoke to several feature owners and checked the status of
    >     > un-finished
    >     >     > features, following are status of features planned to 3.1.0:
    >     >     >
    >     >     > b.1 Merged & Completed features:
    >     >     > * (Sunil) YARN-5881: Support absolute value in
    > CapacityScheduler.
    >     >     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk
    > and
    >     > works
    >     >     > end-to-end.
    >     >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
    >     > services.
    >     >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for
    > zero-rename
    >     >     > commits.
    >     >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
    > of
    >     > Leaf
    >     >     > Queues While Doing Queue Mapping.
    >     >     >
    >     >     > b.2 Features close to finish:
    >     >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting
    > now.
    >     >     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
    >     > completed
    >     >     > and merged to trunk. Except for UI/documentation.
    >     >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
    > done, some
    >     >     > discussions going on about implementation.
    >     >     >
    >     >     > b.3 Tentative features:
    >     >     > * (Arun Suresh). YARN-5972: Support pausing/freezing
    > opportunistic
    >     >     > containers. Only one pending patch. Plan to finish before Jan
    > 7th.
    >     >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
    >     > challenging to
    >     >     > be done before Jan 2018.
    >     >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New
    > SchedulingRequest
    >     > and
    >     >     > anti-affinity support. Tentative will figure out by Jan 1st.
    >     >     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
    > Looks
    >     >     > challenging to be done before Jan 2018.
    >     >     > * (Varun V) YARN-5673: container-executor write. Given security
    >     >     > refactoring of c-e (YARN-6623) is already landed, IMHO other
    > stuff
    >     > may be
    >     >     > moved to 3.2.
    >     >     >
    >     >     > b.4 Additional release drivers
    >     >     > * More exhaustive upgrade testing from 2.x to 3.x.
    >     >     >
    >     >     > c) Regarding branch cut:
    >     >     >
    >     >     > We will keep pointing trunk to 3.1 and cut branch-3.1 until:
    > A. some
    >     >     > feature planned to 3.2 has to be landed on trunk or B. After
    > feature
    >     > freeze
    >     >     > date, whichever comes first.
    >     >     >
    >     >     > I've also talked offline with Vinod to get help on
    > release-management
    >     >     > given this is my first release. He agreed to help do this
    > release
    >     > jointly.
    >     >     >
    >     >     > Thoughts?
    >     >     >
    >     >     > Thanks,
    >     >     > Wangda Tan
    >     >     >
    >     >     > [1] https://lists.apache.org/thread.html/
    >     > c11506c3250c9481852130616b3cb0
    >     >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
    >     >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion
    > in
    >     > (3.1.0)
    >     >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
    >     >     >
    >     >     >
    >     >
    >     >
    >     >
    >
    >
    >
    


---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

Re: Hadoop 3.1.0 release discussion

["Gangumalla, Uma" <um...@intel.com>]

Hi Wangda,

Sorry that we have not started vote on 29th. Daryn is reviewing the branch and he needs a day for his finalized review, then we have to prioritize the comments and decide.
Will keep updated here.

Regards,
Uma


On 1/28/18, 5:31 PM, "Wangda Tan" <wh...@gmail.com> wrote:

    Hi Uma,
    
    Thanks, I saw HDFS-13050 has been resolved 4 hours ago, I don't see any
    other blockers under HDFS-10285. I think you guys should be able to start
    voting thread in time for merging to trunk.
    
    - Wangda
    
    On Mon, Jan 29, 2018 at 3:12 AM, Gangumalla, Uma <um...@intel.com>
    wrote:
    
    > Hi Wangda,
    >
    >
    >
    > Sorry for the delay.
    >
    >
    >
    > >>* (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers:
    > HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the two
    > JIRAs?
    >
    > We have only one blocker now that is HDFS-13050 and we finished the key
    > implementation from HDFS-12995, by HDFS-13075.
    >
    > We are planning to start vote by tomorrow (29th PST time). So, we request
    > you to give us time for running vote. We will keep SPS off by default. So,
    > interested users only can enable explicitly.
    >
    >
    >
    > Regards,
    >
    > Uma
    >
    >
    >
    > *From: *Wangda Tan <wh...@gmail.com>
    > *Date: *Friday, January 26, 2018 at 5:21 PM
    > *To: *Uma Gangumalla <um...@intel.com>
    > *Cc: *"hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
    > yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "
    > common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
    > mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod
    > Kumar Vavilapalli <vi...@hortonworks.com>
    > *Subject: *Re: Hadoop 3.1.0 release discussion
    >
    >
    >
    > Hi All,
    >
    >
    >
    > Just a reminder about feature freeze date.
    >
    >
    >
    > Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days from
    > today), If you've any features which live in a branch and targeted to
    > 3.1.0, please reply this email thread. Ideally, we should finish branch
    > merging before feature freeze date.
    >
    >
    >
    > Here's an updated 3.1.0 feature status:
    >
    >
    >
    > 1. Merged features:
    >
    > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
    >
    > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
    > end-to-end.
    >
    > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
    >
    > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
    > commits.
    >
    > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
    > Queues While Doing Queue Mapping.
    >
    > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
    >
    > * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
    > and merged to trunk. Except for UI/documentation.
    >
    >
    >
    > 2. Features close to finish:
    >
    > * (Uma) HDFS-10285: HDFS SPS. There're two remaining
    > blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of
    > the two JIRAs?
    >
    > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
    > anti-affinity support. (Voting thread started).
    >
    >
    >
    > 3. Tentative features:
    >
    > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
    > containers. Only one pending patch. Plan to finish before Jan 7th.
    >
    > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
    > be done before Jan 2018.
    >
    > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
    > challenging to be done before Jan 2018.
    >
    > * (Varun V) YARN-5673: container-executor write. Given security
    > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may be
    > moved to 3.2.
    >
    >
    >
    > Thanks,
    >
    > Wangda
    >
    >
    >
    >
    >
    > On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma <um...@intel.com>
    > wrote:
    >
    > Sure, Wangda.
    >
    > Regards,
    > Uma
    >
    >
    > On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com> wrote:
    >
    >     Thanks Uma,
    >
    >     Could you update this thread once the merge vote started?
    >
    >     Best,
    >     Wangda
    >
    >     On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <
    > uma.gangumalla@intel.com>
    >     wrote:
    >
    >     > HI Wangda,
    >     >
    >     >  Thank you for the head-up mail.
    >     >  We are in the branch (HDFS-10285) and trying to push the tasks
    > sooner
    >     > before the deadline.
    >     >
    >     > Regards,
    >     > Uma
    >     >
    >     > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com> wrote:
    >     >
    >     >     Hi All,
    >     >
    >     >     Since we're fast approaching previously proposed feature freeze
    > date
    >     > (Jan
    >     >     30, about 13 days from today). If you've any features which live
    > in a
    >     >     branch and targeted to 3.1.0, please reply this email thread.
    > Ideally,
    >     > we
    >     >     should finish branch merging before feature freeze date.
    >     >
    >     >     Here's an updated 3.1.0 feature status:
    >     >
    >     >     1. Merged & Completed features:
    >     >     * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
    >     >     * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
    > works
    >     >     end-to-end.
    >     >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
    > services.
    >     >     * (Steve Loughran): HADOOP-13786: S3Guard committer for
    > zero-rename
    >     > commits.
    >     >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
    > of Leaf
    >     >     Queues While Doing Queue Mapping.
    >     >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
    >     >
    >     >     2. Features close to finish:
    >     >     * (Zhankun) YARN-5983: FPGA support. Majority implementations
    >     > completed and
    >     >     merged to trunk. Except for UI/documentation.
    >     >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done,
    > some
    >     >     discussions going on about implementation.
    >     >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New
    > SchedulingRequest and
    >     >     anti-affinity support. Close to finish, on track to be merged
    > before
    >     > Jan 30.
    >     >
    >     >     3. Tentative features:
    >     >     * (Arun Suresh). YARN-5972: Support pausing/freezing
    > opportunistic
    >     >     containers. Only one pending patch. Plan to finish before Jan
    > 7th.
    >     >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
    > challenging
    >     > to be
    >     >     done before Jan 2018.
    >     >     * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
    > Looks
    >     >     challenging to be done before Jan 2018.
    >     >     * (Varun V) YARN-5673: container-executor write. Given security
    >     > refactoring
    >     >     of c-e (YARN-6623) is already landed, IMHO other stuff may be
    > moved to
    >     > 3.2.
    >     >
    >     >     Thanks,
    >     >     Wangda
    >     >
    >     >
    >     >
    >     >
    >     >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wheeleast@gmail.com
    > >
    >     > wrote:
    >     >
    >     >     > Hi all,
    >     >     >
    >     >     > Congratulations on the 3.0.0-GA release!
    >     >     >
    >     >     > As we discussed in the previous email thread [1], I'd like to
    > restart
    >     >     > 3.1.0 release plans.
    >     >     >
    >     >     > a) Quick summary:
    >     >     > a.1 Release status
    >     >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As of
    > today,
    >     >     > there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
    >     > blockers and
    >     >     > 22 open critical issues.
    >     >     >
    >     >     > a.2 Release date update
    >     >     > Considering delays of 3.0-GA release by month-and-a-half, I
    > propose
    >     > to
    >     >     > move the dates as follows
    >     >     >  - feature freeze date from Dec 15, 2017, to Jan 30, 2018 -
    > last
    >     > date for
    >     >     > any branches to get merged too;
    >     >     >  - code freeze (blockers & critical only) date to Feb 08, 2018;
    >     >     >  - release voting start by Feb 18, 2018, leaving time for at
    > least
    >     > two RCx
    >     >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
    >     >     >
    >     >     > Unlike before, I added an additional milestone for
    >     > release-vote-start so
    >     >     > that we can account for voting time-period also.
    >     >     >
    >     >     > This overall is still 5 1/2 months of release-timeline unlike
    > the
    >     > faster
    >     >     > cadence we hoped for, but this, in my opinion, is the
    > best-updated
    >     > timeline
    >     >     > given the delays of the final release of 3.0-GA.
    >     >     >
    >     >     > b) Individual feature status:
    >     >     > I spoke to several feature owners and checked the status of
    >     > un-finished
    >     >     > features, following are status of features planned to 3.1.0:
    >     >     >
    >     >     > b.1 Merged & Completed features:
    >     >     > * (Sunil) YARN-5881: Support absolute value in
    > CapacityScheduler.
    >     >     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk
    > and
    >     > works
    >     >     > end-to-end.
    >     >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
    >     > services.
    >     >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for
    > zero-rename
    >     >     > commits.
    >     >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
    > of
    >     > Leaf
    >     >     > Queues While Doing Queue Mapping.
    >     >     >
    >     >     > b.2 Features close to finish:
    >     >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting
    > now.
    >     >     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
    >     > completed
    >     >     > and merged to trunk. Except for UI/documentation.
    >     >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
    > done, some
    >     >     > discussions going on about implementation.
    >     >     >
    >     >     > b.3 Tentative features:
    >     >     > * (Arun Suresh). YARN-5972: Support pausing/freezing
    > opportunistic
    >     >     > containers. Only one pending patch. Plan to finish before Jan
    > 7th.
    >     >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
    >     > challenging to
    >     >     > be done before Jan 2018.
    >     >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New
    > SchedulingRequest
    >     > and
    >     >     > anti-affinity support. Tentative will figure out by Jan 1st.
    >     >     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
    > Looks
    >     >     > challenging to be done before Jan 2018.
    >     >     > * (Varun V) YARN-5673: container-executor write. Given security
    >     >     > refactoring of c-e (YARN-6623) is already landed, IMHO other
    > stuff
    >     > may be
    >     >     > moved to 3.2.
    >     >     >
    >     >     > b.4 Additional release drivers
    >     >     > * More exhaustive upgrade testing from 2.x to 3.x.
    >     >     >
    >     >     > c) Regarding branch cut:
    >     >     >
    >     >     > We will keep pointing trunk to 3.1 and cut branch-3.1 until:
    > A. some
    >     >     > feature planned to 3.2 has to be landed on trunk or B. After
    > feature
    >     > freeze
    >     >     > date, whichever comes first.
    >     >     >
    >     >     > I've also talked offline with Vinod to get help on
    > release-management
    >     >     > given this is my first release. He agreed to help do this
    > release
    >     > jointly.
    >     >     >
    >     >     > Thoughts?
    >     >     >
    >     >     > Thanks,
    >     >     > Wangda Tan
    >     >     >
    >     >     > [1] https://lists.apache.org/thread.html/
    >     > c11506c3250c9481852130616b3cb0
    >     >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
    >     >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion
    > in
    >     > (3.1.0)
    >     >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
    >     >     >
    >     >     >
    >     >
    >     >
    >     >
    >
    >
    >
    


---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Hi Uma,

Thanks, I saw HDFS-13050 has been resolved 4 hours ago, I don't see any
other blockers under HDFS-10285. I think you guys should be able to start
voting thread in time for merging to trunk.

- Wangda

On Mon, Jan 29, 2018 at 3:12 AM, Gangumalla, Uma <um...@intel.com>
wrote:

> Hi Wangda,
>
>
>
> Sorry for the delay.
>
>
>
> >>* (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers:
> HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the two
> JIRAs?
>
> We have only one blocker now that is HDFS-13050 and we finished the key
> implementation from HDFS-12995, by HDFS-13075.
>
> We are planning to start vote by tomorrow (29th PST time). So, we request
> you to give us time for running vote. We will keep SPS off by default. So,
> interested users only can enable explicitly.
>
>
>
> Regards,
>
> Uma
>
>
>
> *From: *Wangda Tan <wh...@gmail.com>
> *Date: *Friday, January 26, 2018 at 5:21 PM
> *To: *Uma Gangumalla <um...@intel.com>
> *Cc: *"hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "
> common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod
> Kumar Vavilapalli <vi...@hortonworks.com>
> *Subject: *Re: Hadoop 3.1.0 release discussion
>
>
>
> Hi All,
>
>
>
> Just a reminder about feature freeze date.
>
>
>
> Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days from
> today), If you've any features which live in a branch and targeted to
> 3.1.0, please reply this email thread. Ideally, we should finish branch
> merging before feature freeze date.
>
>
>
> Here's an updated 3.1.0 feature status:
>
>
>
> 1. Merged features:
>
> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>
> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
> end-to-end.
>
> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>
> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
> commits.
>
> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
> Queues While Doing Queue Mapping.
>
> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>
> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
> and merged to trunk. Except for UI/documentation.
>
>
>
> 2. Features close to finish:
>
> * (Uma) HDFS-10285: HDFS SPS. There're two remaining
> blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of
> the two JIRAs?
>
> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
> anti-affinity support. (Voting thread started).
>
>
>
> 3. Tentative features:
>
> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
> containers. Only one pending patch. Plan to finish before Jan 7th.
>
> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
> be done before Jan 2018.
>
> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
> challenging to be done before Jan 2018.
>
> * (Varun V) YARN-5673: container-executor write. Given security
> refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may be
> moved to 3.2.
>
>
>
> Thanks,
>
> Wangda
>
>
>
>
>
> On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma <um...@intel.com>
> wrote:
>
> Sure, Wangda.
>
> Regards,
> Uma
>
>
> On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>
>     Thanks Uma,
>
>     Could you update this thread once the merge vote started?
>
>     Best,
>     Wangda
>
>     On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     wrote:
>
>     > HI Wangda,
>     >
>     >  Thank you for the head-up mail.
>     >  We are in the branch (HDFS-10285) and trying to push the tasks
> sooner
>     > before the deadline.
>     >
>     > Regards,
>     > Uma
>     >
>     > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>     >
>     >     Hi All,
>     >
>     >     Since we're fast approaching previously proposed feature freeze
> date
>     > (Jan
>     >     30, about 13 days from today). If you've any features which live
> in a
>     >     branch and targeted to 3.1.0, please reply this email thread.
> Ideally,
>     > we
>     >     should finish branch merging before feature freeze date.
>     >
>     >     Here's an updated 3.1.0 feature status:
>     >
>     >     1. Merged & Completed features:
>     >     * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     >     * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
> works
>     >     end-to-end.
>     >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
>     >     * (Steve Loughran): HADOOP-13786: S3Guard committer for
> zero-rename
>     > commits.
>     >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
> of Leaf
>     >     Queues While Doing Queue Mapping.
>     >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >
>     >     2. Features close to finish:
>     >     * (Zhankun) YARN-5983: FPGA support. Majority implementations
>     > completed and
>     >     merged to trunk. Except for UI/documentation.
>     >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done,
> some
>     >     discussions going on about implementation.
>     >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New
> SchedulingRequest and
>     >     anti-affinity support. Close to finish, on track to be merged
> before
>     > Jan 30.
>     >
>     >     3. Tentative features:
>     >     * (Arun Suresh). YARN-5972: Support pausing/freezing
> opportunistic
>     >     containers. Only one pending patch. Plan to finish before Jan
> 7th.
>     >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
> challenging
>     > to be
>     >     done before Jan 2018.
>     >     * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
> Looks
>     >     challenging to be done before Jan 2018.
>     >     * (Varun V) YARN-5673: container-executor write. Given security
>     > refactoring
>     >     of c-e (YARN-6623) is already landed, IMHO other stuff may be
> moved to
>     > 3.2.
>     >
>     >     Thanks,
>     >     Wangda
>     >
>     >
>     >
>     >
>     >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wheeleast@gmail.com
> >
>     > wrote:
>     >
>     >     > Hi all,
>     >     >
>     >     > Congratulations on the 3.0.0-GA release!
>     >     >
>     >     > As we discussed in the previous email thread [1], I'd like to
> restart
>     >     > 3.1.0 release plans.
>     >     >
>     >     > a) Quick summary:
>     >     > a.1 Release status
>     >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As of
> today,
>     >     > there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
>     > blockers and
>     >     > 22 open critical issues.
>     >     >
>     >     > a.2 Release date update
>     >     > Considering delays of 3.0-GA release by month-and-a-half, I
> propose
>     > to
>     >     > move the dates as follows
>     >     >  - feature freeze date from Dec 15, 2017, to Jan 30, 2018 -
> last
>     > date for
>     >     > any branches to get merged too;
>     >     >  - code freeze (blockers & critical only) date to Feb 08, 2018;
>     >     >  - release voting start by Feb 18, 2018, leaving time for at
> least
>     > two RCx
>     >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
>     >     >
>     >     > Unlike before, I added an additional milestone for
>     > release-vote-start so
>     >     > that we can account for voting time-period also.
>     >     >
>     >     > This overall is still 5 1/2 months of release-timeline unlike
> the
>     > faster
>     >     > cadence we hoped for, but this, in my opinion, is the
> best-updated
>     > timeline
>     >     > given the delays of the final release of 3.0-GA.
>     >     >
>     >     > b) Individual feature status:
>     >     > I spoke to several feature owners and checked the status of
>     > un-finished
>     >     > features, following are status of features planned to 3.1.0:
>     >     >
>     >     > b.1 Merged & Completed features:
>     >     > * (Sunil) YARN-5881: Support absolute value in
> CapacityScheduler.
>     >     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk
> and
>     > works
>     >     > end-to-end.
>     >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
>     > services.
>     >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for
> zero-rename
>     >     > commits.
>     >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
> of
>     > Leaf
>     >     > Queues While Doing Queue Mapping.
>     >     >
>     >     > b.2 Features close to finish:
>     >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting
> now.
>     >     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
>     > completed
>     >     > and merged to trunk. Except for UI/documentation.
>     >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
> done, some
>     >     > discussions going on about implementation.
>     >     >
>     >     > b.3 Tentative features:
>     >     > * (Arun Suresh). YARN-5972: Support pausing/freezing
> opportunistic
>     >     > containers. Only one pending patch. Plan to finish before Jan
> 7th.
>     >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     > challenging to
>     >     > be done before Jan 2018.
>     >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New
> SchedulingRequest
>     > and
>     >     > anti-affinity support. Tentative will figure out by Jan 1st.
>     >     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
> Looks
>     >     > challenging to be done before Jan 2018.
>     >     > * (Varun V) YARN-5673: container-executor write. Given security
>     >     > refactoring of c-e (YARN-6623) is already landed, IMHO other
> stuff
>     > may be
>     >     > moved to 3.2.
>     >     >
>     >     > b.4 Additional release drivers
>     >     > * More exhaustive upgrade testing from 2.x to 3.x.
>     >     >
>     >     > c) Regarding branch cut:
>     >     >
>     >     > We will keep pointing trunk to 3.1 and cut branch-3.1 until:
> A. some
>     >     > feature planned to 3.2 has to be landed on trunk or B. After
> feature
>     > freeze
>     >     > date, whichever comes first.
>     >     >
>     >     > I've also talked offline with Vinod to get help on
> release-management
>     >     > given this is my first release. He agreed to help do this
> release
>     > jointly.
>     >     >
>     >     > Thoughts?
>     >     >
>     >     > Thanks,
>     >     > Wangda Tan
>     >     >
>     >     > [1] https://lists.apache.org/thread.html/
>     > c11506c3250c9481852130616b3cb0
>     >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>     >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion
> in
>     > (3.1.0)
>     >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>     >     >
>     >     >
>     >
>     >
>     >
>
>
>

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Hi Uma,

Thanks, I saw HDFS-13050 has been resolved 4 hours ago, I don't see any
other blockers under HDFS-10285. I think you guys should be able to start
voting thread in time for merging to trunk.

- Wangda

On Mon, Jan 29, 2018 at 3:12 AM, Gangumalla, Uma <um...@intel.com>
wrote:

> Hi Wangda,
>
>
>
> Sorry for the delay.
>
>
>
> >>* (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers:
> HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the two
> JIRAs?
>
> We have only one blocker now that is HDFS-13050 and we finished the key
> implementation from HDFS-12995, by HDFS-13075.
>
> We are planning to start vote by tomorrow (29th PST time). So, we request
> you to give us time for running vote. We will keep SPS off by default. So,
> interested users only can enable explicitly.
>
>
>
> Regards,
>
> Uma
>
>
>
> *From: *Wangda Tan <wh...@gmail.com>
> *Date: *Friday, January 26, 2018 at 5:21 PM
> *To: *Uma Gangumalla <um...@intel.com>
> *Cc: *"hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "
> common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod
> Kumar Vavilapalli <vi...@hortonworks.com>
> *Subject: *Re: Hadoop 3.1.0 release discussion
>
>
>
> Hi All,
>
>
>
> Just a reminder about feature freeze date.
>
>
>
> Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days from
> today), If you've any features which live in a branch and targeted to
> 3.1.0, please reply this email thread. Ideally, we should finish branch
> merging before feature freeze date.
>
>
>
> Here's an updated 3.1.0 feature status:
>
>
>
> 1. Merged features:
>
> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>
> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
> end-to-end.
>
> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>
> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
> commits.
>
> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
> Queues While Doing Queue Mapping.
>
> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>
> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
> and merged to trunk. Except for UI/documentation.
>
>
>
> 2. Features close to finish:
>
> * (Uma) HDFS-10285: HDFS SPS. There're two remaining
> blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of
> the two JIRAs?
>
> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
> anti-affinity support. (Voting thread started).
>
>
>
> 3. Tentative features:
>
> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
> containers. Only one pending patch. Plan to finish before Jan 7th.
>
> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
> be done before Jan 2018.
>
> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
> challenging to be done before Jan 2018.
>
> * (Varun V) YARN-5673: container-executor write. Given security
> refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may be
> moved to 3.2.
>
>
>
> Thanks,
>
> Wangda
>
>
>
>
>
> On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma <um...@intel.com>
> wrote:
>
> Sure, Wangda.
>
> Regards,
> Uma
>
>
> On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>
>     Thanks Uma,
>
>     Could you update this thread once the merge vote started?
>
>     Best,
>     Wangda
>
>     On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     wrote:
>
>     > HI Wangda,
>     >
>     >  Thank you for the head-up mail.
>     >  We are in the branch (HDFS-10285) and trying to push the tasks
> sooner
>     > before the deadline.
>     >
>     > Regards,
>     > Uma
>     >
>     > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>     >
>     >     Hi All,
>     >
>     >     Since we're fast approaching previously proposed feature freeze
> date
>     > (Jan
>     >     30, about 13 days from today). If you've any features which live
> in a
>     >     branch and targeted to 3.1.0, please reply this email thread.
> Ideally,
>     > we
>     >     should finish branch merging before feature freeze date.
>     >
>     >     Here's an updated 3.1.0 feature status:
>     >
>     >     1. Merged & Completed features:
>     >     * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     >     * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
> works
>     >     end-to-end.
>     >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
>     >     * (Steve Loughran): HADOOP-13786: S3Guard committer for
> zero-rename
>     > commits.
>     >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
> of Leaf
>     >     Queues While Doing Queue Mapping.
>     >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >
>     >     2. Features close to finish:
>     >     * (Zhankun) YARN-5983: FPGA support. Majority implementations
>     > completed and
>     >     merged to trunk. Except for UI/documentation.
>     >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done,
> some
>     >     discussions going on about implementation.
>     >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New
> SchedulingRequest and
>     >     anti-affinity support. Close to finish, on track to be merged
> before
>     > Jan 30.
>     >
>     >     3. Tentative features:
>     >     * (Arun Suresh). YARN-5972: Support pausing/freezing
> opportunistic
>     >     containers. Only one pending patch. Plan to finish before Jan
> 7th.
>     >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
> challenging
>     > to be
>     >     done before Jan 2018.
>     >     * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
> Looks
>     >     challenging to be done before Jan 2018.
>     >     * (Varun V) YARN-5673: container-executor write. Given security
>     > refactoring
>     >     of c-e (YARN-6623) is already landed, IMHO other stuff may be
> moved to
>     > 3.2.
>     >
>     >     Thanks,
>     >     Wangda
>     >
>     >
>     >
>     >
>     >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wheeleast@gmail.com
> >
>     > wrote:
>     >
>     >     > Hi all,
>     >     >
>     >     > Congratulations on the 3.0.0-GA release!
>     >     >
>     >     > As we discussed in the previous email thread [1], I'd like to
> restart
>     >     > 3.1.0 release plans.
>     >     >
>     >     > a) Quick summary:
>     >     > a.1 Release status
>     >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As of
> today,
>     >     > there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
>     > blockers and
>     >     > 22 open critical issues.
>     >     >
>     >     > a.2 Release date update
>     >     > Considering delays of 3.0-GA release by month-and-a-half, I
> propose
>     > to
>     >     > move the dates as follows
>     >     >  - feature freeze date from Dec 15, 2017, to Jan 30, 2018 -
> last
>     > date for
>     >     > any branches to get merged too;
>     >     >  - code freeze (blockers & critical only) date to Feb 08, 2018;
>     >     >  - release voting start by Feb 18, 2018, leaving time for at
> least
>     > two RCx
>     >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
>     >     >
>     >     > Unlike before, I added an additional milestone for
>     > release-vote-start so
>     >     > that we can account for voting time-period also.
>     >     >
>     >     > This overall is still 5 1/2 months of release-timeline unlike
> the
>     > faster
>     >     > cadence we hoped for, but this, in my opinion, is the
> best-updated
>     > timeline
>     >     > given the delays of the final release of 3.0-GA.
>     >     >
>     >     > b) Individual feature status:
>     >     > I spoke to several feature owners and checked the status of
>     > un-finished
>     >     > features, following are status of features planned to 3.1.0:
>     >     >
>     >     > b.1 Merged & Completed features:
>     >     > * (Sunil) YARN-5881: Support absolute value in
> CapacityScheduler.
>     >     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk
> and
>     > works
>     >     > end-to-end.
>     >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
>     > services.
>     >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for
> zero-rename
>     >     > commits.
>     >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
> of
>     > Leaf
>     >     > Queues While Doing Queue Mapping.
>     >     >
>     >     > b.2 Features close to finish:
>     >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting
> now.
>     >     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
>     > completed
>     >     > and merged to trunk. Except for UI/documentation.
>     >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
> done, some
>     >     > discussions going on about implementation.
>     >     >
>     >     > b.3 Tentative features:
>     >     > * (Arun Suresh). YARN-5972: Support pausing/freezing
> opportunistic
>     >     > containers. Only one pending patch. Plan to finish before Jan
> 7th.
>     >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     > challenging to
>     >     > be done before Jan 2018.
>     >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New
> SchedulingRequest
>     > and
>     >     > anti-affinity support. Tentative will figure out by Jan 1st.
>     >     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
> Looks
>     >     > challenging to be done before Jan 2018.
>     >     > * (Varun V) YARN-5673: container-executor write. Given security
>     >     > refactoring of c-e (YARN-6623) is already landed, IMHO other
> stuff
>     > may be
>     >     > moved to 3.2.
>     >     >
>     >     > b.4 Additional release drivers
>     >     > * More exhaustive upgrade testing from 2.x to 3.x.
>     >     >
>     >     > c) Regarding branch cut:
>     >     >
>     >     > We will keep pointing trunk to 3.1 and cut branch-3.1 until:
> A. some
>     >     > feature planned to 3.2 has to be landed on trunk or B. After
> feature
>     > freeze
>     >     > date, whichever comes first.
>     >     >
>     >     > I've also talked offline with Vinod to get help on
> release-management
>     >     > given this is my first release. He agreed to help do this
> release
>     > jointly.
>     >     >
>     >     > Thoughts?
>     >     >
>     >     > Thanks,
>     >     > Wangda Tan
>     >     >
>     >     > [1] https://lists.apache.org/thread.html/
>     > c11506c3250c9481852130616b3cb0
>     >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>     >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion
> in
>     > (3.1.0)
>     >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>     >     >
>     >     >
>     >
>     >
>     >
>
>
>

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Hi Uma,

Thanks, I saw HDFS-13050 has been resolved 4 hours ago, I don't see any
other blockers under HDFS-10285. I think you guys should be able to start
voting thread in time for merging to trunk.

- Wangda

On Mon, Jan 29, 2018 at 3:12 AM, Gangumalla, Uma <um...@intel.com>
wrote:

> Hi Wangda,
>
>
>
> Sorry for the delay.
>
>
>
> >>* (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers:
> HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the two
> JIRAs?
>
> We have only one blocker now that is HDFS-13050 and we finished the key
> implementation from HDFS-12995, by HDFS-13075.
>
> We are planning to start vote by tomorrow (29th PST time). So, we request
> you to give us time for running vote. We will keep SPS off by default. So,
> interested users only can enable explicitly.
>
>
>
> Regards,
>
> Uma
>
>
>
> *From: *Wangda Tan <wh...@gmail.com>
> *Date: *Friday, January 26, 2018 at 5:21 PM
> *To: *Uma Gangumalla <um...@intel.com>
> *Cc: *"hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "
> common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod
> Kumar Vavilapalli <vi...@hortonworks.com>
> *Subject: *Re: Hadoop 3.1.0 release discussion
>
>
>
> Hi All,
>
>
>
> Just a reminder about feature freeze date.
>
>
>
> Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days from
> today), If you've any features which live in a branch and targeted to
> 3.1.0, please reply this email thread. Ideally, we should finish branch
> merging before feature freeze date.
>
>
>
> Here's an updated 3.1.0 feature status:
>
>
>
> 1. Merged features:
>
> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>
> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
> end-to-end.
>
> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>
> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
> commits.
>
> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
> Queues While Doing Queue Mapping.
>
> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>
> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
> and merged to trunk. Except for UI/documentation.
>
>
>
> 2. Features close to finish:
>
> * (Uma) HDFS-10285: HDFS SPS. There're two remaining
> blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of
> the two JIRAs?
>
> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
> anti-affinity support. (Voting thread started).
>
>
>
> 3. Tentative features:
>
> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
> containers. Only one pending patch. Plan to finish before Jan 7th.
>
> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
> be done before Jan 2018.
>
> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
> challenging to be done before Jan 2018.
>
> * (Varun V) YARN-5673: container-executor write. Given security
> refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may be
> moved to 3.2.
>
>
>
> Thanks,
>
> Wangda
>
>
>
>
>
> On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma <um...@intel.com>
> wrote:
>
> Sure, Wangda.
>
> Regards,
> Uma
>
>
> On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>
>     Thanks Uma,
>
>     Could you update this thread once the merge vote started?
>
>     Best,
>     Wangda
>
>     On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     wrote:
>
>     > HI Wangda,
>     >
>     >  Thank you for the head-up mail.
>     >  We are in the branch (HDFS-10285) and trying to push the tasks
> sooner
>     > before the deadline.
>     >
>     > Regards,
>     > Uma
>     >
>     > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>     >
>     >     Hi All,
>     >
>     >     Since we're fast approaching previously proposed feature freeze
> date
>     > (Jan
>     >     30, about 13 days from today). If you've any features which live
> in a
>     >     branch and targeted to 3.1.0, please reply this email thread.
> Ideally,
>     > we
>     >     should finish branch merging before feature freeze date.
>     >
>     >     Here's an updated 3.1.0 feature status:
>     >
>     >     1. Merged & Completed features:
>     >     * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     >     * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
> works
>     >     end-to-end.
>     >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
>     >     * (Steve Loughran): HADOOP-13786: S3Guard committer for
> zero-rename
>     > commits.
>     >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
> of Leaf
>     >     Queues While Doing Queue Mapping.
>     >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >
>     >     2. Features close to finish:
>     >     * (Zhankun) YARN-5983: FPGA support. Majority implementations
>     > completed and
>     >     merged to trunk. Except for UI/documentation.
>     >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done,
> some
>     >     discussions going on about implementation.
>     >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New
> SchedulingRequest and
>     >     anti-affinity support. Close to finish, on track to be merged
> before
>     > Jan 30.
>     >
>     >     3. Tentative features:
>     >     * (Arun Suresh). YARN-5972: Support pausing/freezing
> opportunistic
>     >     containers. Only one pending patch. Plan to finish before Jan
> 7th.
>     >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
> challenging
>     > to be
>     >     done before Jan 2018.
>     >     * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
> Looks
>     >     challenging to be done before Jan 2018.
>     >     * (Varun V) YARN-5673: container-executor write. Given security
>     > refactoring
>     >     of c-e (YARN-6623) is already landed, IMHO other stuff may be
> moved to
>     > 3.2.
>     >
>     >     Thanks,
>     >     Wangda
>     >
>     >
>     >
>     >
>     >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wheeleast@gmail.com
> >
>     > wrote:
>     >
>     >     > Hi all,
>     >     >
>     >     > Congratulations on the 3.0.0-GA release!
>     >     >
>     >     > As we discussed in the previous email thread [1], I'd like to
> restart
>     >     > 3.1.0 release plans.
>     >     >
>     >     > a) Quick summary:
>     >     > a.1 Release status
>     >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As of
> today,
>     >     > there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
>     > blockers and
>     >     > 22 open critical issues.
>     >     >
>     >     > a.2 Release date update
>     >     > Considering delays of 3.0-GA release by month-and-a-half, I
> propose
>     > to
>     >     > move the dates as follows
>     >     >  - feature freeze date from Dec 15, 2017, to Jan 30, 2018 -
> last
>     > date for
>     >     > any branches to get merged too;
>     >     >  - code freeze (blockers & critical only) date to Feb 08, 2018;
>     >     >  - release voting start by Feb 18, 2018, leaving time for at
> least
>     > two RCx
>     >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
>     >     >
>     >     > Unlike before, I added an additional milestone for
>     > release-vote-start so
>     >     > that we can account for voting time-period also.
>     >     >
>     >     > This overall is still 5 1/2 months of release-timeline unlike
> the
>     > faster
>     >     > cadence we hoped for, but this, in my opinion, is the
> best-updated
>     > timeline
>     >     > given the delays of the final release of 3.0-GA.
>     >     >
>     >     > b) Individual feature status:
>     >     > I spoke to several feature owners and checked the status of
>     > un-finished
>     >     > features, following are status of features planned to 3.1.0:
>     >     >
>     >     > b.1 Merged & Completed features:
>     >     > * (Sunil) YARN-5881: Support absolute value in
> CapacityScheduler.
>     >     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk
> and
>     > works
>     >     > end-to-end.
>     >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
>     > services.
>     >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for
> zero-rename
>     >     > commits.
>     >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
> of
>     > Leaf
>     >     > Queues While Doing Queue Mapping.
>     >     >
>     >     > b.2 Features close to finish:
>     >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting
> now.
>     >     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
>     > completed
>     >     > and merged to trunk. Except for UI/documentation.
>     >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
> done, some
>     >     > discussions going on about implementation.
>     >     >
>     >     > b.3 Tentative features:
>     >     > * (Arun Suresh). YARN-5972: Support pausing/freezing
> opportunistic
>     >     > containers. Only one pending patch. Plan to finish before Jan
> 7th.
>     >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     > challenging to
>     >     > be done before Jan 2018.
>     >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New
> SchedulingRequest
>     > and
>     >     > anti-affinity support. Tentative will figure out by Jan 1st.
>     >     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
> Looks
>     >     > challenging to be done before Jan 2018.
>     >     > * (Varun V) YARN-5673: container-executor write. Given security
>     >     > refactoring of c-e (YARN-6623) is already landed, IMHO other
> stuff
>     > may be
>     >     > moved to 3.2.
>     >     >
>     >     > b.4 Additional release drivers
>     >     > * More exhaustive upgrade testing from 2.x to 3.x.
>     >     >
>     >     > c) Regarding branch cut:
>     >     >
>     >     > We will keep pointing trunk to 3.1 and cut branch-3.1 until:
> A. some
>     >     > feature planned to 3.2 has to be landed on trunk or B. After
> feature
>     > freeze
>     >     > date, whichever comes first.
>     >     >
>     >     > I've also talked offline with Vinod to get help on
> release-management
>     >     > given this is my first release. He agreed to help do this
> release
>     > jointly.
>     >     >
>     >     > Thoughts?
>     >     >
>     >     > Thanks,
>     >     > Wangda Tan
>     >     >
>     >     > [1] https://lists.apache.org/thread.html/
>     > c11506c3250c9481852130616b3cb0
>     >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>     >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion
> in
>     > (3.1.0)
>     >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>     >     >
>     >     >
>     >
>     >
>     >
>
>
>

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Hi Uma,

Thanks, I saw HDFS-13050 has been resolved 4 hours ago, I don't see any
other blockers under HDFS-10285. I think you guys should be able to start
voting thread in time for merging to trunk.

- Wangda

On Mon, Jan 29, 2018 at 3:12 AM, Gangumalla, Uma <um...@intel.com>
wrote:

> Hi Wangda,
>
>
>
> Sorry for the delay.
>
>
>
> >>* (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers:
> HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the two
> JIRAs?
>
> We have only one blocker now that is HDFS-13050 and we finished the key
> implementation from HDFS-12995, by HDFS-13075.
>
> We are planning to start vote by tomorrow (29th PST time). So, we request
> you to give us time for running vote. We will keep SPS off by default. So,
> interested users only can enable explicitly.
>
>
>
> Regards,
>
> Uma
>
>
>
> *From: *Wangda Tan <wh...@gmail.com>
> *Date: *Friday, January 26, 2018 at 5:21 PM
> *To: *Uma Gangumalla <um...@intel.com>
> *Cc: *"hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
> yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "
> common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "
> mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod
> Kumar Vavilapalli <vi...@hortonworks.com>
> *Subject: *Re: Hadoop 3.1.0 release discussion
>
>
>
> Hi All,
>
>
>
> Just a reminder about feature freeze date.
>
>
>
> Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days from
> today), If you've any features which live in a branch and targeted to
> 3.1.0, please reply this email thread. Ideally, we should finish branch
> merging before feature freeze date.
>
>
>
> Here's an updated 3.1.0 feature status:
>
>
>
> 1. Merged features:
>
> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>
> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
> end-to-end.
>
> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>
> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
> commits.
>
> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
> Queues While Doing Queue Mapping.
>
> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>
> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
> and merged to trunk. Except for UI/documentation.
>
>
>
> 2. Features close to finish:
>
> * (Uma) HDFS-10285: HDFS SPS. There're two remaining
> blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of
> the two JIRAs?
>
> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
> anti-affinity support. (Voting thread started).
>
>
>
> 3. Tentative features:
>
> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
> containers. Only one pending patch. Plan to finish before Jan 7th.
>
> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
> be done before Jan 2018.
>
> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
> challenging to be done before Jan 2018.
>
> * (Varun V) YARN-5673: container-executor write. Given security
> refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may be
> moved to 3.2.
>
>
>
> Thanks,
>
> Wangda
>
>
>
>
>
> On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma <um...@intel.com>
> wrote:
>
> Sure, Wangda.
>
> Regards,
> Uma
>
>
> On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>
>     Thanks Uma,
>
>     Could you update this thread once the merge vote started?
>
>     Best,
>     Wangda
>
>     On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     wrote:
>
>     > HI Wangda,
>     >
>     >  Thank you for the head-up mail.
>     >  We are in the branch (HDFS-10285) and trying to push the tasks
> sooner
>     > before the deadline.
>     >
>     > Regards,
>     > Uma
>     >
>     > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>     >
>     >     Hi All,
>     >
>     >     Since we're fast approaching previously proposed feature freeze
> date
>     > (Jan
>     >     30, about 13 days from today). If you've any features which live
> in a
>     >     branch and targeted to 3.1.0, please reply this email thread.
> Ideally,
>     > we
>     >     should finish branch merging before feature freeze date.
>     >
>     >     Here's an updated 3.1.0 feature status:
>     >
>     >     1. Merged & Completed features:
>     >     * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     >     * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
> works
>     >     end-to-end.
>     >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
>     >     * (Steve Loughran): HADOOP-13786: S3Guard committer for
> zero-rename
>     > commits.
>     >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
> of Leaf
>     >     Queues While Doing Queue Mapping.
>     >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >
>     >     2. Features close to finish:
>     >     * (Zhankun) YARN-5983: FPGA support. Majority implementations
>     > completed and
>     >     merged to trunk. Except for UI/documentation.
>     >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done,
> some
>     >     discussions going on about implementation.
>     >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New
> SchedulingRequest and
>     >     anti-affinity support. Close to finish, on track to be merged
> before
>     > Jan 30.
>     >
>     >     3. Tentative features:
>     >     * (Arun Suresh). YARN-5972: Support pausing/freezing
> opportunistic
>     >     containers. Only one pending patch. Plan to finish before Jan
> 7th.
>     >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
> challenging
>     > to be
>     >     done before Jan 2018.
>     >     * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
> Looks
>     >     challenging to be done before Jan 2018.
>     >     * (Varun V) YARN-5673: container-executor write. Given security
>     > refactoring
>     >     of c-e (YARN-6623) is already landed, IMHO other stuff may be
> moved to
>     > 3.2.
>     >
>     >     Thanks,
>     >     Wangda
>     >
>     >
>     >
>     >
>     >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wheeleast@gmail.com
> >
>     > wrote:
>     >
>     >     > Hi all,
>     >     >
>     >     > Congratulations on the 3.0.0-GA release!
>     >     >
>     >     > As we discussed in the previous email thread [1], I'd like to
> restart
>     >     > 3.1.0 release plans.
>     >     >
>     >     > a) Quick summary:
>     >     > a.1 Release status
>     >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As of
> today,
>     >     > there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
>     > blockers and
>     >     > 22 open critical issues.
>     >     >
>     >     > a.2 Release date update
>     >     > Considering delays of 3.0-GA release by month-and-a-half, I
> propose
>     > to
>     >     > move the dates as follows
>     >     >  - feature freeze date from Dec 15, 2017, to Jan 30, 2018 -
> last
>     > date for
>     >     > any branches to get merged too;
>     >     >  - code freeze (blockers & critical only) date to Feb 08, 2018;
>     >     >  - release voting start by Feb 18, 2018, leaving time for at
> least
>     > two RCx
>     >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
>     >     >
>     >     > Unlike before, I added an additional milestone for
>     > release-vote-start so
>     >     > that we can account for voting time-period also.
>     >     >
>     >     > This overall is still 5 1/2 months of release-timeline unlike
> the
>     > faster
>     >     > cadence we hoped for, but this, in my opinion, is the
> best-updated
>     > timeline
>     >     > given the delays of the final release of 3.0-GA.
>     >     >
>     >     > b) Individual feature status:
>     >     > I spoke to several feature owners and checked the status of
>     > un-finished
>     >     > features, following are status of features planned to 3.1.0:
>     >     >
>     >     > b.1 Merged & Completed features:
>     >     > * (Sunil) YARN-5881: Support absolute value in
> CapacityScheduler.
>     >     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk
> and
>     > works
>     >     > end-to-end.
>     >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
>     > services.
>     >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for
> zero-rename
>     >     > commits.
>     >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
> of
>     > Leaf
>     >     > Queues While Doing Queue Mapping.
>     >     >
>     >     > b.2 Features close to finish:
>     >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting
> now.
>     >     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
>     > completed
>     >     > and merged to trunk. Except for UI/documentation.
>     >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
> done, some
>     >     > discussions going on about implementation.
>     >     >
>     >     > b.3 Tentative features:
>     >     > * (Arun Suresh). YARN-5972: Support pausing/freezing
> opportunistic
>     >     > containers. Only one pending patch. Plan to finish before Jan
> 7th.
>     >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     > challenging to
>     >     > be done before Jan 2018.
>     >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New
> SchedulingRequest
>     > and
>     >     > anti-affinity support. Tentative will figure out by Jan 1st.
>     >     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
> Looks
>     >     > challenging to be done before Jan 2018.
>     >     > * (Varun V) YARN-5673: container-executor write. Given security
>     >     > refactoring of c-e (YARN-6623) is already landed, IMHO other
> stuff
>     > may be
>     >     > moved to 3.2.
>     >     >
>     >     > b.4 Additional release drivers
>     >     > * More exhaustive upgrade testing from 2.x to 3.x.
>     >     >
>     >     > c) Regarding branch cut:
>     >     >
>     >     > We will keep pointing trunk to 3.1 and cut branch-3.1 until:
> A. some
>     >     > feature planned to 3.2 has to be landed on trunk or B. After
> feature
>     > freeze
>     >     > date, whichever comes first.
>     >     >
>     >     > I've also talked offline with Vinod to get help on
> release-management
>     >     > given this is my first release. He agreed to help do this
> release
>     > jointly.
>     >     >
>     >     > Thoughts?
>     >     >
>     >     > Thanks,
>     >     > Wangda Tan
>     >     >
>     >     > [1] https://lists.apache.org/thread.html/
>     > c11506c3250c9481852130616b3cb0
>     >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>     >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion
> in
>     > (3.1.0)
>     >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>     >     >
>     >     >
>     >
>     >
>     >
>
>
>

Re: Hadoop 3.1.0 release discussion

["Gangumalla, Uma" <um...@intel.com>]

Hi Wangda,

Sorry for the delay.

>>* (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the two JIRAs?
We have only one blocker now that is HDFS-13050 and we finished the key implementation from HDFS-12995, by HDFS-13075.
We are planning to start vote by tomorrow (29th PST time). So, we request you to give us time for running vote. We will keep SPS off by default. So, interested users only can enable explicitly.

Regards,
Uma

From: Wangda Tan <wh...@gmail.com>
Date: Friday, January 26, 2018 at 5:21 PM
To: Uma Gangumalla <um...@intel.com>
Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod Kumar Vavilapalli <vi...@hortonworks.com>
Subject: Re: Hadoop 3.1.0 release discussion

Hi All,

Just a reminder about feature freeze date.

Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days from today), If you've any features which live in a branch and targeted to 3.1.0, please reply this email thread. Ideally, we should finish branch merging before feature freeze date.

Here's an updated 3.1.0 feature status:

1. Merged features:
* (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
* (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works end-to-end.
* (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
* (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename commits.
* (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf Queues While Doing Queue Mapping.
* (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
* (Zhankun) YARN-5983: FPGA support. Majority implementations completed and merged to trunk. Except for UI/documentation.

2. Features close to finish:
* (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the two JIRAs?
* (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and anti-affinity support. (Voting thread started).

3. Tentative features:
* (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic containers. Only one pending patch. Plan to finish before Jan 7th.
* (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to be done before Jan 2018.
* (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks challenging to be done before Jan 2018.
* (Varun V) YARN-5673: container-executor write. Given security refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to 3.2.

Thanks,
Wangda


On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma <um...@intel.com>> wrote:
Sure, Wangda.

Regards,
Uma

On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com>> wrote:

    Thanks Uma,

    Could you update this thread once the merge vote started?

    Best,
    Wangda

    On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <um...@intel.com>>
    wrote:

    > HI Wangda,
    >
    >  Thank you for the head-up mail.
    >  We are in the branch (HDFS-10285) and trying to push the tasks sooner
    > before the deadline.
    >
    > Regards,
    > Uma
    >
    > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com>> wrote:
    >
    >     Hi All,
    >
    >     Since we're fast approaching previously proposed feature freeze date
    > (Jan
    >     30, about 13 days from today). If you've any features which live in a
    >     branch and targeted to 3.1.0, please reply this email thread. Ideally,
    > we
    >     should finish branch merging before feature freeze date.
    >
    >     Here's an updated 3.1.0 feature status:
    >
    >     1. Merged & Completed features:
    >     * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
    >     * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
    >     end-to-end.
    >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
    >     * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
    > commits.
    >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
    >     Queues While Doing Queue Mapping.
    >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
    >
    >     2. Features close to finish:
    >     * (Zhankun) YARN-5983: FPGA support. Majority implementations
    > completed and
    >     merged to trunk. Except for UI/documentation.
    >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
    >     discussions going on about implementation.
    >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
    >     anti-affinity support. Close to finish, on track to be merged before
    > Jan 30.
    >
    >     3. Tentative features:
    >     * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
    >     containers. Only one pending patch. Plan to finish before Jan 7th.
    >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging
    > to be
    >     done before Jan 2018.
    >     * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
    >     challenging to be done before Jan 2018.
    >     * (Varun V) YARN-5673: container-executor write. Given security
    > refactoring
    >     of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to
    > 3.2.
    >
    >     Thanks,
    >     Wangda
    >
    >
    >
    >
    >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com>>
    > wrote:
    >
    >     > Hi all,
    >     >
    >     > Congratulations on the 3.0.0-GA release!
    >     >
    >     > As we discussed in the previous email thread [1], I'd like to restart
    >     > 3.1.0 release plans.
    >     >
    >     > a) Quick summary:
    >     > a.1 Release status
    >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
    >     > there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
    > blockers and
    >     > 22 open critical issues.
    >     >
    >     > a.2 Release date update
    >     > Considering delays of 3.0-GA release by month-and-a-half, I propose
    > to
    >     > move the dates as follows
    >     >  - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last
    > date for
    >     > any branches to get merged too;
    >     >  - code freeze (blockers & critical only) date to Feb 08, 2018;
    >     >  - release voting start by Feb 18, 2018, leaving time for at least
    > two RCx
    >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
    >     >
    >     > Unlike before, I added an additional milestone for
    > release-vote-start so
    >     > that we can account for voting time-period also.
    >     >
    >     > This overall is still 5 1/2 months of release-timeline unlike the
    > faster
    >     > cadence we hoped for, but this, in my opinion, is the best-updated
    > timeline
    >     > given the delays of the final release of 3.0-GA.
    >     >
    >     > b) Individual feature status:
    >     > I spoke to several feature owners and checked the status of
    > un-finished
    >     > features, following are status of features planned to 3.1.0:
    >     >
    >     > b.1 Merged & Completed features:
    >     > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
    >     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
    > works
    >     > end-to-end.
    >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
    > services.
    >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
    >     > commits.
    >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of
    > Leaf
    >     > Queues While Doing Queue Mapping.
    >     >
    >     > b.2 Features close to finish:
    >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
    >     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
    > completed
    >     > and merged to trunk. Except for UI/documentation.
    >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
    >     > discussions going on about implementation.
    >     >
    >     > b.3 Tentative features:
    >     > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
    >     > containers. Only one pending patch. Plan to finish before Jan 7th.
    >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
    > challenging to
    >     > be done before Jan 2018.
    >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest
    > and
    >     > anti-affinity support. Tentative will figure out by Jan 1st.
    >     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
    >     > challenging to be done before Jan 2018.
    >     > * (Varun V) YARN-5673: container-executor write. Given security
    >     > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff
    > may be
    >     > moved to 3.2.
    >     >
    >     > b.4 Additional release drivers
    >     > * More exhaustive upgrade testing from 2.x to 3.x.
    >     >
    >     > c) Regarding branch cut:
    >     >
    >     > We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
    >     > feature planned to 3.2 has to be landed on trunk or B. After feature
    > freeze
    >     > date, whichever comes first.
    >     >
    >     > I've also talked offline with Vinod to get help on release-management
    >     > given this is my first release. He agreed to help do this release
    > jointly.
    >     >
    >     > Thoughts?
    >     >
    >     > Thanks,
    >     > Wangda Tan
    >     >
    >     > [1] https://lists.apache.org/thread.html/
    > c11506c3250c9481852130616b3cb0
    >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org<http://3Cyarn-dev.hadoop.apache.org>%3E
    >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
    > (3.1.0)
    >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
    >     >
    >     >
    >
    >
    >

Re: Hadoop 3.1.0 release discussion

["Gangumalla, Uma" <um...@intel.com>]

Hi Wangda,

Sorry for the delay.

>>* (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the two JIRAs?
We have only one blocker now that is HDFS-13050 and we finished the key implementation from HDFS-12995, by HDFS-13075.
We are planning to start vote by tomorrow (29th PST time). So, we request you to give us time for running vote. We will keep SPS off by default. So, interested users only can enable explicitly.

Regards,
Uma

From: Wangda Tan <wh...@gmail.com>
Date: Friday, January 26, 2018 at 5:21 PM
To: Uma Gangumalla <um...@intel.com>
Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>, "common-dev@hadoop.apache.org" <co...@hadoop.apache.org>, "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>, Vinod Kumar Vavilapalli <vi...@hortonworks.com>
Subject: Re: Hadoop 3.1.0 release discussion

Hi All,

Just a reminder about feature freeze date.

Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days from today), If you've any features which live in a branch and targeted to 3.1.0, please reply this email thread. Ideally, we should finish branch merging before feature freeze date.

Here's an updated 3.1.0 feature status:

1. Merged features:
* (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
* (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works end-to-end.
* (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
* (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename commits.
* (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf Queues While Doing Queue Mapping.
* (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
* (Zhankun) YARN-5983: FPGA support. Majority implementations completed and merged to trunk. Except for UI/documentation.

2. Features close to finish:
* (Uma) HDFS-10285: HDFS SPS. There're two remaining blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of the two JIRAs?
* (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and anti-affinity support. (Voting thread started).

3. Tentative features:
* (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic containers. Only one pending patch. Plan to finish before Jan 7th.
* (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to be done before Jan 2018.
* (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks challenging to be done before Jan 2018.
* (Varun V) YARN-5673: container-executor write. Given security refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to 3.2.

Thanks,
Wangda


On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma <um...@intel.com>> wrote:
Sure, Wangda.

Regards,
Uma

On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com>> wrote:

    Thanks Uma,

    Could you update this thread once the merge vote started?

    Best,
    Wangda

    On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <um...@intel.com>>
    wrote:

    > HI Wangda,
    >
    >  Thank you for the head-up mail.
    >  We are in the branch (HDFS-10285) and trying to push the tasks sooner
    > before the deadline.
    >
    > Regards,
    > Uma
    >
    > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com>> wrote:
    >
    >     Hi All,
    >
    >     Since we're fast approaching previously proposed feature freeze date
    > (Jan
    >     30, about 13 days from today). If you've any features which live in a
    >     branch and targeted to 3.1.0, please reply this email thread. Ideally,
    > we
    >     should finish branch merging before feature freeze date.
    >
    >     Here's an updated 3.1.0 feature status:
    >
    >     1. Merged & Completed features:
    >     * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
    >     * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
    >     end-to-end.
    >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
    >     * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
    > commits.
    >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
    >     Queues While Doing Queue Mapping.
    >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
    >
    >     2. Features close to finish:
    >     * (Zhankun) YARN-5983: FPGA support. Majority implementations
    > completed and
    >     merged to trunk. Except for UI/documentation.
    >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
    >     discussions going on about implementation.
    >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
    >     anti-affinity support. Close to finish, on track to be merged before
    > Jan 30.
    >
    >     3. Tentative features:
    >     * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
    >     containers. Only one pending patch. Plan to finish before Jan 7th.
    >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging
    > to be
    >     done before Jan 2018.
    >     * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
    >     challenging to be done before Jan 2018.
    >     * (Varun V) YARN-5673: container-executor write. Given security
    > refactoring
    >     of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to
    > 3.2.
    >
    >     Thanks,
    >     Wangda
    >
    >
    >
    >
    >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com>>
    > wrote:
    >
    >     > Hi all,
    >     >
    >     > Congratulations on the 3.0.0-GA release!
    >     >
    >     > As we discussed in the previous email thread [1], I'd like to restart
    >     > 3.1.0 release plans.
    >     >
    >     > a) Quick summary:
    >     > a.1 Release status
    >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
    >     > there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
    > blockers and
    >     > 22 open critical issues.
    >     >
    >     > a.2 Release date update
    >     > Considering delays of 3.0-GA release by month-and-a-half, I propose
    > to
    >     > move the dates as follows
    >     >  - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last
    > date for
    >     > any branches to get merged too;
    >     >  - code freeze (blockers & critical only) date to Feb 08, 2018;
    >     >  - release voting start by Feb 18, 2018, leaving time for at least
    > two RCx
    >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
    >     >
    >     > Unlike before, I added an additional milestone for
    > release-vote-start so
    >     > that we can account for voting time-period also.
    >     >
    >     > This overall is still 5 1/2 months of release-timeline unlike the
    > faster
    >     > cadence we hoped for, but this, in my opinion, is the best-updated
    > timeline
    >     > given the delays of the final release of 3.0-GA.
    >     >
    >     > b) Individual feature status:
    >     > I spoke to several feature owners and checked the status of
    > un-finished
    >     > features, following are status of features planned to 3.1.0:
    >     >
    >     > b.1 Merged & Completed features:
    >     > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
    >     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
    > works
    >     > end-to-end.
    >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
    > services.
    >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
    >     > commits.
    >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of
    > Leaf
    >     > Queues While Doing Queue Mapping.
    >     >
    >     > b.2 Features close to finish:
    >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
    >     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
    > completed
    >     > and merged to trunk. Except for UI/documentation.
    >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
    >     > discussions going on about implementation.
    >     >
    >     > b.3 Tentative features:
    >     > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
    >     > containers. Only one pending patch. Plan to finish before Jan 7th.
    >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
    > challenging to
    >     > be done before Jan 2018.
    >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest
    > and
    >     > anti-affinity support. Tentative will figure out by Jan 1st.
    >     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
    >     > challenging to be done before Jan 2018.
    >     > * (Varun V) YARN-5673: container-executor write. Given security
    >     > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff
    > may be
    >     > moved to 3.2.
    >     >
    >     > b.4 Additional release drivers
    >     > * More exhaustive upgrade testing from 2.x to 3.x.
    >     >
    >     > c) Regarding branch cut:
    >     >
    >     > We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
    >     > feature planned to 3.2 has to be landed on trunk or B. After feature
    > freeze
    >     > date, whichever comes first.
    >     >
    >     > I've also talked offline with Vinod to get help on release-management
    >     > given this is my first release. He agreed to help do this release
    > jointly.
    >     >
    >     > Thoughts?
    >     >
    >     > Thanks,
    >     > Wangda Tan
    >     >
    >     > [1] https://lists.apache.org/thread.html/
    > c11506c3250c9481852130616b3cb0
    >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org<http://3Cyarn-dev.hadoop.apache.org>%3E
    >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
    > (3.1.0)
    >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
    >     >
    >     >
    >
    >
    >

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Hi All,

Just a reminder about feature freeze date.

Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days from
today), If you've any features which live in a branch and targeted to
3.1.0, please reply this email thread. Ideally, we should finish branch
merging before feature freeze date.

Here's an updated 3.1.0 feature status:

1. Merged features:
* (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
* (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
end-to-end.
* (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
* (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename commits.
* (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
Queues While Doing Queue Mapping.
* (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
* (Zhankun) YARN-5983: FPGA support. Majority implementations completed and
merged to trunk. Except for UI/documentation.

2. Features close to finish:
* (Uma) HDFS-10285: HDFS SPS. There're two remaining
blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of
the two JIRAs?
* (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
anti-affinity support. (Voting thread started).

3. Tentative features:
* (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
containers. Only one pending patch. Plan to finish before Jan 7th.
* (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to be
done before Jan 2018.
* (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
challenging to be done before Jan 2018.
* (Varun V) YARN-5673: container-executor write. Given security refactoring
of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to 3.2.

Thanks,
Wangda


On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma <um...@intel.com>
wrote:

> Sure, Wangda.
>
> Regards,
> Uma
>
> On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>
>     Thanks Uma,
>
>     Could you update this thread once the merge vote started?
>
>     Best,
>     Wangda
>
>     On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     wrote:
>
>     > HI Wangda,
>     >
>     >  Thank you for the head-up mail.
>     >  We are in the branch (HDFS-10285) and trying to push the tasks
> sooner
>     > before the deadline.
>     >
>     > Regards,
>     > Uma
>     >
>     > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>     >
>     >     Hi All,
>     >
>     >     Since we're fast approaching previously proposed feature freeze
> date
>     > (Jan
>     >     30, about 13 days from today). If you've any features which live
> in a
>     >     branch and targeted to 3.1.0, please reply this email thread.
> Ideally,
>     > we
>     >     should finish branch merging before feature freeze date.
>     >
>     >     Here's an updated 3.1.0 feature status:
>     >
>     >     1. Merged & Completed features:
>     >     * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     >     * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
> works
>     >     end-to-end.
>     >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
>     >     * (Steve Loughran): HADOOP-13786: S3Guard committer for
> zero-rename
>     > commits.
>     >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
> of Leaf
>     >     Queues While Doing Queue Mapping.
>     >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >
>     >     2. Features close to finish:
>     >     * (Zhankun) YARN-5983: FPGA support. Majority implementations
>     > completed and
>     >     merged to trunk. Except for UI/documentation.
>     >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done,
> some
>     >     discussions going on about implementation.
>     >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New
> SchedulingRequest and
>     >     anti-affinity support. Close to finish, on track to be merged
> before
>     > Jan 30.
>     >
>     >     3. Tentative features:
>     >     * (Arun Suresh). YARN-5972: Support pausing/freezing
> opportunistic
>     >     containers. Only one pending patch. Plan to finish before Jan
> 7th.
>     >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
> challenging
>     > to be
>     >     done before Jan 2018.
>     >     * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
> Looks
>     >     challenging to be done before Jan 2018.
>     >     * (Varun V) YARN-5673: container-executor write. Given security
>     > refactoring
>     >     of c-e (YARN-6623) is already landed, IMHO other stuff may be
> moved to
>     > 3.2.
>     >
>     >     Thanks,
>     >     Wangda
>     >
>     >
>     >
>     >
>     >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wheeleast@gmail.com
> >
>     > wrote:
>     >
>     >     > Hi all,
>     >     >
>     >     > Congratulations on the 3.0.0-GA release!
>     >     >
>     >     > As we discussed in the previous email thread [1], I'd like to
> restart
>     >     > 3.1.0 release plans.
>     >     >
>     >     > a) Quick summary:
>     >     > a.1 Release status
>     >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As of
> today,
>     >     > there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
>     > blockers and
>     >     > 22 open critical issues.
>     >     >
>     >     > a.2 Release date update
>     >     > Considering delays of 3.0-GA release by month-and-a-half, I
> propose
>     > to
>     >     > move the dates as follows
>     >     >  - feature freeze date from Dec 15, 2017, to Jan 30, 2018 -
> last
>     > date for
>     >     > any branches to get merged too;
>     >     >  - code freeze (blockers & critical only) date to Feb 08, 2018;
>     >     >  - release voting start by Feb 18, 2018, leaving time for at
> least
>     > two RCx
>     >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
>     >     >
>     >     > Unlike before, I added an additional milestone for
>     > release-vote-start so
>     >     > that we can account for voting time-period also.
>     >     >
>     >     > This overall is still 5 1/2 months of release-timeline unlike
> the
>     > faster
>     >     > cadence we hoped for, but this, in my opinion, is the
> best-updated
>     > timeline
>     >     > given the delays of the final release of 3.0-GA.
>     >     >
>     >     > b) Individual feature status:
>     >     > I spoke to several feature owners and checked the status of
>     > un-finished
>     >     > features, following are status of features planned to 3.1.0:
>     >     >
>     >     > b.1 Merged & Completed features:
>     >     > * (Sunil) YARN-5881: Support absolute value in
> CapacityScheduler.
>     >     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk
> and
>     > works
>     >     > end-to-end.
>     >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
>     > services.
>     >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for
> zero-rename
>     >     > commits.
>     >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
> of
>     > Leaf
>     >     > Queues While Doing Queue Mapping.
>     >     >
>     >     > b.2 Features close to finish:
>     >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting
> now.
>     >     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
>     > completed
>     >     > and merged to trunk. Except for UI/documentation.
>     >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
> done, some
>     >     > discussions going on about implementation.
>     >     >
>     >     > b.3 Tentative features:
>     >     > * (Arun Suresh). YARN-5972: Support pausing/freezing
> opportunistic
>     >     > containers. Only one pending patch. Plan to finish before Jan
> 7th.
>     >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     > challenging to
>     >     > be done before Jan 2018.
>     >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New
> SchedulingRequest
>     > and
>     >     > anti-affinity support. Tentative will figure out by Jan 1st.
>     >     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
> Looks
>     >     > challenging to be done before Jan 2018.
>     >     > * (Varun V) YARN-5673: container-executor write. Given security
>     >     > refactoring of c-e (YARN-6623) is already landed, IMHO other
> stuff
>     > may be
>     >     > moved to 3.2.
>     >     >
>     >     > b.4 Additional release drivers
>     >     > * More exhaustive upgrade testing from 2.x to 3.x.
>     >     >
>     >     > c) Regarding branch cut:
>     >     >
>     >     > We will keep pointing trunk to 3.1 and cut branch-3.1 until:
> A. some
>     >     > feature planned to 3.2 has to be landed on trunk or B. After
> feature
>     > freeze
>     >     > date, whichever comes first.
>     >     >
>     >     > I've also talked offline with Vinod to get help on
> release-management
>     >     > given this is my first release. He agreed to help do this
> release
>     > jointly.
>     >     >
>     >     > Thoughts?
>     >     >
>     >     > Thanks,
>     >     > Wangda Tan
>     >     >
>     >     > [1] https://lists.apache.org/thread.html/
>     > c11506c3250c9481852130616b3cb0
>     >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>     >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion
> in
>     > (3.1.0)
>     >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>     >     >
>     >     >
>     >
>     >
>     >
>
>
>

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Hi All,

Just a reminder about feature freeze date.

Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days from
today), If you've any features which live in a branch and targeted to
3.1.0, please reply this email thread. Ideally, we should finish branch
merging before feature freeze date.

Here's an updated 3.1.0 feature status:

1. Merged features:
* (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
* (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
end-to-end.
* (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
* (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename commits.
* (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
Queues While Doing Queue Mapping.
* (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
* (Zhankun) YARN-5983: FPGA support. Majority implementations completed and
merged to trunk. Except for UI/documentation.

2. Features close to finish:
* (Uma) HDFS-10285: HDFS SPS. There're two remaining
blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of
the two JIRAs?
* (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
anti-affinity support. (Voting thread started).

3. Tentative features:
* (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
containers. Only one pending patch. Plan to finish before Jan 7th.
* (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to be
done before Jan 2018.
* (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
challenging to be done before Jan 2018.
* (Varun V) YARN-5673: container-executor write. Given security refactoring
of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to 3.2.

Thanks,
Wangda


On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma <um...@intel.com>
wrote:

> Sure, Wangda.
>
> Regards,
> Uma
>
> On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>
>     Thanks Uma,
>
>     Could you update this thread once the merge vote started?
>
>     Best,
>     Wangda
>
>     On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     wrote:
>
>     > HI Wangda,
>     >
>     >  Thank you for the head-up mail.
>     >  We are in the branch (HDFS-10285) and trying to push the tasks
> sooner
>     > before the deadline.
>     >
>     > Regards,
>     > Uma
>     >
>     > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>     >
>     >     Hi All,
>     >
>     >     Since we're fast approaching previously proposed feature freeze
> date
>     > (Jan
>     >     30, about 13 days from today). If you've any features which live
> in a
>     >     branch and targeted to 3.1.0, please reply this email thread.
> Ideally,
>     > we
>     >     should finish branch merging before feature freeze date.
>     >
>     >     Here's an updated 3.1.0 feature status:
>     >
>     >     1. Merged & Completed features:
>     >     * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     >     * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
> works
>     >     end-to-end.
>     >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
>     >     * (Steve Loughran): HADOOP-13786: S3Guard committer for
> zero-rename
>     > commits.
>     >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
> of Leaf
>     >     Queues While Doing Queue Mapping.
>     >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >
>     >     2. Features close to finish:
>     >     * (Zhankun) YARN-5983: FPGA support. Majority implementations
>     > completed and
>     >     merged to trunk. Except for UI/documentation.
>     >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done,
> some
>     >     discussions going on about implementation.
>     >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New
> SchedulingRequest and
>     >     anti-affinity support. Close to finish, on track to be merged
> before
>     > Jan 30.
>     >
>     >     3. Tentative features:
>     >     * (Arun Suresh). YARN-5972: Support pausing/freezing
> opportunistic
>     >     containers. Only one pending patch. Plan to finish before Jan
> 7th.
>     >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
> challenging
>     > to be
>     >     done before Jan 2018.
>     >     * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
> Looks
>     >     challenging to be done before Jan 2018.
>     >     * (Varun V) YARN-5673: container-executor write. Given security
>     > refactoring
>     >     of c-e (YARN-6623) is already landed, IMHO other stuff may be
> moved to
>     > 3.2.
>     >
>     >     Thanks,
>     >     Wangda
>     >
>     >
>     >
>     >
>     >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wheeleast@gmail.com
> >
>     > wrote:
>     >
>     >     > Hi all,
>     >     >
>     >     > Congratulations on the 3.0.0-GA release!
>     >     >
>     >     > As we discussed in the previous email thread [1], I'd like to
> restart
>     >     > 3.1.0 release plans.
>     >     >
>     >     > a) Quick summary:
>     >     > a.1 Release status
>     >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As of
> today,
>     >     > there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
>     > blockers and
>     >     > 22 open critical issues.
>     >     >
>     >     > a.2 Release date update
>     >     > Considering delays of 3.0-GA release by month-and-a-half, I
> propose
>     > to
>     >     > move the dates as follows
>     >     >  - feature freeze date from Dec 15, 2017, to Jan 30, 2018 -
> last
>     > date for
>     >     > any branches to get merged too;
>     >     >  - code freeze (blockers & critical only) date to Feb 08, 2018;
>     >     >  - release voting start by Feb 18, 2018, leaving time for at
> least
>     > two RCx
>     >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
>     >     >
>     >     > Unlike before, I added an additional milestone for
>     > release-vote-start so
>     >     > that we can account for voting time-period also.
>     >     >
>     >     > This overall is still 5 1/2 months of release-timeline unlike
> the
>     > faster
>     >     > cadence we hoped for, but this, in my opinion, is the
> best-updated
>     > timeline
>     >     > given the delays of the final release of 3.0-GA.
>     >     >
>     >     > b) Individual feature status:
>     >     > I spoke to several feature owners and checked the status of
>     > un-finished
>     >     > features, following are status of features planned to 3.1.0:
>     >     >
>     >     > b.1 Merged & Completed features:
>     >     > * (Sunil) YARN-5881: Support absolute value in
> CapacityScheduler.
>     >     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk
> and
>     > works
>     >     > end-to-end.
>     >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
>     > services.
>     >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for
> zero-rename
>     >     > commits.
>     >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
> of
>     > Leaf
>     >     > Queues While Doing Queue Mapping.
>     >     >
>     >     > b.2 Features close to finish:
>     >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting
> now.
>     >     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
>     > completed
>     >     > and merged to trunk. Except for UI/documentation.
>     >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
> done, some
>     >     > discussions going on about implementation.
>     >     >
>     >     > b.3 Tentative features:
>     >     > * (Arun Suresh). YARN-5972: Support pausing/freezing
> opportunistic
>     >     > containers. Only one pending patch. Plan to finish before Jan
> 7th.
>     >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     > challenging to
>     >     > be done before Jan 2018.
>     >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New
> SchedulingRequest
>     > and
>     >     > anti-affinity support. Tentative will figure out by Jan 1st.
>     >     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
> Looks
>     >     > challenging to be done before Jan 2018.
>     >     > * (Varun V) YARN-5673: container-executor write. Given security
>     >     > refactoring of c-e (YARN-6623) is already landed, IMHO other
> stuff
>     > may be
>     >     > moved to 3.2.
>     >     >
>     >     > b.4 Additional release drivers
>     >     > * More exhaustive upgrade testing from 2.x to 3.x.
>     >     >
>     >     > c) Regarding branch cut:
>     >     >
>     >     > We will keep pointing trunk to 3.1 and cut branch-3.1 until:
> A. some
>     >     > feature planned to 3.2 has to be landed on trunk or B. After
> feature
>     > freeze
>     >     > date, whichever comes first.
>     >     >
>     >     > I've also talked offline with Vinod to get help on
> release-management
>     >     > given this is my first release. He agreed to help do this
> release
>     > jointly.
>     >     >
>     >     > Thoughts?
>     >     >
>     >     > Thanks,
>     >     > Wangda Tan
>     >     >
>     >     > [1] https://lists.apache.org/thread.html/
>     > c11506c3250c9481852130616b3cb0
>     >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>     >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion
> in
>     > (3.1.0)
>     >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>     >     >
>     >     >
>     >
>     >
>     >
>
>
>

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Hi All,

Just a reminder about feature freeze date.

Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days from
today), If you've any features which live in a branch and targeted to
3.1.0, please reply this email thread. Ideally, we should finish branch
merging before feature freeze date.

Here's an updated 3.1.0 feature status:

1. Merged features:
* (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
* (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
end-to-end.
* (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
* (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename commits.
* (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
Queues While Doing Queue Mapping.
* (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
* (Zhankun) YARN-5983: FPGA support. Majority implementations completed and
merged to trunk. Except for UI/documentation.

2. Features close to finish:
* (Uma) HDFS-10285: HDFS SPS. There're two remaining
blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of
the two JIRAs?
* (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
anti-affinity support. (Voting thread started).

3. Tentative features:
* (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
containers. Only one pending patch. Plan to finish before Jan 7th.
* (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to be
done before Jan 2018.
* (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
challenging to be done before Jan 2018.
* (Varun V) YARN-5673: container-executor write. Given security refactoring
of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to 3.2.

Thanks,
Wangda


On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma <um...@intel.com>
wrote:

> Sure, Wangda.
>
> Regards,
> Uma
>
> On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>
>     Thanks Uma,
>
>     Could you update this thread once the merge vote started?
>
>     Best,
>     Wangda
>
>     On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     wrote:
>
>     > HI Wangda,
>     >
>     >  Thank you for the head-up mail.
>     >  We are in the branch (HDFS-10285) and trying to push the tasks
> sooner
>     > before the deadline.
>     >
>     > Regards,
>     > Uma
>     >
>     > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>     >
>     >     Hi All,
>     >
>     >     Since we're fast approaching previously proposed feature freeze
> date
>     > (Jan
>     >     30, about 13 days from today). If you've any features which live
> in a
>     >     branch and targeted to 3.1.0, please reply this email thread.
> Ideally,
>     > we
>     >     should finish branch merging before feature freeze date.
>     >
>     >     Here's an updated 3.1.0 feature status:
>     >
>     >     1. Merged & Completed features:
>     >     * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     >     * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
> works
>     >     end-to-end.
>     >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
>     >     * (Steve Loughran): HADOOP-13786: S3Guard committer for
> zero-rename
>     > commits.
>     >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
> of Leaf
>     >     Queues While Doing Queue Mapping.
>     >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >
>     >     2. Features close to finish:
>     >     * (Zhankun) YARN-5983: FPGA support. Majority implementations
>     > completed and
>     >     merged to trunk. Except for UI/documentation.
>     >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done,
> some
>     >     discussions going on about implementation.
>     >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New
> SchedulingRequest and
>     >     anti-affinity support. Close to finish, on track to be merged
> before
>     > Jan 30.
>     >
>     >     3. Tentative features:
>     >     * (Arun Suresh). YARN-5972: Support pausing/freezing
> opportunistic
>     >     containers. Only one pending patch. Plan to finish before Jan
> 7th.
>     >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
> challenging
>     > to be
>     >     done before Jan 2018.
>     >     * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
> Looks
>     >     challenging to be done before Jan 2018.
>     >     * (Varun V) YARN-5673: container-executor write. Given security
>     > refactoring
>     >     of c-e (YARN-6623) is already landed, IMHO other stuff may be
> moved to
>     > 3.2.
>     >
>     >     Thanks,
>     >     Wangda
>     >
>     >
>     >
>     >
>     >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wheeleast@gmail.com
> >
>     > wrote:
>     >
>     >     > Hi all,
>     >     >
>     >     > Congratulations on the 3.0.0-GA release!
>     >     >
>     >     > As we discussed in the previous email thread [1], I'd like to
> restart
>     >     > 3.1.0 release plans.
>     >     >
>     >     > a) Quick summary:
>     >     > a.1 Release status
>     >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As of
> today,
>     >     > there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
>     > blockers and
>     >     > 22 open critical issues.
>     >     >
>     >     > a.2 Release date update
>     >     > Considering delays of 3.0-GA release by month-and-a-half, I
> propose
>     > to
>     >     > move the dates as follows
>     >     >  - feature freeze date from Dec 15, 2017, to Jan 30, 2018 -
> last
>     > date for
>     >     > any branches to get merged too;
>     >     >  - code freeze (blockers & critical only) date to Feb 08, 2018;
>     >     >  - release voting start by Feb 18, 2018, leaving time for at
> least
>     > two RCx
>     >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
>     >     >
>     >     > Unlike before, I added an additional milestone for
>     > release-vote-start so
>     >     > that we can account for voting time-period also.
>     >     >
>     >     > This overall is still 5 1/2 months of release-timeline unlike
> the
>     > faster
>     >     > cadence we hoped for, but this, in my opinion, is the
> best-updated
>     > timeline
>     >     > given the delays of the final release of 3.0-GA.
>     >     >
>     >     > b) Individual feature status:
>     >     > I spoke to several feature owners and checked the status of
>     > un-finished
>     >     > features, following are status of features planned to 3.1.0:
>     >     >
>     >     > b.1 Merged & Completed features:
>     >     > * (Sunil) YARN-5881: Support absolute value in
> CapacityScheduler.
>     >     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk
> and
>     > works
>     >     > end-to-end.
>     >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
>     > services.
>     >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for
> zero-rename
>     >     > commits.
>     >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
> of
>     > Leaf
>     >     > Queues While Doing Queue Mapping.
>     >     >
>     >     > b.2 Features close to finish:
>     >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting
> now.
>     >     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
>     > completed
>     >     > and merged to trunk. Except for UI/documentation.
>     >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
> done, some
>     >     > discussions going on about implementation.
>     >     >
>     >     > b.3 Tentative features:
>     >     > * (Arun Suresh). YARN-5972: Support pausing/freezing
> opportunistic
>     >     > containers. Only one pending patch. Plan to finish before Jan
> 7th.
>     >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     > challenging to
>     >     > be done before Jan 2018.
>     >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New
> SchedulingRequest
>     > and
>     >     > anti-affinity support. Tentative will figure out by Jan 1st.
>     >     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
> Looks
>     >     > challenging to be done before Jan 2018.
>     >     > * (Varun V) YARN-5673: container-executor write. Given security
>     >     > refactoring of c-e (YARN-6623) is already landed, IMHO other
> stuff
>     > may be
>     >     > moved to 3.2.
>     >     >
>     >     > b.4 Additional release drivers
>     >     > * More exhaustive upgrade testing from 2.x to 3.x.
>     >     >
>     >     > c) Regarding branch cut:
>     >     >
>     >     > We will keep pointing trunk to 3.1 and cut branch-3.1 until:
> A. some
>     >     > feature planned to 3.2 has to be landed on trunk or B. After
> feature
>     > freeze
>     >     > date, whichever comes first.
>     >     >
>     >     > I've also talked offline with Vinod to get help on
> release-management
>     >     > given this is my first release. He agreed to help do this
> release
>     > jointly.
>     >     >
>     >     > Thoughts?
>     >     >
>     >     > Thanks,
>     >     > Wangda Tan
>     >     >
>     >     > [1] https://lists.apache.org/thread.html/
>     > c11506c3250c9481852130616b3cb0
>     >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>     >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion
> in
>     > (3.1.0)
>     >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>     >     >
>     >     >
>     >
>     >
>     >
>
>
>

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Hi All,

Just a reminder about feature freeze date.

Feature freeze date for 3.1.0 release is Jan 30 PST (about 4 days from
today), If you've any features which live in a branch and targeted to
3.1.0, please reply this email thread. Ideally, we should finish branch
merging before feature freeze date.

Here's an updated 3.1.0 feature status:

1. Merged features:
* (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
* (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
end-to-end.
* (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
* (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename commits.
* (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
Queues While Doing Queue Mapping.
* (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
* (Zhankun) YARN-5983: FPGA support. Majority implementations completed and
merged to trunk. Except for UI/documentation.

2. Features close to finish:
* (Uma) HDFS-10285: HDFS SPS. There're two remaining
blockers: HDFS-12995/HDFS-13050. @Uma could you update what's the ETA of
the two JIRAs?
* (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
anti-affinity support. (Voting thread started).

3. Tentative features:
* (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
containers. Only one pending patch. Plan to finish before Jan 7th.
* (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to be
done before Jan 2018.
* (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
challenging to be done before Jan 2018.
* (Varun V) YARN-5673: container-executor write. Given security refactoring
of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to 3.2.

Thanks,
Wangda


On Mon, Jan 22, 2018 at 1:49 PM, Gangumalla, Uma <um...@intel.com>
wrote:

> Sure, Wangda.
>
> Regards,
> Uma
>
> On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>
>     Thanks Uma,
>
>     Could you update this thread once the merge vote started?
>
>     Best,
>     Wangda
>
>     On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <
> uma.gangumalla@intel.com>
>     wrote:
>
>     > HI Wangda,
>     >
>     >  Thank you for the head-up mail.
>     >  We are in the branch (HDFS-10285) and trying to push the tasks
> sooner
>     > before the deadline.
>     >
>     > Regards,
>     > Uma
>     >
>     > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>     >
>     >     Hi All,
>     >
>     >     Since we're fast approaching previously proposed feature freeze
> date
>     > (Jan
>     >     30, about 13 days from today). If you've any features which live
> in a
>     >     branch and targeted to 3.1.0, please reply this email thread.
> Ideally,
>     > we
>     >     should finish branch merging before feature freeze date.
>     >
>     >     Here's an updated 3.1.0 feature status:
>     >
>     >     1. Merged & Completed features:
>     >     * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     >     * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
> works
>     >     end-to-end.
>     >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
>     >     * (Steve Loughran): HADOOP-13786: S3Guard committer for
> zero-rename
>     > commits.
>     >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
> of Leaf
>     >     Queues While Doing Queue Mapping.
>     >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>     >
>     >     2. Features close to finish:
>     >     * (Zhankun) YARN-5983: FPGA support. Majority implementations
>     > completed and
>     >     merged to trunk. Except for UI/documentation.
>     >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done,
> some
>     >     discussions going on about implementation.
>     >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New
> SchedulingRequest and
>     >     anti-affinity support. Close to finish, on track to be merged
> before
>     > Jan 30.
>     >
>     >     3. Tentative features:
>     >     * (Arun Suresh). YARN-5972: Support pausing/freezing
> opportunistic
>     >     containers. Only one pending patch. Plan to finish before Jan
> 7th.
>     >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
> challenging
>     > to be
>     >     done before Jan 2018.
>     >     * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
> Looks
>     >     challenging to be done before Jan 2018.
>     >     * (Varun V) YARN-5673: container-executor write. Given security
>     > refactoring
>     >     of c-e (YARN-6623) is already landed, IMHO other stuff may be
> moved to
>     > 3.2.
>     >
>     >     Thanks,
>     >     Wangda
>     >
>     >
>     >
>     >
>     >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wheeleast@gmail.com
> >
>     > wrote:
>     >
>     >     > Hi all,
>     >     >
>     >     > Congratulations on the 3.0.0-GA release!
>     >     >
>     >     > As we discussed in the previous email thread [1], I'd like to
> restart
>     >     > 3.1.0 release plans.
>     >     >
>     >     > a) Quick summary:
>     >     > a.1 Release status
>     >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As of
> today,
>     >     > there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
>     > blockers and
>     >     > 22 open critical issues.
>     >     >
>     >     > a.2 Release date update
>     >     > Considering delays of 3.0-GA release by month-and-a-half, I
> propose
>     > to
>     >     > move the dates as follows
>     >     >  - feature freeze date from Dec 15, 2017, to Jan 30, 2018 -
> last
>     > date for
>     >     > any branches to get merged too;
>     >     >  - code freeze (blockers & critical only) date to Feb 08, 2018;
>     >     >  - release voting start by Feb 18, 2018, leaving time for at
> least
>     > two RCx
>     >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
>     >     >
>     >     > Unlike before, I added an additional milestone for
>     > release-vote-start so
>     >     > that we can account for voting time-period also.
>     >     >
>     >     > This overall is still 5 1/2 months of release-timeline unlike
> the
>     > faster
>     >     > cadence we hoped for, but this, in my opinion, is the
> best-updated
>     > timeline
>     >     > given the delays of the final release of 3.0-GA.
>     >     >
>     >     > b) Individual feature status:
>     >     > I spoke to several feature owners and checked the status of
>     > un-finished
>     >     > features, following are status of features planned to 3.1.0:
>     >     >
>     >     > b.1 Merged & Completed features:
>     >     > * (Sunil) YARN-5881: Support absolute value in
> CapacityScheduler.
>     >     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk
> and
>     > works
>     >     > end-to-end.
>     >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
>     > services.
>     >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for
> zero-rename
>     >     > commits.
>     >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation
> of
>     > Leaf
>     >     > Queues While Doing Queue Mapping.
>     >     >
>     >     > b.2 Features close to finish:
>     >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting
> now.
>     >     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
>     > completed
>     >     > and merged to trunk. Except for UI/documentation.
>     >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations are
> done, some
>     >     > discussions going on about implementation.
>     >     >
>     >     > b.3 Tentative features:
>     >     > * (Arun Suresh). YARN-5972: Support pausing/freezing
> opportunistic
>     >     > containers. Only one pending patch. Plan to finish before Jan
> 7th.
>     >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
>     > challenging to
>     >     > be done before Jan 2018.
>     >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New
> SchedulingRequest
>     > and
>     >     > anti-affinity support. Tentative will figure out by Jan 1st.
>     >     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240.
> Looks
>     >     > challenging to be done before Jan 2018.
>     >     > * (Varun V) YARN-5673: container-executor write. Given security
>     >     > refactoring of c-e (YARN-6623) is already landed, IMHO other
> stuff
>     > may be
>     >     > moved to 3.2.
>     >     >
>     >     > b.4 Additional release drivers
>     >     > * More exhaustive upgrade testing from 2.x to 3.x.
>     >     >
>     >     > c) Regarding branch cut:
>     >     >
>     >     > We will keep pointing trunk to 3.1 and cut branch-3.1 until:
> A. some
>     >     > feature planned to 3.2 has to be landed on trunk or B. After
> feature
>     > freeze
>     >     > date, whichever comes first.
>     >     >
>     >     > I've also talked offline with Vinod to get help on
> release-management
>     >     > given this is my first release. He agreed to help do this
> release
>     > jointly.
>     >     >
>     >     > Thoughts?
>     >     >
>     >     > Thanks,
>     >     > Wangda Tan
>     >     >
>     >     > [1] https://lists.apache.org/thread.html/
>     > c11506c3250c9481852130616b3cb0
>     >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>     >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion
> in
>     > (3.1.0)
>     >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>     >     >
>     >     >
>     >
>     >
>     >
>
>
>

Re: Hadoop 3.1.0 release discussion

["Gangumalla, Uma" <um...@intel.com>]

Sure, Wangda.

Regards,
Uma

On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com> wrote:

    Thanks Uma,
    
    Could you update this thread once the merge vote started?
    
    Best,
    Wangda
    
    On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <um...@intel.com>
    wrote:
    
    > HI Wangda,
    >
    >  Thank you for the head-up mail.
    >  We are in the branch (HDFS-10285) and trying to push the tasks sooner
    > before the deadline.
    >
    > Regards,
    > Uma
    >
    > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com> wrote:
    >
    >     Hi All,
    >
    >     Since we're fast approaching previously proposed feature freeze date
    > (Jan
    >     30, about 13 days from today). If you've any features which live in a
    >     branch and targeted to 3.1.0, please reply this email thread. Ideally,
    > we
    >     should finish branch merging before feature freeze date.
    >
    >     Here's an updated 3.1.0 feature status:
    >
    >     1. Merged & Completed features:
    >     * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
    >     * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
    >     end-to-end.
    >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
    >     * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
    > commits.
    >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
    >     Queues While Doing Queue Mapping.
    >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
    >
    >     2. Features close to finish:
    >     * (Zhankun) YARN-5983: FPGA support. Majority implementations
    > completed and
    >     merged to trunk. Except for UI/documentation.
    >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
    >     discussions going on about implementation.
    >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
    >     anti-affinity support. Close to finish, on track to be merged before
    > Jan 30.
    >
    >     3. Tentative features:
    >     * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
    >     containers. Only one pending patch. Plan to finish before Jan 7th.
    >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging
    > to be
    >     done before Jan 2018.
    >     * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
    >     challenging to be done before Jan 2018.
    >     * (Varun V) YARN-5673: container-executor write. Given security
    > refactoring
    >     of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to
    > 3.2.
    >
    >     Thanks,
    >     Wangda
    >
    >
    >
    >
    >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com>
    > wrote:
    >
    >     > Hi all,
    >     >
    >     > Congratulations on the 3.0.0-GA release!
    >     >
    >     > As we discussed in the previous email thread [1], I'd like to restart
    >     > 3.1.0 release plans.
    >     >
    >     > a) Quick summary:
    >     > a.1 Release status
    >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
    >     > there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
    > blockers and
    >     > 22 open critical issues.
    >     >
    >     > a.2 Release date update
    >     > Considering delays of 3.0-GA release by month-and-a-half, I propose
    > to
    >     > move the dates as follows
    >     >  - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last
    > date for
    >     > any branches to get merged too;
    >     >  - code freeze (blockers & critical only) date to Feb 08, 2018;
    >     >  - release voting start by Feb 18, 2018, leaving time for at least
    > two RCx
    >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
    >     >
    >     > Unlike before, I added an additional milestone for
    > release-vote-start so
    >     > that we can account for voting time-period also.
    >     >
    >     > This overall is still 5 1/2 months of release-timeline unlike the
    > faster
    >     > cadence we hoped for, but this, in my opinion, is the best-updated
    > timeline
    >     > given the delays of the final release of 3.0-GA.
    >     >
    >     > b) Individual feature status:
    >     > I spoke to several feature owners and checked the status of
    > un-finished
    >     > features, following are status of features planned to 3.1.0:
    >     >
    >     > b.1 Merged & Completed features:
    >     > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
    >     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
    > works
    >     > end-to-end.
    >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
    > services.
    >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
    >     > commits.
    >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of
    > Leaf
    >     > Queues While Doing Queue Mapping.
    >     >
    >     > b.2 Features close to finish:
    >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
    >     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
    > completed
    >     > and merged to trunk. Except for UI/documentation.
    >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
    >     > discussions going on about implementation.
    >     >
    >     > b.3 Tentative features:
    >     > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
    >     > containers. Only one pending patch. Plan to finish before Jan 7th.
    >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
    > challenging to
    >     > be done before Jan 2018.
    >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest
    > and
    >     > anti-affinity support. Tentative will figure out by Jan 1st.
    >     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
    >     > challenging to be done before Jan 2018.
    >     > * (Varun V) YARN-5673: container-executor write. Given security
    >     > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff
    > may be
    >     > moved to 3.2.
    >     >
    >     > b.4 Additional release drivers
    >     > * More exhaustive upgrade testing from 2.x to 3.x.
    >     >
    >     > c) Regarding branch cut:
    >     >
    >     > We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
    >     > feature planned to 3.2 has to be landed on trunk or B. After feature
    > freeze
    >     > date, whichever comes first.
    >     >
    >     > I've also talked offline with Vinod to get help on release-management
    >     > given this is my first release. He agreed to help do this release
    > jointly.
    >     >
    >     > Thoughts?
    >     >
    >     > Thanks,
    >     > Wangda Tan
    >     >
    >     > [1] https://lists.apache.org/thread.html/
    > c11506c3250c9481852130616b3cb0
    >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
    >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
    > (3.1.0)
    >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
    >     >
    >     >
    >
    >
    >
    


---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

Re: Hadoop 3.1.0 release discussion

["Gangumalla, Uma" <um...@intel.com>]

Sure, Wangda.

Regards,
Uma

On 1/18/18, 10:19 AM, "Wangda Tan" <wh...@gmail.com> wrote:

    Thanks Uma,
    
    Could you update this thread once the merge vote started?
    
    Best,
    Wangda
    
    On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <um...@intel.com>
    wrote:
    
    > HI Wangda,
    >
    >  Thank you for the head-up mail.
    >  We are in the branch (HDFS-10285) and trying to push the tasks sooner
    > before the deadline.
    >
    > Regards,
    > Uma
    >
    > On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com> wrote:
    >
    >     Hi All,
    >
    >     Since we're fast approaching previously proposed feature freeze date
    > (Jan
    >     30, about 13 days from today). If you've any features which live in a
    >     branch and targeted to 3.1.0, please reply this email thread. Ideally,
    > we
    >     should finish branch merging before feature freeze date.
    >
    >     Here's an updated 3.1.0 feature status:
    >
    >     1. Merged & Completed features:
    >     * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
    >     * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
    >     end-to-end.
    >     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
    >     * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
    > commits.
    >     * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
    >     Queues While Doing Queue Mapping.
    >     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
    >
    >     2. Features close to finish:
    >     * (Zhankun) YARN-5983: FPGA support. Majority implementations
    > completed and
    >     merged to trunk. Except for UI/documentation.
    >     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
    >     discussions going on about implementation.
    >     * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
    >     anti-affinity support. Close to finish, on track to be merged before
    > Jan 30.
    >
    >     3. Tentative features:
    >     * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
    >     containers. Only one pending patch. Plan to finish before Jan 7th.
    >     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging
    > to be
    >     done before Jan 2018.
    >     * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
    >     challenging to be done before Jan 2018.
    >     * (Varun V) YARN-5673: container-executor write. Given security
    > refactoring
    >     of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to
    > 3.2.
    >
    >     Thanks,
    >     Wangda
    >
    >
    >
    >
    >     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com>
    > wrote:
    >
    >     > Hi all,
    >     >
    >     > Congratulations on the 3.0.0-GA release!
    >     >
    >     > As we discussed in the previous email thread [1], I'd like to restart
    >     > 3.1.0 release plans.
    >     >
    >     > a) Quick summary:
    >     > a.1 Release status
    >     > We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
    >     > there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
    > blockers and
    >     > 22 open critical issues.
    >     >
    >     > a.2 Release date update
    >     > Considering delays of 3.0-GA release by month-and-a-half, I propose
    > to
    >     > move the dates as follows
    >     >  - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last
    > date for
    >     > any branches to get merged too;
    >     >  - code freeze (blockers & critical only) date to Feb 08, 2018;
    >     >  - release voting start by Feb 18, 2018, leaving time for at least
    > two RCx
    >     >  - release date from Jan 15, 2018, to Feb 28, 2018;
    >     >
    >     > Unlike before, I added an additional milestone for
    > release-vote-start so
    >     > that we can account for voting time-period also.
    >     >
    >     > This overall is still 5 1/2 months of release-timeline unlike the
    > faster
    >     > cadence we hoped for, but this, in my opinion, is the best-updated
    > timeline
    >     > given the delays of the final release of 3.0-GA.
    >     >
    >     > b) Individual feature status:
    >     > I spoke to several feature owners and checked the status of
    > un-finished
    >     > features, following are status of features planned to 3.1.0:
    >     >
    >     > b.1 Merged & Completed features:
    >     > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
    >     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
    > works
    >     > end-to-end.
    >     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
    > services.
    >     > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
    >     > commits.
    >     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of
    > Leaf
    >     > Queues While Doing Queue Mapping.
    >     >
    >     > b.2 Features close to finish:
    >     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
    >     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
    > completed
    >     > and merged to trunk. Except for UI/documentation.
    >     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
    >     > discussions going on about implementation.
    >     >
    >     > b.3 Tentative features:
    >     > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
    >     > containers. Only one pending patch. Plan to finish before Jan 7th.
    >     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
    > challenging to
    >     > be done before Jan 2018.
    >     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest
    > and
    >     > anti-affinity support. Tentative will figure out by Jan 1st.
    >     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
    >     > challenging to be done before Jan 2018.
    >     > * (Varun V) YARN-5673: container-executor write. Given security
    >     > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff
    > may be
    >     > moved to 3.2.
    >     >
    >     > b.4 Additional release drivers
    >     > * More exhaustive upgrade testing from 2.x to 3.x.
    >     >
    >     > c) Regarding branch cut:
    >     >
    >     > We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
    >     > feature planned to 3.2 has to be landed on trunk or B. After feature
    > freeze
    >     > date, whichever comes first.
    >     >
    >     > I've also talked offline with Vinod to get help on release-management
    >     > given this is my first release. He agreed to help do this release
    > jointly.
    >     >
    >     > Thoughts?
    >     >
    >     > Thanks,
    >     > Wangda Tan
    >     >
    >     > [1] https://lists.apache.org/thread.html/
    > c11506c3250c9481852130616b3cb0
    >     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
    >     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
    > (3.1.0)
    >     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
    >     >
    >     >
    >
    >
    >
    


---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Thanks Uma,

Could you update this thread once the merge vote started?

Best,
Wangda

On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <um...@intel.com>
wrote:

> HI Wangda,
>
>  Thank you for the head-up mail.
>  We are in the branch (HDFS-10285) and trying to push the tasks sooner
> before the deadline.
>
> Regards,
> Uma
>
> On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>
>     Hi All,
>
>     Since we're fast approaching previously proposed feature freeze date
> (Jan
>     30, about 13 days from today). If you've any features which live in a
>     branch and targeted to 3.1.0, please reply this email thread. Ideally,
> we
>     should finish branch merging before feature freeze date.
>
>     Here's an updated 3.1.0 feature status:
>
>     1. Merged & Completed features:
>     * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>     end-to-end.
>     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>     * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
> commits.
>     * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>     Queues While Doing Queue Mapping.
>     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>
>     2. Features close to finish:
>     * (Zhankun) YARN-5983: FPGA support. Majority implementations
> completed and
>     merged to trunk. Except for UI/documentation.
>     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>     discussions going on about implementation.
>     * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>     anti-affinity support. Close to finish, on track to be merged before
> Jan 30.
>
>     3. Tentative features:
>     * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>     containers. Only one pending patch. Plan to finish before Jan 7th.
>     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging
> to be
>     done before Jan 2018.
>     * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>     challenging to be done before Jan 2018.
>     * (Varun V) YARN-5673: container-executor write. Given security
> refactoring
>     of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to
> 3.2.
>
>     Thanks,
>     Wangda
>
>
>
>
>     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com>
> wrote:
>
>     > Hi all,
>     >
>     > Congratulations on the 3.0.0-GA release!
>     >
>     > As we discussed in the previous email thread [1], I'd like to restart
>     > 3.1.0 release plans.
>     >
>     > a) Quick summary:
>     > a.1 Release status
>     > We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
>     > there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
> blockers and
>     > 22 open critical issues.
>     >
>     > a.2 Release date update
>     > Considering delays of 3.0-GA release by month-and-a-half, I propose
> to
>     > move the dates as follows
>     >  - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last
> date for
>     > any branches to get merged too;
>     >  - code freeze (blockers & critical only) date to Feb 08, 2018;
>     >  - release voting start by Feb 18, 2018, leaving time for at least
> two RCx
>     >  - release date from Jan 15, 2018, to Feb 28, 2018;
>     >
>     > Unlike before, I added an additional milestone for
> release-vote-start so
>     > that we can account for voting time-period also.
>     >
>     > This overall is still 5 1/2 months of release-timeline unlike the
> faster
>     > cadence we hoped for, but this, in my opinion, is the best-updated
> timeline
>     > given the delays of the final release of 3.0-GA.
>     >
>     > b) Individual feature status:
>     > I spoke to several feature owners and checked the status of
> un-finished
>     > features, following are status of features planned to 3.1.0:
>     >
>     > b.1 Merged & Completed features:
>     > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
> works
>     > end-to-end.
>     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
>     > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>     > commits.
>     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of
> Leaf
>     > Queues While Doing Queue Mapping.
>     >
>     > b.2 Features close to finish:
>     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
>     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
> completed
>     > and merged to trunk. Except for UI/documentation.
>     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>     > discussions going on about implementation.
>     >
>     > b.3 Tentative features:
>     > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>     > containers. Only one pending patch. Plan to finish before Jan 7th.
>     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
> challenging to
>     > be done before Jan 2018.
>     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest
> and
>     > anti-affinity support. Tentative will figure out by Jan 1st.
>     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>     > challenging to be done before Jan 2018.
>     > * (Varun V) YARN-5673: container-executor write. Given security
>     > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff
> may be
>     > moved to 3.2.
>     >
>     > b.4 Additional release drivers
>     > * More exhaustive upgrade testing from 2.x to 3.x.
>     >
>     > c) Regarding branch cut:
>     >
>     > We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
>     > feature planned to 3.2 has to be landed on trunk or B. After feature
> freeze
>     > date, whichever comes first.
>     >
>     > I've also talked offline with Vinod to get help on release-management
>     > given this is my first release. He agreed to help do this release
> jointly.
>     >
>     > Thoughts?
>     >
>     > Thanks,
>     > Wangda Tan
>     >
>     > [1] https://lists.apache.org/thread.html/
> c11506c3250c9481852130616b3cb0
>     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.1.0)
>     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>     >
>     >
>
>
>

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Thanks Uma,

Could you update this thread once the merge vote started?

Best,
Wangda

On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <um...@intel.com>
wrote:

> HI Wangda,
>
>  Thank you for the head-up mail.
>  We are in the branch (HDFS-10285) and trying to push the tasks sooner
> before the deadline.
>
> Regards,
> Uma
>
> On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>
>     Hi All,
>
>     Since we're fast approaching previously proposed feature freeze date
> (Jan
>     30, about 13 days from today). If you've any features which live in a
>     branch and targeted to 3.1.0, please reply this email thread. Ideally,
> we
>     should finish branch merging before feature freeze date.
>
>     Here's an updated 3.1.0 feature status:
>
>     1. Merged & Completed features:
>     * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>     end-to-end.
>     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>     * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
> commits.
>     * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>     Queues While Doing Queue Mapping.
>     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>
>     2. Features close to finish:
>     * (Zhankun) YARN-5983: FPGA support. Majority implementations
> completed and
>     merged to trunk. Except for UI/documentation.
>     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>     discussions going on about implementation.
>     * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>     anti-affinity support. Close to finish, on track to be merged before
> Jan 30.
>
>     3. Tentative features:
>     * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>     containers. Only one pending patch. Plan to finish before Jan 7th.
>     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging
> to be
>     done before Jan 2018.
>     * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>     challenging to be done before Jan 2018.
>     * (Varun V) YARN-5673: container-executor write. Given security
> refactoring
>     of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to
> 3.2.
>
>     Thanks,
>     Wangda
>
>
>
>
>     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com>
> wrote:
>
>     > Hi all,
>     >
>     > Congratulations on the 3.0.0-GA release!
>     >
>     > As we discussed in the previous email thread [1], I'd like to restart
>     > 3.1.0 release plans.
>     >
>     > a) Quick summary:
>     > a.1 Release status
>     > We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
>     > there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
> blockers and
>     > 22 open critical issues.
>     >
>     > a.2 Release date update
>     > Considering delays of 3.0-GA release by month-and-a-half, I propose
> to
>     > move the dates as follows
>     >  - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last
> date for
>     > any branches to get merged too;
>     >  - code freeze (blockers & critical only) date to Feb 08, 2018;
>     >  - release voting start by Feb 18, 2018, leaving time for at least
> two RCx
>     >  - release date from Jan 15, 2018, to Feb 28, 2018;
>     >
>     > Unlike before, I added an additional milestone for
> release-vote-start so
>     > that we can account for voting time-period also.
>     >
>     > This overall is still 5 1/2 months of release-timeline unlike the
> faster
>     > cadence we hoped for, but this, in my opinion, is the best-updated
> timeline
>     > given the delays of the final release of 3.0-GA.
>     >
>     > b) Individual feature status:
>     > I spoke to several feature owners and checked the status of
> un-finished
>     > features, following are status of features planned to 3.1.0:
>     >
>     > b.1 Merged & Completed features:
>     > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
> works
>     > end-to-end.
>     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
>     > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>     > commits.
>     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of
> Leaf
>     > Queues While Doing Queue Mapping.
>     >
>     > b.2 Features close to finish:
>     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
>     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
> completed
>     > and merged to trunk. Except for UI/documentation.
>     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>     > discussions going on about implementation.
>     >
>     > b.3 Tentative features:
>     > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>     > containers. Only one pending patch. Plan to finish before Jan 7th.
>     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
> challenging to
>     > be done before Jan 2018.
>     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest
> and
>     > anti-affinity support. Tentative will figure out by Jan 1st.
>     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>     > challenging to be done before Jan 2018.
>     > * (Varun V) YARN-5673: container-executor write. Given security
>     > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff
> may be
>     > moved to 3.2.
>     >
>     > b.4 Additional release drivers
>     > * More exhaustive upgrade testing from 2.x to 3.x.
>     >
>     > c) Regarding branch cut:
>     >
>     > We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
>     > feature planned to 3.2 has to be landed on trunk or B. After feature
> freeze
>     > date, whichever comes first.
>     >
>     > I've also talked offline with Vinod to get help on release-management
>     > given this is my first release. He agreed to help do this release
> jointly.
>     >
>     > Thoughts?
>     >
>     > Thanks,
>     > Wangda Tan
>     >
>     > [1] https://lists.apache.org/thread.html/
> c11506c3250c9481852130616b3cb0
>     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.1.0)
>     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>     >
>     >
>
>
>

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Thanks Uma,

Could you update this thread once the merge vote started?

Best,
Wangda

On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <um...@intel.com>
wrote:

> HI Wangda,
>
>  Thank you for the head-up mail.
>  We are in the branch (HDFS-10285) and trying to push the tasks sooner
> before the deadline.
>
> Regards,
> Uma
>
> On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>
>     Hi All,
>
>     Since we're fast approaching previously proposed feature freeze date
> (Jan
>     30, about 13 days from today). If you've any features which live in a
>     branch and targeted to 3.1.0, please reply this email thread. Ideally,
> we
>     should finish branch merging before feature freeze date.
>
>     Here's an updated 3.1.0 feature status:
>
>     1. Merged & Completed features:
>     * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>     end-to-end.
>     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>     * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
> commits.
>     * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>     Queues While Doing Queue Mapping.
>     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>
>     2. Features close to finish:
>     * (Zhankun) YARN-5983: FPGA support. Majority implementations
> completed and
>     merged to trunk. Except for UI/documentation.
>     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>     discussions going on about implementation.
>     * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>     anti-affinity support. Close to finish, on track to be merged before
> Jan 30.
>
>     3. Tentative features:
>     * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>     containers. Only one pending patch. Plan to finish before Jan 7th.
>     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging
> to be
>     done before Jan 2018.
>     * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>     challenging to be done before Jan 2018.
>     * (Varun V) YARN-5673: container-executor write. Given security
> refactoring
>     of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to
> 3.2.
>
>     Thanks,
>     Wangda
>
>
>
>
>     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com>
> wrote:
>
>     > Hi all,
>     >
>     > Congratulations on the 3.0.0-GA release!
>     >
>     > As we discussed in the previous email thread [1], I'd like to restart
>     > 3.1.0 release plans.
>     >
>     > a) Quick summary:
>     > a.1 Release status
>     > We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
>     > there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
> blockers and
>     > 22 open critical issues.
>     >
>     > a.2 Release date update
>     > Considering delays of 3.0-GA release by month-and-a-half, I propose
> to
>     > move the dates as follows
>     >  - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last
> date for
>     > any branches to get merged too;
>     >  - code freeze (blockers & critical only) date to Feb 08, 2018;
>     >  - release voting start by Feb 18, 2018, leaving time for at least
> two RCx
>     >  - release date from Jan 15, 2018, to Feb 28, 2018;
>     >
>     > Unlike before, I added an additional milestone for
> release-vote-start so
>     > that we can account for voting time-period also.
>     >
>     > This overall is still 5 1/2 months of release-timeline unlike the
> faster
>     > cadence we hoped for, but this, in my opinion, is the best-updated
> timeline
>     > given the delays of the final release of 3.0-GA.
>     >
>     > b) Individual feature status:
>     > I spoke to several feature owners and checked the status of
> un-finished
>     > features, following are status of features planned to 3.1.0:
>     >
>     > b.1 Merged & Completed features:
>     > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
> works
>     > end-to-end.
>     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
>     > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>     > commits.
>     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of
> Leaf
>     > Queues While Doing Queue Mapping.
>     >
>     > b.2 Features close to finish:
>     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
>     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
> completed
>     > and merged to trunk. Except for UI/documentation.
>     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>     > discussions going on about implementation.
>     >
>     > b.3 Tentative features:
>     > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>     > containers. Only one pending patch. Plan to finish before Jan 7th.
>     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
> challenging to
>     > be done before Jan 2018.
>     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest
> and
>     > anti-affinity support. Tentative will figure out by Jan 1st.
>     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>     > challenging to be done before Jan 2018.
>     > * (Varun V) YARN-5673: container-executor write. Given security
>     > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff
> may be
>     > moved to 3.2.
>     >
>     > b.4 Additional release drivers
>     > * More exhaustive upgrade testing from 2.x to 3.x.
>     >
>     > c) Regarding branch cut:
>     >
>     > We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
>     > feature planned to 3.2 has to be landed on trunk or B. After feature
> freeze
>     > date, whichever comes first.
>     >
>     > I've also talked offline with Vinod to get help on release-management
>     > given this is my first release. He agreed to help do this release
> jointly.
>     >
>     > Thoughts?
>     >
>     > Thanks,
>     > Wangda Tan
>     >
>     > [1] https://lists.apache.org/thread.html/
> c11506c3250c9481852130616b3cb0
>     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.1.0)
>     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>     >
>     >
>
>
>

Re: Hadoop 3.1.0 release discussion

[Wangda Tan <wh...@gmail.com>]

Thanks Uma,

Could you update this thread once the merge vote started?

Best,
Wangda

On Wed, Jan 17, 2018 at 4:30 PM, Gangumalla, Uma <um...@intel.com>
wrote:

> HI Wangda,
>
>  Thank you for the head-up mail.
>  We are in the branch (HDFS-10285) and trying to push the tasks sooner
> before the deadline.
>
> Regards,
> Uma
>
> On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com> wrote:
>
>     Hi All,
>
>     Since we're fast approaching previously proposed feature freeze date
> (Jan
>     30, about 13 days from today). If you've any features which live in a
>     branch and targeted to 3.1.0, please reply this email thread. Ideally,
> we
>     should finish branch merging before feature freeze date.
>
>     Here's an updated 3.1.0 feature status:
>
>     1. Merged & Completed features:
>     * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>     end-to-end.
>     * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>     * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
> commits.
>     * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>     Queues While Doing Queue Mapping.
>     * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>
>     2. Features close to finish:
>     * (Zhankun) YARN-5983: FPGA support. Majority implementations
> completed and
>     merged to trunk. Except for UI/documentation.
>     * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>     discussions going on about implementation.
>     * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>     anti-affinity support. Close to finish, on track to be merged before
> Jan 30.
>
>     3. Tentative features:
>     * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>     containers. Only one pending patch. Plan to finish before Jan 7th.
>     * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging
> to be
>     done before Jan 2018.
>     * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>     challenging to be done before Jan 2018.
>     * (Varun V) YARN-5673: container-executor write. Given security
> refactoring
>     of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to
> 3.2.
>
>     Thanks,
>     Wangda
>
>
>
>
>     On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com>
> wrote:
>
>     > Hi all,
>     >
>     > Congratulations on the 3.0.0-GA release!
>     >
>     > As we discussed in the previous email thread [1], I'd like to restart
>     > 3.1.0 release plans.
>     >
>     > a) Quick summary:
>     > a.1 Release status
>     > We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
>     > there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open
> blockers and
>     > 22 open critical issues.
>     >
>     > a.2 Release date update
>     > Considering delays of 3.0-GA release by month-and-a-half, I propose
> to
>     > move the dates as follows
>     >  - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last
> date for
>     > any branches to get merged too;
>     >  - code freeze (blockers & critical only) date to Feb 08, 2018;
>     >  - release voting start by Feb 18, 2018, leaving time for at least
> two RCx
>     >  - release date from Jan 15, 2018, to Feb 28, 2018;
>     >
>     > Unlike before, I added an additional milestone for
> release-vote-start so
>     > that we can account for voting time-period also.
>     >
>     > This overall is still 5 1/2 months of release-timeline unlike the
> faster
>     > cadence we hoped for, but this, in my opinion, is the best-updated
> timeline
>     > given the delays of the final release of 3.0-GA.
>     >
>     > b) Individual feature status:
>     > I spoke to several feature owners and checked the status of
> un-finished
>     > features, following are status of features planned to 3.1.0:
>     >
>     > b.1 Merged & Completed features:
>     > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>     > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and
> works
>     > end-to-end.
>     > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native
> services.
>     > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>     > commits.
>     > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of
> Leaf
>     > Queues While Doing Queue Mapping.
>     >
>     > b.2 Features close to finish:
>     > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
>     > * (Zhankun) YARN-5983: FPGA support. Majority implementations
> completed
>     > and merged to trunk. Except for UI/documentation.
>     > * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>     > discussions going on about implementation.
>     >
>     > b.3 Tentative features:
>     > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>     > containers. Only one pending patch. Plan to finish before Jan 7th.
>     > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks
> challenging to
>     > be done before Jan 2018.
>     > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest
> and
>     > anti-affinity support. Tentative will figure out by Jan 1st.
>     > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>     > challenging to be done before Jan 2018.
>     > * (Varun V) YARN-5673: container-executor write. Given security
>     > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff
> may be
>     > moved to 3.2.
>     >
>     > b.4 Additional release drivers
>     > * More exhaustive upgrade testing from 2.x to 3.x.
>     >
>     > c) Regarding branch cut:
>     >
>     > We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
>     > feature planned to 3.2 has to be landed on trunk or B. After feature
> freeze
>     > date, whichever comes first.
>     >
>     > I've also talked offline with Vinod to get help on release-management
>     > given this is my first release. He agreed to help do this release
> jointly.
>     >
>     > Thoughts?
>     >
>     > Thanks,
>     > Wangda Tan
>     >
>     > [1] https://lists.apache.org/thread.html/
> c11506c3250c9481852130616b3cb0
>     > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>     > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.1.0)
>     > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>     >
>     >
>
>
>

Re: Hadoop 3.1.0 release discussion

["Gangumalla, Uma" <um...@intel.com>]

HI Wangda,

 Thank you for the head-up mail.
 We are in the branch (HDFS-10285) and trying to push the tasks sooner before the deadline.

Regards,
Uma

On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com> wrote:

    Hi All,
    
    Since we're fast approaching previously proposed feature freeze date (Jan
    30, about 13 days from today). If you've any features which live in a
    branch and targeted to 3.1.0, please reply this email thread. Ideally, we
    should finish branch merging before feature freeze date.
    
    Here's an updated 3.1.0 feature status:
    
    1. Merged & Completed features:
    * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
    * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
    end-to-end.
    * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
    * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename commits.
    * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
    Queues While Doing Queue Mapping.
    * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
    
    2. Features close to finish:
    * (Zhankun) YARN-5983: FPGA support. Majority implementations completed and
    merged to trunk. Except for UI/documentation.
    * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
    discussions going on about implementation.
    * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
    anti-affinity support. Close to finish, on track to be merged before Jan 30.
    
    3. Tentative features:
    * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
    containers. Only one pending patch. Plan to finish before Jan 7th.
    * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to be
    done before Jan 2018.
    * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
    challenging to be done before Jan 2018.
    * (Varun V) YARN-5673: container-executor write. Given security refactoring
    of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to 3.2.
    
    Thanks,
    Wangda
    
    
    
    
    On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com> wrote:
    
    > Hi all,
    >
    > Congratulations on the 3.0.0-GA release!
    >
    > As we discussed in the previous email thread [1], I'd like to restart
    > 3.1.0 release plans.
    >
    > a) Quick summary:
    > a.1 Release status
    > We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
    > there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open blockers and
    > 22 open critical issues.
    >
    > a.2 Release date update
    > Considering delays of 3.0-GA release by month-and-a-half, I propose to
    > move the dates as follows
    >  - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last date for
    > any branches to get merged too;
    >  - code freeze (blockers & critical only) date to Feb 08, 2018;
    >  - release voting start by Feb 18, 2018, leaving time for at least two RCx
    >  - release date from Jan 15, 2018, to Feb 28, 2018;
    >
    > Unlike before, I added an additional milestone for release-vote-start so
    > that we can account for voting time-period also.
    >
    > This overall is still 5 1/2 months of release-timeline unlike the faster
    > cadence we hoped for, but this, in my opinion, is the best-updated timeline
    > given the delays of the final release of 3.0-GA.
    >
    > b) Individual feature status:
    > I spoke to several feature owners and checked the status of un-finished
    > features, following are status of features planned to 3.1.0:
    >
    > b.1 Merged & Completed features:
    > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
    > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
    > end-to-end.
    > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
    > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
    > commits.
    > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
    > Queues While Doing Queue Mapping.
    >
    > b.2 Features close to finish:
    > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
    > * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
    > and merged to trunk. Except for UI/documentation.
    > * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
    > discussions going on about implementation.
    >
    > b.3 Tentative features:
    > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
    > containers. Only one pending patch. Plan to finish before Jan 7th.
    > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
    > be done before Jan 2018.
    > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
    > anti-affinity support. Tentative will figure out by Jan 1st.
    > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
    > challenging to be done before Jan 2018.
    > * (Varun V) YARN-5673: container-executor write. Given security
    > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may be
    > moved to 3.2.
    >
    > b.4 Additional release drivers
    > * More exhaustive upgrade testing from 2.x to 3.x.
    >
    > c) Regarding branch cut:
    >
    > We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
    > feature planned to 3.2 has to be landed on trunk or B. After feature freeze
    > date, whichever comes first.
    >
    > I've also talked offline with Vinod to get help on release-management
    > given this is my first release. He agreed to help do this release jointly.
    >
    > Thoughts?
    >
    > Thanks,
    > Wangda Tan
    >
    > [1] https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0
    > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
    > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
    > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
    >
    >
    

Re: Hadoop 3.1.0 release discussion

["Gangumalla, Uma" <um...@intel.com>]

HI Wangda,

 Thank you for the head-up mail.
 We are in the branch (HDFS-10285) and trying to push the tasks sooner before the deadline.

Regards,
Uma

On 1/17/18, 11:35 AM, "Wangda Tan" <wh...@gmail.com> wrote:

    Hi All,
    
    Since we're fast approaching previously proposed feature freeze date (Jan
    30, about 13 days from today). If you've any features which live in a
    branch and targeted to 3.1.0, please reply this email thread. Ideally, we
    should finish branch merging before feature freeze date.
    
    Here's an updated 3.1.0 feature status:
    
    1. Merged & Completed features:
    * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
    * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
    end-to-end.
    * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
    * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename commits.
    * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
    Queues While Doing Queue Mapping.
    * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
    
    2. Features close to finish:
    * (Zhankun) YARN-5983: FPGA support. Majority implementations completed and
    merged to trunk. Except for UI/documentation.
    * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
    discussions going on about implementation.
    * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
    anti-affinity support. Close to finish, on track to be merged before Jan 30.
    
    3. Tentative features:
    * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
    containers. Only one pending patch. Plan to finish before Jan 7th.
    * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to be
    done before Jan 2018.
    * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
    challenging to be done before Jan 2018.
    * (Varun V) YARN-5673: container-executor write. Given security refactoring
    of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to 3.2.
    
    Thanks,
    Wangda
    
    
    
    
    On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com> wrote:
    
    > Hi all,
    >
    > Congratulations on the 3.0.0-GA release!
    >
    > As we discussed in the previous email thread [1], I'd like to restart
    > 3.1.0 release plans.
    >
    > a) Quick summary:
    > a.1 Release status
    > We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
    > there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open blockers and
    > 22 open critical issues.
    >
    > a.2 Release date update
    > Considering delays of 3.0-GA release by month-and-a-half, I propose to
    > move the dates as follows
    >  - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last date for
    > any branches to get merged too;
    >  - code freeze (blockers & critical only) date to Feb 08, 2018;
    >  - release voting start by Feb 18, 2018, leaving time for at least two RCx
    >  - release date from Jan 15, 2018, to Feb 28, 2018;
    >
    > Unlike before, I added an additional milestone for release-vote-start so
    > that we can account for voting time-period also.
    >
    > This overall is still 5 1/2 months of release-timeline unlike the faster
    > cadence we hoped for, but this, in my opinion, is the best-updated timeline
    > given the delays of the final release of 3.0-GA.
    >
    > b) Individual feature status:
    > I spoke to several feature owners and checked the status of un-finished
    > features, following are status of features planned to 3.1.0:
    >
    > b.1 Merged & Completed features:
    > * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
    > * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
    > end-to-end.
    > * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
    > * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
    > commits.
    > * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
    > Queues While Doing Queue Mapping.
    >
    > b.2 Features close to finish:
    > * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
    > * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
    > and merged to trunk. Except for UI/documentation.
    > * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
    > discussions going on about implementation.
    >
    > b.3 Tentative features:
    > * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
    > containers. Only one pending patch. Plan to finish before Jan 7th.
    > * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
    > be done before Jan 2018.
    > * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
    > anti-affinity support. Tentative will figure out by Jan 1st.
    > * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
    > challenging to be done before Jan 2018.
    > * (Varun V) YARN-5673: container-executor write. Given security
    > refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may be
    > moved to 3.2.
    >
    > b.4 Additional release drivers
    > * More exhaustive upgrade testing from 2.x to 3.x.
    >
    > c) Regarding branch cut:
    >
    > We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
    > feature planned to 3.2 has to be landed on trunk or B. After feature freeze
    > date, whichever comes first.
    >
    > I've also talked offline with Vinod to get help on release-management
    > given this is my first release. He agreed to help do this release jointly.
    >
    > Thoughts?
    >
    > Thanks,
    > Wangda Tan
    >
    > [1] https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0
    > 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
    > [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
    > AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
    >
    >
    

Re: Hadoop 3.1.0 release discussion

[Daniel Templeton <da...@cloudera.com>]

What's the status on resource profiles?  I believe there are still a 
couple of open JIRAs to rethink some of the design choices.

Daniel

On 1/17/18 11:33 AM, Wangda Tan wrote:
> Hi All,
>
> Since we're fast approaching previously proposed feature freeze date (Jan
> 30, about 13 days from today). If you've any features which live in a
> branch and targeted to 3.1.0, please reply this email thread. Ideally, we
> should finish branch merging before feature freeze date.
>
> Here's an updated 3.1.0 feature status:
>
> 1. Merged & Completed features:
> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
> end-to-end.
> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename commits.
> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
> Queues While Doing Queue Mapping.
> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>
> 2. Features close to finish:
> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed and
> merged to trunk. Except for UI/documentation.
> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
> discussions going on about implementation.
> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
> anti-affinity support. Close to finish, on track to be merged before Jan 30.
>
> 3. Tentative features:
> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
> containers. Only one pending patch. Plan to finish before Jan 7th.
> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to be
> done before Jan 2018.
> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
> challenging to be done before Jan 2018.
> * (Varun V) YARN-5673: container-executor write. Given security refactoring
> of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to 3.2.
>
> Thanks,
> Wangda
>
>
>
>
> On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com> wrote:
>
>> Hi all,
>>
>> Congratulations on the 3.0.0-GA release!
>>
>> As we discussed in the previous email thread [1], I'd like to restart
>> 3.1.0 release plans.
>>
>> a) Quick summary:
>> a.1 Release status
>> We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
>> there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open blockers and
>> 22 open critical issues.
>>
>> a.2 Release date update
>> Considering delays of 3.0-GA release by month-and-a-half, I propose to
>> move the dates as follows
>>   - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last date for
>> any branches to get merged too;
>>   - code freeze (blockers & critical only) date to Feb 08, 2018;
>>   - release voting start by Feb 18, 2018, leaving time for at least two RCx
>>   - release date from Jan 15, 2018, to Feb 28, 2018;
>>
>> Unlike before, I added an additional milestone for release-vote-start so
>> that we can account for voting time-period also.
>>
>> This overall is still 5 1/2 months of release-timeline unlike the faster
>> cadence we hoped for, but this, in my opinion, is the best-updated timeline
>> given the delays of the final release of 3.0-GA.
>>
>> b) Individual feature status:
>> I spoke to several feature owners and checked the status of un-finished
>> features, following are status of features planned to 3.1.0:
>>
>> b.1 Merged & Completed features:
>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>> end-to-end.
>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>> commits.
>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>> Queues While Doing Queue Mapping.
>>
>> b.2 Features close to finish:
>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
>> and merged to trunk. Except for UI/documentation.
>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>> discussions going on about implementation.
>>
>> b.3 Tentative features:
>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>> containers. Only one pending patch. Plan to finish before Jan 7th.
>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
>> be done before Jan 2018.
>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>> anti-affinity support. Tentative will figure out by Jan 1st.
>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>> challenging to be done before Jan 2018.
>> * (Varun V) YARN-5673: container-executor write. Given security
>> refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may be
>> moved to 3.2.
>>
>> b.4 Additional release drivers
>> * More exhaustive upgrade testing from 2.x to 3.x.
>>
>> c) Regarding branch cut:
>>
>> We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
>> feature planned to 3.2 has to be landed on trunk or B. After feature freeze
>> date, whichever comes first.
>>
>> I've also talked offline with Vinod to get help on release-management
>> given this is my first release. He agreed to help do this release jointly.
>>
>> Thoughts?
>>
>> Thanks,
>> Wangda Tan
>>
>> [1] https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0
>> 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>> [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org

Re: Hadoop 3.1.0 release discussion

[Daniel Templeton <da...@cloudera.com>]

What's the status on resource profiles?  I believe there are still a 
couple of open JIRAs to rethink some of the design choices.

Daniel

On 1/17/18 11:33 AM, Wangda Tan wrote:
> Hi All,
>
> Since we're fast approaching previously proposed feature freeze date (Jan
> 30, about 13 days from today). If you've any features which live in a
> branch and targeted to 3.1.0, please reply this email thread. Ideally, we
> should finish branch merging before feature freeze date.
>
> Here's an updated 3.1.0 feature status:
>
> 1. Merged & Completed features:
> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
> end-to-end.
> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename commits.
> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
> Queues While Doing Queue Mapping.
> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage.
>
> 2. Features close to finish:
> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed and
> merged to trunk. Except for UI/documentation.
> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
> discussions going on about implementation.
> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
> anti-affinity support. Close to finish, on track to be merged before Jan 30.
>
> 3. Tentative features:
> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
> containers. Only one pending patch. Plan to finish before Jan 7th.
> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to be
> done before Jan 2018.
> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
> challenging to be done before Jan 2018.
> * (Varun V) YARN-5673: container-executor write. Given security refactoring
> of c-e (YARN-6623) is already landed, IMHO other stuff may be moved to 3.2.
>
> Thanks,
> Wangda
>
>
>
>
> On Fri, Dec 15, 2017 at 1:20 PM, Wangda Tan <wh...@gmail.com> wrote:
>
>> Hi all,
>>
>> Congratulations on the 3.0.0-GA release!
>>
>> As we discussed in the previous email thread [1], I'd like to restart
>> 3.1.0 release plans.
>>
>> a) Quick summary:
>> a.1 Release status
>> We started 3.1 release discussion on Sep 6, 2017 [1]. As of today,
>> there’re 232 patches loaded on 3.1.0 alone [2], besides 6 open blockers and
>> 22 open critical issues.
>>
>> a.2 Release date update
>> Considering delays of 3.0-GA release by month-and-a-half, I propose to
>> move the dates as follows
>>   - feature freeze date from Dec 15, 2017, to Jan 30, 2018 - last date for
>> any branches to get merged too;
>>   - code freeze (blockers & critical only) date to Feb 08, 2018;
>>   - release voting start by Feb 18, 2018, leaving time for at least two RCx
>>   - release date from Jan 15, 2018, to Feb 28, 2018;
>>
>> Unlike before, I added an additional milestone for release-vote-start so
>> that we can account for voting time-period also.
>>
>> This overall is still 5 1/2 months of release-timeline unlike the faster
>> cadence we hoped for, but this, in my opinion, is the best-updated timeline
>> given the delays of the final release of 3.0-GA.
>>
>> b) Individual feature status:
>> I spoke to several feature owners and checked the status of un-finished
>> features, following are status of features planned to 3.1.0:
>>
>> b.1 Merged & Completed features:
>> * (Sunil) YARN-5881: Support absolute value in CapacityScheduler.
>> * (Wangda) YARN-6223: GPU support on YARN. Features in trunk and works
>> end-to-end.
>> * (Jian) YARN-5079,YARN-4793,YARN-4757,YARN-6419 YARN native services.
>> * (Steve Loughran): HADOOP-13786: S3Guard committer for zero-rename
>> commits.
>> * (Suma): YARN-7117: Capacity Scheduler: Support Auto Creation of Leaf
>> Queues While Doing Queue Mapping.
>>
>> b.2 Features close to finish:
>> * (Chris Douglas) HDFS-9806: HDFS Tiered Storage. Being voting now.
>> * (Zhankun) YARN-5983: FPGA support. Majority implementations completed
>> and merged to trunk. Except for UI/documentation.
>> * (Uma) HDFS-10285: HDFS SPS. Majority implementations are done, some
>> discussions going on about implementation.
>>
>> b.3 Tentative features:
>> * (Arun Suresh). YARN-5972: Support pausing/freezing opportunistic
>> containers. Only one pending patch. Plan to finish before Jan 7th.
>> * (Haibo Chen). YARN-1011: Resource overcommitment. Looks challenging to
>> be done before Jan 2018.
>> * (Arun Suresh / Kostas / Wangda). YARN-6592: New SchedulingRequest and
>> anti-affinity support. Tentative will figure out by Jan 1st.
>> * (Anu): HDFS-7240: Ozone. Given the discussion on HDFS-7240. Looks
>> challenging to be done before Jan 2018.
>> * (Varun V) YARN-5673: container-executor write. Given security
>> refactoring of c-e (YARN-6623) is already landed, IMHO other stuff may be
>> moved to 3.2.
>>
>> b.4 Additional release drivers
>> * More exhaustive upgrade testing from 2.x to 3.x.
>>
>> c) Regarding branch cut:
>>
>> We will keep pointing trunk to 3.1 and cut branch-3.1 until: A. some
>> feature planned to 3.2 has to be landed on trunk or B. After feature freeze
>> date, whichever comes first.
>>
>> I've also talked offline with Vinod to get help on release-management
>> given this is my first release. He agreed to help do this release jointly.
>>
>> Thoughts?
>>
>> Thanks,
>> Wangda Tan
>>
>> [1] https://lists.apache.org/thread.html/c11506c3250c9481852130616b3cb0
>> 9a0e222f5c2465c015f9906dab@%3Cyarn-dev.hadoop.apache.org%3E
>> [2] "project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
>> AND fixVersion not in (3.0.0,2.9.0) ORDER BY priority DESC”
>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org