2017-10-31 Hadoop 3 release status update

[Andrew Wang <an...@cloudera.com>]

https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3+release+status+
updates

2017-10-31

Lots of progress towards GA, we look on track for cutting RC0 this week. I
ran the versions script to check the branch matches up with JIRA and fixed
things up, and also checked that the changelog and release notes look
reasonable.

Highlights:

   - Resource types vote has passed and will be merged with branch-3.0
   shortly.
   - Down to three blockers on the dashboard, all being actively revved.

Red flags:

   - Still need to validate that resource types is ready to go once it's
   merged.

Previous tracked GA blockers that have been resolved or dropped:

   - Change of ExecutionType
      - YARN-7178
      <https://issues.apache.org/jira/browse/YARN-7178?src=confmacro> - Add
      documentation for Container Update API RESOLVED : Arun got the patch
      in with reviews from Wangda and Haibo.
   - ReservationSystem
      - YARN-4827
      <https://issues.apache.org/jira/browse/YARN-4827?src=confmacro>
- Document
      configuration of ReservationSystem for FairScheduler RESOLVED: Yufei
      and Subru got this in.
   - Rolling upgrade
      - YARN-6142
      <https://issues.apache.org/jira/browse/YARN-6142?src=confmacro> - Support
      rolling upgrade between 2.x and 3.x RESOLVED : Ray resolved this
      since we think it's sufficiently complete.
   - Erasure coding
      - HDFS-12686
      <https://issues.apache.org/jira/browse/HDFS-12686?src=confmacro>
- Erasure
      coding system policy state is not correctly saved and loaded during real
      cluster restart RESOLVED: Resolved this one to incorporate it in
      HDFS-12682

GA blockers:

   - Rolling upgrade
      - HDFS-11096
      <https://issues.apache.org/jira/browse/HDFS-11096?src=confmacro>
- Support
      rolling upgrade between 2.x and 3.xPATCH AVAILABLE: I asked Sean if
      we can downgrade this from blocker
   - Erasure coding
      - HDFS-12682
      <https://issues.apache.org/jira/browse/HDFS-12682?src=confmacro>
- ECAdmin
      -listPolicies will always show SystemErasureCodingPolicies state
as DISABLED
       PATCH AVAILABLE: Actively being worked on and reviewed, should be in
      soon
      - HDFS-11467
      <https://issues.apache.org/jira/browse/HDFS-11467?src=confmacro>
- Support
      ErasureCoding section in OIV XML/ReverseXMLPATCH AVAILABLE: Waiting
      on HDFS-12682, I asked if we can work concurrently

Features merged for GA:

   - Erasure coding
      - Testing is still ongoing at Cloudera, no new bugs found recently
      - Closing on remaining blockers for GA
   - Classpath isolation (HADOOP-11656)
   - HADOOP-13916
      <https://issues.apache.org/jira/browse/HADOOP-13916?src=confmacro>
- Document
      how downstream clients should make use of the new shaded client artifacts
       OPEN: Seems unlikely to make it
   - Compat guide (HADOOP-13714
   <https://issues.apache.org/jira/browse/HADOOP-13714>)
      - HADOOP-14876
      <https://issues.apache.org/jira/browse/HADOOP-14876?src=confmacro>
- Create
      downstream developer docs from the compatibility guidelines PATCH
      AVAILABLE: Patch is being actively revved and reviewed, Robert +1'd,
      Anu posted a big review
      - HADOOP-14875
      <https://issues.apache.org/jira/browse/HADOOP-14875?src=confmacro>
- Create
      end user documentation from the compatibility guidelines PATCH
      AVAILABLE: No patch yet
   - TSv2 alpha 2
   - This was merged, no problems thus far [image: (smile)]
   - API-based scheduler configuration YARN-5734
   <https://issues.apache.org/jira/browse/YARN-5734?src=confmacro> - OrgQueue
   for easy CapacityScheduler queue configuration management RESOLVED
      - Merged, no problems thus far [image: (smile)]
   - HDFS router-based configuration HDFS-10467
   <https://issues.apache.org/jira/browse/HDFS-10467?src=confmacro> -
Router-based
   HDFS federation RESOLVED
      - Merged, no problems thus far [image: (smile)]
   - Resource types YARN-3926
   <https://issues.apache.org/jira/browse/YARN-3926?src=confmacro> - Extend
   the YARN resource model for easier resource-type management and profiles
   RESOLVED
      - Vote has passed, Daniel is currently doing the mechanics of merging
      - Need to also perform final validation post-merge

Dropping the "unmerged features" section since we're not letting in
anything else at this point.