2017-09-19 Hadoop 3 release status update

[Andrew Wang <an...@cloudera.com>]

https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3+release+status+updates

2017-09-19

Sorry for the late update. We're down to one blocker and one EC must do!
Made great progress over the last week and a bit.

We will likely cut RC0 this week.

Highlights:

   - Down to just two blocker issues!

Red flags:

   - HDFS unit tests are quite flaky. Some blockers were filed and then
   resolved or downgraded. More work to do here.

Previously tracked beta1 blockers that have been resolved or dropped:

   - HADOOP-14738 <https://issues.apache.org/jira/browse/HADOOP-14738> (Remove
   S3N and obsolete bits of S3A; rework docs): Committed!
   - HADOOP-14284 <https://issues.apache.org/jira/browse/HADOOP-14284> (Shade
   Guava everywhere): We resolved this since we decided it was unnecessary for
   beta1.
   - YARN-7162 <https://issues.apache.org/jira/browse/YARN-7162> (Remove
   XML excludes file format): Robert committed after review from Junping.
   - HADOOP-14847 <https://issues.apache.org/jira/browse/HADOOP-14847> (Remove
   Guava Supplier and change to java Supplier in AMRMClient and
   AMRMClientAysnc): Committed!
   - HADOOP-14238 <https://issues.apache.org/jira/browse/HADOOP-14238>
(Rechecking
   Guava's object is not exposed to user-facing API): We dropped this off the
   blocker list in the absence of other known issues
   - HADOOP-14835 <https://issues.apache.org/jira/browse/HADOOP-14835> (mvn
   site build throws SAX errors): I committed after further discussion and
   review with Sean Mackrory and Allen. Planning to switch to japicmp for
   later releases.
   - HDFS-12218 <https://issues.apache.org/jira/browse/HDFS-12218> (Rename
   split EC / replicated block metrics in BlockManager): Committed.


beta1 blockers:

   - HADOOP-14771 <https://issues.apache.org/jira/browse/HADOOP-14771>
(hadoop-client
   does not include hadoop-yarn-client): This was committed but then reverted
   since it broke the build. Haibo and Sean are actively pressing towards a
   correct fix.


beta1 features:

   - Erasure coding
      - Resolved a number of must-dos
         - HDFS-7859 (fsimage changes) was committed!
         - HDFS-12395 (edit log changes) was also committed!
         - HDFS-12218 is discussed above.
      - Remaining blockers:
         - HDFS-12447 is to refactor some of the fsimage code, Andrew needs
         to review
      - Also been progress cleaning up the flaky unit tests, still more to
      do
   - Addressing incompatible changes (YARN-6142 and HDFS-11096)
   - Ray has gone through almost all the YARN protos and thinks we're okay
      to move forwards.
      - I think we'll move forward without this committed, given that Sean
      has run it successfully.
   - Classpath isolation (HADOOP-11656)
      - We have just HADOOP-14771 left.
   - Compat guide (HADOOP-13714
   <https://issues.apache.org/jira/browse/HADOOP-13714>)
      - This was committed! Some follow-on work filed for GA.
   - TSv2 alpha 2
   - This was merged, no problems thus far [image: (smile)]

GA features:

   - Resource profiles (Wangda Tan)
      - Merge vote was sent out. Since branch-3.0 has been cut, this can be
      merged to trunk (3.1.0) and then backported once we've completed testing.
   - HDFS router-based federation (Chris Douglas)
   - This is like YARN federation, very separate and doesn't add new APIs,
      run in production at MSFT.
      - If it passes Cloudera internal integration testing, I'm fine
      putting this in for GA.
   - API-based scheduler configuration (Jonathan Hung)
      - Jonathan mentioned that his main goal is to get this in for 2.9.0,
      which seems likely to go out after 3.0.0 GA since there hasn't been any
      serious release planning yet. Jonathan said that delaying this
until 3.1.0
      is fine.
   - YARN native services
      - Still not 100% clear when this will land.