2017-09-22 Hadoop 3 release status update

[Andrew Wang <an...@cloudera.com>]

https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3+release+status+updates

2017-09-22

We've had some late breaking blockers related to Docker support that are
delaying the release. We're on a day-by-day slip at this point.



Highlights:

   - I did a successful test create-release earlier this week.

Red flags:

   - Docker work resulted in some last minute blockers

Previously tracked beta1 blockers that have been resolved or dropped:

   - HADOOP-14771 <https://issues.apache.org/jira/browse/HADOOP-14771>
(hadoop-client
   does not include hadoop-yarn-client): Dropped this from the blocker list as
   it's mainly for documentation purposes
   - HDFS-12247 (Rename AddECPolicyResponse to
   AddErasureCodingPolicyResponse) was committed.

beta1 blockers:

   - YARN-6623 <https://issues.apache.org/jira/browse/YARN-6623> (Add
   support to turn off launching privileged containers in the
   container-executor): This is a newly escalated blocker related to the
   Docker work in YARN. Patch is up but we're still waiting on a commit.
   - HADOOP-14897 <https://issues.apache.org/jira/browse/HADOOP-14897> (Loosen
   compatibility guidelines for native dependencies): Raised by Chris Douglas,
   Daniel will post a patch soon.

beta1 features:

   - Erasure coding
      - Resolved last must-do for beta1!
      - People are looking more at the flaky tests and nice-to-haves
      - Eddy continues to make improvements to block reconstruction
      codepaths
   - Addressing incompatible changes (YARN-6142 and HDFS-11096)
   - Ray has gone through almost all the YARN protos and thinks we're okay
      to move forwards.
      - I think we'll move forward without this committed, given that Sean
      has run it successfully.
   - Classpath isolation (HADOOP-11656)
      - HADOOP-13917
<https://issues.apache.org/jira/browse/HADOOP-13917> (Ensure
      nightly builds run the integration tests for the shaded client):
Sean wants
      to get this in before beta1 if there's time, it's already
catching issues.
      Relies on YETUS-543 which I reviewed, waiting on Allen.
      - HADOOP-14771 might be squeezed in if there's time.
   - Compat guide (HADOOP-13714
   <https://issues.apache.org/jira/browse/HADOOP-13714>)
      - HADOOP-14897 Above mentioned blocker filed by Chris Douglas.
   - TSv2 alpha 2
   - This was merged, no problems thus far [image: (smile)]

GA features:

   - Resource profiles (Wangda Tan)
      - Merge vote was sent out. Since branch-3.0 has been cut, this can be
      merged to trunk (3.1.0) and then backported once we've completed testing.
   - HDFS router-based federation (Chris Douglas)
   - This is like YARN federation, very separate and doesn't add new APIs,
      run in production at MSFT.
      - If it passes Cloudera internal integration testing, I'm fine
      putting this in for GA.
   - API-based scheduler configuration (Jonathan Hung)
      - Jonathan mentioned that his main goal is to get this in for 2.9.0,
      which seems likely to go out after 3.0.0 GA since there hasn't been any
      serious release planning yet. Jonathan said that delaying this
until 3.1.0
      is fine.
   - YARN native services
      - Still not 100% clear when this will land.