2017-12-01 Hadoop 3 release status update

[Andrew Wang <an...@cloudera.com>]

https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3+release+status+updates

2017-12-01

Haven't written one of these in a month. I had high hopes for RC0, but it
failed due to  HADOOP-15058
<https://issues.apache.org/jira/browse/HADOOP-15058?src=confmacro> -
create-release
site build outputs dummy shaded jars due to skipShade PATCH AVAILABLE  which
Sangjin found, and then a number of other blockers were found shortly after
that.

We're back to blocker burndown. My new (realistic) goal is to get 3.0.0 out
before Christmas. We could always use more help with reviews; most things
are patch available.



Highlights:

Red flags:

Previously tracked blockers that have been resolved or dropped:

GA blockers:

   - HDFS-12840
   <https://issues.apache.org/jira/browse/HDFS-12840?src=confmacro> - Creating
   a replicated file in a EC zone does not correctly serialized in
EditLogs PATCH
   AVAILABLE : Has gone through several rounds of review, looks close.
   - HADOOP-15080
   <https://issues.apache.org/jira/browse/HADOOP-15080?src=confmacro> - Cat-X
   transitive dependency on org.json library via json-lib OPEN : New issue,
   waiting on LEGAL but we might need to pull this entire feature.
   - HADOOP-15059
   <https://issues.apache.org/jira/browse/HADOOP-15059?src=confmacro> - 3.0
   deployment cannot work with old version MR tar ball which break rolling
   upgrade PATCH AVAILABLE : Has gone through some review and has a +1 from
   Daryn, could use confirmation from Vinod and others
   - HADOOP-15058
   <https://issues.apache.org/jira/browse/HADOOP-15058?src=confmacro>
- create-release
   site build outputs dummy shaded jars due to skipShade PATCH AVAILABLE :
   Needs review, asked Allen but might need someone else to help.

GA criticals:

   - HDFS-12872
   <https://issues.apache.org/jira/browse/HDFS-12872?src=confmacro> - EC
   Checksum broken when BlockAccessToken is enabled PATCH AVAILABLE : Patch
   needs review
   - YARN-7381
   <https://issues.apache.org/jira/browse/YARN-7381?src=confmacro> - Enable
   the configuration: yarn.nodemanager.log-container-debug-info.enabled PATCH
   AVAILABLE : Has gone through some review and Wangda +1'd, could use
   confirmation from Ray and others

Features merged for GA:

   - Erasure coding
      - Testing is still ongoing at Cloudera, which resulted in  HDFS-12840
      <https://issues.apache.org/jira/browse/HDFS-12840?src=confmacro>
- Creating
      a replicated file in a EC zone does not correctly serialized in EditLogs
       PATCH AVAILABLE  and HDFS-12872
      <https://issues.apache.org/jira/browse/HDFS-12872?src=confmacro> - EC
      Checksum broken when BlockAccessToken is enabled PATCH AVAILABLE .
   - Classpath isolation (HADOOP-11656)
   - No change.
   - Compat guide (HADOOP-13714
   <https://issues.apache.org/jira/browse/HADOOP-13714>)
      - We slid a couple more changes into 3.0.0 after RC0 was cancelled,
      making this work more complete.
   - TSv2 alpha 2
   - No change.
   - API-based scheduler configuration  YARN-5734
   <https://issues.apache.org/jira/browse/YARN-5734?src=confmacro> - OrgQueue
   for easy CapacityScheduler queue configuration management RESOLVED
      - No change.
   - HDFS router-based configuration  HDFS-10467
   <https://issues.apache.org/jira/browse/HDFS-10467?src=confmacro> -
Router-based
   HDFS federation RESOLVED
      - No change.
   - Resource types  YARN-3926
   <https://issues.apache.org/jira/browse/YARN-3926?src=confmacro> - Extend
   the YARN resource model for easier resource-type management and profiles
   RESOLVED
      - Had some post-merge issues that were resolved, nothing outstanding.