2017-09-20 Hadoop 3 release status update

[Andrew Wang <an...@cloudera.com>]

https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3+release+status+updates

2017-09-29

After about a month of slip, RC0 has been sent out for a VOTE. Focus now
turns to GA, where we will attempt to keep the original beta1 target date
(early November).

Highlights:

   - RC0 vote was sent out on Thursday, two binding +1's so far.

Red flags:

   - Resource profiles still has a number of pending subtasks, which is
   concerning from a schedule perspective. I emailed Wangda about this, and we
   need to discuss with other key contributors.
   - Native services has one pending subtask but we haven't gotten
   follow-on reviews from Allen (who -1'd the earlier merge vote). Need to
   confirm that we've satisfied his feedback.

Previously tracked beta1 blockers that have been resolved or dropped:

   - YARN-6623 was pushed out of beta1 to GA, has been committed so we can
   drop it from tracking.
   - HADOOP-14897 <https://issues.apache.org/jira/browse/HADOOP-14897> (Loosen
   compatibility guidelines for native dependencies): Patch committed!

beta1 blockers:

   - None, RC0 is out

GA blockers:

   - YARN-7134
   <https://issues.apache.org/jira/browse/YARN-7134?src=confmacro> -
AppSchedulingInfo
   has a dependency on capacity scheduler OPEN  : this one popped out of
   nowhere, I don't have an update yet.
   - YARN-7178
   <https://issues.apache.org/jira/browse/YARN-7178?src=confmacro> - Add
   documentation for Container Update API OPEN : this also popped out of
   nowhere, no update yet.
   - YARN-7275
   <https://issues.apache.org/jira/browse/YARN-7275?src=confmacro> - NM
   Statestore cleanup for Container updates OPEN : Ditto
   - YARN-4859
   <https://issues.apache.org/jira/browse/YARN-4859?src=confmacro> - [Bug]
   Unable to submit a job to a reservation when using FairScheduler OPEN :
   Ditto
   - YARN-4827
   <https://issues.apache.org/jira/browse/YARN-4827?src=confmacro> - Document
   configuration of ReservationSystem for FairScheduler OPEN : Ditto

Features merged for GA:

   - Erasure coding
      - People are looking more at the flaky tests and nice-to-haves
      - Some bugs reported and being fixed based on testing at Cloudera
      - Need to finish the 3.0 must-do's.
   - Addressing incompatible changes (YARN-6142 and HDFS-11096)
   - Sean has posted a new rev of the rolling upgrade script
      - Some YARN PB backward compat issues that we decided weren't
      blockers and are scheduled for GA
   - Classpath isolation (HADOOP-11656)
      - HADOOP-13917
<https://issues.apache.org/jira/browse/HADOOP-13917> (Ensure
      nightly builds run the integration tests for the shaded client):
Resolved,
      Sean retriggered and determined that this works.
      - HADOOP-14771 is still floating, along with adding documentation.
   - Compat guide (HADOOP-13714
   <https://issues.apache.org/jira/browse/HADOOP-13714>)
      - A few subtasks are targeted at GA
   - TSv2 alpha 2
   - This was merged, no problems thus far [image: (smile)]

Unmerged features:

   - Resource profiles (YARN-3926
   <https://issues.apache.org/jira/browse/YARN-3926> and YARN-7069
   <https://issues.apache.org/jira/browse/YARN-7069>) (Wangda Tan)
      - This has been merged for 3.1.0, YARN-7069 tracks follow on work
      - ~7 patch available subtasks, I asked Wangda to set up a JIRA query
      for tracking this
   - HDFS router-based federation (HDFS-10467
   <https://issues.apache.org/jira/browse/HDFS-10467>) (Inigo Goiri and
   Chris Douglas)
   - Inigo sent out the merge vote
   - API-based scheduler configuration (Jonathan Hung)
      - Jonathan sent out a discuss thread for merge, thinking is early
      next week. Larry did a security-oriented review.
   - YARN native services (YARN-5079
   <https://issues.apache.org/jira/browse/YARN-5079>) (Jian He)
      - Subtasks were filed to address Allen's review comments from the
      previous merge vote, only one pending
      - We need to confirm with Allen that this is ready to go, he hasn't
      been reviewing