2017-10-20 Hadoop 3 release status update

[Andrew Wang <an...@cloudera.com>]

https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3+release+status+updates

2017-10-20

Apologies for skipping the update last week. Here's how we're tracking for
GA.

Highlights:

   - Merge of HDFS router-based federation and API-based scheduler
   configuration with no reported problems. Kudos to the contributors involved!

Red flags:

   - We're making a last-minute push to get resource types (but not
   resource profiles in). Coming this late, it's a risk, but we decided it's
   worthwhile for this feature. See Daniel's yarn-dev email
   <https://lists.apache.org/thread.html/cc51a68f1cf21c641cf77383ee1a13e3b2ca93d2a56f5a441f42b21b@%3Cyarn-dev.hadoop.apache.org%3E>
for
   the full rationale.
   - Still uncovering EC bugs from testing

Previously tracked GA blockers that have been resolved or dropped:

   - YARN-6623
   <https://issues.apache.org/jira/browse/YARN-6623?src=confmacro> - Add
   support to turn off launching privileged containers in the
   container-executor RESOLVED: Committed and resolved
   - Change of ExecutionType
      - YARN-7275
      <https://issues.apache.org/jira/browse/YARN-7275?src=confmacro> - NM
      Statestore cleanup for Container updates RESOLVED : Patch committed,
      resolved.
   - ReservationSystem
      - YARN-4859
      <https://issues.apache.org/jira/browse/YARN-4859?src=confmacro> - [Bug]
      Unable to submit a job to a reservation when using FairScheduler
      RESOLVED: Yufei tested this and found things mostly worked, filed two
      not-blocker followons: YARN-7347
      <https://issues.apache.org/jira/browse/YARN-7347?src=confmacro> - Fixe
      the bug in Fair scheduler to handle a queue named "root.root" OPEN
       and YARN-7348
      <https://issues.apache.org/jira/browse/YARN-7348?src=confmacro> - Ignore
      the vcore in reservation request for fair policy queue OPEN

GA blockers:

   - Change of ExecutionType
      - YARN-7178
      <https://issues.apache.org/jira/browse/YARN-7178?src=confmacro> - Add
      documentation for Container Update API OPEN : Still no update from
      Arun, I pinged it.
   - ReservationSystem
      - YARN-4827
      <https://issues.apache.org/jira/browse/YARN-4827?src=confmacro>
- Document
      configuration of ReservationSystem for FairScheduler OPEN: Yufei said
      he'd work on it as of 2 days ago
   - Rolling upgrade
      - YARN-6142
      <https://issues.apache.org/jira/browse/YARN-6142?src=confmacro> - Support
      rolling upgrade between 2.x and 3.x OPEN : I pinged this and asked
      for a status update
      - HDFS-11096
      <https://issues.apache.org/jira/browse/HDFS-11096?src=confmacro>
- Support
      rolling upgrade between 2.x and 3.xPATCH AVAILABLE: I pinged this and
      asked for a status update
   - Erasure coding
      - HDFS-12682
      <https://issues.apache.org/jira/browse/HDFS-12682?src=confmacro>
- ECAdmin
      -listPolicies will always show policy state as DISABLED OPEN: New
      blocker filed this week, Xiao is working on it
      - HDFS-12686
      <https://issues.apache.org/jira/browse/HDFS-12686?src=confmacro>
- Erasure
      coding system policy state is not correctly saved and loaded during real
      cluster restart OPEN: New blocker filed this week, Sammi is on it
      - HDFS-12686
      <https://issues.apache.org/jira/browse/HDFS-12686?src=confmacro>
- Erasure
      coding system policy state is not correctly saved and loaded during real
      cluster restart OPEN: Old blocker, Huafeng is on it, waiting on
      review from Wei-Chiu or Sammi

Features merged for GA:

   - Erasure coding
      - Continued bug reporting and fixing based on testing at Cloudera.
      - Two new blockers filed this week, mentioned above.
      - Huafeng completed patch to reenable disabled EC tests
   - Classpath isolation (HADOOP-11656)
   - HADOOP-13916
      <https://issues.apache.org/jira/browse/HADOOP-13916?src=confmacro>
- Document
      how downstream clients should make use of the new shaded client artifacts
       IN PROGRESS: I pinged it
   - Compat guide (HADOOP-13714
   <https://issues.apache.org/jira/browse/HADOOP-13714>)
      - HADOOP-14876
      <https://issues.apache.org/jira/browse/HADOOP-14876?src=confmacro>
- Create
      downstream developer docs from the compatibility guidelines PATCH
      AVAILABLE: Daniel has a patch up, revved based on Steve's review
      feedback, waiting on Steve's reply
      - HADOOP-14875
      <https://issues.apache.org/jira/browse/HADOOP-14875?src=confmacro>
- Create
      end user documentation from the compatibility guidelines OPEN: No
      patch yet
   - TSv2 alpha 2
   - This was merged, no problems thus far [image: (smile)]
   - API-based scheduler configuration YARN-5734
   <https://issues.apache.org/jira/browse/YARN-5734?src=confmacro> - OrgQueue
   for easy CapacityScheduler queue configuration management RESOLVED
      - Merged, no problems thus far [image: (smile)]
   - HDFS router-based configuration HDFS-10467
   <https://issues.apache.org/jira/browse/HDFS-10467?src=confmacro> -
Router-based
   HDFS federation RESOLVED
      - Merged, no problems thus far [image: (smile)]

Unmerged features:

   - Resource types / profiles (YARN-3926
   <https://issues.apache.org/jira/browse/YARN-3926> and YARN-7069
   <https://issues.apache.org/jira/browse/YARN-7069>) (Wangda Tan)
      - We're going to try and get in resource types for 3.0.0 and leave
      resource profiles for 3.1.0. Daniel is spearheading this and other major
      contributors like Wangda and Sunil are onboard with the plan. Branch has
      been created and undergone testing, I expect a merge vote ASAP.
      - This is our biggest remaining risk.
   - YARN native services (YARN-5079
   <https://issues.apache.org/jira/browse/YARN-5079>) (Jian He)
      - YARN-7351
      <https://issues.apache.org/jira/browse/YARN-7351?src=confmacro> - High
      CPU usage issue in RegistryDNS PATCH AVAILABLE is blocking
      - Allen is reviewing YARN-7127 with some design-level questions about
      the API and architecture
      - This doesn't look like this is getting in based on YARN-7127
      discussion