Kubernetes Scheduler Backend for Spark: Pre Holiday Update

[Anirudh Ramanathan <fo...@google.com.INVALID>]

Hi all,

We've all been hard at work on the Spark + Kubernetes effort over the past
few weeks. Here's an update on what we've been up to. TL; DR - we're
*nearly* there! Spark 2.3 promises to be an exciting release for the Apache
Spark and Kubernetes communities.

*What's done?* (please see SPARK-18278
<https://issues.apache.org/jira/browse/SPARK-18278> for details!):

   - Core Kubernetes scheduler scaffolding
   - Static executor scheduling
   - Executor failure & recovery semantics
   - Spark application submission semantics
   - Continuous testing in single and multi-node Kubernetes clusters
   - Documentation

*What's in flight?*

   - In Review - Dependency management and documentation for enabling HDFS,
   GCS, S3 and HTTP-based localization of resources. SPARK-22757
   <https://issues.apache.org/jira/browse/SPARK-22757>, SPARK-22648
   <https://issues.apache.org/jira/browse/SPARK-22648>.

It needs acknowledging that this contribution is a result of the
contributions of many many individuals and no small feat. Here's some
statistics speaking to the sheer magnitude of this change.

   - 6k+ lines of code, 1000+ comments on PRs and 100+ rounds of review
   with 10+ unique reviewers (and counting!)
   - At least 2 pull requests that grew so large that they made GitHub throw
   unicorns <https://github.com/503.html> ;)

A special thanks to all the reviewers who spent their time helping us,
especially close to the holiday season, and thanks to the committers for
all the support. Looking forward to an awesome 2018 with both Apache Spark
and Kubernetes scaling even greater heights.

*Special thanks to all the reviewers & release managers*: Marcelo Vanzin,
Mridul Muralidharan, Jiang Xingbo, Reynold Xin, Felix Cheung, Takuya
Ueshin, Sean Owen, Wenchen Fan, Mark Hamstra and Michael Armbrust.

*..and contributors*: Yinan Li, Matt Cheah, Kimoon Kim, Sean Suchter, Shane
Knapp, Erik Erlandson, Andrew Ash, and Ilan Filonenko.

Happy Holidays!
- Anirudh