You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Anirudh Ramanathan <fo...@google.com.INVALID> on 2017/06/23 16:50:21 UTC

An Update on Spark on Kubernetes [Jun 23]

*Project Description: *Kubernetes cluster manager integration that enables
native support for submitting Spark applications to a kubernetes cluster.
The submitted applications can make use of Kubernetes native constructs.

*JIRA*: 18278 <https://issues.apache.org/jira/browse/SPARK-18278>
*Upstream Kubernetes Issue*: 34377
<https://github.com/kubernetes/kubernetes/issues/34377>

*Summary of activity in the last 2 weeks*:

   - PySpark support being added by ifilolenko (Bloomberg)
   - HDFS locality fixes by kimoonkim (PepperData)
   - Minor bug fixes and performance improvement (Tencent)
   - Planning of work around Spark and HDFS on Kubernetes for Q3 2017
   - Weekly meeting notes and discussions (link
   <https://docs.google.com/document/u/1/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit>
   )

*Recent talks and updates*:

   - Spark on Kubernetes (Spark Summit 2017) - link
   <https://spark-summit.org/2017/events/apache-spark-on-kubernetes/>
   - HDFS on Kubernetes - Lessons Learned (Spark Summit 2017) - link
   <https://spark-summit.org/2017/events/hdfs-on-kubernetes-lessons-learned/>


For more details and to contribute to our discussions/efforts, please
attend our weekly public meeting, or join our slack/mailing list. (link
<https://github.com/kubernetes/community/tree/master/sig-big-data>)

Re: An Update on Spark on Kubernetes [Jun 23]

Posted by Reynold Xin <rx...@databricks.com>.
Thanks, Anirudh. This is super helpful!

On Fri, Jun 23, 2017 at 9:50 AM, Anirudh Ramanathan <fo...@google.com>
wrote:

> *Project Description: *Kubernetes cluster manager integration that
> enables native support for submitting Spark applications to a kubernetes
> cluster. The submitted applications can make use of Kubernetes native
> constructs.
>
> *JIRA*: 18278 <https://issues.apache.org/jira/browse/SPARK-18278>
> *Upstream Kubernetes Issue*: 34377
> <https://github.com/kubernetes/kubernetes/issues/34377>
>
> *Summary of activity in the last 2 weeks*:
>
>    - PySpark support being added by ifilolenko (Bloomberg)
>    - HDFS locality fixes by kimoonkim (PepperData)
>    - Minor bug fixes and performance improvement (Tencent)
>    - Planning of work around Spark and HDFS on Kubernetes for Q3 2017
>    - Weekly meeting notes and discussions (link
>    <https://docs.google.com/document/u/1/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit>
>    )
>
> *Recent talks and updates*:
>
>    - Spark on Kubernetes (Spark Summit 2017) - link
>    <https://spark-summit.org/2017/events/apache-spark-on-kubernetes/>
>    - HDFS on Kubernetes - Lessons Learned (Spark Summit 2017) - link
>    <https://spark-summit.org/2017/events/hdfs-on-kubernetes-lessons-learned/>
>
>
> For more details and to contribute to our discussions/efforts, please
> attend our weekly public meeting, or join our slack/mailing list. (link
> <https://github.com/kubernetes/community/tree/master/sig-big-data>)
>