You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Anirudh Ramanathan <fo...@google.com.INVALID> on 2017/06/23 16:50:21 UTC
An Update on Spark on Kubernetes [Jun 23]
*Project Description: *Kubernetes cluster manager integration that enables
native support for submitting Spark applications to a kubernetes cluster.
The submitted applications can make use of Kubernetes native constructs.
*JIRA*: 18278 <https://issues.apache.org/jira/browse/SPARK-18278>
*Upstream Kubernetes Issue*: 34377
<https://github.com/kubernetes/kubernetes/issues/34377>
*Summary of activity in the last 2 weeks*:
- PySpark support being added by ifilolenko (Bloomberg)
- HDFS locality fixes by kimoonkim (PepperData)
- Minor bug fixes and performance improvement (Tencent)
- Planning of work around Spark and HDFS on Kubernetes for Q3 2017
- Weekly meeting notes and discussions (link
<https://docs.google.com/document/u/1/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit>
)
*Recent talks and updates*:
- Spark on Kubernetes (Spark Summit 2017) - link
<https://spark-summit.org/2017/events/apache-spark-on-kubernetes/>
- HDFS on Kubernetes - Lessons Learned (Spark Summit 2017) - link
<https://spark-summit.org/2017/events/hdfs-on-kubernetes-lessons-learned/>
For more details and to contribute to our discussions/efforts, please
attend our weekly public meeting, or join our slack/mailing list. (link
<https://github.com/kubernetes/community/tree/master/sig-big-data>)
Re: An Update on Spark on Kubernetes [Jun 23]
Posted by Reynold Xin <rx...@databricks.com>.
Thanks, Anirudh. This is super helpful!
On Fri, Jun 23, 2017 at 9:50 AM, Anirudh Ramanathan <fo...@google.com>
wrote:
> *Project Description: *Kubernetes cluster manager integration that
> enables native support for submitting Spark applications to a kubernetes
> cluster. The submitted applications can make use of Kubernetes native
> constructs.
>
> *JIRA*: 18278 <https://issues.apache.org/jira/browse/SPARK-18278>
> *Upstream Kubernetes Issue*: 34377
> <https://github.com/kubernetes/kubernetes/issues/34377>
>
> *Summary of activity in the last 2 weeks*:
>
> - PySpark support being added by ifilolenko (Bloomberg)
> - HDFS locality fixes by kimoonkim (PepperData)
> - Minor bug fixes and performance improvement (Tencent)
> - Planning of work around Spark and HDFS on Kubernetes for Q3 2017
> - Weekly meeting notes and discussions (link
> <https://docs.google.com/document/u/1/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit>
> )
>
> *Recent talks and updates*:
>
> - Spark on Kubernetes (Spark Summit 2017) - link
> <https://spark-summit.org/2017/events/apache-spark-on-kubernetes/>
> - HDFS on Kubernetes - Lessons Learned (Spark Summit 2017) - link
> <https://spark-summit.org/2017/events/hdfs-on-kubernetes-lessons-learned/>
>
>
> For more details and to contribute to our discussions/efforts, please
> attend our weekly public meeting, or join our slack/mailing list. (link
> <https://github.com/kubernetes/community/tree/master/sig-big-data>)
>