You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@livy.apache.org by Aliaksandr Sasnouskikh <ja...@gmail.com> on 2020/01/14 11:48:31 UTC
Livy on Kubernetes support
Hi community,
About a year ago I've started to work on the patch to Apache Livy for Spark
on Kubernetes support in the scope of the project I've been working on.
Since that time I've created a PR
https://github.com/apache/incubator-livy/pull/167 which have already been
discussed and reviewed a lot. After finalizing the work in the result of
the PR discussions I've started to split the work introduced in the base PR
into smaller pieces to make it easier to separate the core and aux
functionality, and as a result - easier to review and merge. The first core
PR is https://github.com/apache/incubator-livy/pull/249.
Also I've created the repos with Docker images (
https://github.com/jahstreet/spark-on-kubernetes-docker) and Helm charts (
https://github.com/jahstreet/spark-on-kubernetes-helm) with the possible
stack the users may want to use Livy on Kubernetes with, which potentially
in the future can be partially moved to Livy repo to keep the artifacts
required to run Livy on Kubernetes in a single place.
Until now I've received the positive feedback from more than 10 projects
about the usage of the patch. Several of them could be found in the
discussions of the base PR. Also my repos supporting this feature have
around 35 stars and 15 forks in total and were referenced in Spark related
Stackoverflow and Kubernetes slack channel discussions. So the users use it
already.
You may think "What this guy wants from us then!?"... Well, I would like to
ask for your time and expertise to help push it forward and ideally make it
merged.
Probably before I started coding I should have checked with the
contributors if this feature may have value for the project and how is
better to implement it, but I hope it is never too late;) So I'm here to
share with you the the thoughts behind it.
The idea of Livy on Kubernetes is simply to replicate the logic it has for
Yarn API to Kubernetes API, which can be easily done since the interfaces
for the Yarn API are really similar to the ones of the Kubernetes.
Nevertheless this easy-to-do patch opens Livy the doors to Kubernetes which
seems to be really useful for the community taking into account the
popularity of Kubernetes itself and the latest releases of Spark supporting
Kubernetes as well.
Proposed Livy job submission flow:
- Generate appTag and add
`spark.kubernetes.[driver/executor].label.SPARK_APP_TAG_LABEL` to Spark
config
- Run spark-submit in cluster-mode with Kubernetes master
- Start monitoring thread which resolves Spark Driver and Executor Pods
using the `SPARK_APP_TAG_LABEL`s assigned during the job submission
- Create additional Kubernetes resource if necessary: Spark UI service,
Ingress, CRDs, etc.
- Fetch Spark Pods statuses, Driver logs and other diagnostics information
while Spark Pods are running
- Remove Spark job resources (completed/failed Driver Pod, Service,
ConfigMap, etc.) from the cluster after the job completion/failure after
the configured timeout
The core functionality (covered by
https://github.com/apache/incubator-livy/pull/249):
- Submission of Batch jobs and Interactive sessions
- Caching Driver logs and Kubernetes Pods diagnostics
Aux features (introduced in
https://github.com/apache/incubator-livy/pull/167 and planned to be pushed
after the core PR is merged):
- Proxying Spark UI with Kubernetes Ingress'es and providing the access to
it with the link on the Livy UI while the job is running (Nginx ingress as
an example)
- Proxying Spark History Server with Kubernetes Ingress'es and providing
the access to it with the link on the Livy UI after the job
completion/failure (Nginx ingress as an example)
Misc:
- Add documentation to the Livy web-site and the guide on how to run Livy
on Kubernetes with the possible config options
- Add Docker manifests and Helm charts
Having Livy running on Kubernetes we do get not only the possibility to run
Spark on Kubernetes with the REST API but also get out of the box
integrations:
- Jupyter notebooks (with JupyterHub on Kubernetes) through Sparkmagic
kernel
- Extensive monitoring with Prometheus and Grafana stack
Some more ideas could be found in the readme here:
https://github.com/jahstreet/spark-on-kubernetes-helm.
I would be thankful to everyone paying some time to review the feature I
propose and appreciate any help or advices on the right steps towards
making the feature released.
Kind regards,
Aliaksandr Sasnouskikh
https://github.com/jahstreet/
https://www.linkedin.com/in/jahstreet/
+31 (0) 6 51 29 58 39
Re: Livy on Kubernetes support
Posted by Aliaksandr Sasnouskikh <ja...@gmail.com>.
Hello community,
Recently there was another wave of interest around the *Livy on
Kubernetes* support
PR https://github.com/apache/incubator-livy/pull/249 .
Also multiple times every month I receive questions and feedback from
people trying out this patch in their production environments who
are interested in supporting this by Livy.
I have a list of patches in my backlog to propose on top of the first PR
mentioned above including Helm chart and documentation.
Will be glad to continue supporting Livy on Kubernetes and contributing to
that project, but I need your feedback and help with merging the proposed
patch.
Could you please assist on reviewing it and pushing it forward?
Best regards,
Aliaksandr Sasnouskikh
https://github.com/jahstreet/
https://www.linkedin.com/in/jahstreet/
+31 (0) 6 51 29 58 39
On Tue, 14 Jan 2020 at 23:57, Pat Ferrel <pa...@occamsmachete.com> wrote:
> +1 from another user fwiw. We also have livy containers and helm charts.
> The real problem is deploying a Spark Cluster in k8s. We know of no working
> images for this. The Spark team seems focused on deploying Jobs with k8s,
> which is fine but is not enough. We need to deploy Spark itself. We
> created our own containers and charts for this too.
>
> Is anyone interested in sharing images that work with k8s for Livy and/or
> Spark? Ours are all ASF licensed OSS.
>
> From: Marco Gaido <ma...@gmail.com>
> Reply: dev@livy.incubator.apache.org <de...@livy.incubator.apache.org>
> Date: January 14, 2020 at 2:35:34 PM
> To: dev@livy.incubator.apache.org <de...@livy.incubator.apache.org>
> Subject: Re: Livy on Kubernetes support
>
> Hi Aliaksandr,
>
> thanks for your email and you work on this feature. As I mentioned to you
> in the PR, I agree with you on the usefulness of this feature and you
> have
> a big +1 from me having it in Livy. Unfortunately, it is not my area of
> expertise, so I don't feel confident merging it without other reviewers
> taking a careful look at it.
>
> For the future, I think a better approach would be first to discuss and
> define the architecture with the community, so that it is shared and
> accepted by the whole community before the PR is out. This helps also
> getting people involved and makes easier having them being able to review
> the PR. Anyway, after you have split the PRs, I think they are reasonable
> and we can discuss on them.
>
> Looking forward to have your contribution in Livy.
>
> Thanks,
> Marco
>
> Il giorno mar 14 gen 2020 alle ore 12:48 Aliaksandr Sasnouskikh <
> jahstreetlove@gmail.com> ha scritto:
>
> > Hi community,
> >
> > About a year ago I've started to work on the patch to Apache Livy for
> Spark
> > on Kubernetes support in the scope of the project I've been working on.
> > Since that time I've created a PR
> > https://github.com/apache/incubator-livy/pull/167 which have already
> been
> > discussed and reviewed a lot. After finalizing the work in the result
> of
> > the PR discussions I've started to split the work introduced in the base
> PR
> > into smaller pieces to make it easier to separate the core and aux
> > functionality, and as a result - easier to review and merge. The first
> core
> > PR is https://github.com/apache/incubator-livy/pull/249.
> >
> > Also I've created the repos with Docker images (
> > https://github.com/jahstreet/spark-on-kubernetes-docker) and Helm
> charts (
> > https://github.com/jahstreet/spark-on-kubernetes-helm) with the
> possible
> > stack the users may want to use Livy on Kubernetes with, which
> potentially
> > in the future can be partially moved to Livy repo to keep the artifacts
> > required to run Livy on Kubernetes in a single place.
> >
> > Until now I've received the positive feedback from more than 10
> projects
> > about the usage of the patch. Several of them could be found in the
> > discussions of the base PR. Also my repos supporting this feature have
> > around 35 stars and 15 forks in total and were referenced in Spark
> related
> > Stackoverflow and Kubernetes slack channel discussions. So the users use
> it
> > already.
> >
> > You may think "What this guy wants from us then!?"... Well, I would like
> to
> > ask for your time and expertise to help push it forward and ideally make
> it
> > merged.
> >
> > Probably before I started coding I should have checked with the
> > contributors if this feature may have value for the project and how is
> > better to implement it, but I hope it is never too late;) So I'm here
> to
> > share with you the the thoughts behind it.
> >
> > The idea of Livy on Kubernetes is simply to replicate the logic it has
> for
> > Yarn API to Kubernetes API, which can be easily done since the
> interfaces
> > for the Yarn API are really similar to the ones of the Kubernetes.
> > Nevertheless this easy-to-do patch opens Livy the doors to Kubernetes
> which
> > seems to be really useful for the community taking into account the
> > popularity of Kubernetes itself and the latest releases of Spark
> supporting
> > Kubernetes as well.
> >
> > Proposed Livy job submission flow:
> > - Generate appTag and add
> > `spark.kubernetes.[driver/executor].label.SPARK_APP_TAG_LABEL` to Spark
> > config
> > - Run spark-submit in cluster-mode with Kubernetes master
> > - Start monitoring thread which resolves Spark Driver and Executor Pods
> > using the `SPARK_APP_TAG_LABEL`s assigned during the job submission
> > - Create additional Kubernetes resource if necessary: Spark UI service,
> > Ingress, CRDs, etc.
> > - Fetch Spark Pods statuses, Driver logs and other diagnostics
> information
> > while Spark Pods are running
> > - Remove Spark job resources (completed/failed Driver Pod, Service,
> > ConfigMap, etc.) from the cluster after the job completion/failure
> after
> > the configured timeout
> >
> > The core functionality (covered by
> > https://github.com/apache/incubator-livy/pull/249):
> > - Submission of Batch jobs and Interactive sessions
> > - Caching Driver logs and Kubernetes Pods diagnostics
> >
> > Aux features (introduced in
> > https://github.com/apache/incubator-livy/pull/167 and planned to be
> pushed
> > after the core PR is merged):
> > - Proxying Spark UI with Kubernetes Ingress'es and providing the access
> to
> > it with the link on the Livy UI while the job is running (Nginx ingress
> as
> > an example)
> > - Proxying Spark History Server with Kubernetes Ingress'es and
> providing
> > the access to it with the link on the Livy UI after the job
> > completion/failure (Nginx ingress as an example)
> >
> > Misc:
> > - Add documentation to the Livy web-site and the guide on how to run
> Livy
> > on Kubernetes with the possible config options
> > - Add Docker manifests and Helm charts
> >
> > Having Livy running on Kubernetes we do get not only the possibility to
> run
> > Spark on Kubernetes with the REST API but also get out of the box
> > integrations:
> > - Jupyter notebooks (with JupyterHub on Kubernetes) through Sparkmagic
> > kernel
> > - Extensive monitoring with Prometheus and Grafana stack
> >
> > Some more ideas could be found in the readme here:
> > https://github.com/jahstreet/spark-on-kubernetes-helm.
> >
> > I would be thankful to everyone paying some time to review the feature
> I
> > propose and appreciate any help or advices on the right steps towards
> > making the feature released.
> >
> > Kind regards,
> > Aliaksandr Sasnouskikh
> > https://github.com/jahstreet/
> > https://www.linkedin.com/in/jahstreet/
> > +31 (0) 6 51 29 58 39
> >
>
Re: Livy on Kubernetes support
Posted by Pat Ferrel <pa...@occamsmachete.com>.
+1 from another user fwiw. We also have livy containers and helm charts. The real problem is deploying a Spark Cluster in k8s. We know of no working images for this. The Spark team seems focused on deploying Jobs with k8s, which is fine but is not enough. We need to deploy Spark itself. We created our own containers and charts for this too.
Is anyone interested in sharing images that work with k8s for Livy and/or Spark? Ours are all ASF licensed OSS.
From: Marco Gaido <ma...@gmail.com>
Reply: dev@livy.incubator.apache.org <de...@livy.incubator.apache.org>
Date: January 14, 2020 at 2:35:34 PM
To: dev@livy.incubator.apache.org <de...@livy.incubator.apache.org>
Subject: Re: Livy on Kubernetes support
Hi Aliaksandr,
thanks for your email and you work on this feature. As I mentioned to you
in the PR, I agree with you on the usefulness of this feature and you have
a big +1 from me having it in Livy. Unfortunately, it is not my area of
expertise, so I don't feel confident merging it without other reviewers
taking a careful look at it.
For the future, I think a better approach would be first to discuss and
define the architecture with the community, so that it is shared and
accepted by the whole community before the PR is out. This helps also
getting people involved and makes easier having them being able to review
the PR. Anyway, after you have split the PRs, I think they are reasonable
and we can discuss on them.
Looking forward to have your contribution in Livy.
Thanks,
Marco
Il giorno mar 14 gen 2020 alle ore 12:48 Aliaksandr Sasnouskikh <
jahstreetlove@gmail.com> ha scritto:
> Hi community,
>
> About a year ago I've started to work on the patch to Apache Livy for Spark
> on Kubernetes support in the scope of the project I've been working on.
> Since that time I've created a PR
> https://github.com/apache/incubator-livy/pull/167 which have already been
> discussed and reviewed a lot. After finalizing the work in the result of
> the PR discussions I've started to split the work introduced in the base PR
> into smaller pieces to make it easier to separate the core and aux
> functionality, and as a result - easier to review and merge. The first core
> PR is https://github.com/apache/incubator-livy/pull/249.
>
> Also I've created the repos with Docker images (
> https://github.com/jahstreet/spark-on-kubernetes-docker) and Helm charts (
> https://github.com/jahstreet/spark-on-kubernetes-helm) with the possible
> stack the users may want to use Livy on Kubernetes with, which potentially
> in the future can be partially moved to Livy repo to keep the artifacts
> required to run Livy on Kubernetes in a single place.
>
> Until now I've received the positive feedback from more than 10 projects
> about the usage of the patch. Several of them could be found in the
> discussions of the base PR. Also my repos supporting this feature have
> around 35 stars and 15 forks in total and were referenced in Spark related
> Stackoverflow and Kubernetes slack channel discussions. So the users use it
> already.
>
> You may think "What this guy wants from us then!?"... Well, I would like to
> ask for your time and expertise to help push it forward and ideally make it
> merged.
>
> Probably before I started coding I should have checked with the
> contributors if this feature may have value for the project and how is
> better to implement it, but I hope it is never too late;) So I'm here to
> share with you the the thoughts behind it.
>
> The idea of Livy on Kubernetes is simply to replicate the logic it has for
> Yarn API to Kubernetes API, which can be easily done since the interfaces
> for the Yarn API are really similar to the ones of the Kubernetes.
> Nevertheless this easy-to-do patch opens Livy the doors to Kubernetes which
> seems to be really useful for the community taking into account the
> popularity of Kubernetes itself and the latest releases of Spark supporting
> Kubernetes as well.
>
> Proposed Livy job submission flow:
> - Generate appTag and add
> `spark.kubernetes.[driver/executor].label.SPARK_APP_TAG_LABEL` to Spark
> config
> - Run spark-submit in cluster-mode with Kubernetes master
> - Start monitoring thread which resolves Spark Driver and Executor Pods
> using the `SPARK_APP_TAG_LABEL`s assigned during the job submission
> - Create additional Kubernetes resource if necessary: Spark UI service,
> Ingress, CRDs, etc.
> - Fetch Spark Pods statuses, Driver logs and other diagnostics information
> while Spark Pods are running
> - Remove Spark job resources (completed/failed Driver Pod, Service,
> ConfigMap, etc.) from the cluster after the job completion/failure after
> the configured timeout
>
> The core functionality (covered by
> https://github.com/apache/incubator-livy/pull/249):
> - Submission of Batch jobs and Interactive sessions
> - Caching Driver logs and Kubernetes Pods diagnostics
>
> Aux features (introduced in
> https://github.com/apache/incubator-livy/pull/167 and planned to be pushed
> after the core PR is merged):
> - Proxying Spark UI with Kubernetes Ingress'es and providing the access to
> it with the link on the Livy UI while the job is running (Nginx ingress as
> an example)
> - Proxying Spark History Server with Kubernetes Ingress'es and providing
> the access to it with the link on the Livy UI after the job
> completion/failure (Nginx ingress as an example)
>
> Misc:
> - Add documentation to the Livy web-site and the guide on how to run Livy
> on Kubernetes with the possible config options
> - Add Docker manifests and Helm charts
>
> Having Livy running on Kubernetes we do get not only the possibility to run
> Spark on Kubernetes with the REST API but also get out of the box
> integrations:
> - Jupyter notebooks (with JupyterHub on Kubernetes) through Sparkmagic
> kernel
> - Extensive monitoring with Prometheus and Grafana stack
>
> Some more ideas could be found in the readme here:
> https://github.com/jahstreet/spark-on-kubernetes-helm.
>
> I would be thankful to everyone paying some time to review the feature I
> propose and appreciate any help or advices on the right steps towards
> making the feature released.
>
> Kind regards,
> Aliaksandr Sasnouskikh
> https://github.com/jahstreet/
> https://www.linkedin.com/in/jahstreet/
> +31 (0) 6 51 29 58 39
>
Re: Livy on Kubernetes support
Posted by Marco Gaido <ma...@gmail.com>.
Hi Aliaksandr,
thanks for your email and you work on this feature. As I mentioned to you
in the PR, I agree with you on the usefulness of this feature and you have
a big +1 from me having it in Livy. Unfortunately, it is not my area of
expertise, so I don't feel confident merging it without other reviewers
taking a careful look at it.
For the future, I think a better approach would be first to discuss and
define the architecture with the community, so that it is shared and
accepted by the whole community before the PR is out. This helps also
getting people involved and makes easier having them being able to review
the PR. Anyway, after you have split the PRs, I think they are reasonable
and we can discuss on them.
Looking forward to have your contribution in Livy.
Thanks,
Marco
Il giorno mar 14 gen 2020 alle ore 12:48 Aliaksandr Sasnouskikh <
jahstreetlove@gmail.com> ha scritto:
> Hi community,
>
> About a year ago I've started to work on the patch to Apache Livy for Spark
> on Kubernetes support in the scope of the project I've been working on.
> Since that time I've created a PR
> https://github.com/apache/incubator-livy/pull/167 which have already been
> discussed and reviewed a lot. After finalizing the work in the result of
> the PR discussions I've started to split the work introduced in the base PR
> into smaller pieces to make it easier to separate the core and aux
> functionality, and as a result - easier to review and merge. The first core
> PR is https://github.com/apache/incubator-livy/pull/249.
>
> Also I've created the repos with Docker images (
> https://github.com/jahstreet/spark-on-kubernetes-docker) and Helm charts (
> https://github.com/jahstreet/spark-on-kubernetes-helm) with the possible
> stack the users may want to use Livy on Kubernetes with, which potentially
> in the future can be partially moved to Livy repo to keep the artifacts
> required to run Livy on Kubernetes in a single place.
>
> Until now I've received the positive feedback from more than 10 projects
> about the usage of the patch. Several of them could be found in the
> discussions of the base PR. Also my repos supporting this feature have
> around 35 stars and 15 forks in total and were referenced in Spark related
> Stackoverflow and Kubernetes slack channel discussions. So the users use it
> already.
>
> You may think "What this guy wants from us then!?"... Well, I would like to
> ask for your time and expertise to help push it forward and ideally make it
> merged.
>
> Probably before I started coding I should have checked with the
> contributors if this feature may have value for the project and how is
> better to implement it, but I hope it is never too late;) So I'm here to
> share with you the the thoughts behind it.
>
> The idea of Livy on Kubernetes is simply to replicate the logic it has for
> Yarn API to Kubernetes API, which can be easily done since the interfaces
> for the Yarn API are really similar to the ones of the Kubernetes.
> Nevertheless this easy-to-do patch opens Livy the doors to Kubernetes which
> seems to be really useful for the community taking into account the
> popularity of Kubernetes itself and the latest releases of Spark supporting
> Kubernetes as well.
>
> Proposed Livy job submission flow:
> - Generate appTag and add
> `spark.kubernetes.[driver/executor].label.SPARK_APP_TAG_LABEL` to Spark
> config
> - Run spark-submit in cluster-mode with Kubernetes master
> - Start monitoring thread which resolves Spark Driver and Executor Pods
> using the `SPARK_APP_TAG_LABEL`s assigned during the job submission
> - Create additional Kubernetes resource if necessary: Spark UI service,
> Ingress, CRDs, etc.
> - Fetch Spark Pods statuses, Driver logs and other diagnostics information
> while Spark Pods are running
> - Remove Spark job resources (completed/failed Driver Pod, Service,
> ConfigMap, etc.) from the cluster after the job completion/failure after
> the configured timeout
>
> The core functionality (covered by
> https://github.com/apache/incubator-livy/pull/249):
> - Submission of Batch jobs and Interactive sessions
> - Caching Driver logs and Kubernetes Pods diagnostics
>
> Aux features (introduced in
> https://github.com/apache/incubator-livy/pull/167 and planned to be pushed
> after the core PR is merged):
> - Proxying Spark UI with Kubernetes Ingress'es and providing the access to
> it with the link on the Livy UI while the job is running (Nginx ingress as
> an example)
> - Proxying Spark History Server with Kubernetes Ingress'es and providing
> the access to it with the link on the Livy UI after the job
> completion/failure (Nginx ingress as an example)
>
> Misc:
> - Add documentation to the Livy web-site and the guide on how to run Livy
> on Kubernetes with the possible config options
> - Add Docker manifests and Helm charts
>
> Having Livy running on Kubernetes we do get not only the possibility to run
> Spark on Kubernetes with the REST API but also get out of the box
> integrations:
> - Jupyter notebooks (with JupyterHub on Kubernetes) through Sparkmagic
> kernel
> - Extensive monitoring with Prometheus and Grafana stack
>
> Some more ideas could be found in the readme here:
> https://github.com/jahstreet/spark-on-kubernetes-helm.
>
> I would be thankful to everyone paying some time to review the feature I
> propose and appreciate any help or advices on the right steps towards
> making the feature released.
>
> Kind regards,
> Aliaksandr Sasnouskikh
> https://github.com/jahstreet/
> https://www.linkedin.com/in/jahstreet/
> +31 (0) 6 51 29 58 39
>