You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@livy.apache.org by "nileshrathi345 (via GitHub)" <gi...@apache.org> on 2026/08/10 08:47:31 UTC
[PR] LIVY-979: Create a helm-based integration environment for debugging Livy on Kubernetes [livy]
nileshrathi345 opened a new pull request, #537:
URL: https://github.com/apache/livy/pull/537
## What changes were proposed in this pull request?
Adds a Helm chart under dev/helmchart/ for running Livy on a local Kubernetes cluster (e.g. Docker Desktop) for development and debugging, without cloud dependencies.
**Included:**
- Helm chart for Livy (StatefulSet), Spark History Server, RBAC, Services, PVC/StorageClass, and optional ingress + TLS
- Optional subcharts: ingress-nginx and cert-manager (gated via values)
- `values.yaml`aligned with Livy 1.0.0-SNAPSHOT, Spark 3.5.6, Scala 2.12
- Docs: `README.md` (install, smoke tests, optional Grafana/Loki), `Docker.md` (image build steps)
- `test/validate.sh` — ASF license header checks, version consistency, helm lint / helm template
- ASF license headers on chart source files; `.rat-excludes` updated only for generated `Chart.lock`
**Template fixes found during local testing:**
- Ingress `apiVersion` rendering (trim directive)
- cert-manager / ClusterIssuer / Certificate resources only when cert-manager is enabled
- Spark History Server startup fix (`HADOOP_USER_NAME=root` for local Hadoop user resolution)
This PR is **dev/ops tooling only** — no changes to Livy server runtime code.
JIRA: [LIVY-979](https://issues.apache.org/jira/browse/LIVY-979)
## How was this patch tested?
**Automated / local chart checks:**
- `dev/helmchart/test/validate.sh `— license headers, version keys, helm dependency build, helm lint, helm template
**Manual testing (Docker Desktop Kubernetes, namespace livy-dev)**
- Built local images per `Docker.md` (`livy-local/spark-py:v3.5.6`, `livy-local/livy:spark3.5.6`)
- Deployed with `helm install `/ `helm upgrade`
- Verified pods: `livycluster-0`, ingress-nginx, cert-manager, `spark-history-0`
- **REST via ingress/TLS**: `GET /livy/version` → 200
- **Interactive session**: created pyspark session, ran `1+1`, `sc.parallelize(...).count()`, `spark.range(5).collect()` — all succeeded
- Spark driver and executor pods reached Running on Kubernetes
## Was this patch authored or co-authored using generative AI tooling?
Yes, co-authored by Cursor AI
Note: This PR builds on the original Helm chart work contributed by @askhatri
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-979: Create a helm-based integration environment for debugging Livy on Kubernetes [livy]
Posted by "gyogal (via GitHub)" <gi...@apache.org>.
gyogal commented on code in PR #537:
URL: https://github.com/apache/livy/pull/537#discussion_r3752549111
##########
.rat-excludes:
##########
@@ -30,3 +30,4 @@ logs/*
docs/**/*.html
docs/**/JB/**
venv/*
+dev/helmchart/Chart.lock
Review Comment:
License headers have been added to all files except this one, but since this file is generated by `helm dependency build`, any license headers are overwritten. I checked other Apache projects and adding this file into `.rat-excludes` seems be common practice, so this matches the setup found in other project:
https://github.com/apache/airflow/blob/8c16340ca4903e3974b5342c0fe8ca1f0534395f/.rat-excludes#L118
https://github.com/apache/pulsar-helm-chart/blob/328846044c8e1df50399c28bc63887696b6d58dc/.rat-excludes#L20
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-979: Create a helm-based integration environment for debugging Livy on Kubernetes [livy]
Posted by "nileshrathi345 (via GitHub)" <gi...@apache.org>.
nileshrathi345 commented on PR #537:
URL: https://github.com/apache/livy/pull/537#issuecomment-5237943335
Hi @askhatri — thanks for the original LIVY-979 Helm chart work in this PR.
I've opened an updated revision at [PR #454](https://github.com/apache/livy/pull/454) that keeps your
contribution as the base and adds ASF license headers, Helm 3 updates,
validation script, and manual testing on Docker Desktop Kubernetes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-979: Create a helm-based integration environment for debugging Livy on Kubernetes [livy]
Posted by "gyogal (via GitHub)" <gi...@apache.org>.
gyogal commented on code in PR #537:
URL: https://github.com/apache/livy/pull/537#discussion_r3752549111
##########
.rat-excludes:
##########
@@ -30,3 +30,4 @@ logs/*
docs/**/*.html
docs/**/JB/**
venv/*
+dev/helmchart/Chart.lock
Review Comment:
License headers have been added to all files except this one, but since this file is generated by `helm dependency build`, any license headers are overwritten. I checked other Apache projects and adding this file into `.rat-excludes` seems be common practice, so this matches the setup found in other projects:
https://github.com/apache/airflow/blob/8c16340ca4903e3974b5342c0fe8ca1f0534395f/.rat-excludes#L118
https://github.com/apache/pulsar-helm-chart/blob/328846044c8e1df50399c28bc63887696b6d58dc/.rat-excludes#L20
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-979: Create a helm-based integration environment for debugging Livy on Kubernetes [livy]
Posted by "gyogal (via GitHub)" <gi...@apache.org>.
gyogal commented on code in PR #537:
URL: https://github.com/apache/livy/pull/537#discussion_r3752549111
##########
.rat-excludes:
##########
@@ -30,3 +30,4 @@ logs/*
docs/**/*.html
docs/**/JB/**
venv/*
+dev/helmchart/Chart.lock
Review Comment:
License headers have been added to all files except this one, but since this file is generated by `helm dependency build`, any license headers are overwritten. I checked other Apache projects and adding this file into `.rat-excludes` seems to be common practice, so this matches the setup found in other projects:
https://github.com/apache/airflow/blob/8c16340ca4903e3974b5342c0fe8ca1f0534395f/.rat-excludes#L118
https://github.com/apache/pulsar-helm-chart/blob/328846044c8e1df50399c28bc63887696b6d58dc/.rat-excludes#L20
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-979: Create a helm-based integration environment for debugging Livy on Kubernetes [livy]
Posted by "roczei (via GitHub)" <gi...@apache.org>.
roczei commented on PR #537:
URL: https://github.com/apache/livy/pull/537#issuecomment-5269303126
> could please take a look when possible and review. thanks..!!!
Hi @nileshrathi345,
I don't have experience with Helm charts, so I won't be able to review this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-979: Create a helm-based integration environment for debugging Livy on Kubernetes [livy]
Posted by "nileshrathi345 (via GitHub)" <gi...@apache.org>.
nileshrathi345 commented on PR #537:
URL: https://github.com/apache/livy/pull/537#issuecomment-5269039903
@gyogal, @roczei and @ArnavBalyan could please take a look when possible and review.
thanks..!!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] LIVY-979: Create a helm-based integration environment for debugging Livy on Kubernetes [livy]
Posted by "gyogal (via GitHub)" <gi...@apache.org>.
gyogal merged PR #537:
URL: https://github.com/apache/livy/pull/537
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org