You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@texera.apache.org by "aicam (via GitHub)" <gi...@apache.org> on 2026/06/12 18:25:26 UTC
[D] Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform [texera]
GitHub user aicam created a discussion: Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform
## Problem
Today `bin/k8s` on `main` ships a single Helm chart that targets a **local / on-premise Kubernetes cluster only**. Our AWS (EKS) deployment at `dknet-ai.org` runs the *same* Helm chart, but everything underneath it — the EKS cluster, S3 bucket, IAM/IRSA roles, Elastic IP, DNS, node pools, cert-manager — is created **by hand** and lives nowhere in the repo. None of that substrate is captured as code.
The result: standing up Texera on a fresh AWS account is an undocumented, manual prerequisite checklist that drifts and is hard to reproduce. We'd like to **unify the local and AWS deployment stories** so there's one documented, reproducible path from "empty cluster (or empty AWS account)" to "running Texera," while keeping the Helm chart itself cloud-agnostic.
## Scope of "substrate"
The Helm chart already covers the *application*. The open question is only how we provision/document the *substrate* it runs on:
- EKS cluster + managed node pools
- IAM roles / IRSA for service accounts
- S3 bucket (LakeFS backend), Elastic IP, DNS, cert-manager / Let's Encrypt
## Options considered
### 1. Helm only
Deploy the chart into a cluster that already exists. No AWS substrate creation.
**Pros**
- Simplest — one `helm install`
- Identical for local and AWS — fully cloud-agnostic
- Nothing extra to learn or maintain in the repo
- Keeps the Apache PR scoped to just the chart
**Cons**
- Doesn't create the cluster, S3 bucket, IAM, EIP, node pools, or cert-manager — all manual on a new AWS account
- New-AWS-account UX = a long prerequisite checklist done by hand
- Substrate stays undocumented-as-code → drift, hard to reproduce
### 2. eksctl
AWS CLI that provisions an EKS cluster + node pools + IAM/IRSA + add-ons from a small YAML, via CloudFormation. Helm still deploys on top.
**Pros**
- Purpose-built for EKS; minimal config for a working cluster
- Handles IRSA, node pools, and add-ons natively
- "New AWS account → cluster" in ~one command
- Config file is committable and reproducible
**Cons**
- AWS-only — no use for local or other clouds
- Doesn't manage non-cluster resources well (S3 bucket, EIP, DNS) — you still script those
- CloudFormation-based: slower, less granular state control
- Adds an AWS-specific tool to a repo that's otherwise portable
### 3. Terraform
Declarative IaC that can provision everything: EKS cluster, S3, IAM, EIP, DNS, node pools — and across clouds. Helm deploys on top (or via the Helm provider).
**Pros**
- Most reproducible & reviewable (full state, plan/apply, diffs)
- One tool for all substrate, not just the cluster
- Cloud-agnostic — same tool for AWS / GCP / local
- Can even invoke Helm itself, so one `apply` does substrate + deploy
**Cons**
- Heaviest to set up and maintain (modules, remote state, locking)
- Steeper learning curve; more moving parts
- State management is its own operational burden
- Overkill if you only ever target one EKS cluster
## Our current position: eksctl
We're currently leaning toward **eksctl**, because:
- **GCP and Azure are low priority for now** — the cloud-agnostic advantage of Terraform doesn't pay off yet, so we'd rather not take on its maintenance cost speculatively.
- **We prefer easier maintenance over easier deploying.** eksctl keeps the AWS-specific config small and declarative while letting the Helm chart stay fully portable, rather than introducing Terraform modules + remote state + locking that we'd have to operate.
- eksctl handles the genuinely fiddly EKS-specific parts (IRSA, node pools, add-ons) natively, and the cluster config is committable.
Regardless of the choice, a small amount of non-cluster substrate (S3 bucket, EIP, DNS) would still be scripted alongside.
## We'd love your input
Which approach should we standardize on for upstreaming AWS deployment into `bin/k8s`? Please vote with a reaction on this post and share your reasoning in the comments — especially if you deploy Texera (or similar) on a cloud we haven't prioritized:
- 👍 — **Helm only**
- 🚀 — **eksctl** (our current lean)
- ❤️ — **Terraform**
GitHub link: https://github.com/apache/texera/discussions/5641
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform [texera]
Posted by "aicam (via GitHub)" <gi...@apache.org>.
GitHub user aicam added a comment to the discussion: Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform
Terraform indeed support AWS and on-prem, it sits on top of our stack (Helm) and manage infrastructure. It is used to define set of services and resources to deploy Helm charts on specific cloud providers including Heroku, AWS, GCP, Azure, ... . Our current plan is to standardize our Helm charts and use eksctl to deploy the application on AWS and on-prem.
Terraform is overkill because it is not required for on-perm deployment (including people who use their own clusters) and is also not mandatory for AWS because eksctl is the native AWS tool. In the future when we want to add GCP or any other cloud, we will most likely replace eksctl with Terraform but the efforts to standardize and clean our Helm charts is orthogonal to Terraform or eksctl so we prioritize cleaning our Helm charts and later once we have them, we can add Terraform on top of that.
GitHub link: https://github.com/apache/texera/discussions/5641#discussioncomment-17395200
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform [texera]
Posted by "sshiv012 (via GitHub)" <gi...@apache.org>.
GitHub user sshiv012 added a comment to the discussion: Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform
I'm still getting familiar with the AWS side of the deployment, but here's what stood out reading through bin/k8s.
@chenlica standard eksctl does not support on-prem. It's the **AWS-only** CLI for provisioning Amazon EKS clusters as @aicam mentioned. What should be common for both flavors of deployment is Helm. Not sure if we use [EKS ANYWHERE](https://anywhere.eks.amazonaws.com/) or just barebones k8s on prem, if its eks anywhere then we can maybe reuse it.
One thing I wasn't sure was fully captured in the framing: it might help to separate the application chart from the substrate, because as far as I can tell only the substrate is really in question.
On the chart side, texera-helm already looks closer to cloud-agnostic than I expected, the persistence templates only create hostPath PVs when the storage class is local-path (minio-persistence.yaml, postgresql-persistence.yaml), so other classes fall through to dynamic provisioning, and LakeFS already targets blockstore.type: s3 rather than MinIO-specific APIs. If I'm reading that right, moving to EBS + real S3 is mostly a values change, not a template change.
That leaves the substrate i.e cluster, node pools, IRSA, S3, EIP, DNS, cert-manager as the only layer the eksctl-vs-Terraform question is actually about.
On eksctl: I agree it's the right tool for the cluster layer (node groups, Pod Identity). My one hesitation is that it doesn't get us all the way to "one path to reproduce the deployment" — as @aicam noted, S3/EIP/DNS are out of scpe for it, so if those stay manual we may partly recreate the undocumented-setup problem we're trying to solve. That's where Terraform's plan is appealing: it warns before it deletes a data bucket or changes IAM, detects drift if someone clicks around in the console, and the code doubles as documentation.
But I think it really comes down to who owns the deployment:
- Project maintainers (keep-it-simple, rotating contributors): eksctl + Helm + a committed bootstrap script is probably the lighter path.
- General adopters (integrating into a real org/projects): Terraform is the better thing to offer — portable, composable into existing IaC, self-documenting, and usually the tool they already speak.
So maybe it isn't either/or. We can consider (eventually) shipping a thin, optional Terraform module (or even just documented .tf for S3/IAM/IRSA plus a reference eksctl cluster config), so maintainers stay on the simple path while adopters get something that drops into their stack.
GitHub link: https://github.com/apache/texera/discussions/5641#discussioncomment-17301158
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform [texera]
Posted by "aicam (via GitHub)" <gi...@apache.org>.
GitHub user aicam added a comment to the discussion: Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform
Thanks everyone for your help and suggestions, based on the discussion, we decided to go with Helm + eksctl without Terraform. The main reason is that the people who deploy the service are technical people and we can let them handle the detail they need and we provide base requirements for them to deploy.
GitHub link: https://github.com/apache/texera/discussions/5641#discussioncomment-17330094
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform [texera]
Posted by "aicam (via GitHub)" <gi...@apache.org>.
GitHub user aicam edited a comment on the discussion: Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform
@chenlica Terraform indeed support AWS and on-prem, it sits on top of our stack (Helm) and manage infrastructure. It is used to define set of services and resources to deploy Helm charts on specific cloud providers including Heroku, AWS, GCP, Azure, ... . Our current plan is to standardize our Helm charts and use eksctl to deploy the application on AWS and on-prem.
Terraform is overkill because it is not required for on-perm deployment (including people who use their own clusters) and is also not mandatory for AWS because eksctl is the native AWS tool. In the future when we want to add GCP or any other cloud, we will most likely replace eksctl with Terraform but the efforts to standardize and clean our Helm charts is orthogonal to Terraform or eksctl so we prioritize cleaning our Helm charts and later once we have them, we can add Terraform on top of that.
GitHub link: https://github.com/apache/texera/discussions/5641#discussioncomment-17395200
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica closed the discussion with a comment: Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform
Thanks for the clear explanation.
GitHub link: https://github.com/apache/texera/discussions/5641#discussioncomment-17403392
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica added a comment to the discussion: Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform
I still don't see a clear reason why Terraform is not considered. @aicam Please be more specific about the reasoning.
GitHub link: https://github.com/apache/texera/discussions/5641#discussioncomment-17372372
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform [texera]
Posted by "zuozhiw (via GitHub)" <gi...@apache.org>.
GitHub user zuozhiw closed the discussion with a comment: Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform
I second @aicam here, I also use both terraform and helm charts for work. As Ali said, they are at different layers, and using terraform for one or two envs is a bit overkill and its value start to show more clearly after we need to manage more of envs, terraform has this benefit of managing infra as code, but it also brings a bit extra complexity.
GitHub link: https://github.com/apache/texera/discussions/5641#discussioncomment-17403640
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica added a comment to the discussion: Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform
@aicam To make the reason clear, Terraform doesn't support AWS nor on-prem, right?
GitHub link: https://github.com/apache/texera/discussions/5641#discussioncomment-17360129
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform [texera]
Posted by "aicam (via GitHub)" <gi...@apache.org>.
GitHub user aicam closed a discussion: Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform
## Problem
Today `bin/k8s` on `main` ships a single Helm chart that targets a **local / on-premise Kubernetes cluster only**. Our AWS (EKS) deployment runs the *same* Helm chart, but everything underneath it — the EKS cluster, S3 bucket, IAM/IRSA roles, Elastic IP, DNS, node pools, cert-manager — is created **by hand** and lives nowhere in the repo. None of that substrate is captured as code.
The result: standing up Texera on a fresh AWS account is an undocumented, manual prerequisite checklist that drifts and is hard to reproduce. We'd like to **unify the local and AWS deployment stories** so there's one documented, reproducible path from "empty cluster (or empty AWS account)" to "running Texera," while keeping the Helm chart itself cloud-agnostic.
## Scope of "substrate"
The Helm chart already covers the *application*. The open question is only how we provision/document the *substrate* it runs on:
- EKS cluster + managed node pools
- IAM roles / IRSA for service accounts
- S3 bucket (LakeFS backend), Elastic IP, DNS, cert-manager / Let's Encrypt
## Options considered
### 1. Helm only
Deploy the chart into a cluster that already exists. No AWS substrate creation.
**Pros**
- Simplest — one `helm install`
- Identical for local and AWS — fully cloud-agnostic
- Nothing extra to learn or maintain in the repo
- Keeps the Apache PR scoped to just the chart
**Cons**
- Doesn't create the cluster, S3 bucket, IAM, EIP, node pools, or cert-manager — all manual on a new AWS account
- New-AWS-account UX = a long prerequisite checklist done by hand
- Substrate stays undocumented-as-code → drift, hard to reproduce
### 2. eksctl
AWS CLI that provisions an EKS cluster + node pools + IAM/IRSA + add-ons from a small YAML, via CloudFormation. Helm still deploys on top.
**Pros**
- Purpose-built for EKS; minimal config for a working cluster
- Handles IRSA, node pools, and add-ons natively
- "New AWS account → cluster" in ~one command
- Config file is committable and reproducible
**Cons**
- AWS-only — no use for local or other clouds
- Doesn't manage non-cluster resources well (S3 bucket, EIP, DNS) — you still script those
- CloudFormation-based: slower, less granular state control
- Adds an AWS-specific tool to a repo that's otherwise portable
### 3. Terraform
Declarative IaC that can provision everything: EKS cluster, S3, IAM, EIP, DNS, node pools — and across clouds. Helm deploys on top (or via the Helm provider).
**Pros**
- Most reproducible & reviewable (full state, plan/apply, diffs)
- One tool for all substrate, not just the cluster
- Cloud-agnostic — same tool for AWS / GCP / local
- Can even invoke Helm itself, so one `apply` does substrate + deploy
**Cons**
- Heaviest to set up and maintain (modules, remote state, locking)
- Steeper learning curve; more moving parts
- State management is its own operational burden
- Overkill if you only ever target one EKS cluster
## Our current position: eksctl
We're currently leaning toward **eksctl**, because:
- **GCP and Azure are low priority for now** — the cloud-agnostic advantage of Terraform doesn't pay off yet, so we'd rather not take on its maintenance cost speculatively.
- **We prefer easier maintenance over easier deploying.** eksctl keeps the AWS-specific config small and declarative while letting the Helm chart stay fully portable, rather than introducing Terraform modules + remote state + locking that we'd have to operate.
- eksctl handles the genuinely fiddly EKS-specific parts (IRSA, node pools, add-ons) natively, and the cluster config is committable.
Regardless of the choice, a small amount of non-cluster substrate (S3 bucket, EIP, DNS) would still be scripted alongside.
## We'd love your input
Which approach should we standardize on for upstreaming AWS deployment into `bin/k8s`? Please vote with a reaction on this post and share your reasoning in the comments — especially if you deploy Texera (or similar) on a cloud we haven't prioritized:
- 👍 — **Helm only**
- 🚀 — **eksctl** (our current lean)
- ❤️ — **Terraform**
GitHub link: https://github.com/apache/texera/discussions/5641
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform [texera]
Posted by "aicam (via GitHub)" <gi...@apache.org>.
GitHub user aicam reopened a discussion: Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform
## Problem
Today `bin/k8s` on `main` ships a single Helm chart that targets a **local / on-premise Kubernetes cluster only**. Our AWS (EKS) deployment runs the *same* Helm chart, but everything underneath it — the EKS cluster, S3 bucket, IAM/IRSA roles, Elastic IP, DNS, node pools, cert-manager — is created **by hand** and lives nowhere in the repo. None of that substrate is captured as code.
The result: standing up Texera on a fresh AWS account is an undocumented, manual prerequisite checklist that drifts and is hard to reproduce. We'd like to **unify the local and AWS deployment stories** so there's one documented, reproducible path from "empty cluster (or empty AWS account)" to "running Texera," while keeping the Helm chart itself cloud-agnostic.
## Scope of "substrate"
The Helm chart already covers the *application*. The open question is only how we provision/document the *substrate* it runs on:
- EKS cluster + managed node pools
- IAM roles / IRSA for service accounts
- S3 bucket (LakeFS backend), Elastic IP, DNS, cert-manager / Let's Encrypt
## Options considered
### 1. Helm only
Deploy the chart into a cluster that already exists. No AWS substrate creation.
**Pros**
- Simplest — one `helm install`
- Identical for local and AWS — fully cloud-agnostic
- Nothing extra to learn or maintain in the repo
- Keeps the Apache PR scoped to just the chart
**Cons**
- Doesn't create the cluster, S3 bucket, IAM, EIP, node pools, or cert-manager — all manual on a new AWS account
- New-AWS-account UX = a long prerequisite checklist done by hand
- Substrate stays undocumented-as-code → drift, hard to reproduce
### 2. eksctl
AWS CLI that provisions an EKS cluster + node pools + IAM/IRSA + add-ons from a small YAML, via CloudFormation. Helm still deploys on top.
**Pros**
- Purpose-built for EKS; minimal config for a working cluster
- Handles IRSA, node pools, and add-ons natively
- "New AWS account → cluster" in ~one command
- Config file is committable and reproducible
**Cons**
- AWS-only — no use for local or other clouds
- Doesn't manage non-cluster resources well (S3 bucket, EIP, DNS) — you still script those
- CloudFormation-based: slower, less granular state control
- Adds an AWS-specific tool to a repo that's otherwise portable
### 3. Terraform
Declarative IaC that can provision everything: EKS cluster, S3, IAM, EIP, DNS, node pools — and across clouds. Helm deploys on top (or via the Helm provider).
**Pros**
- Most reproducible & reviewable (full state, plan/apply, diffs)
- One tool for all substrate, not just the cluster
- Cloud-agnostic — same tool for AWS / GCP / local
- Can even invoke Helm itself, so one `apply` does substrate + deploy
**Cons**
- Heaviest to set up and maintain (modules, remote state, locking)
- Steeper learning curve; more moving parts
- State management is its own operational burden
- Overkill if you only ever target one EKS cluster
## Our current position: eksctl
We're currently leaning toward **eksctl**, because:
- **GCP and Azure are low priority for now** — the cloud-agnostic advantage of Terraform doesn't pay off yet, so we'd rather not take on its maintenance cost speculatively.
- **We prefer easier maintenance over easier deploying.** eksctl keeps the AWS-specific config small and declarative while letting the Helm chart stay fully portable, rather than introducing Terraform modules + remote state + locking that we'd have to operate.
- eksctl handles the genuinely fiddly EKS-specific parts (IRSA, node pools, add-ons) natively, and the cluster config is committable.
Regardless of the choice, a small amount of non-cluster substrate (S3 bucket, EIP, DNS) would still be scripted alongside.
## We'd love your input
Which approach should we standardize on for upstreaming AWS deployment into `bin/k8s`? Please vote with a reaction on this post and share your reasoning in the comments — especially if you deploy Texera (or similar) on a cloud we haven't prioritized:
- 👍 — **Helm only**
- 🚀 — **eksctl** (our current lean)
- ❤️ — **Terraform**
GitHub link: https://github.com/apache/texera/discussions/5641
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform [texera]
Posted by "aicam (via GitHub)" <gi...@apache.org>.
GitHub user aicam closed the discussion with a comment: Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform
Thanks @madhusudancs, thanks for your comments today, this is the discussion we had previously
GitHub link: https://github.com/apache/texera/discussions/5641#discussioncomment-17477061
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform [texera]
Posted by "sshiv012 (via GitHub)" <gi...@apache.org>.
GitHub user sshiv012 added a comment to the discussion: Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform
Terraform **supports AWS**, Azure, GCP and other leading cloud providers. As Ian pointed out for on-prem is highly specific, but terraform should support application/platform stuff like helm release, Postgres, MinIO, LakeFS, and the cluster layer for VMs. It would also be self documenting in nature.
The only other complexity I see with it and isn't discussed is for terraform, we would need to manage, store and track the state of deployments, for a team this would be in some datastore that's collaborative like S3, for local deployments or smaller scale or single admin it can be stored locally.
GitHub link: https://github.com/apache/texera/discussions/5641#discussioncomment-17360959
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica closed the discussion with a comment: Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform
@zuozhiw Thanks for the confirmation. Now we have a clear decision.
GitHub link: https://github.com/apache/texera/discussions/5641#discussioncomment-17416749
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform [texera]
Posted by "aicam (via GitHub)" <gi...@apache.org>.
GitHub user aicam added a comment to the discussion: Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform
Terraform supports all Cloud services, however, currently we focus on AWS and onprem, to keep deployment artifacts simple, we keep Helm charts and add eksctl tool which is specific to AWS. Texera will be deployed by technical people so we prefer to keep our deployment templates and code lean and allow different managers use it as they need. At this point, Terraform is somehow overkill for our purpose.
GitHub link: https://github.com/apache/texera/discussions/5641#discussioncomment-17386436
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica added a comment to the discussion: Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform
@aicam : To make it more clear, please answer the following questions:
- Does Teraform supports both AWS and on-prem?
- If so, what is "overkill"?
GitHub link: https://github.com/apache/texera/discussions/5641#discussioncomment-17387434
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica added a comment to the discussion: Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform
Thanks for starting the discussion. We want to support both cloud-based and on-prem deployments. Please confirm if eksctl supports both.
@zuozhiw @shengquan-ni @Yicong-Huang @sshiv012 @parshimers Any input from you?
GitHub link: https://github.com/apache/texera/discussions/5641#discussioncomment-17298591
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform [texera]
Posted by "aicam (via GitHub)" <gi...@apache.org>.
GitHub user aicam edited a discussion: Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform
## Problem
Today `bin/k8s` on `main` ships a single Helm chart that targets a **local / on-premise Kubernetes cluster only**. Our AWS (EKS) deployment runs the *same* Helm chart, but everything underneath it — the EKS cluster, S3 bucket, IAM/IRSA roles, Elastic IP, DNS, node pools, cert-manager — is created **by hand** and lives nowhere in the repo. None of that substrate is captured as code.
The result: standing up Texera on a fresh AWS account is an undocumented, manual prerequisite checklist that drifts and is hard to reproduce. We'd like to **unify the local and AWS deployment stories** so there's one documented, reproducible path from "empty cluster (or empty AWS account)" to "running Texera," while keeping the Helm chart itself cloud-agnostic.
## Scope of "substrate"
The Helm chart already covers the *application*. The open question is only how we provision/document the *substrate* it runs on:
- EKS cluster + managed node pools
- IAM roles / IRSA for service accounts
- S3 bucket (LakeFS backend), Elastic IP, DNS, cert-manager / Let's Encrypt
## Options considered
### 1. Helm only
Deploy the chart into a cluster that already exists. No AWS substrate creation.
**Pros**
- Simplest — one `helm install`
- Identical for local and AWS — fully cloud-agnostic
- Nothing extra to learn or maintain in the repo
- Keeps the Apache PR scoped to just the chart
**Cons**
- Doesn't create the cluster, S3 bucket, IAM, EIP, node pools, or cert-manager — all manual on a new AWS account
- New-AWS-account UX = a long prerequisite checklist done by hand
- Substrate stays undocumented-as-code → drift, hard to reproduce
### 2. eksctl
AWS CLI that provisions an EKS cluster + node pools + IAM/IRSA + add-ons from a small YAML, via CloudFormation. Helm still deploys on top.
**Pros**
- Purpose-built for EKS; minimal config for a working cluster
- Handles IRSA, node pools, and add-ons natively
- "New AWS account → cluster" in ~one command
- Config file is committable and reproducible
**Cons**
- AWS-only — no use for local or other clouds
- Doesn't manage non-cluster resources well (S3 bucket, EIP, DNS) — you still script those
- CloudFormation-based: slower, less granular state control
- Adds an AWS-specific tool to a repo that's otherwise portable
### 3. Terraform
Declarative IaC that can provision everything: EKS cluster, S3, IAM, EIP, DNS, node pools — and across clouds. Helm deploys on top (or via the Helm provider).
**Pros**
- Most reproducible & reviewable (full state, plan/apply, diffs)
- One tool for all substrate, not just the cluster
- Cloud-agnostic — same tool for AWS / GCP / local
- Can even invoke Helm itself, so one `apply` does substrate + deploy
**Cons**
- Heaviest to set up and maintain (modules, remote state, locking)
- Steeper learning curve; more moving parts
- State management is its own operational burden
- Overkill if you only ever target one EKS cluster
## Our current position: eksctl
We're currently leaning toward **eksctl**, because:
- **GCP and Azure are low priority for now** — the cloud-agnostic advantage of Terraform doesn't pay off yet, so we'd rather not take on its maintenance cost speculatively.
- **We prefer easier maintenance over easier deploying.** eksctl keeps the AWS-specific config small and declarative while letting the Helm chart stay fully portable, rather than introducing Terraform modules + remote state + locking that we'd have to operate.
- eksctl handles the genuinely fiddly EKS-specific parts (IRSA, node pools, add-ons) natively, and the cluster config is committable.
Regardless of the choice, a small amount of non-cluster substrate (S3 bucket, EIP, DNS) would still be scripted alongside.
## We'd love your input
Which approach should we standardize on for upstreaming AWS deployment into `bin/k8s`? Please vote with a reaction on this post and share your reasoning in the comments — especially if you deploy Texera (or similar) on a cloud we haven't prioritized:
- 👍 — **Helm only**
- 🚀 — **eksctl** (our current lean)
- ❤️ — **Terraform**
GitHub link: https://github.com/apache/texera/discussions/5641
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform [texera]
Posted by "parshimers (via GitHub)" <gi...@apache.org>.
GitHub user parshimers added a comment to the discussion: Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform
eksctl is just for Amazon's distribution of k8s. Which is fine, as stated GCP/Azure are way less popular, and on prem is always unique. The important thing is that there is no hard binding to AWS and that's already the case.
I think having a well documented set of steps for setting up everything on AWS is a very good idea. IAM in particular is a nightmare without knowing exactly what each service actually needs. People running on prem will have their own particular setups that will need interpretation anyway. For example they probably have some object store, but each one will have it's own configurations for buckets and so on. It's not possible for us as developers to say how that's done on each system a priori. There should be guidelines (e.g. what addresses and ports the various services need, what object stores are known to work, etc.) but not specifics. Maybe an example with some particular set of services would be a good intermediate step, because then the administrator would be able to draw analogies to their environment.
I have gone through Milvus's setup on AWS and that was about as straightforward as it could be. It really would have taken me a very long time to figure out things on my own without that walkthrough.
GitHub link: https://github.com/apache/texera/discussions/5641#discussioncomment-17310476
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica added a comment to the discussion: Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform
@madhusudancs, it will be great to have more advice from you. Thank you.
GitHub link: https://github.com/apache/texera/discussions/5641#discussioncomment-17480317
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica added a comment to the discussion: Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform
@aicam Please confirm why we don't want to use Terraform.
GitHub link: https://github.com/apache/texera/discussions/5641#discussioncomment-17347398
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform [texera]
Posted by "aicam (via GitHub)" <gi...@apache.org>.
GitHub user aicam added a comment to the discussion: Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform
To update our discussion, we decided to go with Terraform after more discussions with industry experts.
GitHub link: https://github.com/apache/texera/discussions/5641#discussioncomment-17502874
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform [texera]
Posted by "aicam (via GitHub)" <gi...@apache.org>.
GitHub user aicam added a comment to the discussion: Approach: unifying AWS (EKS) and on-premise Kubernetes deployment under bin/k8s — Helm-only vs eksctl vs Terraform
The main reason we do not use Terraform is that we currently focus on AWS and onprem only.
GitHub link: https://github.com/apache/texera/discussions/5641#discussioncomment-17355704
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org