You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Rietty (via GitHub)" <gi...@apache.org> on 2026/03/24 23:57:02 UTC
[D] How to deploy to AWS EKS and start up Airflow properly in Production? [airflow]
GitHub user Rietty created a discussion: How to deploy to AWS EKS and start up Airflow properly in Production?
Hello folks! I'm trying to deploy Airflow to our AWS EKS Environments (Dev/UAT/Prod) and am having trouble setting it up properly, specifically the Airflow instance I am trying to install it from the Python Package and get it up and running.
Currently we have some EKS stuff that we deploy so I have followed the same template and got it to at least deploy. However my issue is that the system says not to run `standalone` in production so I'm not sure what to change.
Currently I have a `pyproject.toml` with a simple project, dependencies (`apache-airflow==3.1.7`) and a `[[tool.uv.index]]` for our internal PyPI mirror.
My Dockerfile simply copies these things over, installs `uv` resolves dependencies via `sync` and then just uses `CMD['uv', 'run', 'airflow', 'standalone']` however this complains about the above not to use `standalone` in production.
So my question is how do I properly deploy app properly so:
1. Kubernetes will properly scale workers and such when workloads become high.
2. Handle things like `postgresdb` that are required by `airflow` to function? I assume there's some way to have a built-in one that can be the default. (This is how our old/existing one works.)
3. Is there some documentation about this stuff?
Please don't suggest stuff like Amazon's Managed Airflow (It's very expensive!) or to use `data-on-eks` which is very strongly opinionated and will not work for us as we are not looking to create a new AWS EKS cluster.
GitHub link: https://github.com/apache/airflow/discussions/64196
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] How to deploy to AWS EKS and start up Airflow properly in Production? [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: How to deploy to AWS EKS and start up Airflow properly in Production?
You just need to manage your components in some way - systemd or the like, or even have containers or vms per container. Or you can use Docker Compose or Helm Chart / Kubernetes - but it's generally up to you to configure and deploy Airflow. All possible options are nicely described in https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html - you have all the known installation methods, describing their pros/cons - why you might want to use one, and what are your responsibilities as Deployment Manager.
Maybe you will not get detailed instructions step-by-step to do **precisely what you want** (deployment definition and building is Deployment Manager's responsibility) - you will find enough leads examples, quick-starts, and the Helm Chart of Airflow is used by people in production quite heavily - and it's highly customizable (too highly one might say).
GitHub link: https://github.com/apache/airflow/discussions/64196#discussioncomment-16301854
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] How to deploy to AWS EKS and start up Airflow properly in Production? [airflow]
Posted by "akhoyannh-a11y (via GitHub)" <gi...@apache.org>.
GitHub user akhoyannh-a11y added a comment to the discussion: How to deploy to AWS EKS and start up Airflow properly in Production?
- ****
GitHub link: https://github.com/apache/airflow/discussions/64196#discussioncomment-16302883
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] How to deploy to AWS EKS and start up Airflow properly in Production? [airflow]
Posted by "akhoyannh-a11y (via GitHub)" <gi...@apache.org>.
GitHub user akhoyannh-a11y added a comment to the discussion: How to deploy to AWS EKS and start up Airflow properly in Production?
[]()
GitHub link: https://github.com/apache/airflow/discussions/64196#discussioncomment-16302893
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] How to deploy to AWS EKS and start up Airflow properly in Production? [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: How to deploy to AWS EKS and start up Airflow properly in Production?
Hard to answer it yes/no. It depends mostly I guess on the many ways you choose to deploy it on EKS - each k8s installation has many options, so authoritative answer might not be possible.
But You should try. If you look at issues here-⁰ lots of people use it and you always can customize the chart by having your own and extend it Extending the Chart-with your own things https://share.google/7WCA1oQRqa4rOc17g
GitHub link: https://github.com/apache/airflow/discussions/64196#discussioncomment-16308559
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] How to deploy to AWS EKS and start up Airflow properly in Production? [airflow]
Posted by "Rietty (via GitHub)" <gi...@apache.org>.
GitHub user Rietty added a comment to the discussion: How to deploy to AWS EKS and start up Airflow properly in Production?
Is the helm chart customizable enough to use it with AWS EKS if I configure the variables for it properly?
GitHub link: https://github.com/apache/airflow/discussions/64196#discussioncomment-16302495
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org