You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "pykenny (via GitHub)" <gi...@apache.org> on 2025/06/24 07:01:35 UTC

[D] Kuberentes integration settings in Breeze environment [airflow]

GitHub user pykenny created a discussion: Kuberentes integration settings in Breeze environment

I want to reproduce several issues I encountered when using operator classes under "cncf-kuberentes" provider. To better provide example as proof, I want to make a minimum example with Breeze development environment, but now I'm totally stuck and cannot figure out how to integrate a local Kubernetes cluster to a running Breeze environment (having very limited knowledge about Docker and Kubernetes, while both K8s environment setup and settings required to run these K8s tasks are mostly managed by other teams). Any idea or suggested guide to help with this scenario?

Also, there's no documentation on `breeze-k8s` subcommand, from help doc it seems to be utility that runs Airflow in local Kubernetes cluster, and I'm not certain if it does any help to this scenario.

I also found this issue thread but seems this the has been suspended for more than a year after it's proposed:
 - https://github.com/apache/airflow/issues/40005

GitHub link: https://github.com/apache/airflow/discussions/52131

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk edited a comment on the discussion: Kuberentes integration settings/commands for Breeze environment?

Yes. Breeze `k8s` suite of commands is the wy to go. It has a bit of a difficult in iterating - i.e. every time you make changes to provider / airflow code you need to rebuild and redeploy the image. We have an open issue to improve the iteration speed - https://github.com/apache/airflow/issues/44508 - but we did not have time/energy to work on it and other things took priority.

You have detailed step-by-step instructions how to do it: https://github.com/apache/airflow/blob/main/contributing-docs/testing/k8s_tests.rst#typical-testing-pattern-for-kubernetes-tests  - including the fact that k8s suite of commands work in a "wizard" like way guiding you what are the next steps.

In short:

* it sets up a local venv which has everything configured and all tools (kind, kubectl, helm) automatically installed in this separate env
* it creates a kind cluster of your choice
* It builds a base image that you can use to deploy airflow with (with embedded example dags) 
* it allows you to rebuild the image with your latest changes
* it allows you to deploy the image to the kind cluster
* it uses helm to start complete "airflow" installation in the cluster that you can connect to and interact with

So - the limitation is that every time you make a change you need to:
* rebuild (inclrementally - so it takes seconds) your image
* deploy it to kind cluster
* restart the pods

This is slow - but works. With the opened issue we imagine a workflow where tools like teleport are used to send the modified files via ssh to the running pods and the pods have a way to pick the files up and restart the interpreters  - that would be much faster, workflow but also more complex and brittle. 

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13560418

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny added a comment to the discussion: Kuberentes integration settings/commands for Breeze environment?

Increase RAM to maximum (8GB on my machine) and now only API server startup failed. Looks like a common issue for 3.0 deployment:
 - https://github.com/apache/airflow/discussions/50170

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13571486

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings in Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a discussion: Kuberentes integration settings in Breeze environment?

I want to reproduce several issues I encountered when using operator classes under "cncf-kuberentes" provider. To better provide example as proof, I want to make a minimum example with Breeze development environment, but now I'm totally stuck and cannot figure out how to integrate a local Kubernetes cluster to a running Breeze environment (having very limited knowledge about Docker and Kubernetes, while both K8s environment setup and settings required to run these K8s tasks are mostly managed by other teams). Any idea or suggested guide to help with this scenario?

Also, there's no documentation on `breeze-k8s` subcommand, from help doc it seems to be utility that runs Airflow in local Kubernetes cluster, and I'm not certain if it does any help to this scenario.

I also found this issue thread but seems this the has been suspended for more than a year after it's proposed:
 - https://github.com/apache/airflow/issues/40005

GitHub link: https://github.com/apache/airflow/discussions/52131

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a comment on the discussion: Kuberentes integration settings/commands for Breeze environment?

Increased RAM to maximum (8GB on the machine) and now only API server failed on startup. Looks like a common issue for 3.0 deployment:
 - https://github.com/apache/airflow/discussions/50170

<img width="1340" alt="image" src="https://github.com/user-attachments/assets/ca88ed2c-9dab-4c6a-a971-8769c65d876b" />


GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13571486

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a discussion: Kuberentes integration settings/commands for Breeze environment?

I want to reproduce several issues I encountered when using operator classes under "[cncf-kuberentes](https://github.com/apache/airflow/discussions/52131#discussion-8487393)" provider. To better provide proof to these problems, I want to make a minimum example with Breeze development environment, but now I'm totally stuck and cannot figure out how to integrate a local Kubernetes cluster to a running Breeze environment (having very limited knowledge about Docker and Kubernetes, while both K8s environment setup and settings required to run these K8s tasks are mostly managed by other teams). Any idea or suggested guide to help with this scenario?

Also, there's no documentation on `breeze-k8s` subcommand, from help doc it seems to be utility that runs Airflow in local Kubernetes cluster (such as using Kuberentes Executor?), and I'm not certain if it does any help to this scenario.

I also found this issue thread but seems this has been suspended for more than a year after it's proposed:
- https://github.com/apache/airflow/issues/40005

Here's an unsuccessful attempt to create a K8s cluster locally and let Airflow to gain access to the cluster (with part of the instructions/settings suggested by ChatGPT, if you find them erroneous in some way):

0. We've already have Docker (along with local K8s utilities) installed when installing Breeze.
1. Install [kind](https://kind.sigs.k8s.io/docs/user/quick-start/) and run `kind create cluster --name airflow-testing` to create a cluster named `kind-airflow-testing`. Check cluster information with `kubectl cluster-info --context kind-airflow-testing` command.
2. Now we have the cluster's configuration stored in `~/.kube/config`. Copy the configuration file to the shared mount location under repository root:
```bash
cp ~/.kube/config ./files/kubeconfig
```
3. Add environment variable to `files/airflow-breeze-config/environment_variables.env`:
```bash
KUBECONFIG=/files/kubeconfig
```
By default, K8s utilities will use what `KUBECONFIG` points to to initialize configuration.
4. Start Breeze environment and run Airflow:
```bash
breeze --python 3.9 --backend mysql --mysql-version 8.0 start-airflow
```
5. Build a DAG that uses any operator under "cncf-kuberentes" provider (well, if you, patiently, read through the doc to the [end chapter](https://github.com/apache/airflow/blob/207c4b09cf7c34b740472aca79dadc19c6f69db4/dev/breeze/doc/10_advanced_breeze_topics.rst)), enable it, run it, and welcome all kinds of `NewConnectionError` appearing on the console/Web-UI!

GitHub link: https://github.com/apache/airflow/discussions/52131

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny added a comment to the discussion: Kuberentes integration settings/commands for Breeze environment?

Nope:

```
ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'c608a390-801c-4ff6-938d-1f33f6afc6b6', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'dfc53d28-fc9b-41f3-8478-7a0282759537', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'e10ca922-38fe-4420-891d-6ca2f1253406', 'Date': 'Tue, 01 Jul 2025 05:51:28 GMT', 'Content-Length': '321'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:airflow:airflow-scheduler\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"airflow\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
```

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13625046

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a comment on the discussion: Kuberentes integration settings/commands for Breeze environment?

Now the webserver's dead again and keep failing to restart itself:

<img width="1341" alt="image" src="https://github.com/user-attachments/assets/69b3ec10-6ed8-478a-9474-2a766696b6eb" />

Seems that in a environment with 6GB RAM provided (plus 3GB swap) without running any other thing in Docker, the cluster could suffer from:
 - Long startup time of scheduler
 - Consistent crash of API server (web server)

<img width="1348" alt="image" src="https://github.com/user-attachments/assets/b61eb733-a52b-4b1c-a38c-2ab4921cfa71" />



GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13569444

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings in Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a discussion: Kuberentes integration settings in Breeze environment?

I want to reproduce several issues I encountered when using operator classes under "[cncf-kuberentes](https://github.com/apache/airflow/discussions/52131#discussion-8487393)" provider. To better provide example as proof, I want to make a minimum example with Breeze development environment, but now I'm totally stuck and cannot figure out how to integrate a local Kubernetes cluster to a running Breeze environment (having very limited knowledge about Docker and Kubernetes, while both K8s environment setup and settings required to run these K8s tasks are mostly managed by other teams). Any idea or suggested guide to help with this scenario?

Also, there's no documentation on `breeze-k8s` subcommand, from help doc it seems to be utility that runs Airflow in local Kubernetes cluster, and I'm not certain if it does any help to this scenario.

I also found this issue thread but seems this the has been suspended for more than a year after it's proposed:
 - https://github.com/apache/airflow/issues/40005

GitHub link: https://github.com/apache/airflow/discussions/52131

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a discussion: Kuberentes integration settings/commands for Breeze environment?

I also found this issue thread but seems this the has been suspended for more than a year after it's proposed:
- https://github.com/apache/airflow/issues/40005

Here's an unsuccessful attempt to create a K8s cluster locally and let Airflow to gain access to the cluster (with part of the instructions/settings suggested by ChatGPT):

By default, K8s utilities will use what `KUBECONFIG` points to to initialize configuration.
4. Starts Breeze environment and run Airflow:
```bash
breeze --python 3.9 --backend mysql --mysql-version 8.0 start-airflow
```
5. Build a DAG that uses any operator under "cncf-kuberentes" provider, enable it, run it, and welcome all kinds of `NewConnectionError`!

GitHub link: https://github.com/apache/airflow/discussions/52131

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a comment on the discussion: Kuberentes integration settings/commands for Breeze environment?

Follow all the steps until step `breeze k8s deploy-airflow`, logs:

```
Default value of executor parameter LocalExecutor used.

Deploying Airflow for cluster airflow-python-3.9-v1.30.10
Deploying kind-airflow-python-3.9-v1.30.10 with airflow Helm Chart.
Copied chart sources to /private/var/folders/qg/0mrrt2zs53vbps1qfw7vxqq40000gq/T/chart_5b_pwgnf/chart
Deploying Airflow from /private/var/folders/qg/0mrrt2zs53vbps1qfw7vxqq40000gq/T/chart_5b_pwgnf/chart
I0625 11:36:10.072819   80585 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0625 11:36:10.072840   80585 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0625 11:36:10.072843   80585 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0625 11:36:10.072846   80585 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
NAME: airflow
LAST DEPLOYED: Wed Jun 25 11:36:10 2025
NAMESPACE: airflow
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing Apache Airflow 3.1.0!

Your release is named airflow.
You can now access your dashboard(s) by executing the following command(s) and visiting the corresponding port at localhost in your browser:

Airflow Webserver:     kubectl port-forward svc/airflow-webserver 8080:8080 --namespace airflow
Default Webserver (Airflow UI) Login credentials:
    username: admin
    password: admin
Default Postgres connection credentials:
    username: postgres
    password: postgres
    port: 5432

You can get Fernet Key value by running the following:

    echo Fernet Key: $(kubectl get secret --namespace airflow airflow-fernet-key -o jsonpath="{.data.fernet-key}" | base64 --decode)

###########################################################
#  WARNING: You should set a static webserver secret key  #
###########################################################

You are using a dynamically generated webserver secret key, which can lead to
unnecessary restarts of your Airflow components.

Information on how to set a static webserver secret key can be found here:
https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#webserver-secret-key
Deployed kind-airflow-python-3.9-v1.30.10 with airflow Helm Chart.

Airflow for Python 3.9 and K8S version v1.30.10 has been successfully deployed.

The KinD cluster name: airflow-python-3.9-v1.30.10
The kubectl cluster name: kind-airflow-python-3.9-v1.30.10.


Kubeconfig file in: /AIRFLOW/REPO/ROOT/.build/.k8s-clusters/airflow-python-3.9-v1.30.10/.kubeconfig


KinD Cluster API server URL: http://localhost:24308
Connecting to localhost:13832. Num try: 1
Error when connecting to localhost:13832 : ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
Sleeping for 5 seconds.
(...)
Connecting to localhost:13832. Num try: 25
Error when connecting to localhost:13832 : ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
More than 120 passed. Exiting.

Airflow API server is not available at port 13832. Run `breeze k8s deploy-airflow --python 3.9 --kubernetes-version v1.30.10` to (re)deploy airflow


NEXT STEP: You might now run tests or interact with airflow via shell (kubectl, pytest etc.) or k9s commands:


breeze k8s tests

breeze k8s shell

breeze k8s k9s
```

Seems like the API server's not working. Know something's broken for sure because when looking into the cluster with `k9s` sub-command I saw part of the components are not in normal state:

<img width="1345" alt="image" src="https://github.com/user-attachments/assets/ff623712-b439-4c1a-b1fe-da7c166dc906" />

Then try redeploy with command `breeze k8s deploy-airflow --python 3.9 --kubernetes-version v1.30.10` and it failed without giving useful information:

```
Deploying Airflow for cluster airflow-python-3.9-v1.30.10
Deploying kind-airflow-python-3.9-v1.30.10 with airflow Helm Chart.
Copied chart sources to /private/var/folders/qg/0mrrt2zs53vbps1qfw7vxqq40000gq/T/chart_2atzjmkh/chart
Deploying Airflow from /private/var/folders/qg/0mrrt2zs53vbps1qfw7vxqq40000gq/T/chart_2atzjmkh/chart
I0625 12:09:33.307494   82192 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0625 12:09:33.307523   82192 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0625 12:09:33.307526   82192 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0625 12:09:33.307529   82192 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
Error: INSTALLATION FAILED: cannot re-use a name that is still in use
```

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13568891

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk added a comment to the discussion: Kuberentes integration settings/commands for Breeze environment?

Yes k8s + airflow require a lot of RAM. 8GB is a minimum that I start with . That's the nature of the beast

I think it should be described in the instructionsi linked before - if not , it would be great to add it when. You find out.

Dag's are added in a dockerfile. In any of the commands you can add --verbose flag and see exactly what commands are executed. In case of the k8s workflow there is the step of building k8s imaga and it embeds DGs in the image - look at the content of the dockerfile to find out - I thi k you can find sources of the Dockerfile that extends the base PROD image somewhere in /dev/breeze - look at the kubernetes commands - and follow the steps there - it's just regular Python code, easy to follow.

Not at my PC at this moment but this should give you starting point. Generally breeze is just a Python wrapper that produces regular docker commands and the easiest way to understand what it does is to --verbose and to look at the code.

And if you find thT some descriptions can be added and are missing now - I would really appreciate If you add it where you look for them now - you are probably the closest that you can be to know best which information you miss and where people like you would look for it.

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13571121

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a discussion: Kuberentes integration settings/commands for Breeze environment?

I also found this issue thread but seems this the has been suspended for more than a year after it's proposed:
- https://github.com/apache/airflow/issues/40005

GitHub link: https://github.com/apache/airflow/discussions/52131

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a discussion: Kuberentes integration settings/commands for Breeze environment?

I also found this issue thread but seems this has been suspended for more than a year after it's proposed:
- https://github.com/apache/airflow/issues/40005

0. We've already have Docker (along with local K8s utilities) installed when installing Breeze.
1. Install [kind](https://kind.sigs.k8s.io/docs/user/quick-start/) and run `kind create cluster --name airflow-testing` to create a cluster named `kind-airflow-testing`. Check cluster information with `kubectl cluster-info --context kind-airflow-testing` command.
2. Now we have the cluster's configuration stored in `~/.kube/config`. Copy the configuration file to the shared mount location under repository root:
```bash
cp ~/.kube/config ./files/kubeconfig
```
3. Add environment variable to `files/airflow-breeze-config/environment_variables.env`:
```bash
KUBECONFIG=/files/kubeconfig
```
By default, K8s utilities will use what `KUBECONFIG` points to to initialize configuration.
4. Start Breeze environment and run Airflow:
```bash
breeze --python 3.9 --backend mysql --mysql-version 8.0 start-airflow
```
5. Build a DAG that uses any operator under "cncf-kuberentes" provider (well, one supposed to know where to place DAG files, if one, patiently, read through the whole Breeze doc, right to the [end chapter](https://github.com/apache/airflow/blob/207c4b09cf7c34b740472aca79dadc19c6f69db4/dev/breeze/doc/10_advanced_breeze_topics.rst)), enable it, run it, and welcome all kinds of `NewConnectionError` appearing on the console/Web-UI!

GitHub link: https://github.com/apache/airflow/discussions/52131

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a comment on the discussion: Kuberentes integration settings/commands for Breeze environment?

Update: Increased Docker's RAM and the cluster works now:
<img width="1346" alt="image" src="https://github.com/user-attachments/assets/daf9c0aa-9ae4-42de-bf7c-bfc4c91fd7a8" />

So where should I put the DAGs? Seems that it doesn't load DAG definitions placed under `files/dags` folder.

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13569248

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings in Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a discussion: Kuberentes integration settings in Breeze environment?

I want to reproduce several issues I encountered when using operator classes under "[cncf-kuberentes](https://github.com/apache/airflow/discussions/52131#discussion-8487393)" provider. To better provide proof to these problems, I want to make a minimum example with Breeze development environment, but now I'm totally stuck and cannot figure out how to integrate a local Kubernetes cluster to a running Breeze environment (having very limited knowledge about Docker and Kubernetes, while both K8s environment setup and settings required to run these K8s tasks are mostly managed by other teams). Any idea or suggested guide to help with this scenario?

Also, there's no documentation on `breeze-k8s` subcommand, from help doc it seems to be utility that runs Airflow in local Kubernetes cluster, and I'm not certain if it does any help to this scenario.

I also found this issue thread but seems this the has been suspended for more than a year after it's proposed:
 - https://github.com/apache/airflow/issues/40005

GitHub link: https://github.com/apache/airflow/discussions/52131

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk edited a comment on the discussion: Kuberentes integration settings/commands for Breeze environment?

Wild guess is that airflow is started (and service accounts are configured) to be runnung in "airflow" namespace -  not "default" - look it up, k9s can give you all the possibilities of inspecting running k8s. 

Generally you are in the realm of having everything working - you just need to understand k8s environment, permissions, service accounts etc.  

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13624468

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk edited a comment on the discussion: Kuberentes integration settings/commands for Breeze environment?

Yes k8s + airflow require a lot of RAM. 8GB is a minimum that I start with . That's the nature of the beast

RE: Dag's - I think it should be described in the instructionsi linked before - if not , it would be great to add it when. You find out.

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13571121

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk added a comment to the discussion: Kuberentes integration settings/commands for Breeze environment?

> Increased RAM to maximum (8GB on the machine) and now only API server failed on startup. Looks like a common issue for 3.0 deployment:

Look at k8s error logs and why it does not start. Generally when pod fails to start you can find out in k8s logs why. Without it, it's pure guess. Maybe there is some specific WSL2 setting you have problem with ? Definitely it works out-of-the-box on MacOS and Linux, but if you are on WSL2 and if there are some specific settings, environment etc. it **might** have some errors - pretty much only you can find out what's wrong  by looking at the logs.

Usually k9s will show you them when you go to logs setting - sometimes - when you have crash loopback it might require you to try a few times, but I think using standard kubectl and k9s you should be able to see logs for failed pods as well. 

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13572708

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings in Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a discussion: Kuberentes integration settings in Breeze environment?

I want to reproduce several issues I encountered when using operator classes under "[cncf-kuberentes](https://github.com/apache/airflow/discussions/52131#discussion-8487393)" provider. To better provide proof to these problems, I want to make a minimum example with Breeze development environment, but now I'm totally stuck and cannot figure out how to integrate a local Kubernetes cluster to a running Breeze environment (having very limited knowledge about Docker and Kubernetes, while both K8s environment setup and settings required to run these K8s tasks are mostly managed by other teams). Any idea or suggested guide to help with this scenario?

Also, there's no documentation on `breeze-k8s` subcommand, from help doc it seems to be utility that runs Airflow in local Kubernetes cluster (such as using Kuberentes Executor?), and I'm not certain if it does any help to this scenario.

I also found this issue thread but seems this the has been suspended for more than a year after it's proposed:
 - https://github.com/apache/airflow/issues/40005

GitHub link: https://github.com/apache/airflow/discussions/52131

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a discussion: Kuberentes integration settings/commands for Breeze environment?

I also found this issue thread but seems this the has been suspended for more than a year after it's proposed:
- https://github.com/apache/airflow/issues/40005

GitHub link: https://github.com/apache/airflow/discussions/52131

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a discussion: Kuberentes integration settings/commands for Breeze environment?

I also found this issue thread but seems this has been suspended for more than a year after it's proposed:
- https://github.com/apache/airflow/issues/40005

0. We've already have Docker (along with local K8s utilities) installed when installing Breeze.
1. Install [kind](https://kind.sigs.k8s.io/docs/user/quick-start/) and run `kind create cluster --name airflow-testing` to create a cluster named `kind-airflow-testing`. Check cluster information with `kubectl cluster-info --context kind-airflow-testing` command.
2. Now we have the cluster's configuration stored in `~/.kube/config`. Copy the configuration file to the shared mount location under repository root:
```bash
cp ~/.kube/config ./files/kubeconfig
```
3. Add environment variable to `files/airflow-breeze-config/environment_variables.env`:
```bash
KUBECONFIG=/files/kubeconfig
```
By default, K8s utilities will use what `KUBECONFIG` points to to initialize configuration.
4. Start Breeze environment and run Airflow:
```bash
breeze --python 3.9 --backend mysql --mysql-version 8.0 start-airflow
```
5. Build a DAG that uses any operator under "cncf-kuberentes" provider (well, one supposed to know where to place DAG files, if one, patiently, read through the Breeze doc, right to the [end chapter](https://github.com/apache/airflow/blob/207c4b09cf7c34b740472aca79dadc19c6f69db4/dev/breeze/doc/10_advanced_breeze_topics.rst)), enable it, run it, and welcome all kinds of `NewConnectionError` appearing on the console/Web-UI!

GitHub link: https://github.com/apache/airflow/discussions/52131

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a comment on the discussion: Kuberentes integration settings/commands for Breeze environment?

Recent atttempt:

 - Fix crash issue by turning off some of the resource-consuming programs on the host for more RAM.
 - Modify [`kubernetes_commands.py`](https://github.com/apache/airflow/blob/12cd06d41366fe40ebd8712945d5201c5b0f19b2/dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py#L605) to copy things under `dags` folder into the image (`files` folder is not mounted onto the pods under this mode) during build stage. Once completing `build-k8s-image` and `upload-k8s-image` commands the cluster will automatically run the migration and restart with latest image.
   ```dockerfile
   COPY --chown=airflow:0 dags/ /opt/airflow/dags/
   ```

Minimum test example:

```python
from datetime import timedelta

from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.job import KubernetesJobOperator
from pendulum import datetime

with DAG(
    dag_id="k8s_job_always_fail",
    schedule="0 * * * *",
    start_date=datetime(2022, 12, 8),
    catchup=False,
    tags=["example", "k8s", "retry"],
) as dag:

    failing_task = KubernetesJobOperator(
        task_id="always_fail_job",
        namespace="default",
        image="alpine:latest",
        cmds=["/bin/sh", "-c", "echo failing... && exit 1"],
        retries=3,
        retry_delay=timedelta(seconds=30),
        wait_until_job_complete=True,
        deferrable=True,
        backoff_limit=0,
        do_xcom_push=False,
        get_logs=False,
    )
```

Result: Failed due to lack of permission (Airflow 3 seems to remove log download utils from UI):
<img width="1024" alt="image" src="https://github.com/user-attachments/assets/7fbfce3b-4c04-4471-9541-238de8f5dfe1" />

```
ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '31359fbd-cdfe-40a8-88f0-93df023e7630', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'dfc53d28-fc9b-41f3-8478-7a0282759537', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'e10ca922-38fe-4420-891d-6ca2f1253406', 'Date': 'Tue, 01 Jul 2025 03:26:03 GMT', 'Content-Length': '321'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:airflow:airflow-scheduler\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"default\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
```

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13624390

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a comment on the discussion: Kuberentes integration settings/commands for Breeze environment?

Recent atttempt:

 - Fix crash issue by turning off some of the resource-consuming programs on the host for more RAM.
 - Modify [`kubernetes_commands.py`](https://github.com/apache/airflow/blob/12cd06d41366fe40ebd8712945d5201c5b0f19b2/dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py#L605) to copy things under `dags` folder into the image (`files` folder is not mounted onto the pods under this mode) during build stage. Once completing `build-k8s-image` and `upload-k8s-image` commands the cluster will automatically run the migration and restart with latest image.
   ```dockerfile
   COPY --chown=airflow:0 dags/ /opt/airflow/dags/
   ```

Minimum test example:

```python
from datetime import timedelta

from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.job import KubernetesJobOperator
from pendulum import datetime

with DAG(
    dag_id="k8s_job_always_fail",
    schedule="0 * * * *",
    start_date=datetime(2022, 12, 8),
    catchup=False,
    tags=["example", "k8s", "retry"],
) as dag:

    failing_task = KubernetesJobOperator(
        task_id="always_fail_job",
        namespace="default",
        image="alpine:latest",
        cmds=["/bin/sh", "-c", "echo failing... && exit 1"],
        retries=3,
        retry_delay=timedelta(seconds=30),
        wait_until_job_complete=True,
        deferrable=True,
        backoff_limit=0,
        do_xcom_push=False,
        get_logs=False,
    )
```

Result: Failed due to lack of permission (Airflow 3 seems to remove log download utils from UI, so using screenshot instead):
<img width="1024" alt="image" src="https://github.com/user-attachments/assets/7fbfce3b-4c04-4471-9541-238de8f5dfe1" />

```
ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '31359fbd-cdfe-40a8-88f0-93df023e7630', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'dfc53d28-fc9b-41f3-8478-7a0282759537', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'e10ca922-38fe-4420-891d-6ca2f1253406', 'Date': 'Tue, 01 Jul 2025 03:26:03 GMT', 'Content-Length': '321'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:airflow:airflow-scheduler\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"default\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
```

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13624390

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a comment on the discussion: Kuberentes integration settings/commands for Breeze environment?

Nope:

```
ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'c608a390-801c-4ff6-938d-1f33f6afc6b6', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'dfc53d28-fc9b-41f3-8478-7a0282759537', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'e10ca922-38fe-4420-891d-6ca2f1253406', 'Date': 'Tue, 01 Jul 2025 05:51:28 GMT', 'Content-Length': '321'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:airflow:airflow-scheduler\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"airflow\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
```

Looks like the runner doesn't have enough permission.

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13625046

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk added a comment to the discussion: Kuberentes integration settings/commands for Breeze environment?

Wild gues is that airflow is started (and service accounts are configured) to be runnung in "airflow" namespace -  not default - look it up, k9s can give you all the possibilities of inspecting running k8s. 

Generally you are in the realm of having everything working - you just need to understand k8s environment, permissions, service accounts etc.  

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13624468

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a comment on the discussion: Kuberentes integration settings/commands for Breeze environment?

Recent atttempt:

 - Fix crash issue by turning off some of the resource-consuming programs on host system for more RAM.
 - Modify [`kubernetes_commands.py`](https://github.com/apache/airflow/blob/12cd06d41366fe40ebd8712945d5201c5b0f19b2/dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py#L605) to copy things under `dags` folder into the image (`files` folder is not mounted onto the pods under this mode) during build stage. Once completing `build-k8s-image` and `upload-k8s-image` commands the cluster will automatically run the migration and restart with latest image.
   ```dockerfile
   COPY --chown=airflow:0 dags/ /opt/airflow/dags/
   ```

Minimum test example:

```python
from datetime import timedelta

from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.job import KubernetesJobOperator
from pendulum import datetime

with DAG(
    dag_id="k8s_job_always_fail",
    schedule="0 * * * *",
    start_date=datetime(2022, 12, 8),
    catchup=False,
    tags=["example", "k8s", "retry"],
) as dag:

    failing_task = KubernetesJobOperator(
        task_id="always_fail_job",
        namespace="default",
        image="alpine:latest",
        cmds=["/bin/sh", "-c", "echo failing... && exit 1"],
        retries=3,
        retry_delay=timedelta(seconds=30),
        wait_until_job_complete=True,
        deferrable=True,
        backoff_limit=0,
        do_xcom_push=False,
        get_logs=False,
    )
```

Result: Failed due to lack of permission (Airflow 3 seems to remove log download utils from UI, so using screenshot instead):
<img width="1024" alt="image" src="https://github.com/user-attachments/assets/7fbfce3b-4c04-4471-9541-238de8f5dfe1" />

```
ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '31359fbd-cdfe-40a8-88f0-93df023e7630', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'dfc53d28-fc9b-41f3-8478-7a0282759537', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'e10ca922-38fe-4420-891d-6ca2f1253406', 'Date': 'Tue, 01 Jul 2025 03:26:03 GMT', 'Content-Length': '321'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:airflow:airflow-scheduler\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"default\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
```

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13624390

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a comment on the discussion: Kuberentes integration settings/commands for Breeze environment?

Recent atttempt:

 - Fix crash issue by turning off some of the resource-consuming programs on host system for more RAM.
 - Modify [`kubernetes_commands.py`](https://github.com/apache/airflow/blob/12cd06d41366fe40ebd8712945d5201c5b0f19b2/dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py#L605) to copy things under `dags` folder into the image (`files` folder is not mounted onto the pods under this mode) during build stage. Once completing `build-k8s-image` and `upload-k8s-image` commands the cluster will automatically run the migration and restart using the latest image.
   ```dockerfile
   COPY --chown=airflow:0 dags/ /opt/airflow/dags/
   ```

Minimum test example:

```python
from datetime import timedelta

from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.job import KubernetesJobOperator
from pendulum import datetime

with DAG(
    dag_id="k8s_job_always_fail",
    schedule="0 * * * *",
    start_date=datetime(2022, 12, 8),
    catchup=False,
    tags=["example", "k8s", "retry"],
) as dag:

    failing_task = KubernetesJobOperator(
        task_id="always_fail_job",
        namespace="default",
        image="alpine:latest",
        cmds=["/bin/sh", "-c", "echo failing... && exit 1"],
        retries=3,
        retry_delay=timedelta(seconds=30),
        wait_until_job_complete=True,
        deferrable=True,
        backoff_limit=0,
        do_xcom_push=False,
        get_logs=False,
    )
```

Result: Failed due to lack of permission (Airflow 3 seems to remove log download utils from UI, so using screenshot instead):
<img width="1024" alt="image" src="https://github.com/user-attachments/assets/7fbfce3b-4c04-4471-9541-238de8f5dfe1" />

```
ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '31359fbd-cdfe-40a8-88f0-93df023e7630', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'dfc53d28-fc9b-41f3-8478-7a0282759537', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'e10ca922-38fe-4420-891d-6ca2f1253406', 'Date': 'Tue, 01 Jul 2025 03:26:03 GMT', 'Content-Length': '321'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:airflow:airflow-scheduler\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"default\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
```

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13624390

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a discussion: Kuberentes integration settings/commands for Breeze environment?

I also found this issue thread but seems this the has been suspended for more than a year after it's proposed:
- https://github.com/apache/airflow/issues/40005

Here's an unsuccessful attempt to create a K8s cluster locally and let Airflow to gain access to the cluster:

GitHub link: https://github.com/apache/airflow/discussions/52131

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a discussion: Kuberentes integration settings/commands for Breeze environment?

I also found this issue thread but seems this has been suspended for more than a year after it's proposed:
- https://github.com/apache/airflow/issues/40005

0. We've already have Docker (along with local K8s utilities) installed when installing Breeze.
1. Install [kind](https://kind.sigs.k8s.io/docs/user/quick-start/) and run `kind create cluster --name airflow-testing` to create a cluster named `kind-airflow-testing`. Check cluster information with `kubectl cluster-info --context kind-airflow-testing` command.
2. Now we have the cluster's configuration stored in `~/.kube/config`. Copy the configuration file to the shared mount location under repository root:
```bash
cp ~/.kube/config ./files/kubeconfig
```
3. Add environment variable to `files/airflow-breeze-config/environment_variables.env`:
```bash
KUBECONFIG=/files/kubeconfig
```
By default, K8s utilities will use what `KUBECONFIG` points to to initialize configuration.
4. Start Breeze environment and run Airflow:
```bash
breeze --python 3.9 --backend mysql --mysql-version 8.0 start-airflow
```
5. Build a DAG that uses any operator under "cncf-kuberentes" provider (well, one supposed to know where to place DAG files, if one, patiently, read through the Breeze doc to the [end chapter](https://github.com/apache/airflow/blob/207c4b09cf7c34b740472aca79dadc19c6f69db4/dev/breeze/doc/10_advanced_breeze_topics.rst)), enable it, run it, and welcome all kinds of `NewConnectionError` appearing on the console/Web-UI!

GitHub link: https://github.com/apache/airflow/discussions/52131

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a discussion: Kuberentes integration settings/commands for Breeze environment?

I also found this issue thread but seems this the has been suspended for more than a year after it's proposed:
- https://github.com/apache/airflow/issues/40005

GitHub link: https://github.com/apache/airflow/discussions/52131

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk added a comment to the discussion: Kuberentes integration settings/commands for Breeze environment?

It's really hard to debug errors without looking at logs, it's pretty much guessing - that's what the logs are for.

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13572725

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny added a comment to the discussion: Kuberentes integration settings/commands for Breeze environment?

Follow all the steps until step `breeze k8s deploy-airflow`, logs:

```
Default value of executor parameter LocalExecutor used.

Deploying Airflow for cluster airflow-python-3.9-v1.30.10
Deploying kind-airflow-python-3.9-v1.30.10 with airflow Helm Chart.
Copied chart sources to /private/var/folders/qg/0mrrt2zs53vbps1qfw7vxqq40000gq/T/chart_5b_pwgnf/chart
Deploying Airflow from /private/var/folders/qg/0mrrt2zs53vbps1qfw7vxqq40000gq/T/chart_5b_pwgnf/chart
I0625 11:36:10.072819   80585 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0625 11:36:10.072840   80585 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0625 11:36:10.072843   80585 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0625 11:36:10.072846   80585 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
NAME: airflow
LAST DEPLOYED: Wed Jun 25 11:36:10 2025
NAMESPACE: airflow
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing Apache Airflow 3.1.0!

Your release is named airflow.
You can now access your dashboard(s) by executing the following command(s) and visiting the corresponding port at localhost in your browser:

Airflow Webserver:     kubectl port-forward svc/airflow-webserver 8080:8080 --namespace airflow
Default Webserver (Airflow UI) Login credentials:
    username: admin
    password: admin
Default Postgres connection credentials:
    username: postgres
    password: postgres
    port: 5432

You can get Fernet Key value by running the following:

    echo Fernet Key: $(kubectl get secret --namespace airflow airflow-fernet-key -o jsonpath="{.data.fernet-key}" | base64 --decode)

###########################################################
#  WARNING: You should set a static webserver secret key  #
###########################################################

You are using a dynamically generated webserver secret key, which can lead to
unnecessary restarts of your Airflow components.

Information on how to set a static webserver secret key can be found here:
https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#webserver-secret-key
Deployed kind-airflow-python-3.9-v1.30.10 with airflow Helm Chart.

Airflow for Python 3.9 and K8S version v1.30.10 has been successfully deployed.

The KinD cluster name: airflow-python-3.9-v1.30.10
The kubectl cluster name: kind-airflow-python-3.9-v1.30.10.


Kubeconfig file in: /Users/pyhsieh/dev_repo/airflow_k/.build/.k8s-clusters/airflow-python-3.9-v1.30.10/.kubeconfig


KinD Cluster API server URL: http://localhost:24308
Connecting to localhost:13832. Num try: 1
Error when connecting to localhost:13832 : ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
Sleeping for 5 seconds.
(...)
Connecting to localhost:13832. Num try: 25
Error when connecting to localhost:13832 : ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
More than 120 passed. Exiting.

Airflow API server is not available at port 13832. Run `breeze k8s deploy-airflow --python 3.9 --kubernetes-version v1.30.10` to (re)deploy airflow


NEXT STEP: You might now run tests or interact with airflow via shell (kubectl, pytest etc.) or k9s commands:


breeze k8s tests

breeze k8s shell

breeze k8s k9s
```

Know something's broken because when using the k9s command I saw part of the components are not in normal state:

<img width="1345" alt="image" src="https://github.com/user-attachments/assets/ff623712-b439-4c1a-b1fe-da7c166dc906" />


Then try redeploy again with command `breeze k8s deploy-airflow --python 3.9 --kubernetes-version v1.30.10` and it failed without useful information:

```
Deploying Airflow for cluster airflow-python-3.9-v1.30.10
Deploying kind-airflow-python-3.9-v1.30.10 with airflow Helm Chart.
Copied chart sources to /private/var/folders/qg/0mrrt2zs53vbps1qfw7vxqq40000gq/T/chart_2atzjmkh/chart
Deploying Airflow from /private/var/folders/qg/0mrrt2zs53vbps1qfw7vxqq40000gq/T/chart_2atzjmkh/chart
I0625 12:09:33.307494   82192 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0625 12:09:33.307523   82192 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0625 12:09:33.307526   82192 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0625 12:09:33.307529   82192 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
Error: INSTALLATION FAILED: cannot re-use a name that is still in use
```

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13568891

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a comment on the discussion: Kuberentes integration settings/commands for Breeze environment?

Now the webserver's dead again and keep failing to restart itself:

<img width="1341" alt="image" src="https://github.com/user-attachments/assets/69b3ec10-6ed8-478a-9474-2a766696b6eb" />

Seems that in a environment with 6GB RAM provided (plus 3GB swap) without running any other thing in Docker, the cluster could suffer from:
 - Consistent crash scheduler with extremely long startup
 - Consistent crash of API server (web server) with faster startup

<img width="1345" alt="image" src="https://github.com/user-attachments/assets/9422e2f0-3219-4907-b326-092b2c540f89" />




GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13569444

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a discussion: Kuberentes integration settings/commands for Breeze environment?

I want to reproduce several issues I encountered when using operator classes under "[cncf-kuberentes](https://github.com/apache/airflow/discussions/52131#discussion-8487393)" provider. To better provide proof to these problems, I want to make a minimum example with Breeze development environment, but now I'm totally stuck and cannot figure out how to integrate a local Kubernetes cluster to a running Breeze environment (having very limited knowledge about Docker and Kubernetes, while both K8s environment setup and settings required to run these K8s tasks are mostly managed by other teams). Any idea or suggested guide to help with this scenario?

Also, there's no documentation on `breeze-k8s` subcommand, from help doc it seems to be utility that runs Airflow in local Kubernetes cluster (such as using Kuberentes Executor?), and I'm not certain if it does any help to this scenario.

I also found this issue thread but seems this the has been suspended for more than a year after it's proposed:
 - https://github.com/apache/airflow/issues/40005

GitHub link: https://github.com/apache/airflow/discussions/52131

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a discussion: Kuberentes integration settings/commands for Breeze environment?

I also found this issue thread but seems this the has been suspended for more than a year after it's proposed:
- https://github.com/apache/airflow/issues/40005

0. We've already have Docker (along with local K8s utilities) installed when installing Breeze.
1. Install [kind](https://kind.sigs.k8s.io/docs/user/quick-start/) and run `kind create cluster --name airflow-testing` to create a cluster named `kind-airflow-testing`. Check cluster information with `kubectl cluster-info --context kind-airflow-testing` command.
2. Now we have the cluster's configuration stored in `~/.kube/config`. Copy the configuration file to the shared mount location under repository root:
```bash
cp ~/.kube/config ./files/kubeconfig
```
3. Add environment variable to `files/airflow-breeze-config/environment_variables.env`:
```bash
KUBECONFIG=/files/kubeconfig
```
By default, K8s utilities will use what `KUBECONFIG` points to to initialize configuration.
4. Start Breeze environment and run Airflow:
```bash
breeze --python 3.9 --backend mysql --mysql-version 8.0 start-airflow
```
5. Build a DAG that uses any operator under "cncf-kuberentes" provider, enable it, run it, and welcome all kinds of `NewConnectionError` appearing on the console/Web-UI!

GitHub link: https://github.com/apache/airflow/discussions/52131

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny added a comment to the discussion: Kuberentes integration settings/commands for Breeze environment?

Recent atttempt:

 - Fix crash issue by turning off some of the resource-consuming programs on the host for more RAM.
 - Modify [`kubernetes_commands.py`](https://github.com/apache/airflow/blob/12cd06d41366fe40ebd8712945d5201c5b0f19b2/dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py#L605) to copy things under `dags` folder into the image (`files` folder is not mounted onto the pods under this mode) during build stage. Once completing `build-k8s-image` and `upload-k8s-image` commands the cluster will automatically run the migration and restart with latest image.

Minimum test example:

```python
from datetime import timedelta

from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.job import KubernetesJobOperator
from pendulum import datetime

with DAG(
    dag_id="k8s_job_always_fail",
    schedule="0 * * * *",
    start_date=datetime(2022, 12, 8),
    catchup=False,
    tags=["example", "k8s", "retry"],
) as dag:

    failing_task = KubernetesJobOperator(
        task_id="always_fail_job",
        namespace="default",
        image="alpine:latest",
        cmds=["/bin/sh", "-c", "echo failing... && exit 1"],
        retries=3,
        retry_delay=timedelta(seconds=30),
        wait_until_job_complete=True,
        deferrable=True,
        backoff_limit=0,
        do_xcom_push=False,
        get_logs=False,
    )
```

Result: Failed due to lack of permission (Airflow 3 seems to remove log download utils from UI):
<img width="1024" alt="image" src="https://github.com/user-attachments/assets/7fbfce3b-4c04-4471-9541-238de8f5dfe1" />

```
ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '31359fbd-cdfe-40a8-88f0-93df023e7630', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'dfc53d28-fc9b-41f3-8478-7a0282759537', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'e10ca922-38fe-4420-891d-6ca2f1253406', 'Date': 'Tue, 01 Jul 2025 03:26:03 GMT', 'Content-Length': '321'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:airflow:airflow-scheduler\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"default\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
```

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13624390

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a discussion: Kuberentes integration settings/commands for Breeze environment?

I also found this issue thread but seems this has been suspended for more than a year after it's proposed:
- https://github.com/apache/airflow/issues/40005

0. We've already have Docker (along with local K8s utilities) installed when installing Breeze.
1. Install [kind](https://kind.sigs.k8s.io/docs/user/quick-start/) and run `kind create cluster --name airflow-testing` to create a cluster named `kind-airflow-testing`. Check cluster information with `kubectl cluster-info --context kind-airflow-testing` command.
2. Now we have the cluster's configuration stored in `~/.kube/config`. Copy the configuration file to the shared mount location under repository root:
```bash
cp ~/.kube/config ./files/kubeconfig
```
3. Add environment variable to `files/airflow-breeze-config/environment_variables.env`:
```bash
KUBECONFIG=/files/kubeconfig
```
By default, K8s utilities will use what `KUBECONFIG` points to to initialize configuration.
4. Start Breeze environment and run Airflow:
```bash
breeze --python 3.9 --backend mysql --mysql-version 8.0 start-airflow
```
5. Build a DAG that uses any operator under "cncf-kuberentes" provider, enable it, run it, and welcome all kinds of `NewConnectionError` appearing on the console/Web-UI!

GitHub link: https://github.com/apache/airflow/discussions/52131

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk edited a comment on the discussion: Kuberentes integration settings/commands for Breeze environment?

Yes. Breeze `k8s` suite of commands is the wy to go. It has a bit of a difficult in iterating - i.e. every time you make changes to provider / airflow code you need to rebuild and redeploy the image. We have an open issue to improve the iteration speed - https://github.com/apache/airflow/issues/44508 - but we did not have time/energy to work on it and other things took priority.

You have detailed step-by-step instructions how to do it: https://github.com/apache/airflow/blob/main/contributing-docs/testing/k8s_tests.rst#typical-testing-pattern-for-kubernetes-tests  - including the fact that k8s suite of commands work in a "wizard" like way guiding you what are the next steps.

In short:

* it sets up a local venv which has everything configured and all tools (kind, kubectl, helm) automatically installed in this separate env
* it creates a kind cluster of your choice
* It builds a base image that you can use to deploy airflow with (with embedded example dags) 
* it allows you to rebuild the image with your latest changes
* it allows you to deploy the image to the kind cluster
* it uses helm and current "chart" from sources to start complete "airflow" installation in the cluster that you can connect to and interact with

So - the limitation is that every time you make a change you need to:
* rebuild (inclrementally - so it takes seconds) your image
* deploy it to kind cluster
* restart the pods

This is slow - but works. With the opened issue we imagine a workflow where tools like teleport are used to send the modified files via ssh to the running pods and the pods have a way to pick the files up and restart the interpreters  - that would be much faster, workflow but also more complex and brittle. 

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13560418

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny added a comment to the discussion: Kuberentes integration settings/commands for Breeze environment?

Now the webserver's dead again and keep failing to restart itself:

<img width="1341" alt="image" src="https://github.com/user-attachments/assets/69b3ec10-6ed8-478a-9474-2a766696b6eb" />


GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13569444

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny added a comment to the discussion: Kuberentes integration settings/commands for Breeze environment?

Update: Increased Docker's RAM and the cluster works now:
<img width="1346" alt="image" src="https://github.com/user-attachments/assets/daf9c0aa-9ae4-42de-bf7c-bfc4c91fd7a8" />


GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13569248

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a comment on the discussion: Kuberentes integration settings/commands for Breeze environment?

Follow all the steps until step `breeze k8s deploy-airflow`, logs:

```
Default value of executor parameter LocalExecutor used.

Deploying Airflow for cluster airflow-python-3.9-v1.30.10
Deploying kind-airflow-python-3.9-v1.30.10 with airflow Helm Chart.
Copied chart sources to /private/var/folders/qg/0mrrt2zs53vbps1qfw7vxqq40000gq/T/chart_5b_pwgnf/chart
Deploying Airflow from /private/var/folders/qg/0mrrt2zs53vbps1qfw7vxqq40000gq/T/chart_5b_pwgnf/chart
I0625 11:36:10.072819   80585 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0625 11:36:10.072840   80585 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0625 11:36:10.072843   80585 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0625 11:36:10.072846   80585 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
NAME: airflow
LAST DEPLOYED: Wed Jun 25 11:36:10 2025
NAMESPACE: airflow
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing Apache Airflow 3.1.0!

Your release is named airflow.
You can now access your dashboard(s) by executing the following command(s) and visiting the corresponding port at localhost in your browser:

Airflow Webserver:     kubectl port-forward svc/airflow-webserver 8080:8080 --namespace airflow
Default Webserver (Airflow UI) Login credentials:
    username: admin
    password: admin
Default Postgres connection credentials:
    username: postgres
    password: postgres
    port: 5432

You can get Fernet Key value by running the following:

    echo Fernet Key: $(kubectl get secret --namespace airflow airflow-fernet-key -o jsonpath="{.data.fernet-key}" | base64 --decode)

###########################################################
#  WARNING: You should set a static webserver secret key  #
###########################################################

You are using a dynamically generated webserver secret key, which can lead to
unnecessary restarts of your Airflow components.

Information on how to set a static webserver secret key can be found here:
https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#webserver-secret-key
Deployed kind-airflow-python-3.9-v1.30.10 with airflow Helm Chart.

Airflow for Python 3.9 and K8S version v1.30.10 has been successfully deployed.

The KinD cluster name: airflow-python-3.9-v1.30.10
The kubectl cluster name: kind-airflow-python-3.9-v1.30.10.


Kubeconfig file in: /AIRFLOW/REPO/ROOT/.build/.k8s-clusters/airflow-python-3.9-v1.30.10/.kubeconfig


KinD Cluster API server URL: http://localhost:24308
Connecting to localhost:13832. Num try: 1
Error when connecting to localhost:13832 : ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
Sleeping for 5 seconds.
(...)
Connecting to localhost:13832. Num try: 25
Error when connecting to localhost:13832 : ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
More than 120 passed. Exiting.

Airflow API server is not available at port 13832. Run `breeze k8s deploy-airflow --python 3.9 --kubernetes-version v1.30.10` to (re)deploy airflow


NEXT STEP: You might now run tests or interact with airflow via shell (kubectl, pytest etc.) or k9s commands:


breeze k8s tests

breeze k8s shell

breeze k8s k9s
```

Seems like the API server's not working. Know something's broken for sure because when looking into the cluster with `k9s` sub-command I saw part of the components are not in normal state:

<img width="1345" alt="image" src="https://github.com/user-attachments/assets/ff623712-b439-4c1a-b1fe-da7c166dc906" />

Then try redeploy again with command `breeze k8s deploy-airflow --python 3.9 --kubernetes-version v1.30.10` and it failed without giving useful information:

```
Deploying Airflow for cluster airflow-python-3.9-v1.30.10
Deploying kind-airflow-python-3.9-v1.30.10 with airflow Helm Chart.
Copied chart sources to /private/var/folders/qg/0mrrt2zs53vbps1qfw7vxqq40000gq/T/chart_2atzjmkh/chart
Deploying Airflow from /private/var/folders/qg/0mrrt2zs53vbps1qfw7vxqq40000gq/T/chart_2atzjmkh/chart
I0625 12:09:33.307494   82192 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0625 12:09:33.307523   82192 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0625 12:09:33.307526   82192 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0625 12:09:33.307529   82192 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
Error: INSTALLATION FAILED: cannot re-use a name that is still in use
```

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13568891

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk edited a comment on the discussion: Kuberentes integration settings/commands for Breeze environment?

Wild guess is that airflow is started (and service accounts are configured) to be runnung in "airflow" namespace -  not default - look it up, k9s can give you all the possibilities of inspecting running k8s. 

Generally you are in the realm of having everything working - you just need to understand k8s environment, permissions, service accounts etc.  

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13624468

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a comment on the discussion: Kuberentes integration settings/commands for Breeze environment?

Nope:

```
ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'c608a390-801c-4ff6-938d-1f33f6afc6b6', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'dfc53d28-fc9b-41f3-8478-7a0282759537', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'e10ca922-38fe-4420-891d-6ca2f1253406', 'Date': 'Tue, 01 Jul 2025 05:51:28 GMT', 'Content-Length': '321'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:airflow:airflow-scheduler\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"airflow\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
```

Looks like the runner doesn't have enough permission to spawn a job.

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13625046

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk added a comment to the discussion: Kuberentes integration settings/commands for Breeze environment?

Possibly you need to update helm chart to get it to have the permissions 

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13625583

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a comment on the discussion: Kuberentes integration settings/commands for Breeze environment?

Now the webserver's dead again and keep failing to restart itself:

<img width="1341" alt="image" src="https://github.com/user-attachments/assets/69b3ec10-6ed8-478a-9474-2a766696b6eb" />

Seems that in a environment with 6GB RAM provided (plus 3GB swap) without running any other thing in Docker, the cluster suffers from:
 - Long startup time of scheduler
 - Consistent crash of API server (web server)

<img width="1348" alt="image" src="https://github.com/user-attachments/assets/b61eb733-a52b-4b1c-a38c-2ab4921cfa71" />



GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13569444

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a comment on the discussion: Kuberentes integration settings/commands for Breeze environment?

Now the webserver's dead again and keep failing to restart itself:

<img width="1341" alt="image" src="https://github.com/user-attachments/assets/69b3ec10-6ed8-478a-9474-2a766696b6eb" />

Seems that in a environment with 6GB RAM provided (plus 3GB swap) without running any other thing in Docker, the cluster could suffer from:
 - Consistent crash of scheduler with extremely long startup
 - Consistent crash of API server (web server) with (relatively) fast startup

<img width="1345" alt="image" src="https://github.com/user-attachments/assets/9422e2f0-3219-4907-b326-092b2c540f89" />




GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13569444

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a discussion: Kuberentes integration settings/commands for Breeze environment?

I also found this issue thread but seems this the has been suspended for more than a year after it's proposed:
- https://github.com/apache/airflow/issues/40005

GitHub link: https://github.com/apache/airflow/discussions/52131

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a comment on the discussion: Kuberentes integration settings/commands for Breeze environment?

Follow all the steps until step `breeze k8s deploy-airflow`, logs:

```
Default value of executor parameter LocalExecutor used.

Deploying Airflow for cluster airflow-python-3.9-v1.30.10
Deploying kind-airflow-python-3.9-v1.30.10 with airflow Helm Chart.
Copied chart sources to /private/var/folders/qg/0mrrt2zs53vbps1qfw7vxqq40000gq/T/chart_5b_pwgnf/chart
Deploying Airflow from /private/var/folders/qg/0mrrt2zs53vbps1qfw7vxqq40000gq/T/chart_5b_pwgnf/chart
I0625 11:36:10.072819   80585 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0625 11:36:10.072840   80585 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0625 11:36:10.072843   80585 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0625 11:36:10.072846   80585 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
NAME: airflow
LAST DEPLOYED: Wed Jun 25 11:36:10 2025
NAMESPACE: airflow
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing Apache Airflow 3.1.0!

Your release is named airflow.
You can now access your dashboard(s) by executing the following command(s) and visiting the corresponding port at localhost in your browser:

Airflow Webserver:     kubectl port-forward svc/airflow-webserver 8080:8080 --namespace airflow
Default Webserver (Airflow UI) Login credentials:
    username: admin
    password: admin
Default Postgres connection credentials:
    username: postgres
    password: postgres
    port: 5432

You can get Fernet Key value by running the following:

    echo Fernet Key: $(kubectl get secret --namespace airflow airflow-fernet-key -o jsonpath="{.data.fernet-key}" | base64 --decode)

###########################################################
#  WARNING: You should set a static webserver secret key  #
###########################################################

You are using a dynamically generated webserver secret key, which can lead to
unnecessary restarts of your Airflow components.

Information on how to set a static webserver secret key can be found here:
https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#webserver-secret-key
Deployed kind-airflow-python-3.9-v1.30.10 with airflow Helm Chart.

Airflow for Python 3.9 and K8S version v1.30.10 has been successfully deployed.

The KinD cluster name: airflow-python-3.9-v1.30.10
The kubectl cluster name: kind-airflow-python-3.9-v1.30.10.


Kubeconfig file in: /AIRFLOW/REPO/ROOT/.build/.k8s-clusters/airflow-python-3.9-v1.30.10/.kubeconfig


KinD Cluster API server URL: http://localhost:24308
Connecting to localhost:13832. Num try: 1
Error when connecting to localhost:13832 : ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
Sleeping for 5 seconds.
(...)
Connecting to localhost:13832. Num try: 25
Error when connecting to localhost:13832 : ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
More than 120 passed. Exiting.

Airflow API server is not available at port 13832. Run `breeze k8s deploy-airflow --python 3.9 --kubernetes-version v1.30.10` to (re)deploy airflow


NEXT STEP: You might now run tests or interact with airflow via shell (kubectl, pytest etc.) or k9s commands:


breeze k8s tests

breeze k8s shell

breeze k8s k9s
```

Know something's broken because when using the k9s command I saw part of the components are not in normal state:

<img width="1345" alt="image" src="https://github.com/user-attachments/assets/ff623712-b439-4c1a-b1fe-da7c166dc906" />


Then try redeploy again with command `breeze k8s deploy-airflow --python 3.9 --kubernetes-version v1.30.10` and it failed without useful information:

```
Deploying Airflow for cluster airflow-python-3.9-v1.30.10
Deploying kind-airflow-python-3.9-v1.30.10 with airflow Helm Chart.
Copied chart sources to /private/var/folders/qg/0mrrt2zs53vbps1qfw7vxqq40000gq/T/chart_2atzjmkh/chart
Deploying Airflow from /private/var/folders/qg/0mrrt2zs53vbps1qfw7vxqq40000gq/T/chart_2atzjmkh/chart
I0625 12:09:33.307494   82192 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0625 12:09:33.307523   82192 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0625 12:09:33.307526   82192 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0625 12:09:33.307529   82192 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
Error: INSTALLATION FAILED: cannot re-use a name that is still in use
```

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13568891

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk added a comment to the discussion: Kuberentes integration settings/commands for Breeze environment?

Yes. Breeze `k8s` suite of commands is the wy to go. It has a bit of a difficult in iterating - i.e. every time you make changes to provider / airflow code you need to rebuild and redeploy the image. We have an open issue to improve the iteration speed - https://github.com/apache/airflow/issues/44508 - but we did not have time/energy to work on it and other things took priority.

You have details step-by-step instructions how to do it: https://github.com/apache/airflow/blob/main/contributing-docs/testing/k8s_tests.rst#typical-testing-pattern-for-kubernetes-tests  - including the fact that k8s suite of commands work in a "wizard" like way guiding you what are the next steps.

In short:

* it sets up a local venv which has everything configured and all tools (kind, kubectl, helm) automatically installed in this separate env
* it creates a kind cluster of your choice
* It builds a base image that you can use to deploy airflow with (with embedded example dags) 
* it allows you to rebuild the image with your latest changes
* it allows you to deploy the image to the kind cluster
* it uses helm to start complete "airflow" installation in the cluster that you can connect to and interact with

So - the limitation is that every time you make a change you need to:
* rebuild (inclrementally - so it takes seconds) your image
* deploy it to kind cluster
* restart the pods

This is slow - but works. With the opened issue we imagine a workflow where tools like teleport are used to send the modified files via ssh to the running pods and the pods have a way to pick the files up and restart the interpreters  - that would be much faster, workflow but also more complex and brittle. 

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13560418

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Kuberentes integration settings/commands for Breeze environment? [airflow]

Posted by "pykenny (via GitHub)" <gi...@apache.org>.

GitHub user pykenny edited a comment on the discussion: Kuberentes integration settings/commands for Breeze environment?

Follow all the steps until step `breeze k8s deploy-airflow`, logs:

```
Default value of executor parameter LocalExecutor used.

Deploying Airflow for cluster airflow-python-3.9-v1.30.10
Deploying kind-airflow-python-3.9-v1.30.10 with airflow Helm Chart.
Copied chart sources to /private/var/folders/qg/0mrrt2zs53vbps1qfw7vxqq40000gq/T/chart_5b_pwgnf/chart
Deploying Airflow from /private/var/folders/qg/0mrrt2zs53vbps1qfw7vxqq40000gq/T/chart_5b_pwgnf/chart
I0625 11:36:10.072819   80585 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0625 11:36:10.072840   80585 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0625 11:36:10.072843   80585 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0625 11:36:10.072846   80585 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
NAME: airflow
LAST DEPLOYED: Wed Jun 25 11:36:10 2025
NAMESPACE: airflow
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing Apache Airflow 3.1.0!

Your release is named airflow.
You can now access your dashboard(s) by executing the following command(s) and visiting the corresponding port at localhost in your browser:

Airflow Webserver:     kubectl port-forward svc/airflow-webserver 8080:8080 --namespace airflow
Default Webserver (Airflow UI) Login credentials:
    username: admin
    password: admin
Default Postgres connection credentials:
    username: postgres
    password: postgres
    port: 5432

You can get Fernet Key value by running the following:

    echo Fernet Key: $(kubectl get secret --namespace airflow airflow-fernet-key -o jsonpath="{.data.fernet-key}" | base64 --decode)

###########################################################
#  WARNING: You should set a static webserver secret key  #
###########################################################

You are using a dynamically generated webserver secret key, which can lead to
unnecessary restarts of your Airflow components.

Information on how to set a static webserver secret key can be found here:
https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#webserver-secret-key
Deployed kind-airflow-python-3.9-v1.30.10 with airflow Helm Chart.

Airflow for Python 3.9 and K8S version v1.30.10 has been successfully deployed.

The KinD cluster name: airflow-python-3.9-v1.30.10
The kubectl cluster name: kind-airflow-python-3.9-v1.30.10.


Kubeconfig file in: /AIRFLOW/REPO/ROOT/.build/.k8s-clusters/airflow-python-3.9-v1.30.10/.kubeconfig


KinD Cluster API server URL: http://localhost:24308
Connecting to localhost:13832. Num try: 1
Error when connecting to localhost:13832 : ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
Sleeping for 5 seconds.
(...)
Connecting to localhost:13832. Num try: 25
Error when connecting to localhost:13832 : ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
More than 120 passed. Exiting.

Airflow API server is not available at port 13832. Run `breeze k8s deploy-airflow --python 3.9 --kubernetes-version v1.30.10` to (re)deploy airflow


NEXT STEP: You might now run tests or interact with airflow via shell (kubectl, pytest etc.) or k9s commands:


breeze k8s tests

breeze k8s shell

breeze k8s k9s
```

Seems like the API server's not working. Know something's broken for sure because when looking into the cluster with `k9s` sub-command I saw part of the components are not in normal state:

<img width="1345" alt="image" src="https://github.com/user-attachments/assets/ff623712-b439-4c1a-b1fe-da7c166dc906" />

Then try redeploy with command `breeze k8s deploy-airflow --python 3.9 --kubernetes-version v1.30.10` (yes, I know there's `--upgrade` flag, but the message did not say so) and it failed without giving useful information:

```
Deploying Airflow for cluster airflow-python-3.9-v1.30.10
Deploying kind-airflow-python-3.9-v1.30.10 with airflow Helm Chart.
Copied chart sources to /private/var/folders/qg/0mrrt2zs53vbps1qfw7vxqq40000gq/T/chart_2atzjmkh/chart
Deploying Airflow from /private/var/folders/qg/0mrrt2zs53vbps1qfw7vxqq40000gq/T/chart_2atzjmkh/chart
I0625 12:09:33.307494   82192 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0625 12:09:33.307523   82192 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0625 12:09:33.307526   82192 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0625 12:09:33.307529   82192 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
Error: INSTALLATION FAILED: cannot re-use a name that is still in use
```

GitHub link: https://github.com/apache/airflow/discussions/52131#discussioncomment-13568891

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org