[I] Metadata secret required for workers [airflow]

["devgonvarun (via GitHub)" <gi...@apache.org>]

devgonvarun opened a new issue, #74850:
URL: https://github.com/apache/airflow/issues/74850

   ### Under which category would you file this issue?
   
   Airflow Core
   
   ### Apache Airflow version
   
   3.2.2
   
   ### What happened and how to reproduce it?
   
   Workers in Airflow 3 do not connect directly to the metadata db however they still require a metadata secret.
   
   Deploy a simple multinamespace mode setup. Do not place the metadata secret in the worker pod's namespace (k8sexecutor) and you will see that the tasks will fail. Now place a fake metadata secret in the namespace and you will observe that the tasks will start running. The worker requires this secret but does not actually use it for a connection.
   
   ### What you think should happen instead?
   
   Workers should not need the metadata connection secret. 
   
   ### Operating System
   
   Linux
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Apache Airflow Provider(s)
   
   _No response_
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Official Helm Chart version
   
   1.22.0 (latest released)
   
   ### Kubernetes Version
   
   _No response_
   
   ### Helm Chart configuration
   
   _No response_
   
   ### Docker Image customizations
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Metadata secret required for workers [airflow]

["anzinmhd (via GitHub)" <gi...@apache.org>]

anzinmhd commented on issue #74850:
URL: https://github.com/apache/airflow/issues/74850#issuecomment-6222772013

   I reproduced this with Airflow **3.2.2**, `KubernetesExecutor`, and `multiNamespaceMode: true` on a local Kubernetes cluster.
   
   The scheduler, API server, Dag processor, and database ran in one namespace. A single-task Dag used a pod override to launch its worker in a separate namespace. I provided the worker's required ConfigMaps, service account, and other template-referenced Secrets, but left the metadata Secret absent.
   
   Observed results:
   
   - Without the metadata Secret in the worker namespace, the Airflow worker container remained in `CreateContainerConfigError`. Kubernetes reported `secret "repro322-metadata" not found`.
   - After adding that Secret with a correctly formatted PostgreSQL URI containing deliberately invalid credentials, a fresh Dag run completed successfully. The worker reported `exit_code=0` and `final_state=success` and used the Execution API.
   
   The apparent cause is the shared Helm environment helper. The [worker pod template](https://github.com/apache/airflow/blob/b9aa889577/chart/files/pod-template-file.kubernetes-helm-yaml#L125) includes `standard_airflow_environment`, which adds both `AIRFLOW__DATABASE__SQL_ALCHEMY_CONN` and `AIRFLOW_CONN_AIRFLOW_DB` as mandatory [metadata Secret references](https://github.com/apache/airflow/blob/b9aa889577/chart/templates/_helpers.yaml#L73). Kubernetes must resolve these references before starting the container, even though this Airflow 3 task does not need valid metadata database credentials.
   
   I also compared the local worker template, helper, and default values used for this reproduction with upstream `main` at `b9aa889577`; all three files are identical. This was a runtime test of Airflow 3.2.2, not an execution test of Airflow built from `main`.
   
   I’m preparing a focused chart fix with regression tests and will open a PR shortly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org