[GitHub] [airflow] johngtam opened a new issue #22298: pod_override ignores namespace configuration

[GitBox <gi...@apache.org>]

johngtam opened a new issue #22298:
URL: https://github.com/apache/airflow/issues/22298


   ### Apache Airflow version
   
   2.0.2
   
   ### What happened
   
   I've attempted to use the pod_override as per the documentation [here](https://airflow.apache.org/docs/apache-airflow/2.0.2/executor/kubernetes.html#pod-override). Labels, annotations, and service accounts work. However, attempting to override the namespace does not: I would expect the pod to be created in the namespace that I've set `pod_override` to, but it does not. 
   
   My speculation is that the `pod_generator.py` code is incorrect [here](https://github.com/apache/airflow/blob/2.0.2/airflow/kubernetes/pod_generator.py#L405). The order of reconciliation goes: 
   
   ```
           # Reconcile the pods starting with the first chronologically,
           # Pod from the pod_template_File -> Pod from executor_config arg -> Pod from the K8s executor
           pod_list = [base_worker_pod, pod_override_object, dynamic_pod]
   ```
   
   Note that dynamic pod takes precedence. Note that `dynamic_pod` has the namespace from whatever namespace is [passed](https://github.com/apache/airflow/blob/2.0.2/airflow/kubernetes/pod_generator.py#L373) in. It is initialized from `self.kube_config.kube_namespace` [here](https://github.com/apache/airflow/blob/ac77c89018604a96ea4f5fba938f2fbd7c582793/airflow/executors/kubernetes_executor.py#L245). Therefore, the kube config's namespace takes precedence and will override the namespace of the `pod_override`, because only one namespace can exist (unlike labels and annotations, which can be additive). This unfortunately means pods created by the KubernetesExecutor will run in the config's namespace. 
   
   This code flow remains the same on the latest version of Airflow -- I speculate that the bug still might be in. 
   
   ### What you think should happen instead
   
   If we use pod_override with a namespace, the namespace where the pod is run should take the namespace desired in pod_override.
   
   ### How to reproduce
   
   Create a DAG using a PythonOperator or any other operator. Use the pod_override:
   
   ```
           "pod_override": k8s.V1Pod(
                               metadata=k8s.V1ObjectMeta(namespace="fakenamespace",
                                                         annotations={"test": "annotation"},
                                                         labels={'foo': 'barBAZ'}),
                               spec=k8s.V1PodSpec(
                                       containers=[
                                           k8s.V1Container(
                                               name="base"
                                           )
                                       ],
                                       service_account_name="foobar"
                               )
                           )
   ```
   
   Look at the k8s spec file or use `kubectl` to verify where this pod is attempted to start running. It should be `fakenamespace`, but instead will likely be in the same namespace as whatever is set in the config file.
   
   ### Operating System
   
   EKS
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Other 3rd-party Helm chart
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #22298: pod_override ignores namespace configuration

[GitBox <gi...@apache.org>]

potiuk commented on issue #22298:
URL: https://github.com/apache/airflow/issues/22298#issuecomment-1074219302


   assigned you if you want to fix it @johngtam 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #22298: pod_override ignores namespace configuration

[GitBox <gi...@apache.org>]

boring-cyborg[bot] commented on issue #22298:
URL: https://github.com/apache/airflow/issues/22298#issuecomment-1068618813


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org