[I] Kubernetes task decorator default namespace behaves different than KubernetesPodOperator when in cluster [airflow]

["ketozhang (via GitHub)" <gi...@apache.org>]

ketozhang opened a new issue, #42957:
URL: https://github.com/apache/airflow/issues/42957

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### If "Other Airflow 2 version" selected, which one?
   
   2.9.2
   
   ### What happened?
   
   Given these settings for KuberneteseExecutor:
   
   Section | Key | Value
   -- | -- | --
   kubernetes_executor | in_cluster | True |
   kubernetes_executor | namespace | redacted_not_default |
   
   the behavior of `@task.kubernetes()` vs `KubernetesOperator()` when not setting namespace differs:
   
   - `@task.kubernetes()` defaults to `namespace="default"`
   - `KuberntesPodOperator()` defaults to `namespace=None` which uses the cluster namespace when `in_cluster` is true.
   
   ### What you think should happen instead?
   
   `KubernetesPodOperator` and `@task.kubernetes` should use the in-cluster namespace by default (i.e., the worker pod's namespace).
   
   ### How to reproduce
   
   1. Use a namespace that isn't `default`, let's call it `foobar` for this example
   2. Set the service account to only allow pod API access to `foobar` namespace.
   3. Run this DAG
   
       ```python
       with DAG(...):
           @task.kubernetes(image="hello-world"):
           def foo():
               ...
           taskA = foo()
       
           taskB = KubernetesPodOperator(image="hello-world")
       ```
   
   You'll see K8s API error for the wrong namespace. In my case, the KPO decorator is trying to access the `airflow` namespace despite the pod is running in the correct namespace
   
   ```
   HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:airflow:airflow\" cannot list resource \"pods\" in API group \"\" in the namespace \"airflow\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
   ```
   
   Here's the KPO decorator task's pod spec:
   ```
   metadata:
     annotations:
       dag_id: lensing_model_classification
       run_id: manual__2024-10-11T22:11:31.074729+00:00
       task_id: redacted
       try_number: '4'
     labels:
       airflow-worker: '0'
       airflow_version: 2.9.2
       dag_id: redacted
       kubernetes_executor: 'True'
       run_id: manual__2024-10-11T221131.0747290000-e1a8551c6
       task_id: redacted
       try_number: '4'
     name: redacted
     namespace: 'foobar'
   ...
   ```
   
   ### Operating System
   
   Amazon Linux 2
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-cncf-kubernetes==7.14.0
   
   ### Deployment
   
   Other 3rd-party Helm chart
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Kubernetes task decorator default namespace behaves different than KubernetesPodOperator when in cluster [airflow]

["ketozhang (via GitHub)" <gi...@apache.org>]

ketozhang commented on issue #42957:
URL: https://github.com/apache/airflow/issues/42957#issuecomment-2408207150

   In the meantime you can fix this by explicitly setting the namespace to `None`:
   
   ```python
   with DAG(...):
       @task.kubernetes(image="hello-world", namespace=None):
           ...
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Kubernetes task decorator default namespace behaves different than KubernetesPodOperator when in cluster [airflow]

["romsharon98 (via GitHub)" <gi...@apache.org>]

romsharon98 closed issue #42957: Kubernetes task decorator default namespace behaves different than KubernetesPodOperator when in cluster
URL: https://github.com/apache/airflow/issues/42957


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org