You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "zvarszegi-ro5 (via GitHub)" <gi...@apache.org> on 2025/05/07 23:41:55 UTC

[I] Kubernetes task decorator can be broken by comments [airflow]

zvarszegi-ro5 opened a new issue, #50327:
URL: https://github.com/apache/airflow/issues/50327

   ### Apache Airflow Provider(s)
   
   cncf-kubernetes
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-amazon==8.25.0
   apache-airflow-providers-celery==3.7.2
   apache-airflow-providers-cncf-kubernetes==8.3.3
   apache-airflow-providers-common-io==1.3.2
   apache-airflow-providers-common-sql==1.14.2
   apache-airflow-providers-docker==3.12.2
   apache-airflow-providers-elasticsearch==5.4.1
   apache-airflow-providers-fab==1.2.2
   apache-airflow-providers-ftp==3.10.0
   apache-airflow-providers-google==10.21.0
   apache-airflow-providers-grpc==3.5.2
   apache-airflow-providers-hashicorp==3.7.1
   apache-airflow-providers-http==4.12.0
   apache-airflow-providers-imap==3.6.1
   apache-airflow-providers-microsoft-azure==10.2.0
   apache-airflow-providers-mysql==5.6.2
   apache-airflow-providers-odbc==4.6.2
   apache-airflow-providers-openlineage==1.9.1
   apache-airflow-providers-postgres==5.11.2
   apache-airflow-providers-redis==3.7.1
   apache-airflow-providers-sendgrid==3.5.1
   apache-airflow-providers-sftp==4.10.2
   apache-airflow-providers-slack==8.7.1
   apache-airflow-providers-smtp==1.7.1
   apache-airflow-providers-snowflake==5.6.0
   apache-airflow-providers-sqlite==3.8.1
   apache-airflow-providers-ssh==3.11.2
   
   
   ### Apache Airflow version
   
   v2.9.3  Git Version: .release:81845de9d95a733b4eb7826aaabe23ba9813eba3
   
   ### Operating System
   
   Debian 12
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   Airflow runs on Azure AKS (Kubernetes version `1.30.7`, Kustomize Version: `v5.0.4-0.20230601165947-6ce0bf390ce3`), deployed by `helm` (`version.BuildInfo{Version:"v3.17.0", GitCommit:"301108edc7ac2a8ba79e4ebf5701b0b6ce6a31e4", GitTreeState:"clean", GoVersion:"go1.23.4"}`) using the official helm chart `apache-airflow/airflow` version `1.15.0`. The metadata database is Postgres, version `17.3` from Helm chart `oci://registry-1.docker.io/bitnamicharts/postgresql` version `16.4.11`.
   
   ### What happened
   
   I have a DAG with two tasks, both tasks are decorated by `@task.kubernetes`. The first one passes some information to the second one, and I use the syntactic sugar to simplify XCom -  `task.override(do_xcom_push=True)` - the end result being that my task gets wrapped by the provider code in a new Python script (`/tmp/script.py`) which subsequently gets executed on the pod.
   
   If I put a comment line into the second task, that is not indented "nicely" - e.g. it starts from the beginning of the line - then my second task breaks with an error message similar to the following:  
   ```
   [2025-05-07, 22:22:40 UTC] {pod_manager.py:472} INFO - [base]   File "/tmp/script.py", line 22, in <module>
   [2025-05-07, 22:22:40 UTC] {pod_manager.py:472} INFO - [base]     @task.kubernetes(
   [2025-05-07, 22:22:40 UTC] {pod_manager.py:472} INFO - [base]      ^^^^
   [2025-05-07, 22:22:40 UTC] {pod_manager.py:490} INFO - [base] NameError: name 'task' is not defined
   ```
   This is very confusing because it makes you think it is an import error of some kind but it is clearly not, because it goes away if I move the comments to the right to match the indentation of the surrounding code.
   
   
   
   ### What you think should happen instead
   
   The task should just work and ignore the comments.
   
   ### How to reproduce
   
   The following DAG reproduces the issue (and has only dependencies bundled with Airflow, and only uses the `python:3.12` Docker image). It needs to run in a Kubernetes cluster, however I don't have an easy way to replicate that for the purposes of this bug report:
   ```
   from airflow.decorators import dag
   import pendulum
   
   
   @dag(
       dag_id='kubernetes-task-test-minimal',
       schedule=None,
       start_date=pendulum.datetime(2025, 5, 6, tz="UTC"),
       catchup=False,
   )
   def kubernetes_task_test_minimal():
       from airflow.decorators import task
   
       @task.kubernetes(
           namespace="airflow",
           image="python:3.12",
           name="a-task",
           in_cluster=True,
           get_logs=True,
           container_logs=True,
           on_finish_action='keep_pod',
       )
       def a_task() -> str:
           print("hello A")
           ##################    <- if I move these comment characters to the beginning of the line, that will cause a different exception
           print("running task A now")
           ##################
           return "success A"
   
   
       @task.kubernetes(
           namespace="airflow",
           image="python:3.12",
           name="b-task",
           in_cluster=True,
           get_logs=True,
           container_logs=True,
           on_finish_action='keep_pod',
       )
       def b_task(info: str) -> str:
           print("hello B", info)
   ##################            <- this line will cause the exception in the bug description above
           print("running task B now")
   ##################
           return "success B"
   
       info_a = a_task.override(do_xcom_push=True)()
       info_b = b_task.override(do_xcom_push=True)(info_a)
       print(info_b)
   
   kubernetes_task_test_minimal()
   ```
   
   ### Anything else
   
   The error happens consistently, every time.
   As noted above, if I place the comments at the beginning of the line in the first task, I get a different exception:
   ```
   [2025-05-07, 21:44:33 UTC] {pod_manager.py:472} INFO - [base]   File "/tmp/script.py", line 21
   [2025-05-07, 21:44:33 UTC] {pod_manager.py:472} INFO - [base]     @task.kubernetes(
   [2025-05-07, 21:44:33 UTC] {pod_manager.py:490} INFO - [base] IndentationError: unexpected indent
   ```
   (but even here the line number is not informative and the allegedly wrong indentation is not presented).
   
   I compared the correctly running and the failing contents of the generated script and it seems that the problem is that the `@task.kubernetes()` decorator does not get removed when I have the offending comments in place. The contents of this wrapper script is stored in an environment variable on the pod that is started by the Kubernetes executor and can be read running the following `kubectl` command. For me the generated pod name for `a-task` was `a-task-z87quaiw`, for `b-task` it was `b-task-8dqvpyq0` both running in the `airflow` namespace. Then to see the script generated from the first task I ran:
   ```
   $ kubectl -n airflow get pod a-task-z87quaiw -o go-template='{{range .spec.containers}}{{if eq .name "base"}}{{range .env}}{{if eq .name "__PYTHON_SCRIPT"}}{{.value | base64decode}}{{end}}{{end}}{{end}}{{end}}'
   from __future__ import annotations
   
   
   import json
   import pickle
   import sys
   
   
   if sys.version_info >= (3,6):
       try:
           from airflow.plugins_manager import integrate_macros_plugins
           integrate_macros_plugins()
       except ImportError:
           
           pass
   
   
   arg_dict = {"args": [], "kwargs": {}}
   
   
   # Script
   def a_task() -> str:
       print("hello A")
       ##################
       print("running task A now")
       ##################
       return "success A"
   
   res = a_task(*arg_dict["args"], **arg_dict["kwargs"])
   
   # Write output
   with open(sys.argv[2], "w") as file:
       if res is not None:
           file.write(json.dumps(res)) 
   ```
   And to read the script generated from the second task:
   ```
   $ kubectl -n airflow get pod b-task-8dqvpyq0 -o go-template='{{range .spec.containers}}{{if eq .name "base"}}{{range .env}}{{if eq .name "__PYTHON_SCRIPT"}}{{.value | base64decode}}{{end}}{{end}}{{end}}{{end}}'
   from __future__ import annotations
   
   import json
   import pickle
   import sys
   
   
   if sys.version_info >= (3,6):
       try:
           from airflow.plugins_manager import integrate_macros_plugins
           integrate_macros_plugins()
       except ImportError:
           
           pass
   
   
   with open(sys.argv[1], "rb") as file:
       arg_dict = pickle.load(file)
   
   
   # Script
       @task.kubernetes(
           namespace="airflow",
           image="python:3.12",
           name="b-task",
           in_cluster=True,
           get_logs=True,
           container_logs=True,
           on_finish_action='keep_pod',
       )
       def b_task(info: str) -> str:
           print("hello B", info)
   ##################
           print("running task B now")
   ##################
           return "success B"
   
   res = b_task(*arg_dict["args"], **arg_dict["kwargs"])
   
   # Write output
   with open(sys.argv[2], "w") as file:
       if res is not None:
           file.write(json.dumps(res))
   ```
   As you can see, in the second case,  the `@task.kubernetes` is present without importing or defining `task` hence an the exception is raised. If I properly indent the comments in the second task, then the decorator disappears from the script and everything works again.
   
   
   I looked into the source code a bit and I think it might be worth starting the investigation in the Kubernetes decorator where the task callables are [processed](https://github.com/apache/airflow/blob/2f02e8413fd6dd8c637a2111f6912f5b5d0202f7/airflow/providers/cncf/kubernetes/decorators/kubernetes.py#L82) involving an indentation library and [here](https://github.com/apache/airflow/blob/2f02e8413fd6dd8c637a2111f6912f5b5d0202f7/airflow/providers/cncf/kubernetes/python_kubernetes_script.py#L48) where the source is split on the decorator name.
   
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Kubernetes task decorator can be broken by comments [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #50327:
URL: https://github.com/apache/airflow/issues/50327#issuecomment-2860816581

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Kubernetes task decorator can be broken by comments [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #50327: Kubernetes task decorator can be broken by comments
URL: https://github.com/apache/airflow/issues/50327


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org