[GitHub] [airflow] MatrixManAtYrService opened a new issue #21775: Star expansion on XComArg causes DagImport timeout

[GitBox <gi...@apache.org>]

MatrixManAtYrService opened a new issue #21775:
URL: https://github.com/apache/airflow/issues/21775


   ### Apache Airflow version
   
   2.2.4 (latest released)
   
   ### What happened
   
   This DAG has a problem:  star-expansion can't be used on XComArgs
   
   ```
   from airflow import DAG
   from airflow.decorators import task
   from datetime import datetime
   
   @task
   def xcomify():
       return [1, 2]
   
   @task
   def add(a, b):
       return a + b
   
   with DAG(dag_id="some_numbers", start_date=datetime(1970, 1, 1)) as the_dag:
       add(*xcomify())
   ```
   
   If you check it for errors like `python dags/the_dag.py` it'll just hang without error output until interrupted.
   
   If you include it in your dags folder, you'll get an error:
   
   > Broken DAG: [/usr/local/airflow/dags/star_pos_expansion.py] Traceback (most recent call last):
   >   File "/usr/local/lib/python3.9/site-packages/airflow/models/xcom_arg.py", line 63, in __init__
   >     def __init__(self, operator: BaseOperator, key: str = XCOM_RETURN_KEY):
   >   File "/usr/local/lib/python3.9/site-packages/airflow/utils/timeout.py", line 37, in handle_timeout
   >     raise AirflowTaskTimeout(self.error_message)
   > airflow.exceptions.AirflowTaskTimeout: DagBag import timeout for /usr/local/airflow/dags/star_pos_expansion.py after 30.0s.
   > Please take a look at these docs to improve your DAG import time:
   > * http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/best-practices.html#top-level-python-code
   > * http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/best-practices.html#reducing-dag-complexity, PID: 169
   
   ### What you expected to happen
   
   A more specific error, and faster.  Something that warns that XComArgs can't be expanded as if they were lists (which they might reference, but which is not known by Airflow at dag-parse time).
   
   ### How to reproduce
   
   Run a python file containing the dag above.  Notice that it runs forever without error.
   Add it to your dags list.  Notice that you get a timeout error instead of something useful.
   
   ### Operating System
   
   docker/debian
   
   ### Versions of Apache Airflow Providers
   
   N/A
   
   ### Deployment
   
   Astronomer
   
   ### Deployment details
   
   `astro dev start`
   
   Dockerfile:
   ```
   FROM quay.io/astronomer/ap-airflow:2.2.4-onbuild
   ```
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org