You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/01/13 19:01:33 UTC

[GitHub] [airflow] dinigo opened a new issue #20857: Adding `output` to `templated_fields` makes operator raise `AirflowDagCycleException`

dinigo opened a new issue #20857:
URL: https://github.com/apache/airflow/issues/20857


   ### Apache Airflow version
   
   2.2.3 (latest released)
   
   ### What happened
   
   I'm working on an operator that receives a lot of params, by convention with the rest of the developers one of the named params is called `output`. However when you add this param to the `templated_fields` airflow Fails with the mentioned Exception
   
   It seems like a reasonable name for a param. I've given a look at the `airflow.utils.dag_cycle_tester` but I couln't find the reason why this is raising the exception
   
   
   ```
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py", line 402, in _process_modules
       self.bag_dag(dag=dag, root_dag=dag)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py", line 426, in bag_dag
       self._bag_dag(dag=dag, root_dag=root_dag, recursive=True)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py", line 434, in _bag_dag
       check_cycle(dag)  # throws if a task cycle is found
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/dag_cycle_tester.py", line 76, in check_cycle
       child_to_check = _check_adjacent_tasks(current_task_id, task)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/dag_cycle_tester.py", line 62, in _check_adjacent_tasks
       raise AirflowDagCycleException(msg)
   airflow.exceptions.AirflowDagCycleException: Cycle detected in DAG. Faulty task: test_template_output_task
   
   ```
   
   ### What you expected to happen
   
   Should be able to template this `output` param
   
   ### How to reproduce
   
   Simply try the following
   
   ```python
   from datetime import datetime
   
   from airflow import DAG
   from airflow.models import BaseOperator
   
   
   class TemplateOutputOperator(BaseOperator):
       template_fields = ['output']
   
       def __init__(self, output: str = None, *args, **kwargs):
           super(TemplateOutputOperator, self).__init__(*args, **kwargs)
   
   
   with DAG(
       dag_id='test_template_output',
       start_date=datetime(2021, 10, 22),
       schedule_interval='@daily',
       catchup=False,
       doc_md=__doc__,
   ) as dag:
       TemplateOutputOperator(task_id='test_template_output_task')
   ```
   
   
   Should generate 
   
   ### Operating System
   
   Official docker image
   
   ### Versions of Apache Airflow Providers
   
   Default ones
   
   ### Deployment
   
   Docker-Compose
   
   ### Deployment details
   
   Here's my docker-compose and Dockerfile
   
   ```Dockerfile
   FROM apache/airflow:2.2.3-python3.8
   
   USER root
   RUN apt-get update && apt-get install git -y
   USER airflow
   
   COPY requirements.txt .
   RUN set -ex && pip install --user --requirement requirements.txt
   ```
   
   
   
   ```yaml
   version: '3.9'
   
   services:
     airflow:
       build: .
       secrets:
         - gcp_credentials.json
       environment:
         AIRFLOW__WEBSERVER__NAVBAR_COLOR: orange
         AIRFLOW__WEBSERVER__EXPOSE_CONFIG: True
         AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS: False
         GOOGLE_APPLICATION_CREDENTIALS: /run/secrets/gcp_credentials.json
       command: standalone
       volumes:
         # mounts dags folder
         - ./dags:/opt/airflow/dags
       ports:
         - "8080:8080"
   ```
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] dinigo commented on issue #20857: Adding `output` to `templated_fields` makes operator raise `AirflowDagCycleException`

Posted by GitBox <gi...@apache.org>.
dinigo commented on issue #20857:
URL: https://github.com/apache/airflow/issues/20857#issuecomment-1012622806


   From the docs
   
   > ```python
   > class HelloOperator(BaseOperator):
   > 
   >     template_fields = ["guest_name"]
   >     template_ext = [".sql"]
   > 
   >     def __init__(self, name: str, **kwargs) -> None:
   >         super().__init__(**kwargs)
   >         self.guest_name = name
   > ```
   > _In the example, the template_fields should be `['guest_name']` and not `['name']`_
   
   
   I can see how what's templated is the "property" rather than the "constructor argument". The docs make it semi-clear. My error came from the `output` collyding with an internal `@property` of the `BaseOperator`.
   
   
   Here's a test to prove docs are right
   
   ```python
   import uuid
   from datetime import datetime, timedelta, timezone
   from typing import Any
   
   import pytest
   from airflow import DAG
   from airflow.models import BaseOperator
   from airflow.utils.state import DagRunState, State
   from airflow.utils.types import DagRunType
   
   
   class TemplateOutputOperator(BaseOperator):
       template_fields = ['_output']
   
       def __init__(self, output: str = None, *args, **kwargs):
           self._output = output
           super(TemplateOutputOperator, self).__init__(*args, **kwargs)
   
       def execute(self, context: Any):
           print(f"{self._output=}")
   
   
   DATA_INTERVAL_START = datetime(2021, 9, 13, tzinfo=timezone.utc)
   DATA_INTERVAL_END = DATA_INTERVAL_START + timedelta(days=1)
   
   TEST_DAG_ID = "test_template_output_dag"
   TEST_TASK_ID = "test_template_output_task"
   
   
   @pytest.fixture()
   def dag():
       with DAG(
           dag_id='test_template_output_dag',
           schedule_interval="@daily",
           default_args={"start_date": DATA_INTERVAL_START},
       ) as dag:
           TemplateOutputOperator(
               task_id='test_template_output_task',
               output='{{ task_instance_key_str }}'
           )
       return dag
   
   
   def test_operator_templates_output_param(dag):
       dag_run = dag.create_dagrun(
           state=DagRunState.RUNNING,
           execution_date=datetime.now(timezone.utc),
           data_interval=(DATA_INTERVAL_START, DATA_INTERVAL_END),
           start_date=DATA_INTERVAL_END,
           run_type=DagRunType.MANUAL,
           run_id=uuid.uuid4().hex
       )
       ti = dag_run.get_task_instance(task_id=TEST_TASK_ID)
       ti.task = dag.get_task(task_id=TEST_TASK_ID)
       ti.run(ignore_ti_state=True)
       assert str(ti.task._output).startswith(TEST_DAG_ID + '__' + TEST_TASK_ID)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] dinigo closed issue #20857: Adding `output` to `templated_fields` makes operator raise `AirflowDagCycleException`

Posted by GitBox <gi...@apache.org>.
dinigo closed issue #20857:
URL: https://github.com/apache/airflow/issues/20857


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org