You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/01/13 19:01:33 UTC
[GitHub] [airflow] dinigo opened a new issue #20857: Adding `output` to `templated_fields` makes operator raise `AirflowDagCycleException`
dinigo opened a new issue #20857:
URL: https://github.com/apache/airflow/issues/20857
### Apache Airflow version
2.2.3 (latest released)
### What happened
I'm working on an operator that receives a lot of params, by convention with the rest of the developers one of the named params is called `output`. However when you add this param to the `templated_fields` airflow Fails with the mentioned Exception
It seems like a reasonable name for a param. I've given a look at the `airflow.utils.dag_cycle_tester` but I couln't find the reason why this is raising the exception
```
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py", line 402, in _process_modules
self.bag_dag(dag=dag, root_dag=dag)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py", line 426, in bag_dag
self._bag_dag(dag=dag, root_dag=root_dag, recursive=True)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagbag.py", line 434, in _bag_dag
check_cycle(dag) # throws if a task cycle is found
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/dag_cycle_tester.py", line 76, in check_cycle
child_to_check = _check_adjacent_tasks(current_task_id, task)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/dag_cycle_tester.py", line 62, in _check_adjacent_tasks
raise AirflowDagCycleException(msg)
airflow.exceptions.AirflowDagCycleException: Cycle detected in DAG. Faulty task: test_template_output_task
```
### What you expected to happen
Should be able to template this `output` param
### How to reproduce
Simply try the following
```python
from datetime import datetime
from airflow import DAG
from airflow.models import BaseOperator
class TemplateOutputOperator(BaseOperator):
template_fields = ['output']
def __init__(self, output: str = None, *args, **kwargs):
super(TemplateOutputOperator, self).__init__(*args, **kwargs)
with DAG(
dag_id='test_template_output',
start_date=datetime(2021, 10, 22),
schedule_interval='@daily',
catchup=False,
doc_md=__doc__,
) as dag:
TemplateOutputOperator(task_id='test_template_output_task')
```
Should generate
### Operating System
Official docker image
### Versions of Apache Airflow Providers
Default ones
### Deployment
Docker-Compose
### Deployment details
Here's my docker-compose and Dockerfile
```Dockerfile
FROM apache/airflow:2.2.3-python3.8
USER root
RUN apt-get update && apt-get install git -y
USER airflow
COPY requirements.txt .
RUN set -ex && pip install --user --requirement requirements.txt
```
```yaml
version: '3.9'
services:
airflow:
build: .
secrets:
- gcp_credentials.json
environment:
AIRFLOW__WEBSERVER__NAVBAR_COLOR: orange
AIRFLOW__WEBSERVER__EXPOSE_CONFIG: True
AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS: False
GOOGLE_APPLICATION_CREDENTIALS: /run/secrets/gcp_credentials.json
command: standalone
volumes:
# mounts dags folder
- ./dags:/opt/airflow/dags
ports:
- "8080:8080"
```
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dinigo commented on issue #20857: Adding `output` to `templated_fields` makes operator raise `AirflowDagCycleException`
Posted by GitBox <gi...@apache.org>.
dinigo commented on issue #20857:
URL: https://github.com/apache/airflow/issues/20857#issuecomment-1012622806
From the docs
> ```python
> class HelloOperator(BaseOperator):
>
> template_fields = ["guest_name"]
> template_ext = [".sql"]
>
> def __init__(self, name: str, **kwargs) -> None:
> super().__init__(**kwargs)
> self.guest_name = name
> ```
> _In the example, the template_fields should be `['guest_name']` and not `['name']`_
I can see how what's templated is the "property" rather than the "constructor argument". The docs make it semi-clear. My error came from the `output` collyding with an internal `@property` of the `BaseOperator`.
Here's a test to prove docs are right
```python
import uuid
from datetime import datetime, timedelta, timezone
from typing import Any
import pytest
from airflow import DAG
from airflow.models import BaseOperator
from airflow.utils.state import DagRunState, State
from airflow.utils.types import DagRunType
class TemplateOutputOperator(BaseOperator):
template_fields = ['_output']
def __init__(self, output: str = None, *args, **kwargs):
self._output = output
super(TemplateOutputOperator, self).__init__(*args, **kwargs)
def execute(self, context: Any):
print(f"{self._output=}")
DATA_INTERVAL_START = datetime(2021, 9, 13, tzinfo=timezone.utc)
DATA_INTERVAL_END = DATA_INTERVAL_START + timedelta(days=1)
TEST_DAG_ID = "test_template_output_dag"
TEST_TASK_ID = "test_template_output_task"
@pytest.fixture()
def dag():
with DAG(
dag_id='test_template_output_dag',
schedule_interval="@daily",
default_args={"start_date": DATA_INTERVAL_START},
) as dag:
TemplateOutputOperator(
task_id='test_template_output_task',
output='{{ task_instance_key_str }}'
)
return dag
def test_operator_templates_output_param(dag):
dag_run = dag.create_dagrun(
state=DagRunState.RUNNING,
execution_date=datetime.now(timezone.utc),
data_interval=(DATA_INTERVAL_START, DATA_INTERVAL_END),
start_date=DATA_INTERVAL_END,
run_type=DagRunType.MANUAL,
run_id=uuid.uuid4().hex
)
ti = dag_run.get_task_instance(task_id=TEST_TASK_ID)
ti.task = dag.get_task(task_id=TEST_TASK_ID)
ti.run(ignore_ti_state=True)
assert str(ti.task._output).startswith(TEST_DAG_ID + '__' + TEST_TASK_ID)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dinigo closed issue #20857: Adding `output` to `templated_fields` makes operator raise `AirflowDagCycleException`
Posted by GitBox <gi...@apache.org>.
dinigo closed issue #20857:
URL: https://github.com/apache/airflow/issues/20857
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org