You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/05/18 13:45:09 UTC
[GitHub] [airflow] Jaxing opened a new issue, #23776: Backfill failes due to removed ExternalTaskMarker
Jaxing opened a new issue, #23776:
URL: https://github.com/apache/airflow/issues/23776
### Apache Airflow version
2.2.5
### What happened
I tried to run a backfill on a dag but got
```
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 48, in main
args.func(args)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 92, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/dag_command.py", line 102, in dag_backfill
DAG.clear_dags(
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dag.py", line 1914, in clear_dags
tis = dag.clear(
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 70, in wrapper
return func(*args, session=session, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dag.py", line 1853, in clear
tis = self._get_task_instances(
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dag.py", line 1553, in _get_task_instances
raise AirflowException(f"Could not find dag {tii.dag_id}")
airflow.exceptions.AirflowException: Could not find dag bigquery-processed-data
```
Where `bigquery-processed-data` is an old dag that the current dag previously had a ExternalTaskMarker to.
### What you think should happen instead
I would expect it to ignore the old configuration of this dag and only use the current.
### How to reproduce
Not sure.
Possibly:
1. Create two dags A and B where A has an external task marker to B and B has a external task sensor to A.
2. Run once.
3. Remove dag B and the external task marker.
4. Try to backfill A
### Operating System
Debian GNU/Linux 10 (buster)
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==3.2.0
apache-airflow-providers-celery==2.1.3
apache-airflow-providers-cncf-kubernetes==3.0.0
apache-airflow-providers-docker==2.5.2
apache-airflow-providers-elasticsearch==2.2.0
apache-airflow-providers-ftp==2.1.2
apache-airflow-providers-google==6.7.0
apache-airflow-providers-grpc==2.0.4
apache-airflow-providers-hashicorp==2.1.4
apache-airflow-providers-http==2.1.2
apache-airflow-providers-imap==2.2.3
apache-airflow-providers-microsoft-azure==3.7.2
apache-airflow-providers-mysql==2.2.3
apache-airflow-providers-odbc==2.0.4
apache-airflow-providers-postgres==4.1.0
apache-airflow-providers-redis==2.0.4
apache-airflow-providers-sendgrid==2.0.4
apache-airflow-providers-sftp==2.5.2
apache-airflow-providers-slack==4.2.3
apache-airflow-providers-sqlite==2.1.3
apache-airflow-providers-ssh==2.4.3
### Deployment
Other Docker-based deployment
### Deployment details
Kubernetes deployment on GKE. I can add the yamls if they are relevant but I don't see how
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23776: Backfill failes due to removed ExternalTaskMarker
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23776:
URL: https://github.com/apache/airflow/issues/23776#issuecomment-1131687492
Indeed that's an interesting side-effect. We do not have DAG versioning implemented yet https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-36+DAG+Versioning so those kind of problems are expected. As described in https://airflow.apache.org/docs/apache-airflow/stable/index.html
> Workflows are expected to be mostly static or slowly changing. You can think of the structure of the tasks in your workflow as slightly more dynamic than a database structure would be. Airflow workflows are expected to look similar from a run to the next, this allows for clarity around unit of work and continuity.
We have currently no mechanisms to track version changes for DAGs, so if you do such structural changes, you might expect not only backfill to not work but also UI displays to be misleading. That's what DAG versioning (hopefully we will approve and start implementing on it soon) should address.
Currently, the recommendation is to copy DAGs to another dag in case of structural changes like that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Jaxing commented on issue #23776: Backfill failes due to removed ExternalTaskMarker
Posted by GitBox <gi...@apache.org>.
Jaxing commented on issue #23776:
URL: https://github.com/apache/airflow/issues/23776#issuecomment-1131759190
Ok, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #23776: Backfill failes due to removed ExternalTaskMarker
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #23776: Backfill failes due to removed ExternalTaskMarker
URL: https://github.com/apache/airflow/issues/23776
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org