You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/03 11:09:15 UTC
[GitHub] [airflow] mmpyro opened a new issue, #24159: Inconsistent behavior between backfill and clear commands of airflow cli
mmpyro opened a new issue, #24159:
URL: https://github.com/apache/airflow/issues/24159
### Apache Airflow version
2.1.1
### What happened
I have a dag that runs periodically based on the crontab configuration. For example (5 4 * * *)
After some time the crontab configuration has changed (5 5 * * *) for that dag and we want to backfill past runs dag runs.
**airflow dags backfill cdm.itinerary_filed_fares --start-date '2021-05-18' --end-date '2021-05-23'**
Backfill works fine and run dags for missing time (5 5 * * *).
But when I run clear command
**airflow tasks clear cdm.itinerary_filed_fares --start-date '2021-05-18' --end-date '2021-05-23'**
Clear restart dags for old cront tab (5 4 * * *) instead of (5 5 * * *)
### What you think should happen instead
Backfill and clear cli commands behavior should be consistent.
When crontab has changes both commands should base on the new crontab.
The clear command should rerun only tasks where the run date is based on the new crontab.
### How to reproduce
- Please create some dag and set cron for example */40 * * * *. Wait some time until a few dags were executed.
- Change crontab for the dag to */50 * * * *.
- run backfill on that dag. It should run dags for the backfill time period which has a run date every 50 mins.
- run clear on the same dag, the same time period as backfill, and clear rerun only tasks which have run date every 40 mins.
### Operating System
coreos with containerd
### Versions of Apache Airflow Providers
apache-airflow[crypto,jdbc,kubernetes,postgres]==2.1.1
apache-airflow-providers-cncf-kubernetes==2.0.0 # via apache-airflow
apache-airflow-providers-ftp==2.0.0
apache-airflow-providers-google==2.0.0
apache-airflow-providers-http==2.0.0
apache-airflow-providers-imap==2.0.0
apache-airflow-providers-jdbc==2.0.0
apache-airflow-providers-postgres==2.0.0
apache-airflow-providers-sqlite==2.0.0
### Deployment
Other
### Deployment details
Airflow hosted on GKE Kubernetes cluster on Coreos with containerd node pool.
Dag is running on k8s executor.
Tasks are based on python operator.
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #24159: Inconsistent behavior between backfill and clear commands of airflow cli
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #24159:
URL: https://github.com/apache/airflow/issues/24159#issuecomment-1145855106
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24159: Inconsistent behavior between backfill and clear commands of airflow cli
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24159:
URL: https://github.com/apache/airflow/issues/24159#issuecomment-1145862602
This is not how it works until DAG versioning (https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-36+DAG+Versioning) is implemented. Those inconsistencies are expected.
As described here: https://airflow.apache.org/docs/apache-airflow/stable/index.html#beyond-the-horizon:
> Workflows are expected to be mostly static or slowly changing. You can think of the structure of the tasks in your workflow as slightly more dynamic than a database structure would be. Airflow workflows are expected to look similar from a run to the next, this allows for clarity around unit of work and continuity.
The same is with schedule - if you change the schedule and want consistency, you need to change dag_id and treat the dag as new DAG.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #24159: Inconsistent behavior between backfill and clear commands of airflow cli
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #24159: Inconsistent behavior between backfill and clear commands of airflow cli
URL: https://github.com/apache/airflow/issues/24159
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org