You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/09/23 15:05:34 UTC
[GitHub] [airflow] MatrixManAtYrService opened a new issue #18473: CLI: `airflow dags test { any dag with schedule_interval=None } ` causes error: "No run dates were found"
MatrixManAtYrService opened a new issue #18473:
URL: https://github.com/apache/airflow/issues/18473
### Apache Airflow version
2.2.0b2 (beta snapshot)
### Operating System
ubuntu 20.04
### Versions of Apache Airflow Providers
n/a
### Deployment
Virtualenv installation
### Deployment details
pip install /path/to/airflow/src
### What happened
Given any DAG initialized with: `schedule_interval=None`
Run `airflow dags test mydagname $(date +%Y-%m-%d)` and get an error:
```
INFO - No run dates were found for the given dates and dag interval.
```
This behavior changed in https://github.com/apache/airflow/pull/15397, it used to trigger a backfill dagrun at the given date.
### What you expected to happen
I expected a backfill dagrun with the given date, regardless of whether it fit into the `schedule_interval`.
If AIP-39 made that an unrealistic expectation, then I'd hope for some way to define unscheduled dags which can still be tested from the command line (which, so far as I know, is the fastest way to iterate on a DAG.).
As it is, I keep changing `schedule_interval` back and forth depending on whether I want to iterate via `astro dev start` (which tolerates `None` but does superfluous work if the dag is scheduled) or via `airflow dags test ...` (which doesn't tolerate `None`).
### How to reproduce
Initialize a DAG with: `schedule_interval=None` and run it via `airflow dags test mydagname $(date +%Y-%m-%d)`
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish closed issue #18473: CLI: `airflow dags test { dag w/ schedule_interval=None } ` error: "No run dates were found"
Posted by GitBox <gi...@apache.org>.
dstandish closed issue #18473:
URL: https://github.com/apache/airflow/issues/18473
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #18473: CLI: `airflow dags test { dag w/ schedule_interval=None } ` error: "No run dates were found"
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #18473:
URL: https://github.com/apache/airflow/issues/18473#issuecomment-934621277
> maybe eventually we should get rid of `airflow dags test` entirely and add a flag to `trigger` and `backfill` instead to make them block and emit logs to the CLI (i.o.w. allow the user to run them with `DebugExecutor` instead of the default `LocalExecutor`) to cover the use case. This would make more semantical sense because it'd be obvious to the user what kind of run they are requesting (backfill or manual), and it'd be easier to understand why a backfill does not trigger any runs at all.
Yeah now that you mention it, `dags test` is not not super obvious that it's actually running the task. One could think it is running _tests_ for the dag.
For me, as an airflow user, I rarely use `airflow test` or debug executor.
When I use debug executor it's like this:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #18473: CLI: `airflow dags test { dag w/ schedule_interval=None } ` error: "No run dates were found"
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #18473:
URL: https://github.com/apache/airflow/issues/18473#issuecomment-934406802
I've submitted a hack to restore the behaviour apache/airflow#18742. I'm still not entirely convinced this is the right thing to do; maybe eventually we should get rid of `airflow dags test` entirely and add a flag to `trigger` and `backfill` instead to make them block and emit logs to the CLI (i.o.w. allow the user to run them with `DebugExecutor` instead of the default `LocalExecutor`) to cover the use case. This would make more semantical sense because it'd be obvious to the user what kind of run they are requesting (backfill or manual), and it'd be easier to understand why a backfill does not trigger any runs at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish removed a comment on issue #18473: CLI: `airflow dags test { dag w/ schedule_interval=None } ` error: "No run dates were found"
Posted by GitBox <gi...@apache.org>.
dstandish removed a comment on issue #18473:
URL: https://github.com/apache/airflow/issues/18473#issuecomment-934621277
> maybe eventually we should get rid of `airflow dags test` entirely and add a flag to `trigger` and `backfill` instead to make them block and emit logs to the CLI (i.o.w. allow the user to run them with `DebugExecutor` instead of the default `LocalExecutor`) to cover the use case. This would make more semantical sense because it'd be obvious to the user what kind of run they are requesting (backfill or manual), and it'd be easier to understand why a backfill does not trigger any runs at all.
Yeah now that you mention it, `dags test` is not not super obvious that it's actually running the task. One could think it is running _tests_ for the dag.
For me, as an airflow user, I rarely use `airflow test` or debug executor.
When I use debug executor it's like this:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MatrixManAtYrService commented on issue #18473: CLI: `airflow dags test { dag w/ schedule_interval=None } ` error: "No run dates were found"
Posted by GitBox <gi...@apache.org>.
MatrixManAtYrService commented on issue #18473:
URL: https://github.com/apache/airflow/issues/18473#issuecomment-925904610
I had filed this issue in the wrong repo previously (oops). Here's a comment about it from @uranusjr :
> Hmm, this is some weird inherited design. So airflow dag test triggers backfill runs (in this instance one backfill run on start_date) but ignores all the scheduling limitations specifies by the DAG. It schedules backfills even for a None schedule (which logically never backfills), and ignore start_date, end_date, etc. The timetable implementation unified the scheduling logic and causes a None schedule to never backfill, and a backfill to never happen outside of the DAG and its tasks’ start_date and end_date.
>
> I think we have a design decision to make. We could stick to the current logic and change airflow dag test to trigger manual runs instead, but that’s affect the behaviour of DAGs that have a schedule since now the backfilled time to run more than once (in the previous implementation, if a date was run by airflow dag test, the scheduler won’t schedule a backfill at that time again). Or we need to add some additional logic to the timetable protocol (and users’ custom timetables) to account for this “hey I know this timetable don’t schedule backfills during this time but please give me a backfilling schedule anyway” logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #18473: CLI: `airflow dags test { dag w/ schedule_interval=None } ` error: "No run dates were found"
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #18473:
URL: https://github.com/apache/airflow/issues/18473#issuecomment-934623568
> maybe eventually we should get rid of `airflow dags test` entirely and add a flag to `trigger` and `backfill` instead to make them block and emit logs to the CLI (i.o.w. allow the user to run them with `DebugExecutor` instead of the default `LocalExecutor`) to cover the use case. This would make more semantical sense because it'd be obvious to the user what kind of run they are requesting (backfill or manual), and it'd be easier to understand why a backfill does not trigger any runs at all.
Yeah now that you mention it, `dags test` is not not super obvious that it's actually running the task. One could think it is running _tests_ for the dag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #18473: CLI: `airflow dags test { dag w/ schedule_interval=None } ` error: "No run dates were found"
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #18473:
URL: https://github.com/apache/airflow/issues/18473#issuecomment-926196797
Some additional unorganised thoughts:
1. It’s possible to match the pre-AIP-39 behaviour.
2. But I am not very keen to do it and would prefer other possibilities if possible because the previous design really does not make sense.
3. For this specific use case, changing the test runs to manual would be sufficient. But that may cause issues for DAGs that do have a backfilling schedule (changing the type to manual would result in duplicated runs, one backfilled and one manual).
4. Do we really need `airflow dags test` now since we can do `airflow dags trigger -e`? This has clearer semantic (obviously manual), and `airflow backfill -s ... -e ...` is available if you want backfill runs instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr edited a comment on issue #18473: CLI: `airflow dags test { dag w/ schedule_interval=None } ` error: "No run dates were found"
Posted by GitBox <gi...@apache.org>.
uranusjr edited a comment on issue #18473:
URL: https://github.com/apache/airflow/issues/18473#issuecomment-926519376
That’s a good point. It’s difficult to have the same behaviour the same for `trigger` right now (needs some refactoring), so if this is an important feature I’m inclined to restore the previous behaviour for 2.2 and plan for a deprecation/unification later. (I’m still not entirely convinced we should cover `schedule_interval=None` though, it really feels weird).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #18473: CLI: `airflow dags test { dag w/ schedule_interval=None } ` error: "No run dates were found"
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #18473:
URL: https://github.com/apache/airflow/issues/18473#issuecomment-926520565
cc @ashb for thoughts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MatrixManAtYrService commented on issue #18473: CLI: `airflow dags test { dag w/ schedule_interval=None } ` error: "No run dates were found"
Posted by GitBox <gi...@apache.org>.
MatrixManAtYrService commented on issue #18473:
URL: https://github.com/apache/airflow/issues/18473#issuecomment-926286376
regarding (4), the trouble with using those commands as a substitute for `airflow dags test` is that they don't show you the task logs, so they're not especially useful if your dev workflow looks like:
- edit mydag.py
- `airflow dags test mydag`
- look at task logs
- edit some more
Also, they require that you have a scheduler running, which is a small step, but still an extra setup/teardown to keep track of.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #18473: CLI: `airflow dags test { dag w/ schedule_interval=None } ` error: "No run dates were found"
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #18473:
URL: https://github.com/apache/airflow/issues/18473
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #18473: CLI: `airflow dags test { dag w/ schedule_interval=None } ` error: "No run dates were found"
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #18473:
URL: https://github.com/apache/airflow/issues/18473#issuecomment-926519376
That’s a good point. It’s difficult to have the same behaviour the same for `trigger` right now (needs some refactoring), so I’m inclined to restore the previous behaviour for 2.2 and plan for a deprecation/unification later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr edited a comment on issue #18473: CLI: `airflow dags test { dag w/ schedule_interval=None } ` error: "No run dates were found"
Posted by GitBox <gi...@apache.org>.
uranusjr edited a comment on issue #18473:
URL: https://github.com/apache/airflow/issues/18473#issuecomment-926196797
Some additional unorganised thoughts:
1. It’s possible to match the pre-AIP-39 behaviour.
2. But I am not very keen to do it and would prefer other possibilities if possible because the previous design really does not make sense.
3. For this specific use case, changing the test runs to manual would be sufficient. But that may cause issues for DAGs that do have a backfilling schedule (changing the type to manual would result in duplicated runs, one backfilled and one manual).
4. Do we really need `airflow dags test` now since we can do `airflow dags trigger -e`? This has clearer semantic (obviously manual), and `airflow backfill -s ... -e ...` is available if you want backfill runs instead. Maybe we can match the old behaviour and deprecate the entire command instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org