You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/02/15 11:20:27 UTC
[GitHub] [airflow] BasPH opened a new pull request #7426: [AIRFLOW-6765] Add
CLI "airflow dags test" command
BasPH opened a new pull request #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command
URL: https://github.com/apache/airflow/pull/7426
This PR adds a CLI command "airflow dags test" for running one run of a DAG. Example usage:
```bash
$ airflow dags test example_branch_operator 2018-01-01
[2020-02-15 11:55:39,525] {executor_loader.py:60} INFO - Using executor DebugExecutor
[2020-02-15 11:55:39,526] {dagbag.py:414} INFO - Filling up the DagBag from ...../airflow/dags
[2020-02-15 11:55:39,642] {base_executor.py:74} INFO - Adding to queue: ['<TaskInstance: example_branch_operator.run_this_first 2018-01-01 00:00:00+00:00 [queued]>']
[2020-02-15 11:55:44,624] {taskinstance.py:1033} INFO - Marking task as SUCCESS.dag_id=example_branch_operator, task_id=run_this_first, execution_date=20180101T000000, start_date=20200215T105539, end_date=20200215T105544
[2020-02-15 11:55:44,650] {backfill_job.py:381} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 10 | succeeded: 1 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 10
[2020-02-15 11:55:44,662] {base_executor.py:74} INFO - Adding to queue: ['<TaskInstance: example_branch_operator.branching 2018-01-01 00:00:00+00:00 [queued]>']
[2020-02-15 11:55:49,645] {taskinstance.py:1033} INFO - Marking task as SUCCESS.dag_id=example_branch_operator, task_id=branching, execution_date=20180101T000000, start_date=20200215T105539, end_date=20200215T105549
[2020-02-15 11:55:49,668] {backfill_job.py:381} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 9 | succeeded: 2 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 9
[2020-02-15 11:55:49,686] {base_executor.py:74} INFO - Adding to queue: ['<TaskInstance: example_branch_operator.branch_a 2018-01-01 00:00:00+00:00 [queued]>']
[2020-02-15 11:55:54,636] {taskinstance.py:1033} INFO - Marking task as SUCCESS.dag_id=example_branch_operator, task_id=branch_a, execution_date=20180101T000000, start_date=20200215T105539, end_date=20200215T105554
[2020-02-15 11:55:54,657] {backfill_job.py:381} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 5 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 5
[2020-02-15 11:55:54,668] {base_executor.py:74} INFO - Adding to queue: ['<TaskInstance: example_branch_operator.follow_branch_a 2018-01-01 00:00:00+00:00 [queued]>']
[2020-02-15 11:55:59,644] {taskinstance.py:1033} INFO - Marking task as SUCCESS.dag_id=example_branch_operator, task_id=follow_branch_a, execution_date=20180101T000000, start_date=20200215T105539, end_date=20200215T105559
[2020-02-15 11:55:59,673] {backfill_job.py:381} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 1 | succeeded: 4 | running: 0 | failed: 0 | skipped: 6 | deadlocked: 0 | not ready: 1
[2020-02-15 11:55:59,691] {base_executor.py:74} INFO - Adding to queue: ['<TaskInstance: example_branch_operator.join 2018-01-01 00:00:00+00:00 [queued]>']
[2020-02-15 11:56:04,649] {taskinstance.py:1033} INFO - Marking task as SUCCESS.dag_id=example_branch_operator, task_id=join, execution_date=20180101T000000, start_date=20200215T105539, end_date=20200215T105604
[2020-02-15 11:56:04,671] {dagrun.py:327} INFO - Marking run <DagRun example_branch_operator @ 2018-01-01 00:00:00+00:00: backfill_2018-01-01T00:00:00+00:00, externally triggered: False> successful
[2020-02-15 11:56:04,675] {backfill_job.py:381} INFO - [backfill progress] | finished run 1 of 1 | tasks waiting: 0 | succeeded: 5 | running: 0 | failed: 0 | skipped: 6 | deadlocked: 0 | not ready: 0
[2020-02-15 11:56:04,675] {backfill_job.py:829} INFO - Backfill done. Exiting.
```
The DebugExecutor docs currently suggest to simply call `run()` on a DAG, which would create DagRuns for all intervals between the start_date and end_date/now. I think this is undesirable and for testing purposes you would only want to run one single run of a DAG. Therefore (just like `airflow task test`), it takes a required argument `execution_date` for which a DagRun is created.
---
Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [ ] Description above provides context of the change
- [ ] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
- [ ] Unit tests coverage for changes (not needed for documentation changes)
- [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [ ] Relevant documentation is updated including usage instructions.
- [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
<sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] BasPH commented on issue #7426: [AIRFLOW-6765] Add CLI
"airflow dags test" command
Posted by GitBox <gi...@apache.org>.
BasPH commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command
URL: https://github.com/apache/airflow/pull/7426#issuecomment-612414301
PR: https://github.com/apache/airflow/pull/8251
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] BasPH commented on issue #7426: [AIRFLOW-6765] Add CLI
"airflow dags test" command
Posted by GitBox <gi...@apache.org>.
BasPH commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command
URL: https://github.com/apache/airflow/pull/7426#issuecomment-604879635
Hi @mik-laj, it's on my todo list. Sorry, very busy lately...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on issue #7426: [AIRFLOW-6765]
Add CLI "airflow dags test" command
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command
URL: https://github.com/apache/airflow/pull/7426#issuecomment-586583168
There is `cli` method of `DAG`:
https://github.com/apache/airflow/blob/edcad79b8d1ee359586c843c6e685cc58d6abacf/airflow/models/dag.py#L1393-L1400
Would it be a good idea to update it with the new command? I think currently it doesn't work well...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on issue #7426: [AIRFLOW-6765]
Add CLI "airflow dags test" command
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command
URL: https://github.com/apache/airflow/pull/7426#issuecomment-586582537
Would you mind to put a note about it `TESTING.rst`?
https://github.com/apache/airflow/blob/master/TESTING.rst#dag-testing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #7426: [AIRFLOW-6765] Add CLI
"airflow dags test" command
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command
URL: https://github.com/apache/airflow/pull/7426#issuecomment-601653089
@BasPH Can you add some documentation? Now only a small group of contributors knows about this command.
https://airflow.readthedocs.io/en/latest/tutorial.html#id2
https://airflow.readthedocs.io/en/latest/usage-cli.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7426: [AIRFLOW-6765] Add
CLI "airflow dags test" command
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command
URL: https://github.com/apache/airflow/pull/7426#issuecomment-586613152
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7426?src=pr&el=h1) Report
> Merging [#7426](https://codecov.io/gh/apache/airflow/pull/7426?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/edcad79b8d1ee359586c843c6e685cc58d6abacf?src=pr&el=desc) will **decrease** coverage by `0.29%`.
> The diff coverage is `57.14%`.
[![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7426/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7426?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7426 +/- ##
=========================================
- Coverage 86.66% 86.36% -0.3%
=========================================
Files 878 878
Lines 41163 41168 +5
=========================================
- Hits 35673 35556 -117
- Misses 5490 5612 +122
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7426?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7426/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <ø> (ø)` | :arrow_up: |
| [airflow/utils/cli.py](https://codecov.io/gh/apache/airflow/pull/7426/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9jbGkucHk=) | `71.42% <100%> (ø)` | :arrow_up: |
| [airflow/cli/commands/dag\_command.py](https://codecov.io/gh/apache/airflow/pull/7426/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGFnX2NvbW1hbmQucHk=) | `84.41% <40%> (-1.5%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7426/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7426/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7426/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-45.08%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7426/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.38% <0%> (-25.52%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7426/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7426?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7426?src=pr&el=footer). Last update [edcad79...b11c9e2](https://codecov.io/gh/apache/airflow/pull/7426?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin merged pull request #7426: [AIRFLOW-6765]
Add CLI "airflow dags test" command
Posted by GitBox <gi...@apache.org>.
nuclearpinguin merged pull request #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command
URL: https://github.com/apache/airflow/pull/7426
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #7426: [AIRFLOW-6765] Add CLI
"airflow dags test" command
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command
URL: https://github.com/apache/airflow/pull/7426#issuecomment-586583608
Here is PR about `dag.cli()`:
https://github.com/apache/airflow/pull/6875
Unfortunately, I don't have time to review it again. I will try to do it this week.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #7426: [AIRFLOW-6765] Add CLI
"airflow dags test" command
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command
URL: https://github.com/apache/airflow/pull/7426#issuecomment-604694339
This PR does not contain any tests. Not nice,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services