You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/02/15 11:20:27 UTC

[GitHub] [airflow] BasPH opened a new pull request #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command

BasPH opened a new pull request #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command
URL: https://github.com/apache/airflow/pull/7426
 
 
   This PR adds a CLI command "airflow dags test" for running one run of a DAG. Example usage:
   
   ```bash
   $ airflow dags test example_branch_operator 2018-01-01
   
   [2020-02-15 11:55:39,525] {executor_loader.py:60} INFO - Using executor DebugExecutor
   [2020-02-15 11:55:39,526] {dagbag.py:414} INFO - Filling up the DagBag from ...../airflow/dags
   [2020-02-15 11:55:39,642] {base_executor.py:74} INFO - Adding to queue: ['<TaskInstance: example_branch_operator.run_this_first 2018-01-01 00:00:00+00:00 [queued]>']
   [2020-02-15 11:55:44,624] {taskinstance.py:1033} INFO - Marking task as SUCCESS.dag_id=example_branch_operator, task_id=run_this_first, execution_date=20180101T000000, start_date=20200215T105539, end_date=20200215T105544
   [2020-02-15 11:55:44,650] {backfill_job.py:381} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 10 | succeeded: 1 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 10
   [2020-02-15 11:55:44,662] {base_executor.py:74} INFO - Adding to queue: ['<TaskInstance: example_branch_operator.branching 2018-01-01 00:00:00+00:00 [queued]>']
   [2020-02-15 11:55:49,645] {taskinstance.py:1033} INFO - Marking task as SUCCESS.dag_id=example_branch_operator, task_id=branching, execution_date=20180101T000000, start_date=20200215T105539, end_date=20200215T105549
   [2020-02-15 11:55:49,668] {backfill_job.py:381} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 9 | succeeded: 2 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 9
   [2020-02-15 11:55:49,686] {base_executor.py:74} INFO - Adding to queue: ['<TaskInstance: example_branch_operator.branch_a 2018-01-01 00:00:00+00:00 [queued]>']
   [2020-02-15 11:55:54,636] {taskinstance.py:1033} INFO - Marking task as SUCCESS.dag_id=example_branch_operator, task_id=branch_a, execution_date=20180101T000000, start_date=20200215T105539, end_date=20200215T105554
   [2020-02-15 11:55:54,657] {backfill_job.py:381} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 5 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 5
   [2020-02-15 11:55:54,668] {base_executor.py:74} INFO - Adding to queue: ['<TaskInstance: example_branch_operator.follow_branch_a 2018-01-01 00:00:00+00:00 [queued]>']
   [2020-02-15 11:55:59,644] {taskinstance.py:1033} INFO - Marking task as SUCCESS.dag_id=example_branch_operator, task_id=follow_branch_a, execution_date=20180101T000000, start_date=20200215T105539, end_date=20200215T105559
   [2020-02-15 11:55:59,673] {backfill_job.py:381} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 1 | succeeded: 4 | running: 0 | failed: 0 | skipped: 6 | deadlocked: 0 | not ready: 1
   [2020-02-15 11:55:59,691] {base_executor.py:74} INFO - Adding to queue: ['<TaskInstance: example_branch_operator.join 2018-01-01 00:00:00+00:00 [queued]>']
   [2020-02-15 11:56:04,649] {taskinstance.py:1033} INFO - Marking task as SUCCESS.dag_id=example_branch_operator, task_id=join, execution_date=20180101T000000, start_date=20200215T105539, end_date=20200215T105604
   [2020-02-15 11:56:04,671] {dagrun.py:327} INFO - Marking run <DagRun example_branch_operator @ 2018-01-01 00:00:00+00:00: backfill_2018-01-01T00:00:00+00:00, externally triggered: False> successful
   [2020-02-15 11:56:04,675] {backfill_job.py:381} INFO - [backfill progress] | finished run 1 of 1 | tasks waiting: 0 | succeeded: 5 | running: 0 | failed: 0 | skipped: 6 | deadlocked: 0 | not ready: 0
   [2020-02-15 11:56:04,675] {backfill_job.py:829} INFO - Backfill done. Exiting.
   ```
   
   The DebugExecutor docs currently suggest to simply call `run()` on a DAG, which would create DagRuns for all intervals between the start_date and end_date/now. I think this is undesirable and for testing purposes you would only want to run one single run of a DAG. Therefore (just like `airflow task test`), it takes a required argument `execution_date` for which a DagRun is created.
   
   ---
   Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [ ] Description above provides context of the change
   - [ ] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   <sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] BasPH commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command

Posted by GitBox <gi...@apache.org>.
BasPH commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command
URL: https://github.com/apache/airflow/pull/7426#issuecomment-612414301
 
 
   PR: https://github.com/apache/airflow/pull/8251

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] BasPH commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command

Posted by GitBox <gi...@apache.org>.
BasPH commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command
URL: https://github.com/apache/airflow/pull/7426#issuecomment-604879635
 
 
   Hi @mik-laj, it's on my todo list. Sorry, very busy lately...

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command

Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command
URL: https://github.com/apache/airflow/pull/7426#issuecomment-586583168
 
 
   There is `cli` method of `DAG`:
   https://github.com/apache/airflow/blob/edcad79b8d1ee359586c843c6e685cc58d6abacf/airflow/models/dag.py#L1393-L1400
   
   Would it be a good idea to update it with the new command? I think currently it doesn't work well...

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command

Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command
URL: https://github.com/apache/airflow/pull/7426#issuecomment-586582537
 
 
   Would you mind to put a note about it `TESTING.rst`?
   https://github.com/apache/airflow/blob/master/TESTING.rst#dag-testing

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command
URL: https://github.com/apache/airflow/pull/7426#issuecomment-601653089
 
 
   @BasPH Can you add some documentation? Now only a small group of contributors knows about this command.
   https://airflow.readthedocs.io/en/latest/tutorial.html#id2
   https://airflow.readthedocs.io/en/latest/usage-cli.html

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command

Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command
URL: https://github.com/apache/airflow/pull/7426#issuecomment-586613152
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7426?src=pr&el=h1) Report
   > Merging [#7426](https://codecov.io/gh/apache/airflow/pull/7426?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/edcad79b8d1ee359586c843c6e685cc58d6abacf?src=pr&el=desc) will **decrease** coverage by `0.29%`.
   > The diff coverage is `57.14%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7426/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7426?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff            @@
   ##           master    #7426     +/-   ##
   =========================================
   - Coverage   86.66%   86.36%   -0.3%     
   =========================================
     Files         878      878             
     Lines       41163    41168      +5     
   =========================================
   - Hits        35673    35556    -117     
   - Misses       5490     5612    +122
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7426?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7426/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <ø> (ø)` | :arrow_up: |
   | [airflow/utils/cli.py](https://codecov.io/gh/apache/airflow/pull/7426/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9jbGkucHk=) | `71.42% <100%> (ø)` | :arrow_up: |
   | [airflow/cli/commands/dag\_command.py](https://codecov.io/gh/apache/airflow/pull/7426/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGFnX2NvbW1hbmQucHk=) | `84.41% <40%> (-1.5%)` | :arrow_down: |
   | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7426/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
   | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7426/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
   | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7426/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-45.08%)` | :arrow_down: |
   | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7426/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.38% <0%> (-25.52%)` | :arrow_down: |
   | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7426/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7426?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7426?src=pr&el=footer). Last update [edcad79...b11c9e2](https://codecov.io/gh/apache/airflow/pull/7426?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin merged pull request #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command

Posted by GitBox <gi...@apache.org>.
nuclearpinguin merged pull request #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command
URL: https://github.com/apache/airflow/pull/7426
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command
URL: https://github.com/apache/airflow/pull/7426#issuecomment-586583608
 
 
   Here is PR about `dag.cli()`:
   https://github.com/apache/airflow/pull/6875
   Unfortunately, I don't have time to review it again. I will try to do it this week.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command
URL: https://github.com/apache/airflow/pull/7426#issuecomment-604694339
 
 
   This PR does not contain any tests. Not nice,

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services