You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/02/15 11:21:00 UTC

[jira] [Commented] (AIRFLOW-6765) Test full DAGs with airflow dags test

    [ https://issues.apache.org/jira/browse/AIRFLOW-6765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037496#comment-17037496 ] 

ASF GitHub Bot commented on AIRFLOW-6765:
-----------------------------------------

BasPH commented on pull request #7426: [AIRFLOW-6765] Add CLI "airflow dags test" command
URL: https://github.com/apache/airflow/pull/7426
 
 
   This PR adds a CLI command "airflow dags test" for running one run of a DAG. Example usage:
   
   ```bash
   $ airflow dags test example_branch_operator 2018-01-01
   
   [2020-02-15 11:55:39,525] {executor_loader.py:60} INFO - Using executor DebugExecutor
   [2020-02-15 11:55:39,526] {dagbag.py:414} INFO - Filling up the DagBag from ...../airflow/dags
   [2020-02-15 11:55:39,642] {base_executor.py:74} INFO - Adding to queue: ['<TaskInstance: example_branch_operator.run_this_first 2018-01-01 00:00:00+00:00 [queued]>']
   [2020-02-15 11:55:44,624] {taskinstance.py:1033} INFO - Marking task as SUCCESS.dag_id=example_branch_operator, task_id=run_this_first, execution_date=20180101T000000, start_date=20200215T105539, end_date=20200215T105544
   [2020-02-15 11:55:44,650] {backfill_job.py:381} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 10 | succeeded: 1 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 10
   [2020-02-15 11:55:44,662] {base_executor.py:74} INFO - Adding to queue: ['<TaskInstance: example_branch_operator.branching 2018-01-01 00:00:00+00:00 [queued]>']
   [2020-02-15 11:55:49,645] {taskinstance.py:1033} INFO - Marking task as SUCCESS.dag_id=example_branch_operator, task_id=branching, execution_date=20180101T000000, start_date=20200215T105539, end_date=20200215T105549
   [2020-02-15 11:55:49,668] {backfill_job.py:381} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 9 | succeeded: 2 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 9
   [2020-02-15 11:55:49,686] {base_executor.py:74} INFO - Adding to queue: ['<TaskInstance: example_branch_operator.branch_a 2018-01-01 00:00:00+00:00 [queued]>']
   [2020-02-15 11:55:54,636] {taskinstance.py:1033} INFO - Marking task as SUCCESS.dag_id=example_branch_operator, task_id=branch_a, execution_date=20180101T000000, start_date=20200215T105539, end_date=20200215T105554
   [2020-02-15 11:55:54,657] {backfill_job.py:381} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 5 | succeeded: 3 | running: 0 | failed: 0 | skipped: 3 | deadlocked: 0 | not ready: 5
   [2020-02-15 11:55:54,668] {base_executor.py:74} INFO - Adding to queue: ['<TaskInstance: example_branch_operator.follow_branch_a 2018-01-01 00:00:00+00:00 [queued]>']
   [2020-02-15 11:55:59,644] {taskinstance.py:1033} INFO - Marking task as SUCCESS.dag_id=example_branch_operator, task_id=follow_branch_a, execution_date=20180101T000000, start_date=20200215T105539, end_date=20200215T105559
   [2020-02-15 11:55:59,673] {backfill_job.py:381} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 1 | succeeded: 4 | running: 0 | failed: 0 | skipped: 6 | deadlocked: 0 | not ready: 1
   [2020-02-15 11:55:59,691] {base_executor.py:74} INFO - Adding to queue: ['<TaskInstance: example_branch_operator.join 2018-01-01 00:00:00+00:00 [queued]>']
   [2020-02-15 11:56:04,649] {taskinstance.py:1033} INFO - Marking task as SUCCESS.dag_id=example_branch_operator, task_id=join, execution_date=20180101T000000, start_date=20200215T105539, end_date=20200215T105604
   [2020-02-15 11:56:04,671] {dagrun.py:327} INFO - Marking run <DagRun example_branch_operator @ 2018-01-01 00:00:00+00:00: backfill_2018-01-01T00:00:00+00:00, externally triggered: False> successful
   [2020-02-15 11:56:04,675] {backfill_job.py:381} INFO - [backfill progress] | finished run 1 of 1 | tasks waiting: 0 | succeeded: 5 | running: 0 | failed: 0 | skipped: 6 | deadlocked: 0 | not ready: 0
   [2020-02-15 11:56:04,675] {backfill_job.py:829} INFO - Backfill done. Exiting.
   ```
   
   The DebugExecutor docs currently suggest to simply call `run()` on a DAG, which would create DagRuns for all intervals between the start_date and end_date/now. I think this is undesirable and for testing purposes you would only want to run one single run of a DAG. Therefore (just like `airflow task test`), it takes a required argument `execution_date` for which a DagRun is created.
   
   ---
   Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [ ] Description above provides context of the change
   - [ ] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   <sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Test full DAGs with airflow dags test
> -------------------------------------
>
>                 Key: AIRFLOW-6765
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6765
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: cli
>    Affects Versions: 2.0.0
>            Reporter: Bas Harenslak
>            Assignee: Bas Harenslak
>            Priority: Major
>
> We now have a DebugExecutor for running a full DAG. Similar to airflow test (airflow tasks test in 2.0), it would be nice to have a command "airflow dags test", for running a full DAG with the CLI.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)