You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2019/12/28 23:58:17 UTC
[GitHub] [airflow] tooptoop4 opened a new pull request #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
tooptoop4 opened a new pull request #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
URL: https://github.com/apache/airflow/pull/6943
…_date
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes how to use it.
- All the public functions and the classes in the PR contain docstrings that explain what it does
- If you implement backwards incompatible changes, please leave a note in the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so we can assign it to a appropriate release
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] tooptoop4 closed pull request #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
Posted by GitBox <gi...@apache.org>.
tooptoop4 closed pull request #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
URL: https://github.com/apache/airflow/pull/6943
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
URL: https://github.com/apache/airflow/pull/6943#issuecomment-569519785
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=h1) Report
> Merging [#6943](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/7d97d16c47ba181839fd583b8653d2872dc4290b?src=pr&el=desc) will **decrease** coverage by `0.28%`.
> The diff coverage is `50%`.
[](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6943 +/- ##
==========================================
- Coverage 84.7% 84.41% -0.29%
==========================================
Files 680 680
Lines 38556 38559 +3
==========================================
- Hits 32659 32550 -109
- Misses 5897 6009 +112
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.14% <0%> (-0.14%)` | :arrow_down: |
| [airflow/ti\_deps/deps/runnable\_exec\_date\_dep.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcnVubmFibGVfZXhlY19kYXRlX2RlcC5weQ==) | `100% <100%> (ø)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
| [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=footer). Last update [7d97d16...0cb644b](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
URL: https://github.com/apache/airflow/pull/6943#issuecomment-569519785
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=h1) Report
> Merging [#6943](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/7d97d16c47ba181839fd583b8653d2872dc4290b?src=pr&el=desc) will **decrease** coverage by `0.28%`.
> The diff coverage is `50%`.
[](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6943 +/- ##
==========================================
- Coverage 84.7% 84.41% -0.29%
==========================================
Files 680 680
Lines 38556 38559 +3
==========================================
- Hits 32659 32550 -109
- Misses 5897 6009 +112
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.14% <0%> (-0.14%)` | :arrow_down: |
| [airflow/ti\_deps/deps/runnable\_exec\_date\_dep.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcnVubmFibGVfZXhlY19kYXRlX2RlcC5weQ==) | `100% <100%> (ø)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
| [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=footer). Last update [7d97d16...0cb644b](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
URL: https://github.com/apache/airflow/pull/6943#discussion_r362188175
##########
File path: airflow/config_templates/default_airflow.cfg
##########
@@ -581,6 +581,10 @@ authenticate = False
# DAGs submitted manually in the web UI or with trigger_dag will still run.
use_job_schedule = True
+# Allow externally triggered DagRuns for Execution Dates in the future when True
Review comment:
Can you add a short documentation describing this parameter? I am not sure about the place, but I think you can create a new document in the `howto` directory.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
URL: https://github.com/apache/airflow/pull/6943#issuecomment-569519785
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=h1) Report
> Merging [#6943](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/947fb64f7f16dc65a24f8d337ed1adc0fb28e464?src=pr&el=desc) will **decrease** coverage by `0.3%`.
> The diff coverage is `50%`.
[](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6943 +/- ##
==========================================
- Coverage 84.72% 84.41% -0.31%
==========================================
Files 679 680 +1
Lines 38505 38559 +54
==========================================
- Hits 32623 32550 -73
- Misses 5882 6009 +127
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.14% <0%> (-0.12%)` | :arrow_down: |
| [airflow/ti\_deps/deps/runnable\_exec\_date\_dep.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcnVubmFibGVfZXhlY19kYXRlX2RlcC5weQ==) | `100% <100%> (ø)` | :arrow_up: |
| [airflow/contrib/hooks/cassandra\_hook.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2Nhc3NhbmRyYV9ob29rLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
| [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `91.3% <0%> (-8.7%)` | :arrow_down: |
| [airflow/utils/cli.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9jbGkucHk=) | `71.69% <0%> (-3.04%)` | :arrow_down: |
| ... and [19 more](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=footer). Last update [947fb64...0cb644b](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] tooptoop4 commented on a change in pull request #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
Posted by GitBox <gi...@apache.org>.
tooptoop4 commented on a change in pull request #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
URL: https://github.com/apache/airflow/pull/6943#discussion_r363018119
##########
File path: airflow/jobs/scheduler_job.py
##########
@@ -629,13 +629,21 @@ def _process_task_instances(self, dag, task_instances_list, session=None):
active_dag_runs = []
for run in dag_runs:
self.log.info("Examining DAG run %s", run)
- # don't consider runs that are executed in the future
+ # don't consider runs that are executed in the future unless
+ # specified by config
if run.execution_date > timezone.utcnow():
- self.log.error(
- "Execution date is in future: %s",
- run.execution_date
- )
- continue
+ if conf.getboolean(
Review comment:
@yuqian90 see https://github.com/apache/airflow/pull/7038
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
URL: https://github.com/apache/airflow/pull/6943#issuecomment-569519785
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=h1) Report
> Merging [#6943](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/7d97d16c47ba181839fd583b8653d2872dc4290b?src=pr&el=desc) will **decrease** coverage by `0.28%`.
> The diff coverage is `50%`.
[](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6943 +/- ##
==========================================
- Coverage 84.7% 84.41% -0.29%
==========================================
Files 680 680
Lines 38556 38559 +3
==========================================
- Hits 32659 32550 -109
- Misses 5897 6009 +112
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.14% <0%> (-0.14%)` | :arrow_down: |
| [airflow/ti\_deps/deps/runnable\_exec\_date\_dep.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcnVubmFibGVfZXhlY19kYXRlX2RlcC5weQ==) | `100% <100%> (ø)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
| [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=footer). Last update [7d97d16...0cb644b](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] yuqian90 commented on a change in pull request #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
Posted by GitBox <gi...@apache.org>.
yuqian90 commented on a change in pull request #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
URL: https://github.com/apache/airflow/pull/6943#discussion_r362488927
##########
File path: airflow/config_templates/default_airflow.cfg
##########
@@ -581,6 +581,10 @@ authenticate = False
# DAGs submitted manually in the web UI or with trigger_dag will still run.
use_job_schedule = True
+# Allow externally triggered DagRuns for Execution Dates in the future when True
+# Only has effect if: use_job_schedule = False
+run_future_exec_dates = False
Review comment:
If you need to prevent DAGs with schedule_interval being cron expression from being run too early, why not check if the dag.schedule_interval is None? (instead of forcing `use_job_schedule` to be False globally) ?
That way you can allow both cron expression and externally triggered DAGs to co-exist.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
URL: https://github.com/apache/airflow/pull/6943#issuecomment-569519785
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=h1) Report
> Merging [#6943](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/947fb64f7f16dc65a24f8d337ed1adc0fb28e464?src=pr&el=desc) will **decrease** coverage by `0.3%`.
> The diff coverage is `50%`.
[](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6943 +/- ##
==========================================
- Coverage 84.72% 84.41% -0.31%
==========================================
Files 679 680 +1
Lines 38505 38559 +54
==========================================
- Hits 32623 32550 -73
- Misses 5882 6009 +127
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.14% <0%> (-0.12%)` | :arrow_down: |
| [airflow/ti\_deps/deps/runnable\_exec\_date\_dep.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcnVubmFibGVfZXhlY19kYXRlX2RlcC5weQ==) | `100% <100%> (ø)` | :arrow_up: |
| [airflow/contrib/hooks/cassandra\_hook.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2Nhc3NhbmRyYV9ob29rLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
| [airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=) | `91.3% <0%> (-8.7%)` | :arrow_down: |
| [airflow/utils/cli.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9jbGkucHk=) | `71.69% <0%> (-3.04%)` | :arrow_down: |
| ... and [19 more](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=footer). Last update [947fb64...0cb644b](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] tooptoop4 commented on a change in pull request #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
Posted by GitBox <gi...@apache.org>.
tooptoop4 commented on a change in pull request #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
URL: https://github.com/apache/airflow/pull/6943#discussion_r362387395
##########
File path: airflow/config_templates/default_airflow.cfg
##########
@@ -581,6 +581,10 @@ authenticate = False
# DAGs submitted manually in the web UI or with trigger_dag will still run.
use_job_schedule = True
+# Allow externally triggered DagRuns for Execution Dates in the future when True
+# Only has effect if: use_job_schedule = False
+run_future_exec_dates = False
Review comment:
necessary otherwise cron sched tasks would run too early
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] tooptoop4 commented on a change in pull request #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
Posted by GitBox <gi...@apache.org>.
tooptoop4 commented on a change in pull request #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
URL: https://github.com/apache/airflow/pull/6943#discussion_r362627918
##########
File path: airflow/jobs/scheduler_job.py
##########
@@ -629,13 +629,21 @@ def _process_task_instances(self, dag, task_instances_list, session=None):
active_dag_runs = []
for run in dag_runs:
self.log.info("Examining DAG run %s", run)
- # don't consider runs that are executed in the future
+ # don't consider runs that are executed in the future unless
+ # specified by config
if run.execution_date > timezone.utcnow():
- self.log.error(
- "Execution date is in future: %s",
- run.execution_date
- )
- continue
+ if conf.getboolean(
Review comment:
great!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] yuqian90 commented on a change in pull request #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
Posted by GitBox <gi...@apache.org>.
yuqian90 commented on a change in pull request #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
URL: https://github.com/apache/airflow/pull/6943#discussion_r362490871
##########
File path: airflow/jobs/scheduler_job.py
##########
@@ -629,13 +629,21 @@ def _process_task_instances(self, dag, task_instances_list, session=None):
active_dag_runs = []
for run in dag_runs:
self.log.info("Examining DAG run %s", run)
- # don't consider runs that are executed in the future
+ # don't consider runs that are executed in the future unless
+ # specified by config
if run.execution_date > timezone.utcnow():
- self.log.error(
- "Execution date is in future: %s",
- run.execution_date
- )
- continue
+ if conf.getboolean(
Review comment:
This is regarding my previous comment about requiring `use_job_schedule` to be `False`. Does something like this work?
```python
self.log.info("Examining DAG run %s", run)
# don't consider runs that are executed in the future unless
# schedule_interval is None
if run.execution_date > timezone.utcnow():
if dag.schedule_interval or not conf.getboolean(
'scheduler',
'RUN_FUTURE_EXEC_DATES',
fallback=False):
self.log.error(
"Execution date is in future: %s",
run.execution_date
)
continue
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] yuqian90 commented on a change in pull request #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
Posted by GitBox <gi...@apache.org>.
yuqian90 commented on a change in pull request #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
URL: https://github.com/apache/airflow/pull/6943#discussion_r362354691
##########
File path: airflow/config_templates/default_airflow.cfg
##########
@@ -581,6 +581,10 @@ authenticate = False
# DAGs submitted manually in the web UI or with trigger_dag will still run.
use_job_schedule = True
+# Allow externally triggered DagRuns for Execution Dates in the future when True
+# Only has effect if: use_job_schedule = False
+run_future_exec_dates = False
Review comment:
Why is this necessary? Does it mean if a user wants to allow tasks on DagRuns with execution_date in the future to be executed, they have to turn off `use_job_schedule` globally?
>Only has effect if: use_job_schedule = False
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] tooptoop4 commented on issue #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
Posted by GitBox <gi...@apache.org>.
tooptoop4 commented on issue #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
URL: https://github.com/apache/airflow/pull/6943#issuecomment-570759385
superseded by https://github.com/apache/airflow/pull/7038
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6943: [AIRFLOW-4495] allow externally triggered dags to run for future exec…
URL: https://github.com/apache/airflow/pull/6943#issuecomment-569519785
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=h1) Report
> Merging [#6943](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/7d97d16c47ba181839fd583b8653d2872dc4290b?src=pr&el=desc) will **decrease** coverage by `0.28%`.
> The diff coverage is `50%`.
[](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6943 +/- ##
==========================================
- Coverage 84.7% 84.41% -0.29%
==========================================
Files 680 680
Lines 38556 38559 +3
==========================================
- Hits 32659 32550 -109
- Misses 5897 6009 +112
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.14% <0%> (-0.14%)` | :arrow_down: |
| [airflow/ti\_deps/deps/runnable\_exec\_date\_dep.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcnVubmFibGVfZXhlY19kYXRlX2RlcC5weQ==) | `100% <100%> (ø)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
| [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6943/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=footer). Last update [7d97d16...0cb644b](https://codecov.io/gh/apache/airflow/pull/6943?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services