You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/21 20:46:25 UTC
[GitHub] [airflow] ashb opened a new pull request #11730: Stop scheduler from thinking that upstream_failed tasks are running
ashb opened a new pull request #11730:
URL: https://github.com/apache/airflow/pull/11730
This was messing up the "max_active_runs" calculation, and this fix is a
"hack" until we add a better approach of adding a queued state to
DagRuns -- at which point we don't even have to do this calculation at
all.
Closes #11633
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #11730: Stop scheduler from thinking that upstream_failed tasks are running
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #11730:
URL: https://github.com/apache/airflow/pull/11730#discussion_r509684070
##########
File path: airflow/utils/state.py
##########
@@ -114,6 +114,16 @@ def color_fg(cls, state):
have been interrupted; in any case, it is no longer running.
"""
+ terminal = finished | {
Review comment:
Hmmm, I'm not sure I actually need a new grouping here. Given SKIPPED was in the previous list, perhaps we just need to update the doc string on the previous one and add it there
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #11730: Stop scheduler from thinking that upstream_failed tasks are running
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #11730:
URL: https://github.com/apache/airflow/pull/11730#issuecomment-713867721
/cc @mjpieters
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11730: Stop scheduler from thinking that upstream_failed tasks are running
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11730:
URL: https://github.com/apache/airflow/pull/11730#issuecomment-713922544
[The Workflow run](https://github.com/apache/airflow/actions/runs/320872220) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11730: Stop scheduler from thinking that upstream_failed tasks are running
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11730:
URL: https://github.com/apache/airflow/pull/11730#issuecomment-713899288
[The Workflow run](https://github.com/apache/airflow/actions/runs/320797006) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #11730: Stop scheduler from thinking that upstream_failed tasks are running
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #11730:
URL: https://github.com/apache/airflow/pull/11730#issuecomment-713883409
@mik-laj I changed it just after you reviewed, so wanted to see if you're still happy.
Instead of creating a new `State.terminal` set, I adjusted the existing `State.finished` set (values and updated docs to reflect) after checking all the places in the code where that set was used.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #11730: Stop scheduler from thinking that upstream_failed tasks are running
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #11730:
URL: https://github.com/apache/airflow/pull/11730#discussion_r509700963
##########
File path: airflow/utils/state.py
##########
@@ -114,6 +114,16 @@ def color_fg(cls, state):
have been interrupted; in any case, it is no longer running.
"""
+ terminal = finished | {
Review comment:
Yeah, looking at where it's used (all 8 places!) it's better to just add to this list.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11730: Stop scheduler from thinking that upstream_failed tasks are running
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11730:
URL: https://github.com/apache/airflow/pull/11730#issuecomment-713890339
[The Workflow run](https://github.com/apache/airflow/actions/runs/320768388) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #11730: Stop scheduler from thinking that upstream_failed tasks are running
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #11730:
URL: https://github.com/apache/airflow/pull/11730#discussion_r509682544
##########
File path: airflow/models/dagrun.py
##########
@@ -387,7 +387,7 @@ def update_state(
ti.task = dag.get_task(ti.task_id)
unfinished_tasks = [t for t in tis if t.state in State.unfinished]
- finished_tasks = [t for t in tis if t.state in State.finished | {State.UPSTREAM_FAILED}]
+ finished_tasks = [t for t in tis if t.state in State.terminal]
Review comment:
This was just "tidy up" to use the new property I added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb merged pull request #11730: Stop scheduler from thinking that upstream_failed tasks are running
Posted by GitBox <gi...@apache.org>.
ashb merged pull request #11730:
URL: https://github.com/apache/airflow/pull/11730
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #11730: Stop scheduler from thinking that upstream_failed tasks are running
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #11730:
URL: https://github.com/apache/airflow/pull/11730#discussion_r509682312
##########
File path: airflow/jobs/scheduler_job.py
##########
@@ -1473,7 +1473,7 @@ def _do_scheduling(self, session) -> int:
func.count(TI.execution_date.distinct()),
).filter(
TI.dag_id.in_(list({dag_run.dag_id for dag_run in dag_runs})),
- TI.state.notin_(list(State.finished))
+ TI.state.notin_(list(State.terminal))
Review comment:
This was the fix
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org