You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/11/06 17:19:14 UTC
[GitHub] [airflow] ashb opened a new pull request #12139: Call scheduler "book-keeping" operations less frequently.
ashb opened a new pull request #12139:
URL: https://github.com/apache/airflow/pull/12139
This change makes it so that certain operations in the scheduler are
called on a regular interval, instead of only once at start up, or every
time around the loop:
- adopt_or_reset_orphaned_tasks (detecting SchedulerJobs that died) was
previously only called on start up.
- _clean_tis_without_dagrun was previously called every time around the
scheduling loop, but this isn't so needed to be done every time as
this is a relatively rare cleanup operation
- _emit_pool_metrics doesn't need to be called _every_ time around the
loop, once every 5 seconds is enough.
This uses the built in ["sched" module][sched] to handle the "timers".
[sched]: https://docs.python.org/3/library/sched.html
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #12139: Call scheduler "book-keeping" operations less frequently.
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #12139:
URL: https://github.com/apache/airflow/pull/12139#discussion_r518900321
##########
File path: airflow/config_templates/config.yml
##########
@@ -1492,6 +1492,14 @@
type: string
example: ~
default: "5"
+ - name: clean_tis_without_dagrun
Review comment:
Can you write a guides that describes these configuration options and allows us to familiarize ourselves with these options? The referee documentation is great because it lists all the options, but it's not good documentation for beginners.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #12139: Call scheduler "book-keeping" operations less frequently.
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #12139:
URL: https://github.com/apache/airflow/pull/12139#discussion_r519763121
##########
File path: airflow/config_templates/config.yml
##########
@@ -1492,6 +1492,14 @@
type: string
example: ~
default: "5"
+ - name: clean_tis_without_dagrun
Review comment:
Created https://github.com/apache/airflow/issues/12193 so we do this before 2.0.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb merged pull request #12139: Call scheduler "book-keeping" operations less frequently.
Posted by GitBox <gi...@apache.org>.
ashb merged pull request #12139:
URL: https://github.com/apache/airflow/pull/12139
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org