You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jedcunningham (via GitHub)" <gi...@apache.org> on 2025/02/18 19:58:51 UTC
[PR] Slow down DAG processor refreshing bundles [airflow]
jedcunningham opened a new pull request, #46870:
URL: https://github.com/apache/airflow/pull/46870
Right now we check if any bundles need to be refreshed every loop, which can be very frequent (e.g. 10 times per second or more). This introduces a new config, [dag_processor] bundle_refresh_check_interval, that allows for slowing down how often we do it. Since we send queries to the db, having some control here is helpful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Slow down DAG processor refreshing bundles [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on code in PR #46870:
URL: https://github.com/apache/airflow/pull/46870#discussion_r1962375563
##########
airflow/dag_processing/manager.py:
##########
@@ -447,6 +453,17 @@ def _refresh_dag_bundles(self, known_files: dict[str, set[DagFileInfo]]):
"""Refresh DAG bundles, if required."""
now = timezone.utcnow()
+ # we don't need to check if it's time to refresh every loop - that is way too often
+ last_checked_ago = time.monotonic() - self._bundles_last_refreshed
+ if last_checked_ago < self.bundle_refresh_check_interval:
+ self.log.debug(
+ "Not time to check if DAG Bundles need refreshed yet - skipping. Next check in %.2f seconds",
+ self.bundle_refresh_check_interval - last_checked_ago,
+ )
+ return
+
+ self._bundles_last_refreshed = time.monotonic()
Review Comment:
maybe simpler? take or leave
```suggestion
# we don't need to check if it's time to refresh every loop - that is way too often
next_check = self._bundles_last_refreshed + self.bundle_refresh_check_interval
now_ss = time.monotonic()
if now_ss < next_check:
self.log.debug(
"Not time to check if DAG Bundles need refreshed yet - skipping. "
"Next check in %.2f seconds",
next_check - now_ss,
)
return
self._bundles_last_refreshed = now_ss
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Slow down DAG processor refreshing bundles [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham merged PR #46870:
URL: https://github.com/apache/airflow/pull/46870
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org