You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ep...@apache.org on 2023/04/27 12:23:35 UTC
[airflow] branch v2-6-test updated (37026566b9 -> 3848bfc61f)
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a change to branch v2-6-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
from 37026566b9 Update Release Note
new 568ca2b4c7 Fix dag file processor hertbeat to run only if necessary (#30899)
new 3848bfc61f fixup! Update Release Note
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
RELEASE_NOTES.rst | 1 +
airflow/jobs/dag_processor_job_runner.py | 2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
[airflow] 01/02: Fix dag file processor hertbeat to run only if necessary (#30899)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-6-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 568ca2b4c70af430d81ee725ae3bb3681f99b954
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Thu Apr 27 13:27:33 2023 +0200
Fix dag file processor hertbeat to run only if necessary (#30899)
The standalone file processor as of #30278 introduced accidentally
an artifficial delay between dag processing by adding heartbeat
but missing to set "only_if_necessary" flag to True.
If your dag file processing has been fast (faster than the
scheduler job_heartbeat_sec) this introduced unnecessary pause
between the next dag file processor loop (up until the time
passed), it also introduced inflation of the
dag_processing_last_duration metrics (it would always show minimum
job_heartbeat_sec)
Adding "only_if_necessary" flag fixes the problem.
Fixes: #30593
Fixes: #30884
(cherry picked from commit 00ab45ffb7dee92030782f0d1496d95b593fd4a7)
---
airflow/jobs/dag_processor_job_runner.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/airflow/jobs/dag_processor_job_runner.py b/airflow/jobs/dag_processor_job_runner.py
index 0fda9d9373..afc8f544da 100644
--- a/airflow/jobs/dag_processor_job_runner.py
+++ b/airflow/jobs/dag_processor_job_runner.py
@@ -57,7 +57,7 @@ class DagProcessorJobRunner(BaseJobRunner, LoggingMixin):
self.processor.heartbeat = lambda: perform_heartbeat(
job=self.job,
heartbeat_callback=empty_callback,
- only_if_necessary=False,
+ only_if_necessary=True,
)
def _execute(self) -> int | None:
[airflow] 02/02: fixup! Update Release Note
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-6-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 3848bfc61f6733fe26408b6cb285ee35b95efca5
Author: Ephraim Anierobi <sp...@gmail.com>
AuthorDate: Thu Apr 27 13:23:08 2023 +0100
fixup! Update Release Note
---
RELEASE_NOTES.rst | 1 +
1 file changed, 1 insertion(+)
diff --git a/RELEASE_NOTES.rst b/RELEASE_NOTES.rst
index 9b73f52964..bbb23664b3 100644
--- a/RELEASE_NOTES.rst
+++ b/RELEASE_NOTES.rst
@@ -199,6 +199,7 @@ Improvements
Bug Fixes
"""""""""
+- Fix dag file processor hertbeat to run only if necessary (#30899)
- Fix KubernetesExecutor sending state to scheduler (#30872)
- Count mapped upstream only if all are finished (#30641)
- ExternalTaskSensor: add external_task_group_id to template_fields (#30401)