(airflow) branch main updated: Adds new `triggerer.capacity_left[.]` metric (#41323)

[po...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
     new e46fadc256 Adds new `triggerer.capacity_left[.<hostname>]` metric (#41323)
e46fadc256 is described below

commit e46fadc25679624a15d6775067d3cd2b513742d8
Author: Usiel Riedl <us...@automattic.com>
AuthorDate: Wed Oct 2 09:41:11 2024 +0800

    Adds new `triggerer.capacity_left[.<hostname>]` metric (#41323)
    
    After reducing the default capacity our deployment, it is rather close to the total capacity at certain times, hence it would be useful to be able to create monitoring and alerting based on the left capacity.
    
    The new metric will enable better alerting (not relying on hardcoded capacity values) and even auto-scaling if wished for.
---
 airflow/jobs/triggerer_job_runner.py                                | 6 ++++++
 .../administration-and-deployment/logging-monitoring/metrics.rst    | 3 +++
 2 files changed, 9 insertions(+)

diff --git a/airflow/jobs/triggerer_job_runner.py b/airflow/jobs/triggerer_job_runner.py
index b41af29f37..defde4a164 100644
--- a/airflow/jobs/triggerer_job_runner.py
+++ b/airflow/jobs/triggerer_job_runner.py
@@ -430,9 +430,15 @@ class TriggererJobRunner(BaseJobRunner, LoggingMixin):
         Stats.gauge(
             "triggers.running", len(self.trigger_runner.triggers), tags={"hostname": self.job.hostname}
         )
+
+        capacity_left = self.capacity - len(self.trigger_runner.triggers)
+        Stats.gauge(f"triggerer.capacity_left.{self.job.hostname}", capacity_left)
+        Stats.gauge("triggerer.capacity_left", capacity_left, tags={"hostname": self.job.hostname})
+
         span = Trace.get_current_span()
         span.set_attribute("trigger host", self.job.hostname)
         span.set_attribute("triggers running", len(self.trigger_runner.triggers))
+        span.set_attribute("capacity left", capacity_left)
 
 
 class TriggerDetails(TypedDict):
diff --git a/docs/apache-airflow/administration-and-deployment/logging-monitoring/metrics.rst b/docs/apache-airflow/administration-and-deployment/logging-monitoring/metrics.rst
index 079aa5d397..7ce9b9b765 100644
--- a/docs/apache-airflow/administration-and-deployment/logging-monitoring/metrics.rst
+++ b/docs/apache-airflow/administration-and-deployment/logging-monitoring/metrics.rst
@@ -248,6 +248,9 @@ Name                                                 Description
 ``triggers.running.<hostname>``                      Number of triggers currently running for a triggerer (described by hostname)
 ``triggers.running``                                 Number of triggers currently running for a triggerer (described by hostname).
                                                      Metric with hostname tagging.
+``triggerer.capacity_left.<hostname>``               Capacity left on a triggerer to run triggers (described by hostname)
+``triggerer.capacity_left``                          Capacity left on a triggerer to run triggers (described by hostname).
+                                                     Metric with hostname tagging.
 ==================================================== ========================================================================
 
 Timers