You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ep...@apache.org on 2024/01/14 11:01:34 UTC
(airflow) branch v2-8-test updated (0538ef3edb -> 62ad354b76)
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a change to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
from 0538ef3edb Simplify how mypy "folder" checks are run (#36760)
new d6598aad99 metrics tagging documentation (#36627)
new 005292eefc Add log lookup exception for empty op subtypes (#35536)
new 1a09aa232f Restore function scoped httpx import in file_task_handler for perf (#36753)
new 62ad354b76 explicit string cast required to force integer-type run_ids to be passed as strings instead of integers (#36756)
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
airflow/operators/trigger_dagrun.py | 2 +-
airflow/utils/log/file_task_handler.py | 13 ++++-
.../logging-monitoring/metrics.rst | 62 +++++++++++++++++-----
3 files changed, 62 insertions(+), 15 deletions(-)
(airflow) 01/04: metrics tagging documentation (#36627)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d6598aad99787bcacdb2b963a3a77de5fd4b078f
Author: Gopal Dirisala <39...@users.noreply.github.com>
AuthorDate: Fri Jan 12 07:14:05 2024 +0530
metrics tagging documentation (#36627)
* metrics tagging documentation
(cherry picked from commit 667b842632fbee984b940a3b8b5a1f0bb3749a0f)
---
.../logging-monitoring/metrics.rst | 62 +++++++++++++++++-----
1 file changed, 50 insertions(+), 12 deletions(-)
diff --git a/docs/apache-airflow/administration-and-deployment/logging-monitoring/metrics.rst b/docs/apache-airflow/administration-and-deployment/logging-monitoring/metrics.rst
index a973f4e69e..ebe8b00c95 100644
--- a/docs/apache-airflow/administration-and-deployment/logging-monitoring/metrics.rst
+++ b/docs/apache-airflow/administration-and-deployment/logging-monitoring/metrics.rst
@@ -147,41 +147,59 @@ Name Descripti
``LocalTaskJob``
``local_task_job.task_exit.<job_id>.<dag_id>.<task_id>.<return_code>`` Number of ``LocalTaskJob`` terminations with a ``<return_code>``
while running a task ``<task_id>`` of a DAG ``<dag_id>``.
+``local_task_job.task_exit`` Number of ``LocalTaskJob`` terminations with a ``<return_code>``
+ while running a task ``<task_id>`` of a DAG ``<dag_id>``.
+ Metric with job_id, dag_id, task_id and return_code tagging.
``operator_failures_<operator_name>`` Operator ``<operator_name>`` failures
+``operator_failures`` Operator ``<operator_name>`` failures. Metric with operator_name tagging.
``operator_successes_<operator_name>`` Operator ``<operator_name>`` successes
-``ti_failures`` Overall task instances failures
-``ti_successes`` Overall task instances successes
-``previously_succeeded`` Number of previously succeeded task instances
-``zombies_killed`` Zombie tasks killed
+``operator_successes`` Operator ``<operator_name>`` successes. Metric with operator_name tagging.
+``ti_failures`` Overall task instances failures. Metric with dag_id and task_id tagging.
+``ti_successes`` Overall task instances successes. Metric with dag_id and task_id tagging.
+``previously_succeeded`` Number of previously succeeded task instances. Metric with dag_id and task_id tagging.
+``zombies_killed`` Zombie tasks killed. Metric with dag_id and task_id tagging.
``scheduler_heartbeat`` Scheduler heartbeats
``dag_processing.processes`` Relative number of currently running DAG parsing processes (ie this delta
- is negative when, since the last metric was sent, processes have completed)
-``dag_processing.processor_timeouts`` Number of file processors that have been killed due to taking too long
+ is negative when, since the last metric was sent, processes have completed).
+ Metric with file_path and action tagging.
+``dag_processing.processor_timeouts`` Number of file processors that have been killed due to taking too long.
+ Metric with file_path tagging.
``dag_processing.sla_callback_count`` Number of SLA callbacks received
``dag_processing.other_callback_count`` Number of non-SLA callbacks received
``dag_processing.file_path_queue_update_count`` Number of times we've scanned the filesystem and queued all existing dags
``dag_file_processor_timeouts`` (DEPRECATED) same behavior as ``dag_processing.processor_timeouts``
``dag_processing.manager_stalls`` Number of stalled ``DagFileProcessorManager``
``dag_file_refresh_error`` Number of failures loading any DAG files
-``scheduler.tasks.killed_externally`` Number of tasks killed externally
+``scheduler.tasks.killed_externally`` Number of tasks killed externally. Metric with dag_id and task_id tagging.
``scheduler.orphaned_tasks.cleared`` Number of Orphaned tasks cleared by the Scheduler
``scheduler.orphaned_tasks.adopted`` Number of Orphaned tasks adopted by the Scheduler
``scheduler.critical_section_busy`` Count of times a scheduler process tried to get a lock on the critical
section (needed to send tasks to the executor) and found it locked by
another process.
-``sla_missed`` Number of SLA misses
-``sla_callback_notification_failure`` Number of failed SLA miss callback notification attempts
-``sla_email_notification_failure`` Number of failed SLA miss email notification attempts
+``sla_missed`` Number of SLA misses. Metric with dag_id and task_id tagging.
+``sla_callback_notification_failure`` Number of failed SLA miss callback notification attempts. Metric with dag_id and func_name tagging.
+``sla_email_notification_failure`` Number of failed SLA miss email notification attempts. Metric with dag_id tagging.
``ti.start.<dag_id>.<task_id>`` Number of started task in a given dag. Similar to <job_name>_start but for task
+``ti.start`` Number of started task in a given dag. Similar to <job_name>_start but for task.
+ Metric with dag_id and task_id tagging.
``ti.finish.<dag_id>.<task_id>.<state>`` Number of completed task in a given dag. Similar to <job_name>_end but for task
+``ti.finish`` Number of completed task in a given dag. Similar to <job_name>_end but for task
+ Metric with dag_id and task_id tagging.
``dag.callback_exceptions`` Number of exceptions raised from DAG callbacks. When this happens, it
- means DAG callback is not working.
+ means DAG callback is not working. Metric with dag_id tagging
``celery.task_timeout_error`` Number of ``AirflowTaskTimeout`` errors raised when publishing Task to Celery Broker.
``celery.execute_command.failure`` Number of non-zero exit code from Celery task.
-``task_removed_from_dag.<dag_id>`` Number of tasks removed for a given dag (i.e. task no longer exists in DAG)
+``task_removed_from_dag.<dag_id>`` Number of tasks removed for a given dag (i.e. task no longer exists in DAG).
+``task_removed_from_dag`` Number of tasks removed for a given dag (i.e. task no longer exists in DAG).
+ Metric with dag_id and run_type tagging.
``task_restored_to_dag.<dag_id>`` Number of tasks restored for a given dag (i.e. task instance which was
previously in REMOVED state in the DB is added to DAG file)
+``task_restored_to_dag.<dag_id>`` Number of tasks restored for a given dag (i.e. task instance which was
+ previously in REMOVED state in the DB is added to DAG file).
+ Metric with dag_id and run_type tagging.
``task_instance_created_<operator_name>`` Number of tasks instances created for a given Operator
+``task_instance_created`` Number of tasks instances created for a given Operator.
+ Metric with dag_id and run_type tagging.
``triggerer_heartbeat`` Triggerer heartbeats
``triggers.blocked_main_thread`` Number of triggers that blocked the main thread (likely due to not being
fully asynchronous)
@@ -213,11 +231,18 @@ Name Description
``executor.queued_tasks`` Number of queued tasks on executor
``executor.running_tasks`` Number of running tasks on executor
``pool.open_slots.<pool_name>`` Number of open slots in the pool
+``pool.open_slots`` Number of open slots in the pool. Metric with pool_name tagging.
``pool.queued_slots.<pool_name>`` Number of queued slots in the pool
+``pool.queued_slots`` Number of queued slots in the pool. Metric with pool_name tagging.
``pool.running_slots.<pool_name>`` Number of running slots in the pool
+``pool.running_slots`` Number of running slots in the pool. Metric with pool_name tagging.
``pool.deferred_slots.<pool_name>`` Number of deferred slots in the pool
+``pool.deferred_slots`` Number of deferred slots in the pool. Metric with pool_name tagging.
``pool.starving_tasks.<pool_name>`` Number of starving tasks in the pool
+``pool.starving_tasks`` Number of starving tasks in the pool. Metric with pool_name tagging.
``triggers.running.<hostname>`` Number of triggers currently running for a triggerer (described by hostname)
+``triggers.running`` Number of triggers currently running for a triggerer (described by hostname).
+ Metric with hostname tagging.
=================================================== ========================================================================
Timers
@@ -229,17 +254,30 @@ Name Description
``dagrun.dependency-check.<dag_id>`` Milliseconds taken to check DAG dependencies
``dag.<dag_id>.<task_id>.duration`` Seconds taken to run a task
``dag.<dag_id>.<task_id>.scheduled_duration`` Seconds a task spends in the Scheduled state, before being Queued
+``task.scheduled_duration`` Seconds a task spends in the Scheduled state, before being Queued.
+ Metric with dag_id and task_id tagging.
``dag.<dag_id>.<task_id>.queued_duration`` Seconds a task spends in the Queued state, before being Running
+``task.queued_duration`` Seconds a task spends in the Queued state, before being Running.
+ Metric with dag_id and task_id tagging.
``dag_processing.last_duration.<dag_file>`` Seconds taken to load the given DAG file
+``dag_processing.last_duration`` Seconds taken to load the given DAG file. Metric with file_name tagging.
``dagrun.duration.success.<dag_id>`` Seconds taken for a DagRun to reach success state
+``dagrun.duration.success`` Seconds taken for a DagRun to reach success state.
+ Metric with dag_id and run_type tagging.
``dagrun.duration.failed.<dag_id>`` Seconds taken for a DagRun to reach failed state
+``dagrun.duration.failed`` Seconds taken for a DagRun to reach failed state.
+ Metric with dag_id and run_type tagging.
``dagrun.schedule_delay.<dag_id>`` Milliseconds of delay between the scheduled DagRun
start date and the actual DagRun start date
+``dagrun.schedule_delay`` Milliseconds of delay between the scheduled DagRun
+ start date and the actual DagRun start date. Metric with dag_id tagging.
``scheduler.critical_section_duration`` Milliseconds spent in the critical section of scheduler loop --
only a single scheduler can enter this loop at a time
``scheduler.critical_section_query_duration`` Milliseconds spent running the critical section task instance query
``scheduler.scheduler_loop_duration`` Milliseconds spent running one scheduler loop
``dagrun.<dag_id>.first_task_scheduling_delay`` Seconds elapsed between first task start_date and dagrun expected start
+``dagrun.first_task_scheduling_delay`` Seconds elapsed between first task start_date and dagrun expected start.
+ Metric with dag_id and run_type tagging.
``collect_db_dags`` Milliseconds taken for fetching all Serialized Dags from DB
``kubernetes_executor.clear_not_launched_queued_tasks.duration`` Milliseconds taken for clearing not launched queued tasks in Kubernetes Executor
``kubernetes_executor.adopt_task_instances.duration`` Milliseconds taken to adopt the task instances in Kubernetes Executor
(airflow) 02/04: Add log lookup exception for empty op subtypes (#35536)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 005292eefce8f03d8b0b0a1c637ba8ca96d19601
Author: Victor Chiapaikeo <vc...@gmail.com>
AuthorDate: Fri Jan 12 02:56:54 2024 -0500
Add log lookup exception for empty op subtypes (#35536)
* Add log lookup exception for empty op subtypes
* Use exception catching approach instead to preserve tests
(cherry picked from commit ddcaef45593a5411859327ab2d16ed648073b986)
---
airflow/utils/log/file_task_handler.py | 13 +++++++++----
1 file changed, 9 insertions(+), 4 deletions(-)
diff --git a/airflow/utils/log/file_task_handler.py b/airflow/utils/log/file_task_handler.py
index 3e2561ba75..f86cc77736 100644
--- a/airflow/utils/log/file_task_handler.py
+++ b/airflow/utils/log/file_task_handler.py
@@ -29,6 +29,7 @@ from pathlib import Path
from typing import TYPE_CHECKING, Any, Callable, Iterable
from urllib.parse import urljoin
+import httpx
import pendulum
from airflow.configuration import conf
@@ -78,8 +79,6 @@ def _set_task_deferred_context_var():
def _fetch_logs_from_service(url, log_relative_path):
- import httpx
-
from airflow.utils.jwt_signer import JWTSigner
timeout = conf.getint("webserver", "log_fetch_timeout_sec", fallback=None)
@@ -170,6 +169,9 @@ class FileTaskHandler(logging.Handler):
"""
trigger_should_wrap = True
+ inherits_from_empty_operator_log_message = (
+ "Operator inherits from empty operator and thus does not have logs"
+ )
def __init__(self, base_log_folder: str, filename_template: str | None = None):
super().__init__()
@@ -555,8 +557,11 @@ class FileTaskHandler(logging.Handler):
messages.append(f"Found logs served from host {url}")
logs.append(response.text)
except Exception as e:
- messages.append(f"Could not read served logs: {e}")
- logger.exception("Could not read served logs")
+ if isinstance(e, httpx.UnsupportedProtocol) and ti.task.inherits_from_empty_operator is True:
+ messages.append(self.inherits_from_empty_operator_log_message)
+ else:
+ messages.append(f"Could not read served logs: {e}")
+ logger.exception("Could not read served logs")
return messages, logs
def _read_remote_logs(self, ti, try_number, metadata=None) -> tuple[list[str], list[str]]:
(airflow) 04/04: explicit string cast required to force integer-type run_ids to be passed as strings instead of integers (#36756)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 62ad354b76d7aceafa3e9534b768db3a2eed04b2
Author: Evgeny <ev...@users.noreply.github.com>
AuthorDate: Sat Jan 13 11:19:39 2024 -0500
explicit string cast required to force integer-type run_ids to be passed as strings instead of integers (#36756)
(cherry picked from commit e2335a00cea898d83e17b3eb69959656daae883e)
---
airflow/operators/trigger_dagrun.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/airflow/operators/trigger_dagrun.py b/airflow/operators/trigger_dagrun.py
index 0bca11a801..ab74d4c862 100644
--- a/airflow/operators/trigger_dagrun.py
+++ b/airflow/operators/trigger_dagrun.py
@@ -157,7 +157,7 @@ class TriggerDagRunOperator(BaseOperator):
raise AirflowException("conf parameter should be JSON Serializable")
if self.trigger_run_id:
- run_id = self.trigger_run_id
+ run_id = str(self.trigger_run_id)
else:
run_id = DagRun.generate_run_id(DagRunType.MANUAL, parsed_execution_date)
(airflow) 03/04: Restore function scoped httpx import in file_task_handler for perf (#36753)
Posted by ep...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch v2-8-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 1a09aa232f01c0b4a4857d72be25727ba05d754f
Author: Victor Chiapaikeo <vc...@gmail.com>
AuthorDate: Fri Jan 12 09:53:55 2024 -0500
Restore function scoped httpx import in file_task_handler for perf (#36753)
(cherry picked from commit c792b259699550d06054984f1643bb62df5ea37d)
---
airflow/utils/log/file_task_handler.py | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/airflow/utils/log/file_task_handler.py b/airflow/utils/log/file_task_handler.py
index f86cc77736..6151d0c15f 100644
--- a/airflow/utils/log/file_task_handler.py
+++ b/airflow/utils/log/file_task_handler.py
@@ -29,7 +29,6 @@ from pathlib import Path
from typing import TYPE_CHECKING, Any, Callable, Iterable
from urllib.parse import urljoin
-import httpx
import pendulum
from airflow.configuration import conf
@@ -79,6 +78,9 @@ def _set_task_deferred_context_var():
def _fetch_logs_from_service(url, log_relative_path):
+ # Import occurs in function scope for perf. Ref: https://github.com/apache/airflow/pull/21438
+ import httpx
+
from airflow.utils.jwt_signer import JWTSigner
timeout = conf.getint("webserver", "log_fetch_timeout_sec", fallback=None)
@@ -557,7 +559,9 @@ class FileTaskHandler(logging.Handler):
messages.append(f"Found logs served from host {url}")
logs.append(response.text)
except Exception as e:
- if isinstance(e, httpx.UnsupportedProtocol) and ti.task.inherits_from_empty_operator is True:
+ from httpx import UnsupportedProtocol
+
+ if isinstance(e, UnsupportedProtocol) and ti.task.inherits_from_empty_operator is True:
messages.append(self.inherits_from_empty_operator_log_message)
else:
messages.append(f"Could not read served logs: {e}")