You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ferruzzi (via GitHub)" <gi...@apache.org> on 2026/01/27 23:13:07 UTC
[PR] Executor Synchronous callback workload [airflow]
ferruzzi opened a new pull request, #61153:
URL: https://github.com/apache/airflow/pull/61153
Add support for the Callback workload to be run in the executors. Other executors will need to be updated before they can support the workload, but I tried to make it as non-invasive as I could.
This is the bulk of the work required to allow synchronous callbacks to be used in DeadlineAlerts. For example, this now works in LocalExecutor:
```python
with DAG(
dag_id="sync_deadline",
deadline=DeadlineAlert(
reference=DeadlineReference.FIXED_DATETIME(datetime(1980, 8, 10, 2)),
interval=timedelta(0),
callback=SyncCallback(
SlackWebhookNotifier,
{"text": "Sync Callback; Alert should trigger immediately!"},
)
)
):
EmptyOperator(task_id='empty_task')
```
Co-author: Builds on work handed off by @seanghaeli and research from @ramitkataria; if I did this right then they should be getting co-author credits, I think?
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [ x ] Yes (please specify the tool below)
Cline (Claude Sonnet 4.5) was used for debugging and suggesting some unit test edge cases.
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2738519644
##########
airflow-core/src/airflow/executors/base_executor.py:
##########
@@ -227,10 +229,46 @@ def log_task_event(self, *, event: str, extra: str, ti_key: TaskInstanceKey):
self._task_event_logs.append(Log(event=event, task_instance=ti_key, extra=extra))
def queue_workload(self, workload: workloads.All, session: Session) -> None:
- if not isinstance(workload, workloads.ExecuteTask):
+ if isinstance(workload, workloads.ExecuteTask):
+ ti = workload.ti
+ self.queued_tasks[ti.key] = workload
+ elif isinstance(workload, workloads.ExecuteCallback):
+ self.queued_callbacks[workload.callback.id] = workload
+ else:
raise ValueError(f"Un-handled workload kind {type(workload).__name__!r} in {type(self).__name__}")
- ti = workload.ti
- self.queued_tasks[ti.key] = workload
+
+ def _get_workloads_to_schedule(
+ self, open_slots: int
+ ) -> list[tuple[TaskInstanceKey | str, workloads.All]]:
+ """
+ Select and return the next batch of workloads to schedule, respecting priority policy.
+
+ Priority Policy: Callbacks are scheduled before tasks (callbacks complete existing work).
+ Callbacks are processed in FIFO order. Tasks are sorted by priority_weight (higher priority first).
+
+ :param open_slots: Number of available execution slots
+ """
+ workloads_to_schedule: list[tuple[TaskInstanceKey | str, workloads.All]] = []
+
+ if self.queued_callbacks:
+ for key, workload in self.queued_callbacks.items():
+ if len(workloads_to_schedule) >= open_slots:
+ break
+ workloads_to_schedule.append((key, workload))
+
+ remaining_slots = open_slots - len(workloads_to_schedule)
+ if remaining_slots and self.queued_tasks:
+ sorted_tasks = sorted(
Review Comment:
IMHO, the callbacks should be top priority before the new tasks, and shouldn't be folded in. I feel like we need to treat the callback as "finishing old work" before "accept new work". Given that approach, a new method seemed like the right call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2761352777
##########
providers/celery/src/airflow/providers/celery/executors/celery_executor.py:
##########
@@ -139,17 +140,31 @@ def _process_tasks(self, task_tuples: Sequence[TaskTuple]) -> None:
def _process_workloads(self, workloads: Sequence[workloads.All]) -> None:
# Airflow V3 version -- have to delay imports until we know we are on v3
- from airflow.executors.workloads import ExecuteTask
+ from airflow.executors.workloads import ExecuteCallback, ExecuteTask
Review Comment:
I'll make the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2842164641
##########
providers/edge3/src/airflow/providers/edge3/worker_api/v2-edge-generated.yaml:
##########
Review Comment:
Yeah, I definitely didn't touch that file manually. There's a chance maybe it slipped in when i had a messy merge conflict. I'll try to revert that change and re-run static checks, see if it's re-re-generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2867419959
##########
airflow-core/src/airflow/executors/workloads/callback.py:
##########
@@ -0,0 +1,158 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Callback workload schemas for executor communication."""
+
+from __future__ import annotations
+
+from enum import Enum
+from importlib import import_module
+from pathlib import Path
+from typing import TYPE_CHECKING, Literal
+from uuid import UUID
+
+import structlog
+from pydantic import BaseModel, Field, field_validator
+
+from airflow.executors.workloads.base import BaseDagBundleWorkload, BundleInfo
+
+if TYPE_CHECKING:
+ from airflow.api_fastapi.auth.tokens import JWTGenerator
+ from airflow.models import DagRun
+ from airflow.models.callback import Callback as CallbackModel, CallbackKey
+
+log = structlog.get_logger(__name__)
+
+
+class CallbackFetchMethod(str, Enum):
+ """Methods used to fetch callback at runtime."""
+
+ # For future use once Dag Processor callbacks (on_success_callback/on_failure_callback) get moved to executors
+ DAG_ATTRIBUTE = "dag_attribute"
+
+ # For deadline callbacks since they import callbacks through the import path
+ IMPORT_PATH = "import_path"
+
+
+class CallbackDTO(BaseModel):
+ """Schema for Callback with minimal required fields needed for Executors and Task SDK."""
+
+ id: str # A uuid.UUID stored as a string
+ fetch_method: CallbackFetchMethod
+ data: dict
+
+ @field_validator("id", mode="before")
+ @classmethod
+ def validate_id(cls, v):
+ """Convert UUID to str if needed."""
+ if isinstance(v, UUID):
+ return str(v)
+ return v
+
+ @property
+ def key(self) -> CallbackKey:
+ """Return callback ID as key (CallbackKey = str)."""
+ return self.id
+
+
+class ExecuteCallback(BaseDagBundleWorkload):
+ """Execute the given Callback."""
+
+ callback: CallbackDTO
+
+ type: Literal["ExecuteCallback"] = Field(init=False, default="ExecuteCallback")
+
+ @classmethod
+ def make(
+ cls,
+ callback: CallbackModel,
+ dag_run: DagRun,
+ dag_rel_path: Path | None = None,
+ generator: JWTGenerator | None = None,
+ bundle_info: BundleInfo | None = None,
+ ) -> ExecuteCallback:
+ """Create an ExecuteCallback workload from a Callback ORM model."""
+ if not bundle_info:
+ bundle_info = BundleInfo(
+ name=dag_run.dag_model.bundle_name,
+ version=dag_run.bundle_version,
+ )
+ fname = f"executor_callbacks/{callback.id}" # TODO: better log file template
+
+ return cls(
+ callback=CallbackDTO.model_validate(callback, from_attributes=True),
+ dag_rel_path=dag_rel_path or Path(dag_run.dag_model.relative_fileloc or ""),
+ identity_token=cls.generate_token(str(callback.id), generator),
+ log_path=fname,
+ bundle_info=bundle_info,
+ )
+
+
+def execute_callback_workload(
Review Comment:
Sure, thanks Ferruzzi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2754325537
##########
providers/celery/src/airflow/providers/celery/executors/celery_executor.py:
##########
@@ -139,17 +140,31 @@ def _process_tasks(self, task_tuples: Sequence[TaskTuple]) -> None:
def _process_workloads(self, workloads: Sequence[workloads.All]) -> None:
# Airflow V3 version -- have to delay imports until we know we are on v3
- from airflow.executors.workloads import ExecuteTask
+ from airflow.executors.workloads import ExecuteCallback, ExecuteTask
Review Comment:
I'd propose the second - then in the future once Airflow <3.1 support is dropped all these if conditions will be cleaned and then implicitly the logic can be cleaned-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2771190039
##########
providers/celery/src/airflow/providers/celery/executors/celery_executor.py:
##########
@@ -139,17 +140,31 @@ def _process_tasks(self, task_tuples: Sequence[TaskTuple]) -> None:
def _process_workloads(self, workloads: Sequence[workloads.All]) -> None:
# Airflow V3 version -- have to delay imports until we know we are on v3
- from airflow.executors.workloads import ExecuteTask
+ from airflow.executors.workloads import ExecuteCallback, ExecuteTask
Review Comment:
This is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2791087259
##########
airflow-core/src/airflow/executors/workloads/base.py:
##########
@@ -0,0 +1,82 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""ORM models and Pydantic schemas for BaseWorkload."""
+
+from __future__ import annotations
+
+import os
+from abc import ABC
+from typing import TYPE_CHECKING
+
+from pydantic import BaseModel
+
+if TYPE_CHECKING:
+ from airflow.api_fastapi.auth.tokens import JWTGenerator
+
+
+class BaseWorkload:
+ """
+ Mixin for ORM models that can be scheduled as workloads.
+
+ This mixin defines the interface that scheduler workloads (TaskInstance,
+ ExecutorCallback, etc.) must implement to provide routing information to the scheduler.
+
+ Subclasses must override:
+ - get_dag_id() -> str | None
+ - get_executor_name() -> str | None
+ """
+
+ def get_dag_id(self) -> str | None:
+ """
+ Return the DAG ID for scheduler routing.
+
+ Must be implemented by subclasses.
+ """
+ raise NotImplementedError(f"{self.__class__.__name__} must implement get_dag_id()")
+
+ def get_executor_name(self) -> str | None:
+ """
+ Return the executor name for scheduler routing.
+
+ Must be implemented by subclasses.
+ """
+ raise NotImplementedError(f"{self.__class__.__name__} must implement get_executor_name()")
+
+
+class BundleInfo(BaseModel):
+ """Schema for telling task which bundle to run with."""
+
+ name: str
+ version: str | None = None
+
+
+class BaseWorkloadSchema(BaseModel):
+ """Base Pydantic schema for executor workload DTOs."""
+
+ token: str # The identity token for this workload.
Review Comment:
two hard things.... Yours is much better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2867084862
##########
airflow-core/src/airflow/executors/workloads/callback.py:
##########
@@ -0,0 +1,158 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Callback workload schemas for executor communication."""
+
+from __future__ import annotations
+
+from enum import Enum
+from importlib import import_module
+from pathlib import Path
+from typing import TYPE_CHECKING, Literal
+from uuid import UUID
+
+import structlog
+from pydantic import BaseModel, Field, field_validator
+
+from airflow.executors.workloads.base import BaseDagBundleWorkload, BundleInfo
+
+if TYPE_CHECKING:
+ from airflow.api_fastapi.auth.tokens import JWTGenerator
+ from airflow.models import DagRun
+ from airflow.models.callback import Callback as CallbackModel, CallbackKey
+
+log = structlog.get_logger(__name__)
+
+
+class CallbackFetchMethod(str, Enum):
+ """Methods used to fetch callback at runtime."""
+
+ # For future use once Dag Processor callbacks (on_success_callback/on_failure_callback) get moved to executors
+ DAG_ATTRIBUTE = "dag_attribute"
+
+ # For deadline callbacks since they import callbacks through the import path
+ IMPORT_PATH = "import_path"
+
+
+class CallbackDTO(BaseModel):
+ """Schema for Callback with minimal required fields needed for Executors and Task SDK."""
+
+ id: str # A uuid.UUID stored as a string
+ fetch_method: CallbackFetchMethod
+ data: dict
+
+ @field_validator("id", mode="before")
+ @classmethod
+ def validate_id(cls, v):
+ """Convert UUID to str if needed."""
+ if isinstance(v, UUID):
+ return str(v)
+ return v
+
+ @property
+ def key(self) -> CallbackKey:
+ """Return callback ID as key (CallbackKey = str)."""
+ return self.id
+
+
+class ExecuteCallback(BaseDagBundleWorkload):
+ """Execute the given Callback."""
+
+ callback: CallbackDTO
+
+ type: Literal["ExecuteCallback"] = Field(init=False, default="ExecuteCallback")
+
+ @classmethod
+ def make(
+ cls,
+ callback: CallbackModel,
+ dag_run: DagRun,
+ dag_rel_path: Path | None = None,
+ generator: JWTGenerator | None = None,
+ bundle_info: BundleInfo | None = None,
+ ) -> ExecuteCallback:
+ """Create an ExecuteCallback workload from a Callback ORM model."""
+ if not bundle_info:
+ bundle_info = BundleInfo(
+ name=dag_run.dag_model.bundle_name,
+ version=dag_run.bundle_version,
+ )
+ fname = f"executor_callbacks/{callback.id}" # TODO: better log file template
+
+ return cls(
+ callback=CallbackDTO.model_validate(callback, from_attributes=True),
+ dag_rel_path=dag_rel_path or Path(dag_run.dag_model.relative_fileloc or ""),
+ identity_token=cls.generate_token(str(callback.id), generator),
+ log_path=fname,
+ bundle_info=bundle_info,
+ )
+
+
+def execute_callback_workload(
Review Comment:
I think I actually have the supervisor work done but this PR is already pretty bloated, I'm going to merge this one once I get it green again and follow up with this fix immediately. I'll tag all three of you in that one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2771441095
##########
providers/celery/src/airflow/providers/celery/executors/celery_executor.py:
##########
@@ -139,17 +140,31 @@ def _process_tasks(self, task_tuples: Sequence[TaskTuple]) -> None:
def _process_workloads(self, workloads: Sequence[workloads.All]) -> None:
# Airflow V3 version -- have to delay imports until we know we are on v3
- from airflow.executors.workloads import ExecuteTask
+ from airflow.executors.workloads import ExecuteCallback, ExecuteTask
Review Comment:
The merge conflicts are a bit of a mess but I'm working through them. Can I tag you for an approval once they are done and the CI is green?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2791054642
##########
providers/celery/src/airflow/providers/celery/executors/celery_kubernetes_executor.py:
##########
@@ -100,11 +101,15 @@ def _task_event_logs(self):
def _task_event_logs(self, value):
"""Not implemented for hybrid executors."""
- @property
Review Comment:
Even better :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2738623293
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -945,21 +1042,44 @@ def process_executor_events(
ti_primary_key_to_try_number_map: dict[tuple[str, str, str, int], int] = {}
event_buffer = executor.get_event_buffer()
tis_with_right_state: list[TaskInstanceKey] = []
-
- # Report execution
- for ti_key, (state, _) in event_buffer.items():
- # We create map (dag_id, task_id, logical_date) -> in-memory try_number
- ti_primary_key_to_try_number_map[ti_key.primary] = ti_key.try_number
-
- cls.logger().info("Received executor event with state %s for task instance %s", state, ti_key)
- if state in (
- TaskInstanceState.FAILED,
- TaskInstanceState.SUCCESS,
- TaskInstanceState.QUEUED,
- TaskInstanceState.RUNNING,
- TaskInstanceState.RESTARTING,
- ):
- tis_with_right_state.append(ti_key)
+ callback_keys_with_events: list[str] = []
+
+ # Report execution - handle both task and callback events
+ for key, (state, _) in event_buffer.items():
+ if isinstance(key, TaskInstanceKey):
+ ti_primary_key_to_try_number_map[key.primary] = key.try_number
+ cls.logger().info("Received executor event with state %s for task instance %s", state, key)
+ if state in (
+ TaskInstanceState.FAILED,
+ TaskInstanceState.SUCCESS,
+ TaskInstanceState.QUEUED,
+ TaskInstanceState.RUNNING,
+ TaskInstanceState.RESTARTING,
+ ):
+ tis_with_right_state.append(key)
+ else:
+ # Callback event (key is string UUID)
+ cls.logger().info("Received executor event with state %s for callback %s", state, key)
+ if state in (TaskInstanceState.FAILED, TaskInstanceState.SUCCESS):
+ callback_keys_with_events.append(key)
+
+ # Handle callback completion events
+ for callback_id in callback_keys_with_events:
+ state, info = event_buffer.pop(callback_id)
+ callback = session.get(Callback, callback_id)
+ if callback:
+ # Note: We receive TaskInstanceState from executor (SUCCESS/FAILED) but convert to CallbackState here.
+ # This is intentional - executor layer uses generic completion states, scheduler converts to proper types.
+ if state == TaskInstanceState.SUCCESS:
Review Comment:
This is going to fall under the same fix as https://github.com/apache/airflow/pull/61153#discussion_r2734271925
I was on the fence about this but you've convinced me.... Next revision will have a more generic WorkloadState and WorkloadKey. Code is done, just testing it out now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2836604792
##########
airflow-core/src/airflow/models/callback.py:
##########
@@ -14,6 +14,7 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+
Review Comment:
nit: whitespace accident?
##########
providers/edge3/src/airflow/providers/edge3/worker_api/v2-edge-generated.yaml:
##########
Review Comment:
Interesting that it forces re-generation. But OK.
##########
airflow-core/src/airflow/models/callback.py:
##########
@@ -169,6 +152,14 @@ def get_metric_info(self, status: CallbackState, result: Any) -> dict:
return {"stat": name, "tags": tags}
+ def get_dag_id(self) -> str | None:
+ """Return the DAG ID for scheduler routing."""
+ return self.data.get("dag_id")
+
+ def get_executor_name(self) -> str | None:
+ """Return the executor name for scheduler routing."""
+ return self.data.get("executor")
Review Comment:
How about making this a property of the class?
```suggestion
@property
def dag_id(self) -> str | None:
"""Return the DAG ID for scheduler routing."""
return self.data.get("dag_id")
@property
def executor_name(self) -> str | None:
"""Return the executor name for scheduler routing."""
return self.data.get("executor")
```
##########
airflow-core/src/airflow/models/taskinstance.py:
##########
@@ -802,6 +803,14 @@ def key(self) -> TaskInstanceKey:
"""Returns a tuple that identifies the task instance uniquely."""
return TaskInstanceKey(self.dag_id, self.task_id, self.run_id, self.try_number, self.map_index)
+ def get_dag_id(self) -> str:
+ """Return the DAG ID for scheduler routing."""
+ return self.dag_id
+
+ def get_executor_name(self) -> str | None:
+ """Return the executor name for scheduler routing."""
+ return self.executor
Review Comment:
How about making this a property of the class?
```suggestion
@property
def dag_id(self) -> str:
"""Return the DAG ID for scheduler routing."""
return self.dag_id
@property
def executor_name(self) -> str | None:
"""Return the executor name for scheduler routing."""
return self.executor
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2766693767
##########
airflow-core/src/airflow/executors/local_executor.py:
##########
@@ -87,25 +94,31 @@ def _run_worker(
# Received poison pill, no more tasks to run
return
- if not isinstance(workload, workloads.ExecuteTask):
- raise ValueError(f"LocalExecutor does not know how to handle {type(workload)}")
-
# Decrement this as soon as we pick up a message off the queue
with unread_messages:
unread_messages.value -= 1
- key = None
- if ti := getattr(workload, "ti", None):
- key = ti.key
- else:
- raise TypeError(f"Don't know how to get ti key from {type(workload).__name__}")
- try:
- _execute_work(log, workload, team_conf)
+ # Handle different workload types
+ if isinstance(workload, workloads.ExecuteTask):
+ key = workload.ti.key
+ try:
+ _execute_work(log, workload, team_conf)
+ output.put((key, TaskInstanceState.SUCCESS, None))
+ except Exception as e:
+ log.exception("Task execution failed.")
+ output.put((key, TaskInstanceState.FAILED, e))
+
+ elif isinstance(workload, workloads.ExecuteCallback):
+ key = workload.callback.id
+ try:
+ _execute_callback(log, workload, team_conf)
+ output.put((key, TaskInstanceState.SUCCESS, None))
Review Comment:
Created a WorkloadState TypeAlias which should cover this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2771322767
##########
providers/celery/src/airflow/providers/celery/executors/celery_executor.py:
##########
@@ -139,17 +140,31 @@ def _process_tasks(self, task_tuples: Sequence[TaskTuple]) -> None:
def _process_workloads(self, workloads: Sequence[workloads.All]) -> None:
# Airflow V3 version -- have to delay imports until we know we are on v3
- from airflow.executors.workloads import ExecuteTask
+ from airflow.executors.workloads import ExecuteCallback, ExecuteTask
Review Comment:
Looks much better now. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2850999398
##########
airflow-core/src/airflow/executors/workloads/callback.py:
##########
@@ -0,0 +1,158 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Callback workload schemas for executor communication."""
+
+from __future__ import annotations
+
+from enum import Enum
+from importlib import import_module
+from pathlib import Path
+from typing import TYPE_CHECKING, Literal
+from uuid import UUID
+
+import structlog
+from pydantic import BaseModel, Field, field_validator
+
+from airflow.executors.workloads.base import BaseDagBundleWorkload, BundleInfo
+
+if TYPE_CHECKING:
+ from airflow.api_fastapi.auth.tokens import JWTGenerator
+ from airflow.models import DagRun
+ from airflow.models.callback import Callback as CallbackModel, CallbackKey
+
+log = structlog.get_logger(__name__)
+
+
+class CallbackFetchMethod(str, Enum):
+ """Methods used to fetch callback at runtime."""
+
+ # For future use once Dag Processor callbacks (on_success_callback/on_failure_callback) get moved to executors
+ DAG_ATTRIBUTE = "dag_attribute"
+
+ # For deadline callbacks since they import callbacks through the import path
+ IMPORT_PATH = "import_path"
+
+
+class CallbackDTO(BaseModel):
Review Comment:
> How strongly do you feel about this?
Not really a strong opinion on `DTO` from my side, and I agree that `TaskInstance` is confusingly overloaded here.
Based on your explanation above, fair point -- I think using `DTO` is a good fit here.
##########
airflow-core/src/airflow/executors/workloads/callback.py:
##########
@@ -0,0 +1,158 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Callback workload schemas for executor communication."""
+
+from __future__ import annotations
+
+from enum import Enum
+from importlib import import_module
+from pathlib import Path
+from typing import TYPE_CHECKING, Literal
+from uuid import UUID
+
+import structlog
+from pydantic import BaseModel, Field, field_validator
+
+from airflow.executors.workloads.base import BaseDagBundleWorkload, BundleInfo
+
+if TYPE_CHECKING:
+ from airflow.api_fastapi.auth.tokens import JWTGenerator
+ from airflow.models import DagRun
+ from airflow.models.callback import Callback as CallbackModel, CallbackKey
+
+log = structlog.get_logger(__name__)
+
+
+class CallbackFetchMethod(str, Enum):
+ """Methods used to fetch callback at runtime."""
+
+ # For future use once Dag Processor callbacks (on_success_callback/on_failure_callback) get moved to executors
+ DAG_ATTRIBUTE = "dag_attribute"
+
+ # For deadline callbacks since they import callbacks through the import path
+ IMPORT_PATH = "import_path"
+
+
+class CallbackDTO(BaseModel):
+ """Schema for Callback with minimal required fields needed for Executors and Task SDK."""
+
+ id: str # A uuid.UUID stored as a string
+ fetch_method: CallbackFetchMethod
+ data: dict
+
+ @field_validator("id", mode="before")
+ @classmethod
+ def validate_id(cls, v):
+ """Convert UUID to str if needed."""
+ if isinstance(v, UUID):
+ return str(v)
+ return v
+
+ @property
+ def key(self) -> CallbackKey:
+ """Return callback ID as key (CallbackKey = str)."""
+ return self.id
+
+
+class ExecuteCallback(BaseDagBundleWorkload):
+ """Execute the given Callback."""
+
+ callback: CallbackDTO
+
+ type: Literal["ExecuteCallback"] = Field(init=False, default="ExecuteCallback")
+
+ @classmethod
+ def make(
+ cls,
+ callback: CallbackModel,
+ dag_run: DagRun,
+ dag_rel_path: Path | None = None,
+ generator: JWTGenerator | None = None,
+ bundle_info: BundleInfo | None = None,
+ ) -> ExecuteCallback:
+ """Create an ExecuteCallback workload from a Callback ORM model."""
+ if not bundle_info:
+ bundle_info = BundleInfo(
+ name=dag_run.dag_model.bundle_name,
+ version=dag_run.bundle_version,
+ )
+ fname = f"executor_callbacks/{callback.id}" # TODO: better log file template
+
+ return cls(
+ callback=CallbackDTO.model_validate(callback, from_attributes=True),
+ dag_rel_path=dag_rel_path or Path(dag_run.dag_model.relative_fileloc or ""),
+ identity_token=cls.generate_token(str(callback.id), generator),
+ log_path=fname,
+ bundle_info=bundle_info,
+ )
+
+
+def execute_callback_workload(
Review Comment:
Sorry that I should be more clear.
> The callback is now handled exactly the same as any other TaskInstance, however the executor of choice handles those.
What I mean is that, in the case of `LocalExecutor`, for callbacks we call `_execute_callback`, which in turn calls `execute_callback_workload` directly and executes the callable.
https://github.com/apache/airflow/pull/61153/changes#diff-e6c8f3d778a4f3366b23d8d46a1c9650608ccbf2184e7292f0f7bbf4df37dbebR114
In contrast, for tasks we call `_execute_task`, which calls the `supervise` execution-time entrypoint.
https://github.com/apache/airflow/blob/a11ec7296ce52ae3693e46835a49d8c5e3677341/airflow-core/src/airflow/executors/local_executor.py#L111-L119
The `supervise` TaskSDK entrypoint serves as a kind of isolation layer for executing any callable in Airflow. That’s why I expected we would introduce something like `airflow.sdk.execution_time.supervisor.supervise_callback`, so that callbacks are executed via the TaskSDK execution-time layer instead of running raw Python callbacks directly in the executor process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2734319396
##########
airflow-core/src/airflow/executors/local_executor.py:
##########
@@ -53,6 +53,16 @@
TaskInstanceStateType = tuple[workloads.TaskInstance, TaskInstanceState, Exception | None]
+def _get_executor_process_title_prefix(team_name: str | None) -> str:
Review Comment:
Yeah, I might have botched the rebase before I published this. I'll try to extricate those changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2791302661
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -352,32 +353,37 @@ def _get_team_names_for_dag_ids(
# Return dict with all None values to ensure graceful degradation
return {}
- def _get_task_team_name(self, task_instance: TaskInstance, session: Session) -> str | None:
+ def _get_workload_team_name(self, workload: SchedulerWorkload, session: Session) -> str | None:
"""
- Resolve team name for a task instance using the DAG > Bundle > Team relationship chain.
+ Resolve team name for a workload using the DAG > Bundle > Team relationship chain.
- TaskInstance > DagModel (via dag_id) > DagBundleModel (via bundle_name) > Team
+ SchedulerWorkload > DagModel (via dag_id) > DagBundleModel (via bundle_name) > Team
- :param task_instance: The TaskInstance to resolve team name for
+ :param workload: The SchedulerWorkload to resolve team name for
:param session: Database session for queries
:return: Team name if found or None
"""
+ dag_id = workload.get_dag_id()
+ if dag_id is None:
Review Comment:
Not sure I understand the comment, but to answer what i think you are asking, only callbacks (so far) can have None dag_id, not tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2830604821
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -3201,22 +3313,28 @@ def _try_to_load_executor(self, ti: TaskInstance, session, team_name=NOTSET) ->
# No executor found for that team, fall back to global default
executor = self.executor
else:
- # An executor is specified on the TaskInstance (as a str), so we need to find it in the list of executors
+ # An executor is specified on the workload (as a str), so we need to find it in the list of executors
for _executor in self.executors:
- if _executor.name and ti.executor in (_executor.name.alias, _executor.name.module_path):
+ if _executor.name and workload.get_executor_name() in (
+ _executor.name.alias,
+ _executor.name.module_path,
+ ):
# The executor must either match the team or be global (i.e. team_name is None)
if team_name and _executor.team_name == team_name or _executor.team_name is None:
executor = _executor
if executor is not None:
- self.log.debug("Found executor %s for task %s (team: %s)", executor.name, ti, team_name)
+ self.log.debug("Found executor %s for workload %s (team: %s)", executor.name, workload, team_name)
else:
# This case should not happen unless some (as of now unknown) edge case occurs or direct DB
# modification, since the DAG parser will validate the tasks in the DAG and ensure the executor
# they request is available and if not, disallow the DAG to be scheduled.
# Keeping this exception handling because this is a critical issue if we do somehow find
# ourselves here and the user should get some feedback about that.
- self.log.warning("Executor, %s, was not found but a Task was configured to use it", ti.executor)
+ self.log.warning(
+ "Executor, %s, was not found but a workload was configured to use it",
Review Comment:
I'm just concerned about the fact that I made this pretty extensible so other workloads can be added (triggerer events in the executor maybe? who knows... now that the framework is here, I'm sure folks will come up with ideas) and we'll eventually need to start using the generic term, but maybe today is not that day.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2791325362
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -352,32 +353,37 @@ def _get_team_names_for_dag_ids(
# Return dict with all None values to ensure graceful degradation
return {}
- def _get_task_team_name(self, task_instance: TaskInstance, session: Session) -> str | None:
+ def _get_workload_team_name(self, workload: SchedulerWorkload, session: Session) -> str | None:
"""
- Resolve team name for a task instance using the DAG > Bundle > Team relationship chain.
+ Resolve team name for a workload using the DAG > Bundle > Team relationship chain.
- TaskInstance > DagModel (via dag_id) > DagBundleModel (via bundle_name) > Team
+ SchedulerWorkload > DagModel (via dag_id) > DagBundleModel (via bundle_name) > Team
- :param task_instance: The TaskInstance to resolve team name for
+ :param workload: The SchedulerWorkload to resolve team name for
:param session: Database session for queries
:return: Team name if found or None
"""
+ dag_id = workload.get_dag_id()
+ if dag_id is None:
+ self.log.debug("Workload %s has no dag_id, cannot resolve team name", workload)
Review Comment:
It'll get executed fine, but it falls back on the global executor. Maybe it warrants a `warning`? An `error` feels heavy-handed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2734404010
##########
airflow-core/src/airflow/executors/local_executor.py:
##########
@@ -53,6 +53,16 @@
TaskInstanceStateType = tuple[workloads.TaskInstance, TaskInstanceState, Exception | None]
+def _get_executor_process_title_prefix(team_name: str | None) -> str:
Review Comment:
_execute_callback() is the third time duplicating that exact pattern, so I moved it to a helper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2738906428
##########
airflow-core/src/airflow/executors/base_executor.py:
##########
@@ -402,9 +440,20 @@ def trigger_tasks(self, open_slots: int) -> None:
carrier = Trace.inject()
ti.context_carrier = carrier
- workload_list.append(item)
+ workload_list.append(workload)
+
if workload_list:
- self._process_workloads(workload_list)
+ try:
+ self._process_workloads(workload_list)
+ except AttributeError as e:
+ if any(isinstance(workload, workloads.ExecuteCallback) for workload in workload_list):
Review Comment:
I just re-read that block and.... I don't know what I was thinking. "It made sense at the time" is the best I have on this one. I've made the change, it'll be in the next revision. Thanks for calling it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2843692962
##########
airflow-core/src/airflow/models/taskinstance.py:
##########
@@ -802,6 +803,14 @@ def key(self) -> TaskInstanceKey:
"""Returns a tuple that identifies the task instance uniquely."""
return TaskInstanceKey(self.dag_id, self.task_id, self.run_id, self.try_number, self.map_index)
+ def get_dag_id(self) -> str:
+ """Return the DAG ID for scheduler routing."""
+ return self.dag_id
+
+ def get_executor_name(self) -> str | None:
+ """Return the executor name for scheduler routing."""
+ return self.executor
Review Comment:
To clarify: it wasn't an exception, just confusing and misleading. Plus, based on my reading, if there was ever a mismatch between the two values since there is name reuse, Python will take the value from SQLAlchemy's TaskInstance.dag_id column over this property and that might lead to confusion. But like I said, it should work from a technical standpoint if you feel strongly. .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2766710378
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -883,6 +883,103 @@ def _critical_section_enqueue_task_instances(self, session: Session) -> int:
return len(queued_tis)
+ def _enqueue_executor_callbacks(self, session: Session) -> None:
+ """
+ Enqueue ExecutorCallback workloads to executors.
+
+ Similar to _enqueue_task_instances, but for callbacks that need to run on executors.
+ Queries for QUEUED ExecutorCallback instances and routes them to the appropriate executor.
+
+ :param session: The database session
+ """
+ # Query for QUEUED ExecutorCallback instances
+ from airflow.models.callback import CallbackType
+
+ queued_callbacks = session.scalars(
+ select(ExecutorCallback)
+ .where(ExecutorCallback.type == CallbackType.EXECUTOR)
+ .where(ExecutorCallback.state == CallbackState.QUEUED)
+ .order_by(ExecutorCallback.priority_weight.desc())
+ .limit(conf.getint("scheduler", "max_callback_workloads_per_loop", fallback=100))
+ ).all()
+
+ if not queued_callbacks:
+ return
+
+ # Group callbacks by executor (based on callback executor attribute or default executor)
+ executor_to_callbacks: dict[BaseExecutor, list[ExecutorCallback]] = defaultdict(list)
+
+ for callback in queued_callbacks:
+ # Get the executor name from callback data if specified
+ executor_name = None
+ if isinstance(callback.data, dict):
+ executor_name = callback.data.get("executor")
+
+ # Find the appropriate executor
+ executor = None
+ if executor_name:
+ # Find executor by name - try multiple matching strategies
+ for exec in self.job.executors:
+ # Match by class name (e.g., "CeleryExecutor")
+ if exec.__class__.__name__ == executor_name:
+ executor = exec
+ break
+ # Match by executor name attribute if available
+ if hasattr(exec, "name") and exec.name and str(exec.name) == executor_name:
+ executor = exec
+ break
+ # Match by executor name attribute if available
+ if hasattr(exec, "executor_name") and exec.executor_name == executor_name:
+ executor = exec
+ break
+
+ # Default to first executor if no specific executor found
+ if executor is None:
+ executor = self.job.executors[0] if self.job.executors else None
+
+ if executor is None:
+ self.log.warning("No executor available for callback %s", callback.id)
+ continue
Review Comment:
This should be addressed with the generalization of _executor_to_workloads
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2734244930
##########
airflow-core/src/airflow/executors/base_executor.py:
##########
@@ -402,9 +440,20 @@ def trigger_tasks(self, open_slots: int) -> None:
carrier = Trace.inject()
ti.context_carrier = carrier
- workload_list.append(item)
+ workload_list.append(workload)
+
if workload_list:
- self._process_workloads(workload_list)
+ try:
+ self._process_workloads(workload_list)
+ except AttributeError as e:
+ if any(isinstance(workload, workloads.ExecuteCallback) for workload in workload_list):
Review Comment:
If we know exactly how to check for the unsupported use case, why don't we just check before trying to call `__process_workloads()`? Also, we can check much earlier in the queueing of workloads because we can check the `supports_callback` attr?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2738531022
##########
airflow-core/src/airflow/executors/base_executor.py:
##########
@@ -402,9 +440,20 @@ def trigger_tasks(self, open_slots: int) -> None:
carrier = Trace.inject()
ti.context_carrier = carrier
- workload_list.append(item)
+ workload_list.append(workload)
+
if workload_list:
- self._process_workloads(workload_list)
+ try:
+ self._process_workloads(workload_list)
+ except AttributeError as e:
+ if any(isinstance(workload, workloads.ExecuteCallback) for workload in workload_list):
Review Comment:
Fair, I can make the change. I was trying to remove some nesting and using the try/except block as flow-control felt pythonic, but if it's less clear then I can change it up and add a "fail fast" escape hatch in the next revision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2830069032
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -3201,22 +3313,28 @@ def _try_to_load_executor(self, ti: TaskInstance, session, team_name=NOTSET) ->
# No executor found for that team, fall back to global default
executor = self.executor
else:
- # An executor is specified on the TaskInstance (as a str), so we need to find it in the list of executors
+ # An executor is specified on the workload (as a str), so we need to find it in the list of executors
for _executor in self.executors:
- if _executor.name and ti.executor in (_executor.name.alias, _executor.name.module_path):
+ if _executor.name and workload.get_executor_name() in (
+ _executor.name.alias,
+ _executor.name.module_path,
+ ):
# The executor must either match the team or be global (i.e. team_name is None)
if team_name and _executor.team_name == team_name or _executor.team_name is None:
executor = _executor
if executor is not None:
- self.log.debug("Found executor %s for task %s (team: %s)", executor.name, ti, team_name)
+ self.log.debug("Found executor %s for workload %s (team: %s)", executor.name, workload, team_name)
else:
# This case should not happen unless some (as of now unknown) edge case occurs or direct DB
# modification, since the DAG parser will validate the tasks in the DAG and ensure the executor
# they request is available and if not, disallow the DAG to be scheduled.
# Keeping this exception handling because this is a critical issue if we do somehow find
# ourselves here and the user should get some feedback about that.
- self.log.warning("Executor, %s, was not found but a Task was configured to use it", ti.executor)
+ self.log.warning(
+ "Executor, %s, was not found but a workload was configured to use it",
Review Comment:
Maybe "Task or Callback" instead of "workload"? Will users know what a workload is? Here and elsewhere, anytime it's user facing I'm not sure we want to use "workload", at least not without a considered marketing push to make users aware of what it is? Or maybe they already are and I'm being too cautious 😆
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -979,6 +985,70 @@ def _critical_section_enqueue_task_instances(self, session: Session) -> int:
return len(queued_tis)
+ def _enqueue_executor_callbacks(self, session: Session) -> None:
+ """
+ Enqueue ExecutorCallback workloads to executors.
+
+ Similar to _enqueue_task_instances, but for callbacks that need to run on executors.
+ Queries for QUEUED ExecutorCallback instances and routes them to the appropriate executor.
+
+ :param session: The database session
+ """
+ num_occupied_slots = sum(executor.slots_occupied for executor in self.job.executors)
+ max_callbacks = conf.getint("core", "parallelism") - num_occupied_slots
+
+ if max_callbacks <= 0:
+ self.log.debug("No available slots for callbacks; all executors at capacity")
+ return
+
+ queued_callbacks = session.scalars(
+ select(ExecutorCallback)
+ .where(ExecutorCallback.type == CallbackType.EXECUTOR)
+ .where(ExecutorCallback.state == CallbackState.QUEUED)
+ .order_by(ExecutorCallback.priority_weight.desc())
+ .limit(max_callbacks)
+ ).all()
+
+ if not queued_callbacks:
+ return
+
+ # Route callbacks to executors using the generalized routing method
+ executor_to_callbacks = self._executor_to_workloads(queued_callbacks, session)
+
+ # Enqueue callbacks for each executor
+ for executor, callbacks in executor_to_callbacks.items():
+ for callback in callbacks:
+ if not isinstance(callback, ExecutorCallback):
+ # Can't happen since we queried ExecutorCallback, but satisfies mypy.
+ continue
+ dag_run = None
+ if isinstance(callback.data, dict) and "dag_run_id" in callback.data:
+ dag_run_id = callback.data["dag_run_id"]
+ dag_run = session.get(DagRun, dag_run_id)
+ elif isinstance(callback.data, dict) and "dag_id" in callback.data:
+ # Fallback: try to find the latest dag_run for the dag_id
+ dag_id = callback.data["dag_id"]
+ dag_run = session.scalars(
+ select(DagRun)
+ .where(DagRun.dag_id == dag_id)
+ .order_by(DagRun.execution_date.desc())
+ .limit(1)
+ ).first()
+
+ if dag_run is None:
+ self.log.warning("Could not find DagRun for callback %s", callback.id)
+ continue
+
+ workload = workloads.ExecuteCallback.make(
+ callback=callback,
+ dag_run=dag_run,
+ generator=executor.jwt_generator,
+ )
+
+ executor.queue_workload(workload, session=session)
+ callback.state = CallbackState.RUNNING
Review Comment:
Looks better now, it's going to queued once we queue it with the executor instead of directly to running. This is more correct because there is no guarantee that the executor is able to get the workload running.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2791247636
##########
airflow-core/src/airflow/executors/workloads/task.py:
##########
@@ -0,0 +1,104 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Task workload schemas for executor communication."""
+
+from __future__ import annotations
+
+import uuid
+from pathlib import Path
+from typing import TYPE_CHECKING, Literal
+
+from pydantic import BaseModel, Field
+
+from airflow.executors.workloads.base import BaseDagBundleWorkload, BundleInfo
+
+if TYPE_CHECKING:
+ from airflow.api_fastapi.auth.tokens import JWTGenerator
+ from airflow.models.taskinstance import TaskInstance as TIModel
+ from airflow.models.taskinstancekey import TaskInstanceKey
+
+
+class TaskInstanceDTO(BaseModel):
+ """Schema for TaskInstance with minimal required fields needed for Executors and Task SDK."""
+
+ id: uuid.UUID
+ dag_version_id: uuid.UUID
+ task_id: str
+ dag_id: str
+ run_id: str
+ try_number: int
+ map_index: int = -1
+
+ pool_slots: int
+ queue: str
+ priority_weight: int
+ executor_config: dict | None = Field(default=None, exclude=True)
+
+ parent_context_carrier: dict | None = None
+ context_carrier: dict | None = None
+
+ @property
+ def key(self) -> TaskInstanceKey:
+ """Return the TaskInstanceKey for this task instance."""
+ from airflow.models.taskinstancekey import TaskInstanceKey
+
+ return TaskInstanceKey(
+ dag_id=self.dag_id,
+ task_id=self.task_id,
+ run_id=self.run_id,
+ try_number=self.try_number,
+ map_index=self.map_index,
+ )
+
+
+class ExecuteTask(BaseDagBundleWorkload):
+ """Execute the given Task."""
+
+ ti: TaskInstanceDTO
+ sentry_integration: str = ""
+
+ type: Literal["ExecuteTask"] = Field(init=False, default="ExecuteTask")
+
+ @classmethod
+ def make(
Review Comment:
Here is the actual diff for these two classes with main on the left and the new on the right
I did accidentally drop an existing todo from TaskInstance when I ported it over, which I have added back in, so I'll make another pass on the copypasta and make sure I didn't drop any other comments.
<img width="1267" height="1310" alt="image" src="https://github.com/user-attachments/assets/dfc463fd-644c-4962-bafe-791c866ac6c9" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2734269009
##########
airflow-core/src/airflow/executors/local_executor.py:
##########
@@ -53,6 +53,16 @@
TaskInstanceStateType = tuple[workloads.TaskInstance, TaskInstanceState, Exception | None]
+def _get_executor_process_title_prefix(team_name: str | None) -> str:
Review Comment:
These multi team related changes probably shouldn't be showing up in this diff right?
##########
airflow-core/src/airflow/executors/base_executor.py:
##########
@@ -227,10 +229,46 @@ def log_task_event(self, *, event: str, extra: str, ti_key: TaskInstanceKey):
self._task_event_logs.append(Log(event=event, task_instance=ti_key, extra=extra))
def queue_workload(self, workload: workloads.All, session: Session) -> None:
- if not isinstance(workload, workloads.ExecuteTask):
+ if isinstance(workload, workloads.ExecuteTask):
+ ti = workload.ti
+ self.queued_tasks[ti.key] = workload
+ elif isinstance(workload, workloads.ExecuteCallback):
+ self.queued_callbacks[workload.callback.id] = workload
+ else:
raise ValueError(f"Un-handled workload kind {type(workload).__name__!r} in {type(self).__name__}")
- ti = workload.ti
- self.queued_tasks[ti.key] = workload
+
+ def _get_workloads_to_schedule(
+ self, open_slots: int
+ ) -> list[tuple[TaskInstanceKey | str, workloads.All]]:
+ """
+ Select and return the next batch of workloads to schedule, respecting priority policy.
+
+ Priority Policy: Callbacks are scheduled before tasks (callbacks complete existing work).
+ Callbacks are processed in FIFO order. Tasks are sorted by priority_weight (higher priority first).
+
+ :param open_slots: Number of available execution slots
+ """
+ workloads_to_schedule: list[tuple[TaskInstanceKey | str, workloads.All]] = []
+
+ if self.queued_callbacks:
+ for key, workload in self.queued_callbacks.items():
+ if len(workloads_to_schedule) >= open_slots:
+ break
+ workloads_to_schedule.append((key, workload))
+
+ remaining_slots = open_slots - len(workloads_to_schedule)
+ if remaining_slots and self.queued_tasks:
Review Comment:
nit: remaining_slots is only used once. You could just put something like `open_slots > len(workloads_to_schedule)` in the if expression
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -883,6 +883,103 @@ def _critical_section_enqueue_task_instances(self, session: Session) -> int:
return len(queued_tis)
+ def _enqueue_executor_callbacks(self, session: Session) -> None:
+ """
+ Enqueue ExecutorCallback workloads to executors.
+
+ Similar to _enqueue_task_instances, but for callbacks that need to run on executors.
+ Queries for QUEUED ExecutorCallback instances and routes them to the appropriate executor.
+
+ :param session: The database session
+ """
+ # Query for QUEUED ExecutorCallback instances
+ from airflow.models.callback import CallbackType
+
+ queued_callbacks = session.scalars(
+ select(ExecutorCallback)
+ .where(ExecutorCallback.type == CallbackType.EXECUTOR)
+ .where(ExecutorCallback.state == CallbackState.QUEUED)
+ .order_by(ExecutorCallback.priority_weight.desc())
+ .limit(conf.getint("scheduler", "max_callback_workloads_per_loop", fallback=100))
Review Comment:
Here and down below in the final loop over executors/workloads we're just queueing a static amount each time. But it is the schedulers responsibility now (in the world of multiple executors and now multi-team) to ensure we don't ever schedule more tasks (now, workloads) than we have executor slots for. You can see how we do this math for tasks currently here:
https://github.com/apache/airflow/blob/056e24e023a32dbcd5d0be9da45dc4eede770916/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L854-L871
We need to ensure that math now includes callbacks because they also take up worker slots.
I think this will work for now, as long as this method is always called before the critical section. Since callbacks will increase occupied slots in the executors which should be taken into account in the critical section. BUT this code here needs to ensure it doesn't over subscribe the executors. So some similar logic to the critical section needs to be done here. E.g. we're taking a flat 100 here (by default anyway) but there may only be 20 free executor slots.
##########
airflow-core/src/airflow/executors/base_executor.py:
##########
@@ -227,10 +229,46 @@ def log_task_event(self, *, event: str, extra: str, ti_key: TaskInstanceKey):
self._task_event_logs.append(Log(event=event, task_instance=ti_key, extra=extra))
def queue_workload(self, workload: workloads.All, session: Session) -> None:
- if not isinstance(workload, workloads.ExecuteTask):
+ if isinstance(workload, workloads.ExecuteTask):
+ ti = workload.ti
+ self.queued_tasks[ti.key] = workload
+ elif isinstance(workload, workloads.ExecuteCallback):
+ self.queued_callbacks[workload.callback.id] = workload
+ else:
raise ValueError(f"Un-handled workload kind {type(workload).__name__!r} in {type(self).__name__}")
- ti = workload.ti
- self.queued_tasks[ti.key] = workload
+
+ def _get_workloads_to_schedule(
+ self, open_slots: int
+ ) -> list[tuple[TaskInstanceKey | str, workloads.All]]:
+ """
+ Select and return the next batch of workloads to schedule, respecting priority policy.
+
+ Priority Policy: Callbacks are scheduled before tasks (callbacks complete existing work).
+ Callbacks are processed in FIFO order. Tasks are sorted by priority_weight (higher priority first).
+
+ :param open_slots: Number of available execution slots
+ """
+ workloads_to_schedule: list[tuple[TaskInstanceKey | str, workloads.All]] = []
+
+ if self.queued_callbacks:
+ for key, workload in self.queued_callbacks.items():
+ if len(workloads_to_schedule) >= open_slots:
+ break
+ workloads_to_schedule.append((key, workload))
+
+ remaining_slots = open_slots - len(workloads_to_schedule)
+ if remaining_slots and self.queued_tasks:
+ sorted_tasks = sorted(
Review Comment:
Why didn't you use the existing `order_queued_tasks_by_priority()` method?
##########
airflow-core/src/airflow/executors/base_executor.py:
##########
@@ -402,9 +440,20 @@ def trigger_tasks(self, open_slots: int) -> None:
carrier = Trace.inject()
ti.context_carrier = carrier
- workload_list.append(item)
+ workload_list.append(workload)
+
if workload_list:
- self._process_workloads(workload_list)
+ try:
+ self._process_workloads(workload_list)
+ except AttributeError as e:
+ if any(isinstance(workload, workloads.ExecuteCallback) for workload in workload_list):
Review Comment:
If we know exactly how to check for the unsupported use case, why don't we just check before trying to call `__process_workloads()`? Also, we can check much earlier because we can check much earlier in the queueing of workloads because we can check the `supports_callback` attr?
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -883,6 +883,103 @@ def _critical_section_enqueue_task_instances(self, session: Session) -> int:
return len(queued_tis)
+ def _enqueue_executor_callbacks(self, session: Session) -> None:
+ """
+ Enqueue ExecutorCallback workloads to executors.
+
+ Similar to _enqueue_task_instances, but for callbacks that need to run on executors.
+ Queries for QUEUED ExecutorCallback instances and routes them to the appropriate executor.
+
+ :param session: The database session
+ """
+ # Query for QUEUED ExecutorCallback instances
+ from airflow.models.callback import CallbackType
+
+ queued_callbacks = session.scalars(
+ select(ExecutorCallback)
+ .where(ExecutorCallback.type == CallbackType.EXECUTOR)
+ .where(ExecutorCallback.state == CallbackState.QUEUED)
+ .order_by(ExecutorCallback.priority_weight.desc())
+ .limit(conf.getint("scheduler", "max_callback_workloads_per_loop", fallback=100))
+ ).all()
+
+ if not queued_callbacks:
+ return
+
+ # Group callbacks by executor (based on callback executor attribute or default executor)
+ executor_to_callbacks: dict[BaseExecutor, list[ExecutorCallback]] = defaultdict(list)
+
+ for callback in queued_callbacks:
+ # Get the executor name from callback data if specified
+ executor_name = None
+ if isinstance(callback.data, dict):
+ executor_name = callback.data.get("executor")
+
+ # Find the appropriate executor
+ executor = None
+ if executor_name:
+ # Find executor by name - try multiple matching strategies
+ for exec in self.job.executors:
+ # Match by class name (e.g., "CeleryExecutor")
+ if exec.__class__.__name__ == executor_name:
+ executor = exec
+ break
+ # Match by executor name attribute if available
+ if hasattr(exec, "name") and exec.name and str(exec.name) == executor_name:
+ executor = exec
+ break
+ # Match by executor name attribute if available
+ if hasattr(exec, "executor_name") and exec.executor_name == executor_name:
+ executor = exec
+ break
+
+ # Default to first executor if no specific executor found
+ if executor is None:
+ executor = self.job.executors[0] if self.job.executors else None
+
+ if executor is None:
+ self.log.warning("No executor available for callback %s", callback.id)
+ continue
Review Comment:
This is also missing multi-team logic which we need to stay up to date with at this point. It also is duplicating a lot of the work in `_try_to_load_executor` which is made to do exactly this kind of lookup. I think it's going to save you a bunch of effort and future maintenance to update `_try_to_load_executor` to support workloads generally instead of just ti (basically exactly the type of coding you did in the base executor and local executor changes).
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -883,6 +883,103 @@ def _critical_section_enqueue_task_instances(self, session: Session) -> int:
return len(queued_tis)
+ def _enqueue_executor_callbacks(self, session: Session) -> None:
+ """
+ Enqueue ExecutorCallback workloads to executors.
+
+ Similar to _enqueue_task_instances, but for callbacks that need to run on executors.
+ Queries for QUEUED ExecutorCallback instances and routes them to the appropriate executor.
+
+ :param session: The database session
+ """
+ # Query for QUEUED ExecutorCallback instances
+ from airflow.models.callback import CallbackType
+
+ queued_callbacks = session.scalars(
+ select(ExecutorCallback)
+ .where(ExecutorCallback.type == CallbackType.EXECUTOR)
+ .where(ExecutorCallback.state == CallbackState.QUEUED)
+ .order_by(ExecutorCallback.priority_weight.desc())
+ .limit(conf.getint("scheduler", "max_callback_workloads_per_loop", fallback=100))
+ ).all()
+
+ if not queued_callbacks:
+ return
+
+ # Group callbacks by executor (based on callback executor attribute or default executor)
+ executor_to_callbacks: dict[BaseExecutor, list[ExecutorCallback]] = defaultdict(list)
+
+ for callback in queued_callbacks:
+ # Get the executor name from callback data if specified
+ executor_name = None
+ if isinstance(callback.data, dict):
+ executor_name = callback.data.get("executor")
+
+ # Find the appropriate executor
+ executor = None
+ if executor_name:
+ # Find executor by name - try multiple matching strategies
+ for exec in self.job.executors:
+ # Match by class name (e.g., "CeleryExecutor")
+ if exec.__class__.__name__ == executor_name:
+ executor = exec
+ break
+ # Match by executor name attribute if available
+ if hasattr(exec, "name") and exec.name and str(exec.name) == executor_name:
+ executor = exec
+ break
+ # Match by executor name attribute if available
+ if hasattr(exec, "executor_name") and exec.executor_name == executor_name:
+ executor = exec
+ break
+
+ # Default to first executor if no specific executor found
+ if executor is None:
+ executor = self.job.executors[0] if self.job.executors else None
+
+ if executor is None:
+ self.log.warning("No executor available for callback %s", callback.id)
+ continue
+
+ executor_to_callbacks[executor].append(callback)
Review Comment:
Similar to the above, there is already a `_executor_to_tis` which is doing exactly this but for tis, could be generalized.
##########
airflow-core/src/airflow/executors/local_executor.py:
##########
@@ -87,25 +94,31 @@ def _run_worker(
# Received poison pill, no more tasks to run
return
- if not isinstance(workload, workloads.ExecuteTask):
- raise ValueError(f"LocalExecutor does not know how to handle {type(workload)}")
-
# Decrement this as soon as we pick up a message off the queue
with unread_messages:
unread_messages.value -= 1
- key = None
- if ti := getattr(workload, "ti", None):
- key = ti.key
- else:
- raise TypeError(f"Don't know how to get ti key from {type(workload).__name__}")
- try:
- _execute_work(log, workload, team_conf)
+ # Handle different workload types
+ if isinstance(workload, workloads.ExecuteTask):
+ key = workload.ti.key
+ try:
+ _execute_work(log, workload, team_conf)
+ output.put((key, TaskInstanceState.SUCCESS, None))
+ except Exception as e:
+ log.exception("Task execution failed.")
+ output.put((key, TaskInstanceState.FAILED, e))
+
+ elif isinstance(workload, workloads.ExecuteCallback):
+ key = workload.callback.id
+ try:
+ _execute_callback(log, workload, team_conf)
+ output.put((key, TaskInstanceState.SUCCESS, None))
Review Comment:
Mostly just curious: We still use TaskInstanceState here even those these are callbacks?
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -945,21 +1042,44 @@ def process_executor_events(
ti_primary_key_to_try_number_map: dict[tuple[str, str, str, int], int] = {}
event_buffer = executor.get_event_buffer()
tis_with_right_state: list[TaskInstanceKey] = []
-
- # Report execution
- for ti_key, (state, _) in event_buffer.items():
- # We create map (dag_id, task_id, logical_date) -> in-memory try_number
- ti_primary_key_to_try_number_map[ti_key.primary] = ti_key.try_number
-
- cls.logger().info("Received executor event with state %s for task instance %s", state, ti_key)
- if state in (
- TaskInstanceState.FAILED,
- TaskInstanceState.SUCCESS,
- TaskInstanceState.QUEUED,
- TaskInstanceState.RUNNING,
- TaskInstanceState.RESTARTING,
- ):
- tis_with_right_state.append(ti_key)
+ callback_keys_with_events: list[str] = []
+
+ # Report execution - handle both task and callback events
+ for key, (state, _) in event_buffer.items():
+ if isinstance(key, TaskInstanceKey):
+ ti_primary_key_to_try_number_map[key.primary] = key.try_number
+ cls.logger().info("Received executor event with state %s for task instance %s", state, key)
+ if state in (
+ TaskInstanceState.FAILED,
+ TaskInstanceState.SUCCESS,
+ TaskInstanceState.QUEUED,
+ TaskInstanceState.RUNNING,
+ TaskInstanceState.RESTARTING,
+ ):
+ tis_with_right_state.append(key)
+ else:
+ # Callback event (key is string UUID)
+ cls.logger().info("Received executor event with state %s for callback %s", state, key)
+ if state in (TaskInstanceState.FAILED, TaskInstanceState.SUCCESS):
+ callback_keys_with_events.append(key)
+
+ # Handle callback completion events
+ for callback_id in callback_keys_with_events:
+ state, info = event_buffer.pop(callback_id)
+ callback = session.get(Callback, callback_id)
+ if callback:
+ # Note: We receive TaskInstanceState from executor (SUCCESS/FAILED) but convert to CallbackState here.
+ # This is intentional - executor layer uses generic completion states, scheduler converts to proper types.
+ if state == TaskInstanceState.SUCCESS:
Review Comment:
I think this is fine for now, but would be cool if Callbacks were fully first class citizens in executors. Including executors reporting the right state back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2843822224
##########
airflow-core/src/airflow/models/callback.py:
##########
@@ -14,6 +14,7 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+
Review Comment:
Fixed. I actually thought one of the static checks fixed those. Good catch, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2791491024
##########
airflow-core/src/airflow/executors/workloads/callback.py:
##########
@@ -0,0 +1,158 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Callback workload schemas for executor communication."""
+
+from __future__ import annotations
+
+from enum import Enum
+from importlib import import_module
+from pathlib import Path
+from typing import TYPE_CHECKING, Literal
+from uuid import UUID
+
+import structlog
+from pydantic import BaseModel, Field, field_validator
+
+from airflow.executors.workloads.base import BaseDagBundleWorkload, BundleInfo
+
+if TYPE_CHECKING:
+ from airflow.api_fastapi.auth.tokens import JWTGenerator
+ from airflow.models import DagRun
+ from airflow.models.callback import Callback as CallbackModel, CallbackKey
+
+log = structlog.get_logger(__name__)
+
+
+class CallbackFetchMethod(str, Enum):
+ """Methods used to fetch callback at runtime."""
+
+ # For future use once Dag Processor callbacks (on_success_callback/on_failure_callback) get moved to executors
+ DAG_ATTRIBUTE = "dag_attribute"
+
+ # For deadline callbacks since they import callbacks through the import path
+ IMPORT_PATH = "import_path"
+
+
+class CallbackDTO(BaseModel):
+ """Schema for Callback with minimal required fields needed for Executors and Task SDK."""
+
+ id: str # A uuid.UUID stored as a string
+ fetch_method: CallbackFetchMethod
+ data: dict
+
+ @field_validator("id", mode="before")
+ @classmethod
+ def validate_id(cls, v):
+ """Convert UUID to str if needed."""
+ if isinstance(v, UUID):
+ return str(v)
+ return v
+
+ @property
+ def key(self) -> CallbackKey:
+ """Return callback ID as key (CallbackKey = str)."""
+ return self.id
+
+
+class ExecuteCallback(BaseDagBundleWorkload):
+ """Execute the given Callback."""
+
+ callback: CallbackDTO
+
+ type: Literal["ExecuteCallback"] = Field(init=False, default="ExecuteCallback")
+
+ @classmethod
+ def make(
+ cls,
+ callback: CallbackModel,
+ dag_run: DagRun,
+ dag_rel_path: Path | None = None,
+ generator: JWTGenerator | None = None,
+ bundle_info: BundleInfo | None = None,
+ ) -> ExecuteCallback:
+ """Create an ExecuteCallback workload from a Callback ORM model."""
+ if not bundle_info:
+ bundle_info = BundleInfo(
+ name=dag_run.dag_model.bundle_name,
+ version=dag_run.bundle_version,
+ )
+ fname = f"executor_callbacks/{callback.id}" # TODO: better log file template
Review Comment:
That was an existing TODO I migrated from [the old file](https://github.com/apache/airflow/blame/main/airflow-core/src/airflow/executors/workloads.py#L170). I'd like to leave it for now, this PR is already huge without taking on existing TODOs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2791464599
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -979,6 +985,70 @@ def _critical_section_enqueue_task_instances(self, session: Session) -> int:
return len(queued_tis)
+ def _enqueue_executor_callbacks(self, session: Session) -> None:
+ """
+ Enqueue ExecutorCallback workloads to executors.
+
+ Similar to _enqueue_task_instances, but for callbacks that need to run on executors.
+ Queries for QUEUED ExecutorCallback instances and routes them to the appropriate executor.
+
+ :param session: The database session
+ """
+ num_occupied_slots = sum(executor.slots_occupied for executor in self.job.executors)
+ max_callbacks = conf.getint("core", "parallelism") - num_occupied_slots
+
+ if max_callbacks <= 0:
+ self.log.debug("No available slots for callbacks; all executors at capacity")
+ return
+
+ queued_callbacks = session.scalars(
+ select(ExecutorCallback)
+ .where(ExecutorCallback.type == CallbackType.EXECUTOR)
+ .where(ExecutorCallback.state == CallbackState.QUEUED)
+ .order_by(ExecutorCallback.priority_weight.desc())
+ .limit(max_callbacks)
+ ).all()
+
+ if not queued_callbacks:
+ return
+
+ # Route callbacks to executors using the generalized routing method
+ executor_to_callbacks = self._executor_to_workloads(queued_callbacks, session)
+
+ # Enqueue callbacks for each executor
+ for executor, callbacks in executor_to_callbacks.items():
+ for callback in callbacks:
+ if not isinstance(callback, ExecutorCallback):
+ # Can't happen since we queried ExecutorCallback, but satisfies mypy.
+ continue
+ dag_run = None
+ if isinstance(callback.data, dict) and "dag_run_id" in callback.data:
+ dag_run_id = callback.data["dag_run_id"]
+ dag_run = session.get(DagRun, dag_run_id)
+ elif isinstance(callback.data, dict) and "dag_id" in callback.data:
+ # Fallback: try to find the latest dag_run for the dag_id
+ dag_id = callback.data["dag_id"]
+ dag_run = session.scalars(
+ select(DagRun)
+ .where(DagRun.dag_id == dag_id)
+ .order_by(DagRun.execution_date.desc())
+ .limit(1)
+ ).first()
+
+ if dag_run is None:
+ self.log.warning("Could not find DagRun for callback %s", callback.id)
+ continue
+
+ workload = workloads.ExecuteCallback.make(
+ callback=callback,
+ dag_run=dag_run,
+ generator=executor.jwt_generator,
+ )
+
+ executor.queue_workload(workload, session=session)
+ callback.state = CallbackState.RUNNING
Review Comment:
I'm not sure it's behind, but the callbacks are missing a QUEUED state.
PENDING in the Callback.init()
QUEUED by Deadline.handle_miss
RUNNING here after we've queued it.... yeah, alright, I see what you mean....
I'll add a scheduled state and realign these
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2748549870
##########
providers/celery/src/airflow/providers/celery/executors/celery_executor.py:
##########
@@ -139,17 +140,31 @@ def _process_tasks(self, task_tuples: Sequence[TaskTuple]) -> None:
def _process_workloads(self, workloads: Sequence[workloads.All]) -> None:
# Airflow V3 version -- have to delay imports until we know we are on v3
- from airflow.executors.workloads import ExecuteTask
+ from airflow.executors.workloads import ExecuteCallback, ExecuteTask
Review Comment:
Yeah, for sure, good catch. I can do a soft test like :
```
if isinstance(workload, ExecuteTask):
# handle task
elif type(workload).__name__ == "ExecuteCallback":
# Import only if callback found
from airflow.executors.workloads import ExecuteCallback
```
or a hard gate like
```
def _process_workloads(self, workloads):
from airflow.executors.workloads import ExecuteTask
if AIRFLOW_V_3_2_PLUS:
from airflow.executors.workloads import ExecuteCallback
```
The first option feels better, but the second looks more explicit. Any preference?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2734404010
##########
airflow-core/src/airflow/executors/local_executor.py:
##########
@@ -53,6 +53,16 @@
TaskInstanceStateType = tuple[workloads.TaskInstance, TaskInstanceState, Exception | None]
+def _get_executor_process_title_prefix(team_name: str | None) -> str:
Review Comment:
Moving duplicate code into a helper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2748513235
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -883,6 +883,103 @@ def _critical_section_enqueue_task_instances(self, session: Session) -> int:
return len(queued_tis)
+ def _enqueue_executor_callbacks(self, session: Session) -> None:
+ """
+ Enqueue ExecutorCallback workloads to executors.
+
+ Similar to _enqueue_task_instances, but for callbacks that need to run on executors.
+ Queries for QUEUED ExecutorCallback instances and routes them to the appropriate executor.
+
+ :param session: The database session
+ """
+ # Query for QUEUED ExecutorCallback instances
+ from airflow.models.callback import CallbackType
+
+ queued_callbacks = session.scalars(
+ select(ExecutorCallback)
+ .where(ExecutorCallback.type == CallbackType.EXECUTOR)
+ .where(ExecutorCallback.state == CallbackState.QUEUED)
+ .order_by(ExecutorCallback.priority_weight.desc())
+ .limit(conf.getint("scheduler", "max_callback_workloads_per_loop", fallback=100))
Review Comment:
This should be addressed if I understood correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2766692164
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -883,6 +883,103 @@ def _critical_section_enqueue_task_instances(self, session: Session) -> int:
return len(queued_tis)
+ def _enqueue_executor_callbacks(self, session: Session) -> None:
+ """
+ Enqueue ExecutorCallback workloads to executors.
+
+ Similar to _enqueue_task_instances, but for callbacks that need to run on executors.
+ Queries for QUEUED ExecutorCallback instances and routes them to the appropriate executor.
+
+ :param session: The database session
+ """
+ # Query for QUEUED ExecutorCallback instances
+ from airflow.models.callback import CallbackType
+
+ queued_callbacks = session.scalars(
+ select(ExecutorCallback)
+ .where(ExecutorCallback.type == CallbackType.EXECUTOR)
+ .where(ExecutorCallback.state == CallbackState.QUEUED)
+ .order_by(ExecutorCallback.priority_weight.desc())
+ .limit(conf.getint("scheduler", "max_callback_workloads_per_loop", fallback=100))
+ ).all()
+
+ if not queued_callbacks:
+ return
+
+ # Group callbacks by executor (based on callback executor attribute or default executor)
+ executor_to_callbacks: dict[BaseExecutor, list[ExecutorCallback]] = defaultdict(list)
+
+ for callback in queued_callbacks:
+ # Get the executor name from callback data if specified
+ executor_name = None
+ if isinstance(callback.data, dict):
+ executor_name = callback.data.get("executor")
+
+ # Find the appropriate executor
+ executor = None
+ if executor_name:
+ # Find executor by name - try multiple matching strategies
+ for exec in self.job.executors:
+ # Match by class name (e.g., "CeleryExecutor")
+ if exec.__class__.__name__ == executor_name:
+ executor = exec
+ break
+ # Match by executor name attribute if available
+ if hasattr(exec, "name") and exec.name and str(exec.name) == executor_name:
+ executor = exec
+ break
+ # Match by executor name attribute if available
+ if hasattr(exec, "executor_name") and exec.executor_name == executor_name:
+ executor = exec
+ break
+
+ # Default to first executor if no specific executor found
+ if executor is None:
+ executor = self.job.executors[0] if self.job.executors else None
+
+ if executor is None:
+ self.log.warning("No executor available for callback %s", callback.id)
+ continue
+
+ executor_to_callbacks[executor].append(callback)
Review Comment:
This should be done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2842158174
##########
airflow-core/src/airflow/executors/workloads/callback.py:
##########
@@ -0,0 +1,158 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Callback workload schemas for executor communication."""
+
+from __future__ import annotations
+
+from enum import Enum
+from importlib import import_module
+from pathlib import Path
+from typing import TYPE_CHECKING, Literal
+from uuid import UUID
+
+import structlog
+from pydantic import BaseModel, Field, field_validator
+
+from airflow.executors.workloads.base import BaseDagBundleWorkload, BundleInfo
+
+if TYPE_CHECKING:
+ from airflow.api_fastapi.auth.tokens import JWTGenerator
+ from airflow.models import DagRun
+ from airflow.models.callback import Callback as CallbackModel, CallbackKey
+
+log = structlog.get_logger(__name__)
+
+
+class CallbackFetchMethod(str, Enum):
+ """Methods used to fetch callback at runtime."""
+
+ # For future use once Dag Processor callbacks (on_success_callback/on_failure_callback) get moved to executors
+ DAG_ATTRIBUTE = "dag_attribute"
+
+ # For deadline callbacks since they import callbacks through the import path
+ IMPORT_PATH = "import_path"
+
+
+class CallbackDTO(BaseModel):
+ """Schema for Callback with minimal required fields needed for Executors and Task SDK."""
+
+ id: str # A uuid.UUID stored as a string
+ fetch_method: CallbackFetchMethod
+ data: dict
+
+ @field_validator("id", mode="before")
+ @classmethod
+ def validate_id(cls, v):
+ """Convert UUID to str if needed."""
+ if isinstance(v, UUID):
+ return str(v)
+ return v
+
+ @property
+ def key(self) -> CallbackKey:
+ """Return callback ID as key (CallbackKey = str)."""
+ return self.id
+
+
+class ExecuteCallback(BaseDagBundleWorkload):
+ """Execute the given Callback."""
+
+ callback: CallbackDTO
+
+ type: Literal["ExecuteCallback"] = Field(init=False, default="ExecuteCallback")
+
+ @classmethod
+ def make(
+ cls,
+ callback: CallbackModel,
+ dag_run: DagRun,
+ dag_rel_path: Path | None = None,
+ generator: JWTGenerator | None = None,
+ bundle_info: BundleInfo | None = None,
+ ) -> ExecuteCallback:
+ """Create an ExecuteCallback workload from a Callback ORM model."""
+ if not bundle_info:
+ bundle_info = BundleInfo(
+ name=dag_run.dag_model.bundle_name,
+ version=dag_run.bundle_version,
+ )
+ fname = f"executor_callbacks/{callback.id}" # TODO: better log file template
+
+ return cls(
+ callback=CallbackDTO.model_validate(callback, from_attributes=True),
+ dag_rel_path=dag_rel_path or Path(dag_run.dag_model.relative_fileloc or ""),
+ identity_token=cls.generate_token(str(callback.id), generator),
+ log_path=fname,
+ bundle_info=bundle_info,
+ )
+
+
+def execute_callback_workload(
Review Comment:
I'm not sure what you mean by "within TaskSDK" so I'm going to say no, there's no plan or discussion? The callback is now handled exactly the same as any other TaskInstance, however the executor of choice handles those.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2842164641
##########
providers/edge3/src/airflow/providers/edge3/worker_api/v2-edge-generated.yaml:
##########
Review Comment:
Yeah, I definitely didn't touch that file manually. There's a chance maybe it slipped in when i had a messy merge conflict. I'll try to revert that change and re-run static checks, see if it's re-re-generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2842203835
##########
providers/edge3/src/airflow/providers/edge3/worker_api/v2-edge-generated.yaml:
##########
Review Comment:
I'll take another look,. I didn't do that manually, but if I can figure out a way to not force that change to propagate down that far, I will. I was surprised by that as well, but the CI made the change and I rolled with it figuring Trust the System. If you are also surprised then I'll have a look and see what I can do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2867779657
##########
airflow-core/src/airflow/executors/workloads/callback.py:
##########
@@ -0,0 +1,158 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Callback workload schemas for executor communication."""
+
+from __future__ import annotations
+
+from enum import Enum
+from importlib import import_module
+from pathlib import Path
+from typing import TYPE_CHECKING, Literal
+from uuid import UUID
+
+import structlog
+from pydantic import BaseModel, Field, field_validator
+
+from airflow.executors.workloads.base import BaseDagBundleWorkload, BundleInfo
+
+if TYPE_CHECKING:
+ from airflow.api_fastapi.auth.tokens import JWTGenerator
+ from airflow.models import DagRun
+ from airflow.models.callback import Callback as CallbackModel, CallbackKey
+
+log = structlog.get_logger(__name__)
+
+
+class CallbackFetchMethod(str, Enum):
+ """Methods used to fetch callback at runtime."""
+
+ # For future use once Dag Processor callbacks (on_success_callback/on_failure_callback) get moved to executors
+ DAG_ATTRIBUTE = "dag_attribute"
+
+ # For deadline callbacks since they import callbacks through the import path
+ IMPORT_PATH = "import_path"
+
+
+class CallbackDTO(BaseModel):
+ """Schema for Callback with minimal required fields needed for Executors and Task SDK."""
+
+ id: str # A uuid.UUID stored as a string
+ fetch_method: CallbackFetchMethod
+ data: dict
+
+ @field_validator("id", mode="before")
+ @classmethod
+ def validate_id(cls, v):
+ """Convert UUID to str if needed."""
+ if isinstance(v, UUID):
+ return str(v)
+ return v
+
+ @property
+ def key(self) -> CallbackKey:
+ """Return callback ID as key (CallbackKey = str)."""
+ return self.id
+
+
+class ExecuteCallback(BaseDagBundleWorkload):
+ """Execute the given Callback."""
+
+ callback: CallbackDTO
+
+ type: Literal["ExecuteCallback"] = Field(init=False, default="ExecuteCallback")
+
+ @classmethod
+ def make(
+ cls,
+ callback: CallbackModel,
+ dag_run: DagRun,
+ dag_rel_path: Path | None = None,
+ generator: JWTGenerator | None = None,
+ bundle_info: BundleInfo | None = None,
+ ) -> ExecuteCallback:
+ """Create an ExecuteCallback workload from a Callback ORM model."""
+ if not bundle_info:
+ bundle_info = BundleInfo(
+ name=dag_run.dag_model.bundle_name,
+ version=dag_run.bundle_version,
+ )
+ fname = f"executor_callbacks/{callback.id}" # TODO: better log file template
+
+ return cls(
+ callback=CallbackDTO.model_validate(callback, from_attributes=True),
+ dag_rel_path=dag_rel_path or Path(dag_run.dag_model.relative_fileloc or ""),
+ identity_token=cls.generate_token(str(callback.id), generator),
+ log_path=fname,
+ bundle_info=bundle_info,
+ )
+
+
+def execute_callback_workload(
Review Comment:
Follow-up PR to address that is here: https://github.com/apache/airflow/pull/62645
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi merged PR #61153:
URL: https://github.com/apache/airflow/pull/61153
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2843555171
##########
airflow-core/src/airflow/executors/workloads/callback.py:
##########
@@ -0,0 +1,158 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Callback workload schemas for executor communication."""
+
+from __future__ import annotations
+
+from enum import Enum
+from importlib import import_module
+from pathlib import Path
+from typing import TYPE_CHECKING, Literal
+from uuid import UUID
+
+import structlog
+from pydantic import BaseModel, Field, field_validator
+
+from airflow.executors.workloads.base import BaseDagBundleWorkload, BundleInfo
+
+if TYPE_CHECKING:
+ from airflow.api_fastapi.auth.tokens import JWTGenerator
+ from airflow.models import DagRun
+ from airflow.models.callback import Callback as CallbackModel, CallbackKey
+
+log = structlog.get_logger(__name__)
+
+
+class CallbackFetchMethod(str, Enum):
+ """Methods used to fetch callback at runtime."""
+
+ # For future use once Dag Processor callbacks (on_success_callback/on_failure_callback) get moved to executors
+ DAG_ATTRIBUTE = "dag_attribute"
+
+ # For deadline callbacks since they import callbacks through the import path
+ IMPORT_PATH = "import_path"
+
+
+class CallbackDTO(BaseModel):
Review Comment:
"Two hard things", for sure. How strongly do you feel about this? I think "TaskInstance" by itself is confusingly overloaded (I count four different classes/objects named just TaskInstance, IIRC) and we should start differentiating them instead of relying on the reader checking the import module name to know which TI object that are dealing with. Which all gets compounded by using import aliases some places and not others, etc. We have `import TaskInstance as TI` some places, ` import TaskInstance as TIModel` in some places, `import TaskInstance as TaskInstanceSDK`, etc.
We already have `Response`, `Body`, `Key`, `State` suffixes, so adding a new one isn't a stretch.
I like DTO; it's short, simple, and not uncommon outside of Airflow. We may not be using it anywhere else yet but I don't think that should stop us from introducing it. I looked at the `datamodels` module and it doesn't appear to be related, so naming it that might lead to even more confusion. In fact, I might argue the opposite; it seems to make way more sense to rename `execution_api/datamodels/taskinstance.TaskInstance` to `execution_api/datamodels/taskinstance.TaskInstanceDataModel` than it does to use that name for the Pydantic data transfer object here, and using it here would just add confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2790718525
##########
airflow-core/src/airflow/executors/workloads/task.py:
##########
@@ -0,0 +1,104 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Task workload schemas for executor communication."""
+
+from __future__ import annotations
+
+import uuid
+from pathlib import Path
+from typing import TYPE_CHECKING, Literal
+
+from pydantic import BaseModel, Field
+
+from airflow.executors.workloads.base import BaseDagBundleWorkload, BundleInfo
+
+if TYPE_CHECKING:
+ from airflow.api_fastapi.auth.tokens import JWTGenerator
+ from airflow.models.taskinstance import TaskInstance as TIModel
+ from airflow.models.taskinstancekey import TaskInstanceKey
+
+
+class TaskInstanceDTO(BaseModel):
+ """Schema for TaskInstance with minimal required fields needed for Executors and Task SDK."""
+
+ id: uuid.UUID
+ dag_version_id: uuid.UUID
+ task_id: str
+ dag_id: str
+ run_id: str
+ try_number: int
+ map_index: int = -1
+
+ pool_slots: int
+ queue: str
+ priority_weight: int
+ executor_config: dict | None = Field(default=None, exclude=True)
+
+ parent_context_carrier: dict | None = None
+ context_carrier: dict | None = None
+
+ @property
+ def key(self) -> TaskInstanceKey:
+ """Return the TaskInstanceKey for this task instance."""
+ from airflow.models.taskinstancekey import TaskInstanceKey
+
+ return TaskInstanceKey(
+ dag_id=self.dag_id,
+ task_id=self.task_id,
+ run_id=self.run_id,
+ try_number=self.try_number,
+ map_index=self.map_index,
+ )
+
+
+class ExecuteTask(BaseDagBundleWorkload):
+ """Execute the given Task."""
+
+ ti: TaskInstanceDTO
+ sentry_integration: str = ""
+
+ type: Literal["ExecuteTask"] = Field(init=False, default="ExecuteTask")
+
+ @classmethod
+ def make(
Review Comment:
This is all copy/paste right? No changes to how Airflow tasks are constructed?
##########
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/local_kubernetes_executor.py:
##########
@@ -109,8 +110,8 @@ def queued_tasks(self, value) -> None:
"""Not implemented for hybrid executors."""
@property
- def running(self) -> set[TaskInstanceKey]:
Review Comment:
This executor is also deprecated.
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -979,6 +985,70 @@ def _critical_section_enqueue_task_instances(self, session: Session) -> int:
return len(queued_tis)
+ def _enqueue_executor_callbacks(self, session: Session) -> None:
+ """
+ Enqueue ExecutorCallback workloads to executors.
+
+ Similar to _enqueue_task_instances, but for callbacks that need to run on executors.
+ Queries for QUEUED ExecutorCallback instances and routes them to the appropriate executor.
+
+ :param session: The database session
+ """
+ num_occupied_slots = sum(executor.slots_occupied for executor in self.job.executors)
+ max_callbacks = conf.getint("core", "parallelism") - num_occupied_slots
+
+ if max_callbacks <= 0:
+ self.log.debug("No available slots for callbacks; all executors at capacity")
+ return
+
+ queued_callbacks = session.scalars(
+ select(ExecutorCallback)
+ .where(ExecutorCallback.type == CallbackType.EXECUTOR)
+ .where(ExecutorCallback.state == CallbackState.QUEUED)
+ .order_by(ExecutorCallback.priority_weight.desc())
+ .limit(max_callbacks)
+ ).all()
+
+ if not queued_callbacks:
+ return
+
+ # Route callbacks to executors using the generalized routing method
+ executor_to_callbacks = self._executor_to_workloads(queued_callbacks, session)
+
+ # Enqueue callbacks for each executor
+ for executor, callbacks in executor_to_callbacks.items():
+ for callback in callbacks:
+ if not isinstance(callback, ExecutorCallback):
+ # Can't happen since we queried ExecutorCallback, but satisfies mypy.
+ continue
+ dag_run = None
+ if isinstance(callback.data, dict) and "dag_run_id" in callback.data:
+ dag_run_id = callback.data["dag_run_id"]
+ dag_run = session.get(DagRun, dag_run_id)
+ elif isinstance(callback.data, dict) and "dag_id" in callback.data:
+ # Fallback: try to find the latest dag_run for the dag_id
+ dag_id = callback.data["dag_id"]
+ dag_run = session.scalars(
+ select(DagRun)
+ .where(DagRun.dag_id == dag_id)
+ .order_by(DagRun.execution_date.desc())
+ .limit(1)
+ ).first()
+
+ if dag_run is None:
+ self.log.warning("Could not find DagRun for callback %s", callback.id)
+ continue
+
+ workload = workloads.ExecuteCallback.make(
+ callback=callback,
+ dag_run=dag_run,
+ generator=executor.jwt_generator,
+ )
+
+ executor.queue_workload(workload, session=session)
+ callback.state = CallbackState.RUNNING
Review Comment:
The state management is kind of strange here. For tasks we park them as SCHEDULED when they go through the scheduler, and QUEUED when they go through the executor, and RUNNING when they are actually on a worker and running. It seems like the callback state machine is one step earlier in all cases. Was this intentional?
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -1041,21 +1111,44 @@ def process_executor_events(
ti_primary_key_to_try_number_map: dict[tuple[str, str, str, int], int] = {}
event_buffer = executor.get_event_buffer()
tis_with_right_state: list[TaskInstanceKey] = []
-
- # Report execution
- for ti_key, (state, _) in event_buffer.items():
- # We create map (dag_id, task_id, logical_date) -> in-memory try_number
- ti_primary_key_to_try_number_map[ti_key.primary] = ti_key.try_number
-
- cls.logger().info("Received executor event with state %s for task instance %s", state, ti_key)
- if state in (
- TaskInstanceState.FAILED,
- TaskInstanceState.SUCCESS,
- TaskInstanceState.QUEUED,
- TaskInstanceState.RUNNING,
- TaskInstanceState.RESTARTING,
- ):
- tis_with_right_state.append(ti_key)
+ callback_keys_with_events: list[str] = []
+
+ # Report execution - handle both task and callback events
+ for key, (state, _) in event_buffer.items():
+ if isinstance(key, TaskInstanceKey):
+ ti_primary_key_to_try_number_map[key.primary] = key.try_number
+ cls.logger().info("Received executor event with state %s for task instance %s", state, key)
+ if state in (
+ TaskInstanceState.FAILED,
+ TaskInstanceState.SUCCESS,
+ TaskInstanceState.QUEUED,
+ TaskInstanceState.RUNNING,
+ TaskInstanceState.RESTARTING,
+ ):
+ tis_with_right_state.append(key)
+ else:
+ # Callback event (key is string UUID)
+ cls.logger().info("Received executor event with state %s for callback %s", state, key)
+ if state in (TaskInstanceState.FAILED, TaskInstanceState.SUCCESS):
+ callback_keys_with_events.append(key)
+
+ # Handle callback completion events
+ for callback_id in callback_keys_with_events:
+ state, info = event_buffer.pop(callback_id)
+ callback = session.get(Callback, callback_id)
+ if callback:
Review Comment:
Why would we not have callback if we received an event for it?
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -352,32 +353,37 @@ def _get_team_names_for_dag_ids(
# Return dict with all None values to ensure graceful degradation
return {}
- def _get_task_team_name(self, task_instance: TaskInstance, session: Session) -> str | None:
+ def _get_workload_team_name(self, workload: SchedulerWorkload, session: Session) -> str | None:
"""
- Resolve team name for a task instance using the DAG > Bundle > Team relationship chain.
+ Resolve team name for a workload using the DAG > Bundle > Team relationship chain.
- TaskInstance > DagModel (via dag_id) > DagBundleModel (via bundle_name) > Team
+ SchedulerWorkload > DagModel (via dag_id) > DagBundleModel (via bundle_name) > Team
- :param task_instance: The TaskInstance to resolve team name for
+ :param workload: The SchedulerWorkload to resolve team name for
:param session: Database session for queries
:return: Team name if found or None
"""
+ dag_id = workload.get_dag_id()
+ if dag_id is None:
Review Comment:
Can this only happen in the case of a Callback? Or is this something that can happen to a task now?
##########
providers/celery/src/airflow/providers/celery/executors/celery_kubernetes_executor.py:
##########
@@ -100,11 +101,15 @@ def _task_event_logs(self):
def _task_event_logs(self, value):
"""Not implemented for hybrid executors."""
- @property
Review Comment:
This executor is deprecated (see the note [here](https://airflow.apache.org/docs/apache-airflow-providers-celery/stable/celery_kubernetes_executor.html) for example), we don't have to make updates to this executor to support these new features.
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -352,32 +353,37 @@ def _get_team_names_for_dag_ids(
# Return dict with all None values to ensure graceful degradation
return {}
- def _get_task_team_name(self, task_instance: TaskInstance, session: Session) -> str | None:
+ def _get_workload_team_name(self, workload: SchedulerWorkload, session: Session) -> str | None:
"""
- Resolve team name for a task instance using the DAG > Bundle > Team relationship chain.
+ Resolve team name for a workload using the DAG > Bundle > Team relationship chain.
- TaskInstance > DagModel (via dag_id) > DagBundleModel (via bundle_name) > Team
+ SchedulerWorkload > DagModel (via dag_id) > DagBundleModel (via bundle_name) > Team
- :param task_instance: The TaskInstance to resolve team name for
+ :param workload: The SchedulerWorkload to resolve team name for
:param session: Database session for queries
:return: Team name if found or None
"""
+ dag_id = workload.get_dag_id()
+ if dag_id is None:
+ self.log.debug("Workload %s has no dag_id, cannot resolve team name", workload)
Review Comment:
Should this be a warning or error message? Or even an exception? I'm not sure what this means if it doesn't exist.
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -979,6 +985,70 @@ def _critical_section_enqueue_task_instances(self, session: Session) -> int:
return len(queued_tis)
+ def _enqueue_executor_callbacks(self, session: Session) -> None:
+ """
+ Enqueue ExecutorCallback workloads to executors.
+
+ Similar to _enqueue_task_instances, but for callbacks that need to run on executors.
+ Queries for QUEUED ExecutorCallback instances and routes them to the appropriate executor.
+
+ :param session: The database session
+ """
+ num_occupied_slots = sum(executor.slots_occupied for executor in self.job.executors)
+ max_callbacks = conf.getint("core", "parallelism") - num_occupied_slots
+
+ if max_callbacks <= 0:
+ self.log.debug("No available slots for callbacks; all executors at capacity")
+ return
+
+ queued_callbacks = session.scalars(
+ select(ExecutorCallback)
+ .where(ExecutorCallback.type == CallbackType.EXECUTOR)
+ .where(ExecutorCallback.state == CallbackState.QUEUED)
+ .order_by(ExecutorCallback.priority_weight.desc())
+ .limit(max_callbacks)
+ ).all()
+
+ if not queued_callbacks:
+ return
+
+ # Route callbacks to executors using the generalized routing method
+ executor_to_callbacks = self._executor_to_workloads(queued_callbacks, session)
+
+ # Enqueue callbacks for each executor
+ for executor, callbacks in executor_to_callbacks.items():
+ for callback in callbacks:
+ if not isinstance(callback, ExecutorCallback):
+ # Can't happen since we queried ExecutorCallback, but satisfies mypy.
+ continue
+ dag_run = None
Review Comment:
Why do we do so much work to determine the dag_run? This feels like it's a missing input to the callback workload data object when it's created? At that time we should know the task or dag that created the callback. It feels strange to me that we're trying to figure it out after the fact. But I'm probably just struggling to understand the architecture here.
##########
airflow-core/src/airflow/executors/workloads/callback.py:
##########
@@ -0,0 +1,158 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Callback workload schemas for executor communication."""
+
+from __future__ import annotations
+
+from enum import Enum
+from importlib import import_module
+from pathlib import Path
+from typing import TYPE_CHECKING, Literal
+from uuid import UUID
+
+import structlog
+from pydantic import BaseModel, Field, field_validator
+
+from airflow.executors.workloads.base import BaseDagBundleWorkload, BundleInfo
+
+if TYPE_CHECKING:
+ from airflow.api_fastapi.auth.tokens import JWTGenerator
+ from airflow.models import DagRun
+ from airflow.models.callback import Callback as CallbackModel, CallbackKey
+
+log = structlog.get_logger(__name__)
+
+
+class CallbackFetchMethod(str, Enum):
+ """Methods used to fetch callback at runtime."""
+
+ # For future use once Dag Processor callbacks (on_success_callback/on_failure_callback) get moved to executors
+ DAG_ATTRIBUTE = "dag_attribute"
+
+ # For deadline callbacks since they import callbacks through the import path
+ IMPORT_PATH = "import_path"
+
+
+class CallbackDTO(BaseModel):
+ """Schema for Callback with minimal required fields needed for Executors and Task SDK."""
+
+ id: str # A uuid.UUID stored as a string
+ fetch_method: CallbackFetchMethod
+ data: dict
+
+ @field_validator("id", mode="before")
+ @classmethod
+ def validate_id(cls, v):
+ """Convert UUID to str if needed."""
+ if isinstance(v, UUID):
+ return str(v)
+ return v
+
+ @property
+ def key(self) -> CallbackKey:
+ """Return callback ID as key (CallbackKey = str)."""
+ return self.id
+
+
+class ExecuteCallback(BaseDagBundleWorkload):
+ """Execute the given Callback."""
+
+ callback: CallbackDTO
+
+ type: Literal["ExecuteCallback"] = Field(init=False, default="ExecuteCallback")
+
+ @classmethod
+ def make(
+ cls,
+ callback: CallbackModel,
+ dag_run: DagRun,
+ dag_rel_path: Path | None = None,
+ generator: JWTGenerator | None = None,
+ bundle_info: BundleInfo | None = None,
+ ) -> ExecuteCallback:
+ """Create an ExecuteCallback workload from a Callback ORM model."""
+ if not bundle_info:
+ bundle_info = BundleInfo(
+ name=dag_run.dag_model.bundle_name,
+ version=dag_run.bundle_version,
+ )
+ fname = f"executor_callbacks/{callback.id}" # TODO: better log file template
+
+ return cls(
+ callback=CallbackDTO.model_validate(callback, from_attributes=True),
+ dag_rel_path=dag_rel_path or Path(dag_run.dag_model.relative_fileloc or ""),
+ token=cls.generate_token(str(callback.id), generator),
+ log_path=fname,
+ bundle_info=bundle_info,
+ )
+
+
+def execute_callback_workload(
+ callback: CallbackDTO,
+ log,
+) -> tuple[bool, str | None]:
+ """
+ Execute a callback function by importing and calling it, returning the success state.
+
+ Supports two patterns:
+ 1. Functions - called directly with kwargs
+ 2. Classes that return callable instances (like BaseNotifier) - instantiated then called with context
+
+ Example:
+ # Function callback
+ callback.data = {"path": "my_module.alert_func", "kwargs": {"msg": "Alert!", "context": {...}}}
+ execute_callback_workload(callback, log) # Calls alert_func(msg="Alert!", context={...})
+
+ # Notifier callback
+ callback.data = {"path": "airflow.providers.slack...SlackWebhookNotifier", "kwargs": {"text": "Alert!", "context": {...}}}
+ execute_callback_workload(callback, log) # SlackWebhookNotifier(text=..., context=...) then calls instance(context)
+
+ :param callback: The Callback schema containing path and kwargs
+ :param log: Logger instance for recording execution
+ :return: Tuple of (success: bool, error_message: str | None)
+ """
+ callback_path = callback.data.get("path")
+ callback_kwargs = callback.data.get("kwargs", {})
+
+ if not callback_path:
+ return False, "Callback path not found in data."
+
+ try:
+ # Import the callback callable
+ # Expected format: "module.path.to.function_or_class"
+ module_path, function_name = callback_path.rsplit(".", 1)
+ module = import_module(module_path)
+ callback_callable = getattr(module, function_name)
+
+ log.debug("Executing callback %s(%s)...", callback_path, callback_kwargs)
+
+ # If the callback is a callabale, call it. If it is a class, instantiate it.
+ result = callback_callable(**callback_kwargs)
+
+ # If the callback is a class then it is now instantiated and callable, call it.
+ if callable(result):
+ context = callback_kwargs.get("context", {})
+ log.debug("Calling result with context for %s", callback_path)
+ result = result(context)
+
+ log.info("Callback %s executed successfully.", callback_path)
+ return True, None
Review Comment:
The doc string says
> calling it, returning the success state.
Is `True` here basically casted to some success state later up the call stack?
##########
airflow-core/src/airflow/executors/base_executor.py:
##########
@@ -495,24 +534,26 @@ def running_state(self, key: TaskInstanceKey, info=None) -> None:
"""
self.change_state(key, TaskInstanceState.RUNNING, info, remove_running=False)
- def get_event_buffer(self, dag_ids=None) -> dict[TaskInstanceKey, EventBufferValueType]:
+ def get_event_buffer(self, dag_ids=None) -> dict[WorkloadKey, EventBufferValueType]:
"""
Return and flush the event buffer.
In case dag_ids is specified it will only return and flush events
for the given dag_ids. Otherwise, it returns and flushes all events.
+ Note: Callback events (with string keys) are always included regardless of dag_ids filter.
:param dag_ids: the dag_ids to return events for; returns all if given ``None``.
:return: a dict of events
"""
- cleared_events: dict[TaskInstanceKey, EventBufferValueType] = {}
+ cleared_events: dict[WorkloadKey, EventBufferValueType] = {}
if dag_ids is None:
cleared_events = self.event_buffer
self.event_buffer = {}
else:
- for ti_key in list(self.event_buffer.keys()):
- if ti_key.dag_id in dag_ids:
- cleared_events[ti_key] = self.event_buffer.pop(ti_key)
+ for key in list(self.event_buffer.keys()):
+ # Include if it's a callback (string key) or if it's a task in the specified dags
+ if isinstance(key, str) or key.dag_id in dag_ids:
Review Comment:
I think in the future we'll likely want a more reliable check than just str (since other workload types may also have string keys but require different behaviour).
##########
airflow-core/src/airflow/executors/workloads/callback.py:
##########
@@ -0,0 +1,158 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Callback workload schemas for executor communication."""
+
+from __future__ import annotations
+
+from enum import Enum
+from importlib import import_module
+from pathlib import Path
+from typing import TYPE_CHECKING, Literal
+from uuid import UUID
+
+import structlog
+from pydantic import BaseModel, Field, field_validator
+
+from airflow.executors.workloads.base import BaseDagBundleWorkload, BundleInfo
+
+if TYPE_CHECKING:
+ from airflow.api_fastapi.auth.tokens import JWTGenerator
+ from airflow.models import DagRun
+ from airflow.models.callback import Callback as CallbackModel, CallbackKey
+
+log = structlog.get_logger(__name__)
+
+
+class CallbackFetchMethod(str, Enum):
+ """Methods used to fetch callback at runtime."""
+
+ # For future use once Dag Processor callbacks (on_success_callback/on_failure_callback) get moved to executors
+ DAG_ATTRIBUTE = "dag_attribute"
+
+ # For deadline callbacks since they import callbacks through the import path
+ IMPORT_PATH = "import_path"
+
+
+class CallbackDTO(BaseModel):
+ """Schema for Callback with minimal required fields needed for Executors and Task SDK."""
+
+ id: str # A uuid.UUID stored as a string
+ fetch_method: CallbackFetchMethod
+ data: dict
+
+ @field_validator("id", mode="before")
+ @classmethod
+ def validate_id(cls, v):
+ """Convert UUID to str if needed."""
+ if isinstance(v, UUID):
+ return str(v)
+ return v
+
+ @property
+ def key(self) -> CallbackKey:
+ """Return callback ID as key (CallbackKey = str)."""
+ return self.id
+
+
+class ExecuteCallback(BaseDagBundleWorkload):
+ """Execute the given Callback."""
+
+ callback: CallbackDTO
+
+ type: Literal["ExecuteCallback"] = Field(init=False, default="ExecuteCallback")
+
+ @classmethod
+ def make(
+ cls,
+ callback: CallbackModel,
+ dag_run: DagRun,
+ dag_rel_path: Path | None = None,
+ generator: JWTGenerator | None = None,
+ bundle_info: BundleInfo | None = None,
+ ) -> ExecuteCallback:
+ """Create an ExecuteCallback workload from a Callback ORM model."""
+ if not bundle_info:
+ bundle_info = BundleInfo(
+ name=dag_run.dag_model.bundle_name,
+ version=dag_run.bundle_version,
+ )
+ fname = f"executor_callbacks/{callback.id}" # TODO: better log file template
Review Comment:
Did we want to leave this todo undone? Or was that meant to be finished before merging?
##########
airflow-core/src/airflow/executors/workloads/base.py:
##########
@@ -0,0 +1,82 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""ORM models and Pydantic schemas for BaseWorkload."""
+
+from __future__ import annotations
+
+import os
+from abc import ABC
+from typing import TYPE_CHECKING
+
+from pydantic import BaseModel
+
+if TYPE_CHECKING:
+ from airflow.api_fastapi.auth.tokens import JWTGenerator
+
+
+class BaseWorkload:
+ """
+ Mixin for ORM models that can be scheduled as workloads.
+
+ This mixin defines the interface that scheduler workloads (TaskInstance,
+ ExecutorCallback, etc.) must implement to provide routing information to the scheduler.
+
+ Subclasses must override:
+ - get_dag_id() -> str | None
+ - get_executor_name() -> str | None
+ """
+
+ def get_dag_id(self) -> str | None:
+ """
+ Return the DAG ID for scheduler routing.
+
+ Must be implemented by subclasses.
+ """
+ raise NotImplementedError(f"{self.__class__.__name__} must implement get_dag_id()")
+
+ def get_executor_name(self) -> str | None:
+ """
+ Return the executor name for scheduler routing.
+
+ Must be implemented by subclasses.
+ """
+ raise NotImplementedError(f"{self.__class__.__name__} must implement get_executor_name()")
+
+
+class BundleInfo(BaseModel):
+ """Schema for telling task which bundle to run with."""
+
+ name: str
+ version: str | None = None
+
+
+class BaseWorkloadSchema(BaseModel):
+ """Base Pydantic schema for executor workload DTOs."""
+
+ token: str # The identity token for this workload.
Review Comment:
Put the doc in the name?
```suggestion
identity_token: str
```
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -945,21 +1042,44 @@ def process_executor_events(
ti_primary_key_to_try_number_map: dict[tuple[str, str, str, int], int] = {}
event_buffer = executor.get_event_buffer()
tis_with_right_state: list[TaskInstanceKey] = []
-
- # Report execution
- for ti_key, (state, _) in event_buffer.items():
- # We create map (dag_id, task_id, logical_date) -> in-memory try_number
- ti_primary_key_to_try_number_map[ti_key.primary] = ti_key.try_number
-
- cls.logger().info("Received executor event with state %s for task instance %s", state, ti_key)
- if state in (
- TaskInstanceState.FAILED,
- TaskInstanceState.SUCCESS,
- TaskInstanceState.QUEUED,
- TaskInstanceState.RUNNING,
- TaskInstanceState.RESTARTING,
- ):
- tis_with_right_state.append(ti_key)
+ callback_keys_with_events: list[str] = []
+
+ # Report execution - handle both task and callback events
+ for key, (state, _) in event_buffer.items():
+ if isinstance(key, TaskInstanceKey):
+ ti_primary_key_to_try_number_map[key.primary] = key.try_number
+ cls.logger().info("Received executor event with state %s for task instance %s", state, key)
+ if state in (
+ TaskInstanceState.FAILED,
+ TaskInstanceState.SUCCESS,
+ TaskInstanceState.QUEUED,
+ TaskInstanceState.RUNNING,
+ TaskInstanceState.RESTARTING,
+ ):
+ tis_with_right_state.append(key)
+ else:
+ # Callback event (key is string UUID)
+ cls.logger().info("Received executor event with state %s for callback %s", state, key)
+ if state in (TaskInstanceState.FAILED, TaskInstanceState.SUCCESS):
+ callback_keys_with_events.append(key)
+
+ # Handle callback completion events
+ for callback_id in callback_keys_with_events:
+ state, info = event_buffer.pop(callback_id)
+ callback = session.get(Callback, callback_id)
+ if callback:
+ # Note: We receive TaskInstanceState from executor (SUCCESS/FAILED) but convert to CallbackState here.
+ # This is intentional - executor layer uses generic completion states, scheduler converts to proper types.
+ if state == TaskInstanceState.SUCCESS:
Review Comment:
Still looks like this is using TI state for callbacks? Even though we now have full a full callback state machine from the rest of your changes here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2738461118
##########
airflow-core/src/airflow/executors/local_executor.py:
##########
@@ -87,25 +94,31 @@ def _run_worker(
# Received poison pill, no more tasks to run
return
- if not isinstance(workload, workloads.ExecuteTask):
- raise ValueError(f"LocalExecutor does not know how to handle {type(workload)}")
-
# Decrement this as soon as we pick up a message off the queue
with unread_messages:
unread_messages.value -= 1
- key = None
- if ti := getattr(workload, "ti", None):
- key = ti.key
- else:
- raise TypeError(f"Don't know how to get ti key from {type(workload).__name__}")
- try:
- _execute_work(log, workload, team_conf)
+ # Handle different workload types
+ if isinstance(workload, workloads.ExecuteTask):
+ key = workload.ti.key
+ try:
+ _execute_work(log, workload, team_conf)
+ output.put((key, TaskInstanceState.SUCCESS, None))
+ except Exception as e:
+ log.exception("Task execution failed.")
+ output.put((key, TaskInstanceState.FAILED, e))
+
+ elif isinstance(workload, workloads.ExecuteCallback):
+ key = workload.callback.id
+ try:
+ _execute_callback(log, workload, team_conf)
+ output.put((key, TaskInstanceState.SUCCESS, None))
Review Comment:
I chewed this one a bit.... Its' either a SUCCESS or FAIL and I felt heavy handed adding yet another state machine enum to repeat those values. I have an idea I'll include in the next revision and see what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2738489921
##########
airflow-core/src/airflow/executors/base_executor.py:
##########
@@ -227,10 +229,46 @@ def log_task_event(self, *, event: str, extra: str, ti_key: TaskInstanceKey):
self._task_event_logs.append(Log(event=event, task_instance=ti_key, extra=extra))
def queue_workload(self, workload: workloads.All, session: Session) -> None:
- if not isinstance(workload, workloads.ExecuteTask):
+ if isinstance(workload, workloads.ExecuteTask):
+ ti = workload.ti
+ self.queued_tasks[ti.key] = workload
+ elif isinstance(workload, workloads.ExecuteCallback):
+ self.queued_callbacks[workload.callback.id] = workload
+ else:
raise ValueError(f"Un-handled workload kind {type(workload).__name__!r} in {type(self).__name__}")
- ti = workload.ti
- self.queued_tasks[ti.key] = workload
+
+ def _get_workloads_to_schedule(
+ self, open_slots: int
+ ) -> list[tuple[TaskInstanceKey | str, workloads.All]]:
+ """
+ Select and return the next batch of workloads to schedule, respecting priority policy.
+
+ Priority Policy: Callbacks are scheduled before tasks (callbacks complete existing work).
+ Callbacks are processed in FIFO order. Tasks are sorted by priority_weight (higher priority first).
+
+ :param open_slots: Number of available execution slots
+ """
+ workloads_to_schedule: list[tuple[TaskInstanceKey | str, workloads.All]] = []
+
+ if self.queued_callbacks:
+ for key, workload in self.queued_callbacks.items():
+ if len(workloads_to_schedule) >= open_slots:
+ break
+ workloads_to_schedule.append((key, workload))
+
+ remaining_slots = open_slots - len(workloads_to_schedule)
+ if remaining_slots and self.queued_tasks:
Review Comment:
Totally fair, it felt a little more clear/explicit but maybe I just overthinking it. I'll make the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2743533937
##########
providers/celery/src/airflow/providers/celery/executors/celery_executor.py:
##########
@@ -139,17 +140,31 @@ def _process_tasks(self, task_tuples: Sequence[TaskTuple]) -> None:
def _process_workloads(self, workloads: Sequence[workloads.All]) -> None:
# Airflow V3 version -- have to delay imports until we know we are on v3
- from airflow.executors.workloads import ExecuteTask
+ from airflow.executors.workloads import ExecuteCallback, ExecuteTask
Review Comment:
The `ExecuteCallback` was added leter than 3.0.0, so if you import here it will break in core versions <3.2.0?
Can you either make a (very) late or lazy import such that no upgrade dependency to airflow core is generated and that it is still working with 3.0.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2748511378
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -945,21 +1042,44 @@ def process_executor_events(
ti_primary_key_to_try_number_map: dict[tuple[str, str, str, int], int] = {}
event_buffer = executor.get_event_buffer()
tis_with_right_state: list[TaskInstanceKey] = []
-
- # Report execution
- for ti_key, (state, _) in event_buffer.items():
- # We create map (dag_id, task_id, logical_date) -> in-memory try_number
- ti_primary_key_to_try_number_map[ti_key.primary] = ti_key.try_number
-
- cls.logger().info("Received executor event with state %s for task instance %s", state, ti_key)
- if state in (
- TaskInstanceState.FAILED,
- TaskInstanceState.SUCCESS,
- TaskInstanceState.QUEUED,
- TaskInstanceState.RUNNING,
- TaskInstanceState.RESTARTING,
- ):
- tis_with_right_state.append(ti_key)
+ callback_keys_with_events: list[str] = []
+
+ # Report execution - handle both task and callback events
+ for key, (state, _) in event_buffer.items():
+ if isinstance(key, TaskInstanceKey):
+ ti_primary_key_to_try_number_map[key.primary] = key.try_number
+ cls.logger().info("Received executor event with state %s for task instance %s", state, key)
+ if state in (
+ TaskInstanceState.FAILED,
+ TaskInstanceState.SUCCESS,
+ TaskInstanceState.QUEUED,
+ TaskInstanceState.RUNNING,
+ TaskInstanceState.RESTARTING,
+ ):
+ tis_with_right_state.append(key)
+ else:
+ # Callback event (key is string UUID)
+ cls.logger().info("Received executor event with state %s for callback %s", state, key)
+ if state in (TaskInstanceState.FAILED, TaskInstanceState.SUCCESS):
+ callback_keys_with_events.append(key)
+
+ # Handle callback completion events
+ for callback_id in callback_keys_with_events:
+ state, info = event_buffer.pop(callback_id)
+ callback = session.get(Callback, callback_id)
+ if callback:
+ # Note: We receive TaskInstanceState from executor (SUCCESS/FAILED) but convert to CallbackState here.
+ # This is intentional - executor layer uses generic completion states, scheduler converts to proper types.
+ if state == TaskInstanceState.SUCCESS:
Review Comment:
That's up now, let me know what you think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2791408220
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -979,6 +985,70 @@ def _critical_section_enqueue_task_instances(self, session: Session) -> int:
return len(queued_tis)
+ def _enqueue_executor_callbacks(self, session: Session) -> None:
+ """
+ Enqueue ExecutorCallback workloads to executors.
+
+ Similar to _enqueue_task_instances, but for callbacks that need to run on executors.
+ Queries for QUEUED ExecutorCallback instances and routes them to the appropriate executor.
+
+ :param session: The database session
+ """
+ num_occupied_slots = sum(executor.slots_occupied for executor in self.job.executors)
+ max_callbacks = conf.getint("core", "parallelism") - num_occupied_slots
+
+ if max_callbacks <= 0:
+ self.log.debug("No available slots for callbacks; all executors at capacity")
+ return
+
+ queued_callbacks = session.scalars(
+ select(ExecutorCallback)
+ .where(ExecutorCallback.type == CallbackType.EXECUTOR)
+ .where(ExecutorCallback.state == CallbackState.QUEUED)
+ .order_by(ExecutorCallback.priority_weight.desc())
+ .limit(max_callbacks)
+ ).all()
+
+ if not queued_callbacks:
+ return
+
+ # Route callbacks to executors using the generalized routing method
+ executor_to_callbacks = self._executor_to_workloads(queued_callbacks, session)
+
+ # Enqueue callbacks for each executor
+ for executor, callbacks in executor_to_callbacks.items():
+ for callback in callbacks:
+ if not isinstance(callback, ExecutorCallback):
+ # Can't happen since we queried ExecutorCallback, but satisfies mypy.
+ continue
+ dag_run = None
Review Comment:
Just me being too defensive and planning for an unknown future.
As it stands, a deadline is the only way to have a Callback workload and a deadline callback has the darun_id embedded in the callback.data dict, but I was thinking of some unknown future case (like moving the on_(success|failure)_callback over to this workload paradigm where that may not be the case.
I've simplified it for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2791436144
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -1041,21 +1111,44 @@ def process_executor_events(
ti_primary_key_to_try_number_map: dict[tuple[str, str, str, int], int] = {}
event_buffer = executor.get_event_buffer()
tis_with_right_state: list[TaskInstanceKey] = []
-
- # Report execution
- for ti_key, (state, _) in event_buffer.items():
- # We create map (dag_id, task_id, logical_date) -> in-memory try_number
- ti_primary_key_to_try_number_map[ti_key.primary] = ti_key.try_number
-
- cls.logger().info("Received executor event with state %s for task instance %s", state, ti_key)
- if state in (
- TaskInstanceState.FAILED,
- TaskInstanceState.SUCCESS,
- TaskInstanceState.QUEUED,
- TaskInstanceState.RUNNING,
- TaskInstanceState.RESTARTING,
- ):
- tis_with_right_state.append(ti_key)
+ callback_keys_with_events: list[str] = []
+
+ # Report execution - handle both task and callback events
+ for key, (state, _) in event_buffer.items():
+ if isinstance(key, TaskInstanceKey):
+ ti_primary_key_to_try_number_map[key.primary] = key.try_number
+ cls.logger().info("Received executor event with state %s for task instance %s", state, key)
+ if state in (
+ TaskInstanceState.FAILED,
+ TaskInstanceState.SUCCESS,
+ TaskInstanceState.QUEUED,
+ TaskInstanceState.RUNNING,
+ TaskInstanceState.RESTARTING,
+ ):
+ tis_with_right_state.append(key)
+ else:
+ # Callback event (key is string UUID)
+ cls.logger().info("Received executor event with state %s for callback %s", state, key)
+ if state in (TaskInstanceState.FAILED, TaskInstanceState.SUCCESS):
+ callback_keys_with_events.append(key)
+
+ # Handle callback completion events
+ for callback_id in callback_keys_with_events:
+ state, info = event_buffer.pop(callback_id)
+ callback = session.get(Callback, callback_id)
+ if callback:
Review Comment:
There's a possible edge case like if the dagrun is deleted, the callback is on a cascading delete in the database.... but I don't know how likely that is. I'll refactor it to be a bit more clear and add a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2839544644
##########
airflow-core/src/airflow/executors/workloads/callback.py:
##########
@@ -0,0 +1,158 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Callback workload schemas for executor communication."""
+
+from __future__ import annotations
+
+from enum import Enum
+from importlib import import_module
+from pathlib import Path
+from typing import TYPE_CHECKING, Literal
+from uuid import UUID
+
+import structlog
+from pydantic import BaseModel, Field, field_validator
+
+from airflow.executors.workloads.base import BaseDagBundleWorkload, BundleInfo
+
+if TYPE_CHECKING:
+ from airflow.api_fastapi.auth.tokens import JWTGenerator
+ from airflow.models import DagRun
+ from airflow.models.callback import Callback as CallbackModel, CallbackKey
+
+log = structlog.get_logger(__name__)
+
+
+class CallbackFetchMethod(str, Enum):
+ """Methods used to fetch callback at runtime."""
+
+ # For future use once Dag Processor callbacks (on_success_callback/on_failure_callback) get moved to executors
+ DAG_ATTRIBUTE = "dag_attribute"
+
+ # For deadline callbacks since they import callbacks through the import path
+ IMPORT_PATH = "import_path"
+
+
+class CallbackDTO(BaseModel):
Review Comment:
Naming is hard, but we haven’t introduced the term “DTO” in the core codebase so far. Would it be better to name it “DataModel” instead? This also align with the `datamodels` module naming in API.
##########
airflow-core/src/airflow/executors/workloads/callback.py:
##########
@@ -0,0 +1,158 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Callback workload schemas for executor communication."""
+
+from __future__ import annotations
+
+from enum import Enum
+from importlib import import_module
+from pathlib import Path
+from typing import TYPE_CHECKING, Literal
+from uuid import UUID
+
+import structlog
+from pydantic import BaseModel, Field, field_validator
+
+from airflow.executors.workloads.base import BaseDagBundleWorkload, BundleInfo
+
+if TYPE_CHECKING:
+ from airflow.api_fastapi.auth.tokens import JWTGenerator
+ from airflow.models import DagRun
+ from airflow.models.callback import Callback as CallbackModel, CallbackKey
+
+log = structlog.get_logger(__name__)
+
+
+class CallbackFetchMethod(str, Enum):
+ """Methods used to fetch callback at runtime."""
+
+ # For future use once Dag Processor callbacks (on_success_callback/on_failure_callback) get moved to executors
+ DAG_ATTRIBUTE = "dag_attribute"
+
+ # For deadline callbacks since they import callbacks through the import path
+ IMPORT_PATH = "import_path"
+
+
+class CallbackDTO(BaseModel):
+ """Schema for Callback with minimal required fields needed for Executors and Task SDK."""
+
+ id: str # A uuid.UUID stored as a string
+ fetch_method: CallbackFetchMethod
+ data: dict
+
+ @field_validator("id", mode="before")
+ @classmethod
+ def validate_id(cls, v):
+ """Convert UUID to str if needed."""
+ if isinstance(v, UUID):
+ return str(v)
+ return v
+
+ @property
+ def key(self) -> CallbackKey:
+ """Return callback ID as key (CallbackKey = str)."""
+ return self.id
+
+
+class ExecuteCallback(BaseDagBundleWorkload):
+ """Execute the given Callback."""
+
+ callback: CallbackDTO
+
+ type: Literal["ExecuteCallback"] = Field(init=False, default="ExecuteCallback")
+
+ @classmethod
+ def make(
+ cls,
+ callback: CallbackModel,
+ dag_run: DagRun,
+ dag_rel_path: Path | None = None,
+ generator: JWTGenerator | None = None,
+ bundle_info: BundleInfo | None = None,
+ ) -> ExecuteCallback:
+ """Create an ExecuteCallback workload from a Callback ORM model."""
+ if not bundle_info:
+ bundle_info = BundleInfo(
+ name=dag_run.dag_model.bundle_name,
+ version=dag_run.bundle_version,
+ )
+ fname = f"executor_callbacks/{callback.id}" # TODO: better log file template
+
+ return cls(
+ callback=CallbackDTO.model_validate(callback, from_attributes=True),
+ dag_rel_path=dag_rel_path or Path(dag_run.dag_model.relative_fileloc or ""),
+ identity_token=cls.generate_token(str(callback.id), generator),
+ log_path=fname,
+ bundle_info=bundle_info,
+ )
+
+
+def execute_callback_workload(
Review Comment:
May I ask whether the callback will eventually be executed within TaskSDK instead of within the Scheduler/Executor process in the long term?
Executing the callback in the Scheduler/Executor process does not seem very secure, even though it does not violate the AF3 security model, since the Scheduler/Executor process can directly access the database.
From my perspective, all synchronous workloads will be executed within TaskSDK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2843129395
##########
providers/edge3/src/airflow/providers/edge3/worker_api/v2-edge-generated.yaml:
##########
Review Comment:
Yeah, this was fallout from @o-nikolas's suggestion to rename token to identity_token. That ended up having wider ripples than I expected. With hindsight in full effect, I have reverted that name change for now since maybe that's a breaking change. We can re-do that renaming later if we decide we want it, and that will help simplify this already-large PR a little.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2843610592
##########
airflow-core/src/airflow/models/taskinstance.py:
##########
@@ -802,6 +803,14 @@ def key(self) -> TaskInstanceKey:
"""Returns a tuple that identifies the task instance uniquely."""
return TaskInstanceKey(self.dag_id, self.task_id, self.run_id, self.try_number, self.map_index)
+ def get_dag_id(self) -> str:
+ """Return the DAG ID for scheduler routing."""
+ return self.dag_id
+
+ def get_executor_name(self) -> str | None:
+ """Return the executor name for scheduler routing."""
+ return self.executor
Review Comment:
I see where you are going and tried it that way at first, but it was causing name conflict issues with the SQLALchemy `TaskInstance` model which uses the same name for one of the columns. Add to that the fact that TI and Callback are retrieving that value differently, I felt that having the getters was a more unified/standardized interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "Dev-iL (via GitHub)" <gi...@apache.org>.
Dev-iL commented on PR #61153:
URL: https://github.com/apache/airflow/pull/61153#issuecomment-3932378274
Comments from Opus:
---
## Summary
This PR adds `ExecuteCallback` as a new workload type alongside `ExecuteTask` and `RunTrigger`, enabling synchronous callbacks (e.g., `SyncCallback` wrapping `SlackWebhookNotifier`) to be dispatched through the executor pipeline. It refactors the `workloads.py` single-file module into a `workloads/` package, introduces a `BaseWorkload` ORM mixin for unified scheduler routing, and updates BaseExecutor, LocalExecutor, CeleryExecutor, the scheduler loop, and the deadline model to support callback workloads.
---
## Critical Issues
### 1. Layer Violation: `airflow.models` Importing from `airflow.executors`
The PR introduces imports from `airflow.executors.workloads` into two model files:
```python
# airflow/models/callback.py
from airflow.executors.workloads import BaseWorkload
from airflow.executors.workloads.callback import CallbackFetchMethod
# airflow/models/taskinstance.py
from airflow.executors.workloads import BaseWorkload
```
And adds `BaseWorkload` as a mixin to both `Callback(Base, BaseWorkload)` and `TaskInstance(Base, LoggingMixin, BaseWorkload)`. This creates a **circular dependency risk** and violates the current layering where models are independent of executors.
`CallbackFetchMethod` was originally defined in `airflow/models/callback.py` — this PR moves it to `airflow/executors/workloads/callback.py`, then the model imports it back. The enum definition should stay in the models layer or move to a shared module.
**Recommendation:** Move `BaseWorkload` to `airflow.models.base` or a shared module like `airflow._shared.workload_protocol`. Keep `CallbackFetchMethod` in `airflow.models.callback` (its original location) or move to `airflow._shared`. The executor workloads package should depend on models, not the other way around.
### 2. Arbitrary Code Execution via `import_module` Without Validation
`execute_callback_workload()` in `workloads/callback.py` dynamically imports and executes any Python callable based on a `path` string stored in the database:
```python
module_path, function_name = callback_path.rsplit(".", 1)
module = import_module(module_path)
callback_callable = getattr(module, function_name)
result = callback_callable(**callback_kwargs)
```
If an attacker gains write access to the `callback` table (SQL injection, compromised admin, etc.), they can execute arbitrary code on executor workers. The current `handle_miss` path only stores paths from deserialized SDK callback definitions, but there is no validation at the execution boundary.
**Recommendation:** Add allowlist validation or at minimum verify the path resolves to a subclass of `BaseNotifier` or a registered callback type before executing.
### 3. CeleryExecutor / BaseExecutor Queue Inconsistency
`BaseExecutor.queue_workload` stores callbacks in `self.queued_callbacks`:
```python
self.queued_callbacks[workload.callback.id] = workload
```
But `CeleryExecutor.queue_workload` overrides this and stores callbacks in `self.queued_tasks`:
```python
elif isinstance(workload, workloads.ExecuteCallback):
self.queued_tasks[workload.callback.key] = workload
```
This creates two problems:
1. **`_get_workloads_to_schedule`** in `BaseExecutor` iterates `queued_callbacks` first (for priority), then `queued_tasks`. Since CeleryExecutor puts callbacks in `queued_tasks`, callback-over-task priority is silently broken for Celery.
2. **Slot accounting** (`slots_available`, `slots_occupied`) counts `len(queued_tasks) + len(queued_callbacks)`. For CeleryExecutor, `queued_callbacks` is always empty, so slot math is correct by accident, but the semantic inconsistency is a maintenance trap.
**Recommendation:** Either have CeleryExecutor also use `queued_callbacks` for callback workloads, or document and enforce that `_get_workloads_to_schedule` is not used by CeleryExecutor. A better approach would be to standardize on `queued_callbacks` in `BaseExecutor.queue_workload` (which CeleryExecutor should call via `super()`) and then override `_process_workloads` to handle callbacks from the correct dict.
---
## Architectural Concerns
### 4. Callback Routing Depends on Data Dict, Not ORM Relationships
The scheduler reconstructs `DagRun` for each callback in `_enqueue_executor_callbacks` via:
```python
if isinstance(callback.data, dict) and "dag_run_id" in callback.data:
dag_run = session.get(DagRun, dag_run_id)
elif isinstance(callback.data, dict) and "dag_id" in callback.data:
dag_run = session.scalars(
select(DagRun).where(DagRun.dag_id == dag_id)
.order_by(DagRun.execution_date.desc()).limit(1)
).first()
```
The fallback path (lookup by `dag_id` with `ORDER BY execution_date DESC LIMIT 1`) is dangerous — it could match the **wrong** DagRun if a DAG has multiple concurrent runs. Similarly, executor routing reads `callback.data.get("executor")` from the JSON blob. These should be first-class columns or relationships, not buried in a serialized dict.
**Recommendation:** This is acceptable for an initial implementation but should be tracked as a fast-follow. Convert the existing TODO to a GitHub issue and specifically flag the fallback DagRun lookup as a known correctness risk.
### 5. Shared Parallelism Pool for Tasks and Callbacks
Callbacks share the same `parallelism` slots as tasks:
```python
num_occupied_slots = sum(executor.slots_occupied for executor in self.executors)
max_callbacks = conf.getint("core", "parallelism") - num_occupied_slots
```
And in `BaseExecutor`:
```python
def slots_available(self):
return self.parallelism - len(self.running) - len(self.queued_tasks) - len(self.queued_callbacks)
```
This means a fully-loaded executor could indefinitely delay deadline alert callbacks — exactly the kind of alert that should have guaranteed delivery. Conversely, a burst of callbacks could starve task execution.
**Recommendation:** Consider a reserved callback slot count (e.g., `max_callback_slots = min(N, parallelism * 0.1)`) or at minimum document this limitation. The AIP doesn't specify callback delivery guarantees, but "deadline missed but alert never fired" is a poor user experience.
### 6. `WorkloadKey` Union Type Pollutes Executor Interface
The PR widens `BaseExecutor.running` from `set[TaskInstanceKey]` to `set[WorkloadKey]` (i.e., `set[TaskInstanceKey | str]`) and `event_buffer` similarly. `CallbackKey` is a bare `str` alias defined inside `TYPE_CHECKING`, so at runtime the distinction between a callback key and an arbitrary string is invisible. This forces `type: ignore` comments in the hybrid executors and weakens type safety.
```python
# celery_kubernetes_executor.py
return self.celery_executor.running.union(self.kubernetes_executor.running) # type: ignore[return-value, arg-type]
```
**Recommendation:** Consider a proper `CallbackKey` wrapper class (not just `str` alias) that shares a protocol with `TaskInstanceKey`, or keep callbacks in separate data structures in the executor layer rather than mixing them into `running`/`event_buffer`.
### 7. Dead `DeadlineCallbackState` Enum
The PR adds a `DeadlineCallbackState` enum to `airflow/models/deadline.py`:
```python
class DeadlineCallbackState(str, Enum):
"""All possible states of deadline callbacks once the deadline is missed."""
...
```
This enum is **never referenced** anywhere in the codebase — not in queries, not in state transitions, not in tests. It appears to be leftover from an earlier design iteration.
**Recommendation:** Remove `DeadlineCallbackState` to avoid dead code.
---
## Code Quality Issues
### 8. Typo in Docstring
`workloads/callback.py`:
```python
# If the callback is a callabale, call it. If it is a class, instantiate it.
```
Should be "callable".
### 9. Fragile Callable-vs-Class Heuristic
```python
result = callback_callable(**callback_kwargs)
if callable(result):
context = callback_kwargs.get("context", {})
result = result(context)
```
This assumes: if calling the path returns a callable, it must be a class instance (like `BaseNotifier`) that needs to be called again with context. But any function that returns a callable (e.g., a factory function) would be double-invoked. The `SyncCallback` / `AsyncCallback` distinction should drive explicit dispatch, not runtime inspection of return values.
**Recommendation:** Use the `fetch_method` field or add a `callback_type` field to dispatch explicitly between "call this function" and "instantiate this class, then call the instance."
### 10. `BaseWorkload` Mixin Uses `NotImplementedError` Instead of `abc.ABC`
```python
class BaseWorkload:
def get_dag_id(self) -> str | None:
raise NotImplementedError(f"{self.__class__.__name__} must implement get_dag_id()")
def get_executor_name(self) -> str | None:
raise NotImplementedError(f"{self.__class__.__name__} must implement get_executor_name()")
```
This is a mixin added to `TaskInstance` and `Callback`, both of which are ORM models with hundreds of existing tests. If any code path accidentally calls these on an unpatched subclass, it will raise at runtime rather than being caught by type checking. Consider using `abc.ABC` or a `Protocol` instead.
### 11. `_enqueue_executor_callbacks` Queries for QUEUED but `handle_miss` Sets State to QUEUED via `queue()`
The state lifecycle for executor callbacks is:
```
PENDING (Callback.__init__) → QUEUED (handle_miss → callback.queue()) → RUNNING (_enqueue_executor_callbacks) → SUCCESS/FAILED (process_executor_events)
```
However, `_enqueue_executor_callbacks` queries for `CallbackState.QUEUED` and immediately sets `CallbackState.RUNNING` before the executor has actually started processing. If the executor crashes between queue and execution, the callback is stuck in `RUNNING` with no recovery path. There is no heartbeat or timeout mechanism for callback execution.
**Recommendation:** Consider keeping the state as `QUEUED` until the executor confirms it has started (via event buffer), similar to how task instances transition through `QUEUED → RUNNING` based on executor events. Alternatively, add a callback execution timeout.
### 12. `process_executor_events` Maps Between State Enum Types
The scheduler maps executor-provided `TaskInstanceState` to `CallbackState` for callback events:
```python
# Callback event (key is string UUID)
if state in (TaskInstanceState.FAILED, TaskInstanceState.SUCCESS):
callback_keys_with_events.append(key)
...
if state == TaskInstanceState.SUCCESS:
callback.state = CallbackState.SUCCESS
elif state == TaskInstanceState.FAILED:
callback.state = CallbackState.FAILED
```
The code includes a comment noting this is intentional ("executor layer uses generic completion states, scheduler converts to proper types"). While functionally correct, this coupling means that if either enum changes independently, the mapping silently breaks. Consider defining the mapping in a single place (e.g., a dict) rather than scattered `if/elif` chains.
---
## AIP Alignment
### What the AIP specifies:
- DAG-level deadlines with callbacks that work with the existing Notifier system
- Callbacks must not cause extra DAG file parsing
- OTel metrics from day one
- Future: dedicated sidecar process for callbacks
### PR alignment:
- **Positive:** Callbacks flow through the executor pipeline without parsing DAG files. The existing `deadline_alerts.deadline_missed` metric is preserved.
- **Positive:** Extensible — `supports_callbacks` flag lets executors opt in incrementally.
- **Positive:** Both `AsyncCallback` (via triggerer) and `SyncCallback` (via executor) are now supported.
- **Gap:** No new OTel metric for callback execution outcome (`deadline_alerts_triggered` counter from AIP is not added here; only the existing `deadline_missed` stat is emitted).
- **Gap:** No callback execution duration metric.
- **Deviation:** The AIP mentions a "dedicated sidecar process for callbacks" as future work. This PR routes callbacks through the main executor pool instead, which is a reasonable interim approach but should be documented as a known limitation.
---
## Test Coverage Assessment
### Covered:
- `supports_callbacks` flag on BaseExecutor and LocalExecutor
- `queue_workload` with ExecuteCallback (with and without support)
- Callback prioritization over tasks in `_get_workloads_to_schedule`
- `execute_callback_workload` success, missing path, import error, execution error
- LocalExecutor callback processing (including `queued_callbacks` cleanup)
- DeadlineAlert accepts both AsyncCallback and SyncCallback
### Missing:
- CeleryExecutor end-to-end callback processing (only task-type workloads are tested)
- Scheduler `process_executor_events` handling of callback state transitions (`RUNNING` → `SUCCESS`/`FAILED`)
- Multi-executor callback routing (callback specifies a particular executor)
- Fallback DagRun lookup by `dag_id` when `dag_run_id` is missing (correctness risk)
- Callback cleanup when DagRun is deleted mid-execution
- `_executor_to_workloads` with mixed TaskInstance and ExecutorCallback inputs
- Parallelism exhaustion scenario (all slots occupied, callbacks waiting)
- Callback stuck in RUNNING after executor crash (no recovery test)
---
## Minor Nits
1. The `workloads/callback.py` `CallbackFetchMethod` enum has a `DAG_ATTRIBUTE` member marked "for future use" — consider removing it until it's needed to avoid dead code.
2. The edge worker API schema (`v2-edge-generated.yaml`) renames `TaskInstance` → `TaskInstanceDTO`. Ensure the provider version constraint enforces this for edge workers on older versions.
3. `docs/spelling_wordlist.txt` adds `WorkloadKey` — also add `WorkloadState`, `CallbackDTO`, `SchedulerWorkload` if they appear in docs.
4. `send_task_to_executor` is renamed to `send_workload_to_executor` in `celery_executor_utils.py` with a backward-compat alias. The alias is good, but the import in `celery_executor.py` still references the old name — verify this is intentional and tested.
---
## Verdict
The core design — treating callbacks as first-class executor workloads — is sound and aligns with the AIP's goal of reliable deadline alert delivery. However, the PR needs attention on:
1. **Must fix before merge:** Fix the layer violation (`models` → `executors` import direction); resolve the CeleryExecutor/BaseExecutor queue inconsistency (`queued_callbacks` vs `queued_tasks`); remove dead `DeadlineCallbackState` enum.
2. **Should fix:** Add callback execution validation (allowlist or type check); fix the callable-vs-class heuristic; guard the fallback DagRun lookup; fix the typo.
3. **Track as follow-up:** ORM foreign key for `dag_run_id`; dedicated callback parallelism pool; OTel callback execution metrics; `CallbackKey` wrapper type; callback execution timeout/recovery.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on PR #61153:
URL: https://github.com/apache/airflow/pull/61153#issuecomment-3937740000
Some of those were valid, a lot of it was less useful. I'm not going to address each point in detail, but I have made some small adjustments based on the comment and rebased again to fix yet another merge conflict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2738530762
##########
airflow-core/src/airflow/executors/base_executor.py:
##########
@@ -227,10 +229,46 @@ def log_task_event(self, *, event: str, extra: str, ti_key: TaskInstanceKey):
self._task_event_logs.append(Log(event=event, task_instance=ti_key, extra=extra))
def queue_workload(self, workload: workloads.All, session: Session) -> None:
- if not isinstance(workload, workloads.ExecuteTask):
+ if isinstance(workload, workloads.ExecuteTask):
+ ti = workload.ti
+ self.queued_tasks[ti.key] = workload
+ elif isinstance(workload, workloads.ExecuteCallback):
+ self.queued_callbacks[workload.callback.id] = workload
+ else:
raise ValueError(f"Un-handled workload kind {type(workload).__name__!r} in {type(self).__name__}")
- ti = workload.ti
- self.queued_tasks[ti.key] = workload
+
+ def _get_workloads_to_schedule(
+ self, open_slots: int
+ ) -> list[tuple[TaskInstanceKey | str, workloads.All]]:
+ """
+ Select and return the next batch of workloads to schedule, respecting priority policy.
+
+ Priority Policy: Callbacks are scheduled before tasks (callbacks complete existing work).
+ Callbacks are processed in FIFO order. Tasks are sorted by priority_weight (higher priority first).
+
+ :param open_slots: Number of available execution slots
+ """
+ workloads_to_schedule: list[tuple[TaskInstanceKey | str, workloads.All]] = []
+
+ if self.queued_callbacks:
+ for key, workload in self.queued_callbacks.items():
+ if len(workloads_to_schedule) >= open_slots:
+ break
+ workloads_to_schedule.append((key, workload))
+
+ remaining_slots = open_slots - len(workloads_to_schedule)
+ if remaining_slots and self.queued_tasks:
+ sorted_tasks = sorted(
Review Comment:
This method is great, I meant specifically this lambda you're using to sort and prioritize the tasks. There is already `order_queued_tasks_by_priority()` for doing exactly this right? See the old line 362 for a usage of it. I think it's what you want here, but maybe I'm missing something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2738604506
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -883,6 +883,103 @@ def _critical_section_enqueue_task_instances(self, session: Session) -> int:
return len(queued_tis)
+ def _enqueue_executor_callbacks(self, session: Session) -> None:
+ """
+ Enqueue ExecutorCallback workloads to executors.
+
+ Similar to _enqueue_task_instances, but for callbacks that need to run on executors.
+ Queries for QUEUED ExecutorCallback instances and routes them to the appropriate executor.
+
+ :param session: The database session
+ """
+ # Query for QUEUED ExecutorCallback instances
+ from airflow.models.callback import CallbackType
+
+ queued_callbacks = session.scalars(
+ select(ExecutorCallback)
+ .where(ExecutorCallback.type == CallbackType.EXECUTOR)
+ .where(ExecutorCallback.state == CallbackState.QUEUED)
+ .order_by(ExecutorCallback.priority_weight.desc())
+ .limit(conf.getint("scheduler", "max_callback_workloads_per_loop", fallback=100))
Review Comment:
Ah. I think I see what you are saying, I missed that. I may need to chat offline to clarify, but I'll look into it. Thanks for catching that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2771452624
##########
providers/celery/src/airflow/providers/celery/executors/celery_executor.py:
##########
@@ -139,17 +140,31 @@ def _process_tasks(self, task_tuples: Sequence[TaskTuple]) -> None:
def _process_workloads(self, workloads: Sequence[workloads.All]) -> None:
# Airflow V3 version -- have to delay imports until we know we are on v3
- from airflow.executors.workloads import ExecuteTask
+ from airflow.executors.workloads import ExecuteCallback, ExecuteTask
Review Comment:
Yes but accept that I only am working a few hours in the evening, so please do not fully rely on my availability...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2791211467
##########
airflow-core/src/airflow/executors/workloads/callback.py:
##########
@@ -0,0 +1,158 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Callback workload schemas for executor communication."""
+
+from __future__ import annotations
+
+from enum import Enum
+from importlib import import_module
+from pathlib import Path
+from typing import TYPE_CHECKING, Literal
+from uuid import UUID
+
+import structlog
+from pydantic import BaseModel, Field, field_validator
+
+from airflow.executors.workloads.base import BaseDagBundleWorkload, BundleInfo
+
+if TYPE_CHECKING:
+ from airflow.api_fastapi.auth.tokens import JWTGenerator
+ from airflow.models import DagRun
+ from airflow.models.callback import Callback as CallbackModel, CallbackKey
+
+log = structlog.get_logger(__name__)
+
+
+class CallbackFetchMethod(str, Enum):
+ """Methods used to fetch callback at runtime."""
+
+ # For future use once Dag Processor callbacks (on_success_callback/on_failure_callback) get moved to executors
+ DAG_ATTRIBUTE = "dag_attribute"
+
+ # For deadline callbacks since they import callbacks through the import path
+ IMPORT_PATH = "import_path"
+
+
+class CallbackDTO(BaseModel):
+ """Schema for Callback with minimal required fields needed for Executors and Task SDK."""
+
+ id: str # A uuid.UUID stored as a string
+ fetch_method: CallbackFetchMethod
+ data: dict
+
+ @field_validator("id", mode="before")
+ @classmethod
+ def validate_id(cls, v):
+ """Convert UUID to str if needed."""
+ if isinstance(v, UUID):
+ return str(v)
+ return v
+
+ @property
+ def key(self) -> CallbackKey:
+ """Return callback ID as key (CallbackKey = str)."""
+ return self.id
+
+
+class ExecuteCallback(BaseDagBundleWorkload):
+ """Execute the given Callback."""
+
+ callback: CallbackDTO
+
+ type: Literal["ExecuteCallback"] = Field(init=False, default="ExecuteCallback")
+
+ @classmethod
+ def make(
+ cls,
+ callback: CallbackModel,
+ dag_run: DagRun,
+ dag_rel_path: Path | None = None,
+ generator: JWTGenerator | None = None,
+ bundle_info: BundleInfo | None = None,
+ ) -> ExecuteCallback:
+ """Create an ExecuteCallback workload from a Callback ORM model."""
+ if not bundle_info:
+ bundle_info = BundleInfo(
+ name=dag_run.dag_model.bundle_name,
+ version=dag_run.bundle_version,
+ )
+ fname = f"executor_callbacks/{callback.id}" # TODO: better log file template
+
+ return cls(
+ callback=CallbackDTO.model_validate(callback, from_attributes=True),
+ dag_rel_path=dag_rel_path or Path(dag_run.dag_model.relative_fileloc or ""),
+ token=cls.generate_token(str(callback.id), generator),
+ log_path=fname,
+ bundle_info=bundle_info,
+ )
+
+
+def execute_callback_workload(
+ callback: CallbackDTO,
+ log,
+) -> tuple[bool, str | None]:
+ """
+ Execute a callback function by importing and calling it, returning the success state.
+
+ Supports two patterns:
+ 1. Functions - called directly with kwargs
+ 2. Classes that return callable instances (like BaseNotifier) - instantiated then called with context
+
+ Example:
+ # Function callback
+ callback.data = {"path": "my_module.alert_func", "kwargs": {"msg": "Alert!", "context": {...}}}
+ execute_callback_workload(callback, log) # Calls alert_func(msg="Alert!", context={...})
+
+ # Notifier callback
+ callback.data = {"path": "airflow.providers.slack...SlackWebhookNotifier", "kwargs": {"text": "Alert!", "context": {...}}}
+ execute_callback_workload(callback, log) # SlackWebhookNotifier(text=..., context=...) then calls instance(context)
+
+ :param callback: The Callback schema containing path and kwargs
+ :param log: Logger instance for recording execution
+ :return: Tuple of (success: bool, error_message: str | None)
+ """
+ callback_path = callback.data.get("path")
+ callback_kwargs = callback.data.get("kwargs", {})
+
+ if not callback_path:
+ return False, "Callback path not found in data."
+
+ try:
+ # Import the callback callable
+ # Expected format: "module.path.to.function_or_class"
+ module_path, function_name = callback_path.rsplit(".", 1)
+ module = import_module(module_path)
+ callback_callable = getattr(module, function_name)
+
+ log.debug("Executing callback %s(%s)...", callback_path, callback_kwargs)
+
+ # If the callback is a callabale, call it. If it is a class, instantiate it.
+ result = callback_callable(**callback_kwargs)
+
+ # If the callback is a class then it is now instantiated and callable, call it.
+ if callable(result):
+ context = callback_kwargs.get("context", {})
+ log.debug("Calling result with context for %s", callback_path)
+ result = result(context)
+
+ log.info("Callback %s executed successfully.", callback_path)
+ return True, None
Review Comment:
I modelled it after the existing pattern for the task workload.
The existing task workload pattern was that supervise() returned an `int` exit_code which the calling method bubbles up and _run_worker() catches.
This returns a true/false bool for success or fail, and the calling method (in this case _execute_callback() in local_executor) has an `if not success: raise`. which _run_worker() catches
see _run_worker() in local_executor.py around line 103 for where that happens.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2791062690
##########
airflow-core/src/airflow/executors/workloads/task.py:
##########
@@ -0,0 +1,104 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Task workload schemas for executor communication."""
+
+from __future__ import annotations
+
+import uuid
+from pathlib import Path
+from typing import TYPE_CHECKING, Literal
+
+from pydantic import BaseModel, Field
+
+from airflow.executors.workloads.base import BaseDagBundleWorkload, BundleInfo
+
+if TYPE_CHECKING:
+ from airflow.api_fastapi.auth.tokens import JWTGenerator
+ from airflow.models.taskinstance import TaskInstance as TIModel
+ from airflow.models.taskinstancekey import TaskInstanceKey
+
+
+class TaskInstanceDTO(BaseModel):
+ """Schema for TaskInstance with minimal required fields needed for Executors and Task SDK."""
+
+ id: uuid.UUID
+ dag_version_id: uuid.UUID
+ task_id: str
+ dag_id: str
+ run_id: str
+ try_number: int
+ map_index: int = -1
+
+ pool_slots: int
+ queue: str
+ priority_weight: int
+ executor_config: dict | None = Field(default=None, exclude=True)
+
+ parent_context_carrier: dict | None = None
+ context_carrier: dict | None = None
+
+ @property
+ def key(self) -> TaskInstanceKey:
+ """Return the TaskInstanceKey for this task instance."""
+ from airflow.models.taskinstancekey import TaskInstanceKey
+
+ return TaskInstanceKey(
+ dag_id=self.dag_id,
+ task_id=self.task_id,
+ run_id=self.run_id,
+ try_number=self.try_number,
+ map_index=self.map_index,
+ )
+
+
+class ExecuteTask(BaseDagBundleWorkload):
+ """Execute the given Task."""
+
+ ti: TaskInstanceDTO
+ sentry_integration: str = ""
+
+ type: Literal["ExecuteTask"] = Field(init=False, default="ExecuteTask")
+
+ @classmethod
+ def make(
Review Comment:
Sorry, I've been wresting with the deadline persistence bug (https://github.com/apache/airflow/pull/61702) and got distracted, I had meant to make that comment. Yeah, the tasks and triggerer parts are pure copypasta, just relocating the code and maybe some more generic type hinting like using WorkloadType instead of the more specific TaskType or whatever
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executor Synchronous callback workload [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #61153:
URL: https://github.com/apache/airflow/pull/61153#discussion_r2791260039
##########
airflow-core/src/airflow/executors/base_executor.py:
##########
@@ -495,24 +534,26 @@ def running_state(self, key: TaskInstanceKey, info=None) -> None:
"""
self.change_state(key, TaskInstanceState.RUNNING, info, remove_running=False)
- def get_event_buffer(self, dag_ids=None) -> dict[TaskInstanceKey, EventBufferValueType]:
+ def get_event_buffer(self, dag_ids=None) -> dict[WorkloadKey, EventBufferValueType]:
"""
Return and flush the event buffer.
In case dag_ids is specified it will only return and flush events
for the given dag_ids. Otherwise, it returns and flushes all events.
+ Note: Callback events (with string keys) are always included regardless of dag_ids filter.
:param dag_ids: the dag_ids to return events for; returns all if given ``None``.
:return: a dict of events
"""
- cleared_events: dict[TaskInstanceKey, EventBufferValueType] = {}
+ cleared_events: dict[WorkloadKey, EventBufferValueType] = {}
if dag_ids is None:
cleared_events = self.event_buffer
self.event_buffer = {}
else:
- for ti_key in list(self.event_buffer.keys()):
- if ti_key.dag_id in dag_ids:
- cleared_events[ti_key] = self.event_buffer.pop(ti_key)
+ for key in list(self.event_buffer.keys()):
+ # Include if it's a callback (string key) or if it's a task in the specified dags
+ if isinstance(key, str) or key.dag_id in dag_ids:
Review Comment:
I did partially think of that and made the CallbackKey type alias.
It's just a type alias for a `str` (for now?), and it was a later addition; I should have gone back and used it here when I created it. I'll make that change.
But you are also right that in the future, other workload types may use strings and we may want to adjust this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org