You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "amoghrajesh (via GitHub)" <gi...@apache.org> on 2026/05/05 09:00:16 UTC

Re: [PR] Add Execution API endpoints for callback lifecycle and token swap [airflow]

amoghrajesh commented on code in PR #66141:
URL: https://github.com/apache/airflow/pull/66141#discussion_r3182194392


##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -1258,17 +1258,32 @@ def process_executor_events(
                 )
                 continue
 
+            # Callback state transitions are now driven by the supervisor through
+            # the Execution API (POST /callbacks/{id}/run, PATCH /callbacks/{id}/state).
+            # The in-process events from the executor are kept as a fallback safety
+            # net for cases where the supervisor crashed before reporting a terminal state
+
+            need_to_modify = False
+
             if state == CallbackState.RUNNING:
-                callback.state = CallbackState.RUNNING
                 cls.logger().info("Callback %s is currently running", callback_id)
+                if callback.state == CallbackState.QUEUED:
+                    # The state change QUEUE -> RUNNING must be only executed by api
+                    pass

Review Comment:
   Why do we need this block? Seems to be dead code



##########
airflow-core/src/airflow/api_fastapi/execution_api/routes/callbacks.py:
##########
@@ -0,0 +1,152 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import Annotated
+from uuid import UUID
+
+import structlog
+from cadwyn import VersionedAPIRouter
+from fastapi import Body, HTTPException, Response, Security, status
+from structlog.contextvars import bind_contextvars
+
+from airflow.api_fastapi.auth.tokens import JWTGenerator
+from airflow.api_fastapi.common.db.common import SessionDep
+from airflow.api_fastapi.execution_api.datamodels.callback import CallbackTerminalStatePayload
+from airflow.api_fastapi.execution_api.datamodels.token import TIToken
+from airflow.api_fastapi.execution_api.deps import DepContainer
+from airflow.api_fastapi.execution_api.security import CurrentTIToken, ExecutionAPIRoute, require_auth
+from airflow.models.callback import Callback
+from airflow.utils.state import CallbackState
+
+log = structlog.get_logger(__name__)
+
+router = VersionedAPIRouter(route_class=ExecutionAPIRoute)
+
+
+def _require_self(token: TIToken, callback_id: UUID) -> None:
+    """Mirror the ``ti:self`` enforcement from security.py for callback routes."""
+    if str(token.id) != str(callback_id):
+        raise HTTPException(
+            status_code=status.HTTP_403_FORBIDDEN,
+            detail="Token subject does not match callback id",
+        )
+
+
+@router.post(
+    "/{callback_id}/run",
+    status_code=status.HTTP_204_NO_CONTENT,
+    dependencies=[Security(require_auth, scopes=["token:execution", "token:workload"])],
+    responses={
+        status.HTTP_403_FORBIDDEN: {"description": "Token subject does not match callback id"},
+        status.HTTP_404_NOT_FOUND: {"description": "Callback not found"},
+        status.HTTP_409_CONFLICT: {"description": "Callback is not in a state that can be marked running"},
+    },
+)
+def callback_run(
+    callback_id: UUID,
+    response: Response,
+    session: SessionDep,
+    services=DepContainer,
+    token: TIToken = CurrentTIToken,
+) -> Response:
+    """
+    Mark a callback as RUNNING.
+
+    Mirrors ``PATCH /task-instances/{id}/run``: this is the single endpoint that
+    accepts a workload-scoped token and atomically (a) transitions the callback
+    from QUEUED to RUNNING and (b) issues a fresh execution-scoped token via the
+    ``Refreshed-API-Token`` response header. All subsequent supervisor calls hit
+    execution-only routes.
+    """
+    bind_contextvars(callback_id=str(callback_id))
+    _require_self(token, callback_id)
+
+    callback = session.get(Callback, callback_id)
+    if callback is None:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail={"reason": "not_found", "message": "Callback not found"},
+        )
+
+    # Allow QUEUED → RUNNING transition; treat RUNNING as idempotent so a retried
+    # supervisor start does not 409. Anything else (PENDING / SCHEDULED / terminal)
+    # rejects.
+    if not callback.state == CallbackState.QUEUED:

Review Comment:
   ```suggestion
       if callback.state != CallbackState.QUEUED:
   ```



##########
task-sdk/tests/task_sdk/execution_time/test_callback_supervisor.py:
##########
@@ -228,3 +234,108 @@ def test_handle_requests(
 
         if client_mock:
             mock_client_method.assert_called_once_with(*client_mock.args, **client_mock.kwargs)
+
+
+class TestSuperviseCallback:
+    """supervise_callback drives every callback state transition through the API."""
+
+    def _make_mock_client(self, mocker):
+        client = mocker.Mock(spec=Client)
+        client.callbacks = mocker.Mock()

Review Comment:
   Use spec for mocks, `spec=CallbackOperations`



##########
airflow-core/src/airflow/api_fastapi/execution_api/routes/callbacks.py:
##########
@@ -0,0 +1,152 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import Annotated
+from uuid import UUID
+
+import structlog
+from cadwyn import VersionedAPIRouter
+from fastapi import Body, HTTPException, Response, Security, status
+from structlog.contextvars import bind_contextvars
+
+from airflow.api_fastapi.auth.tokens import JWTGenerator
+from airflow.api_fastapi.common.db.common import SessionDep
+from airflow.api_fastapi.execution_api.datamodels.callback import CallbackTerminalStatePayload
+from airflow.api_fastapi.execution_api.datamodels.token import TIToken
+from airflow.api_fastapi.execution_api.deps import DepContainer
+from airflow.api_fastapi.execution_api.security import CurrentTIToken, ExecutionAPIRoute, require_auth
+from airflow.models.callback import Callback
+from airflow.utils.state import CallbackState
+
+log = structlog.get_logger(__name__)
+
+router = VersionedAPIRouter(route_class=ExecutionAPIRoute)
+
+
+def _require_self(token: TIToken, callback_id: UUID) -> None:
+    """Mirror the ``ti:self`` enforcement from security.py for callback routes."""
+    if str(token.id) != str(callback_id):
+        raise HTTPException(
+            status_code=status.HTTP_403_FORBIDDEN,
+            detail="Token subject does not match callback id",
+        )
+
+
+@router.post(
+    "/{callback_id}/run",
+    status_code=status.HTTP_204_NO_CONTENT,
+    dependencies=[Security(require_auth, scopes=["token:execution", "token:workload"])],
+    responses={
+        status.HTTP_403_FORBIDDEN: {"description": "Token subject does not match callback id"},
+        status.HTTP_404_NOT_FOUND: {"description": "Callback not found"},
+        status.HTTP_409_CONFLICT: {"description": "Callback is not in a state that can be marked running"},
+    },
+)
+def callback_run(
+    callback_id: UUID,
+    response: Response,
+    session: SessionDep,
+    services=DepContainer,
+    token: TIToken = CurrentTIToken,
+) -> Response:
+    """
+    Mark a callback as RUNNING.
+
+    Mirrors ``PATCH /task-instances/{id}/run``: this is the single endpoint that
+    accepts a workload-scoped token and atomically (a) transitions the callback
+    from QUEUED to RUNNING and (b) issues a fresh execution-scoped token via the
+    ``Refreshed-API-Token`` response header. All subsequent supervisor calls hit
+    execution-only routes.
+    """
+    bind_contextvars(callback_id=str(callback_id))
+    _require_self(token, callback_id)
+
+    callback = session.get(Callback, callback_id)
+    if callback is None:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail={"reason": "not_found", "message": "Callback not found"},
+        )
+
+    # Allow QUEUED → RUNNING transition; treat RUNNING as idempotent so a retried
+    # supervisor start does not 409. Anything else (PENDING / SCHEDULED / terminal)
+    # rejects.

Review Comment:
   This comment above says "treat RUNNING as idempotent so a retried supervisor start does not 409" but this condition raises 409 for RUNNING. If the server processes the POST but the response is lost in transit (DB → RUNNING, token minted but never received), a retry hits 409 and the supervisor is stuck with a workload token — the exact bug this PR fixes. Handle RUNNING as a success path (return 204 + a fresh execution token)
   
   



##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -1258,17 +1258,32 @@ def process_executor_events(
                 )
                 continue
 
+            # Callback state transitions are now driven by the supervisor through
+            # the Execution API (POST /callbacks/{id}/run, PATCH /callbacks/{id}/state).
+            # The in-process events from the executor are kept as a fallback safety
+            # net for cases where the supervisor crashed before reporting a terminal state
+
+            need_to_modify = False
+
             if state == CallbackState.RUNNING:
-                callback.state = CallbackState.RUNNING
                 cls.logger().info("Callback %s is currently running", callback_id)
+                if callback.state == CallbackState.QUEUED:
+                    # The state change QUEUE -> RUNNING must be only executed by api
+                    pass
             elif state == CallbackState.SUCCESS:
-                callback.state = CallbackState.SUCCESS
                 cls.logger().info("Callback %s completed successfully", callback_id)
+                if callback.state == CallbackState.RUNNING:
+                    callback.state = CallbackState.SUCCESS
+                    need_to_modify = True
             elif state == CallbackState.FAILED:
-                callback.state = CallbackState.FAILED
-                callback.output = str(info) if info else "Execution failed"
                 cls.logger().error("Callback %s failed: %s", callback_id, callback.output)

Review Comment:
   This is a regression from earlier: `callback.output` is logged here before it is assigned in the conditional block below. 
   
   



##########
task-sdk/src/airflow/sdk/api/datamodels/_generated.py:
##########
@@ -63,6 +63,23 @@ class AssetProfile(BaseModel):
     type: Annotated[str, Field(title="Type")]
 
 
+class State(str, Enum):

Review Comment:
   This file is generated manually? Please regenerate this file from the updated OpenAPI spec rather than hand-writing the models.
   
   



##########
task-sdk/src/airflow/sdk/api/datamodels/_generated.py:
##########
@@ -63,6 +63,23 @@ class AssetProfile(BaseModel):
     type: Annotated[str, Field(title="Type")]
 
 
+class State(str, Enum):

Review Comment:
   We have a prek hook for this that runs when you try to commit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org