You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kaxil (via GitHub)" <gi...@apache.org> on 2025/07/09 00:32:23 UTC

[PR] Run Task failure callbacks on DAG Processor when task is externally k… [airflow]

kaxil opened a new pull request, #53058:
URL: https://github.com/apache/airflow/pull/53058

   …illed
   
   Until https://github.com/apache/airflow/issues/44354 is implemented, tasks killed externally or when supervisor process dies unexpectedly, users have no way of knowing this happened.
   
   This has been a blocker for Airflow 3.0 adoption for some:
   
   - https://github.com/apache/airflow/issues/44354
   - https://apache-airflow.slack.com/archives/C07813CNKA8/p1751057525231389
   
   https://github.com/apache/airflow/issues/44354 is more involved and we might not get to it for Airflow 3.1 -- so this is a good fix until then similar to how we run Dag Run callback.
   
   Ran a example DAG:
   
   ```py
   import time
   
   from airflow.sdk import dag
   from airflow.providers.standard.operators.python import PythonOperator
   @dag
   def my_dag():
   
       def fail_cb(context):
           print("From inside the callback")
           print(context)
   
       def n_s():
           time.sleep(3000)
   
       t = PythonOperator(
           task_id="my_task",
           python_callable=n_s,
           on_failure_callback=fail_cb
       )
   
       t
   
   my_dag()
   
   ```
   
   
   DAG Processor Logs after I killed the Supervisor & task process when the task is running:
   
   ```
   {"timestamp":"2025-07-08T22:48:31.568171Z","level":"info","event":"{'dag': <DAG: my_dag>, 'inlets': [], 'map_index_template': None, 'outlets': [], 'run_id': 'manual__2025-07-08T22:44:37.346022+00:00', 'task': <Task(PythonOperator): my_task>, 'task_instance': RuntimeTaskInstance(id=UUID('0197ec36-656a-75b7-814a-c409886945e2'), task_id='my_task', dag_id='my_dag', run_id='manual__2025-07-08T22:44:37.346022+00:00', try_number=2, dag_version_id=UUID('0197ec0c-669e-7e05-80e3-97c3549dbfda'), map_index=-1, hostname='c9bd6c415dbf', context_carrier={}, task=<Task(PythonOperator): my_task>, max_tries=0, end_date=None, state=None, is_mapped=None, rendered_map_index=None, log_url=None), 'ti': RuntimeTaskInstance(id=UUID('0197ec36-656a-75b7-814a-c409886945e2'), task_id='my_task', dag_id='my_dag', run_id='manual__2025-07-08T22:44:37.346022+00:00', try_number=2, dag_version_id=UUID('0197ec0c-669e-7e05-80e3-97c3549dbfda'), map_index=-1, hostname='c9bd6c415dbf', context_carrier={}, task=<Task(Pytho
 nOperator): my_task>, max_tries=0, end_date=None, state=None, is_mapped=None, rendered_map_index=None, log_url=None), 'outlet_events': <airflow.sdk.execution_time.context.OutletEventAccessors object at 0xffff6cb48880>, 'inlet_events': InletEventsAccessors(_inlets=[], _assets={}, _asset_aliases={}), 'macros': <MacrosAccessor (dynamic access to macros)>, 'params': {}, 'var': {'json': <VariableAccessor (dynamic access)>, 'value': <VariableAccessor (dynamic access)>}, 'conn': <ConnectionAccessor (dynamic access)>, 'dag_run': DagRun(dag_id='my_dag', run_id='manual__2025-07-08T22:44:37.346022+00:00', logical_date=datetime.datetime(2025, 7, 8, 22, 44, 35, 842000, tzinfo=Timezone('UTC')), data_interval_start=datetime.datetime(2025, 7, 8, 22, 44, 35, 842000, tzinfo=Timezone('UTC')), data_interval_end=datetime.datetime(2025, 7, 8, 22, 44, 35, 842000, tzinfo=Timezone('UTC')), run_after=datetime.datetime(2025, 7, 8, 22, 44, 35, 842000, tzinfo=Timezone('UTC')), start_date=datetime.datetime(2025,
  7, 8, 22, 44, 44, 227565, tzinfo=Timezone('UTC')), end_date=None, clear_number=0, run_type=<DagRunType.MANUAL: 'manual'>, conf={}, consumed_asset_events=[]), 'triggering_asset_events': TriggeringAssetEventsAccessor(_events=defaultdict(<class 'list'>, {})), 'task_instance_key_str': 'my_dag__my_task__20250708', 'task_reschedule_count': 0, 'prev_start_date_success': <Proxy at 0xffff6cb48cd0 with factory <function RuntimeTaskInstance.get_template_context.<locals>.<lambda> at 0xffff6c494280>>, 'prev_end_date_success': <Proxy at 0xffff6cb48d30 with factory <function RuntimeTaskInstance.get_template_context.<locals>.<lambda> at 0xffff6c8e5120>>, 'logical_date': DateTime(2025, 7, 8, 22, 44, 35, 842000, tzinfo=Timezone('UTC')), 'ds': '2025-07-08', 'ds_nodash': '20250708', 'ts': '2025-07-08T22:44:35.842000+00:00', 'ts_nodash': '20250708T224435', 'ts_nodash_with_tz': '20250708T224435.842000+0000', 'data_interval_end': DateTime(2025, 7, 8, 22, 44, 35, 842000, tzinfo=Timezone('UTC')), 'data_int
 erval_start': DateTime(2025, 7, 8, 22, 44, 35, 842000, tzinfo=Timezone('UTC')), 'prev_data_interval_start_success': <Proxy at 0xffff6cb48dc0 with factory <function RuntimeTaskInstance.get_template_context.<locals>.<lambda> at 0xffff6c8e51b0>>, 'prev_data_interval_end_success': <Proxy at 0xffff6cb48df0 with factory <function RuntimeTaskInstance.get_template_context.<locals>.<lambda> at 0xffff6c8e5240>>}","chan":"stdout","logger":"processor"}
   {"timestamp":"2025-07-08T22:48:31.568235Z","level":"info","event":"From inside the callback","chan":"stdout","logger":"processor"}
   {"timestamp":"2025-07-08T22:48:31.572041Z","level":"info","event":"{'dag': <DAG: my_dag>, 'inlets': [], 'map_index_template': None, 'outlets': [], 'run_id': 'manual__2025-07-08T22:44:37.346022+00:00', 'task': <Task(PythonOperator): my_task>, 'task_instance': RuntimeTaskInstance(id=UUID('0197ec36-656a-75b7-814a-c409886945e2'), task_id='my_task', dag_id='my_dag', run_id='manual__2025-07-08T22:44:37.346022+00:00', try_number=2, dag_version_id=UUID('0197ec0c-669e-7e05-80e3-97c3549dbfda'), map_index=-1, hostname='c9bd6c415dbf', context_carrier={}, task=<Task(PythonOperator): my_task>, max_tries=0, end_date=None, state=None, is_mapped=None, rendered_map_index=None, log_url=None), 'ti': RuntimeTaskInstance(id=UUID('0197ec36-656a-75b7-814a-c409886945e2'), task_id='my_task', dag_id='my_dag', run_id='manual__2025-07-08T22:44:37.346022+00:00', try_number=2, dag_version_id=UUID('0197ec0c-669e-7e05-80e3-97c3549dbfda'), map_index=-1, hostname='c9bd6c415dbf', context_carrier={}, task=<Task(Pytho
 nOperator): my_task>, max_tries=0, end_date=None, state=None, is_mapped=None, rendered_map_index=None, log_url=None), 'outlet_events': <airflow.sdk.execution_time.context.OutletEventAccessors object at 0xffff6cb48e20>, 'inlet_events': InletEventsAccessors(_inlets=[], _assets={}, _asset_aliases={}), 'macros': <MacrosAccessor (dynamic access to macros)>, 'params': {}, 'var': {'json': <VariableAccessor (dynamic access)>, 'value': <VariableAccessor (dynamic access)>}, 'conn': <ConnectionAccessor (dynamic access)>, 'dag_run': DagRun(dag_id='my_dag', run_id='manual__2025-07-08T22:44:37.346022+00:00', logical_date=datetime.datetime(2025, 7, 8, 22, 44, 35, 842000, tzinfo=Timezone('UTC')), data_interval_start=datetime.datetime(2025, 7, 8, 22, 44, 35, 842000, tzinfo=Timezone('UTC')), data_interval_end=datetime.datetime(2025, 7, 8, 22, 44, 35, 842000, tzinfo=Timezone('UTC')), run_after=datetime.datetime(2025, 7, 8, 22, 44, 35, 842000, tzinfo=Timezone('UTC')), start_date=datetime.datetime(2025,
  7, 8, 22, 44, 44, 227565, tzinfo=Timezone('UTC')), end_date=None, clear_number=0, run_type=<DagRunType.MANUAL: 'manual'>, conf={}, consumed_asset_events=[]), 'triggering_asset_events': TriggeringAssetEventsAccessor(_events=defaultdict(<class 'list'>, {})), 'task_instance_key_str': 'my_dag__my_task__20250708', 'task_reschedule_count': 0, 'prev_start_date_success': <Proxy at 0xffff6cb48ca0 with factory <function RuntimeTaskInstance.get_template_context.<locals>.<lambda> at 0xffff6c494280>>, 'prev_end_date_success': <Proxy at 0xffff6cb48c70 with factory <function RuntimeTaskInstance.get_template_context.<locals>.<lambda> at 0xffff6c8e52d0>>, 'logical_date': DateTime(2025, 7, 8, 22, 44, 35, 842000, tzinfo=Timezone('UTC')), 'ds': '2025-07-08', 'ds_nodash': '20250708', 'ts': '2025-07-08T22:44:35.842000+00:00', 'ts_nodash': '20250708T224435', 'ts_nodash_with_tz': '20250708T224435.842000+0000', 'data_interval_end': DateTime(2025, 7, 8, 22, 44, 35, 842000, tzinfo=Timezone('UTC')), 'data_int
 erval_start': DateTime(2025, 7, 8, 22, 44, 35, 842000, tzinfo=Timezone('UTC')), 'prev_data_interval_start_success': <Proxy at 0xffff6cb488e0 with factory <function RuntimeTaskInstance.get_template_context.<locals>.<lambda> at 0xffff6c8e5240>>, 'prev_data_interval_end_success': <Proxy at 0xffff6cb48880 with factory <function RuntimeTaskInstance.get_template_context.<locals>.<lambda> at 0xffff6c8e51b0>>}","chan":"stdout","logger":"processor"}
   ```
   
   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Run Task failure callbacks on DAG Processor when task is externally killed [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #53058:
URL: https://github.com/apache/airflow/pull/53058#issuecomment-3052565368

   ### Backport failed to create: v3-0-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/16169821259'> Run details </a>
   
   <table>
       <tr>
           <th>Status</th>
           <th>Branch</th>
           <th>Result</th>
       </tr>
       <tr>
           <td>❌</td>
           <td>v3-0-test</td>
           <td><a href="https://github.com/apache/airflow/commit/a5211f2efd5ccc565cbc16baee6144dba09918bc"><img src='https://img.shields.io/badge/Commit-a5211f2-red' alt='Commit Link'></a></td>
       </tr>
   </table>
   
   You can attempt to backport this manually by running:
   
   ```bash
   cherry_picker a5211f2 v3-0-test
   ```
   
   This should apply the commit to the v3-0-test branch and leave the commit in conflict state marking
   the files that need manual conflict resolution.
   
   After you have resolved the conflicts, you can continue the backport process by running:
   
   ```bash
   cherry_picker --continue
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Run Task failure callbacks on DAG Processor when task is externally killed [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #53058:
URL: https://github.com/apache/airflow/pull/53058#discussion_r2194672814


##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -927,10 +928,13 @@ def process_executor_events(
                         bundle_version=ti.dag_version.bundle_version,
                         ti=ti,
                         msg=msg,
+                        context_from_server=TIRunContext(
+                            dag_run=ti.dag_run,
+                            max_tries=ti.max_tries,
+                        ),
                     )
                     executor.send_callback(request)
-                else:
-                    ti.handle_failure(error=msg, session=session)
+                ti.handle_failure(error=msg, session=session)

Review Comment:
   It is intentional actually and also necessary. The code before didn't cater for the fact that the dag processor is no longer part of Scheduler and that the dag processor can no longer set the Task State.
   
   Without it tasks with callbacks only sent notifications but never called `ti.handle_failure()` to update their state (FAILED/RETRY, end_date, etc.) This separates the callback execution and state management.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Run Task failure callbacks on DAG Processor when task is externally k… [airflow]

Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #53058:
URL: https://github.com/apache/airflow/pull/53058#discussion_r2193907922


##########
airflow-core/tests/unit/dag_processing/test_processor.py:
##########
@@ -571,15 +536,284 @@ def fake_collect_dags(self, *args, **kwargs):
 
     spy_agency.spy_on(DagBag.collect_dags, call_fake=fake_collect_dags, owner=DagBag)
 
+    # Create a minimal TaskInstance for the request
+    ti_data = TIDataModel(
+        id=uuid.uuid4(),
+        dag_id="a",
+        task_id="b",
+        run_id="test_run",
+        map_index=-1,
+        try_number=1,
+        dag_version_id=uuid.uuid4(),
+    )
+
     requests = [
         TaskCallbackRequest(
             filepath="A",
             msg="Message",
-            ti=None,
+            ti=ti_data,
             bundle_name="testing",
             bundle_version=None,
         )
     ]
-    _parse_file(DagFileParseRequest(file="A", callback_requests=requests), log=structlog.get_logger())
+    _parse_file(
+        DagFileParseRequest(file="A", bundle_path="test", callback_requests=requests),
+        log=structlog.get_logger(),
+    )
 
     assert called is True
+
+
+class TestExecuteTaskCallbacks:
+    """Test the _execute_task_callbacks function"""
+
+    def test_execute_task_callbacks_failure_callback(self, spy_agency):
+        """Test _execute_task_callbacks executes failure callbacks"""
+        called = False
+        context_received = None
+
+        def on_failure(context):
+            nonlocal called, context_received
+            called = True
+            context_received = context
+
+        with DAG(dag_id="test_dag") as dag:
+            _ = BaseOperator(task_id="test_task", on_failure_callback=on_failure)
+
+        def fake_collect_dags(self, *args, **kwargs):
+            self.dags[dag.dag_id] = dag
+
+        spy_agency.spy_on(DagBag.collect_dags, call_fake=fake_collect_dags, owner=DagBag)
+
+        dagbag = DagBag()
+        dagbag.collect_dags()
+
+        ti_data = TIDataModel(
+            id=uuid.uuid4(),
+            dag_id="test_dag",
+            task_id="test_task",
+            run_id="test_run",
+            try_number=1,
+            dag_version_id=uuid.uuid4(),
+        )
+
+        request = TaskCallbackRequest(
+            filepath="test.py",
+            msg="Task failed",
+            ti=ti_data,
+            bundle_name="testing",
+            bundle_version=None,
+            task_callback_type=TaskInstanceState.FAILED,
+        )
+
+        log = structlog.get_logger()
+        _execute_task_callbacks(dagbag, request, log)
+
+        assert called is True
+        assert context_received is not None
+        assert context_received["dag"] == dag
+        assert "ti" in context_received
+
+    def test_execute_task_callbacks_retry_callback(self, spy_agency):
+        """Test _execute_task_callbacks executes retry callbacks"""
+        called = False
+        context_received = None
+
+        def on_retry(context):
+            nonlocal called, context_received
+            called = True
+            context_received = context
+
+        with DAG(dag_id="test_dag") as dag:
+            _ = BaseOperator(task_id="test_task", on_retry_callback=on_retry)
+
+        def fake_collect_dags(self, *args, **kwargs):
+            self.dags[dag.dag_id] = dag
+
+        spy_agency.spy_on(DagBag.collect_dags, call_fake=fake_collect_dags, owner=DagBag)
+
+        dagbag = DagBag()
+        dagbag.collect_dags()
+
+        ti_data = TIDataModel(
+            id=uuid.uuid4(),
+            dag_id="test_dag",
+            task_id="test_task",
+            run_id="test_run",
+            map_index=-1,
+            try_number=1,
+            dag_version_id=uuid.uuid4(),
+            state=TaskInstanceState.UP_FOR_RETRY,
+        )
+
+        request = TaskCallbackRequest(
+            filepath="test.py",
+            msg="Task retrying",
+            ti=ti_data,
+            bundle_name="testing",
+            bundle_version=None,
+            task_callback_type=TaskInstanceState.UP_FOR_RETRY,
+        )
+
+        log = structlog.get_logger()
+        _execute_task_callbacks(dagbag, request, log)
+
+        assert called is True
+        assert context_received is not None
+        assert context_received["dag"] == dag
+        assert "ti" in context_received
+
+    def test_execute_task_callbacks_with_context_from_server(self, spy_agency):
+        """Test _execute_task_callbacks with context_from_server creates full context"""
+        called = False
+        context_received = None
+
+        def on_failure(context):
+            nonlocal called, context_received
+            called = True
+            context_received = context
+
+        with DAG(dag_id="test_dag") as dag:
+            _ = BaseOperator(task_id="test_task", on_failure_callback=on_failure)
+
+        def fake_collect_dags(self, *args, **kwargs):
+            self.dags[dag.dag_id] = dag
+
+        spy_agency.spy_on(DagBag.collect_dags, call_fake=fake_collect_dags, owner=DagBag)
+
+        dagbag = DagBag()
+        dagbag.collect_dags()
+
+        # Create a mock DagRun

Review Comment:
   ```suggestion
   ```



##########
airflow-core/tests/unit/dag_processing/test_processor.py:
##########
@@ -571,15 +536,284 @@ def fake_collect_dags(self, *args, **kwargs):
 
     spy_agency.spy_on(DagBag.collect_dags, call_fake=fake_collect_dags, owner=DagBag)
 
+    # Create a minimal TaskInstance for the request
+    ti_data = TIDataModel(
+        id=uuid.uuid4(),
+        dag_id="a",
+        task_id="b",
+        run_id="test_run",
+        map_index=-1,
+        try_number=1,
+        dag_version_id=uuid.uuid4(),
+    )
+
     requests = [
         TaskCallbackRequest(
             filepath="A",
             msg="Message",
-            ti=None,
+            ti=ti_data,
             bundle_name="testing",
             bundle_version=None,
         )
     ]
-    _parse_file(DagFileParseRequest(file="A", callback_requests=requests), log=structlog.get_logger())
+    _parse_file(
+        DagFileParseRequest(file="A", bundle_path="test", callback_requests=requests),
+        log=structlog.get_logger(),
+    )
 
     assert called is True
+
+
+class TestExecuteTaskCallbacks:
+    """Test the _execute_task_callbacks function"""
+
+    def test_execute_task_callbacks_failure_callback(self, spy_agency):
+        """Test _execute_task_callbacks executes failure callbacks"""
+        called = False
+        context_received = None
+
+        def on_failure(context):
+            nonlocal called, context_received
+            called = True
+            context_received = context
+
+        with DAG(dag_id="test_dag") as dag:
+            _ = BaseOperator(task_id="test_task", on_failure_callback=on_failure)
+
+        def fake_collect_dags(self, *args, **kwargs):
+            self.dags[dag.dag_id] = dag
+
+        spy_agency.spy_on(DagBag.collect_dags, call_fake=fake_collect_dags, owner=DagBag)
+
+        dagbag = DagBag()
+        dagbag.collect_dags()
+
+        ti_data = TIDataModel(
+            id=uuid.uuid4(),
+            dag_id="test_dag",
+            task_id="test_task",
+            run_id="test_run",
+            try_number=1,
+            dag_version_id=uuid.uuid4(),
+        )
+
+        request = TaskCallbackRequest(
+            filepath="test.py",
+            msg="Task failed",
+            ti=ti_data,
+            bundle_name="testing",
+            bundle_version=None,
+            task_callback_type=TaskInstanceState.FAILED,
+        )
+
+        log = structlog.get_logger()
+        _execute_task_callbacks(dagbag, request, log)
+
+        assert called is True
+        assert context_received is not None
+        assert context_received["dag"] == dag
+        assert "ti" in context_received
+
+    def test_execute_task_callbacks_retry_callback(self, spy_agency):
+        """Test _execute_task_callbacks executes retry callbacks"""
+        called = False
+        context_received = None
+
+        def on_retry(context):
+            nonlocal called, context_received
+            called = True
+            context_received = context
+
+        with DAG(dag_id="test_dag") as dag:
+            _ = BaseOperator(task_id="test_task", on_retry_callback=on_retry)

Review Comment:
   ```suggestion
               BaseOperator(task_id="test_task", on_retry_callback=on_retry)
   ```



##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -927,10 +928,13 @@ def process_executor_events(
                         bundle_version=ti.dag_version.bundle_version,
                         ti=ti,
                         msg=msg,
+                        context_from_server=TIRunContext(
+                            dag_run=ti.dag_run,
+                            max_tries=ti.max_tries,
+                        ),
                     )
                     executor.send_callback(request)
-                else:
-                    ti.handle_failure(error=msg, session=session)
+                ti.handle_failure(error=msg, session=session)

Review Comment:
   Was this intentional? Don't think so?



##########
airflow-core/tests/unit/dag_processing/test_processor.py:
##########
@@ -571,15 +536,284 @@ def fake_collect_dags(self, *args, **kwargs):
 
     spy_agency.spy_on(DagBag.collect_dags, call_fake=fake_collect_dags, owner=DagBag)
 
+    # Create a minimal TaskInstance for the request
+    ti_data = TIDataModel(
+        id=uuid.uuid4(),
+        dag_id="a",
+        task_id="b",
+        run_id="test_run",
+        map_index=-1,
+        try_number=1,
+        dag_version_id=uuid.uuid4(),
+    )
+
     requests = [
         TaskCallbackRequest(
             filepath="A",
             msg="Message",
-            ti=None,
+            ti=ti_data,
             bundle_name="testing",
             bundle_version=None,
         )
     ]
-    _parse_file(DagFileParseRequest(file="A", callback_requests=requests), log=structlog.get_logger())
+    _parse_file(
+        DagFileParseRequest(file="A", bundle_path="test", callback_requests=requests),
+        log=structlog.get_logger(),
+    )
 
     assert called is True
+
+
+class TestExecuteTaskCallbacks:
+    """Test the _execute_task_callbacks function"""
+
+    def test_execute_task_callbacks_failure_callback(self, spy_agency):
+        """Test _execute_task_callbacks executes failure callbacks"""
+        called = False
+        context_received = None
+
+        def on_failure(context):
+            nonlocal called, context_received
+            called = True
+            context_received = context
+
+        with DAG(dag_id="test_dag") as dag:
+            _ = BaseOperator(task_id="test_task", on_failure_callback=on_failure)
+
+        def fake_collect_dags(self, *args, **kwargs):
+            self.dags[dag.dag_id] = dag
+
+        spy_agency.spy_on(DagBag.collect_dags, call_fake=fake_collect_dags, owner=DagBag)
+
+        dagbag = DagBag()
+        dagbag.collect_dags()
+
+        ti_data = TIDataModel(
+            id=uuid.uuid4(),
+            dag_id="test_dag",
+            task_id="test_task",
+            run_id="test_run",
+            try_number=1,
+            dag_version_id=uuid.uuid4(),
+        )
+
+        request = TaskCallbackRequest(
+            filepath="test.py",
+            msg="Task failed",
+            ti=ti_data,
+            bundle_name="testing",
+            bundle_version=None,
+            task_callback_type=TaskInstanceState.FAILED,
+        )
+
+        log = structlog.get_logger()
+        _execute_task_callbacks(dagbag, request, log)
+
+        assert called is True
+        assert context_received is not None
+        assert context_received["dag"] == dag
+        assert "ti" in context_received
+
+    def test_execute_task_callbacks_retry_callback(self, spy_agency):
+        """Test _execute_task_callbacks executes retry callbacks"""
+        called = False
+        context_received = None
+
+        def on_retry(context):
+            nonlocal called, context_received
+            called = True
+            context_received = context
+
+        with DAG(dag_id="test_dag") as dag:
+            _ = BaseOperator(task_id="test_task", on_retry_callback=on_retry)
+
+        def fake_collect_dags(self, *args, **kwargs):
+            self.dags[dag.dag_id] = dag
+
+        spy_agency.spy_on(DagBag.collect_dags, call_fake=fake_collect_dags, owner=DagBag)
+
+        dagbag = DagBag()
+        dagbag.collect_dags()
+
+        ti_data = TIDataModel(
+            id=uuid.uuid4(),
+            dag_id="test_dag",
+            task_id="test_task",
+            run_id="test_run",
+            map_index=-1,
+            try_number=1,
+            dag_version_id=uuid.uuid4(),
+            state=TaskInstanceState.UP_FOR_RETRY,
+        )
+
+        request = TaskCallbackRequest(
+            filepath="test.py",
+            msg="Task retrying",
+            ti=ti_data,
+            bundle_name="testing",
+            bundle_version=None,
+            task_callback_type=TaskInstanceState.UP_FOR_RETRY,
+        )
+
+        log = structlog.get_logger()
+        _execute_task_callbacks(dagbag, request, log)
+
+        assert called is True
+        assert context_received is not None
+        assert context_received["dag"] == dag
+        assert "ti" in context_received
+
+    def test_execute_task_callbacks_with_context_from_server(self, spy_agency):
+        """Test _execute_task_callbacks with context_from_server creates full context"""
+        called = False
+        context_received = None
+
+        def on_failure(context):
+            nonlocal called, context_received
+            called = True
+            context_received = context
+
+        with DAG(dag_id="test_dag") as dag:
+            _ = BaseOperator(task_id="test_task", on_failure_callback=on_failure)
+
+        def fake_collect_dags(self, *args, **kwargs):
+            self.dags[dag.dag_id] = dag
+
+        spy_agency.spy_on(DagBag.collect_dags, call_fake=fake_collect_dags, owner=DagBag)
+
+        dagbag = DagBag()
+        dagbag.collect_dags()
+
+        # Create a mock DagRun
+        dag_run = DagRun(
+            dag_id="test_dag",
+            run_id="test_run",
+            logical_date=timezone.utcnow(),
+            start_date=timezone.utcnow(),
+            run_type="manual",
+        )
+        dag_run.run_after = timezone.utcnow()
+
+        ti_data = TIDataModel(
+            id=uuid.uuid4(),
+            dag_id="test_dag",
+            task_id="test_task",
+            run_id="test_run",
+            try_number=1,
+            dag_version_id=uuid.uuid4(),
+        )
+
+        context_from_server = TIRunContext(
+            dag_run=dag_run,
+            max_tries=3,
+        )
+
+        request = TaskCallbackRequest(
+            filepath="test.py",
+            msg="Task failed",
+            ti=ti_data,
+            bundle_name="testing",
+            bundle_version=None,
+            task_callback_type=TaskInstanceState.FAILED,
+            context_from_server=context_from_server,
+        )
+
+        log = structlog.get_logger()
+        _execute_task_callbacks(dagbag, request, log)
+
+        assert called is True
+        assert context_received is not None
+        # When context_from_server is provided, we get a full RuntimeTaskInstance context
+        assert "dag_run" in context_received
+        assert "logical_date" in context_received
+
+    def test_execute_task_callbacks_not_failure_callback(self, spy_agency):
+        """Test _execute_task_callbacks when request is not a failure callback"""
+        called = False
+
+        def on_failure(context):
+            nonlocal called
+            called = True
+
+        with DAG(dag_id="test_dag") as dag:
+            _ = BaseOperator(task_id="test_task", on_failure_callback=on_failure)
+
+        def fake_collect_dags(self, *args, **kwargs):
+            self.dags[dag.dag_id] = dag
+
+        spy_agency.spy_on(DagBag.collect_dags, call_fake=fake_collect_dags, owner=DagBag)
+
+        dagbag = DagBag()
+        dagbag.collect_dags()
+
+        ti_data = TIDataModel(
+            id=uuid.uuid4(),
+            dag_id="test_dag",
+            task_id="test_task",
+            run_id="test_run",
+            try_number=1,
+            dag_version_id=uuid.uuid4(),
+            state=TaskInstanceState.SUCCESS,
+        )
+
+        request = TaskCallbackRequest(
+            filepath="test.py",
+            msg="Task succeeded",
+            ti=ti_data,
+            bundle_name="testing",
+            bundle_version=None,
+            task_callback_type=TaskInstanceState.SUCCESS,
+        )
+
+        log = structlog.get_logger()
+        _execute_task_callbacks(dagbag, request, log)
+
+        # Should not call the callback since it's not a failure callback
+        assert called is False
+
+    def test_execute_task_callbacks_multiple_callbacks(self, spy_agency):
+        """Test _execute_task_callbacks with multiple callbacks"""
+        call_count = 0
+
+        def on_failure_1(context):
+            nonlocal call_count
+            call_count += 1
+
+        def on_failure_2(context):
+            nonlocal call_count
+            call_count += 1
+
+        with DAG(dag_id="test_dag") as dag:
+            _ = BaseOperator(task_id="test_task", on_failure_callback=[on_failure_1, on_failure_2])

Review Comment:
   ```suggestion
               BaseOperator(task_id="test_task", on_failure_callback=[on_failure_1, on_failure_2])
   ```



##########
airflow-core/tests/unit/jobs/test_scheduler_job.py:
##########
@@ -6621,6 +6625,75 @@ def test_activate_referenced_assets_with_multiple_conflict_asset_in_one_dag(
         for i in range(100):
             assert f"it's duplicate {i}" in dag_warning.message
 
+    def test_scheduler_passes_context_from_server_on_heartbeat_timeout(self, dag_maker, session):
+        """Test that scheduler passes context_from_server when handling heartbeat timeouts."""
+        with dag_maker(dag_id="test_dag", session=session):
+            _ = EmptyOperator(task_id="test_task")
+
+        dag_run = dag_maker.create_dagrun(run_id="test_run", state=DagRunState.RUNNING)
+
+        mock_executor = MagicMock()
+        scheduler_job = Job(executor=mock_executor)
+        self.job_runner = SchedulerJobRunner(scheduler_job)
+
+        # Create a task instance that appears to be running but hasn't heartbeat
+        ti = dag_run.get_task_instance(task_id="test_task")
+        ti.state = TaskInstanceState.RUNNING
+        ti.queued_by_job_id = scheduler_job.id
+        # Set last_heartbeat_at to a time that would trigger timeout
+        ti.last_heartbeat_at = timezone.utcnow() - timedelta(seconds=600)  # 10 minutes ago
+        session.merge(ti)
+        session.commit()
+
+        # Run the heartbeat timeout check
+        self.job_runner._find_and_purge_task_instances_without_heartbeats()
+
+        # Verify TaskCallbackRequest was created with context_from_server
+        mock_executor.send_callback.assert_called_once()
+        callback_request = mock_executor.send_callback.call_args[0][0]
+
+        assert isinstance(callback_request, TaskCallbackRequest)
+        assert callback_request.context_from_server is not None
+        assert callback_request.context_from_server.dag_run.logical_date == dag_run.logical_date
+        assert callback_request.context_from_server.max_tries == ti.max_tries
+
+    def test_scheduler_passes_context_from_server_on_task_failure(self, dag_maker, session):
+        """Test that scheduler passes context_from_server when handling task failures."""
+        with dag_maker(dag_id="test_dag", session=session):
+            _ = EmptyOperator(task_id="test_task", on_failure_callback=lambda: print("failure"))

Review Comment:
   ```suggestion
               EmptyOperator(task_id="test_task", on_failure_callback=lambda: print("failure"))
   ```



##########
airflow-core/tests/unit/jobs/test_scheduler_job.py:
##########
@@ -6621,6 +6625,75 @@ def test_activate_referenced_assets_with_multiple_conflict_asset_in_one_dag(
         for i in range(100):
             assert f"it's duplicate {i}" in dag_warning.message
 
+    def test_scheduler_passes_context_from_server_on_heartbeat_timeout(self, dag_maker, session):
+        """Test that scheduler passes context_from_server when handling heartbeat timeouts."""
+        with dag_maker(dag_id="test_dag", session=session):
+            _ = EmptyOperator(task_id="test_task")
+
+        dag_run = dag_maker.create_dagrun(run_id="test_run", state=DagRunState.RUNNING)
+
+        mock_executor = MagicMock()
+        scheduler_job = Job(executor=mock_executor)
+        self.job_runner = SchedulerJobRunner(scheduler_job)
+
+        # Create a task instance that appears to be running but hasn't heartbeat
+        ti = dag_run.get_task_instance(task_id="test_task")
+        ti.state = TaskInstanceState.RUNNING
+        ti.queued_by_job_id = scheduler_job.id
+        # Set last_heartbeat_at to a time that would trigger timeout
+        ti.last_heartbeat_at = timezone.utcnow() - timedelta(seconds=600)  # 10 minutes ago
+        session.merge(ti)
+        session.commit()
+
+        # Run the heartbeat timeout check
+        self.job_runner._find_and_purge_task_instances_without_heartbeats()
+
+        # Verify TaskCallbackRequest was created with context_from_server
+        mock_executor.send_callback.assert_called_once()
+        callback_request = mock_executor.send_callback.call_args[0][0]
+
+        assert isinstance(callback_request, TaskCallbackRequest)
+        assert callback_request.context_from_server is not None
+        assert callback_request.context_from_server.dag_run.logical_date == dag_run.logical_date
+        assert callback_request.context_from_server.max_tries == ti.max_tries
+
+    def test_scheduler_passes_context_from_server_on_task_failure(self, dag_maker, session):
+        """Test that scheduler passes context_from_server when handling task failures."""
+        with dag_maker(dag_id="test_dag", session=session):
+            _ = EmptyOperator(task_id="test_task", on_failure_callback=lambda: print("failure"))
+
+        dag_run = dag_maker.create_dagrun(run_id="test_run", state=DagRunState.RUNNING)
+
+        # Create a task instance that's running
+        ti = dag_run.get_task_instance(task_id="test_task")
+        ti.state = TaskInstanceState.RUNNING
+        # ti.queued_by_job_id = 90000

Review Comment:
   ```suggestion
   ```



##########
airflow-core/tests/unit/dag_processing/test_processor.py:
##########
@@ -571,15 +536,284 @@ def fake_collect_dags(self, *args, **kwargs):
 
     spy_agency.spy_on(DagBag.collect_dags, call_fake=fake_collect_dags, owner=DagBag)
 
+    # Create a minimal TaskInstance for the request
+    ti_data = TIDataModel(
+        id=uuid.uuid4(),
+        dag_id="a",
+        task_id="b",
+        run_id="test_run",
+        map_index=-1,
+        try_number=1,
+        dag_version_id=uuid.uuid4(),
+    )
+
     requests = [
         TaskCallbackRequest(
             filepath="A",
             msg="Message",
-            ti=None,
+            ti=ti_data,
             bundle_name="testing",
             bundle_version=None,
         )
     ]
-    _parse_file(DagFileParseRequest(file="A", callback_requests=requests), log=structlog.get_logger())
+    _parse_file(
+        DagFileParseRequest(file="A", bundle_path="test", callback_requests=requests),
+        log=structlog.get_logger(),
+    )
 
     assert called is True
+
+
+class TestExecuteTaskCallbacks:
+    """Test the _execute_task_callbacks function"""
+
+    def test_execute_task_callbacks_failure_callback(self, spy_agency):
+        """Test _execute_task_callbacks executes failure callbacks"""
+        called = False
+        context_received = None
+
+        def on_failure(context):
+            nonlocal called, context_received
+            called = True
+            context_received = context
+
+        with DAG(dag_id="test_dag") as dag:
+            _ = BaseOperator(task_id="test_task", on_failure_callback=on_failure)
+
+        def fake_collect_dags(self, *args, **kwargs):
+            self.dags[dag.dag_id] = dag
+
+        spy_agency.spy_on(DagBag.collect_dags, call_fake=fake_collect_dags, owner=DagBag)
+
+        dagbag = DagBag()
+        dagbag.collect_dags()
+
+        ti_data = TIDataModel(
+            id=uuid.uuid4(),
+            dag_id="test_dag",
+            task_id="test_task",
+            run_id="test_run",
+            try_number=1,
+            dag_version_id=uuid.uuid4(),
+        )
+
+        request = TaskCallbackRequest(
+            filepath="test.py",
+            msg="Task failed",
+            ti=ti_data,
+            bundle_name="testing",
+            bundle_version=None,
+            task_callback_type=TaskInstanceState.FAILED,
+        )
+
+        log = structlog.get_logger()
+        _execute_task_callbacks(dagbag, request, log)
+
+        assert called is True
+        assert context_received is not None
+        assert context_received["dag"] == dag
+        assert "ti" in context_received
+
+    def test_execute_task_callbacks_retry_callback(self, spy_agency):
+        """Test _execute_task_callbacks executes retry callbacks"""
+        called = False
+        context_received = None
+
+        def on_retry(context):
+            nonlocal called, context_received
+            called = True
+            context_received = context
+
+        with DAG(dag_id="test_dag") as dag:
+            _ = BaseOperator(task_id="test_task", on_retry_callback=on_retry)
+
+        def fake_collect_dags(self, *args, **kwargs):
+            self.dags[dag.dag_id] = dag
+
+        spy_agency.spy_on(DagBag.collect_dags, call_fake=fake_collect_dags, owner=DagBag)
+
+        dagbag = DagBag()
+        dagbag.collect_dags()
+
+        ti_data = TIDataModel(
+            id=uuid.uuid4(),
+            dag_id="test_dag",
+            task_id="test_task",
+            run_id="test_run",
+            map_index=-1,
+            try_number=1,
+            dag_version_id=uuid.uuid4(),
+            state=TaskInstanceState.UP_FOR_RETRY,
+        )
+
+        request = TaskCallbackRequest(
+            filepath="test.py",
+            msg="Task retrying",
+            ti=ti_data,
+            bundle_name="testing",
+            bundle_version=None,
+            task_callback_type=TaskInstanceState.UP_FOR_RETRY,
+        )
+
+        log = structlog.get_logger()
+        _execute_task_callbacks(dagbag, request, log)
+
+        assert called is True
+        assert context_received is not None
+        assert context_received["dag"] == dag
+        assert "ti" in context_received
+
+    def test_execute_task_callbacks_with_context_from_server(self, spy_agency):
+        """Test _execute_task_callbacks with context_from_server creates full context"""
+        called = False
+        context_received = None
+
+        def on_failure(context):
+            nonlocal called, context_received
+            called = True
+            context_received = context
+
+        with DAG(dag_id="test_dag") as dag:
+            _ = BaseOperator(task_id="test_task", on_failure_callback=on_failure)

Review Comment:
   ```suggestion
               BaseOperator(task_id="test_task", on_failure_callback=on_failure)
   ```



##########
airflow-core/tests/unit/dag_processing/test_processor.py:
##########
@@ -571,15 +536,284 @@ def fake_collect_dags(self, *args, **kwargs):
 
     spy_agency.spy_on(DagBag.collect_dags, call_fake=fake_collect_dags, owner=DagBag)
 
+    # Create a minimal TaskInstance for the request
+    ti_data = TIDataModel(
+        id=uuid.uuid4(),
+        dag_id="a",
+        task_id="b",
+        run_id="test_run",
+        map_index=-1,
+        try_number=1,
+        dag_version_id=uuid.uuid4(),
+    )
+
     requests = [
         TaskCallbackRequest(
             filepath="A",
             msg="Message",
-            ti=None,
+            ti=ti_data,
             bundle_name="testing",
             bundle_version=None,
         )
     ]
-    _parse_file(DagFileParseRequest(file="A", callback_requests=requests), log=structlog.get_logger())
+    _parse_file(
+        DagFileParseRequest(file="A", bundle_path="test", callback_requests=requests),
+        log=structlog.get_logger(),
+    )
 
     assert called is True
+
+
+class TestExecuteTaskCallbacks:
+    """Test the _execute_task_callbacks function"""
+
+    def test_execute_task_callbacks_failure_callback(self, spy_agency):
+        """Test _execute_task_callbacks executes failure callbacks"""
+        called = False
+        context_received = None
+
+        def on_failure(context):
+            nonlocal called, context_received
+            called = True
+            context_received = context
+
+        with DAG(dag_id="test_dag") as dag:
+            _ = BaseOperator(task_id="test_task", on_failure_callback=on_failure)

Review Comment:
   ```suggestion
               BaseOperator(task_id="test_task", on_failure_callback=on_failure)
   ```
   
   Shouldn't need this.



##########
airflow-core/tests/unit/jobs/test_scheduler_job.py:
##########
@@ -6621,6 +6625,75 @@ def test_activate_referenced_assets_with_multiple_conflict_asset_in_one_dag(
         for i in range(100):
             assert f"it's duplicate {i}" in dag_warning.message
 
+    def test_scheduler_passes_context_from_server_on_heartbeat_timeout(self, dag_maker, session):
+        """Test that scheduler passes context_from_server when handling heartbeat timeouts."""
+        with dag_maker(dag_id="test_dag", session=session):
+            _ = EmptyOperator(task_id="test_task")

Review Comment:
   ```suggestion
               EmptyOperator(task_id="test_task")
   ```



##########
airflow-core/tests/unit/jobs/test_scheduler_job.py:
##########
@@ -6621,6 +6625,75 @@ def test_activate_referenced_assets_with_multiple_conflict_asset_in_one_dag(
         for i in range(100):
             assert f"it's duplicate {i}" in dag_warning.message
 
+    def test_scheduler_passes_context_from_server_on_heartbeat_timeout(self, dag_maker, session):
+        """Test that scheduler passes context_from_server when handling heartbeat timeouts."""
+        with dag_maker(dag_id="test_dag", session=session):
+            _ = EmptyOperator(task_id="test_task")
+
+        dag_run = dag_maker.create_dagrun(run_id="test_run", state=DagRunState.RUNNING)
+
+        mock_executor = MagicMock()
+        scheduler_job = Job(executor=mock_executor)
+        self.job_runner = SchedulerJobRunner(scheduler_job)
+
+        # Create a task instance that appears to be running but hasn't heartbeat
+        ti = dag_run.get_task_instance(task_id="test_task")
+        ti.state = TaskInstanceState.RUNNING
+        ti.queued_by_job_id = scheduler_job.id
+        # Set last_heartbeat_at to a time that would trigger timeout
+        ti.last_heartbeat_at = timezone.utcnow() - timedelta(seconds=600)  # 10 minutes ago
+        session.merge(ti)
+        session.commit()
+
+        # Run the heartbeat timeout check
+        self.job_runner._find_and_purge_task_instances_without_heartbeats()
+
+        # Verify TaskCallbackRequest was created with context_from_server
+        mock_executor.send_callback.assert_called_once()
+        callback_request = mock_executor.send_callback.call_args[0][0]
+
+        assert isinstance(callback_request, TaskCallbackRequest)
+        assert callback_request.context_from_server is not None
+        assert callback_request.context_from_server.dag_run.logical_date == dag_run.logical_date
+        assert callback_request.context_from_server.max_tries == ti.max_tries
+
+    def test_scheduler_passes_context_from_server_on_task_failure(self, dag_maker, session):
+        """Test that scheduler passes context_from_server when handling task failures."""
+        with dag_maker(dag_id="test_dag", session=session):
+            _ = EmptyOperator(task_id="test_task", on_failure_callback=lambda: print("failure"))
+
+        dag_run = dag_maker.create_dagrun(run_id="test_run", state=DagRunState.RUNNING)
+
+        # Create a task instance that's running
+        ti = dag_run.get_task_instance(task_id="test_task")
+        ti.state = TaskInstanceState.RUNNING
+        # ti.queued_by_job_id = 90000
+        session.merge(ti)
+        session.commit()
+
+        # Mock the executor to simulate a task failure
+        mock_executor = MagicMock(spec=BaseExecutor)
+        mock_executor.has_task = mock.MagicMock(return_value=False)
+        scheduler_job = Job(executor=mock_executor)
+        self.job_runner = SchedulerJobRunner(scheduler_job)
+
+        # Simulate executor reporting task as failed
+        # from airflow.executors.base_executor import TaskInstanceStateType

Review Comment:
   ```suggestion
   ```



##########
airflow-core/tests/unit/dag_processing/test_processor.py:
##########
@@ -571,15 +536,284 @@ def fake_collect_dags(self, *args, **kwargs):
 
     spy_agency.spy_on(DagBag.collect_dags, call_fake=fake_collect_dags, owner=DagBag)
 
+    # Create a minimal TaskInstance for the request
+    ti_data = TIDataModel(
+        id=uuid.uuid4(),
+        dag_id="a",
+        task_id="b",
+        run_id="test_run",
+        map_index=-1,
+        try_number=1,
+        dag_version_id=uuid.uuid4(),
+    )
+
     requests = [
         TaskCallbackRequest(
             filepath="A",
             msg="Message",
-            ti=None,
+            ti=ti_data,
             bundle_name="testing",
             bundle_version=None,
         )
     ]
-    _parse_file(DagFileParseRequest(file="A", callback_requests=requests), log=structlog.get_logger())
+    _parse_file(
+        DagFileParseRequest(file="A", bundle_path="test", callback_requests=requests),
+        log=structlog.get_logger(),
+    )
 
     assert called is True
+
+
+class TestExecuteTaskCallbacks:
+    """Test the _execute_task_callbacks function"""
+
+    def test_execute_task_callbacks_failure_callback(self, spy_agency):
+        """Test _execute_task_callbacks executes failure callbacks"""
+        called = False
+        context_received = None
+
+        def on_failure(context):
+            nonlocal called, context_received
+            called = True
+            context_received = context
+
+        with DAG(dag_id="test_dag") as dag:
+            _ = BaseOperator(task_id="test_task", on_failure_callback=on_failure)
+
+        def fake_collect_dags(self, *args, **kwargs):
+            self.dags[dag.dag_id] = dag
+
+        spy_agency.spy_on(DagBag.collect_dags, call_fake=fake_collect_dags, owner=DagBag)
+
+        dagbag = DagBag()
+        dagbag.collect_dags()
+
+        ti_data = TIDataModel(
+            id=uuid.uuid4(),
+            dag_id="test_dag",
+            task_id="test_task",
+            run_id="test_run",
+            try_number=1,
+            dag_version_id=uuid.uuid4(),
+        )
+
+        request = TaskCallbackRequest(
+            filepath="test.py",
+            msg="Task failed",
+            ti=ti_data,
+            bundle_name="testing",
+            bundle_version=None,
+            task_callback_type=TaskInstanceState.FAILED,
+        )
+
+        log = structlog.get_logger()
+        _execute_task_callbacks(dagbag, request, log)
+
+        assert called is True
+        assert context_received is not None
+        assert context_received["dag"] == dag
+        assert "ti" in context_received
+
+    def test_execute_task_callbacks_retry_callback(self, spy_agency):
+        """Test _execute_task_callbacks executes retry callbacks"""
+        called = False
+        context_received = None
+
+        def on_retry(context):
+            nonlocal called, context_received
+            called = True
+            context_received = context
+
+        with DAG(dag_id="test_dag") as dag:
+            _ = BaseOperator(task_id="test_task", on_retry_callback=on_retry)
+
+        def fake_collect_dags(self, *args, **kwargs):
+            self.dags[dag.dag_id] = dag
+
+        spy_agency.spy_on(DagBag.collect_dags, call_fake=fake_collect_dags, owner=DagBag)
+
+        dagbag = DagBag()
+        dagbag.collect_dags()
+
+        ti_data = TIDataModel(
+            id=uuid.uuid4(),
+            dag_id="test_dag",
+            task_id="test_task",
+            run_id="test_run",
+            map_index=-1,
+            try_number=1,
+            dag_version_id=uuid.uuid4(),
+            state=TaskInstanceState.UP_FOR_RETRY,
+        )
+
+        request = TaskCallbackRequest(
+            filepath="test.py",
+            msg="Task retrying",
+            ti=ti_data,
+            bundle_name="testing",
+            bundle_version=None,
+            task_callback_type=TaskInstanceState.UP_FOR_RETRY,
+        )
+
+        log = structlog.get_logger()
+        _execute_task_callbacks(dagbag, request, log)
+
+        assert called is True
+        assert context_received is not None
+        assert context_received["dag"] == dag
+        assert "ti" in context_received
+
+    def test_execute_task_callbacks_with_context_from_server(self, spy_agency):
+        """Test _execute_task_callbacks with context_from_server creates full context"""
+        called = False
+        context_received = None
+
+        def on_failure(context):
+            nonlocal called, context_received
+            called = True
+            context_received = context
+
+        with DAG(dag_id="test_dag") as dag:
+            _ = BaseOperator(task_id="test_task", on_failure_callback=on_failure)
+
+        def fake_collect_dags(self, *args, **kwargs):
+            self.dags[dag.dag_id] = dag
+
+        spy_agency.spy_on(DagBag.collect_dags, call_fake=fake_collect_dags, owner=DagBag)
+
+        dagbag = DagBag()
+        dagbag.collect_dags()
+
+        # Create a mock DagRun
+        dag_run = DagRun(
+            dag_id="test_dag",
+            run_id="test_run",
+            logical_date=timezone.utcnow(),
+            start_date=timezone.utcnow(),
+            run_type="manual",
+        )
+        dag_run.run_after = timezone.utcnow()
+
+        ti_data = TIDataModel(
+            id=uuid.uuid4(),
+            dag_id="test_dag",
+            task_id="test_task",
+            run_id="test_run",
+            try_number=1,
+            dag_version_id=uuid.uuid4(),
+        )
+
+        context_from_server = TIRunContext(
+            dag_run=dag_run,
+            max_tries=3,
+        )
+
+        request = TaskCallbackRequest(
+            filepath="test.py",
+            msg="Task failed",
+            ti=ti_data,
+            bundle_name="testing",
+            bundle_version=None,
+            task_callback_type=TaskInstanceState.FAILED,
+            context_from_server=context_from_server,
+        )
+
+        log = structlog.get_logger()
+        _execute_task_callbacks(dagbag, request, log)
+
+        assert called is True
+        assert context_received is not None
+        # When context_from_server is provided, we get a full RuntimeTaskInstance context
+        assert "dag_run" in context_received
+        assert "logical_date" in context_received
+
+    def test_execute_task_callbacks_not_failure_callback(self, spy_agency):
+        """Test _execute_task_callbacks when request is not a failure callback"""
+        called = False
+
+        def on_failure(context):
+            nonlocal called
+            called = True
+
+        with DAG(dag_id="test_dag") as dag:
+            _ = BaseOperator(task_id="test_task", on_failure_callback=on_failure)

Review Comment:
   ```suggestion
               BaseOperator(task_id="test_task", on_failure_callback=on_failure)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Run Task failure callbacks on DAG Processor when task is externally killed [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #53058:
URL: https://github.com/apache/airflow/pull/53058#discussion_r2194231172


##########
airflow-core/src/airflow/dag_processing/processor.py:
##########
@@ -238,6 +238,66 @@ def _execute_dag_callbacks(dagbag: DagBag, request: DagCallbackRequest, log: Fil
             Stats.incr("dag.callback_exceptions", tags={"dag_id": request.dag_id})
 
 
+def _execute_task_callbacks(dagbag: DagBag, request: TaskCallbackRequest, log: FilteringBoundLogger) -> None:
+    if not request.is_failure_callback:
+        log.warning(
+            "Task callback requested but is not a failure callback",
+            dag_id=request.ti.dag_id,
+            task_id=request.ti.task_id,
+            run_id=request.ti.run_id,
+        )
+        return
+
+    dag = dagbag.dags[request.ti.dag_id]
+    task = dag.get_task(request.ti.task_id)
+
+    if request.task_callback_type is TaskInstanceState.UP_FOR_RETRY:
+        callbacks = task.on_retry_callback
+    else:
+        callbacks = task.on_failure_callback
+
+    if not callbacks:
+        log.warning(
+            "Callback requested but no callback found",
+            dag_id=request.ti.dag_id,
+            task_id=request.ti.task_id,
+            run_id=request.ti.run_id,
+        )
+        return
+
+    callbacks = callbacks if isinstance(callbacks, Sequence) else [callbacks]
+    ctx_from_server = request.context_from_server
+
+    if ctx_from_server is not None:
+        rti = RuntimeTaskInstance.model_construct(

Review Comment:
   ```suggestion
           runtime_ti = RuntimeTaskInstance.model_construct(
   ```
   
   nit



##########
task-sdk/src/airflow/sdk/api/datamodels/_generated.py:
##########
@@ -519,7 +519,7 @@ class TIRunContext(BaseModel):
     next_method: Annotated[str | None, Field(title="Next Method")] = None
     next_kwargs: Annotated[dict[str, Any] | str | None, Field(title="Next Kwargs")] = None
     xcom_keys_to_clear: Annotated[list[str] | None, Field(title="Xcom Keys To Clear")] = None
-    should_retry: Annotated[bool, Field(title="Should Retry")]
+    should_retry: Annotated[bool | None, Field(title="Should Retry")] = False

Review Comment:
   Hmm, this doesn't sound right, its always a boolean right?



##########
airflow-core/src/airflow/dag_processing/processor.py:
##########
@@ -238,6 +238,66 @@ def _execute_dag_callbacks(dagbag: DagBag, request: DagCallbackRequest, log: Fil
             Stats.incr("dag.callback_exceptions", tags={"dag_id": request.dag_id})
 
 
+def _execute_task_callbacks(dagbag: DagBag, request: TaskCallbackRequest, log: FilteringBoundLogger) -> None:
+    if not request.is_failure_callback:
+        log.warning(
+            "Task callback requested but is not a failure callback",
+            dag_id=request.ti.dag_id,
+            task_id=request.ti.task_id,
+            run_id=request.ti.run_id,
+        )
+        return
+
+    dag = dagbag.dags[request.ti.dag_id]
+    task = dag.get_task(request.ti.task_id)
+
+    if request.task_callback_type is TaskInstanceState.UP_FOR_RETRY:
+        callbacks = task.on_retry_callback
+    else:
+        callbacks = task.on_failure_callback
+
+    if not callbacks:
+        log.warning(
+            "Callback requested but no callback found",
+            dag_id=request.ti.dag_id,
+            task_id=request.ti.task_id,
+            run_id=request.ti.run_id,
+        )
+        return
+
+    callbacks = callbacks if isinstance(callbacks, Sequence) else [callbacks]
+    ctx_from_server = request.context_from_server
+
+    if ctx_from_server is not None:
+        rti = RuntimeTaskInstance.model_construct(
+            **request.ti.model_dump(exclude_unset=True),
+            task=task,
+            _ti_context_from_server=ctx_from_server,
+            max_tries=ctx_from_server.max_tries,
+        )
+    else:
+        rti = RuntimeTaskInstance.model_construct(
+            **request.ti.model_dump(exclude_unset=True),
+            task=task,
+        )
+    context = rti.get_template_context()
+
+    def get_callback_representation(callback):
+        with contextlib.suppress(AttributeError):
+            return callback.__name__
+        with contextlib.suppress(AttributeError):
+            return callback.__class__.__name__
+        return callback
+
+    for idx, callback in enumerate(callbacks):
+        callback_repr = get_callback_representation(callback)
+        log.info("Executing Task callback at index %d: %s", idx, callback_repr)

Review Comment:
   ```suggestion
          log.info("Executing task callback #%d: %s", idx, callback_repr)
   ```
   nit



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Run Task failure callbacks on DAG Processor when task is externally killed [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on PR #53058:
URL: https://github.com/apache/airflow/pull/53058#issuecomment-3057767118

   Backport PR: https://github.com/apache/airflow/pull/53143


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Run Task failure callbacks on DAG Processor when task is externally killed [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil merged PR #53058:
URL: https://github.com/apache/airflow/pull/53058


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Run Task failure callbacks on DAG Processor when task is externally killed [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #53058:
URL: https://github.com/apache/airflow/pull/53058#discussion_r2194662126


##########
task-sdk/src/airflow/sdk/api/datamodels/_generated.py:
##########
@@ -519,7 +519,7 @@ class TIRunContext(BaseModel):
     next_method: Annotated[str | None, Field(title="Next Method")] = None
     next_kwargs: Annotated[dict[str, Any] | str | None, Field(title="Next Kwargs")] = None
     xcom_keys_to_clear: Annotated[list[str] | None, Field(title="Xcom Keys To Clear")] = None
-    should_retry: Annotated[bool, Field(title="Should Retry")]
+    should_retry: Annotated[bool | None, Field(title="Should Retry")] = False

Review Comment:
   This is auto-generated and there are more cases where defaults have `None` type auto-added. Will handle that separately



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Run Task failure callbacks on DAG Processor when task is externally killed [airflow]

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #53058:
URL: https://github.com/apache/airflow/pull/53058#discussion_r2194800569


##########
airflow-core/src/airflow/dag_processing/processor.py:
##########
@@ -238,6 +238,66 @@ def _execute_dag_callbacks(dagbag: DagBag, request: DagCallbackRequest, log: Fil
             Stats.incr("dag.callback_exceptions", tags={"dag_id": request.dag_id})
 
 
+def _execute_task_callbacks(dagbag: DagBag, request: TaskCallbackRequest, log: FilteringBoundLogger) -> None:
+    if not request.is_failure_callback:
+        log.warning(
+            "Task callback requested but is not a failure callback",
+            dag_id=request.ti.dag_id,
+            task_id=request.ti.task_id,
+            run_id=request.ti.run_id,
+        )
+        return
+
+    dag = dagbag.dags[request.ti.dag_id]
+    task = dag.get_task(request.ti.task_id)
+
+    if request.task_callback_type is TaskInstanceState.UP_FOR_RETRY:
+        callbacks = task.on_retry_callback
+    else:
+        callbacks = task.on_failure_callback
+
+    if not callbacks:
+        log.warning(
+            "Callback requested but no callback found",
+            dag_id=request.ti.dag_id,
+            task_id=request.ti.task_id,
+            run_id=request.ti.run_id,

Review Comment:
   ```suggestion
               dag_id=request.ti.dag_id,
               task_id=request.ti.task_id,
               run_id=request.ti.run_id,
               ti_id=request.ti.id,
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org